Which the article mentions:

    ...due to its 40GB+ size.

and is visible in the screenshot: https://torrentfreak.com/images/hateful.png

Interestingly, that's less than a 50GB Blu-Ray, so tell me, why did we need a different disc format for UHD Blu-Ray?

UHD Blu-ray doesn't introduce new disc formats, but compatibility with the variant does require support for 3- and 4-layer discs (2-layer, 50GB, discs are also still supported for UHD). These aren't new for UHD, but they were introduced after the regular 1080p standards were set--nothing really stops a 1080p movie from using them too, except compatibility with very old players.

Still, as other commenters note, these large disc sizes accommodate for high quality. If we have a 100GB disc format, and you want the 4K video to appear in the best possible quality, you will choose some encoding options that fill up most of that 100GB disc, minus whatever the special features and audio tracks cost. Most consumers won't care about the size: it's on a disc, they're not ripping it to a hard drive of their own.

And a YIFY 1080p release is less than a 4.5GB DVD, why did we need a different disc format for regular Blu-Ray?

A VCD release fits on a 700MB CD, why did we need a different disc format for DVD?

You can make video files come out any size you like - even 1 MB - if you're willing to make the tradeoffs (on both quality and time-to-encode dimensions). There's also a limited amount of hardware you can put into a set-top player while keeping a consumer-friendly price point.

You could still sell the set-top player (which aren't really at a "low" price point) with H.265 as the codec on the current-gen Blu-Ray disc - reducing manufacturing costs.

Normally those kind of decisions are optimized by free market competition. It doesn't seem to have kicked-in in this case, probably due to the elephants in the room (piracy and streaming) having zero capex cost.

Hence the HD-DVD which packed the 1080p and 720p videos onto the older media standard.

HD-DVD was a separate type of physical disk. It could store more data than DVD but less than Bluray.

I suspect that the Hateful Eight compresses considerably better then most movies due to the amount of time outside in the endless snow. If it compresses 20% better then a movie of a similar length (like one of the LOTR), one of those movies could go over the 50GB size.

Because Hollywood has higher quality standards than pirates?

This is just false, passionate fans are absolutely going to have higher quality standards than some hollywood executives.

Even just glancing around the more specific private piracy sites you can see far more love and care placed on releases than most commercial releases.

Tell that to the pirate encoders who had to deband movies because the official releases were so bad.

In the early/mid 2000s there was a whole scene around manually applying inverse telecine to badly produced commercial anime releases, with custom tools like YATTA.

Don't get me started about pre-widescreen episodes of Family Guy. I don't know how they ruined them, but they're all ruined with horrible, burned-in interlacing artefacts.