We use freebsd jails and a lightweight in house orchestration tool written in Rust. We are running hundreds of Ryzen machines with 64 cores. Our costs compared to running equivalent on Amazon is so much less. We estimate our costs are about 6x lower than AWS and we have far better performance in terms of networking, CPU, and disk write speed.
Jails has been a pleasure to work with! We even dynamically scale up and down resources as needed.
We use bare metal machines on Interserver. But there a quite a few good data centers worth considering.
I stay quasi-anonymous on here, but I worked at a company that bursted -- at the time -- to approx 1.5m requests per second. Average traffic was approx 400k rps.
It was run on naked boxes in many pops worldwide. We heavily used the jvm, which dramatically simplifies application distribution. Boxes were all netboot; they came up, grabbed the OS, and based on some pretty simple config figured out what jars to download and run.
We also costed out a move to aws, and it was somewhere between 5 and 6x as expensive as buying our own hardware.
Thousands of boxes worldwide managed with a team of two fulltime sysadmins plus two more primarily engaged in dev with sysadmin duties as-needed, plus remote hands services in the various datacenters.
As a tangent to this, I highly recommend https://bloom.host for anyone looking for just a few cores worth of Ryzen performance. They’re primarily a gaming server host, but their general-purpose VPS are absolutely stellar, and the support is top-notch. I tested them out after they were recommended on another HN thread, and got a 3x speedup based on a few heavy postgres queries I benchmarked on a $20/mo Digital Ocean instance vs. a $20/mo Bloom instance.
Data center Ryzen really knocks it out of the park.
Jails has been a pleasure to work with! We even dynamically scale up and down resources as needed.
We use bare metal machines on Interserver. But there a quite a few good data centers worth considering.