Author here: This article was intended to explain some modern hardware bottlenecks (and non-bottlenecks), but unexpectedly ended up covering a bunch of Linux kernel I/O stack issues as well :-) AMA
I just love this article. Especially when the norm is always about scaling out instead of scaling up. We can have 128 Core CPU, 2TB Memory, PCI-E 4.0 SSD, ( and soon PCI-E 5.0 ). We could even fit a Petabyte in 1U for SSD Storage.
I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes. ( Only to be taken apart by Facebook )
So Thank you for raising awareness. Hopefully the pendulum is swinging back to conceptually simple design.
>I also have a 380 GB Intel Optane 905P SSD for low latency writes
I would love to see that. Although I am waiting for someone to do a review on the Optane SSD P5800X [1]. Random 4K IOPS up to 1.5M with lower than 6us Latency.
>> I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes.
With 1TB of RAM you can have 256 bytes for every person on earth live in memory. With SSD either as virtual memory or keeping an index in RAM, you can do meaningful work in real time, probably as fast as the network will allow.
When I first moved to the bay area, the company that hired me asked me what kind of computer I wanted and gave me a budget (like $3000 or something)... I spent a few days crafting a parts list so I could build an awesome workstation. Once I sent it over they were like "Uh, we just meant which macbook do you want?" and kind of gave me some shade about it. They joked, so how are you going to do meetings or on call?
I rolled with it, but really wondered if they knew I could get 2x the hardware and have a computer at home and at work for less money than the MBP ... Most of the people didnt seem to understand that laptop CPUs are not the same as desktop/workstation ones, especially when they hit thermal down throttling.
Last but one job boss offered me an iMac Pro, I asked if I could just have the equivalent money for hardware and he said sure.
Which is how I ended up with an absolute monster of a work machine, these days I WFH and while work issued me a Macbook Pro it sits on the shelf behind me.
Fedora on a (still fast) Ryzen/2080 and 2x4K 27" screens vs a Macbook Pro is a hilarious no brainer for me.
Upgrading soon but can't decide whether I need the 5950X or merely want it - realistically except for gaming I'm nowhere near tapping out this machine (and it's still awesome for that an VR which is why the step-son is about to get a in his words "sick" PC).
I mean it would have been a totally valid answer to say that you intended to use a $600 laptop as effectively a thin client, and spend $2400 on a powerful workstation PC to drive remotely.
Was a macbookpro user for a decade+ then dropped laptops for desktop machines, first imacpro, currently ryzen 12 core. I no longer understand why I had a laptop for so long. Status I guess (only talking about me)
You should look at cpu usage. There is a good chance all your interrupts are hitting cpu-0. you can run hwloc to see what chiplet the pci cards are on and handle interrupts on those cores.
Thanks for the "hwloc" tip. I hadn't thought about that.
I was thinking of doing something like that. Weirdly I got sustained throughput differences when I killed & restarted fio. So, if I got 11M IOPS, it stayed at that level until I killed fio & restarted. If I got 10.8M next, it stayed like it until I killed & restarted it.
This makes me think that I'm hitting some PCIe/memory bottleneck, dependent on process placement (which process happens to need to move data across infinity fabric due to accessing data through a "remote" PCIe root complex or something like that). But then I realized that Zen 2 has a central IO hub again, so there shouldn't be a "far edge of I/O" like on current gen Intel CPUs (?)
But there's definitely some workload placement and I/O-memory-interrupt affinity that I've wanted to look into. I could even enable the NUMA-like-mode from BIOS, but again with Zen 2, the memory access goes through the central infinity-fabric chip too, I understand, so not sure if there's any value in trying to achieve memory locality for individual chiplets on this platform (?)
So there are 2 parts to cpu affinity. a)
cpu assigned to ssd for handling interrupts and b) cpu assigned to fio. numactl is your friend for experimenting with with changing fio affinity.
The PCIe is all on a single IO die, but internally it is organized into quadrants that can produce some NUMA effects. So it is probably worth trying out the motherboard firmware settings to expose your CPU as multiple NUMA nodes, and using the FIO options to allocate memory only on the local node, and restricting execution to the right cores.
Yep, I enabled the "numa-like-awareness" in BIOS and ran a few quick tests to see whether the NUMA-aware scheduler/NUMA balancing would do the right thing and migrate processes closer to their memory over time, but didn't notice any benefit. But yep I haven't manually locked down the execution and memory placement yet. This placement may well explain why I saw some ~5% throughput fluctuations only if killing & restarting fio and not while the same test was running.
I have done some tests on AMD servers and I the Linux scheduler does a pretty good job.
I do however get noticeable (a couple percent) better performance by forcing the process to run on the correct numa node.
Make sure you get as many numa domains as possible in your BIOS settings.
I recommend using numactl with the cpu-exclusive and mem-exclusive flags. I have noticed a slight perfomance drop when RAM cache fills beyond the sticks local to the cpus doing work.
One last comment is that you mentioned interrupts being "stiped" among CPUs. I would recommend pinning the interrupts from one disk to one numa-local CPU and using numactl to run fio for that disk on the same CPU.
An additional experiment is to, if you have enough cores, pin interrupts to CPUs local to disk, but use other cores on the same numa node for fio. That has been my most successful setup so far.
I have the same box, but with the 32 core CPU and fewer NVMe drives. I've not poked at all the PCIe slots yet, but all that I've looked at are in NUMA node 1. This includes the on board M.2 slots. It is in NPS=4 mode.
Mine goes only up to 2 NUMA nodes (as shown in numactl --hardware), despite setting NPS4 in BIOS. I guess it's because I have only 2 x 8-core chiplets enabled (?)
I think that in addition to allocating a queue per CPU, you need to be able to allocate a MSI(-X) vector per CPU. That shouldn't be a problem for the Samsung 980 PRO, since it supports 128 queues and 130 interrupt vectors.
Good question. I don't ever read kernel code as a starting point, only if some profiling or tracing tool points me towards an interesting function or codepath. And interesting usually is something that takes most CPU in perf output or some function call with an unusually high latency in ftrace, bcc/bpftrace script output. Or just a stack trace in a core- or crashdump.
As far as mindset goes - I try to apply the developer mindset to system performance. In other words, I don't use much of what I call the "old school sysadmin mindset", from a time where better tooling was not available. I don't use systemwide utilization or various get/hit ratios for doing "metric voodoo" of Unix wizards.
The developer mindset dictates that everything you run is an application. JVM is an application. Kernel is an application. Postgres, Oracle are applications. All applications execute one or more threads that run on CPU or do not run on CPU. There are only two categories of reasons why a thread does not run on CPU (is sleeping): The OS put the thread to sleep (involuntary blocking) or the thread voluntarily wanted to go to go to sleep (for example, it realized it can't get some application level lock).
And you drill down from there. Your OS/system is just a bunch of threads running on CPU, sleeping and sometimes communicating with each other. You can directly measure all of these things easily nowadays with profilers, no need for metric voodoo.
I have written my own tools to complement things like perf, ftrace and BPF stuff - as a consultant I regularly see 10+ year old Linux versions, etc - and I find sampling thread states from /proc file system is a really good (and flexible) starting point for system performance analysis and even some drilldown - all this without having to install new software or upgrading to latest kernels. Some of the tools I showed in my article too:
Excellent article, thank you!
I really like the analysis and profiling part of the evaluation.
I also have some experience in I/O performance in linux -- we measured 30GiB/s in a pcie Gen3 box (shameless plug[0]).
I have one question / comment: did you use multiple jobs for the BW (large IO) experiments? If yes, then did you set randrepeat to 0? I'm asking this because fio by default uses the same sequence of offsets for each job, in which case there might be data re-used across jobs. I had verified that with blktrace a few years back, but it might have changed recently.
Looks interesting! I wonder whether there'd be interesting new database applications on NVMe when doing as small as 512 byte I/Os (with more efficient "IO engine" than Linux bio, that has too high CPU overhead with such small requests).
I mean, currently OLTP RDBMS engines tend to use 4k, 8k (and some) 16k block size and when doing completely random I/O (or, say traversing an index on customer_id that now needs to read random occasional customer orders across years of history). So you may end up reading 1000 x 8 kB blocks just to read 1000 x 100B order records "randomly" scattered across the table from inserts done over the years.
Optane persistent memory can do small, cache line sized I/O I understand, but that's a different topic. When being able to do random 512B I/O on "commodity" NVMe SSDs efficiently, this would open some interesting opportunities for retrieving records that are scattered "randomly" across the disks.
edit: to answer your question, I used 10 separate fio commands with numjobs=3 or 4 for each and randrepeat was set to default.
At Netflix, I'm playing with an EPYC 7502P with 16 NVME and dual 2x100 Mellanox ConnectX6-DX NICs. With hardware kTLS offload, we're able to serve about 350Gb/s of real customer traffic. This goes down to about 240Gb/s when using software kTLS, due to memory bandwidth limits.
>we're able to serve about 350Gb/s of real customer traffic.
I still remember the post about breaking 100Gbps barrier, that was may be in 2016 or 17 ? And wasn't that long ago it was 200Gbps and if I remember correct it was hitting memory bandwidth barrier as well.
And now 350Gbps?!
So what's next? Wait for DDR5? Or moving to some memory controller black magic like POWER10?
Yes, before hardware inline kTLS offload, we were limited to 200Gb/s or so with Naples. With Rome, its a bit higher. But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck.
The current bottleneck is IO related, and its unclear what the issue is. We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s
> But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck.
For a while now I had operated under the assumption that CPU-based crypto with AES-GCM was faster than most hardware offload cards. What makes the Mellanox NIC perform better?
I.e.: Why does memory bandwidth matter to TLS? Aren't you encrypting data "on the fly", while it is still resident in the CPU caches?
> We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s
Something I explained to a colleague recently is that a modern CPU gains or loses more computer power from a 1° C temperature difference in the room's air than my first four computers had combined.
You're basically complaining that you're unable to get a mere 10% of the expected throughput. But put in absolute terms, that's 40 Gbps, which is about 10x more than what a typical server in 2020 can put out on the network. (Just because you have 10 Gbps NICs doesn't mean you can get 10 Gbps! Try iperf3 and you'll be shocked that you're lucky if you can crack 5 Gbps in practice)
> For a while now I had operated under the assumption that CPU-based crypto with AES-GCM was faster than most hardware offload cards. What makes the Mellanox NIC perform better?
> I.e.: Why does memory bandwidth matter to TLS? Aren't you encrypting data "on the fly", while it is still resident in the CPU caches?
It may depend on what you're sending. Netflix's use case is generally sending files. If you're doing software encryption you would load the plain text file into memory (via the filesystem/unified buffer cache), then write the (session specific) encrypted text into separate memory, then tell give that memory to the NIC to send out.
If the NIC can do the encryption, you would load the plain text into memory, then tell the NIC to read from that memory to encrypt and send out. That saves at least a write pass, and probably a read pass. (256 MB of L3 cache on latest EPYC is a lot, but it's not enough to expect cached reads from the filesystem to hit L3 that often, IMHO)
If my guestimate is right, a cold file would go from hitting memory 4 times to hitting it twice. And a file in disk cache would go from 3 times to once; the CPU doesn't need to touch the memory if it's in the disk cache.
Not that this is a totally different case from encrypting dynamic data that's necessarily touched by the CPU.
> You're basically complaining that you're unable to get a mere 10% of the expected throughput. But put in absolute terms, that's 40 Gbps, which is about 10x more than what a typical server in 2020 can put out on the network. (Just because you have 10 Gbps NICs doesn't mean you can get 10 Gbps! Try iperf3 and you'll be shocked that you're lucky if you can crack 5 Gbps in practice)
I had no problem serving 10 Gbps of files on a dual Xeon E5-2690 (v1; a 2012 CPU), although that CPU isn't great at AES, so I think it only did 8 Gbps or so with TLS; the next round of servers for that role had 2x 10G and 2690 v3 or v4 (2014 or 2016; but I can't remember when we got them) and thanks to better AES instructions, they were able to do 20 G (and a lot more handshakes/sec too). If your 2020 servers aren't as good as my circa 2012 servers were, you might need to work on your stack. OTOH, bulk file serving for many clients can be different than a single connection iperf.
> If my guestimate is right, a cold file would go from hitting memory 4 times to hitting it twice. And a file in disk cache would go from 3 times to once; the CPU doesn't need to touch the memory if it's in the disk cache.
> I.e.: Why does memory bandwidth matter to TLS? Aren't you encrypting data "on the fly", while it is still resident in the CPU caches?
I assume NF's software pipeline is zero copy, so if TLS is done in the NIC data only gets read from memory once when it is DMA'd to the NIC. With software TLS you need to read the data from memory (assuming it's not already in cache, which given the size of data NF deals with is unlikely), encrypt it, then write it back out to main memory so it can be DMA'd to the NIC. I know Intel has some fancy tech that can DMA directly to/from the CPU's cache, but I don't think AMD has that capability (yet).
> Try iperf3 and you'll be shocked that you're lucky if you can crack 5 Gbps in practice
Easy line rate if you crank the MTU all the way to 9000 :D
> modern CPU gains or loses more computer power from a 1° C temperature difference in the room's air
If you're using the boost algorithm rather than a static overclock, and when that boost is thermally limited rather than current limited. With a good cooler it's not too hard to always have thermal headroom.
> Easy line rate if you crank the MTU all the way to 9000 :D
In my experience jumbo frames provide at best an improvement of about 20% in rare cases, such as ping-pong UDP protocols such as TFTP or Citrix PVS streaming.
Which NICs would you recommend for me to buy for testing at least 1x100 Gbps (ideally 200 Gbps?) networking between this machine (PCIe 4.0) and an Intel Xeon one that I have with PCIe 3.0. Don't want to spend much money, so the cards don't need to be too enterprisey, just fast.
And - do such cards even allow direct "cross" connection without a switch in between?
Great article, I learned! Can you tell me if you looked into aspects of the NVMe device itself, such as whether it supports 4K logical blocks instead of 512B? Use `nvme id-ns` to read out the supported logical block formats.
Doesn't seem to support 4k out of the box? Some drives - like Intel Optane SSDs allow changing this in firmware (and reformatting) with a manufacturer's utility...
$ lsblk -t /dev/nvme0n1
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
nvme0n1 0 512 0 512 512 0 none 1023 128 0B
$ sudo nvme id-ns -H /dev/nvme0n1 | grep Size
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
Thanks for checking. SSD review sites never mention this important detail. For some reason the Samsung datacenter SSDs support 4K LBA format, and they are very similar to the retail SSDs which don't seem to. I have the a retail 970 Evo that only provides 512.
I just checked my logs, and none of Samsung's consumer NVMe drives have ever supported sector sizes other than 512B. They seem to view this feature as part of their product segmentation strategy.
Some consumer SSD vendors do enable 4kB LBA support. I've seen it supported on consumer drives from WD, SK hynix and a variety of brands using Phison or SMI SSD controllers (including Kingston, Seagate, Corsair, Sabrent). But I haven't systematically checked to see which brands consistently support it.
At least early WD Black models don't really seem to have 4K LBA support. The format option is listed, but it refuses to actually run the command to reformat the drive to the new "sector" size.
Put your system to sleep and wake it back up. (I use `rtcwake -m mem -s 10`). Power-cycling the drive like this resets whatever security lock the motherboard firmware enables on the drive during the boot process, allowing the drive to accept admin commands like NVMe format and ATA secure erase that would otherwise be rejected. Works on both the WD Black SN700 and SN750 models, doesn't seem to be necessary on the very first (Marvell-based) WD Black or the latest SN850.
I think that's the second-gen WD Black, but the first one that had their in-house SSD controller rather than a third-party controller. The marketing and packaging didn't prominently use a more specific model number to distinguish it from the previous WD Black, but on the drive's label it does say "PC SN700". Also, the first-gen WD Black was 256GB and 512GB capacities, while the later generations are 250/500/1000/2000GB. Firmware version strings for the first-gen WD Black were stuff like "B35200WD", while the SN700/720/730/750 family have versions like "102000WD" and "111110WD". So I would definitely expect your drive to require the sleep-wake cycle before it'll let you reformat to 4k sectors.
But this thread gets into details that are more esoteric than what I cover in most reviews, which are written with a more Windows-oriented audience in mind. Since I do most of my testing on Linux and have an excess of SSDs littering my office, I'm well-equipped to participate in a thread like this.
I highly recommend reddit.com/r/NewMaxx as the clearinghouse for consumer SSD news and Q&A. I'm not aware of a similarly comprehensive forum for enterprise storage, where this thread would probably be a better fit.
Regardless of what sector size you configure the SSD to expose, the drive's flash translation layer still manages logical to physical mappings at a 4kB granularity, the underlying media page size is usually on the order of 16kB, and the erase block size is several MB. So what ashift value you want to use depends very much on what kind of tradeoffs you're okay with in terms of different aspects of performance and write endurance/write amplification. But for most flash-based SSDs, there's no reason to set ashift to anything less than 12 (corresponding to 4kB blocks).
There are downsides to forcing the OS/FS to always use larger block sizes for IO. You might simply be moving some write amplification out of the SSD and into the filesystem, while losing some performance in the process. Which is why it really depends on your workload, and to some extent on the specific SSD in question. I'm not convinced that ashift=14 is a sensible one size fits all recommendation, even if we're talking only about recent-model consumer-grade NAND SSDs.
3) Why? As a performance troubleshooter consultant+trainer, I regularly have to reproduce complex problems that show up only under high concurrency & load - stuff that you can't just reproduce in a VM in a laptop.
4) Fun - seeing if the "next gen" hardware's promised performance is actually possible!
FYI I have some videos from my past complex problem troubleshooting adventures, mostly Oracle stuff so far and some Linux performance troubleshooting:
