Even without NFS, using mmap requires being real careful about signals - SIGBUS can be raised any time the underlying file operation fails, including because someone else truncated the file, or because the underlying storage had an error (disk error, removed media, network storage). And, as this post so eloquently illustrates (and through my personal experience), handling SIGBUS/SIGSEGV cleanly in a multithreaded program on POSIX is incredibly painful.
Honestly, pread is just a much better solution for 90% of use cases, and it works for large files on 32-bit systems (mmap does not!). If you're doing largely sequential things, fread/fseek often work remarkably well as they handle all the caching for you.
mmap tends to shine performance-wise if you need random access to a file but access certain parts of the file frequently (for example, accessing the index in a header + contents of the file), because the page cache is literally designed for this type of usage. But the performance improvement is rarely worth the technical complexity.
Alternatively one can run a separated process that does mmap and runs the calculations or whatever that needs to access the file as quickly as possible and do the the straightforward recovery in the parent process when the child process dies. The drawback is the need to some form of RPC, but there a lot of libraries to do that without much hustle.
If you are already copying things around (between address spaces even), you might as well just use pread or lseek+read, which at that point is likely a much better choice overall.
It would be better to mmap the file into N single-threaded processes instead of mmapping the file into 1 N-threaded process. This is exactly why signals are process-global instead of thread-local. The intent is that they were used for inter-process communication. The whole notion of processes and signals predates the notion of a thread, and threads are essentially a performance optimization for 30 or 40 year old hardware. If you use single-threaded processes then all of the processes can MAP_SHARED map the file and then they all get access to the file contents as each process is updating the file. And they all also have their own address space and can crash separately from each other.
That's because mmap() predates multithreading (at least as a usable production approach rather than a lab experiment). It's best for IPC, not sharing between threads.
I forget sometimes that the young'ns never grew up in a world without threads, and often never learned how unix really works.
>This is exactly why signals are process-global instead of thread-local.
Signals can be sent either to a process or to a specific thread under linux. Signals sent to the process are handled by an arbitrary thread. There's a bunch of gotchas and whatnot past that.
> Mixing signals and threads is pretty much trash.
Agreed, except for SIGSEGV, SIGBUS, and SIGPIPE, which are guaranteed to be delivered, if at all, to the thread which triggered the condition. (Unless, of course, they were raised explicitly.) Still trash in the sense that to do it right you're depending on installing global signal handlers and (very likely) arranging for thread-local storage (actual TLS isn't even async-signal safe; see my post elsethread regarding the sigaltstack trick), so these tricks are really only practical for the core application and not something you can stash in a library to perform transparently.
FWIW, none of this behavior is specific to Linux. Linux follows POSIX rather well in this regard.
You can do that with a MAP_ANONYMOUS | MAP_SHARED mapping too: that kind of mapping is writable by both parent and child, but isn't backed by a disk file and so can't be truncated or surprise-removed. The article's points about mmap infelicity applies mostly to mappings of disk files. Anonymous mappings don't have the same problems.
Anonymous mappings are backed by swap and may be overcommitted, it's still possible to catch signals in a wide variety of circumstances
There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid mmap whenever possible -- it's clear almost nobody fully understands it
If a process was swapped out and a fault fails to bring a page back due to an IO error, you can at least catch (I think) SIGBUS. But this just reinforces the point: nobody really understands virtual memory, even people like us that think they do
So should we extend your conclusion above to the following?
"There is probably enough evidence in this thread to use it as a reference for why typical apps should avoid virtual memory whenever possible -- it's clear almost nobody fully understands it"
I'd suggest that is ludicrous, and for the same reason your original conclusion is also excessive.
It is not constructive to form a sweeping generalization from a statement and then claim the sweeping generalization is ludicrous, implying the original statement is ludicrous. :)
My first comment was in reply to one claiming anonymous memory did not have the same problems as file-backed memory, indicating the parent did not understand they are the same thing. The subsequent reply was to another comment continuing to claim anonymous memory was somehow safer, both instances supporting the notion that most people in this thread don't seem to understand mmap at all.
What we're examining is a powerful (and consequently hazardous) OS feature that often provides only marginal performance improvement, yet introduces many exotic error paths into a program that have their own exotic problems (memory access in thread A can raise SEGV in thread B, async-signal safety), that 7 hours' commenting has not been sufficient to fully capture. This thread is full of upvoted miscomprehension, bad advice (spawn a child to deal with SEGV!?), obviously incorrect solutions (signalfd), and yet still manages to completely omit some critical characteristics of mmap, for example, that faults take a VM-global semaphore -- mmap can easily destroy multithreaded app performance in a way read() is immune to, because nobody expects file IO in one thread to cause malloc() latency in another.
If this isn't evidence for "avoid this feature wherever possible", I really don't know what is.
As if "avoid virtual memory" is substantially more of a "sweeping generalization" than "avoid mmap."
If you apply the same reasoning that you've used to conclude that everyone should avoid mmap, than you are led directly to the conclusion that everyone should avoid virtual memory.
The "same problems" that you are pointing out are possible with anonymous memory aren't unique to memory you get directly from mmap, they also apply to all memory, period. mmap'ed anonymous memory might have th same problems as file-backed memory, but those are the same problems that .text and .bss have.
The "powerful (and consequently hazardous) OS feature" here isn't mmap, it's literally virtual memory. At the moment you concede that memory may be backed by something besides physical memory at any point in time, you get the possibility of all those "exotic error paths."
The exotic error paths are always there, but you don't always need to handle them. You can push some error handling to other systems, such as clients and supervisors / orchestrators. The reason why mmap with files is tractable is because we have a good error handling strategy (remap with zeroes, mark error) and we have a few understandable reasons why we might expect the error (IO errors, lost media/network failures, or even truncate). In general the problem that IO is done outside of direct syscalls like write() can be difficult even when you're not using mmap, like when Postgres was losing errors when calling fsync.
But when you have an IO error in your swap file, go ahead, eat the SIGBUS and die. This is fine.
Let's try and simplify things here: the post we're both currently commenting on relates to doing file IO via nmap. Of course 'avoiding all use of virtual memory' is ridiculous, but nobody except you is suggesting that, and you continue to suggest it even after a long reply.
The "powerful (and consequently hazardous) OS feature" here is using mmap for general file IO, it:
- introduces resources leaks many developers can't profile
- introduces VM bottlenecks 99% of developers can't profile
- introduces random segfaults delivered to arbitrary threads in the running process, leading to crashes many developers can't diagnose
- the mmap() interface itself is fundamentally unsafe in that it allows partially overwriting random bits of VM (MAP_FIXED) with file views, and worse still, allows those mappings to be read-write
Once again, nobody has ever suggested avoiding virtual memory except you -- once again, that is impossible in a modern environment, but it is more than possible, and ultimately incredibly sensible, not to mention entirely on topic with regards to this thread and the article it is attached to, to suggest avoiding use of mmap for general file IO
- Nobody ever said mmap was always faster than the alternative. If you care about performance then you should do whole-application performance testing with and without features enabled (like mmap IO). This is not unusual, there are plenty of aspects of performance that are counter-intuitive, where speeding up one part of your program causes a seemingly unrelated part of your part of your program to slow down.
- The signals are SIGBUS, and they can be intercepted, mapped with zeroes, and the errors can be propagated back to your app code later. This is not trivial but neither is it outrageous.
- You can overwrite arbitrary memory with read(), too, you just have to pass it a pointer to something you want to overwrite. mmap() is not any less safe. Recall that typical use of MAP_FIXED is so you can overwrite an existing mmap() region with something else, not so you can nuke random parts of your address space.
Keep in mind that you are, if nothing else, an indirect user of mmap(). The question is whether using mmap() directly is advantageous for your applicaiton. "Yes" is not an unreasonable nor outrageous answer for some applications.
> nobody has ever suggested avoiding virtual memory except you
Do you know what the "anonymous" in "anonymous mapping" means? You are the one that started asserting that anonymous mapping from mmap have the same difficulties as mapping of normal files and therefore too dangerous to use.
> paragraph, n.: a distinct section of a piece of writing, usually dealing with a single theme and indicated by a new line, indentation, or numbering
In the original comment you will find two of these, the former correcting an error in the parent comment, the latter making an observation based on the obvious brainwrong riddled throughout this thread
I'm done replying, you're of course free to continue checking in hazardous and suspect file IO code, as the rest of us are free to giggle at such things before ripping them out
This simply won't die, will it? I mean, while we're at it, let's advocate abandonment of all higher level languages because essentially they all boil down to machine code, and nobody could recommend working directly with machine code any more, could they.
Please clarify how you think that statement is incorrect. As far as I'm concerned, it won't die because that's how virtual memory works, and you have something going on in your head that is either wrong or massive hairsplitting.
I would have expected a better understanding from someone bloviating about how the try of the commenters are too thick to understand mmap and virtual memory.
All* allocation is mmap, really. All mmap does is dedicate a region of address space to some kind of backing storage. The particular kind of backing storage makes all the difference. The problem is that people colloquially use "mmap" to mean "mmap of a conventional disk file" and don't mean all the other kinds of mmap out there, so discussions can become confusing.
* Ther's sbrk too, but it's just a fancy legacy path that amounts to the same thing as anonymous mmap
The article points out the infelicity as applied to mappings of files into multi-threaded programmes. What
fpoling suggests is a solution to the POSIX signals problem where anonymous mmaps are not an option.
Honestly, pread is just a much better solution for 90% of use cases, and it works for large files on 32-bit systems (mmap does not!). If you're doing largely sequential things, fread/fseek often work remarkably well as they handle all the caching for you.
mmap tends to shine performance-wise if you need random access to a file but access certain parts of the file frequently (for example, accessing the index in a header + contents of the file), because the page cache is literally designed for this type of usage. But the performance improvement is rarely worth the technical complexity.