At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine.
The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.
(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)
I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think.
I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year.
Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :)
(Hmmm, now I'm trying to figure out you are, send me an email if you will :)
> I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert.
Which is why I'm happy others have paved the way.
A huge amount of web apps can get by with a Postgres backed queue that won't break a sweat handling a few hundred writes per second on a low end cloud hosted VPS.
While leveraging something like Oban you don't even need to be an expert. You can just use the library, throw work into it with a reasonable amount of foresight / best practices and you can start developing features in your app.
The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database.
(Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)