“What I wasn’t told was that we had 70,000 hard-coded references to Sacramento”.
Except they had many people tell him it was a bad idea.
In the end, it was indeed a bad idea. Unfortunately, his loyal lieutenants get rewarded for “kicking ass” while the person who knew it was a bad idea probably started looking for opportunities elsewhere.
The only people who would be left in the end are people who worship him unquestionably.
For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. “In retrospect, the whole Sacramento shutdown was a mistake,” Musk would admit in March 2023. “I was told we had redundancy across our data centers. What I wasn’t told was that we had 70,000 hard-coded references to Sacramento. And there’s still shit that’s broken because of it.”
"What I wasn’t told was that we had 70,000 hard-coded references to Sacramento"
You can't play the "I wasn't told" card, this is something that I would hope 99% of engineers would have warned about, and if Elon was the "coder" he claims to be he should have known himself
"The challenge is his personality and demeanor can turn on a dime going from excited to angry. Since it was hard to read what mood he might be in and what his reaction would be to any given thing, people quickly became afraid of being called into meetings or having to share negative news with him." -Esther Crawford
It's not an autobiography. Whatever is written to have been said here comes from multiple sources. That also means that whatever isn't written as having been said we know was not said.
This is a ludicrous assertion. What if he said something to only one person in privacy, and there literally aren't multiple sources other than him (and the other party to the conversation might have incentive not to divulge anything he didn't approve of)? Not being an autobiography doesn't somehow imply that we know every single thing he's ever said.
You're talking about a hypothetical scenario that doesn't even match the conversation described.
The recounted conversation involved multiple individuals, many of whom have an adversarial relationship with the subject and would be inclined to sue. Misrepresentations of facts could have meaningful reputational damage.
If you're the author you do not risk going to publication without verification that the gist of the conversation is accurate. Your editor literally will not let you. The publisher's reputation is on the line too.
What purpose is served by grossly misrepresenting the number of servers in a datacenter and who said what about their status with the very real risk that they might sue you?
Elon is the kind of deep-pocketed person that lawyers take cases on pro-bono against.
Reminds me of my first start up. Every time we ran out of money or good plans, we had to rearrange the office. Movement is activity is purpose is not dead&gone I guess.
Let he who has never written a permanent "temporary hack" cast the first stone.
This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.
> and a company culture not focused on continuous improvement.
You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.
Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism.
Some of the best people I've ever worked with come from Amazon though.
Amazon is one of the few top tier companies with a generally good reputation where I know engineers who have left voluntarily. The amount of baling wire and duct tape holding it together is only one reason.
> She started to give a lot more details, but after a minute, Musk interrupted.
> “This is making my brain hurt,” he said.
> “I’m sorry, that was not my intention,” she replied in a measured monotone.
> “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.”
156,000 individual physical servers (remember, it said 5200 racks * 30) in one building and not being the only building _to serve 140 character tweets_ IS fucking absurd.
This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate.
This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice.
And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate.
It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit.
Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
First of all, their infrastructure is globally distributed. This was one data center of many.
Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly.
I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.
If you're doing your job properly and you can't be expected to know something coming in, you shouldn't just make extreme changes wholesale on a whim without any sort of check that it won't break things.
In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines.
And yet nothing catastrophic happened as a result.
The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase.
I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone.
Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
> And yet nothing catastrophic happened as a result.
Musk himself blames the DeSantis Spaces debacle on this poorly planned server move. If that's not catastrophic, I don't know what is.
There's almost certainly entire teams of on-call engineers pulling heroic shifts because of this. I'm sure Musk DGAF, but those poor souls losing their holidays, weekends and time with their friends and family probably care.
Their advertisers have been leaving in droves, Twitter itself has been breaking itself in new and profound ways since his take over and the site itself was broken (per Musk's own admission) as a result of the move. Profitability is in the dumpster, they're dealing with multiple court cases for refusing to abide by their legal obligations in regards to severance AND rent. And this entire critical move was caused by their lease running out with zero plans in place due to the financial crashing of Twitter.
I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
> I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
The problem is people like yourself (nothing personal) acting like there's sides to be on and that it matters. Twitter was a stupid company before and is still a stupid company after. Bystanders are stupid for being waaaaaaaaay too invested in the story and the outcome. Tribalism is fucking cancer.
The era of risk free money created a world-scale reality distortion field where people believed that a company like Twitter -- with no novel product, no paying customers, no growth trajectory and a cash burn rate that could put men on the fucking moon -- is a viable, thriving business.
Brand equity is only powerful for as long as you have money in your pocket and the rent isn't fucking due. And advertisers are fickle.
I don't know how anyone who read "The Emperor's New Clothes" as a child could give the slightest shit about Twitter, but here we are.
Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences
Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else
If you buy a totally rotten building, what parts do you save? The foundation's fucked. The floors can be fallen through. There's black mold everywhere.
It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference.
I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional.
The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
emphasis on the first clause, because it seems to be something elmu struggles with
Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there
Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself
these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad.
Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!”
he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own
and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable
Can you say more about why his behavior would make you disinterested in the Isaacson book? I'd imagine the fact that it seems there are parallels between both men would make it an interesting read especially for someone who could compare and contrast it with the Jobs book.
Because Steve Jobs was a smart a-hole. Musk is just an a-hole by all accounts. Nobody wants to read a book about a pampered South African neo-nazi racist and how he has consistently taken credit for other people's work while peddling to dictators across the world and treating his employees as actual slaves.
Wow they used real servers. That goes pretty well with their lack of a dev/test environment. And they had hard coded references to infrastructure, because they were always in a hurry improving their product. Extending that character limit took 500k engineering person years, including the devops team. Great job!
Really, Musk sounds like an amusing guy to work with. "Fuck, let's do it" is an attitude more execs should take.
Twitter has been one of the most expensive human undertakings in history and it's just so mediocre. If it isn't actually harmful to us.
In fact since Musk bought it, people have been tripping over each other rushing to proclaim just how mediocre and unnecessary it is. So much so that they don't even need it anymore.
And yet when anyone points this out, these same people expend an enormous amount of energy pillorying that person and shouting incoherences about Musk.
Except they had many people tell him it was a bad idea.
In the end, it was indeed a bad idea. Unfortunately, his loyal lieutenants get rewarded for “kicking ass” while the person who knew it was a bad idea probably started looking for opportunities elsewhere.
The only people who would be left in the end are people who worship him unquestionably.