I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast cultural change.
My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.
Obvious, right?
What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development.
Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.
> My observation is that Agile is all about ... and for the benefit of software developers ...
No, it's the opposite! Agile is bad news for software developers:
1) hardware developers will insist on having predetermined requirements and will stick to them; at most, they'll throw the mantra: "Software will fix it!";
2) software users will use Agile as an excuse to avoid at all cost deciding what they want and limit themselves to bikeshedding after the release;
3) those idiot project leaders who manage by ticking cases on an Excel printout will explain you how Agile is a wonderful panacea which will solve all your problems: you will "just" have to write the code and you'll have no excuse to miss the arbitrarily imposed deadline.
No no no, not at all! Agile is for those who ponder about developing software, but not for those who do the job!
Paraphrasing here. "Agile is the worst software development process, except all the rest"
More seriously, doing big upfront design was a really bad idea. Agile and more specifically Scrum brings a number of ceremonies to keep the "managers" happy and allow the transition. When and if ever companies understand that they can trust their software engineers they can let them to their job. These ceremonies are more like a security blanked for the non software engineers who are frequently clueless about how software is developed.
I prefer to focus on the Agile principles. My favorite is "Continuous attention to technical excellence and good design enhances agility."
> Agile is the worst software development process, except all the rest.
That's not my opinion. I'll quote Alex' worlds:
In the past decade or so, we’ve seen a lot of “fad development methodologies” crop up. [...] They all promise the same thing – cheaper, better software built faster – and they all generally fail to deliver. Like fad diets, these methodologies try to avoid a simple, painful truth: you have to know what to build before you can start building it. (http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile)
"Faster" (and "cheaper") are the problems here. I believe Scrum embedded this flaw in its very language by calling the iterative cycle "sprints". You don't run a marathon by breaking it into a bunch of sprints - sprinting in a marathon is foolish. But the language suggests speed.
If the Scrum cycle were called "iterations" rather than "sprints", things would be a lot better. Language matters.
This is an interesting language problem. I think part of the reason the words are “agile” and “sprint” is that it’s essentially a way for engineering to communicate iterative progress, convince management that yes, work is getting done, and that really, we’re moving as fast as we can. That there is deception in the wording is, imho, a pragmatic response to management never seeming to understand software development at face value.
Eng: “No, we can’t move any faster. This is literally the best we can do. Doing this right, and doing it as fast as possible, involves care and intentionality.”
Mgmt; “Can’t you just work more hours right now?”
“No, that will not help.”
“You guys are lazy. We regularly pull 70 hour weeks to get all the meetings and emails in.”
“You know what? Next week, we are going to deliver x,y, and z. Promise. It may mean the overall vision takes longer to execute, but I want to show you how hard we work. We are getting a ton done. We are moving as fast as we can. We are sprinting! Now leave us alone and let us do the work necessary to execute the vision.”
Mgmt: “Hey other managers, look how much MY team is kicking ass. They are delivering x, y, and z ON SCHEDULE AND AS PROMISED. They are sprinting, and they basically never tell me ‘no’! Sure, we don’t need y or z to execute the vision, but how else were we to know that? If we hadn’t tried it out? They just sigh and go back to their caves, EVERY WEEK. I’m the best manager.”
It’s basically a way to educate about iterative development while giving management a way to save face, no matter how inefficient it may be. A healthy org could do it without the deception, but corporations are krakens.
>If the Scrum cycle were called "iterations" rather than "sprints", things would be a lot better.
Agree. To me, the real value of Scrum when done right is the focus on delivering increments of value in a sustainable, repeatable way. Scrum provides a means and the existing hype/buzz/literature gives me some leverage in dealing with non-technical managers and getting them to buy-in to the process.
My favorite feature of Scrum on paper, as designed, is the idea that any change to the requirements for a sprint means the sprint ends right there, and a new sprint is started. It's great for "Just leave us the hell alone for two weeks, and we'll deliver what we promised we'll deliver over these two weeks".
Of course, it never works that way. The sprint becomes an arbitrary schedule, and the work of the sprint becomes mutable. Ugh.
Sudden, OMG THIS IS IMPORTANT WE MUST DO THIS RIGHT NOW changes in direction are the bane of productivity.
Add to that: you have to be prepared for sudden changes in functionality and scope.
If you dont then the painful truth mentioned will only apply to utopia (never been there but I hear everyone is rich and thin with infinite throughput).
That is the thing. You rarely know what you need to build. With big upfront design your are speculating, the design is a house of cards. Agile amplifies learning. You learn as you build because you get early and frequent feedback.
Agile relies on having very capable product owners who understand what needs to built. Waterfall relies on design documents. Which will be all wrong anyway if you didn't have a capable product owner driving the vision of the product.
> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
These aren't part of Agile 101, but the Agile community has been discussing these things for years. I first heard Mary Poppendieck discuss Agile contracts in 2006 or so [1]. Pat Reed has been talking about Agile accounting since at least 2014, and I doubt that was the beginning of it [2]. My book discusses techniques for making commitments to stakeholders, and it was written in 2007 [3].
Literally Google the phrase "Agile" + anything you wrote and you'll find a wealth of resources. If you're not finding that information, it's because you're not looking.
The agile community is currently an amorphous blob of people connected by words like "Agile!", "Scrum", "TDD" and so on. Agile was stolen from its creators and mutated into a corporate version - which I have to say I find pretty darn funny.
Point is, that info is certainly not included in any of the Agile resources I've seen. The only one that handles some of this is Ambler's D.A.D, but I didn't bother reading that for a simple reason: after seeing various Agile methods (mostly Scrum) fail so much, I've come to the conclusion that Agile development processes do not represent a real-world solution.
There's always an excuse why Scrum fails in practice, but in the end it requires such a paradigm shift from our typical development approaches in order to succeed, that it's about as useful as a diet plan that only works on super-athletes.
XP is in zombie mode for a decade, Scrum is only good for mobile apps and small websites, Kanban is not even a way of making software. What are we left with? Scaled Agile, LeSS, which even the original Agile people bitterly complain about? These won't work either, but I know some consultants make a crapton of money selling them to corps.
> I've come to the conclusion that Agile development processes do not represent a real-world solution.
I think one of the problems regarding this whole discussion, whenever it stirs up, is that Agile (as defined in the original manifesto) is explicitly not about any particular processes, but still most of the critique against Agile seems to actually be against different kinds of "agile processes". The manifesto explicitly states "Individuals and interactions over processes and tools".
In real world, one of course needs actual processes, and all kinds of consults are trying to ride the wave to lucrative contracts. But since the Agile Manifesto is about values, not processes, you cannot steal it from its creators. You can pervert the meaning and utilize some braindead processes in your company and call them "agile", but that doesn't lessen the worth of the original values.
I know that that's a highly unsatisfactory answer, a kind of a cop-out, but I think applying more to Agile than it's meant to be is part of the problems surrounding it today.
Edit: those "you"s are meant for passive voice. No personal offense or commentary meant.
The road from Agile values to having something that can be used to ship quality software is way too long. "Eat well and do sports" is correct, yet ineffectual advice, since many countries still have a big problem with obesity, diabetes and other modern diseases.
There are two things undermining "the values" in practice:
1) The economic and social environment doesn't encourage an engineering/scientific approach to SW development. Nor does it encourage eating well or having an active lifestyle.
2) People need a lot of help to implement those fuzzy, high-level values and will always reach for some process / diet / plan.
All things considered, it's probably better to be agile, but how do you actually deliver a quality, successful SW project? Let's talk about that instead of fuzzy old Agile.
I agree with your conclusion fully. Also your link to "eat well and do sports" is kind of exactly what I was after. I mean no one is saying that eating well is bad for your health, right? What I was trying to convey is that I think similarly of the Agile Manifesto itself.
Because we shouldn't infer that eating well is bad because a lot of people try to eat well and are still obese, we shouldn't infer that Agile as in the Manifesto is bad because lots of companies get burned by Scrum. We should be coming to conclusions like "hey, maybe this Scrum thing we tried is shit, let's try some other process that seems to align with these fuzzy agile values we believe in".
The problem I have with the current discourse around the matter is that as long as we keep bashing "The Agile Values" when we mean "agile processes as implemented in framework ZZZ", we are not having the discussion of "hey, which process would align better with these values while letting us ship quality software rapidly".
I find the whole idea of one size fits all processes dubious. I don't have a solution for this and I haven't seen any such process work reliably in large scale projects.
Scrum quite clearly can't tolerate the peculiarities and pressures of a large scale mixed HW-SW projects for instance. Those will turn the whole sprint / PO / independent SW team arrangement into mush.
A long time ago I did some waterfall-RUP mix which didn't feel worse compared to the Scrum I did recently, but it did result in better documentation and traceability.
If I had the chance I would take bits and pieces from several processes and experiment to see which work for a particular company and department, perhaps even at project level. Things that come to mind: Kanban-like task management, regular retrospectives, an exploratory design phase at the beginning, documentation (FFS!), history-based estimation + careful team-private time tracking, a project manager that doesn't also manage the team and doesn't have too much power, regular customer-feedback cycles, CI/CD pipeline, etc.
The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.
Agile is such a vague thing that it's impossible to say whether it works.
So let's take Scrum, which is probably the face of Agile, whether its parents want it or not: Scrum doesn't work either for a lot of projects. Typically the implementors are blamed, but if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself.
Maybe the real problem with software is that it's become too unstable because of market forces and all these methods designed to tame that are just a band-aid. Releasing a new version of an OS yearly seems to be a pretty silly idea...
At the current level of complexity and churn, everything seems to deliver less than impressive results in the end, but people are kind of used to perpetual crappy software.
> if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself.
I have heard that sentiment many times but it doesn't really make sense to me.
If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then?
Or do you mean that the method should be designed in a way that only companies who will succeed will adopt?
By your standard, is there any successful method to this area?
Yes, I would blame the choice of the method if most patients are too lazy or busy. A good choice would instead work for busy and lazy patients or would convince them to be less lazy and/or busy :) Multiple options should be offered depending on the patient.
I've given actually a similar example regarding the "eat well and do sports" advice. It's not bad advice, it is correct, but it's not working for a large segment of the population, causing massive problems for health care and later (or now?) society.
Instead of cheap advice maybe the doctor should prescribe physical therapy with a specialist where the patient can be educated on the what and the how. Or the appropriate gov organisation should ban or heavily tax soft drinks and crappy foods and make good, healthy food affordable and available.
If a doctor tells patients to go for a run a few times a week with no additional resources and training advice, the doctor just set the patient to fail. Because they will train ineffectively getting little benefit for a lot of effort. Some will get injured from overdoing it. Others will stop improving after just few weeks and then end in frustration.
If a particular method is not compatible with duties and needs of a normal employed person with a familly, then that method constitues a bad advice for such person.
> If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then?
Yes, if a medical intervention largely fails in real-world practice, whether or not it is through non-compliance, that's a strike against it as an intervention. On which is more successful in practice, even if worse in the ideal case of perfect compliance, is a better intervention.
Similarly, a development methodology which fails with real world orgs, even if the failure is due to “incorrect” implementation, is a worse methodology than one which produces better results in practice.
> Agile is such a vague thing that it's impossible to say whether it works.
Agile methods aren't really particularly usable across orgs for the same reason prior methods weren't; the Agile Manifesto states “individuals and interactions over processes and tools”, but the Agile community is all about processes and tools at the same level, if with different content, than the processes and tools the Manifesto was talking about. How do you prioritize individuals and interactions? The Agile literature doesn't really address this. (Lean, for instance, does through higher-level processes and tools by which teams establish ownership and control of the main processes and tools, which meta-processes are subject to themselves so that he team owns the meta-process as well as the lower-level process.)
> Alot of big companies do fine (financially) with the waterfall methodology.
Because it works for them. Right tool for the right job maybe.
I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.
I guess. Ime. all project are are combination of waterfall and agile. Wrt. infrastructure the initial architecture plan needs to be almost all-encompassing. On the other hand you need to be able to react to new information and changing circumstances.
Being religious with methodology is going to backfire 99% of the time.
> Because it works for them. Right tool for the right job maybe.
They are able to deliver software in spite of waterfall rather than because of waterfall. They just keep injecting time and money until they 'declare' their projects to be complete.
Very insightful observation. As someone who has worked in companies just like the one you described, fully agile startups and horrible hybrids invetween, I think that the biggest failure of the application of agile is that persons outside of the development bubble see agile as a solution to old late, overspent, under delivered solutions. Really the agile manifesto needs to be applied top down. Frameworks like SAFe attempt to solve this. In truth the most powerful application of ‘agile’ every entity in an organisation can apply is to fail as fast as possible. third party is slow to interact and hard to deal with? Switch provider or offer assistance. Hiring staff is tough? Look internally at recruitment team and remuneration offered. Project is late and costing to much? Cancel it, don’t pump more more resource in
The UK has a big infrastructure project called Crossrail, which is running about £2bn over budget is two years late.
You can't just walk away from a project on that scale. Nor can you get 10% of the way in, realise you've made some fundamental errors, and start again.
In reality project management comes down to three problems - finding and optimising competence, dealing with politics and corruption, and minimising danger from external threats.
When projects fail it's almost always because something went wrong in at least one of those areas.
Agile can sometimes help with optimising competence, but only if it's run competently. It won't magically transform any org or situation that lacks fundamental competence and/or has problems with the other two areas.
I thought the problem with Crossrail wasn't that the track wouldn't be ready, it was that the stations wouldn't be ready?
In that case, you could concentrate effort on key stations and only add the remaining stations to the timetable when they are available.
(I don't know the actual details so I may be missing something. All I know is that there are Barnett consequentials for Crossrail that hands money to Scotland, Wales and Northern Ireland but nothing for we English who don't live in the South East).
I think the reason for that hostility is that, since it was decided software needs to be made and software developers are the only ones who can make it, tasks that don't involve making software may be better handled by others.
No one has solved contract writing, and of course developers need to contribute sometimes, but the less time they spend on contracts, the more energy that is put into delivery of working products. Working products soothe stakeholders better than developers detailing why there's no product yet.
A lot of success with Agile comes down to having great administrators/managers/etc who are capable of dealing with as much of the surrounding efforts needed to make software in an organization as possible.
> My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.
No, it is not. It is definitely not for developers benefit. It is sorta kinda good for junior developers I think, but mostly it is for management. It effectively ignores human psychology, treats developers as machines and individual developer is mostly powerless with no responsibility, no accountability and zero autonomy. It effectively amounts to large dose of micromanagement.
It may sound weird given agile rhetoric and some its processes are really good and improvement, but in it is does creates pleasant work environment in practice.
> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
> These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development.
My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things.
What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want.
In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process.
But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.
The Agile Methodology is to argue about The Agile Methodology.
--
The only durable thing I've learned about methodologies:
Alistair Cockburn
"Characterizing People as Non-linear, First Order Components in Software Development." Oops. https://alistair.cockburn.us is "under construction". Sorry.
Anywho. My current position is "just tell me what you want, I'll adapt." As a recovering methodology junkie, I can no longer stand talking about it. If everyone is on the same page (roles, rules, expectations, etc), I'm happy. If someone dictates this ceremony by fiat, minimizing discussion, all the better.
Its problematic that Waterfall/Agile stuff regularly gets upvoted to the front page, and then the majority of commenters don't seem to read the article before posting some tangentially related anecdotes on the vague topic of "Agile".
This article is making the point that a "Waterfall Method" has never actually been proposed by academia, so it is incorrect to speak about an Agile/Waterfall dichotomy. In other words: The Waterfall Methodology is not a thing and only seems to be used as a "straw man" foil.
Personally I think we should also be a little more critical about who defines "Agile", how it differs from common sense, and what research has actually been done on it, but thats another matter.
"The Waterfall Methodology is not a thing and only seems to be used as a "straw man" foil."
Waterfall is a 'thing' because it existed long before software or computers.
I view it as essentially the most obvious form of project management, which is simply a phased approach to solving a problem: gather requirements, plan, design, do, test, fix problems etc..
I think that’s the core problem, too: it works fine for well-defined needs in mature categories — things like buying printers or renting office space, where all of the options are usually capable of doing the job to at least a basic level. Large organizations have tons of experience muddling through that process and it’s tempting to think that software is similar, especially because many vendors are will to perjure themselves saying so.
The problem was accepting that custom software doesn’t work in the same way and reconciling the huge differences in how much customization people request versus what they are willing to pay.
The word Agile has become so loaded that if you want to side track a meeting just mention you will be using Agile. Nothing else of value will be discussed in the meeting because people will either vigorously object, support it or argue on what exactly is Agile. This brings me to your statement:
> Agile, how it differs from common sense
Thank you for mentioning common sense. No one size fits all. I have never followed one methodology to the absolute T. I borrow from all the methodologies I have read about. I use KanBan boards I learnt about at a Agile workshop but don't have daily scrum meetings. I have SLDC like development plans but never hesistate to meet clients to seek clarity when needed or even to show them what I have done. Sometimes I show them final product at the end of the implementation because it can confuse clients to show them a half finished product. Sometimes it is possible to show a half finished product.
I must confess I haven't read the article but had to post about Agile :-).
I would at this point argue agile is the new waterfall. It's so common that the phrase "we're doing agile" vaguely hints at a few practices that may or may not be relevant to the discussion. Mostly it means "we have standups" and "we promoted some juniors to scrum master and pm roles".
It's mostly devolved to ritualistic bullshit performed dogmatically by corporate idiots. Every shitty third rate software sweat shop does agile/scrum these days. You need to look pretty hard to find companies that are not very eagerly proclaiming they do agile.It means absolutely nothing until you qualify what that actually means for you.
Those claiming they are doing it "right" or by the book are the worst because that means they read some stuff without grasping the meaning of what they read. Basically, the agile manifesto signers have come out arguing against that. That was the whole point of that manifesto. They basically provided us with tools, practices, and ways of working that we're supposed to adapt to our context as needed. They don't say: do this dogmatically and follow my instructions by the letter. They actually say don't do that but if you are getting started you might as well do this and then adapt as needed.
That's sort of the same thing that went wrong with waterfall. The article points out a few historical twists to historical context of that. I read the paper by Royce long time ago and found it to be quite modern and insightful. It doesn't mention waterfalls at all (as the article points out). It also points out that, waterfall as we know and love it, basically was institutionalized bureaucratic government organizations using it to sugar coat heavy processes that they came up with that got subsequently adopted as "the way to do things" by equally bureaucratic large software houses they worked with. It got a bad reputation for said corporations making obscene amounts of money with obscenely bad software delivered late and never on budget.
This pattern is something that is a constant and was referred to as the software crisis in the 1960s already and lead to Margaret Hamilton along with other famous pioneers in computing coining the notion of software engineering in the 1970s. She was of course working on the lunar program at Nasa (speaking of bureaucratic organizations). The software crisis is alive and kicking. Many scrum/agile organizations continue to peddle atrociously bad software developed. That doesn't mean that nothing has changed but you should beware of anyone claiming to know the one and only way to do stuff.
There's a lot of common sense that inspired agile and waterfall. I'd argue the modern take on this is to replace manual processes with automation to facilitate rapid flow of changes from inception to production. Another trend is to decouple decision points to facilitate asynchronous processes and communication. For example, you don't bundle features into releases. Instead you ship pull requests straight to CI and CD pipelines. That means that there are no central planning bottlenecks for delivering new features and they can be developed on their own schedules. This happens continually. Most big brand websites you use, update many times per day. Iterations and sprints are kind of obsolete as a way of orchestrating release schedules.
That practice is now creeping into the enterprise. If you are doing scrum and being told not to do this, that's something that you need to challenge.
It is but a symptom of the real problem of organizations blindly adopting 'Agile' out of no more than a desire to stay trendy and the developers taking an interest in the subject out of seeing that the move is an abject failure more often than not.
I think waterfall was adopted for software development, not by accident, but because it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing.
In areas where changes to the spec are expensive, like building a bridge, waterfall methodology makes a lot of sense. Now that dev tools are so good and computers so powerful, spec changes are cheap, so waterfall is inappropriate. However, there may be times where it is appropriate; maybe a controller for a pacemaker or an oil refinery.
I see it as using the right process for the right job.
In contrast, the Empire State Building was not built with a waterfall approach. It was delivered on time and 18% under budget. I saw Mary Poppendieck, who has written some excellent books on software development, give a fantastic talk on it. There's a write-up of a related talk of hers here: https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of...
My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. I think it's basically King Canute writ large. (Large, and into a zillion GANTT charts.)
I do agree that it's inappropriate for modern software projects. But I think this is also suspect:
> maybe a controller for a pacemaker
I've talked with a number of medical-device people, including Nancy van Schooenderwoert, who has been involved in the Agile world for at least a decade:
They believe that Agile approaches are way better even for life-critical systems. You do need much more rigorous testing, but given the way Agile approaches deliver core functionality early, you end up having way more time for that testing, and many more opportunities to discover ways to improve the testing approach.
That's a fascinating article on the construction of the Empire State Building. Thanks for sharing it.
However, what the article describes is fundamentally a waterfall methodology. The innovative part was breaking the large project into several overlapping stages of parallel projects, but the project management was still much more waterfall than agile.
The waterfall methodology is not a product of "naive fantasies" but of business realities. In many cases changing the plan after you get started is phenomenally expensive. AN agile approach of building three stories and then deciding you want to try a different support structure or to add another sub-basement would be absurd.
We all have a natural bias to assume that people in the past were foolish for not doing things the way we do, but for the most part, they were every bit as smart as we are and were making perfectly rational decisions.
In terms of the pacemaker example, I've never built one myself, but I suspect that the constraints are not only the need for additional testing but that the functionality and hardware itself would require regulatory approval, and so a waterfall approach to the project might make a lot of sense.
The developers could, of course, use an Agile or iterative approach to deliver on those requirements, but I think a waterfall approach might be a better fit for the overall project.
Well, at least _some_ aspects of a 50k feet Waterfall are not avoidable in any construction project, as you can‘t redo parts of the foundation when you‘re building the 50th floor.
Something that works quite in the same way (not as strongly, but still significantly so) in software by the way is re-modelling of the underlying entity schema.
Which by the way is my favorite beef with process-driven agile methods and why I‘m leading my projects with an initial phase of data model and UX sessions until it‘s exactly clear _what_ needs to be built and _how_. Only then does it make sense to build any working code IMHO.
Scrum and friends come with a complete disregard for distinguishing the stuff that‘s easy to change later on vs. the parts that are not.
You can either work to minimize the amount of change you'll have and identify the places where it's more likely, so you can add flexibility there, or you can be caught by surprise every time and pray that the next change won't suddenly throw years of work away. Both are perfectly sane attitudes on different contexts.
You generally can change those things even in a building but it gets exponentially more expensive, and at some point such deep changes are better done by a full careful rewrite - building a second building.
I don't believe the pacemaker example is necessarily a great one. In order to meet the risk-mitigation requirements for device approval, you have to document the design and the various decisions that led you to that point. If you're working in an agile fashion, you end up documenting a bunch of stuff that isn't in the product and doesn't get used, and at best your technical file is going to be much bigger than it needs to be (at worst, you confuse the assessor and don't get approval).
That's not to say you can't use agile methods, but I think the sensible way would be to use agile to explore the solution space and learn what to build, but continue using a formal methodology in parallel construction to document the development of the device.
It is definitely not Waterfall. Waterfall's distinguishing characteristic is that the total plan is known before any material action happens. For the Empire State Building example, that emphatically isn't true.
And that's one of the parts that is fantasy: total knowability. Bosses want that so they can indulge another fantasy: complete control. Perfect plan, perfect, obedience, perfect outcome.
> I've never built one myself, but I suspect that the constraints
Right. Note that here you are indulging in fantasy, in the face of me pointing to expert practitioners saying otherwise. This too is a characteristic of the adoption of waterfall. As the original article says, the paper that defined it said it was a naive, bad idea. But that stopped nothing. People ignored that and made it a template, ignoring the experts.
You should spend 30 minutes talking with a project coordinator for any construction job and you'll change your mind.
My company recently opened its first retail store in a new Caruso mall, and it was a considerably eye-opening experience. The pace (and cost) is frustrating, but the end result is generally fantastic and that's more than I can say for most software projects.
Even for a simple job, the number of things that need to align to allow for a dump truck to deliver gravel to a construction site is non-trivial. Along with permits and what times the truck can arrive depending on whatever bureaucratic bullshit is at play, you also need to make sure the rest of the crew is in place to receive the delivery and put it to use. Don't forget about the things that need to be done even before the gravel can be delivered, and things dependent on the delivery that begin on that task is complete.
This isn't a software project. A simple truck delivery being late can require dozens of people to get involved to reschedule everything else that needed to come after it - this is non-trivial, expensive, and can impact hundreds of people depending on how much the schedule slips since we often dealing with hourly labor.
"But it's only changing one delivery" you say? The delivery was probably handled by a contractor, which has other clients and other responsibilities and may not be able to reschedule when it's convenient for you. Multiply this by every contractor or 3rd party you're working with and suddenly Metcalfe's law is punching you in the face.
The waterfall approach exists because it works - on average - considerably better than everything else that's been tried before it, regardless of whatever cosmic forces we think are at play.
> A simple truck delivery being late can require dozens of people to get involved to reschedule everything else that needed to come after it
Looks like security people telling ops people they'll need to keep some extra data, and then coordinating with development how they are even going to collect it. Or developers moving ahead on their tasks while somebody later discovers that a schema migration will break a different application.
It's not exactly waterfal that solves this kind of problem. It's the other way around, rigid specifications make the problem worse.
I did spend a lot more than 30 minutes with a project coordinator, and complex construction projects seem to align remarkably well with software projects. Spikes, iterative approaches, severe changes in project goals/architecture while construction is already in progress, budget cuts just before the projects is almost done... it's all there. "We paid an unbelievable amount for a Gantt chart, and it was already outdated when it was delivered" he said.
> I did spend a lot more than 30 minutes with a project coordinator, and complex construction projects seem to align remarkably well with software projects.
I don't understand how you can say this with a straight face. Material costs for software projects are nonexistent (and don't come at me with AWS blabla, it's still way less than when building physical things). Hitting delete on what you've done so far costs nothing. Construction is pretty straightforward in that even if there might be different approaches to building a certain object (like a wall), these different approaches are agreed upon (as in people agree each is a valid approach and will choose which based on the pros/cons of each). Unlike in software where there exists no agreement in how to approach a problem and each programmer out there will probably provide a different solution.
I think you're quite a bit delusional here. Construction industry and the software industry are asymmetrically different from each other in the complexities you'd need to deal with.
For example, there might be many ways to do the same thing in the software industry, but you might have already dedicated yourself to proving that a given solution works "good enough". So hitting "delete" on that solution would require unmaintainable at that step amount of coordination between many teams who are using your code.
Construction doesn't deal with constant changes to the laws of physics, but software industry has a churn rate of APIs of about 1 new version every 6 months, all of which need to be constantly overseen and accounted for.
Did construction industry have anything like GDPR happen to it in the last 20 years?
> Did construction industry have anything like GDPR happen to it in the last 20 years?
You mean like regulations and building codes so that we don't have incidents where apartment buildings collapse like in China?
Some tests to get certain architecture certification require them to literally memorize different rules from everything to how close an AC unit can be to certain things. There are a LOT of rules to follow.
I believe we might be thinking about different project sizes here. I'm not speaking about building a single-family home.
The complexities of construction projects arise at a much larger project volume compared to software projects. The equivalence to a single-family home (which agreeably bears considerable material costs, and already requires a couple of people to build) might be a simple RoR application. Very straightforward, and to some extend every professional agrees on how to execute on that.
But if you accept a certain multiplicator in project costs (say, two orders of magnitude) before you compare a construction project to a software project, you will notice a lot of challenges commonly attributed to software projects.
In my experience, they are primarily a communication tool like my well-worn paperback copy of The Fellowship of the Ring is a communication tool. A way of communicating a beloved fantasy.
You claim that the date and time for the glass delivery for the windows is fixed before the diggers start hauling dirt to make room for the foundation?
That a construction isn't constantly scrambling to cover expected-unexpected incidents like a machine breakdown or shortstaff or a contractor screwup?
Not fixed, but date slippage is much more impactful in these classes of projects compared to software.
> That a construction isn't constantly scrambling to cover expected-unexpected incidents like a machine breakdown or shortstaff or a contractor screwup
They are, but thanks to great tooling and process they've been able to deal with it pretty well all things considered.
Did you read the article I linked to? I specifically gave an example of how one of the world's most prominent buildings -- far larger than a retail store -- was delivered not via waterfall planning, but flow orientation. And done on time and well under budget, something waterfall plans rarely manage to do in the construction industry?
So your theory that waterfall always works considerably better is demonstrably false.
As an aside, my brother builds houses, and I've spent a great deal more than 30 minutes talking with him about this. I agree houses are not software. But he agrees that waterfall thinking is often much more a problem for him than a solution.
> So your theory that waterfall always works considerably better is demonstrably false.
Uhmm, no? You pointed to a single building and you want to use that one exception to prove a point? Constructed during a time with considerably less federal & local bureaucracy? Seriously?
You claimed that waterfall worked considerably better. I am offering you proof that it doesn't. Your one example is a store in a mall that you spent 30 minutes discussing. My example is the Empire State Building, which a process expert spent significant time researching.
I'm open to the notion that waterfall processes work well somewhere. But my experience is generally that person in industry A says, "Oh, it really works in industry B." When I talk to somebody in industry B, they say, "Oh, I'm sure it works better in industry C." (Or, often, "industry A".)
Your theory seems to be that the fact that waterfall is the now-dominant paradigm means it must be good. That was the theory in software, and still often is, despite theoretical and practical demonstrations otherwise. So it shouldn't be inconceivable that it dominates in other industries even though it doesn't work well there either.
> My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects.
That is itself pretty naive. There is a reason why waterfall used to be popular, but it no longer is. Is it because people's ego's have ebbed? Doubtful. It's more likely that something else happened in the interim.
The obvious change is the proliferation of instant communications. If communicating is hard and expensive, it often makes more sense to change things as little as possible. If it's easy, then change becomes less expensive.
The internet gave us instant communication (e-mail, video conferencing, github, etc). Tools like these made agile more attractive. Humanity didn't just become enlightened.
You are overgeneralizing one (small) enabling factor of agile as the whole reason. Maybe based on your experience in major corps.
Agile doesn't start at the corp level. It starts at the individual and team level (think scrum) and maybe penetrates to the large strategic decisions on a department/corp level. Communication is cheap for teams in the same room or even people in the same building.
A far greater effect is the internet itself and software/services that is/are distributed through the internet. This made shipping new versions cheap, gave an abundant access to metrics and fast feedback cycles. Cheap shipping and fast feedback enable frequent iterations.
When you had to burn your software to CD and mail it to the customer releasing minor updates multiple times a week or even just every few months was not an option. You also got far less feedback and had basically no usage metrics.
You make some good points, but it still boils down to instant communication.
Updating software quickly is difficult without internet. Getting feedback quickly and cheaply also isn't.
So you're right that agile isn't just enabled by instant communication among developers, but I still think it's enabled by fast communication generally, including to and from the end-user.
Not true at all. It is not about instant communication but about transparency and bug reporting tools.
In the past, you got an app and you were lucky if the dev provided a marketing phone number for a drone to talk to. (Here is where IBM and such made bank, they did actual on site support.) Or perhaps an email address that was read manually in a year.
Nowadays, a crash is automatically detected and loses some send feedback going is included in many apps for additional reports, preferably landing in a CRM directly accessed by consumers.
The one problem left to tackle is marketing selling things that are not ready nor even tried out.
> It is not about instant communication but about transparency and bug reporting tools.
Good transparency and bug reporting tools are only possible with instant communication.
If you need to call a meeting and have senior devs fly-in somewhere to report them, bug reporting is difficult, and addressing those bugs are even more difficult. If all it takes is instantly opening a github issue, you can find out about the issue and address it nearly instantly.
Waterfall is still quite popular, just under different names. Most of the "agile" projects I come across are really mini-waterfall. And some supposedly agile processes, like SAFe, don't even really qualify as "mini". The main difference is that some Agile labels and rituals have adopted. But there are still large deliverables whose plans are completed in advance of production beginning. Delivery is heavily or entirely weighted toward the end, with no expectation of iteration or product improvement based on early release. The major difference with the waterfall of yore is that the chunk size is not 18-36 months, but more 2-4 months.
In contrast, modern Agile shops release early and often, with product choices for an initiative being informed by the real-world data generated by actual use. As an example, last I did a startup, our team of 6 was releasing a few times per day. The product manager would have an idea, find the minimum essence of it, and we would build and release that. User tests and live user data would quickly tell us whether the experiment pointed in the right direction; we'd then either double down on our bet and build a little more or we'd change direction. The average cycle time between having an idea and testing it was under a week for us. In mini-waterfall shops, that number is generally months.
The reason the 18-36-month version of waterfall died out was not modern communication tools. When my dad was a developer in the 70s and 80s, you had nothing like remote work. There was a special room that the computer was in, and everybody had directly connected terminals in an office they went to every day. Some places were pure waterfall. A few were doing what we'd now call Agile, releasing new software every few days and seeing how it helped.
From the way my dad tells it, the difference in the waterfall and non-waterfall shops was the extent to which executives preferred a sense of ultimate control over trusting expert staff. I agree that hasn't changed. What did change to force many companies to a shorter cycle was the change in expectations created by the rise of the internet. 18 months seemed a very long time on the web. (Although as late as 2004, I knew about one prominent brand name that did annual releases for their website.) So you'll see that consumer software now comes in smaller chunk sizes. That's sometimes true in the enterprise context and sometimes isn't. I know of 18-24-month projects at large, traditional organizations with controlling executives, captive user bases, and low-competition business environments.
So waterfall hasn't gone away entirely; it may have gotten a facelift and been forced to retreat to the margins, but it's still definitely there.
> the difference in the waterfall and non-waterfall shops was the extent to which executives preferred a sense of ultimate control
So you're saying that the reason shops have moved to agile is because executives have changed their psychology and are now far more comfortable with relinquishing control than in the past? Hard to accept.
What's much easier to accept is that communication tools have reduced the cost of changing specs and building consensus, allowing the same number of people to make far more changes in the same amount of time with the same amount of effort.
The incentives of an executive (or generally, of a human) haven't changed for a few thousand years.
> So you're saying that the reason shops have moved to agile is because executives have changed their psychology and are now far more comfortable with relinquishing control than in the past?
I am explicitly not saying that. In fact, the very next sentence from the one you quote is agreeing that the variation in psychology has not changed.
If you finish reading that paragraph, you'll see I'm saying what changed is consumer expectation given the rise of the Internet. People were ok with new versions every 18 months when they had to come in the mail and be manually installed. But the web got people used to frequent improvements and fast responsiveness. This caused pressure on executives to shift at least to mini-waterfall.
That change in expectations has differing impact based on the industry, and how much executives are isolated from that. Large companies with captive customers can more easily get away with longer waterfall processes, especially when the product is for an internal audience that can't switch tools. Startups and consumer-focused products tend to feel the pressure more, and so stick with much shorter cycles, some of them even doing what I'd consider true Agile.
So in sum, I'm saying some shops became Agile and many more just went to mini-waterfall with Scrum labels, the variation being partly due to the existing psychological and cultural variation, but mainly due to the extent to which customers and/or users demanded more responsiveness.
That’s an interesting viewpoint, in my experience almost 100% of budget overruns and ensuing lack of quality comes from changing requirements or rather truly thinking through the requirements for the first time after the project is about halfway done. It would be my dream to work on a project that has everything figured out upfront, alas unlike bridges, every software project is pretty unique, otherwise why are you rebuilding something that’s already done?
In my experience disastrous projects come of spending too much time thinking through requirements and not enough time actually validating them - what's perceived as "changing requirements" is more often than not actually a case of misunderstanding the requirement in the first place. Dozens of people in months of meetings can very easily specify something that is obviously impossible as soon as someone spends 5 minutes with a working prototype.
That's why I've found Agile so much more effective - if you get a system that works end-to-end at the earliest possible stage it becomes so much clearer what's possible and what's not, and your design discussions are much more productive. You still have to make design decisions and tradeoffs, but you get to make them much later on when you have much more information to work with.
Oddly, most truly disastrous projects I have seen, have had no small amount of over engineering.
There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs.
Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.
I've seen this too. My conclusion is that this happens when people who are strong communicators and networkers - influential even if they don't think of themselves that way - either misunderstand the need or value of some component, or they wilfully use the project to do a thing that they personally want to do. This "project as a vehicle for something else" is really catastrophic and is often a way for someone to exit the organisation with a gilded CV. What's important is that it screws everyone else. On the other hand, the obsessed super engineer is not being malicious, they believe wholeheartedly that they are right. The thing is to recognise which is which... one is a cancer that must be cut out fast, the other is a hugely valuable contributor who is misaligned (and in this case much listening is in order as it might be the project is misaligned).
Twitter also failed regularly. The cost of failure varies, but in government work their are non zero risks of your failures making national news. For example the healthcare.gov’s failure the scale.
Twitter only scaled well when they started over rather than trying to use Agle to bolt on add hock solutions.
Yeah, I don't think software changes are actually cheap at all.
The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements.
> Now that dev tools are so good and computers so powerful, spec changes are cheap
I‘d like to differ. Changing the spec in every iteration is going to end you up with a suboptimal architecture, because at some point devs will stop tearing it all apart and just refactor „a bit“. Of course, that’s different for various project sizes - a mega project like FB obviously has to allow changed requirements, all the time. But I‘m not sure that should actually be called a project.
An atomic project should be of finite duration, and it should be finishable. Ideally not longer than 6 months. For those, requirements should not change much after a short cool-down phase. Just like tasks/stories/features/epics, projects are made up of different levels with very different aspects.
I find the ideas presented in the article appealing, because they are refreshingly realistic even for one-off freelance projects to pull off.
I've heard before that Making software is like making a bridge... Except the design part takes almost all of the time, and building takes minutes at most.
It also works better than agile when you’re buying software and you can’t just sign on on something without knowing what it’ll cost. Which is 99% of the time in the real world.
I think the golden middleway is somewhere in between, where you do the upfront analysis and figure out what to do, why you’re doing it and with whom, but don’t lock everything down tight in an unchangable requirement specification.
I think methologies like service design can be really great, if you don’t know what you’re trying to do. I think agile methologies are decent for the actual development of software, but a software project is so much more than that, including benefit realisation.
I think it’s embarrassing that we can plan and build a organic house for thousands of employees, and deliver on time and within budget, but we can’t build a piece of software to handle the digital workflow of labelling and storing documents.
Hell, we plan city infrastructure and succeed, every year without going over budget. But building a piece of software to digitise vacation registration? Fuck no, that’ll go one year over time and cost four times what we expected, even though you could rightly build the same thing in a few months as a single focused developer.
I’ve seen all sorts of project management models in action, none of them work for software. But agile is especially stupid from the buyers perspective. When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either.
Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.
'Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.'
Even if this is the case, PM still responsible for blown deadline, because they did not estimate their teams delivery velocity correctly.
They should either:
a) give longer estimate, if their team quality is low.
b) pay more to hire better, more productive developers.
A good PM will also account for uncertainty in devs estimation, by padding etc
Predictability of delivery is what ultimately counts, not giving optimistic deadlines.
> When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either.
You hope you don't. But halfway into construction, everyone might realize the blueprints attached to that contract don't cover some detail you really need. Then if you want that done, you'll have to pay extra. That's why change orders are so common in construction.
I'm quite sure the work preparation stage of construction (design, technical drawings, calculations, planning, etc) is usually rather iterative. It's just that when creating software, the project is done when we complete that stage.
Quite plausible. Software is ontologically different from the real world where everything takes times and lots of energy. And also you don't have recursive towers of processing stages. Software attracts the idea of agile/adaptive/recomputed almost analog (continuous stream) that is clashing with clear and solid planned steps.
I'm don't think software is necessarily "different" than many other types of projects. To implement agile project management you need high-quality instant communication. Tools like e-mail, video-conferencing, and Github made agile possible.
In a pre-internet era, where every change could take weeks to permeate an organization, change became extraordinarily expensive. With faster communication, the cost of change decreased, and made agile development more attractive.
It's not the same. A computing system is theoretically capable of recompiling itself on the fly; traditional projects don't even think of that kind of frequency.
Waterfall seems more like a factory assembly line to me, on each step someone will add a component and at the end you will have the full product, its just in software usually you don't know what the end product should be when you begin.
It's amazing how many times over my career I've watched and participated in debates on software development methodologies. HN seems to have one at least annually.
The author of the post does a good job summing up the issues. I only skimmed it but have to agree with everything I read.
I can add that there's a time and place for Waterfall: when requirements are well-known and static and when lots of oversight is needed. I can also add that no methodology has successfully dealt with the mega-projects we read about in software engineering case studies.
Building stuff, whether it's software or anything other product, is hard. It's messy. Often the requirements are not well thought out. In the situations I normally deal with, a requester can state a high-level strategic goal for the organization, but not how the software can get us to it. That's where my team gets involved. One engineer I respect highly once said that we need to get something in front of the requester quickly so that we can find out what (s)he doesn't want. By process of elimination, the idea foments in the requester's mind and requirements become more concrete.
Again, though, some organizations are very static or they deal with software whose failure can be catastrophic. In those situations, the highly bureaucratic Waterfall method can be helpful at least for slowing down the process.
I think the "Mythical Man-Month" does a great job accounting for the process and implementation of a mega-project. But as Brooks even stated, there's "No Silver Bullet."
I don't know about an accident, necessarily. It's got two important things going for it.
1. It lets everyone involved cover their ass. "All I'm responsible for is my function. There was a problem with my inputs, so no wonder my outputs suck. Don't blame me, my inputs are all right there in black and white, and you signed off on them."
2. It gives people the comforting feeling that they can predict the future, and then do things like plan budgets or negotiate fixed price sales contracts. Predicting the future is irresistible for people who handle money. As fictional as it might be.
So those two things, I think they have a momentum of their own, and without constant vigilance and effort, for most domains, waterfall development is a sad, inevitable default.
Completely agree. The recent discussion here about Jira confirms this: Any new software development methodology -- however well-intentioned -- eventually will be distorted into the waterfall model with a sexier name. The incentives for doing so are much more powerful than the incentives for sticking with a truly better development process. If we want to change this, we have to change those incentives.
I agree with you, except I’d say that the ideal of Agile, the manifesto itself, responding to change, etc., is a significantly worse workflow process than Waterfall (which is also bad).
Agile prescribes so many things that are absolutely pushover, toothless concepts in the face of basic corporate bureaucracy that it really is a case where it’s the tool’s own fault that everyone uses it wrong.
I’ve seen Waterfall used successfully. Not often, but on a few defense research projects it was great.
I have never once seen or heard of any Agile process anywhere actually working with any degree of success.
I’ve heard plenty of people indulging some No True Scotsman nonsense about how a “real” Agile process can succeed, and then proceeding to write a diatribe about their “real Agile” workplace that is clearly so fucked with dysfunction that the person writing can’t even realize they are only shipping software in spite of Agile and not the least bit because of it. But I’ve never ever seen or heard of it actually causing success, not across any of the countless blog posts or anecdata on HN, and not across 6 different jobs at companies of various ages, sizes and capitalization levels all using Agile withvarying degrees of formality, pragmatism, Agile coaching and consulting, etc.
None of it made any difference. Agile just hinders people. It is not capable of doing otherwise.
Now, having said all that, I do agree that for processes that can work, processes that are not overly prescriptive, let different teams do things different ways at different times based on different goals or different personnel, let teams ignore the customer when needed, enforce long-term planning when needed, do pure research with no time boxing when needed, etc., then for those kinds of common sense, non-Agile processes, we have to change the incentives so that people are aligned to care about the objective truth and objective outcomes required to get the job done.
It wasn't perfect, and learning how to do it was painful, but once we got going it worked great.
We didn't adhere strongly to any one guiding principle, but being able to use sprints to align work across multiple teams, and as a planning mechanism for management was incredibly useful.
"Our breakdown of that feature means it'll take 3 sprints in total, the last of those sprints the work is split across 2 devs, if one of those devs gets reassigned it'll take 4 sprints."
Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating, and knowing how far to break work down. There is of course a trade off between gains from breaking work down, and time spent doing the breakdown! Several months were rather painful when we went too far one way, and then inevitably the next few months swung too far in the other direction.
But in the end? Yeah it all worked out. We figured out what length of sprint worked for us, what type of estimation system to use, and how much work to put into our planning sessions.
FWIW one of the huge benefit of doing the work breakdowns is being able to see opportunities to parallelize work, and also to see natural stopping points.
My team was lucky to have someone almost full-time dedicated to managing our workflow, without that the entire process wouldn't have been tenable. The tools just suck up too much time.
I guess it would all hinge on what exactly is meant by “we developed a process that worked for us based on agile core principles.”
The risk of No True Scotsman applied to Agile is that you go and find a process that works, and because it works, it gets labeled as Agile, even though it may (likely) have nothing to do with anything that could be said to resemble Agile.
> The risk of No True Scotsman applied to Agile is that you go and find a process that works, and because it works, it gets labeled as Agile
That's more consistent with the Agile Manifesto than calling any of the concrete development methodologies that come out of the Agile community “Agile” merely because they emerge from that community.
Choosing the process that works in practice for your team in their concrete situation is pretty much the whole of the Agile Manifesto. The failure of the Agile community is that it has almost no literature or body of knowledge oriented toward how you do that, just a bunch of concrete development practices without tools for adapting them or choosing among them.
OTOH, tools operationalizing essentially the values of the Agile Manifesto do exist in the Lean community. (Which is why I sometimes, when feeling particularly snarky, call Lean “Agile, but for engineers” or “Agile, but usable”.)
If this is true, it means Agile has no definition, it’s a vacuous term. If “Agile” == whatever works, then Agile’s not a thing, it cannot differentiate between what works and what doesn’t, and it can’t predict in advance what will work.
It reminds me of a great LessWrong quote, “If you can explain every outcome equally well, you have zero knowledge.”
> If this is true, it means Agile has no definition, it’s a vacuous term.
No, it's just the definition is not in the space you (and to be fair, lots of other people) seem to want it to be in.
> If “Agile” == whatever works, then Agile’s not a thing, it cannot differentiate between what works and what doesn’t, and it can’t predict in advance what will work.
“Whatever works” is definitely a thing; however, the kind of thing it is, as you note, is not a tool for predicting what will work.
“Agile”, though, while not that kidnd of tool, is not exactly “whatever works”, either (and here I'm talking about the Manifesto; it's worth noting that the Manifesto, the Agile community, and the concrete methods labeled as “Agile” are not precisely in alignment); it is a critique of (accusation against? hypothesis about?) the status quo at the time it was written attributing its disappointing performance in finding what works to three errors:
(1) Failing to recognize that what works, at the level development practices were generally recommended, is not a universal across software development, but a function of the particular members of the team and their relations, which not only vary between efforts, but vary over time within one effort.
(2) Prioritizing written artifacts of the development process more than is justified by their utility to software development.
(3) Preference for arms-length negotiation and implementation to collaborative involvement of the customer with the development effort.
No, really, it would mean the term is literally vacuous, as in an actual tautology, Agile succeeds because Agile is defined to be whatever has succeeded. It’s not really possible to disagree on it.
Breaking down features into smaller chunks isn't waterfall. There seems to be some confusion that there are two ways of delivering: agile vs waterfall and if it isn't one then it is the other.
Agile software development is about 'advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change'. See https://en.wikipedia.org/wiki/Agile_software_development
For a very large feature, agile deliveries will split that larger feature into chunks and deliver those. That reduces risk and can get feedback ('adapt') to how that 'chunk' has gone.
There is one problem in that the design, vision or story (high level requirement) can get lost in all of this. This is how many agile projects end up, doing churn instead of satisfying the user more with every iteration.
Typically this results in a mishmash and never finishing. (instead the thing is dropped)
Bioshock itself still has game breaking bugs nobody will ever fix - and its story wasn't too iterative.
The process is not immune to building the wrong thing at all - devs always want to side with easier vs better and agile pushes them even more by forcing things to be independently chunked...
> “Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating”
I’m skeptical of this, because even working in a team that had many person-decades of experience of estimating software delivery, the consensus opinion was that nobody’s estimates are worth anything and that the business would actually be worse off using estimates even as mere guidelines than just working on things, hitting blockers, and adjust expectations as you go.
But regardless it sounds like your process was fairly different from Agile. The only superficial similarity was having some notion of a regular sprint cadence, but cadence based scheduling is common to many project management strategies. It sounds like you had something very pragmatic for your exact situation, which is great — that’s the whole goal, the very thing Agile paradoxically does not conceptually support.
I've worked on an agile team (specifically XP). The idea there was that estimates diverge from reality past 3 weeks. If the estimate is bigger than 3 weeks, break it down into smaller pieces and estimate those.
Also, XP was very into feedback. "How much did we get done in the last two weeks? How much did we think we were going to get done? Let's use that to adjust our estimates going forward." This was actually done with story points, so that when we "adjusted", the units of estimation stayed the same (that is, an item that was 4 story points before was still 4 story points), but we realized that the team was only doing 12 story points per 2-week sprint, instead of the 20 that we expected.
And yes, it worked very well for us, and it was truly agile.
And I disagree with you that agile doesn't allow something tuned for your exact situation. Or rather, agile allows that, even if Agile doesn't. We'd sometimes say "For the next two sprints, we're going to tweak our approach in this specific way. If it works, we'll keep doing it that way." That is, our process was one of the parameters we could adjust.
> I've worked on an agile team .... The idea there was that estimates diverge from reality past 3 weeks. If the estimate is bigger than 3 weeks, break it down into smaller pieces and estimate those.
That's not really all that agile. The agile teams I've worked on try to break things down much smaller, ideally half-days or less. Any estimate of how complex some task is, is often off by a factor of 2, sometimes much more. Only by breaking down tasks into small bits can you really start getting a handle on your velocity and complexity of a project.
It’s so funny to hear things like this. I’ve worked in places where veteran, highly trained Agile coaches insisted that estimation had to be highly accurate at the quarter-long OKR level all the way down to daily programming task level.
It’s part of the problem with Agile. It means all things to all people, very much like denominations of a mainstream religion.
You’ve got your small estimates people, you’ve got your big estimates people. You’ve got your no-meetings people, you’ve got your tons-of-meetings people, you’ve got your every-iteration-has-to-be-shippable people and you’ve got your what-about-fundamental-research-(and-don’t-you-dare-say-spikes-or-timeboxing) people.
And all of them believe their way of doing it “is Agile.” So the term pretty much has no meaning in practice apart from some juvenile Ten Commandments sort of phenomenon where the only commonality in all of it is time-waster meetings like stand-ups and backlog grooming.
> all of them believe their way of doing it “is Agile.” ... the only commonality in all of it is time-waster meetings like stand-ups and backlog grooming.
Have you ever tried alternative approaches to software development? I suspect that the main reason you're frustrated with Agile is that you have never dealt with it's predecessors. There is undoubtedly wasted time in Agile, but with other processes, it seems like the goal is to maximize the wasted time.
I do believe smaller chunks are better than larger ones, but I'll happily take larger chunks over going back to a waterfall process.
I have worked on many Waterfall projects while at a defense research lab in the late 90s / early 00s. For several projects, it worked quite well, though none of this has anything to do with Waterfall being good. Waterfall is pretty bad, but success is at least possible because it allows for long-term commitment to a plan that can’t (as) easily be subverted by politics as in Agile, where politics doesn’t even need to do the pretense of coming up with an excuse because the built in excuse to mandate ill thought out context switches is ever present.
What I have seen really work are self-organizing policies where teams decide on their needs on a rolling basis. No prescriptive requirements.
For example, Agile might suggest teams can determine what type of sprint length matters for them, within some generally narrow range of sprint lengths.
But a good process will have no concept of a sprint or regular cadence, since it will be allowed to be different for each person and vary by project.
Another example may be that Agile offers flexibility to do research tasks as “spike” tickets or time-boxed regular tickets, so that the scope of a research task is forced to fit in with the iterative nature of Agile.
But a good process will not define a limit on the scope of a research task and will trust the engineer or implementer to choose when to curtail the scope based on the that person’s trusted understanding of the trade-off of probing more vs doing work on some other area, and that other people are likely not in a position to know better if more research investment is better than context switching.
Agile might suggest that at the completion of every sprint, the overall project is in a deployable / shippable state.
A good process would instead defer to the engineers or researchers to own the decision on a case by case basis about whether extended “partially implemented” states are better than the intermittent overhead of needing each change to satisfy delivery constraints.
Basically Agile pays lip service to infinite flexibility, but in reality is quite dogmatic about prescribed process. With Agile you are “free to customize” a series of prescribed global properties like fixed cadence, iterative delivery, etc., but you’re not actually free to arbitrarily disregard those things at whatever moments you need to disregard them.
The minimum condition of any good process is that the engineers and implementers are empowered to ignore and override any and all aspects of the process’s prescribed requirements at any time for any reason, and that the principle thing the business does is to trust the engineers and implementers to selectively disregard fixed processes at the exact moments when doing so is what’s best for the product & customer.
For example one way that Waterfall permits this where Agile doesn’t is to allow engineers to refuse scope change requests from customers. It can be crucial for making a customer happy to tell them “no” and refuse to change your existing plan in the face of new requests. You won’t always do this, of course. Often you will make changes that customers demand. But a good process leaves that freedom to veto a customer request up to the engineers and trusts that they won’t abuse that freedom. Agile on the other hand intrinsically revokes that freedom and takes the attitude of not at all trusting engineers to make that decision.
Agile, despite the lip service, categorically disallows this by its very definition, all the way back to the Agile principles themselves.
Not true at all. Every increment is supposed to be shippable and better quality than the one before. So the chunks cannot be too small or you risk never shipping by never shipping critical functionality that is hard to chunk, or instead you end up with long stretches where software does not work and is not testable.
This also ignores the fact that there are evolutionary dead ends and digging yourself out of such a hole is hard.
> you risk never shipping by never shipping critical functionality that is hard to chunk
You don't have to be perfect at dividing up a big project, but you should at least try. Breaking down big pieces of functionality into smaller pieces is one of the core tenants of Agile.
In fact, your approach has far more risk. By refusing to try to break things down into smaller units, you risk not shipping anything because every feature is deemed "critical".
> I’m skeptical of this, because even working in a team that had many person-decades of experience of estimating software delivery, the consensus opinion was that nobody’s estimates are worth anything and that the business would actually be worse off using estimates even as mere guidelines than just working on things, hitting blockers, and adjust expectations as you go.
I usually agree, but we were in a bit of a special situation:
1. In house designed and built custom RTOS
2. In house designed and built custom hardware platform
3. Our entire UI framework was highly customized by us, for our specific needs
It turns out, when you control all the variables, making estimations becomes possible. It isn't easy, but it is possible!
> the very thing Agile paradoxically does not conceptually support.
Early on, a few of the more senior engineers wanted everything to be done as one big design up front. That is after all how embedded hardware projects are normally delivered. (And for hardware that makes a lot of sense, lead times are a thing!)
The idea of shipping embedded firmware every 3 weeks is considered insane by many people. Heck I think that throughout the industry it is still considered insane.
But if we had not pushed for "agile", users would have suffered. They wouldn't have gotten the incredible stream of features that they did, and the teams wouldn't have been forced to maintain constant stability and code quality.
That idea of "always ready to ship" is one of the tenants of agile. Of course having milestones is common, but "our code is always good" is not common, and it is one of agile's differentiators and adhering to that leads to systematic changes in how teams function.
2. In house designed and built custom hardware platform
3. Our entire UI framework was highly customized by us, for our specific needs
It turns out, when you control all the variables, making estimations becomes possible. It isn't easy, but it is possible!
That's an interesting set of observations there. I've been wondering for a while whether we -- as a field -- have been systematically underestimating the cost of using libraries. To my eye, the anecdata definitely seem to be on the side of projects going more smoothly when the dependencies are few and well understood.
Does anyone know of anyone doing research into this kind of thing?
I'd personally say libraries save time in net, but they add unpredictability.
Especially for experienced devs who have written framework code before. "How long will it take you to write a graphics rendering routine that you've previously written on another project" is easier to estimate than "does this feature in the graphics library we are using work?"
The library probably works, sure, but when it doesn't the problem can become an almost unbounded time sink.
Yes, I think I'd agree with that. Good judgement can potentially help to avoid the unbounded time sinks. I've met a few people who's opinions I'd generally trust in that regard, but it seems to be a hard problem, and so far I'm unconvinced by any claimed heuristic solution (in particular, the approach of picking "active" libraries on the basis of lots of commits on Github seems more likely to get you churn, rather than support when you most need it).
I also think there are some break-points where adding libraries can change the nature of a project significantly. Think of old-school Javascript, loaded directly with a few script tags. In general "new style" JS (with many small libraries, a bundler, and probably some other tooling) brings some noticeable advantages (at least for larger applications), but you definitely lose something relative to the "no tooling, just hit reload" edit-test cycle.
>I have never once seen or heard of any Agile process anywhere actually working with any degree of success.
I worked on two internal tools at Etsy that used the most agile of agile processes (no product manager or project manager, no estimation, no timelines and burndown charts, just the people who used the tools and the people writing the tools, a big physical board with cards on it representing the work that needs to get done in priority ordering, and daily standups with the manager of the engineering team. The whole team would have retros every two weeks that were kind of social, kind of project management, kind of technical.
They were the two most wildly successful projects I've ever worked on and the tools were well loved and last I checked a couple years ago, at least one was still being used about 4 years on. They solved the users' problems extremely well.
It’s hilarious for me to hear this about Etsy. I live not far in Brooklyn and have several friends who have worked there. I specifically did not apply to a machine learning role at Etsy after talking to them about the software engineering culture there.
Maybe I received wrong data, but according to their reports anyway, it sounded like Etsy would be almost a case study in bad Agile processes.
I feel like you could be very right or very wrong, but have no way to tell because you are emphatic without being specific. What are the indicators of bad agile processes, and what would be better processes?
No process "causes success", so if that's your standard, nothing will ever meet it. Also, there is no single "Agile process" that is actually Agile. The point is for teams to continuously improve over time in a direction that shortens feedback loops, decreases waste, and improves quality.
Anyhow, I've used these approaches very happily in a startup context. In particular, I've used processes derived from Extreme Programming, so we used practices like pair programming, frequent pair rotation, test-driven development, small units of work, continuous deployment, weekly retrospectives, continuous process improvement, and a number of other things. A+++, would do again. And all the developers and business people said the same thing.
Part of the problem you may be having is that most of what people call "Agile" is what I'd call "mini-waterfall". It's where a waterfall-oriented organization decided Agile was fashionable so they borrowed a bunch of labels, cargo-culted some rituals, turned up their release frequency slightly, and declared themselves awesome. I agree this is shitty nonsense. But we didn't trademark the term "Agile" at the beginning, so anybody can use the term to mean anything they like. I think this slide began circa 2007, and in 2011 I wrote about what a mess things were: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...
> Agile prescribes so many things that are absolutely pushover, toothless concepts in the face of basic corporate bureaucracy
Agile prescribes exactly one thing; adapting processes and tools to use those that work for producing working software for the actual people on the team, working in their actual context.
> I’ve seen Waterfall used successfully.
And pretty much all the concrete methods sold as “Agile” are processes that have been used successfully somewhere, too, that's usually what motivates documenting them.
Moreover, Lean (which is essentially the system in which the values expressed in the Agile Manifesto are concretely operationalized, unlike what happens in the “Agile” community where you have the Manifesto reject rote application of canned processes, and then the literature of the community is almost entirely canned processes for rote application rather than discussion of how to adapt processes to teams and situations) shows how Agile values can work, as well as “Agile” canned processes.
> “And pretty much all the concrete methods sold as “Agile” are processes that have been used successfully somewhere, too, that's usually what motivates documenting them.”
Agile is a gimmick to sell. That’s what motivates documenting different examples of so-called Agile.
It’s similar to the age old scam of sending out mailers to 2^n different sequences of buy / sell advice for n different investment decisions, and then only focusing on the people who received the advice that turned out to be correct in every case.
Say that some organization “uses Agile” (whatever that means), but that the engineers deliver a product in spite of the processes that exist “because of Agile” and would have delivered better results with less risk had those processes not existed.
In that case, Agile will still be labeled as a success story and trumpeted around as a documented example for consultants to sell, despite Agile having a negative effect.
It boils down to a causal inference problem: what would have been delivered in the counterfactual situation of some methodology that did not impose any “Agile” processes?
I’m just pointing out that in every implementation of Agile I’ve personally worked in, heard about from friends or read about in HN comments (including this thread), all of the details appear to suggest it really is just this counterfactual problem.
Agile has a subtractive, hindering effect on a process of delivering software, and due to engineers doing extra work to overcome Agile and deliver things in spite of it, the end result is that, no thanks to Agile, organizations still ship things and make customer’s happy. Just not as happy as if they did not approach work with Agile process management.
I experienced 1) in another context. We went from traditional coarse task per employee to atomic operation dedicated employees, to increase throughput. The system became blind. Everybody was applying it's tiny task on the stream, except there were always a tiny % of ambiguity that nobody cared enough (we all cared, but the stream pushed and no one upstream / manager wanted to stop and solve issues). We generated a never ending list of crap that got rejected by the final commit step.
To my reading, the article suggests that the reason why we have waterfall style development is because some guy at the DoD misread Winston Royce's paper.
"In 1985 the US Departement of Defense wanted a documented process for software development: DOD-STD-2167. Since governments and companies reused these requirements, it radiated across the world from there"
I contend that there are more fundamental reasons for this, so even if this quirk of history had never occurred, we'd still see a lot of waterfall development.
It’s no accident. Waterfall is driven by government procurement regulations. If the components of the waterfall look uncannily like a high level work breakdown structure of a project... that’s because it is.
The historical philosophy of government procurement is not just an outcome, but a competitive process. In many circumstances each part of delivery let to bid, and bidders performing design work cannot implement.
Even today, design-build (aka agile) is fairly limited in most procurement scenarios.
The thing that’s hard for people to grasp is that equity in procurement often yields increased cost and timelines. If vendors don’t price risk into bids, they lose money and fail to deliver. The government compensates with more stringent requirements that make it harder to bid, which shrinks the bidding pool, etc.
Indeed. The documents referenced in the OP seem like the eventual attempt at squeezing software development into the confines of Public Law 93-400[0], which predated the DOD Standard doc from OP by a good decade, and the Federal Acquisition Regulation which was an evolution of the Armed Services Procurement Regulation originally published in 1947!
In my opinion, these concepts are fundamental to the successful functioning of a monopsony like the government, who is also subject to it's own equitable competition laws.
Interesting, I always thought that spec-first and rigorous testing methodologies came from limitations in technology. For example without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution. I cannot imagine doing agile on a widely distributed software project as it is done today with games and their day-one patches.
>without the capacity for constant updates software had to be as close to perfect and bug free as practical before distribution
In some cases, there is a downside to incorrect behavior that is worse than "oops we'll have to patch it." This category includes anything that handles money, industrial equipment, scientific data, or anything that isn't a computer game or adtech.
Adtech loses income, not money. There is a vast difference.
Additionally you can have rollbacks - strategy applied by many banks when things break.
Funny enough, safe reverts and rollbacks require specialized up front design, it does not happen on its own. Safe automatic updates with no state loss? Same.
> Adtech loses income, not money. There is a vast difference.
Placing bids is (potentially) spending money. A bug in your adtech platform can absolutely bankrupt you in the very direct way of spending more money than you have.
From the script of The Mother of all Demos (its 50th anniversary noted elsewhere in HN)[1]:
"...the general approach for us, empirical but pursuing this monstrous goal, monstrously difficult to like building and trying empirically, and we’re approaching it evolutionary-wise because we feel that it’s a whole system problem, you need to get a person in that environment, working and looking at the many aspects of his working system that are involved in his effectiveness. That’s many more things than just these computerated tools. And in a large system like that, we need to do it evolutionary-wise because we can’t be analytic enough about it at any one point, to decide what best our next thing should be. We can only decide from here as well as we can analyze it, where we can invest our next resources to get the most return that will increase of the effectiveness of the system we have and this item down here is the term boot-strapping applied in a slightly loose sense. We’re applying that to our approach".
A good thing about waterfall is the documentation it generates. We switched to Agile 10 years ago, didn't manage the outsourcing all that well and now we are in the same position as then but without the documentation. More and more I wonder how we can get teams to write about what they are doing and teach the next team how everything works... sadly people suspect that such efforts will be used to outsource and offshore, but without them it's impossible to pick apart what's happened and rationally work out where to go next.
I've seen this a number of times. People (management and team alike) have said to me things along the lines of: "We don't have documentation, we are agile." Which, of course is not what the agile manifesto says, but is how some people have taken it.
Other sayings are "The spec is in the tickets, there is no user manual."
"We have unit tests." and so on.
All this makes trouble - Especially when you consider in order to change a thing, you should aim to understand it's salient parts in the first place - but perhaps that is not agile or the scrum way according to some.
Thinking that "Working software over comprehensive documentation" means "don't write documentation" is one of the biggest misunderstandings about agile and I feel your pain.
I really wish we went away from saying things like "OOP is bad, Agile is bad, Waterfall is bad" to realizing that pretty much any methodology can work well or not so well depending on the type of work, the people involved and lot of other factors. The same people who mess up waterfall will also mess up Agile and the people can't do OOP in a reasonable way will not do any better with FP.
In the end it comes down to constantly adapting to problems after openly admitting these problems.
In my experience even if you're not "the same people who mess up" everything, doing it right is actually kinda _hard_, and advice on how to figure out how to do it right is useful. Which is what these methodologies are, different peoples/communities advice.
Waterfall is far from perfect, but it has its place in many industries, where well customers have precisely-defined specifications.
In software, people usually don't know exactly what they want, let alone be able to precisely define it. This situation lends itself perfectly for rapid-prototyping to discover what works fast... which is basically an agile approach.
A factor overlooked in the comments below (and in the article) is that prior to roughly 2000, projects tended to be slow-moving and could be defined by specifications, requirements, and design docs before any serious coding was undertaken. So the stair-step approach referred to as waterfall was not unreasonable.
In my experience, though, what was unreasonable was the sequence "build, then debug." This meant spending huge amounts of time in debugging after the code was written and made it impossible to keep projects to a schedule because the debug/fix/rerun cycle was open-ended.
This, in my view, is where Agile has rightfully distinguished itself--by having tests developed in parallel with code, thereby making debugging an "every once in a while" activity rather than a whole endless-seeming cycle.
So we ended up with "Waterfall" being a historic accident, "Agile" turning into a cult, and with everybody knowing that great teams self-organize to top it all.
I've done a number of projects in both "waterfall"[0] and Agile.
On the "waterfall" projects, we wrote lots of documentation. LOTS. Business cases for every major feature. Requirements documents with serial numbered acceptance criteria. Design documents explaining exactly how we were going to implement the requirements, with every REQ bullet needing to be accounted for explicitly. Test plans showing how each DES bullet was implemented correctly and how it proved the REQ was satisfied. And it all had to be correct and fresh. We weren't allowed to have anything disagree or slip in or drop out. If it wasn't in the REQs, it didn't go into the software.
That meant if we discovered constraints that necessitated a change in design, we had to change the requirements. If you had to change requirements, it meant you had to reevaluate the business case.
And that meant we were prototyping while we were writing the documentation. The documentation wasn't done until the software was about 80% complete. By that time, we were mostly just working the test plans and resolving defect tickets to get to 100%.
Some of the best, longest lasting software I've ever written.
All of the "Agile" projects I've been on are what I call BTSOOP development: "By The Seat Of Our Pants". There was no attention paid to documentation at all. It was hell getting PMs to write any tickets, let alone ones that were in scope. And of the tickets that were there, most of the devs ignored them, even if the PM didn't. Some people would mark tickets done that they just didn't want to do, hoping the PM would just blanket-accept tickets at the end of the Sprint. Which he most frequently did.
The difference wasn't actually process. It was accountability for failing to adhere to the designated process. The Agile shops all had corporate policy on how projects would be ran, and nobody was ever taken to task for it. The "waterfall" shops didn't lack continuous integration and improvement or responsiveness to change, we just wouldn't sign off on a particular document until we knew it represented the final product, with linear dependencies on when different levels of assets could be approved (not when they could be worked on).
[0] our product manager called it "the NASA way", and we fairly closely followed their documentation standards, if not their engineering ones, but we were all pretty junior at the time. IDK why I was made senior other than being the best programmer there, which at the time wasn't saying much.
There is no such thing as “the waterfall methodology”. It was a strawman set up by the Agile camp to help peddle their wares. Everyone who was around back in the day knows this. If you want to know what methodology was actually being used pre-Agile check out https://en.wikipedia.org/wiki/Dynamic_systems_development_me... , you might be surprised at how familiar it looks...
If you're designing something that needs to be high-quality, safe, well-documented and is difficult to change later on, say an elevator, waterfall is a good model to consider. For a webapp startup, hell no.
It seems that hardly a week goes by without me seeing another 'agile is better than waterfall' post. We are so busy flogging this dead (and possibly mythical) straw man that we are not examining the real problems of current practices.
Good grief.. this comments section is a disaster.
Agile is far from perfect but for software development it is FAR superior to waterfall. If you think differently then you either have zero experience delivering software over the last 10 years or you work for an organization that it is clearly in denial.
I have to agree with this. What people need to understand is Scrum clearly states the framework itself is not intended to solve your problems. Rather, it's set up to unveil your problems so you and your team can solve them yourselves.
So, if you have a ton of issues while using Scrum, that's actually kind of the point. The real issue is not the framework that unveils your issues, but rather the level at which a team is empowered to fix their own issues.
Scrum is both powerful and useless depending on how empowered teams are to manage their own work. It's really that simple.
I haven't really ever worked in a company where they used straight waterfall in practice; it has usually been design and documentation up front with the main aspects discovered, followed by coding and updating some or all of the documents as you go in an itertive sort of manner re-designing as you go. Typically, the documents updated were screen shots and data formats. You can't call that agile perhpas, but I don't think you can call it waterfall, either.
My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.
Obvious, right?
What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development.
Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.