When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process.
I consulted to the DoD and this pattern was repeated, over and over again.
Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built.
It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.
I've participated on several >$10M Canadian Gov't IT projects over the years and you nailed it.
Business groups will fight tooth-and-nail to keep their existing processes - partly to maintain status-quo, but also because the folks who represent the business know little about system's analysis/design, so they just simply aren't in a mindset to re-engineer.
Combine this with the vendor's (not so hidden) agenda of milking the gov't and it's a recipe for disaster... every damn time.
I've consulted to the Canadian Government on both a data science project and on cyber security as well. The technical people inside GC know what they need and it isn't IBM building custom software for something like payroll. They should use off the shelf stuff for the 95% of employees that have run of the mill circumstances and just use humans to handle the super long tail of bespoke needs that no other Canadian employer has to deal with.
The problem is that the political side of the government doesn't trust their own technical staff and they get convinced by these horrible consulting companies like IBM that the only way to write good software is to spend hundreds of millions on it but don't worry it will pay for itself with all the cost savings on staff.
They try to minimize risk through bureaucracy but they end up getting the opposite of what they want. Less useable, less secure software at 50x the price. What they would do if they understood software is communicate to the public about how good software projects need fast iteration and that mistakes might be made sometimes, but that the important thing is that they are easy and fast to fix and that there are risk mitigation strategies to stop mass leaks when penetrators get into networks.
Also, the CSE should just be empowered to take GC servers offline whenever they don't conform to a predefined list of security practices. It's 2018 the fact that all of GC isn't on HTTPS / HSTS is quite frustrating and it requires all departments to get on HSTS preload lists because of the stupid way gc.ca is a subdomain of .ca.
A factor here that might be overlooked (and I'm not making excuses for the bad process you're describing) is that changing business processes means re-training all your non-technical staff (and everything that comes with that - training materials, manuals, forms, public-facing processes, etc).
In a unionized environment like the Canadian federal government, that would likely incur not only a fairly high cost (but less than a failed technology project for sure), it's also a massive HR undertaking that business-side managers and leaders want to avoid at all costs.
So they'd rather push all that work onto the technology teams to ensure process remains the same, even if it's the wrong thing to do.
My personal anecdote on this: years ago in a past job, I led a team that worked on a project for Bell Canada to build a retention portal for their call centre agents, and the requirements were full of crazy and sub-optimal flows and features, and every single one we questioned was justified with "any other way would require too much re-training of unionized call centre staff".
In one case, for a call centre in New Brunswick, the union actually had a clause that major process re-training could only be done every 2 years, regardless of cost.
Governments everywhere would function a lot better if there were more people like you in them.
I'm probably preaching to the choir, but government managers are deathly afraid to stick their necks out and say "no" to the ridiculous spec-by-committee approach to projects.
It's a lot easier to shift the blame to contractors, particularly when most projects take so long that they're obsolete by the time they go live.
I've seen quite a lot of this too, though from other vantage points.
I don't disagree with anything you say, and would probably go down a similar route. These projects have enormous and unacknowledged risk. One of those risks (the most common pone) is that the project does actually arrive at completion, is good enough to go into use, but it's still pretty crap. Not better than what could have been achieved at a fraction of the cost. You should (as CEO, CIO) take this as a given, and act accordingly. IE, minimize these projects and find alternatives.
So, this isn't a criticism...
Here's the problem as I see it: Custom, "enterprise^" software is important, very. A lot of things could work a lot better if we had better enterprise software, much better. Especially for government, the ability to produce good software that works is tremendous. Transport systems could be better, welfare systems could be better, democratic systems could be much higher information, education.... Google maps really raised the minimum standard for public transport "timetables," especially buses. This changes public transport, enabling multi-stop routes (you would never find them otherwise), casual use, tourist use...
So... How do we do this better? Personally, I think the recurring failure details are probably a red herring. You imply problems like not having a designer, just a dump of requirements from anyone who has the juice to make them. Problems like cargo-cult design around existing processes, rather than adapting to new tools. IMO these are symptoms of bad meta-systems, the "economy" around making decisions, buying services, designing solutions, etc. Also, naturally, the incentives.
Will this be a problem forever?
^I'm at a loss for a good general term. Hopefully everyone understands what I mean.
I have had a similar experience. People hate change, and will do their best to not change their processes, ever, unless someone makes them (which rarely happens). They'd rather spends hundreds of millions of dollars inventing some new piece of software just so that don't have to spend a minute pressing a button they don't want to press.
I am convinced 90% of corporate IT software would be irrelevant if the businesses were willing to change 5 or 10% of their process.
When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process.
I consulted to the DoD and this pattern was repeated, over and over again.
Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built.
It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.