I wonder if anyone has ever been wrongfully convicted because of poor forensic software. Those programmers are incredibly unethical people.
I'm familiar with the food-on-the-table argument, but wow, possibly getting innocent people locked up just because you don't know what you're doing is something. It can't feel good to receive the double whammy of knowing you're incompetent and that you're incompetent enough to ruin lives.
Would I do it? I don't know. Through good fortune, I've never been, as a programmer, in the situation where I worried about losing my job and being unable to find one. So maybe those of us who feel we would never do something like this (work on critical health software while ignorant, or on forensic software while clueless) just don't know that it is easy when your livelihood is threatened. Still really unethical, though.
I used to work in pressure equipment control software.
Quite literally, the software malfunctioning (ignoring physical safety mechanisms which should prevent it) could literally cause an explosion that would destroy the lab, send the equipment rocketing in to the air if not properly bolted in to concrete, and would kill anyone that happened to be in the area when it happened. The pressure differential of the shockwave was enough to seriously injure you with sudden compression and was strong enough to tare limbs off. Even with physical failsafes, that only reduced the damage to a sudden jet of highly pressurized, unbreathable air, which was dangerous in its own ways.
I would have quit that job (even if it meant going back to painting houses or tutoring high school kids or serving coffee) if it had for a moment required that I not take every opportunity to make sure the software was as safe and predictable as it needed to be. This included consulting with staff from Boeing on good engineering practices for mission critical safety gear, using languages outside of the most popular ones (ie, not using popular ones and instead using languages with good tooling for safety and reliability), etc.
I don't think it's acceptable we hold people who make forensic software to a different standard than those who make airplanes or industrial safety systems - they're ruining lives just the same.
> I wonder if anyone has ever been wrongfully convicted because of poor forensic software.
Not only this, they've been convicted by digital expert witnesses just being lazy in their analysis and only looking at the surface level.
One of my favorite digital forensic instructors from my time in college had almost helped send a guy to prison for child pornography by half-assing his analysis before he decided that he was going to do the very best job he could even if he hated the guys guts, took a second look at the evidence before submitting his report, and discovered that it was likely a coworker who had looked up the child porn on the accused guy's work computer.
I think it's reasonable to assume that people have also been wrongfully convicted due to buggy forensics software as well, given just how hard it is to get it right.
There exists software which is designed to confuse them and convincingly plant evidence; and while that software is imperfect, they are powerless against that.
I've been a digital forensics software developer for a decade. There's plenty of buggy software, yes, but I've never heard of software that deliberately plants evidence.
FLETC and other training classes instill in investigators the necessity of manually verifying and testing their results for their reports and/or court testimony.
Most cases involve child pornography. Most of the time its known child pornography (i.e., there's a known victim, there are likely other folks who have been convicted of possessing the images). Most of the time the suspect takes a plea bargain.
It's the rare case that involves figuring out who-did-what-when events on a computer based on trace artifacts. So, the good news is that they are not the norm; the bad news is that only the best examiners (on either prosecution or defense) can typically handle them.
This struck a nerve because that's how I feel in my position. I really had no idea what went into the products that I'm not working on. I feel ignorant, but someone's giving my code the okay and occasionally throwing some back, so I must be doing something right. Someone said, "He looks like he can handle this job" and here I am with no prior experience in this particular field (I did have some programming experience). I wonder how many juniors these forensic software companies employ?
On one hand, mission-critical companies should only hire people who are both experts in programming, experts in safe and reliable programming, and experts in the domain that they programming in. (note: actual experts, not "i've-done-it-once-or-twice-so-i'm-the-department-expert-because-no-one-else-is-really-working-on-it" expert)
On the other hand, it's incredibly hard to find all three of those things in sufficient numbers to staff a team capable of handling large and complicated software. With employees in general changing positions more often (sometimes on whim alone, nevermind what the world needs), it's even harder to cultivate 5+ year employees who really know their way around the software and the business.
There is just a huge volume of information and practice required to become that individual, and you really run into daily practical concerns if we were to only hire these experts. You have to be fed challenges that teach you so you can eventually become a real expert, but you need competent oversight -- a mentor, really. And sometimes you just don't have the luxury of tackling problems that are 'small enough'. Sometimes your mentor really doesn't have time to teach to the level of detail needed because let's face it, there's still a business behind all of this. And then they should be competent in security, performance, and a host of other skills because software is just so connected and reliant on the other parts of the system.
So now you need someone with all of those previous skills and the ability to teach people well. These newly minted experts will have to pass down the knowledge as well.
How do you cultivate experts without also creating a little danger to your customers and their customers and maybe even the general population? At some point, that trainee is going to have to make change and implement things without a safety net behind him. He's got to do this enough until he becomes that actual expert, but that takes a really fucking long time, and I don't think the market is really motivating people to take that route ("Why not just make some websites for 80k a year and save myself all of that stress and an early death?").
If someone else makes a big mistake, most of that time you spent to become better becomes worthless when your firm hits the front page. Now you're associated with someone else's problem and that's going to affect your chances of a job. So not only do you have to watch out for your own mistakes, but also your peers' mistakes as well. While you may not have to resort to flipping burgers, you probably won't be able to get a job in a critical environment for a while.
Not to diminish the difficulty of accomplishing this, but there is a good answer to your question: it requires both a strong culture of quality assurance (this is what you're talking about with mentors and watching each others' backs), and a strong specialization of quality assurance, made up of the experts. Unfortunately, many software organizations treat the specialization like a joke of a useless roadblock and only pay lip service to the culture. I think this may be because lots of software truly doesn't need to be very high quality, while other software truly does, and determining which kind you're making and justifying the additional cost of quality (which is enormous) is not straightforward.
In the case of this specific article, I find it odd that the blame is laid at the non-forensic-expert software engineers, rather than the companies that employ them, seemingly without the support of a tightly integrated set of experts. There is nothing wrong with dividing labor between creating software and defining and verifying what the software does. Both jobs are difficult in specialized ways. It seems that, in the opinion of one expert at least, these companies are just doing a poor job of QA.
If you want to be horrified, read about the ludicrous bullshit that is a breathalyzer. The fundamental problem seems to be three fold: first, as anyone not an idiot would expect, animals process alcohol differently so at best you have a model relating alcohol intake with alcohol present in the lungs. Second, the programming appears to be ludicrously bad [1]. Pigs and manufacturers have fought and fought to avoid having to share code behind these devices or present any proof that they measure what they purport to measure besides, "Trust me." When competent engineers have finally gotten their hands on code, you'll be unsurprised to discover it's full of bugs. Third, even accepting the bac to gas model arguendo, it's an open question if these devices measure what they purport to measure.
Just a few representative issues when competent software engineers have got their hands on the code:
- Of the available 12-bits of A/D precision, just 4-bits (most-significant)
are used in the actual calculation. This sorts each raw blood-alcohol reading
into one of 16 buckets. (I wonder how they biased the rounding on that.)
- Out of range A/D readings are forced to the high or low limit. This must
happen with at least 32 consecutive readings before any flags are raised.
- There is no feedback mechanism for the software to ensure that actuated
devices, such as an air pump and infrared sensor, are actually on or off
when they are supposed to be.
- The software first averages the initial two readings. Then it averages the
third reading with that average. Then the fourth reading is averaged in,
etc. No comments or documentation explains the use of this formula, which
causes the final reading to have a weight of 0.5 in the final value and the
one before that to have a weight of 0.25, etc.
- Out of range averages are forced to the high or low limit too.
- Static analysis with lint produced over 19,000 warnings about the code
(that’s about three errors for ever five lines of source code).
I would suggest we move to an impairment model of punishment rather than a blood alcohol. Regardless, it's a mockery of a justice system to have defendants convicted based on the flawed source code of a flawed model of human biology, that prosecutors have successfully loopholed their way into not having to defend, and that has essentially no publicly available evidence proving it measures what it purports to measure.
edit: and just so we're clear, drunk drivers are utter assholes. But it's a basic principle that you need a high likelihood of guilt to punish people. These breathalyzers seem like utter bullshit, used mostly as a pseudo-scientific gloss to help pigs + prosecutors punish those whom they wish to punish.
I'm familiar with the food-on-the-table argument, but wow, possibly getting innocent people locked up just because you don't know what you're doing is something. It can't feel good to receive the double whammy of knowing you're incompetent and that you're incompetent enough to ruin lives.
Would I do it? I don't know. Through good fortune, I've never been, as a programmer, in the situation where I worried about losing my job and being unable to find one. So maybe those of us who feel we would never do something like this (work on critical health software while ignorant, or on forensic software while clueless) just don't know that it is easy when your livelihood is threatened. Still really unethical, though.