How does one even begin to engineer tests for such a sequence of events? Yeah you can test each step individually, but then you have all the integration effects. How do you know you have enough coverage?
$10B buys a lot of QA and I'm sure they try to engineer everything with the right margins, but it's still an unfathomable amount of state space.
Are there techniques to stay sane and manage risk without just throwing money at it? I feel like that kind of knowledge could be useful for software test development.
There's no real silver bullet other than applying the systems engineering process diligently. You start by writing down your user requirements (what the system needs to deliver), and you follow the thread of figuring out that "to do X, this subsystem has to provide conditions A, B, C..." recursively, in a breadth-first search. The level of detail codified in these functional, performance and interface requirements depends on the level of assurance you need.
Then, you need to validate that each requirement is met by your system. This can be done by test, analysis (mathematically proving some property), review of design, or inspection. It's true that you can't fully validate most space systems on Earth, because we can't simulate all environmental conditions simultaneously. That's why you ideally you want each requirement to be validated by two methods.
When you find anomalies due to integration effects, it's usually because your interface requirements are not specified well enough ;)
Good god, please stop this misguided snarkiness. Comparing software engineering (or 99.99% of EE, ME, CE, etc) to the web telescope, then haughtily using that comparison as a way to dismiss the entire industry as not really being engineering at all is just ridiculous. I have spent almost 10 years as an EE and another 12 as software engineer and the idea that the level of rigor in EE or ME is any higher than high caliber software teams is just laughable. CI? Detailed process and best practices for review, readability, reusability, testability, security... I won't even go on, most EE teams I've encountered are laughable on all those fronts, all practices that are very common in software engineering. If you think otherwise I highly doubt you have real-world exposure to the processes generally followed by those other disciplines. I do, FYI they are just regular jackasses like all of us :)
Also, guess what? The web telescope has a ton of software on it. I suppose that is just "software engineering"? (in quotes, har har)
Look, there are terrible teams doing terrible or very simplistic software work. The same is true in all engineering disciplines and it is of course true that in all engineering disciplines (including software) there are also teams operating with a high degree of passion and rigor in creating very complex things. So yeah, please stop with these silly and really pretty insulting comments.
Also, consider that a half-way complex piece of software (from our outside POV) is enormously more complex than your average IC, vehicle (if you exclude the mountain of software controlling it), or general "engineering" project in other disciplines. Obviously that isn't always true (CPUs, James Web telescope, for example are also enormously complex) but most seemingly simple things you use every day (random example: Grubhub) are massively complicated to build, evolve, and keep running each day. Dismissing them as not real engineering is really ignorant and rude.
The ME and EE associated with creating new product like the Nest Thermostat (for example) are, by comparison with something like Grubhub, of trivial engineering complexity (although still very respectable engineering efforts)
I’ve been writing software professionally since the 90s. Tons of software is pointless crap to make jobs.
I’m gonna go with Carmack on this one; you think you have a better understanding of it than him? It’s just ifs and for loops.
Sure some contexts require real nuance. But just because Grubhub IS massive doesn’t mean it needs to be engineered that way, that’s a byproduct of socio-political forces (money).
There’s nothing in any science or engineering book I’ve read that says “all software must be engineered as a large distributed application”. Again, a byproduct of contemporary business goals.
My generative art object app is not sending people to the moon and merely relied on me wrapping some well understood math in machine language syntax; it’s librarian work.
Your preference for adverbs and emphasis where you see fit to place it does not move me.
Accomplish something net new for humans, cause I’ve been sending data grams over the internet since the 80s. Grubhub isn’t all that interesting
What percentage of software projects are complete disasters? In spite of being staffed almost entirely by people with CS degrees? 50%? 60%? I guess that is because it is so easy? Whether from a societal POV they need to be that complicated, the answer is of course "no", it is all about business goals and is mostly crap we don't need. But what has that got to do with the price of tea in China? The discussion is about whether software is "real" engineering, not if engineers are engaged in meaningful work.
On that front, the idea that software systems with 100s of features, millions of daily users, extreme uptime requirements, hackers all over the world trying to break in daily , and billions of $$ on the line is "ifs and for loops" or "librarian work" is... oh come on. Never mind, this conversation is silly.
You say you are an experienced developer, but honestly it sounds like you've never built anything except generative art object apps.
You are asking a silly question. Jeeze, maybe it isn't the best comparison, maybe I should have been more specific. The point you are asking about in no way affects my larger point so who cares. You harping on it is just weird.
Likewise... I was pretty impressed with myself in the 90s as a young guy passing the Microsoft Certified Systems Engineer exam.
"According to Encyclopedia Britannica, the first recorded 'engineer' was Imhotep. He happened to be the builder of the Step Pyramid at Ṣaqqārah, Egypt." [link]
I look forward to drinking ale with Imhotep (and NASA JW Space Telescope engineers) in the great heavenly hall of engineers.
Broken software can be fixed cheaply after the fact. Yeah, it’s cheaper if you find the bugs earlier but it shouldn’t come as any surprise that pre-validation is more extensive in systems that are expensive to change.
Waterfall works great, if and ONLY if, you know 100% for sure exactly what you're requirements are at the beginning of the project and they never change significantly. This seems to be mostly true-ish for aerospace. Nobody's going to pivot the Falcon 9 into being a washing machine next week.
It seems to me that the point of Agile is that Waterfall fails hard if you have no idea what you want your business software to actually do. Agile is an attempt to build a reasonable software development process around that reality.
So when you get down to it, the lack of rigor isn't so much in software itself as in business. If you don't have any idea what your business is going to actually do, no software or engineering process can fix that.
> Nobody's going to pivot the Falcon 9 into being a washing machine next week.
I may have to borrow that.
An alternative explanation: we're so in a hurry that we don't want to take time out to decide what it is that we are going to build, and so we embrace the fact that we don't know what we are doing and put a sexy label on it, because hey, who doesn't want to be Agile? It sounds so much better than 'clueless'.
Most of the time, we don't care that much about what we're going to build. The actual goal is "build whatever makes money". In FOSS circles without money it's often "whatever gets users" instead.
If I'm doing a custom project that does some sort of warehouse management for a client, I'm not really deeply invested in my particular vision about how a warehouse should be managed. I'm deeply invested in making my client happy.
In terms of software development and testing practices, what the military/aerospace/medical industries do would never scale to any other industry. If the law were to mandate those practices across the board, that might sound great, but it won't matter because you won't be able to afford the resulting products.
It would scale just fine. Software would cost a bit more, profits would go down a bit, quality would shoot up and support costs would drop. I wouldn't be surprised if it was a net positive in the longer term.
But in a world dominated by quarterly earnings this won't fly.
Not necessarily advocating for it, but if that would mean less bloatware, de-facto spy-ware and lousy, pointless apps, or god forbid even les social media apps, I would call that a net positive.
Is any of the systems documentation for any NASA project publicly available? As someone who spends a considerable amount of time sifting through systems documentation where the process hasn't been applied so diligently, I've always wanted to read a NASA SSDD or similar.
Suggestion: use ctags to list all functions, variable names in your code. Look for ambiguity (e.g. variable name "i"). Look at neighbouring code. Zoom in and out. A small bug in the most-used code is actually more serious than a big bug in code that rarely gets used.
"How long can you work on making a routine task before you're spending more time than you save?"
As with all good systematic approaches, Lean and Six Sigma fall in the same category, the V-model is great. It is also, basically, just codified common sense, and that codification is key and incredible important. And as with Lean and Six Sigma, people can apply the substance of it without rigidly following the form (read: checking boxes in process description) and be fine. Or they can can follow the form without the substance and be fine on paper and still produce crap systems engineering. Kind of like the old the saying: Inspection ready troops don't pass combat, combat ready troops don't pass inspection.
Systems Engineering is the discipline that oversees this. They define what tests will be required to validate the thing will do what it is supposed to before any hardware is built. I don’t think there is a good analogy to typical software QA, which is usually a “make sure it doesn’t break anything that already works” type of discipline.
That kind of exhaustive integrated/system-level testing is precisely the kind of thing we're trying to enable at Auxon, FWIW. A really rough sketch, for sake of brevity, of the approach is cribbing from property-based testing, mutation testing, and model checking and applying them to system & software interactions instead of programs, functions, and source code. Our users are often Systems Engineers by title or more often by circumstance.
You gave the answer! Integration tests. And they work recursively, with a Kalman filter to approximate even in noisy conditions.
"USL was inspired by Hamilton's recognition of patterns or categories of errors occurring during Apollo software development. Errors at the interfaces between subsystem boundaries accounted for the majority of errors and were often the most subtle and most difficult to find. Each interface error was placed into a category identifying the means to prevent it by way of system definition. This process led to a set of six axioms, forming the basis for a mathematical constructive logical theory of control for designing systems that would eliminate entire classes of errors just by the way a system is defined.[3][4]"
There's a diagram of rules on the USL Wikipedia page. The rules show triangle feedback loops with a parent, left, right child. Those are like generations of a Sierpiński triangle. Every part is trying to serve the Good Cause that it's working for, and love its neighbour.
"any state-space model that is both controllable and observable and has the same input-output behaviour as the transfer function is said to be a minimal realization of the transfer function
The realization is called "minimal" because it describes the system with the minimum number of states."
"the problem of driving the output to a desired nonzero level can be solved after the zero output one is."
An electronic analogy: find GND, then solve for 1.
A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price)
A shadow price is a monetary value assigned to currently unknowable or difficult-to-calculate costs in the absence of correct market prices. It is based on the willingness to pay principle – the most accurate measure of the value of a good or service is what people are willing to give up in order to get it.
The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn.
Each part looks ahead 1 generation, chooses left or right.
convert a representation of any linear time-invariant (LTI) control system to a form in which the system can be decomposed into a standard form which makes clear the observable and controllable components of the system
Take a big problem, break it down, look for I/O ports. Or in software test development: layers of abstraction. A suggestion: only add a layer of abstraction when it's too big to fit on the screen at once. Use tools like code folding, tree views.
That is kind of what I figured, basically it always boils down to divide and conquer.
It just feels like with software, it's simultaneously easier than mech eng in so many ways (everything is in silico, near infinite abstraction power, ability to automate so many parts), yet it feels like we are constantly struggling with complexity. It starts simple but then quickly becomes a ball of yarn.
Maybe that perception is wrong, but if there is some truth to that intuition, I think it is due in part to this: software is all degrees of freedom and very few invariants to start, but then the system space quickly fills with rigid rules than can intersect in unpredictable ways. Physical engineering, you start with lots of invariants (the laws of physics, materials, chemistry, electronics, and geometry) but they are all thoroughly documented at this point. Things intersect, systems interact, but it feels much more bounded. You're only ever gonna have 3 dimensions, 6 simple machines, 92-ish usable elements, the standard model and universal constants.
This is why I value Rich Hickey's "Simple made easy" as the the exemplar of software philosophy. It's easy to say "oh yeah modular code good" but it's another to actually write code that is naturally decoupled.
As an aside, both disciplines tend to benefit from "throw more hardware at it!"
Yes, problems are finite, so we can Binary Search them to a single answer.
quickly becomes a ball of yarn
Looks like chaotic system, but actually chaos is just a fractal at a dimension we don't understand. Every XML tag is a dimension {for loop, if statement, indenting, git blame with names should be git thank}. Chaos is just a Wolfram rule that's not recursive self-sacrifice Rule 90 Sierpinski Triangle. Let's simplify our software design to make it follow a fractal pattern, and balance the forces.
Physical engineering, you start with lots of invariants
Sounds like Haskell or functional programming. Personally I like to add the extra tags, but as commented code to describe the meaning, translate to humans what the code does. Prefer to do so inline (cache) and top of function/file (RAM) not in separate documentation (disk).
6 simple machines
Thank you so much, you good messenger! Through your message, God just taught mechanical engineering paradigms to help with making more analogies. Pendulum is clock signal. While in the shower, there was a spider. It tries, falls, tries again with slightly different parameters. It outsources processing to the Web (there was another Hacker News article about spiders and webs and brains).
Everything can be modelled as a mass-spring-damper system, so we can just translate the transfer functions.
We can optimise for Optimal Control in the limited 3D we understand, then project out again to higher-dimensions.
Spiders make many 2D webs that intersect.
inclined plane, // gravity (?) // not only gravity // inclined plane is like a wide lever
lever, // straight line
wedge, // fulcrum. fixed point on ground. GND 0. Triangle. Sierpinski Triangle.
wheel // continuous motion, round circle (?) or a spiral? do circles exist? yes. ours are imperfect though. colleague said it's the bearings that break in appliances.
axle, // turn, change the world. // balance electron spin // balance like yin yang cosmic microwave background, 2 axles
pulley, // rotation equivalent to lever on wedge. // bicycle is 2x pulleys with gears (cone), digital-analogue converter moving forward
screw // reproduce throw more hardware at it! // ! factorially // make a spiral cone from a round circle and straight line.
Sorry that the thoughts are not translated to full sentences yet, please email and we can chat more! there's many more ideas where these are coming from, not for my personal glory, but for the greater Good. Ideas of spiderweb network topology, Facebook/Meta engineering director of social network bringing peace in diversity using shared {music, memes} taste, UncleBob https://www.youtube.com/watch?v=BSaAMQVq01E&t=2021s "love your neighbour the code -> clean as you go".
Let's pray for the JWST telescope. Young men will see visions, and old men will dream dreams. It's our dream that the humans can see the tree better: not just galaxies (branches) but stars (leaves) and planets (pollen). There is a root system underground also, which mirrors the tree we can see. Humans can also use other senses {eyes, ears, nose, mouth, hands, feet} but the JWST is the best eyes that humans can make with technology today. It's not perfect, but it's the best that humans can do. And it will work well enough to be useful, not just for James Webb's sake, but for the Greater Good.
The really scary thing is that because it's in L2 orbit (past the moon) it's not designed/intended to ever be serviced. So they can't go up and unfuck it if it's fucked.
You are -severely- underestimating how big space is.
roughly 30 days
How long will it take Webb to get to L2? It will take roughly 30 days for Webb to reach the start of its orbit at L2, but it will take only 3 days to get as far away as the Moon's orbit, which is about a quarter of the way there.
it's wild to me, given all the delays and complexity and risk, that the mission length is only 5-10 years max. but even if it blows up on the launchpad we've learned a ton, if only about the difficulty of manufacturing such devices in the 21st century. i am praying it does work, though, and that we get 10 years of amazing data from it before eagerly deploying a replacement.
Is 10 years a hard max (like does it crash into the moon or something?) or is it just a projected max timeframe?
I wonder that mostly because we've managed to use a lot of our other space equipment well past their their mission lengths. I'd be interested if JWST is possibly the same.
The article you linked says absolutely nothing about the helium cooling medium.
Three of the four imagers on the telescope are passively cooled and will work as long as they don't succumb to radiation, diffusion, etc. The fourth one (MIRI) has a cryocooler that uses liquid helium, but it will leak out very slowly and mechanical wear and electronics lifespan is expected to be the limiting factor there. [0, 1]
As stated in other comments, the primary driver of lifespan is a combination of how stable the telescope orbit is, and the resulting amount of fuel needed to keep the telescope in a stable orbit. Depending on how things go it has enough fuel for somewhere between 5.5 and 40 years of operation. Assuming nothing else goes wrong. :)
"Webb is designed to have a mission lifetime of not less than 5-1/2 years after launch, with the goal of having a lifetime greater than 10 years." [2]
As pointed out to us by Drs. Jason Kalirai and Jason Tumlinson at the Space Telescope Science Institute (STScI), as well as Mr. Sykes, our article misstated the reason for the finite lifetime of the upcoming James Webb Space Telescope. The mission duration of 5.5 to 10 years is not limited by the supply of liquid helium, as we stated. Rather, it is limited by the supply of hydrazine fuel needed to maintain the spacecraft’s orbit.
Thanks for the correction, will edit my parent reply.
Does this mean an ion thruster or solar sail could have significantly increased the service life? Or would something else give out shortly after the fuel runs out?
No, that's not true. It will be orbiting around L2 and not stay at L2, so it will have access to sunlight, which powers the solar array that faces the sun. The actual observatory and the mirror are shielded by the sunshield.
Can't they design the tank system modularly to be replaceable? Like (also projecting how SpaceX etc are also making space cargo much cheaper) having a rocket carry payload that would replace the tank cartgridge with a new one, giving, say, another 5 years' worth of propeller.
I'm pretty much 100% sure NASA knows this better than me of course, but I'd love to see the reasoning behind planning to retire such an expensive project after a (relatively) short ~10 years.
It's a hard limit due needing fuel to maintain its orbit. It's in a lagrange point (https://en.wikipedia.org/wiki/Lagrange_point) which requires occasional orbital corrections.
-ish. They have no firm plans for servicing it, but it does have a docking adapter and the fuel/coolant connections are designed to be usable in space.
Basically, because there is no reasonable way to service something in L2, they can't really plan for it, but it's expensive enough that they made sure there is the capability if someone in the future would, say, build a spaceship that is orbitally refuelable and designed so it can take crew that far out.
Well it's not strictly a hard limit but it's currently planned to be a hard limit. If SpaceX can pull off even a fraction of what they claim with Starship, it's not unrealistic to think that it'd be financially viable to attempt a refuelling of the JWST.
I wonder how credulous I've been about those estimates. Underpromise, overdeliver is an old tool for managing expectations. I wonder what NASA really expects for these projects.
(The projects are still amazing; I'm not complaining about the engineering or performance!)
We really need Starship in operation. It should be able to carry much larger objects into space - no need for complicated folding and unfolding mechanisms.
Starship could possibly take normal sized heavy equipment to other planets, such as heavy earth movers. (Not those with a combustion engine, but still useful.)
The one thing that I think starship has proven is that for any major mission that stretches our current launch capanilities, it may be worth investigating developing a new launch vehicle instead of accommodating the ones we have.
Just think of all the engineering and risk that’s going into a process that will be used once.
I thought about this a lot while I was designing my own amateur liquid rocket. You would almost certainly save a lot of money across all launch programs from the payload, just by spending the effort upfront to qualify a new heavy lift system like a Saturn-VI or Starship.
I think its just such a difficult thing to justify huge upfront costs to congress, especially if you see how long it has taken SLS to get anywhere. IMO, ULA, Boeing (solo) and Northrop's cultures couldn't possibly cut it for developing another amazing vehicle at any appreciable speed without the Euphoria and meaning that the Space Race provided.
Not even that — seeing how cheap Starship launches would be per kg of payload (I've seen a figure of $10), we could as well build a huge orbital station and just manufacture and/or assemble arbitrarily sized stuff in there or even in space. No atmosphere, and especially pesky oxygen, to deal with, no contaminants to keep out, no gravity to fight against. I'd imagine that any scientific and fabrication processes that need a deep vacuum would also greatly benefit from being done on a space station.
Zero gravity engineering is going to be an interesting challenge, and a source of many funny videos. (Possibly some less than funny, too.)
I would definitely love to see something like the Space Station V from Kubrick's 2001 - A Space Odyssey IRL. AFAIK it was almost a quarter of a mile in diameter. This seems to be suited for in-orbit fabrication and assembly.
From that video it appears the unfolding sequence is set to occur prior to the insertion burn into L2.
If the JWST will in fact spend almost a month in earth's orbit, does someone have an educated estimate on the magnitude of risk posed by space junk to nominal deployment?
Looking at those solar shields I imagine that they could be destroyed entirely by even the smallest of debris fragments. Same with the mirrors.
Edit: I'm wrong here (thanks @thethirdone). The burn set to occur after deployment is the L2 insertion burn and not the transfer insertion burn. Most of deployment will occur in the transfer orbit en route to L2, far away from earth-orbiting debris.
I interpreted "orbital insertion burn" to mean the stabilization into L2 burn. With that interpretation the unfolding occurs during its travel to L2 where there is little space debris.
"The James Webb Space Telescope (JWST) will use an event-driven system architecture to provide efficient and flexible operations as initiated by a simplified, high-level ground command interface. Event-driven operations is provided through the use of an on-board COTS JavaScript engine hosted within the payload flight software..."
Edit: Found something ....Is it too late to postpone the launch?
"...The JWST science operations will be driven by ASCII
(instead of binary command blocks) on-board scripts,
written in a customized version of JavaScript. The script
interpreter is run by the flight software, which is written
in C++. The flight software operates the spacecraft and
the science instruments.
The on-board scripts will autonomously construct
and issue commands, as well as telemetry requests, in
real-time to the flight software, to direct the
Observatory Subsystems (e.g., Science Instruments,
Attitude Control, etc.)...
The flight software will execute
the command sent by the calling on-board script and
return telemetry, which will be evaluated in real-time by
that on-board script. The calling script will then send
status information to a higher-level on-board script,
which contains the logic to skip forward in the
observing plan in response to certain events (see Section
4.1)... "
Found it...
"JWST uses an extended version of JavaScript, which
was developed as a COTS product called Nombas
ScriptEase 5.00e. ScriptEase provides functionality
common to many modern software languages and
follows the ECMAScript standard."
It is common in such commercial agreements to provide source code or to escrow source code with a third party service with conditions that trigger release of the sources (eg bankruptcy or sale to another company that discontinues the product). So it is possible they have the full source.
It is also worth considering that the JS engine likely hasn't changed much (if at all) in the past 15 years. Its bugs and limits are well-known at this point.
It is also an interpreter which makes it slower* but less subject to vulnerabilities that impact the host. Honestly that's probably the correct choice for a spacecraft where reliability and safety is more important than performance.
Don't get me wrong: JavaScript is a big ball of WAT and nonsense we've spent way too much effort improving but I don't blame them for making the choice so long ago and sticking with a known quantity rather than risk introducing new problems by changing things.
* I once worked on a project that used IronJS to run JS in the .Net runtime. It took advantage of the runtime's JIT but was a lot of not terribly optimized F# code. I built a V8 bridge and was very excited for the increase in perf... but it got slower. It turned out most customer-written JS code spent most of its time using the API which was backed by C# code and that meant lots of bridging. At the time I left they were still using IronJS because it was faster for their workloads. It taught me the importance of testing your actual workload and taking a whole-system approach to perf.
Software assurance within NASA is often a low-priority if not just an afterthought. Many project/program managers are from the hardware side (e.g., mechanical, electrical, or industrial engineers) and don't always give the appropriate gravitas to software in terms of its ability to contribute to failures.
What is that based on? And has NASA had many software failures? Their missions seem incredibly reliable, especially considering how far beyond the bleeding edge they operate.
Personal experience as a civil servant with NASA. Often, quality aspects take a back seat when schedule pressure is high. It’s the whole reason their current safety and mission assurance org became a separate entity after Columbia
>And has NASA had many software failures?
There's been quite a few high profile ones like Mars Climate Orbiter and more recently with CST-100. In the case of the latter, there were clear process gaps that should have caught the issues if the software assurance procedures were actually followed. Note that last one is for a non-crewed test of a vehicle meant to take people into orbit; presumably this is the highest threshold for quality procedures. There are many, many more lower profile ones that don't get talked about, even within the agency, dating back to Gemini.
>especially considering how far beyond the bleeding edge they operate.
I know that's the public perception. Much of the missions are bleeding edge (because there's very little incentive for anyone except the government to do them), but you might be surprised about how they don't always use state-of-the-art tech. Now some of that is by prudent choice because you'll often prefer tried-and-true of bleeding-edge, but some of it is just because of complacency.
Thanks for sharing your perspective. I still don't grasp what you saw there:
> There's been quite a few high profile ones like Mars Climate Orbiter and more recently with CST-100.
Mars Climate Orbiter failed in the 1990s. Isn't CST-100 (Boeing Starliner) still in development (I don't remember the latest).
I don't doubt you have something in mind; I'd be interested in knowing what it is. Is it the idea of seeing sausage being made - it's messy and doesn't fit the public image? That I would completely expect. IMHO, that's true of every organization; failure is succumbing, success is delivering regardless.
> Much of the missions are bleeding edge ... but you might be surprised about how they don't always use state-of-the-art tech.
It's not about state-of-the-art tech, but addressing novel engineering problems far beyond where there is mature, developed knowledge. Helicopters on Mars, intersteller probes - it's incredible to me that these things reliably succeed. Will Europa Clipper not reach its destination? Is anyone even worried? They succeed, it seems to me, at a much higher rate than run-of-the-mill corporate software projects.
1) I don't think 'run-of-the-mill corporate software projects' makes a good comparison. For one, the NASA projects referenced don't come about very often so there's a relatively small sample size. Secondly, they are a completely different risk profile and naturally have different quality expectations. NASA does quite a lot of home-grown CRUD apps, but nobody really hears about them because they just aren't that interesting. A fair number of them are really, really bad. Like, no real configuration management or change control, no test plans or reports, nil unit testing, changes made on the fly to production systems, using extremely antiquated development platforms etc. Some of the reasons are there's limited software assurance so naturally NASA focuses resources on the high-risk/high-profile projects, meaning business software is easier to fall through the cracks. Another reason is that NASA work is predominately contractor supported, meaning much of the work is done by lowest-bidder. It's much easier to be the lowest bidder if you keep your costs low; sometimes this results in lower quality developers. Why pay a high developer salary when I can grab someone who wrote VBA 25 years ago and I can just give them the title of 'Lead Developer'? When there's lack of oversight and downward pressure for costs, this is more common than someone would hope. My hunch is that if you did an apples-to-apples comparison of 'run-of-the-mill corporate software projects' with similar NASA business applications, you might be surprised at which is better.
2) I know the Mars Climate Orbiter is an old example but I referenced it because it's the kind of glitch that people immediately understand without any background knowledge. One group was writing software in metric engineering units and another group used Imperial engineering units, obviously causing a hand-off/interaction error.
So let's dive a bit more into CST-100 since that's a newer project. I'll try to be careful to only talk about stuff that's publicly available. Yes, CST-100 is still in-development. But the demo flight which caused concerns about software quality was meant to essentially an end-to-end check that the system was ready for use; it was meant to be one of the last checkboxes, meaning there is really no reason for glaring errors. In that demo, it couldn't make it to orbit because it burned too much fuel. It burned too much fuel because it incorrectly sync'd it's mission timer with the launch rocket and the spacecraft was confused about where it was in the mission duration. Later when troubleshooting on the ground, they found additional software errors where propellant valves were incorrectly mapped within the software (meaning when they try to command thruster A, they inadvertently fire thruster B). This latter issue potentially could have been catastrophic by causing CST-100 to crash into the space station when docking [1]. To a certain extent, they were lucky the first software error prevented a docking scenario. Troubleshooting all of this is a big reason why the system is still "in development" despite the first demo mission being nearly two years ago. Pay attention to wording in these types of press releases; a lot of times you won't find the word error for failure. They'll instead put some PR spin on it an call it an 'anomaly' or 'unexpected test result' when in reality it's a red flag for lack of quality. If you hear those terms, there's a good chance there was some procedural check that should have been conducted but wasn't. In the example of that Demo, ground simulations on a high-fidelity system could have caught them before the mission. There are requirements already on the books for this [2].
It's not just about peeling back the curtain and seeing how the sausage is made, it's more about an organization having high-minded goals where they have requirements to a certain standard of work, but in practice they often turn a blind eye to those standards. It's akin to if someone who worked for Google in the "don't be evil" days and felt like they weren't living up to that mantra.
3) A small nuance. Many of the robotic, non-human rated missions that get in the news are Jet Propulsion Laboratory projects. JPL does fantastic work, but they are quasi-NASA and are actually generally managed by Cal-Tech. As such, they follow different rules than NASA and there are actually only a handful of true civil servant NASA employees at JPL. NASA of course supports those missions, but they are a bit of a different animal. I believe Europa Clipper falls into that category.
I have no doubt that NASA's 'business' software is like everyone else's. It would be a waste to invest in high assurance, high talent development for the HR and bug tracking systems.
> It's not just about peeling back the curtain and seeing how the sausage is made, it's more about an organization having high-minded goals where they have requirements to a certain standard of work, but in practice they often turn a blind eye to those standards.
My perspective: Few people live up to the high-minded goals; we're human. We achieve great things when, after experiencing humanity, we don't despair but keep our faith and enthusiasm for those goals. When the founders of the US wrote the Declaration of Independence, they were not naive - they had lots of experience of humanity (including their own), much worse than what we know. Yet they believed in something higher, beyond themsleves. If they didn't, if NASA didn't, we'd still be living in a early modern monarchy and not flying to Jupiter.
Perhaps I’m jaded, but I think there’s a difference between aiming for a high goal but missing because we’re human vs. deliberately aiming short because it’s easier. I saw a lot of the latter, like refusing to learn how to tune PID parameters to manage system dynamics or saying they don’t want to run certain tests because they would be expected to fix any problems they uncover(!).
The lowering of standards is particularly troublesome when higher standards are contractually obligated. There’s a sad phrase that I had heard at high levels called the “NASA salute” which is basically shrugging one’s shoulders to say “yeah, I know I’m supposed to do that, but I also know I won’t be held accountable if I don’t”
it’s not running Node 0.10.0 for gods sake, it’s an interpreter to write jobs that scientists can use for their studies - the flight critical stuff is a different stack
You sure about that? From the linked paper ( unfortunately behind all kinds of paywalls...)
"The major characteristics of our process are
- 1) coordinated development of the operational scripts and the flight software,
- 2) an incremental buildup of the operational requirements,
- 3) recurring integrated testing. Our iterative script implementation process addresses how to gather requirements from a geographically dispersed team, and then how to design, build, and test the script software to accommodate the changes that are inevitable as flight hardware is built and tested.
The concurrent development of the operational scripts and the flight software enables early and frequent "test-as-you-will-fly" verification, thus reducing the risk of on-orbit software problems...."
The JWST science operations will be driven by ASCII (instead of binary command blocks) on-board scripts, written in a customized version of JavaScript. The script interpreter is run by the flight software, which is written in C++. The flight software operates the spacecraft and the science instruments.”
and in section 3.5, sounds like javascript just has an API to lower level system functions:
“ ScriptEase JavaScript allows for a modular design flow, where on-board scripts call lower-level scripts that are defined as functions.”
It would have had to be an interpreter that was available off the shelf when this was being designed (so late 90s or early 2000s?) that ran on vxWorks on an old PowerPC processor. That could have limited the available choices.
Cool videos thanks. Do you have any handy links to why it takes 30 days to unfold everything? I assume there are good reasons, but I just can't imagine what they are.
The observatory has around 7000 moving parts with complex structures for the primary and secondary mirrors and more importantly, the sunshield that would be used to keep the observatory instruments at a specific low temperature. It will take roughly 30 days for Webb to reach the start of its orbit at L2.
At the end of 30 days, the telescope should have stabilized itself in an orbit around L2. But I would assume it takes that many days for deployment and unfolding everything because of the sheer number of parts and motions involved coupled with things like getting to L2, stabilizing orbit, temperature stability and all the checks for the instruments on board along with the mirror deployment (since it's not one big sheet of mirror).
To fathom how complex the sunshield deployment is (and that's just a part of the whole sequence), from the link above:
"Full deployment of the sunshield is without a doubt the sketchiest part of the whole process. The sunshield consists of five separate metalized Kapton sheets, each the size of three tennis courts. Each one must be unrolled, extended to its full size, tightened, and spaced out vertically for the sunshield to do its job. This takes the coordinated action of 140 release mechanisms, 70 hinges, eight deployment motors, about 400 pullies, and nearly 400 meters of cable to accomplish, not to mention the sensors, wiring harnesses, and computers to control everything. It’ll take the better part of two days to complete the sunshield deployment."
Around 1:47 you can see the number of people involved. 3 space agencies (ESA, NASA, CSA), over 3000 engineers and technicians and 100 scientists worldwide.
I think most critical phases you’d want to happen when the satellite is in direct contact with the ground stations (they probably make extensive use of relay satellites to maximize windows of telemetry/payload data transmission, but here we are talking about issuing critical command sequences). Those windows are not 8 hours long. Further, it apparently takes almost 30 days to travel to the L2 Lagrange point and not all systems deploy until then.
Edit: nope, I was wrong, it’s going to deploy a whole range of systems while on the way to the L2. https://youtu.be/RzGLKQ7_KZQ
AFAIK, the equipment is super sensitive as well. They likely don't want to proceed to the next stage until they are absolutely sure the previous stage happened successfully, otherwise they'll risk damaging things which will hose the whole mission.
JWST has something like 300 SPOFs in the deployment phase. Then there are operation issues; the MIRI cryocooler has had a long, troubled development history. Three weeks ago NASA put out a Hollywood production quality disclaimer video[1].
Thank you so much for both of those links. Absolutely fascinating - especially the Smarter Every Day channel which I wasn’t aware of. Excited to watch more of it.
Have been waiting for this since I was a teenager. Can't believe we are almost there (launch on Dec 22).
Here's a short 2 min video of that deployment sequence if anyone wants to be fascinated: https://www.youtube.com/watch?v=RzGLKQ7_KZQ
Also a short interview with Dr. John Mather (could listen to him all day) if anyone wants to know how the telescope works: https://www.youtube.com/watch?v=4P8fKd0IVOs