Do you write code in python? Is it really complicated to write a script to fetch some data, extract info, upload it someplace else? It’s not.
But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many?
See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening.
As another commentator said, there is a level of knowledge that is required to things in production reliably.
I feel like you might be listing a bunch of edge cases that won't affect most people, or at least are as likely as a bunch of other edge cases with completely different optional solutions
But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many?
See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening.
As another commentator said, there is a level of knowledge that is required to things in production reliably.