It's only half of the solution though. If the models are trained in a closed way, they can prioritize values encoded during training even if that's not what you want (example: ask the open Chinese models about Tiananmen). It's not beyond imagining that these models would e.g. try to send your data to authorities or advertisers when their training says so, even if you run them locally.

So the full solution would be models trained in an open verifiable way and running locally.

The model is only generating tokens without touching the network at all, right? How would it send data away?

Lots of people I know run models in "yolo" mode or the equivalent as well, which means it could just invoke curl or telnet to exfiltrate data.

Vibe supply chain attacks are coming btw.

Wdym? You vibe code your software. Are you saying the LLM will spit out malware?

Sooner or later, yes. What stops it , other than layers of imperfect process? And it's the perfect vector to exploit anyone who doesn't review and understand the generated code before running it locally

I think it's nice to be able to do things like rename nested structs and keep wire compatibility when upgrading two parts of the system at different schedules. Protos are neat. Think like a proto.

(Not saying the signing problem in OP is invalid of course. Just a different problem.)

Yeah. I'd say half of the work is Gödel numbering and the other half is the diagonal lemma.

This is the most apt answer I've read thus far.

But maybe not for long. When we get long-running AIs, the knowledge locked inside the AI's thinking might supplant docs once again. Like if you had an engineer working at your company for a long time and knowing everything. With all the problems that implies, of course.

at any time you can ask the model to produce documents given the latest state of the code base and at an altitude you choose.

If you break the rig on a mature oil deposit, there is a chance you will make the remaining petroleum/gas unreachable for the foreseeable future (at least at an acceptable price point). So you reduce the total oil quantity humanity will be able to extract.

Yeah. Even more than that, I think "prompt injection" is just a fuzzy category. Imagine an AI that has been trained to be aligned. Some company uses it to process some data. The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries. Pick your poison.

> The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries.

You can handle the CSAM at another level. There can be a secondary model whose job is to scan all data for CSAM. If it detects something, start whatever the internal process is for that.

The "base" model shouldn't arbitrarily refuse to operate on any type of content. Among other things... what happens if NCMEC wants to use AI in their operations? What happens if you're the DoJ trying to find connections in the unredacted Epstein files?

We want a human level of discretion.

Organizations struggle even letting humans use their discretion. Pretty much every retail worker has encountered a rigidly enforced policy that would be better off ignored in most cases.

Yes, because humans would never fall for instructions embedded in data. If they did we'd surely have a name for something like that ;)

By the way, when was the last time you looked out of your window?

Yeah, any kind of aid (e.g. food or medicine) allows the people you're aiding to spend more on the military if they want. I guess the only way around it is to set limits on someone's military capability and make aid conditional on not crossing these limits.

I agree sci-fi is an outlier on this, but I also think all stories compete on setting to some extent. Fantasy most obviously (Tolkien, JK Rowling). But also for example the Jazz Age setting of The Great Gatsby contributed a lot to the novel's popularity and was a bit fictionalized, hard boiled detective writers like Hammett or Chandler wrote about a crime-filled world that was fictionalized for appeal, historical romances about lords and ladies are super fictionalized and so on. Writers try to put appeal into everything, that's why they're writers.

Larry Niven isn't referring to merely an "unusual" setting in his quote (which I've never managed to find referenced online, unfortunately), but to the way in science fiction you are creating the setting from scratch. Gatsby is set in the Jazz Age, and you can pick up some aspects of it from that, but it is still in the stock set of settings the author expected you to have some ideas about, so it doesn't explain how cars work or how doors open. And by that, I don't mean the sort of "explain" at an engineering level, but things like "how combadges work" in Star Trek, i.e., when they work, when they don't, what can be sent on them, what failures they are prone to, etc. Even something as fantastic as Tolkien is still generally set in a particular milieu and he is adding very skillful and numerous brush strokes to a genre that existed already.

You've read many stories set in all the settings you mentioned. You have never read a story in which the fundamental shape of space-time is two time dimensions and two space dimensions before, unless you have also read Dichronauts. This is the supplementary material to the novel, which is mostly not in the novel and is not the story itself, just the background: https://gregegan.net/DICHRONAUTS/01/World.html You don't need that provided for something set in the Jazz Age, or a fantasy story explicitly based on myths that had been floating around for centuries, or a historical fantasy. Someone could write some equivalent, but you don't need it; it's already loaded into your head. That's the point.

I'm thinking more and more that there's an ethical problem with using LLMs for programming. You might be reusing someone's GPL code with the license washed off. It's especially worrisome if the results end up in a closed product, competing with the open source project and making more money than it. Of course neither you nor the AI companies will face any consequence, the government is all-in and won't let you be hurt. But ethically, people need to start asking themselves some questions.

For me personally, in my projects there's not a single line of LLM code. At most I ask LLMs for advice about specific APIs. And the more I think about it, the more I want to stop doing even that.

The same here. I find big AI-corpos pretty evil and drastically misaligned with broader well-being of the society.

I would also add: if you're paying, supporting their cause with your money.

Sometimes I would like to have magical make-my-project tool for my selfish reasons; sometimes I know it would be a bad choice to fall behind on what's to come. But I really, really don't want to support that future.