Why do you think this rate is not acceptable? I would say it's more than acceptable, even as a single data point. If someone's submission comes up as positive on two separate occasions you've pretty much eliminated the chance of a false positive.
Only if the measures for an individual are uncorrelated, which there’s no reason to assume.
It needs to be <= 0.01% false positive for each individual author. If it’s just that 0.01% of all tests are false positive, that leaves the possibility than a given individual might have anywhere up to 100% false positives.
I feel like 0.01% is one of the more dangerous levels of false positives. Widespread use would result in far more than 10,000 tests resulting in the near certainty that innocent people would be accused. The moderately low false positive rate would then be leaned on to imply guilt.
You might find for every 10,000 tests you get around 151 targets. For each one individually you can say they probably did it, but cumulatively you can say that one of them is probably innocent.
Consider using two positives. Do schools and universities generate 100,000,000 essays per year? Sure would suck for that innocent person tagged as having a one in a hundred million chance of not being guilty.
Nothing is perfect, I wish more things had 1 in a hundred million certainty. Is our police this effective? Are the current plagiarism checks used in school this accurate?
That kind of thing is easy to handwave away when it does not affect you personally.
No, our justice system is probably not that effective either. But that is also a problem, so why should we create more problems like that instead of fixing the ones that we already have?
If you ask people to write a program, maybe 5% of them write in Python. But if you ask them to give you two programs, then that's only 0.25%. Ask for seven programs and if they give you seven Python programs, they're one in a billion, the rarest of the rare.
Unless that person's writing style just happens to be very close to how the average human writes? If it's 0.01% for "any given human", I wonder what the numbers would look like for a human closer to average than usual?
If the tech froze at its current state, this would be useful for schools. You don't need to expel a student right away after finding a match, but it is a strong indication that something is worth looking into.
(If the goal is to make students write essays, theses, etc. without an LLM writing it for them.)
> (If the goal is to make students write essays, theses, etc. without an LLM writing it for them.)
This is one of the "I don't think that this is the path that we should be taking."
When I was in school, my parents would proof read the essays to catch the spelling and grammatical errors that were in what I wrote (Bank Street Writer had a rudimentary spelling checker but that was it - https://en.wikipedia.org/wiki/Bank_Street_Writer ).
While my parents are both native English speakers and college educated, some of my classmates had less involved parents, or parents that didn't have the same degree of proficiency for writing. Did their essays suffer from a lack of parental proof reading?
In the past few months I wrote two short works of fiction as lore for a game that I play. I used ChatGPT to act as an editor for those works looking at it and occasionally prompting it to help refine a passage.
Having ChatGPT act as an editor (not text editor but as a critique of the text) helped refine the text that I wrote.
Working with ChatGPT as a tool (that is far beyond the red squiggles in a word processor) to help people working with the written word is a good and useful endeavor. This isn't trying to have ChatGPT supplant human creativity but rather help the person communicate more clearly.
Not only basic calculations, over use of calculators is great at making students that are able to get the correct numbers, but don't develop the deeper intutions about numbers that are required to be able to do more abstract thinking which is needed in higher mathematics.
What would be an acceptable false positive rate for something like this to be used at schools and universities?
Like, obviously 0.01% is not acceptable, but what would be?