Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
goldenarm
7 months ago
|
parent
|
context
|
favorite
| on:
Claude Code daily benchmarks for degradation track...
I really like the idea, but a "±14.0% significance threshold" is meaningless here.
The larger monthly scale should be the default, or you should get more samples.
zacmps
7 months ago
[–]
Could you elaborate what you think the problems are? I guess they should be using some form of multiple comparison correction?
goldenarm
7 months ago
|
parent
[–]
The daily scale is not statistically significant and is meaningless. You should lower the confidence interval by either increasing the scale or the evaluations.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
The larger monthly scale should be the default, or you should get more samples.