That means that 5% of the time GPUs were not doing any training, e.g. the server... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

WithinReason on April 19, 2024 | parent | context | favorite | on: Llama 3 8B is almost as good as Wizard 2 8x22B

That means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpathy: https://twitter.com/karpathy/status/1781028605709234613

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact