Interesting example on the difference between Sonnet and Opus (from https://chat.lmsys.org/):
Here's the prompt: "Fix the subtle grammatical errors here: ```As the notice above states, it is not recommended using the latest CDN version directly in production. Though Babylon's team won't do it deliberatly, it is always possible that something went wrong during build or deployment of a specific version and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to do keep the server online.```"
Sonnet's response: "Here's the corrected version:
As the notice above states, it is not recommended to use the latest CDN version directly in production. Though Babylon's team won't do it deliberately, it is always possible that something went wrong during the build or deployment of a specific version, and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online.
Changes:
1. "using" changed to "to use"
2. "deliberatly" changed to "deliberately"
3. Comma added after "deployment of a specific version"
4. "do keep" changed to "keep""
Opus response: "Here is the corrected version:
As the notice above states, it is not recommended to use the latest CDN version directly in production. Though Babylon's team won't do it deliberately, it is always possible that something might go wrong during the build or deployment of a specific version and that the CDN could be updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online."
Here's GPT-4 for good measure: "Here's the corrected version:
```As the notice above states, it is not recommended to use the latest CDN version directly in production. Although the Babylon team won't do it deliberately, it is always possible that something went wrong during the build or deployment of a specific version and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online.```"
Only Opus correctly identifies that something going wrong is a hypothetical future event, and not something it should be discussing as having happened in the past.
Here's the prompt: "Fix the subtle grammatical errors here: ```As the notice above states, it is not recommended using the latest CDN version directly in production. Though Babylon's team won't do it deliberatly, it is always possible that something went wrong during build or deployment of a specific version and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to do keep the server online.```"
Sonnet's response: "Here's the corrected version:
As the notice above states, it is not recommended to use the latest CDN version directly in production. Though Babylon's team won't do it deliberately, it is always possible that something went wrong during the build or deployment of a specific version, and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online.
Changes: 1. "using" changed to "to use" 2. "deliberatly" changed to "deliberately" 3. Comma added after "deployment of a specific version" 4. "do keep" changed to "keep""
Opus response: "Here is the corrected version:
As the notice above states, it is not recommended to use the latest CDN version directly in production. Though Babylon's team won't do it deliberately, it is always possible that something might go wrong during the build or deployment of a specific version and that the CDN could be updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online."
Here's GPT-4 for good measure: "Here's the corrected version:
```As the notice above states, it is not recommended to use the latest CDN version directly in production. Although the Babylon team won't do it deliberately, it is always possible that something went wrong during the build or deployment of a specific version and that the CDN was updated with some code that will break your production application. We also can't guarantee 100% uptime, though we do our best to keep the server online.```"
Only Opus correctly identifies that something going wrong is a hypothetical future event, and not something it should be discussing as having happened in the past.