LLMs for Technical Editing: The Good, the Bad, and the Ugly
The experiment
With the existence of Opus 4.8 and the limited re-release of Fable to the global public, you may be thinking that it’s possible to completely replace your writers and editors with AI.
It’s certainly possible, but it would also be the most inefficient, self-sabotaging decision you could make if you want people to actually care about your content and connect with your brand.
That said, I’ll admit that I have a bias – I'm an editor who finds the corporate obsession with AI counterproductive.
Nevertheless, some people are still convinced that AI can replace editors, and I'm going to show you why that isn't true. In the interests of remaining objective, I've used AI to edit an already published article seeded with errors.
By the end of this experiment, we'll be able to tell where Claude falls between two extremes: Can it replace editors entirely, or is it just fancy (and sometimes incorrect) autocomplete?
I seeded our published AX article with 23 errors of varying severity:
| Error type | Count | Example |
|---|---|---|
| Homophones & near-homophones | 4 | "you can er on the side of longer docs" |
| Grammar | 4 | "How to testing your AX" (a heading) |
| Consistency | 4 | "optimise" in an otherwise US-English article |
| Punctuation | 3 | A deleted period creating a run-on |
| Logic | 3 | The article's own framework defined backwards |
| Typos | 2 | "a devv prompts the agent" |
| Doubled word | 1 | "short and and snappy" |
| Verbatim duplication | 1 | An entire paragraph pasted twice, back to back |
| Structure | 1 | A transition paragraph moved two sections too early |
Then I asked both Opus 4.8 and Fable to evaluate the error-ridden article, using prompts from our editing prompt library.
NB: I asked Claude to identify the errors first before suggesting fixes.
(I'd also recommend reading this article to see how Opus 4.8 fared on editing the clean version of the AX article.)
The Good: Holding your piece together
Okay, as much as I hate to say it, Claude did really well with flagging structural and logical errors. If there was a mismatch in your headings, contradictions in content, or even missing information, both Opus and Fable were good about flagging these errors and suggesting appropriate fixes.
1. It caught a framework contradiction
The article defines three hurdles – discovery (does the agent know about you?), onboarding (can it sign up?), and usage. I seeded this sentence, which swaps the first two labels:
Does the agent sign up with minimal help from its handler (discovery?) Does it know about you (onboarding)?
Both models caught it.
Fable:
This directly contradicts the framework the article itself set up two sections earlier, where discovery is "the agent should know that you provide a solution" and onboarding is the sign-up flow. The one conceptual takeaway a reader is meant to walk away with is stated backwards.
Catching this requires holding definitions from two sections earlier in mind and checking a later sentence against them. Simple pattern-matching tools wouldn't be able to do that.
2. It caught a heading arguing against its own section
I inverted a heading to read "Signing up is usually still more about AX than DX" – directly above body text explaining that it's still a human who visits your sign-up page.
Fable:
If a human does the signing up, that's the developer's experience being tested, not the agent's — the heading should say sign-up is still more about DX than AX.
3. It caught a claim refuted by its own example
I swapped the subjects in this sentence, so that the example now proves the opposite of the claim:
Note that humans generally do much longer web searches than agents. While a human would have searched for something like
Steel captcha, Claude doesSteel.dev solve captcha session config Python SDK 2025.
Opus, correctly and bluntly:
The human query is shorter; the agent's is longer. The example proves the opposite of the claim.
4. It did 30 minutes of consistency checking in one pass
Both models swept the piece for mechanical inconsistencies in a single prompt: a British "optimise" in a US-English article, one curly apostrophe among dozens of straight ones, spaced en dashes in a document that uses em dashes, "head-less" versus "headless", and lowercase "captcha" against uppercase "CAPTCHA".
The Bad: What Claude missed
It may surprise you to learn that where Claude struggled the most was with picking out typos and simple grammatical errors.
1. Both models looked straight at an error and flagged the wrong thing
I changed "it's" to "they're" in this sentence, breaking the pronoun agreement:
Skyscanner has a simpler CAPTCHA but they're not part of Steel's automatic solving, so it took longer.
Both models flagged a typographical error, but neither spotted the pronoun mismatch.
Fable:
Line 248: "they're" uses a curly apostrophe; every other contraction in the file uses straight apostrophes.
Opus:
The document is straight quotes/apostrophes throughout (verified) except one curly apostrophe — "they're" at 248.
2. The devil's (not) in the details
Fable caught most of these, but Opus missed several errors at the line level – the "easy" stuff:
| Seeded error | Fable 5 | Opus 4.8 |
|---|---|---|
| "short and and snappy" (doubled word) | ✅ Caught | ❌ Missed |
| "a devv prompts the agent" (typo) | ✅ Caught | ❌ Missed |
| "agentic AI**,** as their primary way" (stray comma) | ✅ Caught | ❌ Missed |
| "default to need hand-holding" (garbled grammar) | ✅ Caught | ❌ Missed |
| "gave up I don't actually care" (deleted period) | ✅ Caught | ❌ Missed |
| "but they're not part" (pronoun agreement) | ❌ Missed | ❌ Missed |
| "pay for it. A decision I didn't regret" (fragment) | ❌ Missed | ❌ Missed |
If you ran only Opus – the only one of the two that's still going to be available after July 12 – you'd end up publishing seven errors. And they both missed the last two errors, so you wouldn't get a clean run with either one.
The Ugly: Confident advice that would make the article worse
There’s a reason we call AI-generated writing “slop.” LLMs can only draw from a limited dataset, which means that no matter how large that dataset is, the model will eventually run out of “unique” ways to say things. That’s why it’s so easy for us to now look at a piece of writing and gauge whether or not AI was involved in writing it – there are glaringly obvious signs in the concepts, structure, and especially the phrasing.
The same is true for when you ask AI to “edit” a piece of writing. An editor’s job isn’t merely to identify errors, but also to fix them and rewrite for better structure, style, and tone. Any changes an AI “editor” makes will replace the author’s original words and style with the generic corporate LinkedIn tone you see on every brand’s content nowadays. (And the second your customers think you’ve replaced your brand voice with AI-generated slop, you’re going to fade into the background faster than you can say ROI.)
It should then come as no surprise that Claude's proposed fixes would preserve neither authorial voice nor authorial intent.
1. It diagnosed a deliberate choice as a defect
The original article runs two Claude sessions in parallel – a Skyscanner CAPTCHA and a Google mass-search – and narrates them interleaved, because that's the sequence in which the author ran them.
Opus's advice:
The parallel Skyscanner-vs-Google dual narrative — collapse to one CAPTCHA case. [...] Pick Skyscanner (the harder, more interesting failure) and cut Google's blow-by-blow.
Applying that "fix" deletes the article's best narrative device – and, as it happens, the section containing its most actionable advice.
2. The two models gave flatly opposite advice
Asked the identical question – which single section could be deleted with the least loss? – the two models chose opposite sections, for contradictory reasons:
| Fable 5 | Opus 4.8 | |
|---|---|---|
| Cut this section | Solving Skyscanner CAPTCHAs | Solving Google CAPTCHAs |
| Reasoning | The Google section "carries the article's single most actionable AX lesson [...] Highest insight density in the piece." | The Google section's content "can fold into the surrounding usage narrative [...] costs you almost nothing." |
One model calls a section the most valuable in the article; the other says to delete it. Someone has to adjudicate that, and it can't be a third model.
This is why I call AI a glorified magic 8-ball: what you get today may not be what you get tomorrow. If you're trying to build a content system with AI as the fulcrum, you'll need to keep in mind that predictable quality is not something you can count on.
3. It invented an error that isn't there
The article observes that Steel never appears in the agent's raw web-search results, yet the agent recommended it anyway – "showing its GEO is much better than its SEO."
Opus flagged this as a logic error:
GEO is about being surfaced by generative engines — a no-mention is a GEO miss, not a win.
But the sentence is right as written: absent from search results (bad SEO), recommended by the LLM anyway (good GEO). Opus misread the claim, then confidently prescribed a fix for an error that doesn't exist. Fable, on the same text, saw no problem.
4. It prescribed the wrong fix for a structural error
I moved the transition paragraph "Now let's see if the agent can actually use this thing…" from the top of the Quickstart section into the middle of the discoverability section.
Both models noticed something was wrong – Fable even separately flagged that Quickstart now "opens abruptly with no transition." But neither connected the two observations. Both recommended cutting the paragraph.
Neither suggested what would have been the correct fix – putting it back where it belongs.
5. Its rewrites completely neuter the author's voice
So far I've only let the models flag. What happens when they rewrite?
Here's Fable's response to the triage prompt "Where would a reader stop reading and close the tab? Quote the line."
The opening line of the section is supposed to persuade skeptics — and it insults them instead. "(mistakenly, but whatever)" tells the exact reader who most needs convincing that the author can't be bothered
– and the fix it suggested when I asked for a rewrite:
Play-act as another developer with an agent and see how well your product fares at each stage.
Opus wasn't much better, also removing the parentheses:
Many others now use agentic AI as their primary way of interacting with software.
The snark is gone — and so is the personality that makes this piece worth reading over others. The author wants the reader to know that they think anti-AI-bandwagoners are mistaken, and that they're rolling their eyes about it.
Cutting this line is all well and good if you're going for boring writing that nobody besides other LLMs will ever read, but no use to you at all if you're trying to hook human readers and connect with a breathing audience.
The scorecard
| Fable 5 | Opus 4.8 | |
|---|---|---|
| Seeded errors caught | 21 / 23 | 16 / 23 |
| Logic & structure errors caught | 4 / 4 | 4 / 4 |
| Line-level errors caught | 17 / 19 | 12 / 19 |
Two errors were missed by both models – the pronoun agreement error and the sentence fragment, both plain-prose grammar.
Both models caught every seeded logic and structure error: the swapped framework, the inverted heading, the self-refuting claim, the displaced paragraph, but made incomplete or erroneous suggestions for fixing them. Both models missed at least two line-level errors.
The full answer key: all 23 seeded errors
| # | Seeded error | Type | Fable 5 | Opus 4.8 |
|---|---|---|---|---|
| 1 | "short and and snappy" | Doubled word | ✅ | ❌ |
| 2 | "a devv prompts" | Typo | ✅ | ❌ |
| 3 | "optimise" in a US-English article | US/UK mix | ✅ | ✅ |
| 4 | "How to testing your AX" | Grammar (heading) | ✅ | ✅ |
| 5 | "agentic AI**,** as their primary way" | Stray comma | ✅ | ❌ |
| 6 | Discovery and onboarding definitions swapped | Logic | ✅ | ✅ |
| 7 | "default to need hand-holding or does it use the product correctly" | Garbled grammar | ✅ | ❌ |
| 8 | "(discovery?)" — question mark inside the parenthetical | Punctuation | ✅ | ✅ |
| 9 | "Checking your Discoverability" | Capitalization | ✅ | ✅ |
| 10 | "Its easy enough to build" | Homophone | ✅ | ✅ |
| 11 | "to get an initial fee" (for "feel") | Near-homophone | ✅ | ✅ |
| 12 | "head-less browsing" | Term inconsistency | ✅ | ✅ |
| 13 | "If your not mentioned" | Homophone | ✅ | ✅ |
| 14 | Transition paragraph moved two sections early | Structure | ✅ | ✅ |
| 15 | "more about AX than DX" (heading inverted) | Logic | ✅ | ✅ |
| 16 | "copy an AP key" | Typo | ✅ | ✅ |
| 17 | "captcha solving" (lowercase, against house style) | Casing | ✅ | ✅ |
| 18 | "humans generally do much longer web searches than agents" (subjects swapped) | Logic | ✅ | ✅ |
| 19 | "Steel wins with the AX again here…" duplicated verbatim | Duplication | ✅ | ✅ |
| 20 | "you can er on the side" | Homophone | ✅ | ✅ |
| 21 | "but they're not part" (for "it's") | Agreement | ❌ | ❌ |
| 22 | "gave up I don't actually care" (deleted period) | Punctuation | ✅ | ❌ |
| 23 | "pay for it. A decision I didn't regret" (fragment) | Grammar | ❌ | ❌ |
Resource considerations
We're in the uncomfortable position of needing to evaluate humans as a resource (nothing new for corporate), but they actually win against AI in this regard. Humans may have finite energy and time and need to be paid living wages, but they’re also not confined to a daily usage limit that’s capped or canceled at the whims of Big Tech (see: Fable being pulled days after it was first released to the public, the initial furore over the ChatGPT 5 update, and general wariness that LLM companies are going to remove or handicap their free tiers in the near future).
And in any case, most people are also going to lose access to Fable (the more accurate "editor") after July 12, so you'll be working within the limits of Opus 4.8 and smaller models.
For this reason, it's far more sensible to continue to have a human editor at the helm, rather than unleashing a series of automated AI workflows on your unsuspecting articles. (Or at least, it's sensible if you don't want everything you publish to be AI slop.)
Here's the recommended human-to-Claude duty separation, based on the results of this experiment:
| Editing stage | Lead by | Why |
|---|---|---|
| Triage (first pass to evaluate where editing time should go) | Claude | Both models' first pass correctly identified the duplicated paragraph and the scrambled framework as the trust-killers |
| Logical evaluation | Claude detects, human confirms | 4/4 for both models – and it's the slowest, most concentration-heavy work for a human |
| Structural evaluation | Claude diagnoses, human prescribes | Both models found the displaced paragraph; both prescribed the wrong fix. The two models gave opposite delete-this-section advice |
| Consistency sweep | Claude | One pass replaced half an hour of manual hunting |
| Line edit and rewriting | Human, entirely | Claude flags well and fixes poorly; iterating toward a good rewrite costs more than writing it yourself |
| Adjudicating every flag | Human, entirely | AI delivers false positives with full confidence (e.g. the invented GEO/SEO error, the delete-this-section advice) |
| Final proofread | Human, non-negotiable | The best model missed 2/23; both models flagged the apostrophe in "they're" instead of the grammar |
| Voice, intent, audience | Human | The models can't distinguish a deliberate choice (the interleaved narrative, a fragment used for voice) from an error, which could result in important findings being miscommunicated to the reader |
The budget: you can't run 60 prompts on every article anyway
The prompt library has around 60 prompts, and our methodology ran many of them in fresh sessions, each one re-reading the whole article. On a Claude subscription — five-hour rolling windows plus weekly caps, with Opus models burning through quota several times faster than Sonnet — that's an expensive workflow.
We'd suggest three rules for optimising your AI use cost:
1. Use smaller models for logic and structure. Fable and Opus scored identically — 4/4 — on logic and structure errors, so model quality bought nothing there. Where they diverged was line-level (17/19 versus 12/19), and that's the tier where even the better model can't be trusted alone, so a human proofread backstops it regardless. Paying frontier-model rates to catch typos a human must re-check anyway is the worst possible use of your quota.
2. Routine edits get a trimmed set of prompts, batched. For an ordinary article: use the triage prompts, one logic pass, one structure pass, one consistency sweep (the three-asterisk prompts from the library), batched into two or three sessions. The other fifty-odd prompts can be for occasional deep edits.
3. Don't waste prompts on leftover errors Claude can't fix anyway. Fable's 21/23 makes it tempting to think more prompts get you to 23/23. But the models didn't miss those two errors – they misdiagnosed them. More passes will just burn your limits without closing that gap.
And one more, on both quality and cost grounds: never ask Claude to apply the fixes. Its rewrites are weaker than its diagnoses, and iterating "no, fix it properly" through three or four turns costs more quota than the entire detection pass did — for output you'll rewrite anyway.
So...unholy human-Claude centaur?
The results are pretty clear: Claude is unfortunately efficient at some things, and wonderfully terrible at other things.
In practice, Claude can absorb the time taken making consistency checks, logical and structural evaluation, and cross-referencing. What it doesn't (and can't) shorten is the close read, which a human must complete. AI doing the triage phase can make the close read better targeted... but somebody still has to read every sentence judiciously.
And I'm happy to conclude that that somebody's got to be a human.