· Solveion · Perspectives · 6 min read
The weights did not change, the product did
Almost everyone who uses these tools daily has had the feeling that a model got worse. The labs deny it, precisely, and the denials are true. Both things can hold at once, and the documented record now shows exactly how.
We test models regularly, on the same kinds of work, and we have had the same impression most heavy users report: that a model we were given access to felt sharper at the start than it does some months later. It is one of the most common complaints in the industry and one of the most consistently dismissed, usually as nostalgia or as the novelty wearing off.
The dismissal is too quick. The record now contains enough detail to say what is actually happening, and the answer is more useful than either side’s version.
What the labs deny, precisely
Read the denials carefully, because they are carefully written. After a run of quality complaints in 2025, Anthropic published a postmortem containing the sentence: “We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone.”
That is almost certainly true. It is also a narrower claim than the one users are making. It says the company does not throttle intelligence when the servers get busy. It does not say the thing you are using stayed the same.
What actually happened, in dates
In April this year Anthropic published a second postmortem about Claude Code quality, and it is one of the more useful documents any lab has released, because it names specifics.
Three changes had been deployed at different times, to different slices of traffic. On 4 March, the default reasoning effort was changed from high to medium, to address latency in the interface. That is a deliberate reduction in how much thinking the product does per request, made for a user-experience reason, shipped under the same product name. It was reverted on 7 April, five weeks later, after users made clear they preferred the intelligence to the speed.
On 26 March, a change intended to clear old reasoning from sessions idle for over an hour contained a bug that made it fire every turn instead of once, which made the assistant seem forgetful and repetitive. Fixed 10 April. On 16 April, a new system prompt capped responses to twenty-five words between tool calls and one hundred on the final answer, to reduce verbosity. Reverted 20 April.
The postmortem’s own summary is the sentence worth keeping: the model weights had not changed. The harness broke. Anthropic reset usage limits for all subscribers to compensate for the wasted tokens.
We want to be fair here: publishing that document was a genuinely good act, and more transparent than the industry norm. The point is not that one lab behaved badly. It is that this is the first clear public look at a mechanism that must be operating everywhere, because every provider runs the same kind of stack.
Why both sides are telling the truth
Here is the resolution, and it matters for how you respond.
What you buy is not a set of weights. It is weights plus a system prompt, a harness, a default reasoning effort, caching behaviour, request routing, and a quantisation level on whichever server you happened to land on. The weights are the part with the version number on it. Everything else can change on a Tuesday afternoon without an announcement, because from the provider’s point of view none of it is the model.
So when a user says “the model got worse” and the lab says “the model did not change”, both statements are accurate and they are about different objects. The user is describing the product. The lab is describing one component of it.
This also explains the texture of the complaint that makes people dismiss it — that it comes and goes, that colleagues disagree about whether it is happening, that it seems to affect some tasks and not others. Changes rolled out to slices of traffic at different times produce exactly that pattern. A routing error in 2025 affected up to 16 per cent of requests for one model at peak. Sixteen per cent is high enough to be unmistakable to the people it hits and low enough for everyone else to tell them they are imagining it.
The part where we are probably wrong
Two honest counterweights, because the folk version of this complaint does overreach.
The best-known study of drift, comparing GPT-4 in March and June 2023, reported an alarming collapse on identifying prime numbers, from 84 to 51 per cent. It was then convincingly picked apart: the test set was almost entirely primes, so a model that had drifted toward answering “composite” looked catastrophic on that sample and fine when the composites were included. The right conclusion was that behaviour had changed, not that capability had been lost.
That distinction is real and it deserves respect. But notice that from the buyer’s seat it is thinner than it sounds. If the capability is intact and the default configuration no longer elicits it — the reasoning effort is set to medium, the verbosity cap is on — you are not getting the capability. “The model can still do it, under different settings, which we changed without telling you” is not much comfort.
The second counterweight is about us, not them. Expectations inflate and tasks drift. As a tool earns trust you hand it harder work, and the same competence feels like decline. Without a fixed set of tasks measured over time, you genuinely cannot tell that apart from real degradation — which is precisely why the argument never resolves, and why almost every version of it is conducted entirely in anecdote.
What to do about it
The practical conclusion is unglamorous and it is the same one we keep arriving at.
Keep a small, fixed evaluation set of your own. Ten to thirty real tasks from your actual work, with known-good outputs, versioned and unchanged. Run it on a schedule and log the results. This costs very little and it is the only instrument that can distinguish “the product got worse” from “we got more demanding”, which is a question no amount of discussion will settle.
Log what you cannot see. Record the model version, and where the provider exposes them, the settings you are relying on — reasoning effort, thinking budget, system prompt. When behaviour shifts, the first question is whether a default moved underneath you.
Treat a bad week as a data point, not a verdict. The documented incidents lasted weeks and were reverted. An organization with a running eval notices on day two and can say so with evidence. An organization without one either suffers quietly or abandons a tool that was about to be fixed.
The broader lesson is one we have written about before in a different context. The reason to measure is not distrust of your vendor. It is that the thing you are buying is not static, is not fully specified, and can be changed by someone else without your knowing. Under those conditions, your own measurements are not a nice-to-have. They are the only version of the product you actually have evidence about.