Testing an LLM judge when the human labels are biased.
How should an LLM's scores be evaluated when the human scores they're compared with are unreliable? On Hostelworld, guests rate seven aspects of each stay, but 40% give all seven the same score. This thesis evaluates an LLM scoring pipeline across 342 hostels and three models, without treating those ratings as correct. It tests whether LLMs separate the aspects better than guests do, which pipeline design choices entangle the scores, how stable they stay when LLMs have less information available, and how fine-grained the resulting scores are.
A hostel rated well on one aspect tends to be rated well on all seven. Psychologists call this the halo effect: one overall impression shapes every specific judgment. The clearest sign comes from guests of the same hostel, who all share one address: across 357 hostels, guests who rate cleanliness 4 or lower rate the location 1.8 points below their hostel's average.
Guests' seven scores mostly rise and fall together (correlation 0.77). Claude's scores from the same reviews correlate at 0.36, and the decorrelation follows the pattern of how more differentiating reviewers rate. →
Changing how Claude is asked lowers the overlap between aspects. 60% of that drop comes from scoring one aspect per call, and 40% from removing the list of aspects from the schema. →
With only a few reviews, the guests' average is more stable. Claude and DeepSeek favour certain last digits, so differences of a few hundredths mean nothing. →
Models tell the aspects apart,
and not at random.
All three models rank hostels almost exactly like guests do, but their seven aspect scores are far less alike. Where they come apart matches the guests who did not give every aspect the same score.
LLM scores are stable between runs,
but need many reviews.
Two separate tests of how far a single score can be trusted.
Each hostel was scored three times with identical settings. A score is only useful if a rerun lands close to it, measured against how far hostels differ from each other. All three models pass: the spread between hostels is 31 times the run-to-run variation for Claude, 50 times for Jev and 6 times for DeepSeek. That is enough to rank hostels. But DeepSeek's reruns vary as much as its prompt versions do, so subtle variations from prompt design cannot be measured.
Each hostel was scored again from random samples of its reviews, from about 5 to 43 reviews. At every size, the guests' own average gets closer to its full-sample value than the models do.
64% of DeepSeek's scores
end in a 5.
A score of 0.85 looks precise to the hundredth. Add the scores ending in 0 and it is 81%: in practice DeepSeek scores in steps of 0.05. Claude is not much better, with 84% of its scores ending in 2, 5 or 8. So differences of a few hundredths carry no information.
For finer scores, use a model that returns a probability per rating level and average the levels by probability, as Jev does over ten levels. Note that the guests’ score and Jev’s are both averages, so they are finer-grained by design, not because they read the text better.
Caching sets the price,
and the prompt order.
Per token, Claude costs 5 to 31 times as much as DeepSeek. What matters is the cost per hostel, and that depends on how the scoring is designed and how each provider caches. Scoring one aspect per call means eleven calls over the same reviews. With caching they cost Claude 2.2× a single call, not 11×.
Providers cache differently
On Bedrock, Claude reads cached tokens at a 90% discount, which cuts the per-aspect bill by 79%. DeepSeek through OpenRouter gives about 50% off and hits the cache less reliably, saving only 22%, so its per-aspect scoring costs 9× its single call.
Batch or cache, not both
Batch processing halves the price in exchange for answers within 24 hours, but it cannot be combined with caching. For Claude's per-aspect calls, caching saves more. For single calls, batching wins and brings DeepSeek to about 0.26¢ per hostel.
Cheap runs buy reliability
At under a cent per hostel, scoring every hostel several times and averaging the runs becomes affordable, which addresses the run-to-run variation from chapter 02. Jev goes further: at 0.08¢ per hostel it is the cheapest setup measured, and also the most stable between runs.
Six lessons that transfer
to agent design.
The setting is hostel reviews, but the same design questions come up in any LLM pipeline or agent.
Tool schemas create subtle biases.
Claude scored one aspect per call, but the tool it answered through listed all seven. That list alone kept 40% of the overlap that separate calls should have removed. The model reads a tool definition as context, not just as a format. Expose only the fields a step needs, and review a schema change like a prompt change.
Isolate judgments that must stay independent.
Scoring all aspects in one answer caused the other 60%. LLMs are autoregressive, so every score leaned on the ones already written.
A score with two decimals doesn’t carry two decimals of information.
Claude ended four in five of its scores in 2, 5 or 8; DeepSeek preferred 0 and 5. Ties are therefore more frequent than the underlying data would suggest, which makes choosing reasonable tiebreakers important.
Clean runs don’t prove a failure mode is absent.
DeepSeek occasionally returned 0.0 for all seven aspects: valid JSON that passed every schema check and would have entered the results as normal scores. Aside from regular benchmarking, check generated scores for rare, implausible values to catch long-tail errors.
Design the prompt around the cache.
Prompt caches only reuse an identical prefix. Put everything that stays the same across calls first, the tool schema included, and the part that changes last, even if that puts the instruction after the content. For Claude, this made separate per-aspect calls 79% cheaper.
Benchmark with the assumption that the LLM might be more accurate than the human-annotated reference dataset.
40% of guests gave every aspect the same score, so a model that matched their ratings perfectly would have copied that halo. When the reference labels may be worse than the model, agreement stops being the goal. Test for what a good score must do instead: separate what should differ, agree with the most careful annotators, and stay stable across reruns and smaller samples.