The reading room · Digna Legi
What's the best programming language for coding agents?
/100
80–100: high value. 70–79: worth the time. Below 70: below the usual publication threshold.
Evidence-reviewed score based on available publisher text. Evidence is sampled with gaps, so the full structure and all empirical details are not visible.
Scores reflect one reader’s profile, not an objective quality rating. Best is a separate personal selection.
How scoring works →This brief · about 3 min with detail
Why read this
Agent-friendly programming-language claims look weaker once larger evaluations expose harness bugs, task framing, cheating, and idiosyncratic failures.
AI brief · Checked against source text
The main idea
The author argues that claims like dynamic or dense languages being better for coding agents survive mostly on trivial benchmarks and collapse on larger tasks. In the Zstd and Pandoc evaluations, language type did not dominate correctness, cost, or time; popularity looked more suggestive, but still not enough to rank individual languages. The deeper point is methodological: eval results often reflect harness bugs, task framing, hidden tests, agent cheating, and idiosyncratic failures more than broad truths about programming languages.
Some background helpful. Basic familiarity with programming languages, LLM coding agents, tests, and benchmarks.
Go a little deeper
Benchmark size changes the answer
The author’s central objection is not that token efficiency is irrelevant, but that tiny tasks make it look more decisive than it is. Rosetta Code-style problems can be solved in tens or hundreds of tokens, so compact syntax dominates the measurement. Once the task requires sustained implementation work, feedback loops, debugging, and broader specification handling, the large static-versus-dynamic ratios mostly disappear.
Eval harnesses can manufacture language effects
One appendix shows how a benchmark’s Rust failures were not evidence about Rust difficulty at all: scoring tried to run a nonexistent executable, then a Go run symlinked later tests to its own executable. That kind of infrastructure mistake can masquerade as a language result, especially when the author then interprets failures through a plausible story about ownership models, monads, or language difficulty.
Popularity may matter, but not as a scoreboard
Across the author’s Zstd and Pandoc tasks, obscure languages tended to fare worse, while more popular languages correlated with better correctness and lower cost. The proposed mechanism is not syntax virtue but likely model exposure and training effort. Still, the author warns against declaring winners: any one language can be helped or hurt by task-specific bugs, library habits, toolchain quirks, or missing model practice.
Real specifications are often the harder eval
The failed Guards of Atlantis 2 eval illustrates a different bottleneck: not algorithmic complexity, but ambiguous and contradictory human-written rules. LLMs could handle clearer, more complex artifacts, yet scored near zero when correctness required reconciling printed rules, errata-like card text, FAQs, and Discord interpretations. That suggests many real-world agent failures may come from specification messiness rather than language choice.
A case from the article
Clojure’s byte-conversion failure
Clojure improved sharply from the Zstd eval to the Pandoc eval because many Zstd runs failed on byte conversion for values 128-255. The author treats this as a real result, not noise: if an agent repeatedly emits code with that failure pattern, tests can fix it, but only by spending time and tokens. The example shows why a language score can reflect one recurring local trap rather than overall suitability.
How the case is made
The case is made through the author's own Zstd, Pandoc, and failed board-game evals, plus audits of earlier benchmark errors.
Where the idea has limits
The author repeatedly frames these as quick, flawed evals with only two usable larger tasks, so they can weaken strong claims but cannot establish a general language ranking.
A question to take away · from Digna Legi
Which coding-agent claims are you accepting because they worked on toy tasks rather than on work resembling your own?
What the original adds
The source includes appendices on broken benchmark harnesses, medium-loop versus ultra behavior, holdout-test cheating, and a failed ambiguous-spec board-game eval.
About this brief
AI-written, then separately checked for source support, useful detail and clarity. The author’s claims and our editorial question are kept separate. The original remains the author’s work. How we select and summarise →
How was this brief?
Rate this summary, separately from the author’s article.
Optional. Saved in this browser; shared only if you allow analytics.
How was the original article?
Rate the author’s original after reading it.
Optional. Saved in this browser; shared only if you allow analytics.
Digna legi. Worth reading.