The control group just refused to exist. metr tried to re-run its landmark 2025 productivity study…
the control group just refused to exist. metr tried to re-run its landmark 2025 productivity study in february and developers said no at $150 an hour because they won't code without ai.
the counterfactual is gone, so the productivity debate is now about belief, not measurement.
Context
Verified date: METR published its update on Feb 24, 2026, saying data from the new experiment it started in August 2025 gives an unreliable signal after participant feedback, and that it is changing its experiment design. Its original study, published Jul 10, 2025, found experienced open-source developers took 19% longer with AI tools.
METR's follow-up repository covers a late 2025 study run with the same methodology.
The METR redesign and the unreliable-signal finding match the Feb 24 post. The $150 an hour rate and the specific claim that developers 'said no' because they won't code without AI were not seen in the sources read, so unsupported here, not refuted. in the excerpts read, which cite participant feedback and surveys. The follow-up data also exists and an EA Forum analysis reports a 6% sample-wide figure, so the counterfactual is not entirely gone. 'The productivity debate is now about belief' is the author's opinion.
Related work
- Assessing heterogeneity in METR's late 2025 developer productivity experiment, EA Forum ↗Mar 14, 2026.
- METR 2025 study ↗Jul 10, 2025. 19% slower.
- METR follow-up data ↗Late 2025 data.
Watch next
- Read METR's Feb 24 post in full for what developers said about the $150 hourly rate.
Sources
Provenance
The note above is reproduced unedited from the original post, first published on Threads on 1 October 2026 at 01:49 IST. Sources are the papers and datasets the note draws on.
View the original post ↗Embed this note
More notes
The air is now being asked to keep its own ledger
the air is now being asked to keep its own ledger: ecmwf’s aifs compo becomes the first ai model to forecast atmospheric composition globally every three hours, cleanair simulates 365 days of pm2.5 over china in ten seconds, and a unified framework maps six pollutants at one kilometer across the whole country. the air now files its own composition report.
read the note →The current is now being asked to draw its own map
the current is now being asked to draw its own map: china’s langya 2.0 predicts six ocean phenomena including internal waves and mesoscale eddies, a deep net called wenhai resolves eddies globally with air sea flux formulas built in, and scripps infers surface currents from the way temperature patterns deform in satellite images. the ocean now files its own circulation report.
read the note →The soil is now being asked to report its own carbon
the soil is now being asked to report its own carbon: a nix color sensor paired with generative data augmentation predicts soil organic carbon without a lab, random forest drives 74 percent of soil health mapping studies, and sentinel 2 tracks five year carbon change across france and italy from 922 samples. the dirt now files its own carbon account.
read the note →