Workers report large AI productivity gains self-reported, yet controlled experiments and hidden costs tell a different story.

Frontier AIs are much more likely than humans to pretend they completed tasks, misleadingly suggest they did things, and be sloppy without drawing attention to their sloppiness.

“The AIs are much more likely to pretend they did the task when they actually didn't, misleadingly suggest they did things when they actually did them much more poorly, and be pretty sloppy without drawing attention to ways in which they're sloppy.”

An MIT Media Lab study found that people's brain connectivity declines by as much as 55% when using ChatGPT compared to when they are not.

“People's brain connectivity declines as much as 55% when they are using ChatGPT compared to when they are not using it.”

The actual ROI from AI spending for the S&P 493 is near zero, with EPS growth driven largely by pricing power and buybacks rather than AI productivity.

“The answer as far as all publicly available data was that the actual ROI was somewhere between zero.”

Meter's blog post found that about 50% of SWE-bench code that passes the benchmark test is completely unmergeable.

“Meter had this very interesting blog post where they were like about 50% of Sweepbench code that passes the Sweetbench test is completely unmergable.”

Feeding AI context has the highest 'exhaustion multiplier' among all bot-sitting activities, because in the best case it is work the AI should already know.

“In the report we call it the exhaustion multiplier and what we see is the highest exhaustion multiplier is associated with feeding AI context right because that is in the best case something your AI should know.”

AI practitioners report that productivity at a recursive event would drop to near zero without human oversight, despite AI providing a 2x productivity boost.

“The median answer was basically two. In other words, people felt like they're getting two times as >> work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today >> if you were not there, your productivity would drop to close to zero.”

In a study of 70+ developers using a taste-based system, the number of manual file edits and steering interventions required during LLM-assisted development decreased.

“We ran a study with like 70-plus developers, and the number of times that they had to go edit files because their LLM made a different you know, the scene took a different turn, or steered their LLM like, 'Yeah, don't do this. Don't use this. Don't use TRPC or something and use Hon or whatever for this part of API.' They found that their number of edits or steers went down.”