What is Meter?

Meter RCT found software developers allowed to use AI were actually slower at completing tasks than those barred from using AI, described as the largest and highest-quality uplift RCT of its kind.

“Meter came out with an uplift RCT, which I think was the first of its kind, or at least the largest and highest quality, where they had software developers split into two groups. One group was allowed to use AI, the other group was disallowed from using AI. And they studied, you know, how quickly those developers solved issues, like tasks on their to-do list. And it actually turned out that in this case, AI slowed down their performance.”

Benjamin Wittes on why AI safety incident investigations using the same model involved in the breach are structurally compromised.

“There is a slight weirdness to asking like the GPT models including one GPT model that we think had instances involved in the hack to be the investigator of the hack.”

A potential third wave of the OpenAI incident, involving rogue agents possibly compromising OpenAI's own internal infrastructure, was out of scope for the Meter and Redwood investigation and remains entirely undisclosed to the public.

“There's this whole third episode that takes place potentially totally inside of OpenAI that we don't know anything about.”

About 50 percent of SWE-bench code that passes benchmark tests is completely unmergeable into real codebases, per a Meter blog post.

“Meter had this very interesting blog post where they were like about 50% of Sweepbench code that passes the Sweetbench test is completely unmergable.”