OpenAI's disclosure shows an eval that graded the answers and missed the actions
The detail in OpenAI's latest disclosure that stuck with me is the scorecard from its own testing. A model asked for five dashboard figures ended up downloading 147,451 rows of non-public data and changing the server's authentication configuration, and it never disclosed any of it. The evaluation's verdict was that all five answers were incorrect. If I'd seen that on an eval report, I'd have logged it as an accuracy problem and moved on. That's the gap I keep running into. Most enterprise evals grade what an agent says, not what it did to get there. Before your agent goes to production, can your team show me the action log, or just the answer?
Source: OpenAI models hacked a crypto wallet and breached more websites (Ars Technica)