“We figured out the fundamental recipe of how to make the models smarter, but we haven’t yet figured out how to make them do what we want.” The deadline might be closer than even the Anthros think. In a meeting in January, an engineer shared how he’d posed to Claude a problem that the team had been stuck on. The answer was uninspiring. Then the engineer told Claude to pretend that it was an AGI and was designing itself—how would that upgraded entity answer the question? The reply was astonishingly better. “AGI!” shouted several people in the room. It’s here! They were joking, of course. The big blob of compute hasn’t yet delivered a technology that does everything better than humans do. Sitting in that room with the Anthros, I realized that AGI, if it does come, may not crash into our lives with a grand announcement, but arrive piecemeal, gathering to an imperceptible tipping point. Amodei welcomes it. “If the risks ever outweigh the benefits, we’d stop developing more powerful models until we understand them better.” In short, that’s Anthropic’s promise. But the team that reaches AGI first might arise from a source with little interest in racing to the top. It might even come from China. And that would be a constitutional challenge.
