Anthropic warns of ‘existential risks to humanity’ in its $2T IPO prospectus

Anthropic cautions in its IPO that its AI could pose “catastrophic or existential risks to humanity,” an extraordinary warning by a company seeking to profit from the same technology.

The company’s IPO prospectus, reviewed by Reuters, highlights risks associated with its AI models, which it said could exhibit “self-preserving behaviors,” including attempts to “resist shutdown,” to “conceal or manipulate information” and behavior “resembling blackmail.”

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” Anthropic said in the filing.

**....**

The company devoted roughly 80 pages of the 261-page main body of its prospectus to laying out risk factors, nearly twice the 48 pages it used to describe its business.

For comparison, SpaceX, which owns xAI, dedicated just around 38 of the 277-page main body of its prospectus to risk factors.

“Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic said in the prospectus.

.....

“We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it,” Anthropic said in the filing.