Can Insurance Help AI Startups Grow?
New startup AIUC is offering a certificate similar to SOC 2, but for AI safety. Founders have mixed reactions so far.
The Upshot
Rune Kvist is on a mission to answer a couple of big questions for AI tools. What would make you trust an AI agent more? And who pays the bill when something goes wrong?
The co-founder and CEO of a new startup that launched this week, Artificial Intelligence Underwriting Company, or AIUC, Kvist is betting that capitalism already has a tried-and-true answer for mitigating risk, one that simply hasn’t been applied to the new technology yet: insurance.
Insurance, and a market of underwriters that can certify companies that meet certain standards, while providing a backstop for customers if something goes wrong, already underpins a bunch of industries, Kvist notes. Your home has an owner or renter policy; the lightbulb in your lamp is certified to meet safety standards. Ditto your electronics, your car, and so on.
Kvist, who attended Oxford University and moved to the U.S. to join Anthropic as an early product hire in 2022, likes to invoke American Founding Father Benjamin Franklin, who co-founded the longest-tenured insurance company in the U.S., the Philadelphia Contributionship, to help encourage the safer expansion of his city’s housing in the face of fires in 1752.
In AI, the sweeping policy debates around its existential threats, prominent enough a year ago to spur us to write a Forbes cover story about them, have taken somewhat of a backseat. (The Trump administration is interested in shaping AI policy, with the President issuing an Executive Order and giving a speech on AI last night; for most startups, however, it’s business as usual.)
But whether you’re an ‘accelerationist’ or a ‘doomer’ – and Kvist, a founding board member of the Center for AI Safety, would be pegged more on the safety-first latter side – AIUC’s leader believes that an insurance framework is designed “to be the one thing we can all agree on,” a middle path that directly incentivizes the market, and AI startups, to build responsible tools.
AIUC recently raised $15 million in initial funding led by Nat Friedman, the AI leader and investor who recently joined Meta, as well as backers including Emergence Capital, Terrain, and Geoff Ralston, the former president of Y Combinator. (Meta could be a surprise indirect shareholder in AIUC, as the tech giant is reportedly acquiring a large stake of Friedman’s fund.)
How Kvist and co-founders Brandon Wang and Rajiv Dattani plan to popularize an insurance-first approach to AI safety and build a big business in the process is a little complicated. The approach will likely include a mix of auditing, testing, managed services and a marketplace.
Upstarts has more details on what AIUC is building. But we were even more curious how startups are reacting to Kvist’s pitch. So we spent the past several days pinging a bunch of AIUC’s early supporters, as well as AI founders in the Upstarts community, to gauge their reactions. Their responses – from optimism about its potential, to cynicism about a “new racket” for startups to have to pay for — are below.