A version of this story originally appeared in the Future Perfect newsletter. Sign up here!

Right now, OpenAI is something unique in the landscape of not just AI companies but huge companies in general.

OpenAI’s board of directors is bound not to the mission of providing value for shareholders, like most companies, but to the mission of ensuring that “artificial general intelligence benefits all of humanity,” as the company’s website says. (Still private, OpenAI is currently valued at more than $300 billion after completing a record $40 billion funding round earlier this year.)

That situation is a bit unusual, to put it mildly, and one that is increasingly buckling under the weight of its own contradictions.

For a long time, investors were happy enough to pour money into OpenAI despite a structure that didn’t put their interests first, but in 2023, the board of the nonprofit that controls the company — yep, that’s how confusing it is — fired Sam Altman for lying to them.

It was a move that definitely didn’t maximize shareholder value, was at best very clumsily handled, and made it clear that the nonprofit’s control of the for-profit could potentially have huge implications — especially for its partner Microsoft, which has poured billions into OpenAI.

Altman’s firing didn’t stick — he returned a week later after an outcry, with much of the board resigning. But ever since the firing, OpenAI has been considering a restructuring into, well, more of a normal company.

Under this plan, the nonprofit entity that controls OpenAI would sell its control of the company and the assets that it owns. OpenAI would then become a for-profit company — specifically a public benefit corporation, like its rivals Anthropic and X.ai — and the nonprofit would walk away with a hotly disputed but definitely large sum of money in the tens of billions, presumably to spend on improving the world with AI.

There’s just one problem, argues a new open letter by legal scholars, several Nobel-prize winners, and a number of former OpenAI employees: The whole thing is illegal (and a terrible idea).

Their argument is simple: The thing the nonprofit board currently controls — governance of the world’s leading AI lab — makes no sense for the nonprofit to sell at any price. The nonprofit is supposed to act in pursuit of a highly specific mission: making AI go well for all of humanity. But having the power to make rules for OpenAI is worth more than even a mind-bogglingly large sum of money for that mission.

“Nonprofit control over how AGI is developed and governed is so important to OpenAI’s mission that removing control would violate the special fiduciary duty owed to the nonprofit’s beneficiaries,” the letter argues. Those beneficiaries are all of us, and the argument is that a big foundation has nothing on “a role guiding OpenAI.”

And it’s not just saying that the move is a bad thing. It’s saying that the board would be illegally breaching their duties if they went forward with it and the attorneys general of California and Delaware — to whom the letter is addressed because OpenAI is incorporated in Delaware and operates in California — should step in to stop it.

I’ve previously covered the wrangling over OpenAI’s potential change of structure. I wrote about the challenge of pricing the assets owned by the nonprofit, and we reported on Elon Musk’s claim that his own donations early in OpenAI’s history were misappropriated to make the for-profit.

This is a different argument. It’s not a claim that the nonprofit’s control of the for-profit ought to produce a higher sale price. It’s an argument that OpenAI, and what it may create, is literally priceless.

OpenAI’s mission “is to ensure that artificial general intelligence is safe and benefits all of humanity,” Tyler Whitmer, a nonprofit lawyer and one of the letter’s authors, told me. “Talking about the value of that in dollars and cents doesn’t make sense.”

Are they right on the merits? Will it matter? That’s substantially up to two people: California Attorney General Robert Bonta and Delaware Attorney General Kathleen Jennings. But it’s a serious argument that deserves a serious hearing. Here’s my attempt to digest it.

When OpenAI was founded in 2015, its mission sounded absurd: to work towards the safe development of artificial general intelligence — which, it clarifies now, means artificial intelligence that can do nearly all economically valuable work — and ensure that it benefited all of humanity.

Many people thought such a future was a hundred years away or more. But many of the few people who wanted to start planning for it were at OpenAI.

They founded it as a nonprofit, saying that was the only way to ensure that all of humanity maintained a claim to humanity’s future. “We don’t ever want to be making decisions to benefit shareholders,” Altman promised in 2017. “The only people we want to be accountable to is humanity as a whole.”

Worries about existential risk, too, loomed large. If it was going to be possible to build extremely intelligent AIs, it was going to be possible — even if it were accidental — to build ones that had no interest in cooperating with human goals and laws. “Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity,” Altman said in 2015.

Thus the nonprofit. The idea was that OpenAI would be shielded from the relentless incentive to make more money for shareholders — the kind of incentive that could drive it to underplay AI safety — and that it would have a governance structure that left it positioned to do the right thing. That would be true even if that meant shutting down the company, merging with a competitor, or taking a major (dangerous) product off the market.

“A for-profit company’s obligation is to make money for shareholders,” Michael Dorff, a professor of business law at the University of California Los Angeles, told me. “For a nonprofit, those same fiduciary duties run to a different purpose, whatever their charitable purpose is. And in this case, the charitable purpose of the nonprofit is twofold: One is to develop artificial intelligence safely, and two is to make sure that artificial intelligence is developed for the benefit of all humanity.”

“OpenAI’s founders believed the public would be harmed if AGI was developed by a commercial entity with proprietary profit motives,” the letter argues. In fact, the letter documents that OpenAI was founded precisely because many people were worried that AI would otherwise be developed within Google, which was and is a massive commercial entity with a profit motive.

Even in 2019, when OpenAI created a “capped for-profit” structure that would let them raise money from investors and pay the investors back up to a 100x return, they emphasized that the nonprofit was still in control. The mission was still not to build AGI and get rich but to ensure its development benefited all of humanity.

“We’ve designed OpenAI LP to put our overall mission — ensuring the creation and adoption of safe and beneficial AGI — ahead of generating returns for investors. … Regardless of how the world evolves, we are committed — legally and personally — to our mission,” the company declared in an announcement adopting the new structure.

OpenAI made further commitments: To avoid an AI “arms race” where two companies cut corners on safety to beat each other to the finish line, they built into their governing documents a “merge and assist” clause where they’d instead join the other lab and work together to make the AI safe. And thanks to the cap, if OpenAI did become unfathomably wealthy, all of the wealth above the 100x cap for investors would be distributed to humanity. The nonprofit board — meant to be composed of a majority of members who had no financial stake in the company — would have ultimate control.

In many ways the company was deliberately restraining its future self, trying to ensure that as the siren call of enormous profits grew louder and louder, OpenAI was tied to the mast of its original mission. And when the original board made the decision to fire Altman, they were acting to carry out that mission as they saw it.

Now, argues the new open letter, OpenAI wants to be unleashed. But the company’s own arguments over the last 10 years are pretty convincing: The mission that they set forth is not one that a fully commercial company is likely to pursue. Therefore, the attorneys general should tell them no and instead work to ensure the board is resourced to do what 2019-era OpenAI intended the board to be resourced to do.

What about a public benefit corporation?

OpenAI, of course, doesn’t intend to become a fully commercial company. The proposal I’ve seen floated is to become a public benefit corporation.

“Public benefit corporations are what we call hybrid entities,” Dorff told me. “In a traditional for-profit, the board’s primary duty is to make money for shareholders. In a public benefit corporation, their job is to balance making money with public duties: They have to take into account the impact of the company’s activities on everyone who is affected by them.”

The problem is that the obligations of public benefit corporations are, for all practical purposes, unenforceable. In theory, if a public benefit corporation isn’t benefitting the public, you — a member of the public — are being wronged. But you have no right to challenge it in court.

“Only shareholders can launch those suits,” Dorff told me. Take a public benefit corporation with a mission to help end homelessness. “If a homeless advocacy organization says they’re not benefitting the homeless, they have no grounds to sue.”

Only OpenAI’s shareholders could try to hold it accountable if it weren’t benefitting humanity. And “it’s very hard for shareholders to win a duty-of-care suit unless the directors acted in bad faith or were engaging in some kind of conflict of interest,” Dorff said. “Courts understandably are very deferential to the board in terms of how they choose to run the business.”

That means, in theory, a public benefit corporation is still a way to balance profit and the good of humanity. In practice, it’s one with the thumb hard on the scales of profit, which is probably a significant part of why OpenAI didn’t choose to restructure to a public benefit corporation back in 2019.

“Now they’re saying we didn’t foresee that,” Sunny Gandhi of Encode Justice, one of the letter’s signatories, told me. “And that is a deliberate lie to avoid the truth of — they originally were founded in this way because they were worried about this happening.”

But, I challenged Gandhi, OpenAI’s major competitors Anthropic and X.ai are both public benefit corporations. Shouldn’t that make a difference?

“That’s kind of asking why a conservation nonprofit can’t convert to being a logging company just because there are other logging companies out there,” he told me. In this view, yes, Anthropic and X both have inadequate governance that can’t and won’t hold them accountable for ensuring humanity benefits from their AI work. That might be a reason to shun them, protest them or demand reforms from them, but why is it a reason to let OpenAI abandon its mission?

I wish this corporate governance puzzle had never come to me, said Frodo

Reading through the letter — and speaking to its authors and other nonprofit law and corporate law experts — I couldn’t help but feel badly for OpenAI’s board. (I have reached out to OpenAI board members for comment several times over the last few months as I’ve reported on the nonprofit transition. They have not returned any of those requests for comment.)

The very impressive suite of people responsible for OpenAI’s governance have all the usual challenges of being on the board of a fast-growing tech company with enormous potential and very serious risks, and then they have a whole bunch of puzzles unique to OpenAI’s situation. Their fiduciary duty, as Altman has testified before Congress, is to the mission of ensuring AGI is developed safely and to the benefit of all humanity.

But most of them were selected after Altman’s brief firing with, I would argue, another implicit assignment: Don’t screw it up. Don’t fire Sam Altman. Don’t terrify investors. Don’t get in the way of some of the most exciting research happening anywhere on Earth.

What, I asked Dorff, are the people on the board supposed to do, if they have a fiduciary duty to humanity that is very hard to live up to? Do they have the nerve to vote against Altman? He was less impressed than me with the difficulty of this plight. “That’s still their duty,” he said. “And sometimes duty is hard.”

That’s where the letter lands, too. OpenAI’s nonprofit has no right to cede its control over OpenAI. Its obligation is to humanity. Humanity deserves a say in how AGI goes. Therefore, it shouldn’t sell that control at any price.

It shouldn’t sell that control even if it makes fundraising much more convenient. It shouldn’t sell that control even though its current structure is kludgy, awkward, and not meant for handling a challenge of this scale. Because it’s much, much better suited to the challenge than becoming yet another public benefit corporation would be. OpenAI has come further than anyone imagined toward the epic destiny it envisioned for itself in 2015.

But if we want the development of AGI to benefit humanity, the nonprofit will have to stick to its guns, even in the face of overwhelming incentive not to. Or the state attorneys general will have to step in.