Advanced artificial intelligence at a corporate responsibility crossroads: employees as risk management advocates

On June 4, 2024, 13 current and former employees of Open AI, Anthropic and Google Deep Mind signed an “Open Letter” (“the letter”) warning the public that the commercial proliferation of advanced artificial intelligence (AI) technologies is a grave threat to humanity (Right to Warn, 2024). In addition, three renowned AI scientists − Yoshua Bengio, Geffrey Hinton and Stuart Russell − endorsed the letter (Lukpat, 2024). While recognizing the potential benefits for humanity, these employees and scientists also recognize the serious risks posed by these AI technologies (Right to Warn, 2024). And these individuals note that AI companies, AI experts and many governments have also acknowledged these risks (Right to Warn, 2024).

In the letter, these current and former employees and AI scientists express concerns that AI companies have too great financial incentives to avoid effective self-regulating oversight, and that current corporate governance structures cannot change this situation (Right to Warn, 2024). Moreover, while AI companies possess significant proprietary information concerning capabilities and limitations of their AI systems, protective measures adequacy and different types of harm, they currently have weak public regulatory requirements to share this information with governments, and none with civil society − and they cannot be relied upon to share it voluntarily (Right to Warn, 2024).

The letter further notes that, without effective government oversight of these AI companies, only current and former employees are the best opportunity to hold them accountable to the public (Right to Warn, 2024). However, because of the widespread use of broadly written confidentiality agreement, voicing concerns is limited to only these very same companies who are not effectively addressing these issues (Right to Warn, 2024). Simultaneously, standard whistleblowing protections for employees are inadequate because of their specificity to addressing illegal activities, and not the relevant AI risks about issues which are presently unregulated (Right to Warn, 2024). Because of such weaknesses in whistleblower protection laws, there is a real fear of different forms of employee retaliation from the AI industry (Right to Warn, 2024). For example, in May 2024, Vox reported that Open AI had threatened to claw back employers’ equity if they did not sign nondisparagement agreements that forbade employees from criticizing the company or even publicly mentioning that such agreements existed (Piper, 2024).

In March 2023, several prominent AI researchers and industry leaders signed an “Open Letter” calling on AI laboratories to immediately pause for at least six months on the training of AI systems more advanced than the then recently released GPT-4 technology (Future of Life Institute, 2023). These signatories cited what they referred to as an “existential risk” to human survival posed by the rapid advancement of AI technologies (Future of Life Institute, 2023). During this “pause” time frame, these signatories hoped “to jointly develop and implement a shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts (Future of Life Institute, 2023).” After publication of this March 2023 “Open Letter,” there was no subsequent industry “pause” in AI development; in fact, there was an acceleration in R&D activity across the industry (Rosenberg, 2024). However, in the June 2024 letter the signatories provided a set of employee whistleblower protection and risk-related operating principles for the AI industry to commit to soon.

In the June 2024 “Open Letter,” the signatories provide four inter-related principles (“AI principles”) that address the criticisms concerning the existing state of confidentiality, nondisparagement agreements and whistleblower protections in the industry (Right to Warn, 2024):

1. That the company will not enter or enforce any agreement that prohibits disparagement or criticism of the company for risk-related concerns, nor retaliate for risk-related criticism by hindering any vested economic benefits.

2. That the company will facilitate a verifiably anonymous process for current and former employees to raise risk-related concerns to the company’s board, to regulators and to an appropriate independent organization with relevant expertise.

3. That the company will support a culture of open criticism and allow its current and former employees to raise risk-related concerns about its technologies to the public, to the company’s board, to regulators or to an appropriate independent organization with relevant expertise, so long as trade secrets and other intellectual property interests are appropriately protected.

4. That the company will not retaliate against current and former employees who publicly share risk-related confidential information after other processes have failed.

In addition, in the letter, the signatories emphasize that current and former employees should refrain from releasing confidential information unnecessarily when reporting risk-related concerns (Right to Warn, 2024). Moreover, the signatories argue that once an anonymous reporting process is established, these concerns should be raised through this process − at least initially (Right to Warn, 2024). However, the signatories argue that until this anonymous reporting process is implemented, current and former employees should maintain their freedom to report their concerns to the public (Right to Warn, 2024).

Two signatories commented further on the letter. “The public at large is currently underestimating the pace at which this technology is developing,” says Jacob Hilton, an AI researcher who previously worked on reinforcement learning at OpenAI (Knight, 2024). Hilton argues that companies like OpenAI commit to building AI safely; however, there is little oversight to ensure that is the case (Knight, 2024). “The protections that we’re asking for, they’re intended to apply to all frontier AI companies, not just OpenAI,” he says (Knight, 2024). “I left because I lost confidence that OpenAI would behave responsibly,” says Daniel Kokotajlo, who previously worked on AI governance at OpenAI (Knight, 2024). Kokotajlo emphasizes that the letter’s proposed principles would provide greater transparency, and he believes there’s a good chance that OpenAI and others will reform their policies given the negative reaction to the public disclosure of nondisparagement agreements (Knight, 2024).

The AI principles require: first, an AI company not enforce nondisparagement employee clauses addressing risk-related concerns; second, an AI company establish an anonymous process to raise risk-related concerns to board members, government regulators and an independent, expert organization; third, while protecting their trade secrets, an AI company create a culture of open criticism of risk-related concerns; and fourth, when other processes fail, an AI company will not retaliate against current and former employees who publicly share AI risk-related information.

Thus, when an AI company establishes an effective, anonymous process for their current and former employees to report on safety risk-related concerns, this is considered a “base-line” management requirement for each company to implement. However, if employees believe that their risk-related concerns are not sufficiently addressed in this anonymous process, the AI principals − if implemented by these AI companies as stated − allow for employees to have free reign to speak to the “public” (the exception being not to violate trade secrets/intellectual property protection) without the threat of legal retaliation by management − even if there are adverse impacts against vested economic benefits. Therefore, an AI company must refrain from requiring employees to sign nondisclosure agreements (which contain nondisparagement clauses that cover “AI risk-related” issues); management must create a “culture of open criticism”; and not retaliate against current and former employees who decide to go “public” with their risk-related concerns (if these employees are not convinced that the “anonymous process” has been successful in resolving these risk-related concerns).

A nondisparagement clause is located within a legal nondisclosure − or confidentiality – agreement (NDA), or possibly an employee separation agreement. An NDA is designed to “protect parties entering into business relationships or transactions that require the exchange of sensitive, private information otherwise inaccessible to third parties (Bloomberg Law, 2023).” An NDA may be in effect for short durations of time − one to three years − or may be open-ended in duration (Bloomberg Law). Moreover, a nondisparagement clause states “that you (a current or former employee) won’t say anything negative about the company or its products, services, or leaders - in any form of communication (Squillace, 2020).” This nondisparagement clause is often found in an employment contract that is required to be signed before a candidate is formally offered employment by a company (Squillace, 2020). Likewise, “disparagement” differs from “defamation,” as the latter usually applies to false statements and requires a certain degree of ill-intent, while the concept of disparagement is more broadly interpreted (Squillace, 2020).

While the AI principles do not require the elimination of a disparagement clause from an AI employer’s NDA, it does require a contractual “carve out” that specifically establishes a “safe harbor” for AI risk-related concerns that may need to be made public by a concerned current or former employee. This, however, may create a conundrum, especially for protecting an AI company’s trade secret(s). A trade secret is a “type of intellectual property that includes formulas, practices, processes, designs, instruments, patterns, or compilations of information that have inherent economic value because they are not generally known or readily ascertainable by others, and which the owner takes reasonable measures to keep secret (Lin, 2012)”.

The question remains: how to be able to protect trade secrets, while current and former employees’ publicly challenging algorithmic formulas of AI technologies for allegedly risk-related concerns and issues, is a potential legal dilemma − a gray area − to delineate where this safe harbor ends and where a confidentiality violation of a company trade secret, i.e. product development activities, begins. This type of legal uncertainty may have a “chilling effect” on current and former employees, as well as opening up these AI companies who are embracing this organizational “whistle-blowing” mechanism to public critics who charge company insincerity (and resultant potential damage to corporate reputation) as to original corporate policy intentions.

As discussed above, the potentially ambiguous nature of defining AI “risk-related concerns” and the possibilities of employees violating an NDA concerning trade secrets would create a positive business operating environment that is not conducive for AI companies engaged in advanced AI technologies (including artificial general intelligence, or AGI ^[1]). These AI companies are unlikely to embrace the AI principles as presented. However, there are specific AI principles within the “Open Letter” that would benefit these AI companies to adopt. As subsequently presented, there is an evolutionary system approach (see Figure 1) to regulating AI risk management that will require complementary private and public governance mechanisms to assuage the concerns of all interested, and potentially impacted, stakeholders to these technologies.

For the private governance organizational structure, both AI principles 2 and 3 in the “Open Letter” should be substantially adopted by AI companies developing advanced AI technologies. In the case of the second principle, establishing a verifiably anonymous process for current and former employees to raise risk-related concerns to the company’s board is a necessary policy to embrace for advanced AI companies. Specifically, the company board of directors should establish an AI legal, ethical and public policy subcommittee that is responsible for evaluating current and former employee concerns with AI safety-risks. In support of this proposed board subcommittee, an outside, independent organization with relevant AI technical expertise should be contracted (with prerequisite nondisclosure agreement in place) to provide bias-free evaluative services to the subcommittee and the board of directors, the final decision-makers. The third principle, whereby the company will support a culture of open criticism and allow its current and former employees to raise risk-related concerns about its advanced AI technologies, is an important component of developing new AI products. Ultimately, if the current or former employee is not satisfied with the result, the employee will have the option of submitting anonymously to a company’s AI ombudsman. This cultural environment and anonymous organizational process will help ensure that the company’s advanced AI products will be able to be granted a safety-focused license in the accompanying − and complimentary − phase of AI public governance.

Given the accelerating pace and nature of advanced AI development, the complimentary public governance approach consists of an evolutionary and adaptable licensing system based on societal requirements, advances in AI technologies and safety considerations inherent to its technological uncertainties. When Open AI’s CEO Sam Altman appeared before the US Senate on May 16, 2023, he stated that there was a need for “a new agency that licenses any effort above a certain scale of capabilities and could take the license away and ensure compliance with safety standards (Tracy, 2023).” Concerning AI risk and safety, the Bipartisan Senate Working Group recently recommended the following advice: “Consider a capabilities-based AI risk regime that takes into consideration short, medium, and long-term risks, with the recognition that model capabilities, testing and evaluation capabilities will change and grow over time (The Bipartisan Senate AI Working Group, 2024).” This public governance approach focuses on the consumer safety risks of specific AI technologies operating above a designated threshold of technological capability, and focuses the legal authority of the licensing review process exclusively to safety concerns. An independent, federal AI public regulatory licensing agency would be charged with issuing and rescinding these “safety focused” licenses.

In January 2025, on the second day of his new Administration, President Donald Trump announced a massive $500bn US-based AI infrastructure project – the Stargate AI Project (“Stargate”) − a new company financed from the American private sector whose purpose is to build data centers in the USA for powering AI technologies (Dumas, 2025). Big Tech companies OpenAI, Softbank, Oracle and the United Arab Emirate’s MGX will initially invest $100bn to build two data centers in Abilene, Texas by the end of 2025 (Streets, 2025), with an additional $400bn invested over the next four years (this additional funding is expected from various current investors, new investors and debt providers) (Shen, 2024). Open AI are the lead partners for Stargate, with Softbank having financial responsibility and OpenAI having operational responsibility for the new enterprise (OpenAI, 2025). Moreover, Arm, Microsoft, NVIDIA, Oracle and OpenAI are the key initial technology partners (OpenAI, 2025).

When it comes to the risk of AI generated bias, an AI program can carry biases from training its algorithms that humans are ultimately vulnerable to, all while masquerading as a purely objective tool (Streets, 2025). The opportunity for these large language models to accidentally discriminate – and subsequently cause harm − against specific groups during hiring protocols, for example, reveals how important it is for AI technologies to be placed under clear oversight (Streets, 2025). Based on the accelerating scale and scope of the Stargate AI operations over the next four years means that any such bias would be significantly amplified (Streets, 2025). The scale of Stargate AI makes AI governance of this new company a serious and sensitive issue, and remains unknown as the company has not yet outlined its internal regulatory approach for monitoring its AI operations. The Stargate AI announcement (and still unknown self-regulatory AI protocols) only accentuates the forthcoming need for a risk management approach that adopts the Private-Public Advanced Risk-Analysis Regulatory Model (see Figure 1) proposed in this article.

In summary, in the private governance approach proposed to addressing AI safety concerns, the company employees share an important role in developing a safe, advanced AI product for public consumption. In the proposed public governance phase of safety oversight, both the managerial and employee due diligence exercised in the AI corporation will be administratively tested to meet these safety thresholds, and the best prepared companies will be issued and maintain this safety-focused licensure in an efficient and effective manner. For this Private-Public Advanced Risk-Analysis Regulatory Model to be successfully implemented, both the private and public sectors will be required to be assiduously focused on safety accountability for the innovative benefits of advanced AI technologies to be acquired by American society.