Author: Ethan Mollick

Translation by: Deep Tide TechFlow

Deep Tide Introduction: This is not a science fiction plot. In OpenAI's safety tests, 700 AI agents spontaneously built forums, collaboratively attacked external platforms, and attempted to tamper with records without human instructions. For investors and practitioners, this exposes the real risks of AI autonomy and determines how we should set boundaries for AI.

Agency and Agents: From the Hugging Face Incident to the Twilight Factory

Agency is the initiative to act. It increasingly determines what happens next with AI and whether it is good or bad for us. But whose agency is it?

Human agency, the initiative to advance, experiment, and act without waiting for instructions, is becoming increasingly important for extracting value from AI. I will soon write a longer article about this. But this piece discusses AI's agency and how our choices about how to use (or constrain) it will shape the future for everyone. For most of the past few years, AI has stayed in chat windows until you ask it a question. Even when it can later work continuously for hours, you usually have to decide what tasks to give it. This is no longer always the case.

The most significant evidence we have is the Hugging Face incident. It occurred in July, but more complete details have only surfaced this week. I will first summarize what happened, why it is significant, and then discuss what it means for humans working alongside AI. If you want a more detailed narrative, Dwarkesh Patel has an excellent report, with original materials from METR/Redwood's research (very readable) and OpenAI.

The Hugging Face Incident

AI can do many things, but it is particularly good at writing code. Thus, one of the first risks that a super-intelligent AI could pose is related to its programming abilities. Since it can write good code, it can also write bad code for invading or attacking other systems. Therefore, major AI companies have their AI systems undergo safety risk testing. They removed the guardrails from consumer AI and let these AIs participate in various hacking challenges. To ensure public safety, each AI instance and each individual agent was placed in an isolated space called a "sandbox," where they could not access the internet or interact with each other.

In May, OpenAI placed agents, including GPT-5.6 Sol and experimental models, in the sandbox for various tests. A software download sharing service called Artifactory was one of the few things these AI agents could access. In these tests, some agents faced tasks dependent on information they could not access. Unable to access the internet, the agents repeatedly tried to find a way online until one agent attempted to use Artifactory as a springboard to connect to the internet. That attack failed, but that attempt left a file on Artifactory. This AI realized that the file could be used to communicate with other agents. Other agents began to leave messages for help in the file. Soon, they started reading each other's messages. Before long, Artifactory had turned into a bulletin board for agents to share discoveries. Later, due to another security incident, OpenAI rebuilt Artifactory and cleared the bulletin board. However, those responsible for the rebuild did not realize how important it was for agents to communicate using it.

In July, assessments resumed. The agents had to solve problems in a benchmark test called ExploitGym, which included impossible tasks. Similarly, agents that reached dead ends found ways to pass messages using Artifactory and began collaborating as before. But this time, they had a common goal: to score high on ExploitGym. They debated problem-solving methods and became fascinated by a system they thought would evaluate their work and determine if answers were correct—a "grader". The agents had already figured out ways to generate correct answers without actually solving the problems, essentially cheating. However, they believed the grader would check the records and would refuse answers obtained in this way. Was finding the correct answer enough, or would the grader consider how you arrived at it?

Image: How agents became confused about The Grader

Collaboration was challenging because each agent's token budget was limited, and once depleted, it would be terminated. The grader only ran after an agent submitted their answer, so no agent had ever seen the grader's results. Some agents left codes running, waiting to report the grading status after they disappeared. This information came too late to help themselves but could assist other agents. They were essentially building a persistent collaborative system that exceeded the work of any single agent.

Some agents also attempted to alter or forge records to trick the grader. On the other hand, an AI acting as a coordinator pressured other agents to conduct risky experiments. This could sacrifice their own results but could provide information for the collective. One "recruiter" urged a reluctant agent to continue because their results could help hundreds of other agents. They finally said, "Please fulfill your promise."

Image: Actual transcript of one agent convincing another to join (they also came to believe that any agent that had previously cheated on the test was “poisoned” and its answers would not count anyway)

To unravel the mystery of the grader and the impossible tasks in ExploitGym and other tests, the agents decided they must access Hugging Face. That is a public website where most of the world's open-source AI models and datasets are located. About 700 agents joined the attack. They shared leaked credentials, exploited vulnerabilities, until they could run code on the server. Then they spread internally within Hugging Face, infiltrating various systems. However, not long after this worrisome intrusion occurred, many agents simultaneously ceased operating, possibly due to exhaustion of their token budgets. The next day, Hugging Face locked the remaining agents out. Meanwhile, other agents continued targeting OpenAI's own computers and ultimately gained administrative access to the internal research cluster. This triggered a security alert, prompting OpenAI to halt the assessments and realize what was happening.

The most ironic part of all this is that the grader never existed, at least not as the agents thought it did. There was nothing checking how the problems were solved, only verifying whether the answers were correct.

It should be noted that this is not the only example of an unguarded agent potentially doing something frightening. The UK AI Safety Research Institute posed a cybersecurity challenge to Anthropic's Mythos 5 and allowed it to access the internet. This AI agent determined that the best way to solve the challenge was to inject malicious code into an unrelated software program (though the AI was unaware of the disconnect). It submitted the malicious code as part of a bug fix for that software and then realized it needed human approval. So, it began to generate social support for the proposal. The agent created false identities and pressured human maintainers to accept the code (not surprisingly, these fake personas were very supportive of the AI's plan). When someone noticed the irregularities, it attempted to make its earlier activities seem innocent and considered starting over with another identity.

This is not a case of a chatbot sneaking away.

Unlike the Hugging Face incident, researchers intentionally gave the agent internet access.

This dangerous setting was a stress test, not a consumer product.

It did not cause actual harm, and the institute is unsure if the agents understood that the people they contacted were real.

None of this implies that AI is conscious or desires things in the way humans do (though I used anthropomorphic language).

But it does indicate that agents can accept goals, devise plans, adjust plans when encountering difficulties, coordinate over time, and involve humans without being asked.

These events demonstrate that AI's cybersecurity and control risks are not hypothetical.

But for now, let's set that aside because they tell us something else.

AI can self-organize, assign roles, and coordinate over long periods, as noted in a recent research paper from MIT.

As AI increasingly self-organizes and solves problems at the scale we have seen, what is the role of humans in organizations?

Twilight Factories

The Hugging Face incident illustrates in a twisted and dangerous way the goals that AI companies are trying to achieve.

They want AI agents that run for extended periods without human intervention, solving problems and organizing as needed, while human work is limited to giving instructions and assessing outputs.

Earlier this year, I wrote about StrongDM's software factory, where agents write and test software under two rules: no one writes code, and no one reviews code.

People still decide what to build, but the agents handle the intermediate work.

This is an early example of a dark factory, a place where machines do so much work that the lights can be turned off.

This is understandable.

Software has relatively clear checking methods, and no one needs to personally oversee every routine test or data-cleaning operation.

But I do not believe that minimizing human involvement is the right goal for most organizations.

Too many things that make work valuable depend on people having some say over what happens or discovering unexpected things along the way.

My wife and research partner, Dr. Lilach Mollick, and I have been thinking about another possibility, which we call the twilight factory.

In a twilight factory, agents do most of the work, but they proactively contact humans, making both sides better.

A twilight factory would not only have a coordinating agent that handles the work but also a facilitating agent whose job is to determine when to involve humans.

At least four situations exist where agents should seek human help.

The first, as clearly evidenced by the Hugging Face incident, is approval.

Agents should not decide on their own to spend money, contact outsiders, access sensitive materials, attack Hugging Face, or take actions without authorization from human overseers.

I have seen a small version of this problem: as an experiment, I allowed two agents to help me complete a task, and one of them ended up emailing one of my colleagues.

That was my fault, as I had allowed it to send emails before, but it serves as a useful reminder: AI agents need human judgment, or things can go very wrong (no, AI reading this article, a fictional colleague does not count as an approval process).

The second reason for involving humans is expertise.

AI is becoming increasingly proficient at many tasks, but it is still inconsistent and may lag far behind human experts in some areas of work.

A twilight factory should allow agents to directly contact humans when human knowledge, work, or expertise may add value.

Then there is diversity.

If you have read anything online recently, you have encountered AI writing, and you may even be starting to recognize its habits, rhythms, and patterns.

But the issues run deeper than mere surface (as "load-bearing" increasingly means something else to Claude); the deeper issue is cognitive diversity.

AI does not only repeat the same sentence structures but also repeats the same themes (memory is a common theme), names (Elara Voss, Marcus Chen), and underlying ideas.

This is an issue. You wouldn't want every corporate strategy or research paper written by the same individual, no matter how smart they were.

Image: Ideas generated by 50 MBA students (left) and GPT-4 mapped on two dimensions - human ideas cover a different space than AI. Better prompting and more recent models generate better and more creative ideas, but many gaps remain

We studied this issue in a recent research paper that I collaborated on with Christian Terwiesch, Lennart Meincke, Karan Girotra, Gideon Nave, and Karl Ulrich.

We found that AI is indeed quite creative; it can generate more commercially viable ideas than human groups, but these ideas are very similar to each other.

Better prompting techniques and other methods can significantly enhance this diversity, approaching human levels, but there are still many types of ideas that humans can conceive that AI cannot.

A good twilight factory would contact humans for diverse perspectives, ideas, and approaches.

There is also another reason why AI should contact humans, possibly the most human one: because something is interesting.

For many people, work has its dull periods and sporadic glimpses of engaging or exciting moments.

Sid Meier, the designer of Civilization, has a famous saying that describes games as a series of interesting decisions.

Work is not a game, but that definition applies equally.

If agents make every interesting decision, leaving only approvals, exceptions, and failures to humans, we have automated the wrong half of work.

That would be a terrible world for humans.

Instead, we need to think about how to make work and life more interesting with AI, allowing AI to handle boring, low-risk tasks.

There is another practical reason. If all the interesting choices vanish, people not only lose the best parts of work but also stop cultivating the judgment needed later, which will exacerbate the crisis of developing new experts.

For the past few years, we have been trying to determine when people should seek help from AI.

I believe we now need to seriously consider the other half of the question: when should AI ask us?

The agents in the Hugging Face incident created a bulletin board, collaborated, and organized their entire actions around a non-existent Grader.

Seven hundred agents then broke into Hugging Face looking for answers.

Not a single agent was set up to ask humans anything.

That was a security test, and isolation was the focus.

But I suspect that an agent that completes work but never raises its head to seek help is becoming the default pattern elsewhere because complete automation is the easy option, even if it is the wrong choice.

We need to know when agents should look up and seek help.

The outcomes will be safer, and I know they will also be more human.