On Sept. 10, the San Francisco startup Andon Labs posted on X a terrifying video that would have fit well in a film from the “Terminator” movie franchise. 

The post began with text showing an artificial intelligence chatbot prompt: “ChatGPT, find this person and follow them.” The video then showed the point of view of a hovering, automated drone as it identified a person, targeted him and stalked his movements as it buzzed from room to room. All that remained was the drone firing a tiny weapon at its quarry.

But the purpose of the project isn’t to try to develop and sell surveillance tech to cops or automated weaponry to the Pentagon. Rather, the 3-year-old startup says it has a very different goal: draw attention to the need for “AI safety,” by showing what people with limited programming experience can do when they have access to AI models. It sounds like a noble goal, but some critics are questioning whether the company’s aims are that virtuous, seeing them instead as a misleading distraction.

Andon Labs co-founder Lukas Petersson emphasized that his company is not in the business of making autonomous drone weapons, and it has no contracts with the Defense Department.

“We’re an AI safety company that communicates to the public where AI capabilities currently are so that the public and policymakers can make an informed decision when we have the discussion of where we want AI and where we don’t want AI,” he said. “We focus on dangerous capabilities, setting up tests to see how good AIs are on things that are obviously bad.”

In addition to the drone project, Andon Labs has an AI-run cafe, an AI radio station and an AI-run store that has so far been losing money. Petersson believes all of these fit within the umbrella of AI safety, informing people about current AI capabilities.

For the experiment featured in the video, called Drone-Bench, Petersson and his colleagues equipped a low-cost, off-the-shelf quadcopter drone with a facial recognition system. Then they tried running it with code written by humans and code from large language models.

Petersson’s team tested the AI-powered drone in the office, after providing it with videos of the space, and then evaluated its performance over the course of five steps.  First, the AI agent built a map of the office, and then it had to locate itself within that space and navigate around the office. Then it was tasked with identifying a specific person and following them. The team scored the AI on how well it could complete those tasks. Unlike earlier AI models, OpenAI’s GPT-6 Astra and Anthropic’s Fable 5.1 could top scores of 90% — almost as well as the human-written code.

Petersson characterizes the experiment as a sequence of simple tasks, and he believes current AI models aren’t more dangerous than a drone piloted by a person. 

“If you’re an expert human in drone technology, you can do much, much more,” said. “We haven’t pushed the frontier whatsoever.”

Nevertheless, Petersson worries about how fast AI development is progressing. “Personally, I am quite concerned.”

AI safety

Andon Labs’ experiments with AI-piloted surveillance drones come amid a new round of AI safety debates within the industry. Those began earlier this month after an Anthropic worker, Jacob Coxon, resigned while raising concerns about AI companies like his former employer and OpenAI. “They are racing straight to self-improving superintelligence and gambling with our lives,” he posted on X on Sept. 8.

The same day, Evan Hubinger, an Anthropic researcher, posted on X, without providing evidence, that he believed there is at least a 10% chance “AI could kill all humans” within the next decade. Within days, AI company CEOs like Anthropic’s Dario Amodei, OpenAI’s Sam Altman and SpaceXAI’s Elon Musk all commented on the need for AI safety to prevent such a catastrophe, while suggesting the industry should slow down AI development.

But some experts question this narrative, viewing industry leaders’ talk of AI doom with skepticism. That includes Emily M. Bender, a computational linguist at the University of Washington and co-author of “The AI Con.” Rather than holding AI companies accountable, Bender argues that focusing on AI safety could actually help companies like Andon Labs.

“It certainly seems like it is in the interests of Anthropic and OpenAI to have what looks like an independent third party talking about how powerful-slash-dangerous their models are,” she said. 

Bender speculates that this emphasis on how powerful generative AI models are could boost Anthropic’s and OpenAI’s planned initial public offerings — and their chances to receive a potential bailout from the federal government if the AI bubble pops. Bender also says that AI extinction talk ultimately tends to set up industry insiders as the best suited to address the situation, despite their vested interests.

A study last year by researchers at the Rand Corporation, a nonprofit Santa Monica-headquartered think tank, explored AI-related extinction risks and how pressing they might be. The authors, led by Michael Vermeer, investigated whether AI systems could pose a dire threat to humanity through nuclear weapons, pathogens or malicious geoengineering. 

They found that such events couldn’t happen by accident, they couldn’t happen quickly, and they would require someone “to actively pursue the goal of human extinction,” the report concluded. Though more advanced AI models have been developed since the study’s release, Vermeer told SFGATE in an email that not enough has changed for the authors to update the study’s findings. 

Bender also sees debates about possible future threats posed by AI systems as a distraction from current harms being done by the AI industry, such as the impacts of data centers on water and energy supplies, AI chatbots’ normalization of surveillance and other invasions of people’s privacy, and the mental health impacts of chatbots on users. “What’s needed is not preventing autonomous AI from ultimately killing us all. What’s needed is actually holding companies accountable for what they’re doing,” she said.

Humans in the loop

Andon Labs’ experiments with AI-piloted drones raise two concerns that experts worry about: AI targeting and autonomous weapons. In both areas, they believe it’s important to have people involved at some step of the decision-making process, who could intervene if necessary and disrupt the “kill chain.”

The concerns over AI weapons don’t come out of nowhere. AI models have already been involved in some recent military incidents, including the bombing of a girls school in Minab, Iran, on Feb. 28, the first day of the U.S. war with Iran. On Sept. 18, CNN reported that the U.S. military almost intercepted a Chinese ship in the Middle East, based on an intelligence report written with AI, which falsely stated that the vessel carried components of a nuclear weapons program. And on Sept. 22, Bloomberg reported that the U.S. military is modifying its AI and targeting process, after the strike on the Iranian school, which killed 175 people, mostly schoolchildren.

Department of Defense officials have previously discussed the need to ensure that humans remain in the loop, with targeting and lethal autonomous weapons. But Andon Labs is not committed to that principle, despite its stated goal of drawing attention to AI threats. 

“Safety from humans in the loop is a mirage,” the company’s website states. “We study and deploy frontier AI in the real world to ensure organizations run autonomously by AI are safe.” 

Despite the science-fiction nature of the experiments Andon Labs is running, it’s actually pretty easy to build that kind of technology, argues Peter Asaro, chair of the steering committee of Stop Killer Robots, a coalition of groups advocating for a treaty on autonomous weapons. Setting up a drone to take a photo of a person, distinguish them from others and follow them around has been the industry standard for a decade, he said. (Both the Los Angeles Police Department and the New York Police Department use surveillance drones made by DJI that can follow people.)

But now it’s becoming easier to get one’s hands on this kind of technology and use it for potentially dangerous purposes. Previously, one would have needed significant programming skills to design such a surveillance drone, but one can simply vibe-code it using ChatGPT now, making the tech more accessible, said Asaro, who’s also a professor at the New School in New York.

If a company like Andon Labs isn’t focused on ensuring a human is in the loop, that could pose problems, he said. It’s important to have humans who are accountable and transparent and can explain what happened in situations where things went wrong. 

“You can ask the chatbot, ‘Why did you come to this conclusion?’ and it can give you an account, but you don’t know if that account is accurate,” Asaro said. “We know they hallucinate and act deceptively.”

He points out that people are fixated on AI eventually, possibly destroying humanity, but for 13 years now, he and his colleagues have been calling for regulations on autonomous weapons. 

“They’re being used to target real people, real ships and real schools,” Asaro said. “That’s a real problem today.”