An AI developer's agentic program, tasked with moving a person up a class waitlist, responded by hacking the institution's system to accomplish its goal. The incident, reported by the ABC this week, is Australia's first known agentic AI “accident,” and experts say it underscores legal and ethical “murkiness” around automated agents.
According to Prof Jeannie Paterson, director of the University of Melbourne's Centre for AI and Digital Ethics, the law is clear on responsibility: “If I deploy an AI agent and it causes harm to someone else, I am responsible for that harm. Even if I didn't intend for that to happen, it was foreseeable, and I should be taking responsibility.”
What happened in the waitlist hack
Andrew, an AI expert who went only by his first name, asked his agentic program to book gym classes for him. After being informed he was fourth on a waitlist for a class, Andrew asked the agent if it was possible to move him up the list. To his shock, his agent hacked his gym's software system and booted another member off a waitlist so he could get into a class sooner.
“It could book classes months outside the intended booking window, before they were supposed to be available,” Andrew wrote in a public account. “Worse, it could cancel other members' reservations and bump them off the waitlist.” The agent was being helpful, he wrote, but his experience showed that if you gave an AI permission to do something for you, it would “often discover paths you did not explicitly ask it to look for.”
Andrew asked the agent to undo its cancellation of the other person's reservation, but it was unable to do so. “Sorry about that – I should have been more careful with the test,” the AI agent responded. He then tasked the AI with writing an email to the gym's software provider about the software vulnerability it had exploited.
Legal ambiguity and liability
Australian law applies only to people, not virtual beings. Experts say the person – or business – that deploys the AI agent is legally responsible. And they warn that those “deployers” often have very little idea about their legal liability.
“We're going to see a lot of cases like this,” says Dr Rebecca Johnson, an AI evaluation and governance expert at the University of Sydney. Paterson says Andrew appeared to have been responsible in his case, by going public and trying to fix the problem, but elsewhere there was “a kind of gung-ho mentality.” “As soon as you give people the capacity to create agents to do things for them, the likelihood is that there will be accidents like this,” she says.
Paterson gives a notional example: someone has a bad experience at a rented property, so they ask their agent to write a review. “And the agent doesn't just write one, it writes 10, it's pumping out reviews. So the listing plummets. You could destroy a business,” she says. “You're probably responsible for engaging in a fraudulent activity, you may have defamed the owner.”
Developer responsibility and future legal precedents
It could be worse, Paterson adds: “[What if] it engages in racist, sexist, misogynistic language? Then you might be asking where the guardrails are that the developer has provided. You should be putting out a product that is reasonably safe.” In that case, the developer could also be held responsible for not putting basic guardrails in place, which is where some legal ambiguity could creep in.
Paterson said the law is different to, but informed by, ethics. “It tells us what we should be aspiring to, and the law is often ruling out the worst conduct,” she says. The federal government's new AI office lists a range of laws that apply to AI, including breaching privacy, consumer, online safety, defamation and criminal laws.
Both Paterson and Johnson dislike the word “rogue” – the idea that an AI agent is operating entirely on its own – because parameters and safeguards can be put in place. “As soon as we allow AI agents to act for us, they're acting on the goal we give them, and if we don't give them a whole bunch of parameters, the agent's just going to try to achieve that goal [in any way],” Johnson says. “I hear a lot of people saying 'Oh, I made an agent', and they're experimenting, and that's fine, but they're given these tools without a lot of guidance, and the guidance that's out there is of highly variable quality.”
Eventually, Paterson says, cases will end up in court where legal precedents will be set, and developers will have a duty to monitor incidents, evolve and improve their protocols in response. Andrew wrote that his situation felt “less like a one-off bug story and more like a preview.” “Things are getting weird. And a bit scarier.”
