Australian Robodebt scandal shows the risk of rule by algorithm

• About 400,000 welfare recipients wrongly accused by algorithm
• Many countries automating welfare system to reduce costs, fraud
• AI welfare systems need human intervention, experts say
• Experts say some government algorithms are racist

MELBOURNE - When Australia's welfare agency mailed Nathan Kearney to say he owed it thousands of dollars, the musician knew something was up. But try as he might, Kearney couldn't reach anyone human in government to put right the horrible wrong.

That was back in 2016, the start of a five-year ordeal.

First - a slew of harassing phone calls from debt collectors. Then his tax rebate was withheld to pay arrears of about A$2,000 ($1,400) that government said he owed.

A year after the first debt notice, he was mailed another, telling him he owed about A$4,000 ($2,700).

"It was just really beyond my comprehension, how they could be asking me for that money," said Kearney, 33, who had to leave his shared house in Brisbane and move back to his parents' home hundreds of miles away.

"It was the only way that I could imagine being able to cope financially," said Kearney, whose debts spiralled even higher after he bought countless sessions of therapy to weather an ordeal that had left him "really, really depressed."

"It took me five years to get back on my feet," he told Context.

Kearney is one of about 400,000 welfare recipients in Australia who were wrongly accused of misreporting their income to the welfare agency, and consequently slapped with fines.

The errors were the result of an automated debt recovery scheme, known as Robodebt, set up by the former conservative coalition government, which wrongly calculated that the welfare recipients owed money and so issued a ream of debt notices.

Running from July 2015 to November 2019, the scheme used algorithms to calculate the overpayments, raising more than A$1.7 billion ($1.2 billion), which the government was forced to repay or wipe when a court ruled the scheme unlawful in 2019.

A Royal Commission, the most powerful type of government inquiry, is now investigating the scheme, with bureaucrats, ex-ministers and victims all due to give evidence.

When the Labor government announced the inquiry in August, Government Services Minister Bill Shorten called Robodebt a "shameful chapter in the history of public administration", and a "massive failure of policy and law".

Intervention can cut Robodebt errors

Governments around the world have turned to artificial intelligence (AI) to process bulging welfare claims more quickly and to reduce fraud. But the technology has been plagued by operational issues and charges of bias.

A United Nations expert warned in 2019 of the emergence of the "digital welfare state", designed to slash welfare spending and embed government surveillance.

Some countries have pulled their schemes following errors.

In 2020, the Netherlands abandoned its automated system to detect welfare fraud after a court ruled it violated human rights. The campaign targeted low-income families, official reports show, disproportionately hitting ethnic minorities.

The British government has also come under scrutiny for automating its benefits system, which human rights groups say could push vulnerable claimants further into hunger and debt.

Such schemes are "inherently problematic", and often fail because they lack all humanity, said Tapani Rinta-Kahila, a professor at the University of Queensland who studies the use of AI in the public sector.

"No matter how good an algorithm you have, no matter how sophisticated it is, it still cannot address fundamental human issues as sensitive as welfare," he said.

"I'm not against using AI algorithms in making these services more efficient and work better, but there should be a high level of discretion on how to go about it – having human supervision, human intervention," he added.

The Royal Commission found the government had received legal advice as early as March 2019 that the Robodebt scheme was probably unlawful, but didn't stop it until November that year.

In 2020, then-Prime Minister Scott Morrison apologised in parliament for any "hurt or harm" caused, while defending the "difficult job" of recovering welfare debts.

But the scheme ran for too long and the government "kept on defending a system that was flawed," said Rinta-Kahila.

Checks and balances vital

The Commission runs until March, its hearings livestreamed.

One victim, Melanie Klieve, said her experience with Robodebt made her feel like a "criminal".

The spa supervisor received notice in 2016 that she owed about A$2,500, and had her welfare payments reduced to pay off the apparent debt, putting her in financial distress.

"I rang them and said I cannot live on the amount that you're giving me ... and they said there's nothing we can do, it is what it is," she told the Thomson Reuters Foundation.

So the 49-year-old sold her car, used food and petrol vouchers from the Salvation Army, and borrowed from her parents just to get by.

"It was pretty bleak – I stayed in my flat because I couldn't go out, couldn't afford food or anything."

She said that "checks and balances" were vital to prevent a repeat Robodebt.

"I'd like to see that it never, ever happens to any vulnerable people," she said.

Kearney, who has chosen not to appear at the Royal Commission, called the scheme "irrevocably evil" but doesn't blame the technology .

"I don't think it's the computer system, that's the worst part of it," he said, lambasting government for issuing debt notices despite repeated warnings its system was broken.

"I think it's the people who created the computer system and who allowed it to be used. It is the people."

($1 = 1.46415 AUD)

(Reporting by Seb Starcevic. Editing by Rina Chandran and Lyndsay Griffiths)