When the Algorithm Fails
Vignettes that tell tales of companies pushing their technologies forward, ignoring regulation, conventional wisdom, and societal norms fill recent history. We limp along, believing government will catch up with the exponential pace technology has set and figure it out, but what happens in today’s modern machine learning, AI-driven world when the algorithms fail? What happens when the machine isn’t proffering advise, but rendering a decision and it’s wrong? More concerning, what happens when it’s the government using the technology and it’s not always right? At last season’s Outsell Leadership Council meetings, CEOs were hot on the topic about what to do when the machines are wrong. Whose liable? Does liability now move from the user to the provider of solutions? Published in our recent CEO Roundup:
What are the ethics of liability, and who will be responsible if and when algorithms take over? Human beings can usually articulate, however imperfectly, why they took the actions that they did. Simple rule-based computer programs leave a trail. But the latest cognitive systems cannot explain or justify their decisions. Why did the autonomous vehicle behave the way it did when its brakes failed? Who is responsible when it is hacked? Will data companies need insurance coverage for forward forecasts or will the usual legalese in marketing and delivery footnotes suffice? Will they need to advise customers that they should not rely, for business purposes, on the expensive systems they have just purchased?
In an AI world, we move from delivering search algorithms and lists of “stuff” (so yesterday, Google) to delivering answers. What happens when the answer is wrong? Or the answer needs some human overlay but humans fail to add judgement and common sense? The legal morass begins, and it gets murkier when the situation is around governments own use of this stuff.
We’re about to see this play out in real life, and it’s unfortunate at best. This past week the San Francisco Chronicle reported a computer tool cited as a factor in a local slaying. An algorithm that assigns risk scores to SF criminal defendants supported the release of a 19-year-old who within days gunned down a 71-year-old stranger. The city has been experimenting with the tool, which was designed by a foundation in Texas that’s fighting for criminal justice reform and uses the technology to predict which defendants to release while awaiting trial. It’s used in dozens of counties in the country and apparently has to do with helping determine bail affordability. I don’t know the ins and outs of the case, but I do know it’s pretty uncool when the DA and public defenders say we have to use caution before blaming the tool. “One tragedy doesn’t render it broken and it needs studying.”
Tell that to the poor guy who died.
There is indeed the possibility of human culpability because the law does not require judges to heed the algorithm’s suggestions. The article goes on to explain that some were shocked that the guy was released. As is always the case, people involved aren’t speaking due to “judicial ethics rules,” but this is a case when it’s actually the judicial system applying the technology. What a vicious circle, chase-your-tail moment.
There are legal issues here. There are ethics issues. There might be basic “training” issues as in “use the algorithm as input” but make the final decision a human one. Then one day, some human is going to make the wrong “human decision.” When an algorithm says “no” and a person overrules it, you all know what is going to hit the fan.
Anthea C. Stratigos is Co-founder & CEO of Outsell, Inc., the leading research and advisory firm focused exclusively on data, media, information, and technology. Get professional and personal lessons from a career spent mentoring successful leaders. Tell your story or ask a question — confidentially. Ask Anthea!