The problem of AI ethics, and laws about AI

This post is by Benedict Evans from Essays – Benedict Evans

In the late 1990s, the UK Post Office deployed a new point-of-sale computer system, built for it by Fujitsu. Almost immediately, post-masters, who are self-employed and generally small independent retailers, started reporting that it was showing shortfalls of cash; the Post Office reacted by launching prosecutions for theft. Close to a thousand people were convicted over the next 15 years, many more were falsely accused and settled, and there were at least four suicides.

Meanwhile, Fujitsu and the Post Office knew that the system was full of bugs that could cause false shortfalls to appear, but Fujitsu and Post Office staff went to court and testified that the system was working correctly and that theft was the only explanation. This has now, understandably, become a huge scandal.

I think about this case every time I hear about AI Ethics and every time people talk about regulating AI. Fujitsu was not building machine learning or LLMs – it was 1970s technology. But we don’t look at this scandal and say that we need Database Ethics, or that the solution is a SQL regulator. This was an institutional failure inside Fujitsu and inside the Post Office, and in the court system failing to test the evidence properly. And, to be clear, the failure was not that there were bugs, but in refusing to acknowledge the bugs. But either way, to take the language that people now use to worry about AI: a computer, running indeterminate software that was hard to diagnose or (Read more…)