Medical AI must not lead to Post Office Horizon-like scandal, says Northumberland academic and councillor
The use of Artificial Intelligence (AI) in medical settings must not replicate the Post Office ‘Horizon’ scandal, a computer expert and Northumberland County councillor has warned.
Councillors from across the North East gathered at Gateshead Civic Centre to hear about the potential future uses of AI throughout the NHS.
Dr Michael Smith informed elected members that while AI is not at the stage of genuinely affecting patients, work will need to be undertaken to ensure people remain at the helm of technological change.
Dr Smith also stated that care will need to be taken in relation to what any AI system is fed in order to arrive at medical conclusions, emphasising possible discrepancies between population groups.
Paul Ezhilchelvan, councillor for the Cramlington South East ward and reader in distributing computing at Newcastle University, said: “You alluded to it, one of the problems with this software is it is not perfectly reliable.
“You alluded to it, that 5% of the time it can fail. Building on top of that is how we are going to cope with the administrative system, those 5% of cases.
“It won’t come out immediately, it’ll come out slowly, obviously. If we don’t have human centric processes in place then we will have a scandal much bigger than Horizon.
“In the Horizon scandal, the system software was bad, but the people tried to conceal it. I’m not pointing fingers.”
Coun Ezhilchelvan continued: “We need to be building some serious human centric process to monitor those 5% of cases. We must be open to suggestions that things are not as perfect as it seems.”
Dr Smith responded: “We are not quite there yet with the things that have been implemented are essentially safety nets for existing human processes rather than machine led processes like Horizon.
“Once we get to the next stage, when we start thinking about those things, and the clear ethical boundaries, we will be thinking about it and incorporating those things.
“How do we build those appeal mechanisms, so you can go and say I think the machine has made a mistake and you’re not just met with a computer that says no. That attitude is just unacceptable.”