How humans and AI should work together in critical care

While AI is transforming how the entire software industry works (side note: Or according to some: how everyone will soon be out of work.), adoption in more risk-averse industries such as clinical practice lags behind. One major obstacle is that AI-based systems fail to account for the broad sociotechnical system in which humans and AI collaborate: advancements in AI must be accompanied by building human competencies. Contrary to popular belief, more capable systems do not necessarily reduce human effort or errors (side note: The crash of Air France 447, caused by over-reliance on automation, was a notable example of this.), and may paradoxically even lead to increased demands for highly skilled personnel. Another major issue is a lack of transparency in today’s black-box AI systems, which raises the question of who should be responsible for the system’s outcomes.

This study tackles the question of how humans working in intensive care units (ICUs) can effectively collaborate with AI based on surveys and interviews with (I’d say, a fairly large number of) data scientists and clinicians.

The study covers four levels of automation in which AI:

1. does not provide real value over humans
2. can improve human performance, but humans are always needed
3. could act alone, but human input increases reliability
4. is able to fully automate tasks without human input

For six core ICU tasks:

1. monitoring patient data
2. documenting clinical information
3. analysing medical data
4. prescribing medication or treatment
5. diagnostic decision-making
6. interacting with patients.

About 72% of surveyed data scientists agree that AI could eventually fully automate monitoring of patient data. AI is especially suited for this task because even experienced clinicians are overwhelmed with the amounts of data that must be monitored. Moreover, occasional system failures are unlikely to cause patients to die.

A large majority of data scientists also agree that AI could eventually automate documentation of clinical information – a task which many clinicians find boring – although human assistance would increase reliability.

Something similar can be said about automating the generation of analysis reports of medical data, which would enable clinicians to focus fully on interpreting analyses.

Things start to get a bit trickier when it comes to prescribing medication or treatment. Most data scientists agree that this task can be supported by AI, but would always require humans in the lead to ensure that suggestions are approved by a medical professional who is accountable and takes legal liability.

The same arguments hold for diagnostic decision-making: many data scientists have social and ethical concerns about augmenting diagnostic decision-making with AI, even if it is technologically feasible to do so. Some also feel that using AI for this would undermine physicians’ role identities.

Regarding interaction with patients, there is strong agreement that this is not something that should be augmented or automated by AI, based on social concerns and what it means to be human.

Like the surveyed data scientists, clinicians acknowledge the benefits of automating tasks, as long as systems are highly reliable. Otherwise they would have to constantly supervise the AI, which defeats the whole purpose of saving time.

Clinicians are particularly hopeful about a future in which clinical documentation is fully automated, and only needs to be checked for potential errors.

They agree with the data scientists when it comes to analysis of medical data and prescription of medication or treatment: the ultimate decision power must remain with humans and doing otherwise is dangerous.

However, whereas data scientists believed that clinicians would be worried about loss of status or power if diagnostic decision-making were automated, not a single clinician voiced such concerns. Instead, they focused entirely on the benefits that AI could bring to patients. Their only real worry appears to be that medicine cannot be generalised, and each patient must be assessed as a unique human being.

Finally, clinicians again agree with data scientists that interaction with patients should be left to humans. For many clinicians, interaction with patients is one of the reasons why they chose their professions, and they hope that AI can help them invest more time in it.