AI in medicine

Where AI is already doing clinical work

Start by being specific, because “artificial intelligence” covers everything from a logistic regression risk score to a large language model, and a candidate who uses the phrase as a single object is easy to dislodge. What has actually arrived in the NHS is narrow: systems trained on one well-defined task, producing one kind of output, sitting inside a workflow that a human still owns.

• Imaging and diagnostics. Automated grading of retinal photographs in diabetic eye screening, tools that push suspected intracranial haemorrhage or large-vessel occlusion up the reporting queue, and chest X-ray triage. AI-supported reading in breast screening belongs on the list too, but in the UK it sits in trials rather than routine practice.
• Triage and risk stratification. Deterioration and sepsis early-warning models, tools that rank surgical waiting lists by clinical priority, and models that predict who is unlikely to attend so the slot can be used.
• Ambient documentation. Speech tools that listen to a consultation and draft the note, the letter and the coding, returning time that currently goes on typing.
• Operations. Theatre scheduling, bed and demand forecasting, rota optimisation and supply logistics — unglamorous, and probably where the largest near-term saving actually sits.

Imaging is furthest ahead because the problem is well posed: a fixed input, a defined label, and enormous archives of historical examples with known outcomes. American regulators authorised the first autonomous diagnostic system — retinal photograph screening for diabetic retinopathy, reported without a specialist reviewing the image — in 2018. In breast screening, interim results published in 2023 from a randomised trial in Sweden reported that AI-supported reading picked up at least as many cancers as standard double reading while cutting the screen-reading workload substantially. Trials inside the NHS breast screening programme were under way at the time of writing, so AI reading is not yet routine here.

Ambient documentation appears to be spreading fastest, and for an unglamorous reason: it does not touch the diagnosis. A tool that drafts the note leaves the clinical decision entirely with the clinician while giving back a slice of a working day that is heavily consumed by typing. It is not risk-free — a language model can introduce content that was never said, the doctor remains responsible for the accuracy of the record they sign, and whether a given product is regulated as a medical device depends on what it claims to do rather than on how it is marketed. NHS England has set out expectations for organisations deploying these tools.

The research case, stated without overclaiming

The strongest example of AI changing science rather than administration is protein structure prediction. In 2020 DeepMind’s AlphaFold system predicted three-dimensional protein structures from amino acid sequence at an accuracy competitive with experimental methods for a large share of targets, and from 2022 the predicted structures for essentially every protein in the major sequence databases — more than 200 million of them — were released openly. The 2024 Nobel Prize in Chemistry recognised the work, shared with the separate achievement of computational protein design.

Why that matters is a question of cost. Determining a structure experimentally, by crystallography or cryo-electron microscopy, could take months or years for a single protein and was often the reason a research question went unanswered. A credible starting model now takes minutes and costs nothing, which changes what a structural biology question is worth attempting and gives structure-based drug design, enzyme engineering and the interpretation of disease-causing mutations a much better starting point.

The overclaim to avoid is the one candidates reach for. A prediction is a hypothesis with a confidence score attached, not a measurement. Proteins are not static objects; many do their work through movement, disorder and interaction with partners the prediction does not include. More importantly, structure was never the rate-limiting step in drug development. The overwhelming majority of candidate drugs fail in clinical trials for lack of efficacy or unacceptable toxicity in human beings, and nothing about a better starting structure fixes that. This shortens one early stage of a pipeline that still takes the better part of a decade.

Where it fails: bias, black boxes and deskilling

Bias, and why it is worse in a model than in a person

The best-documented failure mode is what happens when the training data does not represent everyone the tool will meet. Dermatology is the standard example: models that classify skin lesions from photographs learn from whatever image archives exist, and repeated audits of the large open dermatology datasets have found that images of the darkest Fitzpatrick skin types make up a very small fraction of the total. A classifier trained on that distribution can validate well overall and perform worse precisely in the group where melanoma is already diagnosed later and does worse.

The general principle matters more than the example. A model learns the distribution it is shown, including the inequities baked into who historically got scanned, referred, coded and believed. A widely used American algorithm that identified patients for extra care management used prior healthcare spending as a proxy for health need; because less had historically been spent on Black patients at the same level of illness, the model systematically selected fewer of them. The analysis was published in Science in 2019, and the code contained no variable for ethnicity at all. Bias entered through the choice of label, which is why auditing for it is harder than reading the source.

Two extensions are worth owning. The first is that this is not unique to AI: pulse oximeters have been shown to overestimate oxygen saturation in people with darker skin, a discrepancy that mattered during the pandemic, and a review commissioned by the UK government reported on equity in medical devices in 2024 covering optical devices and AI-enabled tools together. The second is the structural point. Human error is idiosyncratic and roughly independent — two radiologists miss different things. A model’s error is systematic: the same mistake, made identically, on every patient it touches, which is why the same accuracy figure means something different when it is one algorithm rather than a thousand clinicians. If this argument is unfamiliar, health inequalities sets up the underlying pattern.

The black box: why a doctor cannot own a reason they cannot see

Many of the best-performing models are not interpretable in the way a clinician needs. You can see the input and the output, but not a reason you could interrogate, challenge or teach. That is a professional problem rather than a technical curiosity, because the standard a doctor is held to is not “was the answer correct” but “can you justify the decision you made”. The edition of Good Medical Practice that took effect in January 2024 expects doctors to work within the limits of their competence and to be prepared to justify their decisions; “the software said so” is not a reason, and a clinician cannot take responsibility for a decision they are unable to interrogate.

The counter-argument is strong enough that raising it yourself scores well. Medicine already accepts interventions whose mechanism nobody can fully explain — lithium and paracetamol are the usual examples — on the strength of evidence that they work, and on that view what matters is not whether the reasoning can be inspected but whether performance has been established in the population you are applying it to. Insisting on interpretability may then mean choosing a less accurate tool, which is itself a harm. The reply is that a doctor discharges their duty patient by patient rather than in aggregate: trial evidence tells you the tool is usually right, not why it is saying this about this patient, which is what consent, challenge and appeal each need. Where the balance sits is genuinely unsettled, and saying so is better than pretending either side is obvious.

Post-hoc explanations such as heatmaps over an image help, but they are models of the model and can be unstable. There is a related failure that interviewers love: models have been shown to key on artefacts rather than pathology — the laterality marker, the scanner make, the presence of a chest drain, a ruler placed beside a mole — because those features correlated with the label in training. A system that appears to detect disease may partly be detecting which hospital took the image, which works beautifully until it is deployed somewhere else.

Automation bias and deskilling

Then there is what the tool does to the human. Automation bias is well described outside medicine and inside it: under time pressure people accept incorrect prompts, and — the sharper edge — abandon their own correct judgement when the machine disagrees. The earlier generation of computer-aided detection in mammography was adopted very widely in the United States before large studies found it did not improve screening accuracy overall. A tool can be accurate in a trial and unhelpful in a clinic if it changes how the human reads.

Deskilling is the slow version of the same problem. If a generation of trainees never reads a plain film unaided, the human backstop degrades exactly when it is most needed: when the tool is offline, or when the patient in front of you is outside the distribution it was trained on. That argues for a design principle worth stating in an interview — automation is safest where the human keeps a genuine task rather than a rubber stamp, which is why a second-reader configuration behaves very differently from one where the model reads first and the human confirms.

Data, regulation and the liability gap

Models need data, and NHS data is confidential patient information. Britain has already run this argument twice in public. The care.data programme was abandoned in 2016 after failing to carry public consent, and in 2017 the Information Commissioner found that the Royal Free London NHS Foundation Trust had not complied with data protection law when around 1.6 million patient records were shared with DeepMind to test an acute kidney injury application. The clinical intent in that case was not seriously in doubt; the legal basis and the reasonable expectations of patients were. Public trust is a hard constraint on this field rather than a soft one, and the national data opt-out exists because the public asked for a lever.

Consent for training data is the sharper version of the problem. Most patients have no idea that their imaging might be used to build a commercial model, and the standard mitigations each have a hole in them. Anonymisation is not absolute for high-dimensional data — a face can be reconstructed from a head CT — and consent given today for research uses nobody has yet imagined is difficult to call meaningful. Secure data environments, where researchers bring their code to the data rather than taking copies away, are the current direction of travel, and they are a governance answer rather than a complete one.

Regulation runs into a structural mismatch. In the UK, software intended for a medical purpose is a medical device regulated by the MHRA, and NICE has used an early value assessment route to make conditional recommendations while further evidence is gathered. But device approval assumes a fixed product: you evaluate a thing and then that thing is what ships. A continuously learning model changes after approval, so what was assessed is no longer what is deployed. The direction regulators have moved in — developed furthest by the American approach of agreeing a change control plan up front, and reflected in UK proposals — is to assess a pre-specified envelope of change, covering what the model may learn from, under what monitoring, within what performance bounds, rather than only the frozen version. In practice, most tools deployed in the NHS today are locked rather than continuously learning, precisely because the alternative is so hard to govern.

1. A defined clinical question, and use of the tool only for the intended purpose it was actually validated against.
2. Local validation on your own population before go-live, rather than reliance on the manufacturer’s published performance in someone else’s cohort.
3. An explicit human role written into the workflow — triage prompt, second reader, or autonomous — so that everyone knows what accepting an output commits them to.
4. Post-deployment monitoring including performance broken down by subgroup, because drift and disparity are invisible unless somebody is looking for them.
5. A route for a clinician to disagree without penalty, and for a patient to ask that a human reviews the decision.

Liability is the question nobody has answered. If an algorithm is wrong and a patient is harmed, there are three plausible defendants: the clinician who accepted or overrode the output, the trust that procured the tool and designed the workflow around it, and the manufacturer whose product may be defective. At the time of writing, professional and legal responsibility for a clinical decision in the UK still rests with the clinician and the organisation, and there is no settled body of case law apportioning the manufacturer’s share. Saying clearly that the position is unresolved, and explaining why it is hard, is a stronger answer than a confident wrong one.

Will AI replace doctors?

This question comes up constantly, in panels and in MMI stations, and most candidates answer the mood rather than the question. The move that works is to separate three things that get collapsed together: tasks, roles and accountability. Software substitutes for tasks. Roles are bundles of tasks and can be re-bundled. Accountability is a legal and professional relationship, and it does not automate at all.

The defensible landing, on what is known now, is augmentation and role change rather than replacement — though it is a judgement about an open question rather than a settled fact, and it is stronger if you say what the change costs. Doctors will need skills the curriculum barely teaches yet: appraising a tool’s evidence base, knowing its intended use and its validated population, recognising when the patient in front of you sits outside it, and explaining to that patient what part software played. Role change is not free either — it can compress training pathways, hollow out the routine work that trainees learn on, and concentrate the interesting cases in fewer hands.

Running the whole topic through the four pillars

This article sits in stage five for a reason: it is an ethics question wearing a technology costume, and it collapses without the machinery from the four pillars. Running any AI scenario through them takes fifteen seconds and immediately gives you a structure that is not a list of opinions.

• Autonomy. Does the patient know a model was involved, can they ask for human review, and is consent meaningful when nobody in the room can explain the reasoning behind the recommendation?
• Beneficence. The gains are real and should be conceded generously: earlier detection, fewer missed findings, faster stroke pathways, and time given back to the consultation is itself a clinical good.
• Non-maleficence. False positives cascade into anxiety, invasive follow-up and overdiagnosis, and the harms of a flawed model arrive at scale and in the same direction for everyone affected.
• Justice. Who was in the training data, who gets the tool, and who is excluded by an access route that quietly assumes a smartphone, data allowance and confident English?

Practise it the same way you practise any contested topic: state the strongest version of the optimistic case first, then the specific failure mode, then what would have to be true for you to change your mind. Answering a dilemma live gives you the four-move structure that survives a timer, and genomic medicine and screening, the next stop on this path, runs almost exactly the same argument about prediction, consent and inequality through a different technology.