Publication date: November 04, 2026
Errors, bias, opacity, data security and the accountability gap – what patients, clinicians and regulators need to watch
Artificial intelligence is already reading scans, ranking medical literature, predicting kidney failure and managing hospital beds. Its potential to make medicine faster, more precise and more personalised seems almost limitless. Yet the same technology brings a distinct set of risks: algorithms that fail silently, tools used without adequate training, models that reproduce existing inequalities, decisions that no one can explain, patient data that is exposed or re-used, and a legal framework that still struggles to say who is responsible when something goes wrong. This article maps those risks and outlines what is needed to manage them.
Interest in the use of artificial intelligence (AI) in healthcare has grown rapidly in recent years. Medical science is constantly expanding its horizons and generating ever larger and more diverse volumes of information. AI plays a key role in this evolution, becoming increasingly integrated into medical practice and offering healthcare systems in Europe a way to respond to the pressures they face.
Today, AI is used in medicine mainly for the quantification of medical images, the automated analysis of genetic data, disease prediction, medical robotics, telemedicine and virtual assistants. The COVID-19 pandemic significantly accelerated the development and deployment of AI applications in medical and clinical settings, as AI technologies became a central element of the response to the global health crisis.
AI is now applied across radiology, ophthalmology, dermatology and pathology. In cardiology, it is transforming clinical practice through the automated processing of cardiac data: machine learning models improve the diagnosis of heart disease by analysing complex imaging such as echocardiograms and electrocardiograms, help assess cardiac risk and predict treatment outcomes, contributing to better patient care and a lower burden of cardiovascular disease. Although AI is less widespread in nephrology, progress is being made in the diagnosis of kidney disease. Machine learning models can analyse ultrasound images of the kidneys and histopathological findings to identify pathological changes and assess disease severity, and AI can predict acute kidney injury from clinical and laboratory data, enabling faster diagnosis and treatment of patients in potentially life-threatening conditions. In biomedical research, AI is used primarily to analyse scientific publications: platforms such as PubMed use machine learning algorithms to rank search results, and newer tools such as the Best Match algorithm improve access to clinical knowledge.
In surgery, decisions sometimes have to be made quickly and without complete certainty about the diagnosis or the patient’s likely response to treatment. A lack of data or insufficient evidence can push clinicians towards mental shortcuts and intuition, which may lead to errors and bias. Traditional decision-support tools often fail to account for these limitations. AI-based models show promising efficacy in predicting risk for individual patients, which may prove particularly valuable in cardiovascular medicine.
AI is not confined to clinical medicine; it is also used in healthcare administration. One argument for deploying AI in administrative work is that errors there are usually less critical than errors in the clinical setting. AI-based applications can play a key role in organising patient flow, for example by managing bed availability, which is an important factor in surgery cancellations. Another administrative area is clinical coding – the process of extracting information from clinical records and classifying it under systems such as the International Classification of Diseases (ICD) or diagnosis-related groups (DRGs). Coding accuracy is crucial for cost recovery, management and analytics. Although computer-assisted coding has existed for some time, AI can significantly increase the accuracy and transparency of the process.
Despite these benefits, concerns about the possible negative effects of AI in medicine have grown in recent years. This fast-developing field raises considerable concern among healthcare professionals, patients and the public regarding clinical safety, equal access, privacy and security, appropriate use, liability and regulation. These risks can lead not only to harm to patients and citizens but also to a loss of public trust in AI algorithms and in doctors themselves.
One of the main challenges is the unavoidable possibility of algorithmic error, which can have serious consequences for patients. Errors may have different causes and manifest themselves at different stages of the diagnostic and therapeutic process. A misdiagnosis may result in unnecessary treatment or in a serious illness being overlooked. In emergency departments, inappropriate interventions may follow from imprecise diagnoses or incorrect triage prioritisation.
An important aspect is the way data is collected and used in the machine learning process. Noise in input data, whether caused by technological imperfections or human factors, can significantly reduce the predictive accuracy of AI algorithms. Ultrasound imaging, for example, although widely used in clinical practice, is prone to operator-dependent scanning errors that can lead to incorrect results.
Dataset shift – a change in the statistical distribution of the data on which an algorithm was trained – is another major problem. It may result from differences between populations, from differing data acquisition protocols across medical facilities or from the use of different diagnostic technologies. A model trained on data from one type of scanner, for instance, may lose accuracy when applied to data from another.
AI algorithms may also have difficulty adapting to unexpected changes in the clinical environment. What a radiologist would immediately recognise as an artefact on an X-ray may be interpreted by an AI system as a pathological change, leading to false-positive results and unnecessary further investigation or intervention.
To minimise these risks, standardised methods for the assessment and approval of AI solutions by the competent regulatory authorities are essential. In keeping with the principles of ethics and safety, AI algorithms should be designed as tools that support doctors rather than replace them. It is also important that AI solutions remain dynamic, with the ability to keep learning from new data and clinical situations – under appropriate controls.
Implementing such solutions will, however, require significant investment in infrastructure and technology, as well as appropriate integration policies in healthcare facilities. Collaboration between all actors – scientists, physicians, regulators and technology manufacturers – will be crucial for the successful introduction of AI into healthcare.
The role of the human factor in the use of AI in medicine cannot be underestimated. Even the most advanced algorithms are prone to error, and their effectiveness depends on proper use by end users, including doctors, other medical staff and patients. Without adequate understanding and skill, there is a risk that AI tools will be used incorrectly, leading to wrong diagnoses, treatment plans and clinical decisions that endanger patients’ health.
Clinicians and medical staff must not only have access to AI technology but also be properly trained to use it. Current medical curricula often contain too little on artificial intelligence, leaving doctors to acquire new knowledge and skills quickly in order to use these tools effectively in practice. In addition, the absence of clinical experts from the design and development of AI technology can result in tools that are difficult to use and poorly matched to real clinical needs.
Patients, too, must be aware of and able to use medical AI appropriately. As more and more AI applications and tools become available to the general public, there is a risk that patients will rely on them without understanding their limitations and the uncertainty surrounding their accuracy. It is therefore important to inform patients about both the possibilities and the limits of AI in healthcare and to encourage a healthy, critical evaluation of the information these tools provide.
The development of medical AI applications also creates an urgent need for regulation and supervision. For years, many AI-based applications reached the market without sufficient evaluation of their clinical effectiveness and safety. The EU has now begun to close this gap: AI-enabled medical devices are subject to the Medical Devices Regulation (MDR), the AI Act (Regulation (EU) 2024/1689) classifies most of them as high-risk systems with phased-in obligations, and the European Health Data Space regulation sets rules for the secondary use of health data. Clear regulatory guidelines and assessment and approval procedures that reflect specific clinical and ethical requirements remain essential, however, and enhanced regulatory and supervisory efforts can help ensure that AI technologies are tailored to end-user needs and meet the highest standards of safety and effectiveness.
Ultimately, minimising the risk of error and misuse of medical AI requires the broad involvement of all stakeholders – physicians, medical staff, patients, regulators and technology manufacturers. Working together to develop, implement and evaluate AI in healthcare can benefit both patients and the healthcare system as a whole by improving diagnostic accuracy, treatment effectiveness and access to care.
Inequality in healthcare is a complex, multi-factor problem. Many people experience discrimination based on gender, age, ethnicity, socio-economic status or geographical location. These inequalities may be systemic, resulting from socio-economic neglect, lack of access to healthcare or direct discrimination by medical staff.
One of the main sources of these inequalities is human bias, which can lead to patients being treated differently on the basis of characteristics such as race or gender. Research conducted in the United States has shown that Black patients often receive inferior medical care compared with white patients. Similarly, women frequently experience delayed diagnoses for certain conditions, such as endometriosis, and their pain is more likely than men’s to be attributed to psychological causes.
The introduction of AI into healthcare seems promising, but without proper regulation it may exacerbate these inequalities. Medical algorithms can be biased because they are trained on data that is incomplete, unrepresentative or itself the product of biased decisions. If diagnostic algorithms are trained mainly on data from white patients, for example, they may be less effective at diagnosing disease in patients of other ethnic backgrounds. There is also a risk of perpetuating existing inequalities where AI models are trained on data that reflects them.
Addressing these issues requires close supervision and evaluation of medical algorithms and assurance that training data is representative and balanced across population characteristics such as race, gender, age and socio-economic status. It is also necessary to involve a broad range of experts – sociologists, biomedical ethicists, public health professionals and patients – to ensure that AI models are fair and effective for all social groups.
Amid growing interest in medical AI, an algorithm developed by Google for breast cancer screening sparked debate about the role these technologies can play in improving diagnosis and care. The algorithm delivered promising results, improving both the speed and reliability of screening and in some cases outperforming radiologists. Its ability to generalise to populations outside the regions on which it was trained attracted further interest.
The enthusiasm was accompanied by concern. The absence of details about the algorithm itself and of key technical descriptions led some to question the transparency and safety of the tool. This concern was echoed in the media and the AI community, where critics argued that a lack of full transparency could compromise both the usefulness and the safety of the algorithm.
Transparency in healthcare AI operates on two levels: traceability and explainability. Traceability refers to the documentation of the AI development process, including the model, the training data and post-deployment monitoring of the tool. In practice, existing tools are often delivered without full traceability, which undermines users’ trust.
Explainability concerns the transparency of the decisions AI makes. In the case of algorithms based on deep neural networks, often referred to as “black boxes”, the absence of this transparency makes it difficult for clinicians and other stakeholders to understand the basis on which a decision was reached.
Various mitigation measures have been proposed to improve transparency and trust. An “AI passport” documenting key information about a model, together with tools for tracking the use of algorithms after deployment, could significantly improve traceability. Involving clinical end users from the earliest stages of algorithm development can also help develop effective strategies for explaining AI decisions.
Finally, regulators have an important role to play. By treating traceability and explainability as prerequisites for certification, they can set standards that encourage greater transparency in the development and application of AI algorithms in medicine.
The growing use of AI in healthcare, particularly during and after the COVID-19 pandemic, has highlighted the risks to privacy, confidentiality and the protection of patient data. Lack of informed consent may lead to breaches such as the disclosure of protected data or the re-use of patient data for other purposes.
Opaque AI algorithms and complex consent forms limit patient autonomy and hinder shared decision-making with the physician. A well-known example is the 2016 case in which patient records held by a London NHS trust were transferred to DeepMind without the patients’ informed consent.
AI in healthcare also carries the risk of data breaches, which can lead to identity theft and other cyberattacks. In 2020, for instance, a data breach at Cense AI exposed sensitive data relating to more than 2.5 million patients.
The re-use of data is a further risk, potentially leading to unauthorised use of patient data for purposes unrelated to healthcare. There is a need to raise awareness and skills regarding privacy and security risks and to extend the regulatory and legal framework. Support for decentralised approaches to AI and continued research into the security of cloud-based systems are also crucial.
The concept of “algorithmic accountability” is gaining importance wherever AI is used, and especially in healthcare. It refers to the principle that people – not AI systems – are responsible for errors or improper actions of algorithms. The introduction of AI tools in medicine brings new challenges in this respect, and a sound solution to the question of responsibility is crucial for public acceptance of and trust in these technologies.
There is still considerable uncertainty as to who is responsible for AI-related medical errors, which calls for the adaptation of regulations and the legal framework to the specific characteristics of these technologies. One such challenge for policymakers and legislators is the “many hands” problem – the difficulty of allocating responsibility among the various entities involved in the development, implementation and use of AI. At EU level, the revised Product Liability Directive (Directive (EU) 2024/2853), which expressly covers software and AI systems, is a first step towards addressing this, while the proposed AI Liability Directive was withdrawn, leaving fault-based liability largely to national law.
A further difficulty lies in identifying the precise cause of an error involving medical AI. The cause may lie in the algorithm, in the data used to train it or in a misunderstanding of how the AI operates in clinical practice. The absence of uniform ethical and legal standards across the AI industry makes these problems harder to resolve.
Responsibility in medical AI is also closely linked to the clarity and transparency of algorithms: the less understandable an algorithm is, the harder it becomes to determine who is responsible. Proposals to address this include establishing processes that define the respective roles of AI developers and users, and setting up dedicated regulatory bodies.
Over the past five years, numerous medical AI algorithms have been proposed for a wide range of healthcare applications. Yet even where these technologies are clinically proven, safe and ethical, their implementation faces many practical obstacles. Healthcare professionals are often slow to adopt new technologies in daily practice. The implementation phase is critical, and beyond the creation of the algorithm itself, other factors must be taken into account, such as the limitations and quality of medical data and changes in the doctor-patient relationship.
The quality of medical data is essential for the effective use of medical AI, but such data is often unstructured and noisy. It must therefore be verified, quality-controlled and labelled, which requires considerable time and money. In addition, the introduction of AI may change the relationship between patients and healthcare professionals, creating new challenges in communicating disease risk information and updating clinical guidelines.
Bringing AI into clinical practice also requires interoperability and integration with existing health systems. Manufacturers and healthcare organisations will need to establish standard operating procedures and work together to ensure that new AI tools interoperate with existing technologies such as genetic sequencing and electronic patient records.
The use of AI in medicine brings numerous benefits, but it also involves significant challenges and risks. Beyond technical aspects such as data quality and interoperability, changes in the relationship between patients and healthcare professionals and the need to update clinical guidelines must also be considered. Implementing AI in clinical practice requires collaboration between technology manufacturers and healthcare professionals to ensure the safe and effective integration of new tools into existing healthcare systems. Ultimately, a clear-eyed awareness of these risks is the precondition for harnessing the potential of AI to improve healthcare.