Health Canada publishes guiding principles for AI and machine learning in medical devices

Health Canada has released a set of guiding principles designed to promote the safe and effective use of medical devices enabled by artificial intelligence and machine learning.

The principles, published in conjunction with the U.S. Food and Drug Administration and the U.K. Medicines and Healthcare products Regulatory Agency, are a step toward modernizing the regulatory approach to medical devices and harmonizing it internationally.

Unlike medical devices of the past, the potential of artificial intelligence/machine learning (AI/ML) medical devices lies in their power to learn from real-world data and improve performance over time. Their impact on healthcare could be transformative, but the devices also carry risks that regulators are ill-equipped to assess and manage.

As regulators adapt, they must strike a balance between ensuring AI/ML medical devices are safe and effective without stifling innovation or impeding patient access to potentially life-saving technologies.

In an interview with Research Money, Muhammad Mamdani, the director of the Temerty Centre for AI Research and Education in Medicine (T-CAIREM) at the University of Toronto, said the guiding principles are urgently needed and a step in the right direction. 

“I don't think we can move at a slow pace. These are going to radically change how we do business in healthcare. It's not 50 years from now. It's already happening,” he said.

He added that Unity Health Toronto, where he is Vice President of Data Science and Advanced Analytics, is already using an AI-driven early warning system that gives clinicians 24 to 48 hours advance notice that a patient's condition is likely to deteriorate. Similar systems are in place nationally in the U.K., he said. 

The 10 guiding principles include implementing good software engineering and security practices, ensuring data sets are representative of the intended patient population, monitoring the performance of the AI/ML model and the human-AI team, and providing clear, essential information to users.

“AI is early days, and healthcare and AI is even earlier days. These principles help establish the rules of engagement around how to move forward in what has the potential to be the largest paradigm shift in healthcare,” said Aaron Leibtag, co-founder and CEO of Pentavere, in an interview with Research Money.

The Toronto-based company uses AI to extract information from unstructured healthcare data, such as clinical notes. Its technology works upstream of medical devices to deliver evidence that could be used by medical devices.

Both Leibtag and Mamdani said the guiding principles carry weight coming from regulators in Canada, the U.S. and the U.K., and that the alignment across jurisdictions is important.

In a statement to Research Money, Health Canada said the international alignment helps companies and the healthcare system, and that conversations with "trusted, like-minded regulators" are ongoing. Health Canada is an active participant in international work on artificial intelligence and machine learning, including the International Medical Device Regulators Forum’s AI working group and the World Health Organization/International Telecommunication Union AI4Health focus group. 

Toward a set of standards

Medical devices are the workhorses of medicine, from latex gloves and sutures to pacemakers and cochlear implants. The latest AI-powered tools include imaging devices that can help physicians detect respiratory illnesses, cancer and other serious medical conditions, and software tools to detect and predict sepsis. They could deliver earlier disease detection, more accurate diagnoses, and the development of personalized therapeutics.

Health Canada has been working to adapt its regulatory approach to digital health technologies since 2018, when it launched the Medical Devices Action Plan. In 2020, the agency launched the Medical Device Directorate in recognition of the fast pace of medical device development and the need to regulate them across the product lifecycle from pre-market development to post-market surveillance.

It is also establishing a new regulatory Advanced Therapeutic Products (ATPs) Pathway to authorize innovative products in a more flexible and agile manner than can be done today. ATPs are drugs or devices, such as AI/ML medical devices, that the agency's current regulations were not designed to handle.

Health Canada has already authorized a handful of AI/ML medical devices, but the new tools use algorithms that are "locked," meaning they don't adapt over time. One device, XrAI, is a tool developed by 1QBit that can identify lung abnormalities that could be associated with cancer or COVID-19 on chest radiographs. Early in the pandemic, the company received funding from the Digital Technology Supercluster to accelerate deployment to hospitals in British Columbia, Saskatchewan and Ontario. 

The guiding principles will help establish the “Good Machine Learning Practices” — a quality management system for AI/ML medical devices — needed to ensure they do no harm.

Lack of relevant data could hinder the field

Mamdani said he was impressed with the guiding principles, but expected to see more focus on mitigating bias. New imaging tools have been developed to distinguish between malignant and benign breast cancer tumours, helping physicians diagnose the disease earlier and more accurately. But unless the algorithms are developed and tested on diverse populations of women, they have the potential to perpetuate disparities in care, especially for women of colour.

In Canada, he said, it may even be difficult to assess whether algorithmic or data biases are leading to discrimination because Canadian hospitals often don’t collect data on race. “The problem here is, do you even have data on race to know that it's biased against certain races? In many cases, we don't even have that data,” he said.

To ensure new products are fair, they need to be trained on diverse and representative clinical datasets. However, women and people of colour have been historically underrepresented in medical research, as have people with disabilities. Age and geography are also important considerations.

Mamdani also stressed the importance of the clinical setting. For AI models to perform optimally, they should be trained on data that is representative of their intended patient population. A new article by Canadian law experts echoes this view, questioning to what extent AI/ML-powered products developed and trained on data from U.S. hospitals are generalizable to Canadians. The authors warned that the products “may reflect the socio-economic biases that impact access to health care in the U.S.”

Leibtag applauded the principles’ emphasis on using the appropriate data for training and testing. But, he said, that assumes that the right data is available to achieve the desired outcome from your machine learning device.

As Pentavere demonstrated, a large amount of useful medical data in Canada is siloed by jurisdiction and inaccessible, or captured in the kind of unstructured text the company mines for insights. He said we need to think about “how to unleash high-quality data as a starting point” for research and innovation in Canada.

He added that the real test for the guiding principles is bringing them to life. “The challenge ahead is to take policy and support real use cases, so taking a crawl, walk, run approach toward system transformation," he said.

Mamdani said he expects the guiding principles to evolve into standards. “What we need are the actual requirements that align with the principles,” he said.

In a statement, Health Canada said it is working on new guidance to help medical device manufacturers to further understand the factors they should consider in their algorithm development as well as Health Canada’s expectations for safety, effectiveness, and quality. This guidance is expected to be posted for public consultation in 2022.

R$