Google: Detecting Heart Rate Through Smartphone Camera AI
A team at Google has published research into how smartphones can measure heart rate passively while people use them normally. The work centres on machine learning methods that analyse facial video captured by front-facing cameras.
Resting heart rate is a biomarker that could indicate cardiovascular health and long-term risk. According to the research, high resting heart rate is associated with adverse cardiovascular events and certain chronic illnesses. Around five billion people worldwide own a smartphone with the hardware needed to track this metric.
Deep learning for health monitoring
In 2022, Google demonstrated a method for measuring heart rate by placing a finger over a smartphone camera. The new research introduces what the company calls passive heart rate monitoring. This system tracks heart rate in the background during normal smartphone use.
The technology uses the front-facing camera to record video of a user's face. A deep learning model then estimates heart rate from these recordings. According to Google, the system achieves a mean absolute percentage error of less than 10%. This meets industry accuracy standards across all skin tones.
Eric S. Teasley, Product Manager and Ming-Zer Poh, Staff Research Scientist at Google Research, say: "To our knowledge, PHRM marks the first large-scale demonstration of passive HR and daily RHR monitoring during everyday smartphone use."
The pair add: "As the only rPPG (remote photoplethysmography) method to meet heart rate accuracy standards for people of all skin tones – even in unpredictable real-world conditions – it sets a new standard for the field. It also represents the first use of rPPG to estimate daily RHR, achieving wearable-level accuracy across all skin tones."
Neural networks process facial video
The system measures heart rate by detecting how light interacts with skin each time blood pulses through it. An on-device software pipeline processes short clips of facial video. Temporal shift convolutional neural networks then predict heart rate from these clips.
Previous studies in this area have underrepresented people with dark skin. Melanin makes the signals more difficult for cameras to detect. Google's team developed the system using more than 350,000 video clips from nearly 700 participants.
The researchers tested participants in both laboratory and real-world settings. They devoted model training to the most complex cases. The team used the Monk Skin Tone Scale to ensure representation across skin tones.
According to the research, participants with light and medium skin tones each comprised at least 25% of the datasets. Participants with dark skin tones comprised at least 33%. This makes it the largest and most diverse remote photoplethysmography study to date.
Testing across multiple conditions
Researchers trained the system to handle various conditions in laboratory settings. They recorded facial video and simultaneous electrocardiogram data from 365 study participants. The passive heart rate monitoring system outperformed 15 of the leading published remote photoplethysmography models on the same test.
The team also trained the model on real-world data through a study with 231 participants. These participants installed a data collection app on their phones and used them normally. They wore an electrocardiogram chest strap and a Fitbit tracker during the study.
The app recorded an average of 231 video clips per day during this phase. The researchers note that optimising camera exposure could improve performance. Reducing the impact of excessive head movement could also be explored in subsequent work.
Google intends to make its data and modelling resources available to qualified researchers.
