阅读说明：本技术 一种基于图像处理的驾驶员心率识别方法 (Image processing-based driver heart rate identification method ) 是由 陈昌川 刘凯 王海宁 代少升 刘科征 吴占杰 于 2021-06-10 设计创作，主要内容包括：本发明的内容为实现了一种基于图像处理的驾驶员心率识别方法,克服了接触式识别心率的缺点,实现了对驾驶员身体心跳的监测,实现驾驶员心率的识别,具体技术方案包括以下6个部分。感兴趣区域选取：驾驶员感兴趣区域的选取基于人脸68个特征点,选取感兴趣区域。特征提取：在感兴趣区域的基础上,提取其中的绿色通道特征作为驾驶员心率的真实表现,由于感兴趣区域内的特征提取易受光照分布不均的影响,本发明采用算法消除光照误差。转换成频域：构建小段绿色通道序列值,缩减误差,经FFT求得频率值。噪声过滤：过滤掉不符合常理的噪声。心率提取：根据绿色通道频率值与心率的关系求得粗略的心率值。心率平稳：根据上下帧,经算法求得平滑稳定的心率值。本发明具备非接触式,不会给驾驶司机造成不舒适的感受,同时避免传统设备心率识别携带不方便的问题。(The invention aims to realize a method for identifying the heart rate of a driver based on image processing, overcomes the defect of identifying the heart rate in a contact way, monitors the heartbeat of the body of the driver, and identifies the heart rate of the driver, and the specific technical scheme comprises the following 6 parts. Selecting an interested area: the selection of the driver interested region is based on 68 characteristic points of the human face, and the interested region is selected. Feature extraction: on the basis of the region of interest, the green channel characteristics are extracted to be used as the real representation of the heart rate of the driver, and because the characteristic extraction in the region of interest is easily influenced by the uneven distribution of illumination, the method adopts an algorithm to eliminate the illumination error. Conversion to the frequency domain: and constructing a small segment of green channel sequence value, reducing errors, and obtaining a frequency value through FFT. Noise filtering: filtering out noise that does not comply with the laws. Heart rate extraction: and obtaining a rough heart rate value according to the relation between the green channel frequency value and the heart rate. The heart rate is stable: and according to the upper frame and the lower frame, obtaining a smooth and stable heart rate value through an algorithm. The invention has non-contact type, can not cause uncomfortable feeling to a driver, and simultaneously avoids the problem of inconvenient heart rate identification and carrying of the traditional equipment.)

1. A method for recognizing the heart rate of a driver based on image processing is characterized by comprising the following steps:

region of Interest (ROI) selection: the method for selecting the interesting regions of the drivers adopts 68 characteristic points based on the human faces, and since the forehead region of the drivers rarely participates in rich facial expressions and has little relative change, the regions of the forehead parts of the drivers, 20 and 23, are selected as the interesting regions 1, and the regions between the eyes and the nose of the human faces rarely participate in facial expressions, the regions between the characteristic points of the human faces, 3, 13, 40, 41, 46 and 47, are selected as the interesting regions 2.

Feature extraction: on the basis of the region of interest, the characteristic value of the green channel is extracted to be used as the real expression of the heart rate of the driver, and the characteristic extraction in the region of interest is easily influenced by the uneven distribution of illumination, so that the uneven distribution of illumination is eliminated by selecting a k-means clustering algorithm.

Conversion to the frequency domain: and (3) reducing errors by subtracting a mean value in a mode of constructing a small segment of green channel sequence value through the clustered green channel value, and then obtaining a converted sequence frequency value through fast Fourier change.

Noise filtering: by means of a butterworth band-pass filter, heart rate values which do not conform to the normal laws are filtered out.

Heart rate extraction: and according to the peak value detection, obtaining the peak value of the sequence frequency value, and then obtaining a rough heart rate value according to the relation between the green channel frequency value and the heart rate.

The heart rate is stable: and according to the rough heart rate value, adopting a stable heart rate value of the relationship between the upper frame and the lower frame of the heart rate, and then further smoothing and stabilizing the heart rate value through a Kalman filter and a long-time and short-time memory network to obtain a final smooth and stable heart rate value.

2. The image processing-based driver heart rate recognition method according to claim 1, characterized in that: and positioning a characteristic interested region on the basis of 68 characteristic points of the human face, and then selecting an optimal value as a real reaction value of a green channel of the interested region by adopting a mean value of the green channel obtained by a k-means clustering algorithm. And then filtering out noise by using a Butterworth band-pass filter, then solving a frequency value corresponding to the characteristic region by using FFT (fast Fourier transform), solving a roughly predicted heart rate value according to the corresponding relation between the frequency value and the heart rate, further smoothing and stabilizing the heart rate value by using a Kalman filter and the relation between the Kalman filter and the upper frame and the lower frame of the heart rate according to the roughly predicted heart rate value, and then sending the smoothed heart rate value into an LSTM network to obtain a final stable and stable heart rate value.

3. The image processing-based driver heart rate recognition method according to claim 2, characterized in that: the driver heart rate identification comprises the following steps:

for the RGB color three-channel image acquired by the camera, a signal containing a heart rate component can be obtained by selecting an interested area and then by means of signal processing, so that the selection of the interested area is established on the basis of face detection and 68 feature points of the face. The calculation of the mean value of the green channel features of the facial interesting region is easily influenced by the environment interference such as uneven illumination distribution, and the illumination distribution of a small region in the extracted facial interesting region is found to be even by the calculus principle. Therefore, the interested region is divided into a plurality of small blocks, the mean value is obtained in each small block, all the mean values are clustered by adopting a k-means clustering algorithm, and the heart rate signal closest to the true value is obtained.

4. The image processing-based driver heart rate recognition method according to claim 3, characterized in that: the feature extraction specifically comprises the following steps:

selecting Butterworth filter as band-pass filter for filtering out undesired signalA Discrete Fourier Transform (DFT) value that is too low or too high in the heart rate. Magnitude squared function | H of Butterworth filter_a(jΩ)|²Can be expressed by equation (1):

in formula (1), N is referred to as the order of the filter, and when Ω is 0, | H_a(jΩ)|＝1；Ω＝Ω_cWhen the temperature of the water is higher than the set temperature,Ω_cis a 3dB cut-off frequency. The speed of amplitude reduction is related to the order N, when N is larger, the more flat the pass band is, the narrower the transition band is, and the faster the amplitude reduction speed of the transition band and the stop band is, the smaller the error of the total frequency response and the ideal low-pass filter is.

Replacing j omega by s, and squaring the amplitude by a function | H_a(jΩ)|²Written as a function of s:

the complex variable s ═ σ + j Ω, this formula shows that the magnitude square function has 2 Ν poles, pole s_kRepresented by the formula:

in formula (3), k is 0,1,2, …, 2N-1. 2n poles with radius omega_cAre equidistantly distributed, also called butterworth circles, at intervals of pi/N rad. To form a causally stable filter, H_a(s) is composed of N poles of s left half-plane of 2 Ν poles, as shown in equation (4), and H is also_aThe (-s) is made up of the right half of the N poles.

5. The image processing-based driver heart rate recognition method according to claim 4, wherein: the coarse heart rate signal comprises the steps of:

FFT is an efficient DFT that can be divided into time-wise and frequency-wise decimation algorithms, assuming that the length of the finite-length sequence x (N) is N, and N ═ 2 is satisfied^MM is a natural number, x (N) is decomposed into two subsequences of N/2 points according to the parity term of N, as shown in formula (5), and then the DFT of x (N) is formula (6).

And solving the frequency value of DFT of the mean value of the green channel filtered by the Butterworth band-pass filter, and obtaining the DFT frequency variation value through FFT. Due to the property of FFT, the length is shortened by half after FFT conversion, so that the frequency value of the image frame number corresponding to the maximum frequency value can be obtained, and the corresponding frequency is multiplied by 60 to obtain the roughly estimated heartbeat value.

6. The image processing-based driver heart rate recognition method according to claim 5, wherein: the stabilizing the heart rate includes the steps of:

(1) the kalman filter is an algorithm for performing optimal estimation on the system state by inputting and outputting observation data through a system by using a linear system state equation, and the optimal estimation can also be regarded as a filtering process because the observation data contains noise and interference of the system. Kalman filtering can make a foundation prediction on the next step of the system in a dynamic system containing uncertain information, and a Kalman filter can still truly reflect the occurrence situation even along with interference, so that the Kalman filter is very suitable for a rough heart rate filtering noise scene, and the Kalman filter is selected to filter a heartbeat value which has too large heart rate change and is not in accordance with the rationality.

(2) Because the human heart rate has little correlation with external environment variables such as time, illumination and the like, and the human heart rate shows regular up-and-down fluctuation states around the average heart rate on the graph for a long time, 1 normalized heartbeat value m is input_i. And (4) continuously eliminating the too high and too low heart beat values according to a formula (7), wherein i +1 represents the heart beat value at the current moment, and further smoothing the heart beat value curve. The heart rate signal output by the LSTM is further stabilized, the noise is further eliminated, and the noise is smoothed and stabilized, so that better visual experience is brought to an experiencer.

m_i+1＝0.8*m_i+0.2m_i+1 (7)。

Technical Field

The invention relates to a method for recognizing the heart rate of a driver, and relates to the fields of signal processing technology, image processing technology, medical engineering, electrotherapy, human neurology, medical instrument design and the like.

Background

Estimation and detection of heart rate is important to determine a person's physiological and psychological state. The traditional heart rate measurement is that equipment such as electrode ECG closely contacts human skin, and then obtains signals such as heartbeat, blood pressure on obtaining the tester, or through wearing intelligent equipment, adopts the method of Photo Plethysmography (PPG) to measure, like intelligent bracelet, treadmill etc.. The traditional heart rate monitoring method needs to be in close contact with human body surface skin, brings extremely uncomfortable experience to an experiencer, and provides a remote photoplethysmography (RPPG) technology for a learner, wherein the RPPG is the same as a traditional PPG principle, and due to the fact that periodic changes of blood and blood capillaries in human body skin tissues can be caused by each heartbeat, the absorption and reflection of light can also form periodic changes, human eyes cannot observe the periodic changes, but can analyze video images collected by a high-definition camera, and therefore heart rate detection is achieved. Therefore, this method is also called super-sensing heart rate detection. The device has the advantages that the device is not only not required to be worn by detected people, but also can be used for long-time physiological monitoring, for example, the long-time physiological monitoring of the old at home, and has wide application value. The following disadvantages exist for the traditional ECG and PPG driver heart rate detection approach: 1) the device needs to be in close direct contact with human skin, belongs to an invasive signal acquisition mode, and brings extremely uncomfortable experience to a driver. 2) Expensive equipment needs to be worn, while the acquired signals are susceptible to driver handling and environmental disturbances, such as: when a driver drives, behaviors such as turning head, blinking, speaking and the like and electromagnetic field interference all cause interference on an ECG signal, and the interference has great influence on subsequent heart rate identification. 3) The device is not portable and the use place is limited. Because the existing ECG or PPG heart rate acquisition technology needs to be in close contact with human skin, the requirement of normal driving of a vehicle-mounted driver cannot be met, and the use occasion is limited.

Disclosure of Invention

The invention aims to realize a method for identifying the heart rate of a driver based on image processing, overcomes the defect of identifying the heart rate in a contact way, monitors the heartbeat of the body of the driver, and identifies the heart rate of the driver, and the specific technical scheme comprises the following 6 parts.

(1) Region of Interest (ROI) selection: the method for selecting the interesting regions of the drivers adopts 68 characteristic points based on the human faces, and since the forehead region of the drivers rarely participates in rich facial expressions and has little relative change, the regions of the forehead parts of the drivers, 20 and 23, are selected as the interesting regions 1, and the regions between the eyes and the nose of the human faces rarely participate in facial expressions, the regions between the characteristic points of the human faces, 3, 13, 40, 41, 46 and 47, are selected as the interesting regions 2.

(2) Feature extraction: on the basis of the region of interest, the green channel characteristics are extracted to serve as the real representation of the heart rate of the driver, and the characteristic extraction in the region of interest is easily influenced by the uneven distribution of illumination, so that the uneven distribution of illumination is eliminated by selecting a k-means clustering algorithm.

(3) Conversion to the frequency domain: and (3) reducing errors by subtracting a mean value in a mode of constructing a small segment of green channel sequence value through the clustered green channel value, and then obtaining a converted sequence frequency value through fast Fourier change.

(4) Noise filtering: by means of a butterworth band-pass filter, heart rate values which do not conform to the normal laws are filtered out.

(5) Heart rate extraction: and according to the peak value detection, obtaining the peak value of the sequence frequency value, and then obtaining a rough heart rate value according to the relation between the green channel frequency value and the heart rate.

(6) The heart rate is stable: and according to the rough heart rate value, adopting a stable heart rate value of the relationship between the upper frame and the lower frame of the heart rate, and then further smoothing and stabilizing the heart rate value through a Kalman filter and a long-time and short-time memory network to obtain a final smooth and stable heart rate value.

Compared with the traditional technology for measuring the heart rate of the driver, the invention has the advantages that: 1. the facial image of the driver is collected in the form of the vehicle-mounted camera, the heart rate change of the driver is identified in a non-contact mode, and the non-contact type heart rate recognition device is provided with the non-contact type heart rate recognition device and cannot cause uncomfortable feeling to the driver. 2. The heart rate of the driver is identified through image processing, and the problem that the heart rate identification of the traditional equipment is inconvenient to carry is avoided. 3. The realization method and the equipment are simple, the acquisition equipment can obtain image information only by one common vehicle-mounted camera, and the heart rate of the driver can be obtained by some subsequent image processing and signal processing algorithms. 4. The method has the advantages of prominent target motion change information, small data volume, saved storage space, high processing speed and high precision.

Drawings

FIG. 1 is a general flowchart of a method for recognizing heart rate of a driver based on image processing according to the present invention

FIG. 2 is a region of interest map

FIG. 3 is a graph of k-means algorithm clustering green true values

FIG. 4 is a graph of unfiltered time domain values

FIG. 5 is a graph of time domain values after noise filtering

FIG. 6 is a frequency domain plot after FFT

FIG. 7 is a graph of heart rate rough values

FIG. 8 is a heart rate graph smoothed by a Kalman filter

FIG. 9 is a diagram of an LSTM heart rate smoothing network

FIG. 10 is a graph of heart rate after smoothing of upper and lower frames

FIG. 11 is a graph of the final output smoothed stable heart rate

Detailed Description

The invention is used for providing a method for recognizing the heart rate of a driver based on image processing, and in order to make the technical scheme and the effect of the invention clearer and clearer, the following describes a specific implementation mode of the invention in detail with reference to the attached drawings.

As shown in fig. 1, the method for identifying the heart rate of the driver based on image processing comprises four stages, namely a stage of selecting and eliminating an illumination error, a stage of extracting features, a stage of roughly extracting a heart rate signal and a stage of stably stabilizing the heart rate for an area of interest. The characteristic region selection of the interesting region and the illumination error elimination stage is based on 68 characteristic points of a human face, the illumination error is eliminated according to the calculus principle, the characteristic extraction stage comprises the noise error elimination of a heart rate sequence and the noise filtration of a Butterworth band-pass filter, the rough heart rate signal stage is mainly based on the conversion from a time sequence to a frequency domain, the frequency domain value after the conversion is used for obtaining a frequency domain threshold value according to peak detection and is converted into a heart rate value according to a correlation formula, and the smooth and stable heart rate stage is used for obtaining a smooth and stable heart rate value according to the rough heart rate value by adopting a Kalman filter and a long-short time memory network.

1. Selecting the interested area and eliminating the illumination error stage: for an RGB (red, green and blue) color three-channel image acquired by a camera, an average value of three or more channels can be obtained by selecting an interested area, then a signal containing a heart rate component is obtained by means of signal processing, and then corresponding frequency is obtained by means of fast Fourier change or peak detection and the like. The ROI is selected on the basis of face detection and 68 feature points of a face, and because the forehead area of a driver rarely participates in rich facial expressions and the relative change is not large, the region of the forehead part of the driver from 20 to 23 is selected as the ROI1, and the region from the eyes to the nose of the face also rarely participates in facial expressions, the region between the face feature points 3, 13, 40, 41, 46 and 47 is selected as the ROI2, and the region of interest is selected as shown in fig. 2. The calculation of the mean value of the green channel features of the facial interesting region is easily influenced by the environment interference such as uneven illumination distribution, and the illumination distribution of a small region in the extracted facial interesting region is found to be even by the calculus principle. Therefore, the region of interest is divided into a plurality of small blocks, an average value is obtained in each small block, all the average values are clustered by adopting a k-means clustering algorithm, the heart rate signal closest to the true value is obtained, the effect graph after clustering is shown in fig. 3, and the square block is the value after clustering.

2. A characteristic extraction stage: the Butterworth filter is a common electronic filter, also called maximum flat filter, and is mainly characterized in that a frequency curve in a pass band is flat to the maximum, no ripple exists, a stop band is gradually reduced to 0, and a good filtering effect is achieved. Therefore, the patent selects a Butterworth filter as a band-pass filter to filter out Discrete Fourier Transform (DFT) values with too low or too high heart rate, which are not ideal. Magnitude squared function | H of Butterworth filter_a(jΩ)|²Can be expressed by equation (1):

in formula (1), N is referred to as the order of the filter, and when Ω is 0, | H_a(jΩ)|＝1；Ω＝Ω_cWhen the temperature of the water is higher than the set temperature,Ω_cis a 3dB cut-off frequency. The speed of amplitude reduction is related to the order N, when N is larger, the more flat the pass band is, the narrower the transition band is, and the faster the amplitude reduction speed of the transition band and the stop band is, the smaller the error of the total frequency response and the ideal low-pass filter is.

Replacing j omega by s, and squaring the amplitude by a function | H_a(jΩ)|²Written as a function of s:

the complex variable s ═ σ + j Ω, this formula shows that the magnitude square function has 2 Ν poles, pole s_kRepresented by the formula:

in formula (3), k is 0,1,2, …, 2N-1. 2n poles with radius omega_cAre equidistantly distributed, also called butterworth circles, at intervals of pi/N rad. To form a causally stable filter, H_a(s) is composed of N poles of s left half-plane of 2 Ν poles, as shown in equation (4), and H is also_aThe (-s) is made up of the right half of the N poles.

This patent adopts the block to solve the mean value and k-means cluster to solve the mean value to the green channel value of 400 frames of images, lets the green channel value of each frame therein subtract the mean value, obtains unfiltered time domain value, as shown in fig. 4, then lets this 400 frames of green channel mean value frequency domain pass through butterworth band-pass filter, filters out the too high or too low noise value that does not accord with the rationality, obtains the time domain value after filtering, as shown in fig. 5.

3. Coarse heart rate signal phase: FFT is an efficient DFT that can be divided into time-wise and frequency-wise decimation algorithms, assuming that the length of the finite-length sequence x (N) is N, and N ═ 2 is satisfied^MM is a natural number, x (N) is decomposed into two subsequences of N/2 points according to the parity term of N, as shown in formula (5), and then the DFT of x (N) is formula (6).

Because of the fact that

Therefore, it is not only easy to use

Wherein X₁(k) And X₂(k) Are respectively x₁(r) and x₂N/2 point DFT of (r), i.e.

Due to X₁(k) And X₂(k) Are all periodic by N/2, andthus X (k) may in turn be represented by

The method calculates the DFT frequency value of the average value of 400 frames of green channels filtered by the Butterworth band-pass filter, and obtains the DFT frequency change value of the 400 frames of images shown in figure 6 through FFT. Due to the property of FFT, the length of the 400 frames of images is shortened by half after the FFT conversion, the 400 frames of images become 200 frames of images, and the frequency peak value is concentrated around 63, so that the frequency value of the image frame number corresponding to the maximum frequency value can be obtained, and the corresponding frequency is multiplied by 60 to obtain the roughly estimated heartbeat value.

4. A stable heart rate stage:

(1) the kalman filter is an algorithm for performing optimal estimation on the system state by inputting and outputting observation data through a system by using a linear system state equation, and the optimal estimation can also be regarded as a filtering process because the observation data contains noise and interference of the system. Kalman filtering can make a prediction based on the next step of the system in a dynamic system containing uncertain information, and the Kalman filter can still truly reflect the occurrence condition even along with interference, so that the Kalman filter is selected to filter the heartbeat value with the heart rate changing too much and not conforming to the conventional law. Similarly, 400 frames of images are taken as a sequence, and a heart rate value which is not filtered by the Kalman filter is drawn, so that the heartbeat value which is not filtered by the Kalman filter is found to have large amplitude variation and be very unstable. Therefore, the heartbeat value obtained roughly is processed by a Kalman filter to obtain a value shown in FIG. 8, and the fact that the heartbeat value is stable and changes regularly after passing through the Kalman filter can be found.

(2) The overall LSTM network for heart rate estimation is defined to be a 3-layer structure, and because the correlation between the heart rate of a person and external environment variables such as time, illumination and the like is not large, and the regular up-and-down fluctuation state around the average heart rate appears on a graph for a long time, the network selected by the user adopts an input value, and inputs 1 normalized heartbeat value m_iThe overall network flow chart is shown in fig. 9. And (3) continuously eliminating the over-high and over-low heart rate values according to a formula (13), wherein i +1 represents the heart rate value at the current moment, further smoothing a heart rate value curve, continuously taking 400 frames of images, and drawing to obtain a heart rate smoothing curve as shown in the figure 10. The smooth signal can still have great noise interference, and the up-down mutation state with a certain amplitude is presented on the coordinate axis, so that poor visual experience is brought to an experiencer. To this end, the patent sends the smoothed signal to the global LSTM network, and also plots the output 400 frames of predicted heart beat values to the waveform of fig. 11, which is found to be comparable to that of the signal obtained from the global LSTM networkIn the smoothed Kalman filter signal, the heart rate signal output by the LSTM is further stabilized, the noise is further eliminated, and the smoothed and stabilized heart rate signal brings better visual experience to the experiencer.

m_i+1＝0.8*m_i+0.2m_i+1 (13)。