阅读说明：本技术 脉搏数检测装置以及脉搏数检测程序 (Pulse rate detection device and pulse rate detection program ) 是由 迈克尔·琼斯 于 2020-03-27 设计创作，主要内容包括：本发明的目的在于进行可靠性较高的脉搏数的输出。脉搏数检测装置(1)考虑干扰因素并且将脉搏信号的SN比作为基础来评价检测出的脉搏数的可靠度。脉搏数检测装置(1)设置用于显示脉搏数的脉搏数显示基准1和脉搏数显示基准2,在SN比为前者以上的情况下,显示脉搏数,在SN比为后者以上且小于前者的情况下,将与最近的过去的脉搏数之间的差比规定基准小作为条件来进行显示。在SN比小于脉搏数显示基准2的情况下,设为可靠度较低而不显示。另外,脉搏数检测装置(1)考虑基于对象者(11)的脸部的活动的活动干扰和基于照射对象者(11)的脸部的光的变化的光干扰,在活动干扰或光干扰的频域中的峰值频率与脉搏数接近规定基准以上的情况下,设为检测出的脉搏数的可靠度由于这些干扰而较低,而不显示脉搏数。(The purpose of the present invention is to output a pulse rate with high reliability. The pulse rate detection device (1) evaluates the reliability of the detected pulse rate on the basis of the SN ratio of the pulse signal while taking into account the disturbance factor. A pulse rate detection device (1) is provided with a pulse rate display reference (1) and a pulse rate display reference (2) for displaying the pulse rate, and displays the pulse rate on the condition that the SN ratio is greater than or equal to the former, and the difference between the pulse rate and the latest past pulse rate is less than a predetermined reference when the SN ratio is greater than or equal to the latter and less than the former. When the SN ratio is smaller than the pulse rate display reference 2, the display is not performed with low reliability. The pulse rate detection device (1) considers a motion disturbance caused by the motion of the face of the subject person (11) and an optical disturbance caused by the change of light with which the face of the subject person (11) is irradiated, and when the peak frequency in the frequency domain of the motion disturbance or the optical disturbance and the pulse rate approach a predetermined reference or more, the reliability of the detected pulse rate is set to be low due to these disturbances and the pulse rate is not displayed.)

1. A pulse rate detection device is characterized by comprising:

a moving image acquisition unit that acquires a moving image obtained by imaging a body surface of a subject person;

a pulse rate acquisition unit that acquires a pulse rate of the subject person based on a change in the pixel value of the body surface in the acquired dynamic image;

a reliability acquisition unit that acquires reliability of the acquired pulse rate; and

and an output unit configured to output the acquired pulse rate when the acquired reliability is equal to or higher than a predetermined reliability reference and when the acquired reliability is lower than the reliability reference and a change in the acquired pulse rate is within a predetermined appropriate range.

2. The pulse rate detection device according to claim 1,

the change in the pulse rate is a difference between the acquired pulse rate and a latest pulse rate, and when the difference is equal to or smaller than a predetermined value, the acquired pulse rate is assumed to be within an appropriate range, and the output unit outputs the acquired pulse rate.

3. The pulse rate detection device according to claim 1 or 2,

the pulse rate acquisition unit acquires the pulse rate from a pulse rate peak in the frequency domain of the acquired moving image,

the reliability acquisition unit acquires the reliability based on the acquired SN ratio of the pulse rate.

4. The pulse rate detection device according to claim 3,

the pulse rate detection device includes a reduction unit that reduces the reliability when an interference peak value in a frequency domain of an interference factor that reduces the accuracy of the acquired pulse rate and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

5. The pulse rate detection device according to claim 4,

the disturbance factor is activity of the body surface, and the reduction unit reduces the reliability when an activity peak in a frequency domain of the activity of the body surface in the moving image and the acquired pulse rate peak approach a predetermined amount or more.

6. The pulse rate detection device according to claim 4 or 5,

the disturbance factor is a fluctuation of light irradiated on the body surface, and the reduction unit reduces the reliability when a peak value of light in a frequency domain of the fluctuation of light in the moving image and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

7. A pulse rate detection program that realizes the following functions by a computer:

a dynamic image acquisition function of acquiring a dynamic image obtained by photographing a body surface of a subject person;

a pulse rate acquisition function of acquiring a pulse rate of the subject person based on a change in the pixel value of the body surface in the acquired dynamic image;

a reliability acquisition function of acquiring reliability of the acquired pulse rate; and

and an output function that outputs the acquired pulse rate when the acquired reliability is equal to or higher than a predetermined reliability criterion and when the acquired reliability is lower than the reliability criterion and a change in the acquired pulse rate is within a predetermined appropriate range.

Technical Field

The present invention relates to a pulse rate detection device and a pulse rate detection program, and relates to detecting a pulse rate by image processing, for example.

Background

In order to grasp the health and physiological state of the subject, it is important to detect the pulse rate. Generally, the pulse rate is detected by attaching an instrument to a subject, but there is a high demand for easier detection, and a technique for detecting the pulse rate of the subject in a non-contact manner has been actively studied.

This enables, for example, monitoring the pulse rate of the driver of the vehicle to further improve traffic safety.

The technique of non-patent document 1 is known as a technique for detecting the pulse rate of a subject person in a non-contact manner. In this technique, the arm of the subject person is photographed by a camera, and the pulse rate is detected by acquiring changes in color, brightness, and the like of the skin from the camera image. Since the brightness and color of the body surface change according to the blood flow, the pulse rate can be detected by image processing the video.

However, when the pulse rate is detected in an environment where there is a disturbance such as a movement of a subject or a change in ambient light, there is a problem in that the reliability (reliability) of the detected pulse rate is unknown.

For example, when the pulse rate is detected for a certain period, if the reliable detection is 86% and the remaining detection is unreliable due to interference, it is not known which part is reliable.

Even when the reliability of the detected value can be evaluated, there is a problem that if the evaluation is strictly performed, a period during which the pulse rate can be detected becomes short.

Non-patent document 1: "Non-contact monitoring technologies-Principles and applications," D.Teichmann, C.Bruser, B.Eilebrecht, A.Abbas, N.Blanik, and S.Leonhardt, Con.Proc.IEEE Eng., Med.biol.Soc.34th Ann.Int.Conf., San Diego, CA, USA,2012, pp.1302-1305.

Disclosure of Invention

The purpose of the present invention is to output a pulse rate with high reliability.

(1) In order to achieve the above object, the present invention according to claim 1 provides a pulse rate detection device including: a moving image acquisition unit that acquires a moving image obtained by imaging a body surface of a subject person; a pulse rate acquisition unit that acquires a pulse rate of the subject person based on a change in pixel values of the body surface in the acquired moving image; a reliability acquisition unit configured to acquire reliability of the acquired pulse rate; and an output unit configured to output the acquired pulse rate when the acquired reliability is equal to or higher than a predetermined reliability reference and when the acquired reliability is lower than the reliability reference and a change in the acquired pulse rate is within a predetermined appropriate range.

(2) The invention described in claim 2 provides the pulse rate detection device described in claim 1, wherein the change in the pulse rate is a difference between the acquired pulse rate and a latest pulse rate, and when the difference is equal to or less than a predetermined value, the acquired pulse rate is assumed to be within an appropriate range, and the output means outputs the acquired pulse rate.

(3) The invention described in claim 3 provides the pulse rate detection device described in claim 1 or 2, wherein the pulse rate acquisition means acquires the pulse rate from a pulse rate peak in a frequency domain of the acquired moving image, and the reliability acquisition means acquires the reliability based on an SN ratio of the acquired pulse rate.

(4) The invention described in claim 4 provides the pulse rate detection device described in claim 3, wherein the pulse rate detection device includes a reduction unit that reduces the reliability when an interference peak value in a frequency domain of an interference factor that reduces the accuracy of the acquired pulse rate and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

(5) The invention described in claim 5 provides the pulse rate detection device described in claim 4, wherein the disturbance factor is a movement of the body surface, and the reduction means reduces the reliability when a movement peak value in a frequency domain of the movement of the body surface in the moving image and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

(6) The invention described in claim 6 provides the pulse rate detection device described in claim 4 or 5, wherein the noise factor is a fluctuation of light irradiated on the body surface, and the reduction means reduces the reliability when a peak value of light in a frequency domain of the fluctuation of light in the moving image and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

(7) The invention described in claim 7 provides a pulse rate detection program that realizes, by a computer, the following functions: a dynamic image acquisition function of acquiring a dynamic image obtained by photographing a body surface of a subject person; a pulse rate acquisition function of acquiring a pulse rate of the subject person based on a change in pixel values of the body surface in the acquired moving image; a reliability acquisition function of acquiring reliability of the acquired pulse rate; and an output function that outputs the acquired pulse rate when the acquired reliability is equal to or higher than a predetermined reliability reference and when the acquired reliability is lower than the reliability reference and a change in the acquired pulse rate is within a predetermined appropriate range.

According to the present invention, the pulse rate is output based on the reliability, and thus the pulse rate with high reliability can be output.

Drawings

Fig. 1 is a diagram for explaining the configuration of the pulse rate detection device 1.

Fig. 2 is a flowchart for explaining the pulse rate detection process.

Fig. 3 is a diagram for explaining various settings performed on an image.

Fig. 4 is a flowchart for explaining the pulse rate detection process.

Fig. 5 is a flowchart for explaining the reference setting process.

Fig. 6 is a flowchart for explaining the calculation process of the average Q of the detection regions.

Fig. 7 is a flowchart for explaining the calculation process of the average Q in the region other than the measurement region.

Fig. 8 is a flowchart for explaining the calculation process of the peak frequency and the SN ratio of orig _ Q.

Fig. 9 is a flowchart for explaining the calculation processing of the peak frequency and the SN ratio of disp _ Q.

Fig. 10 is a flowchart for explaining the active interference processing.

Fig. 11 is a flowchart for explaining the calculation process of the peak frequency and the SN ratio of conj _ Q.

Fig. 12 is a flowchart for explaining the optical disturbance processing.

Fig. 13 is a flowchart for explaining the anchor process.

Fig. 14 is a flowchart for explaining the procedure of the attention display processing.

Detailed Description

(1) Brief description of the embodiments

The pulse rate detection device 1 (fig. 1) evaluates the reliability of the detected pulse rate based on the SN ratio of the pulse signal, and also evaluates the pulse rate in consideration of disturbance factors based on activity or light.

The pulse rate detection device 1 is provided with a pulse rate display reference 1 for displaying the pulse rate and a pulse rate display reference 2 lower than the display reference, and displays the pulse rate on condition that the SN ratio is equal to or higher than the former, and the difference from the latest past pulse rate is smaller than a predetermined reference on condition that the SN ratio is equal to or higher than the latter and smaller than the former.

When the SN ratio is smaller than the pulse rate display reference 2, the display is not performed with low reliability.

In the pulse rate detection device 1, the disturbance factor disturbing the detection of the pulse rate is considered to be a motion disturbance based on the motion of the face of the subject person 11 and a light disturbance based on the change of light irradiating the face of the subject person 11, and when the peak frequency in the frequency domain of the motion disturbance or the light disturbance and the pulse rate are close to a predetermined reference or more, the reliability of the detected pulse rate is low due to these disturbances and the pulse rate is not displayed.

(2) Detailed description of the embodiments

Fig. 1 is a diagram for explaining the configuration of a pulse rate detection device 1 according to the present embodiment.

The pulse rate detection device 1 is mounted on, for example, a vehicle, and monitors the pulse rate of a passenger (a subject person such as a driver or a passenger in a passenger seat) and the physiological state such as the physical condition and the tension of the driver.

In addition, the pulse rate detection device can be used for detecting and monitoring the pulse rate of a patient or a person suffering from a disaster in a medical field, a disaster field, or the like, or can be used for being installed in a home or a commercial facility to allow a user to easily detect the pulse rate.

As shown in fig. 1 (a), the pulse rate detection device 1 includes a CPU (Central Processing Unit) 2, a ROM (Read Only Memory) 3, a RAM (Random Access Memory) 4, a display Unit 5, an input Unit 6, an output Unit 7, a camera 8, a storage Unit 10, and the like, and detects (or estimates or measures) the pulse rate of a subject person 11.

The CPU2 is a central processing unit that performs various information processing and control based on programs stored in the storage unit 10, the ROM3, and the like.

In the present embodiment, the pulse rate of the subject person 11 is detected by performing image processing on the moving image captured by the camera 8.

This detection can be performed in the visible light region or the infrared region, but in the present embodiment, visible light is used as an example.

Further, illumination for irradiating the subject person 11 with visible light or infrared light may be provided.

The ROM3 is a read-only memory and stores basic programs, parameters, and the like for operating the pulse rate detection device 1.

The RAM4 is a readable and writable memory, and provides a work memory for the CPU2 to operate.

In the present embodiment, the sub-CPU 2 detects a pulse wave from the skin portion of an image by developing and storing an image (a still image of 1 frame) constituting a moving image or storing the calculation result.

The skin portion may be any portion where the body surface such as the face, hands and feet is exposed, but in the present embodiment, the pulse rate is detected from the surface of the face (face) as an example.

The display unit 5 is configured using a display device such as a liquid crystal screen, and displays an operation screen of the pulse rate detection device 1, display of the pulse rate, and information necessary for operation of the pulse rate detection device 1 such as attention concerning the reliability of the detected pulse rate.

The input unit 6 is configured using an input device such as a touch panel provided so as to overlap with the display device, and accepts input of various information depending on presence or absence of a touch on the screen display.

The output unit 7 is an interface for outputting various information to an external device, and is capable of outputting a detected pulse rate or outputting an alarm when the pulse rate changes, for example.

The output unit 7 can output the output to another control device such as a control device for controlling the vehicle. The control device that receives the output of the pulse rate from the output unit 7 can determine, for example, drowsiness, a tension state, and the like of the driver, and perform control for the driver, for example, control for vibrating a steering wheel or a seat to wake drowsiness, output of a warning sound or a message, and the like.

Further, as the control for the vehicle, at least one of the inter-vehicle distance control, the vehicle speed control, and the brake control may be performed based on the tension state of the driver determined based on the pulse rate.

For example, when it is determined that the driver is in a high stress state exceeding a predetermined value, the control device controls the inter-vehicle distance to be larger than a reference value, controls the vehicle speed to be equal to or smaller than a predetermined vehicle speed, and performs deceleration processing by an automatic braking operation if the vehicle speed is equal to or larger than the predetermined vehicle speed.

The camera 8 is a moving image capturing camera, is configured using an optical system including lenses and an image element for converting an image formed by the lenses into an electric signal, and is provided so that the vicinity of the face of the subject person 11 becomes a capturing screen.

When a moving image is captured in the visible light region and the pulse rate is detected in the infrared region, an infrared camera equipped with an infrared image sensor is used as the camera 8.

The camera 8 images the subject person 11 at a predetermined frame rate, and outputs a moving image composed of these continuous images (still images).

The image is composed of an arrangement of pixels (pixels) which are the smallest units constituting the image.

The storage unit 10 is configured using a storage medium such as a hard disk or an EEPROM (Electrically Erasable Programmable Read-Only Memory), and stores a pulse rate detection program for detecting a pulse wave by the CPU2, other programs, and data.

The data of the pulse rate detected by the CPU2 in accordance with the pulse rate detection program is temporarily stored in the RAM4, and is output to the outside as needed, or is stored in the storage unit 10.

The pulse rate detection program is a program for causing the CPU2 to perform pulse wave detection processing.

The CPU2 executes a pulse rate detection program to perform information processing such as setting of a measurement area in an image, detection of a pulse rate, evaluation of reliability of a detected pulse rate, and output control of a pulse rate according to the reliability.

Fig. 1 (b) shows an example of detecting the pulse rate.

When the signal Q including the pulse wave component is transformed by FFT (Fast Fourier Transform), a pulse rate peak appears in the vicinity of 60bpm (beats per minute). This is caused by the change in the skin color of the subject person 11 due to the pulse, which becomes a detection value of the pulse rate of the subject person 11.

Even if the brightness of infrared rays is used, the same pulse rate peak can be obtained.

Fig. 2 is a flowchart for explaining the pulse rate detection process performed by the pulse rate detection device 1.

The following processing is performed by the CPU2 of the pulse rate detection device 1 based on the pulse rate detection program stored in the storage unit 10.

First, the pulse rate detection device 1 performs a reference setting process for setting various references for evaluating the reliability of the detected pulse rate, the degree of interference, and the like (step 5).

Next, the pulse rate detection device 1 acquires a moving image (frame image) captured by the camera 8 by storing the image in the RAM4 (step 10).

In this way, the pulse rate detection device 1 includes a moving image acquisition unit that acquires a moving image obtained by imaging the surface of the body of the subject person.

Then, the pulse rate detection device 1 sets a measurement area in the acquired image of the moving image (step 15), and stores the measurement area in the RAM 4.

As shown in fig. 3 (a), the pulse rate detection device 1 detects the face of the subject person 11 in the image 20 by a known technique, and sets a measurement region 22 of the face.

Returning to fig. 2, when the pulse rate detection device 1 sets the measurement area 22, it determines whether the size of the measurement area 22 is sufficient (step 20).

If the measurement area 22 is not large enough (step 20; no), the pulse rate detection device 1 returns to step 10 to acquire a (next) image and set the measurement area 22. Alternatively, the procedure may return to step 15 to reset the measurement region 22 in the same image.

On the other hand, when the measurement region 22 has a sufficient size (step 20; yes), the pulse rate detection device 1 performs a process of calculating the average Q (orig _ Q) of the measurement region 22 (step 25).

This is to average the Q value in the measurement region 22, i.e., the region indicated by the diagonal lines in fig. 3 (b), and then detect the pulse rate from the temporal change in the Q value.

Orig _ Q is an abbreviation for the average Q of the measurement region 22, and hereinafter, the abbreviation is described in parentheses as necessary.

Returning to fig. 2, the pulse rate detection device 1 then calculates the center position (disp _ Q) of the measurement region 22 (step 30).

As shown in fig. 3 (c), this is a calculation of the position of the center 24 of the measurement region 22, and then is used in evaluating the activity disturbance based on the activity of the face.

More specifically, for example, when the face of the subject person 11 moves while the vehicle is driving, the movement becomes a disturbance factor (movement disturbance) that degrades the accuracy of detecting the pulse rate. Thus, the activity of the hub 24 is utilized to detect the activity disturbance.

As will be described later, when the peak frequency of the motion at the center 24 in the frequency domain is closer to the pulse rate than a predetermined amount, the pulse rate detection device 1 determines that the reliability of the pulse rate is low due to the presence of motion disturbance, and decreases the reliability of the pulse rate.

As described above, the pulse rate detection device 1 includes a reduction unit that reduces the reliability of the pulse rate when the peak value of the pulse rate and the peak value of the pulse rate in the frequency domain of the disturbance factor that reduces the accuracy of the pulse rate are close to each other by a predetermined amount or more.

The disturbing factor is the motion of the body surface, and the reducing means reduces the reliability when the peak of the motion and the peak of the pulse rate in the frequency domain of the motion of the body surface in the moving image are close to each other by a predetermined amount or more.

Returning to fig. 2, the pulse rate detector 1 then performs a process of calculating an average Q (conj _ Q) in a region other than the measurement region 22 (step 35).

This is used for evaluating the optical interference by averaging the Q values of the background region of the measurement region 22, which is indicated by the diagonal lines in fig. 3 (d), except for the measurement region 22.

More specifically, for example, while the vehicle is driving, the state of light that irradiates the face of the subject person 11 changes with time, but this change becomes a disturbing factor (optical disturbance) that degrades the accuracy of pulse rate detection.

The change in the state of the light with which the face is irradiated needs to be known when the change due to the pulse wave is not included, but since the change appears in the background of the face, that is, in the region other than the measurement region 22, the change is detected by the conj _ Q.

As will be described later, when the peak frequency of the conj _ Q in the frequency domain is closer to the pulse rate than the predetermined amount, the pulse rate detection device 1 determines that the reliability of the pulse rate is low due to the presence of the optical disturbance, and decreases the reliability of the pulse rate.

As described above, the disturbing factor is a fluctuation of light irradiated to the body surface, and the reducing means reduces the reliability when the peak value of light and the peak value of pulse rate in the frequency domain of the fluctuation of light in the moving image are close to a predetermined amount or more.

Returning to fig. 2, when the above-described processing is performed, the pulse rate detection device 1 determines whether or not the time elapsed after the start of the image acquisition has elapsed during the FFT window period (step 40).

If the window period has not elapsed (step 40; no), the pulse rate detection device 1 returns to step 10 and performs the same process on the next image.

In the present embodiment, the window period is set to 10 seconds, and the above-described processing is performed on images recorded in time series in a moving image of 10 seconds.

Since 10-second images are prepared in steps 10 to 40 in this manner, they can be converted into the frequency domain by integrating them in the time direction by the FFT algorithm.

More specifically, orig _ Q, disp _ Q, conj _ Q of 10 seconds can be transformed by FFT to obtain frequency components of pulse rate, motion disturbance, and optical disturbance, respectively.

Fig. 4 is a continuation of fig. 2.

When the window period has elapsed (step 40; yes), the pulse rate detector 1 calculates the peak frequency (HR) of orig _ Q and the SN ratio (step 50).

HR represents the pulse rate of the subject person 11, and the SN ratio of the pulse rate signal based on FFT is a basis for evaluating the reliability of the detected pulse rate.

Next, the pulse rate detection device 1 performs a process of calculating the peak frequency and the SN ratio of disp _ Q (step 55). This is the process of detecting active interference.

Next, the pulse rate detection device 1 determines whether the SN ratio of disp _ Q is larger than MR _ SNR _ C and whether the peak frequency of disp _ Q (i.e., the peak frequency of the active disturbance) is close to HR (step 60).

Here, MR _ SNR _ C is a reference value of the SN ratio of the active interference, and is set by the reference setting processing of step 5 as will be described later.

When the SN ratio of disp _ Q is larger than MR _ SNR _ C and the peak frequency of disp _ Q is close to HR (step 60; yes), the pulse rate detection device 1 determines that there is active disturbance and performs active disturbance processing (step 65).

On the other hand, if the condition of step 60 is not satisfied (step 60; no), or after step 65, the pulse rate detector 1 performs the process of calculating the peak frequency and the SN ratio of conj _ Q (step 70). This is the process of detecting optical interference.

Next, the pulse rate detection device 1 determines whether the SN ratio of conj _ Q is greater than LR _ SNR _ C and whether the peak frequency of conj _ Q (i.e., the peak frequency of optical interference) is close to HR (step 75).

Here, LR _ SNR _ C is a reference value of the SN ratio of the optical interference, and is set by the reference setting processing of step 5 as will be described later.

When the SN ratio of conj _ Q is larger than LR _ SNR _ C and the peak frequency of conj _ Q is close to HR (step 75; yes), the pulse rate detection device 1 determines that there is optical interference and performs optical interference processing (step 80).

On the other hand, if the condition of step 75 is not satisfied (step 75; no), or after step 80, the pulse rate detection device 1 performs a pulse rate display process based on the reliability by an Anchor Processing (step 85).

The above-described processes will be described below with reference to flowcharts.

Fig. 5 is a flowchart for explaining the reference setting process (step 5 in fig. 2).

First, the pulse rate detection device 1 stores the pulse rate display reference 1(HR _ SNR _ C1) in the RAM4 and sets it (step 105).

This is the first criterion for evaluating the reliability, and if the reliability of the pulse rate is equal to or higher than HR _ SNR _ C1, the pulse rate detection device 1 outputs the pulse rate with a high reliability.

In this way, the pulse rate detection device 1 includes: a reliability acquisition unit that acquires the reliability of the pulse rate; and an output unit that outputs the pulse rate when the reliability is equal to or higher than a predetermined reliability reference.

Next, the pulse rate detection device 1 stores the pulse rate display reference 2(HR _ SNR _ C2) in the RAM4 and sets it (step 110).

This is a second criterion for evaluating reliability, and is a value smaller than HR _ SNR _ C1.

When the reliability of the pulse rate does not satisfy HR _ SNR _ C1 but satisfies HR _ SNR _ C2, the pulse rate detection device 1 displays the pulse rate on the condition that the detected pulse rate is close to the latest past pulse rate.

This is for making use of the fact that the pulse rate does not change rapidly to remedy a slightly poor reliability, and thereby the pulse rate detection device 1 can increase the effective detection period of the pulse rate.

As described above, the output unit included in the pulse rate detection device 1 outputs the acquired pulse rate when the reliability of the pulse rate is equal to or higher than the predetermined reliability reference (pulse rate display reference 1) and when the reliability of the pulse rate is lower than the reliability reference and the change in the pulse rate is within the predetermined appropriate range.

Here, the change in the pulse rate is a difference between the pulse rate detected this time and the latest pulse rate, and when the difference is equal to or less than a predetermined value (equal to or less than an anchor reference described later), it is assumed that the pulse rate is within an appropriate range and the output means outputs the pulse rate.

Next, the pulse rate detection device 1 stores the pulse rate/disturbance error reference (δ HR) in the RAM4 to set it (step 115).

In the present embodiment, δ HR is set to 5bpm, and the presence or absence of interference is determined in the range of ± 5bpm of HR.

For example, if the peak value of the activity disturbance or the light disturbance is within the range of ± 5bpm of HR, the pulse rate detection apparatus 1 determines that these disturbances are present on the condition that the condition (steps 60, 75) relating to the SN ratios of these disturbance factors themselves is satisfied.

Next, the pulse rate detection device 1 stores the motion disturbance reference (MR _ SNR _ C) in the RAM4 to set it (step 120), and also stores the light disturbance reference (LR _ SNR _ C) in the RAM4 to set it (step 125).

This is the basis for the SN ratio of the interference factor itself used in steps 60, 75.

Next, the pulse rate detection device 1 stores the anchor reference (HR _ ANC _ C) in the RAM4 to set it (step 130), and also stores an appropriate initial value of the anchor pulse rate (HR _ ANC) in the RAM4 to initialize it (step 135).

The anchor pulse rate is a variable to which the latest past pulse rate is added in advance, and the anchor criterion is a criterion for determining the closeness between the detected pulse rate and the latest pulse rate. That is, when the difference between the latest pulse rate detected this time and the latest pulse rate in the past is equal to or less than the anchor criterion, the pulse rate detection device 1 determines that the latest pulse rate is within the appropriate range.

Fig. 6 is a flowchart for explaining the process of calculating the average Q (orig _ Q) of the measurement region 22 (step 25 in fig. 2).

First, the pulse rate detector 1 converts the color space of the image 20 stored in the RAM4 from RGB to YIQ, and stores the image 20 based on the converted Q value in the RAM4 (step 150). In the subsequent processing, the image 20 based on the Q value is used. The image 20 of the Q value is created in this way because the Q value is suitable for detecting the pulse rate.

Next, the pulse rate detection device 1 adjusts (calibrates) the Q value of each pixel so as to cope with the influence of the characteristics of the camera 8 (step 155). This is to correct the performance variation of the pixels constituting the frame image by scaling the value of each pixel.

Next, the pulse rate detector 1 calculates an average value (orig _ Q) by averaging the Q values of the pixels in the measurement region 22, and stores the calculated average value in the RAM4 (step 160).

Then, the pulse rate detection device 1 returns to the main routine.

Fig. 7 is a flowchart for explaining the calculation process (step 35 in fig. 2) of the average Q (conj _ Q) in the region other than the measurement region 22.

First, the pulse rate detection device 1 specifies a portion other than the measurement region 22 in the image 20 stored in the RAM4, excluding the measurement region 22 (step 180).

Then, the pulse rate detection device 1 calculates an average value (conj _ Q) by averaging the Q values of the pixels in the portion other than the specified measurement region 22, and stores the average value in the RAM4 (step 185).

Then, the pulse rate detection device 1 returns to the main routine.

Fig. 8 is a flowchart for explaining the process of calculating the peak frequency (HR) and the SN ratio of orig _ Q (step 50 in fig. 4).

First, the pulse rate detector 1 reads out orig _ Q of 10 seconds stored in the RAM4, performs fourier transform using FFT, generates FFT _ HR, and stores it in the RAM4 (step 205).

FFT _ HR is data obtained by converting orig _ Q from a time domain to a frequency domain (as shown in fig. 1 (b), for example), and a maximum peak value (pulse rate signal) based on the pulse rate appears in the vicinity of 60 bpm.

Therefore, the pulse rate detection apparatus 1 determines the frequency (HR) of the maximum peak value of the FFT _ HR and stores it in the RAM4 (step 210). This corresponds to the pulse rate of the subject person 11.

As described above, the pulse rate detection device 1 includes a pulse rate acquisition unit that acquires the pulse rate of the subject based on a change in the pixel value (for example, Q value or infrared brightness) of the body surface in the moving image, and that acquires the pulse rate from the pulse rate peak in the frequency domain of the moving image.

Next, the pulse rate detection apparatus 1 generates a mask (HR _ M1) of the maximum peak frequency ± 5bpm and stores it in the RAM4 (step 215).

HR _ M1 is data with 1 at HR + -5 bpm and 0 at the other ranges. The HR _ M1 is used in a section for extracting a pulse rate signal from the FFT waveform when calculating the SN ratio later, together with HR _ M2 described later.

Next, the pulse rate detection apparatus 1 generates a mask (HR _ M2) of the harmonic of the maximum peak value ± 5bpm and stores it in the RAM4 (step 220).

This is because the second harmonic of the pulse rate appears also in the frequency domain of the FFT performed by the pulse rate detection device 1, and therefore the mask thereof is also created.

Next, the pulse rate detection device 1 generates one mask (HR _ M) by adding HR _ M1 and HR _ M2 stored in the RAM4 and stores it in the RAM4 (step 225).

HR _ M is a mask with 1 at a part of + -5 bpm centered on HR and + -5 bpm centered on the second harmonic of HR and 0 at the other parts.

Next, the pulse rate detection device 1 creates an interference mask for HR _ M (HRN _ M is 1-HR _ M) and stores it in the RAM4 (step 230).

HRN _ M is obtained by inverting 1 and 0 of HR _ M, and is 0 at a part of + -5 bpm centered on HR and + -5 bpm centered on the second harmonic of HR, and is 1 at the other part.

Next, the pulse rate detection device 1 normalizes the FFT _ HR stored in the RAM4, and stores the normalized data (FFT _ HRn) in the RAM4 (step 235).

The pulse rate detection device 1 calculates the normalization by (FFT _ HRn ═ maximum value of FFT _ HR/FFT _ HR).

Next, the pulse rate detector 1 substitutes the FFT _ HRn stored in the RAM4 into the expression "num" shown in fig. 8 to calculate num, and stores the result in the RAM4 (step 240).

The pulse rate detection device 1 analyzes moving image data accumulated for 10 seconds by FFT, but repeats this processing for the latest data every 1 second. Therefore, the pulse rate detection device 1 generates the latest values of FFT _ HRn, HR _ M, and HRn _ M in time series every 1 second.

Therefore, in the calculation of num and then den, i pieces of data are added in order to make the deviation of the data uniform.

Further, the expression num is an inner product of a vector having a square of FFT _ hrn (i) as a component and HR _ m (i) which is a vector having 01 of the mask as a component, and a value proportional to the power of the signal of a portion of ± 5bpm centered on HR and a portion of ± 5bpm centered on the second harmonic of HR is obtained. That is, the pulse rate detection device 1 extracts a portion corresponding to the pulse rate signal from the FFT _ hrn (i), and calculates a value proportional to the power thereof (for normalization).

Next, the pulse rate detector 1 calculates den by substituting FFT _ HRn stored in RAM4 into the expression of den shown in fig. 8 in the same manner, and stores the result in RAM4 (step 245).

HRN _ m (i) is obtained by inverting HR _ m (i), and thus den obtains a value proportional to the power of the signal of the portion other than the portion of ± 5bpm centered on HR and the portion of ± 5bpm centered on the second harmonic of HR. That is, the pulse rate detector 1 extracts a portion not corresponding to the pulse rate signal from the FFT _ hrn (i), and calculates a value proportional to the power thereof (for normalization).

Next, the pulse rate detector 1 substitutes num and den stored in the RAM4 into the numerator and denominator of the expression for HR _ SNR shown in the figure to generate HR _ SNR which is the basis of the reliability of the pulse wave, and stores it in the RAM4 (step 250).

Then, the pulse rate detection device 1 returns to the main routine.

The expression HR _ SNR is a general expression representing an SN ratio, and the degree of a signal with respect to noise is represented by dividing num by den.

The pulse rate detection device 1 uses HR _ SNR as an initial value of the reliability of the pulse rate, and reduces the HR _ SNR or controls the display of the pulse rate according to the influence of the disturbance factor.

In this way, the reliability acquisition unit included in the pulse rate detection device 1 acquires the reliability based on the SN ratio of the pulse rate.

Fig. 9 is a flowchart for explaining the calculation process of the peak frequency (MR) and the SN ratio of disp _ Q (step 55 of fig. 4).

The order of processing is the same as in the case of orig _ Q, and the description of the common parts is simplified.

First, the disp _ Q of 10 seconds stored in the RAM4 is read out and fourier-transformed by FFT. The pulse rate detection device 1 thus generates FFT _ MR, which is an FFT value of disp _ Q, and stores it in the RAM4 (step 255).

FFT _ MR is a frequency component of the active disturbance, and the pulse rate detection device 1 determines the frequency (MR) of the maximum peak of FFT _ MR and stores it in the RAM4 (step 260).

Next, the pulse rate detection apparatus 1 generates a mask (MR _ M1) of the maximum peak frequency ± 5bpm and stores it in the RAM4 (step 265).

Next, the pulse rate detection device 1 generates a mask (MR _ M2) of the harmonic of the maximum peak ± 5bpm and stores it in the RAM4 (step 270).

Next, the pulse rate detection device 1 adds MR _ M1 and MR _ M2 stored in the RAM4 to generate one mask (MR _ M) and stores it in the RAM4 (step 275).

Next, the pulse rate detection device 1 creates an interference mask for MR _ M (MRN _ M is 1 to MR _ M) and stores it in the RAM4 (step 280).

Next, the pulse rate detector 1 normalizes the FFT _ MR stored in the RAM4, and stores the normalized data (FFT _ MRn) in the RAM4 (step 285).

The pulse rate detection device 1 calculates the normalization by (FFT _ MRn is the maximum value of FFT _ MR/FFT _ MR).

Next, the pulse rate detector 1 substitutes the FFT _ MRn stored in the RAM4 into the expression "num" shown in fig. 9 to calculate num, and stores the result in the RAM4 (step 290).

Next, the pulse rate detection device 1 calculates den by substituting FFT _ MRn stored in RAM4 into the expression of den shown in fig. 9 in the same manner, and stores the result in RAM4 (step 295).

Next, the pulse rate detector 1 substitutes num and den stored in the RAM4 into the MR _ SNR expression shown in fig. 9 to generate an MR _ SNR based on the active disturbance, and stores the MR _ SNR in the RAM4 (step 300).

Then, the pulse rate detection device 1 returns to the main routine.

In step 60 of FIG. 4, the MR _ SNR is used as the SN ratio of disp _ Q.

Fig. 10 is a flowchart for explaining the active interference processing (step 65 of fig. 4).

The pulse rate detecting apparatus 1 sets the activity disturbance flag to 1 (step 305).

The pulse rate detection device 1 also stores the active disturbance flag in the RAM4, sets the active disturbance flag to 0 when there is no active disturbance, and sets the active disturbance flag to 1 when there is active disturbance.

Next, the pulse rate detection device 1 performs "note: activity disturbance "etc. and an attention display of the subject matter that the activity disturbance was generated (step 310).

Further, the pulse rate detection device 1 updates HR _ SNR to HR _ SNR _ C2-1 and stores it in the RAM4 (step 315).

Then, the pulse rate detection device 1 returns to the main routine.

In this way, when there is an activity disturbance, the pulse rate detection device 1 decreases the reliability from HR _ SNR to a value 1 smaller than HR _ SNR _ C2 because the reliability of the pulse rate is low, and does not display the pulse rate in the subsequent anchor processing.

Fig. 11 is a flowchart for explaining the process of calculating the peak frequency (LR) and the SN ratio of conj _ Q (step 70 in fig. 4).

The order of processing is the same as in the case of orig _ Q, and the description of the common parts is simplified.

First, the conj _ Q of 10 seconds stored in the RAM4 is read out and subjected to fourier transform by FFT. The pulse rate detection device 1 thus generates FFT _ LR as the FFT value of conj _ Q and stores it in the RAM4 (step 350).

FFT _ LR is a frequency component of the optical interference, and the pulse rate detection device 1 determines the frequency (LR) of the maximum peak value of FFT _ LR and stores it in the RAM4 (step 355).

Next, the pulse rate detection device 1 generates a mask (LR _ M1) of the maximum peak frequency ± 5bpm and stores it in the RAM4 (step 360).

Next, the pulse rate detection device 1 generates a mask (LR _ M2) of the harmonic of the maximum peak ± 5bpm and stores it in the RAM4 (step 365).

Next, the pulse rate detector 1 adds LR _ M1 and LR _ M2 stored in the RAM4 to generate one mask (LR _ M) and stores the mask in the RAM4 (step 370).

Next, the pulse rate detection device 1 creates an interference mask for LR _ M (LRN _ M is 1-LR _ M) and stores it in the RAM4 (step 375).

Next, the pulse rate detector 1 normalizes the FFT _ LR stored in the RAM4, and stores the normalized data (FFT _ LRn) in the RAM4 (step 380).

The pulse rate detection device 1 calculates the normalization by (FFT _ LRn is the maximum value of FFT _ LR/FFT _ LR).

Next, the pulse rate detection device 1 substitutes the FFT _ LRn stored in the RAM4 into the expression "num" shown in fig. 11 to calculate num, and stores the result in the RAM4 (step 385).

Next, the pulse rate detector 1 calculates den by substituting FFT _ LRn stored in RAM4 into the expression of den shown in fig. 11 in the same manner, and stores the result in RAM4 (step 390).

Next, the pulse rate detector 1 substitutes num and den stored in the RAM4 into the LR _ SNR expression shown in fig. 11 to generate LR _ SNR based on optical disturbance, and stores the LR _ SNR in the RAM4 (step 395).

Then, the pulse rate detection device 1 returns to the main routine.

In step 75 of fig. 4, the LR _ SNR is used as the SN ratio of the conj _ Q.

Fig. 12 is a flowchart for explaining the optical disturbance processing (step 80 in fig. 4).

The pulse rate detection device 1 sets the light interference flag stored in the RAM4 to 1 (step 405).

The pulse rate detection device 1 stores the light interference flag in the RAM4, sets the light interference flag to 0 when there is no light interference, and sets the light interference flag to 1 when there is light interference.

Next, the pulse rate detection device 1 performs "note: light interference "and the like, and an attention display indicating that the light interference is generated (step 410).

Further, the pulse rate detection device 1 updates HR _ SNR to HR _ SNR _ C2-1 and stores it in the RAM4 (step 415).

Then, the pulse rate detection device 1 returns to the main routine.

In this way, when there is optical interference, the pulse rate detection device 1 decreases the reliability from HR _ SNR to a value 1 smaller than HR _ SNR _ C2 because the reliability of the pulse rate is low, and does not display the pulse rate in the subsequent anchor processing.

Fig. 13 is a flowchart for explaining the anchor processing (step 85 of fig. 4).

The anchor process is a process of controlling the display of the pulse rate according to the reliability of the pulse rate.

First, the pulse rate detector 1 reads HR _ SNR (reliability) and HR _ SNR _ C1 (pulse rate display reference 1) from the RAM4 and compares the magnitudes (step 450).

When HR _ SNR (reliability) is equal to or higher than HR _ SNR _ C1 (pulse rate display reference 1) (step 450; yes), the pulse rate detection device 1 substitutes HR (detected pulse rate) into HR _ ANC (anchor pulse rate) and stores it in the RAM4 (step 455). This is processing for storing the latest past pulse rate in HR _ ANC.

Then, the pulse rate detection device 1 displays HR (pulse rate) on the display unit 5 (step 460), and returns to the main routine.

On the other hand, when HR _ SNR (reliability) is smaller than HR _ SNR _ C1 (pulse number display reference 1) (step 450; no), the pulse rate detection device 1 determines whether HR _ SNR (reliability) is equal to or greater than HR _ SNR _ C2 (pulse number display reference 2) and whether the difference between HR (detected pulse rate) and HR _ ANC (most recent past pulse rate) is equal to or less than HR _ ANC _ C (anchor reference) (step 465).

When these conditions are satisfied (step 465; yes), the pulse rate detection device 1 substitutes HR (detected pulse rate) into HR _ ANC (anchor pulse rate) and stores it in the RAM4 (step 470), and displays HR (detected pulse rate) on the display unit 5 (step 460), and returns to the main routine.

On the other hand, if at least one of the conditions is not satisfied (step 465; no), the pulse rate detection device 1 performs the attention display process (step 475), and returns to the main routine.

Fig. 14 is a flowchart for explaining the procedure of the attention display processing of step 475 (fig. 13).

The pulse rate detecting apparatus 1 determines whether or not the motion disturbance flag stored in the RAM4 is set (step 500).

In the case where the activity disturbance flag is set (step 500; no), the pulse rate detection apparatus 1 returns to the anchor process.

On the other hand, if the active disturbance flag is not set (step 500; YES), the pulse rate detection device 1 further determines whether or not the light disturbance flag is set (step 505).

If the light interference flag is set (step 505; no), the pulse rate detection device 1 returns to the anchor process.

On the other hand, if the light interference flag is not set (step 505; yes), the pulse rate detection device 1 displays "caution: the weak pulse wave signal "(step 510), returns to the anchor process.

The embodiments described above can provide the following effects.

(1) The reliability of the detected pulse rate can be quantitatively evaluated.

(2) By inferring the characteristics of the disturbance, it is possible to suppress confusion of the activity disturbance or the light disturbance with the pulse.

(3) Since it is possible to suppress confusion between the motion disturbance or the optical disturbance and the pulse rate, the reliability of the detected pulse rate is improved.

(4) Even when the reliability of the detected pulse rate is low to some extent, the reliability can be improved by comparing the pulse rate with the pulse rate in the recent past (1 second ago) based on the characteristics of the human body such that the pulse rate does not change abruptly, and the rate at which the pulse rate can be detected can be increased.

(5) The period during which the pulse rate can be detected can be determined based on the display of the reliability of the pulse rate.

In the present embodiment, when there is disturbance, the pulse rate is not displayed with HR _ SNR being HR _ SNR _ C2-1, but the degree of disturbance may be divided more finely and the display of the pulse rate may be controlled according to the degree.

In the present embodiment, the display/non-display of the pulse rate is controlled based on the reliability, but various output methods can be employed in which the reliability is recorded together with the pulse rate and all the pulse rates are output, or the output destination is set to, for example, a control system of the vehicle.

In addition, in the present embodiment, the following configuration can also be provided.

(first configuration) a pulse rate detection device, comprising: a moving image acquisition unit that acquires a moving image obtained by imaging a body surface of a subject person; a pulse rate acquisition unit that acquires a pulse rate of the subject person based on a change in pixel values of the body surface in the acquired moving image; a reliability acquisition unit configured to acquire reliability of the acquired pulse rate; and an output unit configured to output the acquired pulse rate when the acquired reliability is equal to or higher than a predetermined reliability reference.

A pulse rate detection device according to the first aspect is characterized in that the pulse rate acquisition means acquires the pulse rate from a peak value of the pulse rate in the frequency domain of the acquired moving image, and the reliability acquisition means acquires the reliability based on an SN ratio of the acquired pulse rate.

(third configuration) the pulse rate detection device according to the second configuration is characterized in that the pulse rate detection device includes a reduction unit that reduces the reliability when an interference peak value in a frequency domain of an interference factor that reduces the accuracy of the acquired pulse rate and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

(fourth configuration) the pulse rate detector according to the third configuration, wherein the disturbance factor is a motion of the body surface, and the reduction means reduces the reliability when a motion peak in a frequency domain of the motion of the body surface in the moving image and the acquired pulse rate peak approach a predetermined amount or more.

A pulse rate detector according to the third or fourth aspect, wherein the disturbance factor is fluctuation of light irradiated onto the body surface, and the reducing unit reduces the reliability when a peak value of light in a frequency domain of the fluctuation of light in the moving image and the acquired pulse rate peak value are close to each other by a predetermined amount or more.

(sixth configuration) a pulse rate detection program that realizes the following functions by a computer: a dynamic image acquisition function of acquiring a dynamic image obtained by photographing a body surface of a subject person; a pulse rate acquisition function of acquiring a pulse rate of the subject person based on a change in pixel values of the body surface in the acquired moving image; a reliability acquisition function of acquiring reliability of the acquired pulse rate; and an output function that outputs the acquired pulse rate when the acquired reliability is equal to or greater than a predetermined reliability reference.

Description of reference numerals: 1 … pulse rate detection means; 2 … CPU; 3 … ROM; 4 … RAM; a display portion of 5 …; 6 … input; 7 … output; an 8 … camera; 10 … storage part; 11 … subject person; 20 … images; 22 … measurement area; 24 … center.