阅读说明：本技术 驾驶员注意力监测的系统和方法 (System and method for driver attention monitoring ) 是由 郑志强 孙健 于 2020-05-29 设计创作，主要内容包括：本申请公开了一种驾驶员注意力监测的系统和方法,属于车联网技术领域。该系统包括车载设备和检测服务器。车载设备,用于对驾驶员行为视频进行第一检测,如果在驾驶员行为视频中检测出注意力分散行为,则在驾驶员行为视频中确定包含检测出的注意力分散行为的候选报警视频,向检测服务器发送候选报警视频。检测服务器,用于对候选报警视频进行第二检测,如果在候选报警视频中检测出注意力分散行为,则确定候选报警视频为报警视频。采用本申请,可以有效提高确定出的报警视频的准确率。(The application discloses a system and a method for monitoring driver attention, and belongs to the technical field of Internet of vehicles. The system comprises the vehicle-mounted equipment and the detection server. And the vehicle-mounted equipment is used for carrying out first detection on the driver behavior video, determining candidate alarm videos containing the detected distractive behaviors in the driver behavior video if the distractive behaviors are detected in the driver behavior video, and sending the candidate alarm videos to the detection server. And the detection server is used for carrying out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, determining the candidate alarm video as the alarm video. By the aid of the method and the device, accuracy of the determined alarm video can be effectively improved.)

1. A system for driver attention monitoring, characterized in that the system comprises an in-vehicle device and a detection server, wherein,

the vehicle-mounted equipment is used for carrying out first detection on a driver behavior video, determining a candidate alarm video containing detected distractive behaviors in the driver behavior video if the distractive behaviors are detected in the driver behavior video, and sending the candidate alarm video to the detection server;

and the detection server is used for carrying out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, determining that the candidate alarm video is the alarm video.

2. The system of claim 1, wherein the first detection has a higher false positive rate than the second detection.

3. The system of claim 2, wherein the distraction behavior includes a first type of distraction behavior and a second type of distraction behavior, the first type of distraction behavior being less difficult to detect than the second type of distraction behavior, the first type of distraction behavior having a higher risk coefficient than the second type of distraction behavior.

4. The system of claim 3, wherein the first type of distraction includes one or more of yawning, eye closing, making a call, smoking a cigarette, unbelting, leaving a driver's seat, blocking a camera, disengaging a steering wheel with both hands, and disengaging a steering wheel with one hand;

the second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

5. The system according to claim 3 or 4, wherein the vehicle-mounted device is further configured to issue an alarm signal if the first type of distraction is detected in the driver behavior video; or

The detection server is further used for sending an alarm message to the vehicle-mounted equipment if the second type of distraction behavior is detected in the candidate alarm video; and the vehicle-mounted equipment is also used for responding to the alarm message and sending out an alarm signal.

6. The system according to any one of claims 1 to 4, wherein the vehicle-mounted device is further configured to, if a distraction behavior is detected in the driver behavior video, acquire a position of a vehicle, a vehicle speed, and a behavior occurrence time, and transmit the position, the vehicle speed, and the behavior occurrence time to the detection server;

the detection server is further configured to determine that the position, the vehicle speed, and the behavior occurrence time are alarm additional information if a distraction behavior is detected in the candidate alarm video.

7. The system of claim 6, wherein the detection server is further configured to, if distraction behavior is detected in the candidate alarm video, grab a target image frame in the candidate alarm video, and determine that the target image frame is a distraction behavior presentation picture.

8. The system of claim 7, further comprising an alert platform, wherein the detection server is further configured to send the alert video, the alert additional information, and the distraction behavior presentation picture to the alert platform.

9. A method for driver attention monitoring, which is applied to a system for driver attention monitoring, the system comprises an on-board device and a detection server, and the method comprises the following steps:

the vehicle-mounted equipment carries out first detection on a driver behavior video, if the distraction behavior is detected in the driver behavior video, a candidate alarm video containing the detected distraction behavior is determined in the driver behavior video, and the candidate alarm video is sent to the detection server;

and the detection server carries out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, the candidate alarm video is determined to be the alarm video.

10. The method of claim 9, wherein the first detection has a higher false positive rate than the second detection.

11. The method of claim 10, wherein the distractive acts include a first type of distractive act and a second type of distractive act, wherein the first type of distractive act is less difficult to detect than the second type of distractive act, and wherein the first type of distractive act has a higher risk coefficient than the second type of distractive act;

the first type of distraction includes one or more of yawning, closing eyes, making a call, smoking, not fastening a safety belt, getting away from a driver seat, shielding a camera, disengaging a steering wheel with two hands and disengaging the steering wheel with one hand;

the second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

12. The method of claim 11, further comprising:

if the vehicle-mounted equipment detects the first type of distraction behavior in the driver behavior video, sending an alarm signal;

if the detection server detects the second type of distraction behavior in the candidate alarm video, sending an alarm message to the vehicle-mounted equipment; and the vehicle-mounted equipment responds to the alarm message and sends out an alarm signal.

13. The method according to any one of claims 9-12, further comprising:

if the vehicle-mounted equipment detects the distraction behavior in the driver behavior video, acquiring the position, the speed and the behavior occurrence time of a vehicle, and sending the position, the speed and the behavior occurrence time to the detection server;

and if the detection server detects the distraction behavior in the candidate alarm video, determining the position, the vehicle speed and the behavior occurrence time as alarm additional information, capturing a target image frame in the candidate alarm video, and determining the target image frame as a distraction behavior display picture.

14. The method of claim 13, wherein the system further comprises an alert platform, the method further comprising:

and the detection server sends the alarm video, the alarm additional information and the attention dispersion behavior display picture to the alarm platform.

Technical Field

The application relates to the technical field of vehicle networking, in particular to a system and a method for monitoring attention of a driver.

Background

With the increasing popularity of automobiles, the potential safety hazards brought by the behavior of distracting drivers are increasing. Therefore, it is important to monitor the attention of the driver and to discover the safety hazard.

A system for driver attention monitoring in the related art includes an in-vehicle device embedded in an automobile and an alarm platform. The vehicle-mounted equipment is connected with a camera installed on the automobile, the camera collects driver behavior videos in real time, and the driver behavior videos are transmitted to the vehicle-mounted equipment. The vehicle-mounted equipment detects the behavior video of the driver, and sends out an alarm signal when the distraction behavior is detected. Meanwhile, the vehicle-mounted equipment can also send an alarm video to the alarm platform, so that an administrator can check and confirm the attention dispersion behavior of the driver through the alarm platform.

In carrying out the present application, the applicant has found that the related art has at least the following problems:

due to the fact that the vehicle-mounted equipment is limited in computing capacity and cannot use complex algorithms, accuracy of detecting the attention dispersion behaviors is low, and accuracy of the determined alarm video is low.

Disclosure of Invention

The embodiment of the application provides a system and a method for monitoring the attention of a driver, which can solve the technical problems in the related art. The technical scheme of the driver attention monitoring system and the driver attention monitoring method is as follows:

in a first aspect, a system for driver attention monitoring is provided, the system comprising an in-vehicle device and a detection server, wherein,

the vehicle-mounted equipment is used for carrying out first detection on a driver behavior video, determining a candidate alarm video containing detected distractive behaviors in the driver behavior video if the distractive behaviors are detected in the driver behavior video, and sending the candidate alarm video to the detection server;

and the detection server is used for carrying out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, determining that the candidate alarm video is the alarm video.

In one possible implementation, the false detection rate of the first detection is higher than the false detection rate of the second detection.

In one possible implementation, the distraction behavior includes a first type of distraction behavior and a second type of distraction behavior, the first type of distraction behavior has a lower detection difficulty than the second type of distraction behavior, and the risk coefficient of the first type of distraction behavior is higher than the risk coefficient of the second type of distraction behavior.

In a possible implementation manner, the vehicle-mounted device is further configured to send out an alarm signal if the first type of distraction behavior is detected in the driver behavior video.

In a possible implementation manner, the detection server is further configured to send an alarm message to the vehicle-mounted device if the second type of distraction behavior is detected in the candidate alarm video;

and the vehicle-mounted equipment is also used for responding to the alarm message and sending out an alarm signal.

In one possible implementation, the first type of distraction includes one or more of yawning, closing eyes, making a call, smoking a cigarette, not wearing a seat belt, leaving a driver's seat, blocking a camera, disengaging a steering wheel with both hands, and disengaging a steering wheel with one hand;

the second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

In a possible implementation manner, the vehicle-mounted device is further configured to, if a distraction behavior is detected in the driver behavior video, acquire a position, a vehicle speed, and a behavior occurrence time of a vehicle, and send the position, the vehicle speed, and the behavior occurrence time to the detection server;

the detection server is further configured to determine that the position, the vehicle speed, and the behavior occurrence time are alarm additional information if a distraction behavior is detected in the candidate alarm video.

In a possible implementation manner, the detection server is further configured to, if a distraction behavior is detected in the candidate alarm video, capture a target image frame in the candidate alarm video, and determine that the target image frame is a distraction behavior display picture.

In a possible implementation manner, the system further includes an alarm platform, and the detection server is further configured to send the alarm video, the alarm additional information, and the distraction behavior display picture to the alarm platform.

In a second aspect, a method for driver attention monitoring is provided, and the method is applied to a system for driver attention monitoring, wherein the system comprises an on-board device and a detection server, and the method comprises the following steps:

the vehicle-mounted equipment carries out first detection on a driver behavior video, if the distraction behavior is detected in the driver behavior video, a candidate alarm video containing the detected distraction behavior is determined in the driver behavior video, and the candidate alarm video is sent to the detection server;

and the detection server carries out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, the candidate alarm video is determined to be the alarm video.

In one possible implementation, the false detection rate of the first detection is higher than the false detection rate of the second detection.

In one possible implementation, the distraction behavior includes a first type of distraction behavior and a second type of distraction behavior, the first type of distraction behavior has a lower detection difficulty than the second type of distraction behavior, and the risk coefficient of the first type of distraction behavior is higher than the risk coefficient of the second type of distraction behavior.

In one possible implementation, if after detecting distracting behavior in the driver behavior video, the method further comprises:

and if the vehicle-mounted equipment detects the first type of distraction behavior in the driver behavior video, sending an alarm signal.

In one possible implementation, if attention deficit behavior is detected in the candidate alarm video, the method further includes:

if the detection server detects the second type of distraction behavior in the candidate alarm video, sending an alarm message to the vehicle-mounted equipment;

and the vehicle-mounted equipment responds to the alarm message and sends out an alarm signal.

In one possible implementation, the first type of distraction includes one or more of yawning, closing eyes, making a call, smoking a cigarette, not wearing a seat belt, leaving a driver's seat, blocking a camera, disengaging a steering wheel with both hands, and disengaging a steering wheel with one hand;

the second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

In one possible implementation, the method further includes:

if the vehicle-mounted equipment detects the distraction behavior in the driver behavior video, acquiring the position, the speed and the behavior occurrence time of a vehicle, and sending the position, the speed and the behavior occurrence time to the detection server;

and if the detection server detects the attention dispersion behavior in the candidate alarm video, determining the position, the vehicle speed and the behavior occurrence time as alarm additional information.

In one possible implementation, the method further includes:

and if the detection server detects the distraction behavior in the candidate alarm video, capturing a target image frame in the candidate alarm video, and determining that the target image frame is a distraction behavior display picture.

In one possible implementation, the system further includes an alarm platform, and the method further includes:

and the detection server sends the alarm video, the alarm additional information and the attention dispersion behavior display picture to the alarm platform.

The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:

according to the technical scheme, the candidate alarm video detected by the vehicle-mounted equipment is set, and whether the candidate alarm video is the alarm video or not can be determined only after the candidate alarm video is detected by the detection server for the second time. Therefore, due to the fact that the detection server has high computing capacity, a complex algorithm with high accuracy can be used, the accuracy of the alarm video determined through the second detection of the detection server is high, and the accuracy of the alarm video is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a flow chart of a method for driver attention monitoring provided by an embodiment of the present application;

FIG. 2 is a flow chart of a method for driver attention monitoring provided by an embodiment of the present application;

FIG. 3 is a flow chart of a method for driver attention monitoring provided by an embodiment of the present application;

FIG. 4 is a flow chart of a method for driver attention monitoring provided by an embodiment of the present application;

FIG. 5 is a flow chart of a method for driver attention monitoring provided by an embodiment of the present application;

fig. 6 is a flowchart of a method for monitoring driver attention according to an embodiment of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

The embodiment of the application provides a driver attention monitoring method which can be realized by a system for monitoring driver attention, wherein the system comprises an on-board device and a detection server.

As shown in FIG. 1, a flow chart of a method of driver attention monitoring is provided, which may include the steps of:

in step 101, the vehicle-mounted device performs first detection on a driver behavior video, determines a candidate warning video including the detected distractive behavior in the driver behavior video if the distractive behavior is detected in the driver behavior video, and transmits the candidate warning video to the detection server.

In one possible implementation, the false detection rate of the first detection is higher than the false detection rate of the second detection.

In one possible implementation, the distraction behavior includes a first type of distraction behavior and a second type of distraction behavior, the first type of distraction behavior has a lower detection difficulty than the second type of distraction behavior, and the risk coefficient of the first type of distraction behavior is higher than the risk coefficient of the second type of distraction behavior.

In one possible implementation, if the on-board device detects a first type of distracting behavior in the video of driver behavior, an alarm signal is issued.

In one possible implementation, the first type of distraction includes one or more of yawning, closing the eyes, making a phone call, smoking a cigarette, not wearing a seat belt, leaving the driver's seat, blocking the camera, disengaging the steering wheel with both hands, and disengaging the steering wheel with one hand. The second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

In one possible implementation, if the vehicle-mounted device detects the distraction behavior in the driver behavior video, the position, the vehicle speed and the behavior occurrence time of the vehicle are acquired, and the position, the vehicle speed and the behavior occurrence time are sent to the detection server.

In step 102, the detection server performs a second detection on the candidate alarm video, and determines that the candidate alarm video is an alarm video if the distraction behavior is detected in the candidate alarm video.

In one possible implementation, if the detection server detects the second type of distraction behavior in the candidate alarm video, an alarm message is sent to the vehicle-mounted device. And the vehicle-mounted equipment responds to the alarm message and sends out an alarm signal.

In one possible implementation, if the detection server detects distracting behavior in the candidate alarm video, the location, the vehicle speed, and the behavior occurrence time are determined as the alarm additional information.

In a possible implementation manner, if the detection server detects the distraction behavior in the candidate alarm video, the target image frame is captured in the candidate alarm video, and the target image frame is determined to be the distraction behavior display picture.

In a possible implementation manner, the system for monitoring the attention of the driver further comprises an alarm video, and the detection server sends the alarm video, the alarm additional information and the display picture of the attention dispersion behavior to the alarm platform.

According to the scheme shown in the embodiment of the application, the candidate alarm video detected by the vehicle-mounted equipment is set, and whether the candidate alarm video is the alarm video can be determined only after the candidate alarm video is detected by the detection server for the second time. Therefore, due to the fact that the detection server has high computing capacity, a complex algorithm with high accuracy can be used, the accuracy of the alarm video determined through the second detection of the detection server is high, and the accuracy of the alarm video is improved.

It should be noted that the alarm platform may be run on the detection server, so that when the detection server determines that the candidate alarm video is an alarm video, the alarm video may be stored. And the candidate alarm video which is not the alarm video can be discarded, so that the processing resource of the detection server is saved.

The alarm platform may not be operated on the detection server, and the alarm platform is independent of the detection server. The system for monitoring the attention of the driver further comprises an alarm platform, and after the detection server determines that the candidate alarm video is the alarm video, the alarm video can be sent to the alarm platform.

For better understanding of the present application, the following description will be made for the detection of distraction, which may be a first detection or a second detection:

using yawning behavior detection as an example:

first, in the image of the driver behavior video, the position of the driver's mouth is located. Then, feature points of the driver's mouth are extracted, coordinates of the respective feature points are calculated, and the distance of the feature points is calculated from the coordinates of the feature points. Finally, whether the driver opens the mouth and the time for opening the mouth can be judged according to the distances of the characteristic points in the plurality of continuous images. Thus, it is determined whether the driver has yawning behavior.

Taking eye closing behavior detection as an example:

first, in an image of a driver behavior video, the position of the eyes of the driver is located. Then, binarization processing is carried out on the eyes of the driver, and an eye edge image is obtained. And finally, calculating the complexity of the eye edge, and if the complexity of the eye edge is larger, indicating that the eyes of the driver are open. If the complexity of the eye edge is small, the driver is in the eye closing state.

As shown in fig. 2, a flow chart of a method for driver attention monitoring is provided, which is mainly described for the case where a first type of distraction behavior is detected, and which may comprise the steps of:

in step 201, the vehicle-mounted device performs first detection on a driver behavior video.

The vehicle-mounted equipment can be embedded equipment arranged on the vehicle, the vehicle-mounted equipment can be connected with camera equipment arranged on the vehicle, and the camera equipment can collect the driver behavior video of the driver.

In implementation, the vehicle-mounted device can acquire the driver behavior video from the camera device in real time and perform first detection on the driver behavior video.

In one possible implementation, the first detection may be a detection performed by a deep learning algorithm, and the corresponding processing procedure of step 201 may be as follows, and the video of the driver behavior is input into the first attention detection neural network model for the first detection.

In implementation, the driver behavior video is input into the first attention detection neural network model, and each image frame in the driver behavior video may be respectively input into the first attention detection neural network model; alternatively, a plurality of consecutive image frames (i.e., a video segment) in the driver behavior video may be input into the first attention monitoring neural network model, which is not limited in this application. In the first case, an image frame including a distraction behavior is detected, and in the second case, a video segment including a distraction behavior is detected.

In addition, the phenomenon that the vehicle-mounted equipment cannot timely remind a driver to concentrate attention due to the fact that the vehicle-mounted equipment fails to detect the distraction behavior and fails to send out the alarm signal is avoided. The detection rate of the distraction behavior of the first detection can be made higher, that is, the detection threshold of the first detection is set to be lower, all distraction behaviors can be detected as far as possible, and the omission ratio is reduced.

The method of determination of the detection threshold may be as follows:

and training the first attention detection neural network model by using a large number of samples of the driver behavior video containing the distractive behaviors, and counting false detection rates corresponding to different detection thresholds, wherein the false detection rates are used for representing the rate of false detection of the non-distractive behaviors as the distractive behaviors.

Then, a two-dimensional relationship diagram of the false detection rate and the detection threshold is drawn on a rectangular coordinate system, and since the detection rate of the distractive behavior decreases as the detection threshold increases, in order to ensure the detection rate of the distractive behavior, the detection threshold represented by an inflection point whose slope does not decrease significantly any more is selected as the detection threshold of the first detection.

A lower detection threshold value is selected according to the method, so that the low omission factor of the vehicle-mounted equipment can be ensured, and the driver is warned in time when the situation that the attention of the driver is dispersed occurs, so that the accident is prevented.

In step 202, if the vehicle-mounted device detects a first type of distraction behavior in the driver behavior video, an alarm signal is issued.

The alarm signal can be a sound alarm signal, a vibration alarm signal or a light alarm signal. The alarm signal may also be a combination of two or three of the above mentioned alarm signals, e.g. an audible and visual alarm signal.

The detection difficulty of the first type of distraction is lower than that of the second type of distraction, and the risk coefficient of the first type of distraction is higher than that of the second type of distraction.

The first category of distraction includes one or more of yawning, closing the eyes, making a call, smoking a cigarette, not wearing a safety belt, leaving the driver's seat, blocking the camera, disengaging the steering wheel with both hands, and disengaging the steering wheel with one hand.

The second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

In implementation, if the first type of distracted behaviors are detected in the driver behavior video, the vehicle-mounted equipment sends out an alarm signal to remind the driver of concentrating attention and avoid danger.

In one possible implementation, if the first detection is that each image frame of the driver behavior video is input into the first attention detection model, the corresponding processing of step 202 is to issue an alarm signal if an image frame containing a first type of distraction behavior is detected in the driver behavior video.

In another possible implementation, if the first detection is that each video segment in the video of driver behavior is input into the first attention detection model, the corresponding processing of step 202 is to issue a warning signal if a video segment containing the first type of distraction behavior is detected in the video of driver behavior.

In step 203, the vehicle-mounted device determines candidate alarm videos containing the detected first-class distraction behaviors in the driver behavior video, and sends the candidate alarm videos to the detection server.

In implementation, after the first type of distracted behaviors are detected in the driver behavior video, candidate alarm videos with a preset time length (such as 12s) before and after the behavior occurrence time can be intercepted from the driver behavior video, and the candidate alarm videos are sent to the detection server.

In one possible implementation, if the first detection is that each image frame of the driver behavior video is input into the first attention detection model, the corresponding processing procedure of step 203 may be as follows, if an image frame containing a first type of distraction behavior is detected in the driver behavior video, a candidate warning video containing the image frame is intercepted in the driver behavior video, and the candidate warning video is sent to the detection server.

In another possible implementation manner, if the first detection is that each video segment in the driver behavior video is input into the first attention detection model, the corresponding processing procedure of step 203 may be as follows, if a video segment containing the first type of distraction behavior is detected in the driver behavior video, a candidate alarm video containing the video segment is intercepted in the driver behavior video, and the candidate alarm video is sent to the detection server.

In addition, in addition to sending the candidate warning videos, information such as the position of the vehicle, the vehicle speed, the occurrence time of the distraction behavior and the like can be sent to the detection server, so that the monitoring authority can better judge the risk of the distraction behavior, for example, the higher the vehicle speed is, the higher the risk of the distraction behavior is. The corresponding processing procedure of step 203 may be as follows, when the first type of distraction behavior is detected, acquiring the position, the vehicle speed and the behavior occurrence time of the vehicle, and sending the position, the vehicle speed and the behavior occurrence time of the vehicle to the detection server.

The processing sequence of step 202 and step 203 may be that step 202 is performed first, and then step 203 is performed, or that step 203 is performed first and then step 202 is performed, or may be performed simultaneously, which is not limited in the present application.

In step 204, the detection server performs a second detection on the candidate alert video.

And the false detection rate of the first detection is higher than that of the second detection. The detection rate of the second detection is lower than the detection rate of the first detection.

In implementation, since the detection threshold of the vehicle-mounted device in the first detection of the driving video is low, the detection rate of the first detection is high, and the omission factor is low, but this inevitably makes the false detection rate high. Therefore, if the alarm video sent by the vehicle-mounted device is directly sent to the alarm platform, excessive error alarm videos (i.e., some alarm videos substantially without distractive behaviors) exist on the alarm platform, and thus the waste of the alarm platform resources is caused.

Therefore, the method and the device for screening the candidate alarm videos sent by the vehicle-mounted equipment are provided by the detection server. The detection server has rich hardware resources, strong computing capability and system scheduling capability, and can operate a complex detection algorithm with high accuracy, so that the detection server can accurately screen candidate alarm videos. It will be appreciated that the second detection should have a lower detection rate of distractive behaviour than the first detection, and a lower false detection rate than the first detection.

In one possible implementation, the second detection may be a detection performed by a deep learning algorithm, and the corresponding processing procedure of step 204 may be as follows, and the candidate alarm video is input into the second attention monitoring neural network model for the second detection.

In implementation, the candidate alarm videos are input into the second attention monitoring neural network model, and each image frame in the candidate alarm videos may be respectively input into the second attention monitoring neural network model; alternatively, a plurality of consecutive image frames (i.e., one video segment, which may be the entire candidate alarm video or a part of the candidate alarm video) in the candidate alarm video may be input into the second attention monitoring neural network model, which is not limited in this application. In the first case, an image frame including a distraction behavior is detected, and in the second case, a video segment including a distraction behavior is detected.

By this operation, the influence of false detection in which the detection threshold is increased is further reduced in order to increase the detection rate of the distraction behavior.

In step 205, if the detection server detects a first type of distraction behavior in the candidate alarm video, the candidate alarm video is determined to be an alarm video.

In implementation, if the detection server still detects the first type of distraction behavior in the candidate alarm video, which indicates that the detection of the vehicle-mounted device is correct, the candidate alarm video is determined to be the alarm video.

In a possible implementation manner, for convenience of viewing by an administrator, the corresponding processing procedure of step 205 may be as follows, if the first type of distraction behavior is detected in the candidate alarm video, capture a target image frame in the alarm video, and determine that the target image frame is a distraction behavior showing picture.

In implementation, the target image frame is captured in the alarm video to be used as the attention distraction behavior showing picture. When the administrator confirms the distracted behavior of the driver, the administrator can directly know the distracted behavior of the driver through the display picture of the distracted behavior without opening the alarm video. Thus, the workload of the administrator is reduced.

In addition, in a possible implementation manner, if the detection server receives the candidate warning video and also receives the position, the vehicle speed, and the behavior occurrence time of the vehicle, the corresponding processing procedure of step 205 may be as follows, and if the first type of distraction behavior is detected in the candidate warning video, the position, the vehicle speed, and the behavior occurrence time of the vehicle are determined as the warning additional information. Thus, it is convenient for the supervising authority to make a better judgment on the risk of the distraction behavior, for example, the higher the vehicle speed, the stronger the risk of the distraction behavior.

It should be noted that the alarm platform may be operated on the detection server, and the alarm platform may store the determined alarm video, the attention dispersion behavior display picture, and the alarm additional information, so as to be conveniently viewed by the staff. For candidate alarm videos that are not alarm videos, the candidate alarm videos may be discarded, and the positions, speeds, and behavior occurrence times of vehicles corresponding to the candidate alarm videos may be discarded.

As shown in fig. 3, the warning platform may also be independent of the detection server, and the system for monitoring the attention of the driver further includes the warning platform. The detection server may send the determined alarm video, the distraction behavior presentation picture, and the alarm additional information to the alarm platform. And for the candidate alarm video which is not the alarm video, the candidate alarm video is not sent to the alarm platform, and the candidate alarm video and the position, the speed and the action occurrence time of the corresponding vehicle can be discarded.

According to the technical scheme, the candidate alarm video detected by the vehicle-mounted equipment is set, and whether the candidate alarm video is the alarm video or not can be determined only after the candidate alarm video is detected by the detection server for the second time. Therefore, due to the fact that the detection server has high computing capacity, a complex algorithm with high accuracy can be used, the accuracy of the alarm video determined through the second detection of the detection server is high, and the accuracy of the alarm video is improved.

In addition, the detection server can screen the alarm video sent by the vehicle-mounted device, so that the detection threshold value of the first detection of the vehicle-mounted device can be set to be lower, namely the detection rate of the distraction behavior of the first detection can be higher. Therefore, the vehicle-mounted equipment can send out alarm signals as far as possible when the driver is in the inattentive behavior, and therefore the driver can be reminded timely. Moreover, due to the presence of the detection server, even if the detection rate of the distractive behavior of the first detection is high, the problem that a large amount of wrong alarm data is sent to the alarm platform does not occur.

As shown in fig. 4, a flow chart of yet another method for driver attention monitoring is provided, which is mainly described with respect to the case where the second type of distraction behavior is detected, and which may comprise the steps of:

in step 301, the vehicle-mounted device performs a first detection on the driver behavior video.

The vehicle-mounted equipment can be embedded equipment arranged on the vehicle, the vehicle-mounted equipment can be connected with camera equipment arranged on the vehicle, and the camera equipment can collect the driver behavior video of the driver.

In implementation, the vehicle-mounted device can acquire the driver behavior video from the camera device in real time and perform first detection on the driver behavior video.

In one possible implementation, the first detection may be a detection performed by a deep learning algorithm, and the corresponding processing of step 301 may be as follows, and the video of the driver behavior is input into the first attention detection neural network model for the first detection.

In implementation, the driver behavior video is input into the first attention detection neural network model, and each image frame in the driver behavior video may be respectively input into the first attention detection neural network model; alternatively, a plurality of consecutive image frames (i.e., a video segment) in the driver behavior video may be input into the first attention monitoring neural network model, which is not limited in this application. In the first case, an image frame including a distraction behavior is detected, and in the second case, a video segment including a distraction behavior is detected.

In addition, the phenomenon that the vehicle-mounted equipment cannot timely remind a driver to concentrate attention due to the fact that the vehicle-mounted equipment fails to detect the distraction behavior and fails to send out the alarm signal is avoided. The detection rate of the distraction behavior of the first detection can be made higher, that is, the detection threshold of the first detection is set to be lower, all distraction behaviors can be detected as far as possible, and the omission ratio is reduced.

The method of determination of the detection threshold may be as follows:

and training the first attention detection neural network model by using a large number of samples of the driver behavior video containing the distractive behaviors, and counting false detection rates corresponding to different detection thresholds, wherein the false detection rates are used for representing the rate of false detection of the non-distractive behaviors as the distractive behaviors.

Then, a two-dimensional relationship diagram of the false detection rate and the detection threshold is drawn on a rectangular coordinate system, and since the detection rate of the distractive behavior decreases as the detection threshold increases, in order to ensure the detection rate of the distractive behavior, the detection threshold represented by an inflection point whose slope does not decrease significantly any more is selected as the detection threshold of the first detection.

A lower detection threshold value is selected according to the method, so that the low omission factor of the vehicle-mounted equipment can be ensured, and the driver is warned in time when the situation that the attention of the driver is dispersed occurs, so that the accident is prevented.

In step 302, if the second type of distracted behavior is detected in the driver behavior video, candidate warning videos including the detected second type of distracted behavior are determined in the driver behavior video, and the candidate warning videos are transmitted to the detection server.

The detection difficulty of the first type of distraction is lower than that of the second type of distraction, and the risk coefficient of the first type of distraction is higher than that of the second type of distraction.

The first category of distraction includes one or more of yawning, closing the eyes, making a call, smoking a cigarette, not wearing a safety belt, leaving the driver's seat, blocking the camera, disengaging the steering wheel with both hands, and disengaging the steering wheel with one hand.

The second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes. The second category of distraction behavior is less distinctive than the first category of distraction behavior.

In the implementation, the detection difficulty of the second type of distraction behavior is high, and in addition, the calculation capability of the vehicle-mounted device is poor, and the running algorithm is simple, so that the false detection rate of the vehicle-mounted device for detecting the second type of distraction behavior is high. Therefore, if the vehicle-mounted device is set to send out the alarm signal after detecting the second type of distracted behaviors, the vehicle-mounted device can send out the alarm signal under the condition that the driver does not have the distracted behaviors, so that the driver is interfered and the driving experience of the driver is influenced.

Therefore, after the vehicle-mounted equipment detects the second type of distraction behavior, the vehicle-mounted equipment does not send out the alarm signal. Instead, the candidate warning video is transmitted to the in-vehicle device, and whether to send out a warning signal is determined based on the feedback information of the detection server. Moreover, since the second type of distraction has a specificity that persists for a period of time, the above arrangement does not cause too late an alarm signal to be issued.

Specifically, after the second type of distracted behavior is detected in the driver behavior video, candidate alarm videos with a preset time duration (for example, 12s) before and after the behavior occurrence time may be captured from the driver behavior video, and the candidate alarm videos may be sent to the detection server.

In one possible implementation, if the first detection is that each image frame of the driver behavior video is input into the first attention detection model, the corresponding processing procedure of step 302 may be as follows, if an image frame containing the second type of distraction behavior is detected in the driver behavior video, a candidate warning video containing the image frame is intercepted in the driver behavior video, and the candidate warning video is sent to the detection server.

In another possible implementation manner, if the first detection is that each video segment in the driver behavior video is input into the first attention detection model, the corresponding processing procedure of step 302 may be as follows, if a video segment containing the second type of attention distraction behavior is detected in the driver behavior video, a candidate alarm video containing the video segment is intercepted in the driver behavior video, and the candidate alarm video is sent to the detection server.

In addition, in addition to sending the candidate warning videos, information such as the position of the vehicle, the vehicle speed, the occurrence time of the distraction behavior and the like can be sent to the detection server, so that the monitoring authority can better judge the risk of the distraction behavior, for example, the higher the vehicle speed is, the higher the risk of the distraction behavior is. The corresponding processing procedure of step 302 may be as follows, when the second type of distraction behavior is detected, obtaining the position, the vehicle speed and the behavior occurrence time of the vehicle, and sending the position, the vehicle speed and the behavior occurrence time of the vehicle to the detection server.

In step 303, the detection server performs a second detection on the candidate alert video.

And the false detection rate of the first detection is higher than that of the second detection. The detection rate of the second detection is lower than the detection rate of the first detection.

In implementation, since the detection threshold of the vehicle-mounted device in the first detection of the driving video is low, the detection rate of the first detection is high, and the omission factor is low, but this inevitably makes the false detection rate high. Therefore, if the candidate alarm videos sent by the vehicle-mounted device are directly sent to the alarm platform, excessive error alarm videos (i.e., some alarm videos substantially without distractive behaviors) exist on the alarm platform, and thus the resource waste of the alarm platform is caused.

Therefore, the detection server is configured to screen candidate alarm videos sent by the vehicle-mounted equipment. The detection server has rich hardware resources, strong computing capability and system scheduling capability, and can operate a complex detection algorithm with high accuracy, so that the detection server can accurately screen candidate alarm videos. It will be appreciated that the second detection should have a lower detection rate of distractive behaviour than the first detection, and a lower false detection rate than the first detection.

In a possible implementation manner, the second detection may be a detection performed by a deep learning algorithm, and the corresponding processing procedure in step 303 may be as follows, and the candidate alarm video is input into the second attention monitoring neural network model for the second detection.

In implementation, the candidate alarm videos are input into the second attention monitoring neural network model, and each image frame in the candidate alarm videos may be respectively input into the second attention monitoring neural network model; alternatively, a plurality of consecutive image frames (i.e., one video segment, which may be the entire candidate alarm video or a part of the candidate alarm video) in the candidate alarm video may be input into the second attention monitoring neural network model, which is not limited in this application. In the first case, an image frame including a distraction behavior is detected, and in the second case, a video segment including a distraction behavior is detected.

It should be noted that the candidate alarm videos may be classified into a candidate alarm video for the first type of distraction behavior and a candidate alarm video for the second type of distraction behavior. For the two types of candidate alarm videos, the same algorithm may be used for the second detection, and different algorithms may also be used for the second detection, which is not limited in the present application.

In step 304, if the detection server detects the second type of distraction behavior in the candidate alarm video, it is determined that the candidate alarm video is an alarm video.

In implementation, if the detection server still detects the second type of distraction behavior in the candidate alarm video, which indicates that the detection of the vehicle-mounted device is correct, the candidate alarm video is determined to be the alarm video.

In one possible implementation manner, for convenience of viewing by an administrator, the corresponding processing procedure of step 304 may be as follows, if the second type of distraction behavior is detected in the candidate alarm video, capture the target image frame in the alarm video, and determine that the target image frame is the distraction behavior showing picture.

In implementation, the target image frame is captured in the alarm video to be used as the attention distraction behavior showing picture. When the administrator confirms the distracted behavior of the driver, the administrator can directly know the distracted behavior of the driver through the display picture of the distracted behavior without opening the alarm video. Thus, the workload of the administrator is reduced.

In addition, in a possible implementation manner, if the detection server receives the position, the vehicle speed, and the behavior occurrence time of the vehicle while receiving the candidate alarm video, the corresponding processing procedure of step 304 may be as follows, if the second type of distraction behavior is detected in the candidate alarm video, determining the position, the vehicle speed, and the behavior occurrence time of the vehicle as the alarm additional information. Thus, it is convenient for the supervising authority to make a better judgment on the risk of the distraction behavior, for example, the higher the vehicle speed, the stronger the risk of the distraction behavior.

It should be noted that the alarm platform may be operated on the detection server, and the alarm platform may store the determined alarm video, the attention dispersion behavior display picture, and the alarm additional information, so as to be conveniently viewed by the staff. For candidate alarm videos that are not alarm videos, the candidate alarm videos may be discarded, and the positions, speeds, and behavior occurrence times of vehicles corresponding to the candidate alarm videos may be discarded.

As shown in fig. 5, the warning platform may also be independent of the detection server, and the system for monitoring the attention of the driver further includes the warning platform. The detection server may send the determined alarm video, the distraction behavior presentation picture, and the alarm additional information to the alarm platform. And for the candidate alarm video which is not the alarm video, the candidate alarm video is not sent to the alarm platform, and the candidate alarm video and the position, the speed and the action occurrence time of the corresponding vehicle can be discarded.

In step 305, the detection server sends an alarm message to the in-vehicle device.

In implementation, because the detection server has strong computing power and high detection accuracy, if the detection server still detects the second type of distraction behavior, the detection server can send an alarm message to the vehicle-mounted device.

If the detection server does not detect the second type of distraction behavior, the detection server may send no alarm message to the in-vehicle device, or send no message to the in-vehicle device.

In step 306, the vehicle-mounted device responds to the alarm message and sends out an alarm signal.

In implementation, the vehicle-mounted equipment receives the alarm message sent by the detection server and sends out an alarm signal in response to the alarm message.

If the vehicle-mounted equipment receives the non-alarm message sent by the detection server, or the vehicle-mounted equipment does not receive the alarm message sent by the detection server within the set time length, the vehicle-mounted equipment does not send out the alarm signal.

According to the technical scheme, the candidate alarm video detected by the vehicle-mounted equipment is set, and whether the candidate alarm video is the alarm video or not can be determined only after the candidate alarm video is detected by the detection server for the second time. Therefore, due to the fact that the detection server has high computing capacity, a complex algorithm with high accuracy can be used, the accuracy of the alarm video determined through the second detection of the detection server is high, and the accuracy of the alarm video is improved.

And the detection server with strong computing power can accurately detect the candidate alarm video containing the second type of distraction behavior sent by the vehicle-mounted equipment, and sends an alarm message to the vehicle-mounted equipment when the detection server detects the second type of distraction behavior again, so that the vehicle-mounted equipment sends an alarm signal. Therefore, the method can accurately detect the second type of distraction behavior with unobvious characteristics and high detection difficulty.

As shown in FIG. 6, a flow chart of another method of driver attention monitoring is provided that generally illustrates the detection of a first type of distraction behavior and a second type of distraction behavior, as follows.

(1) Algorithm configuration

According to training and researching of a detection algorithm based on a convolutional neural network, two sets of detection algorithms are summarized, wherein one detection algorithm is a lightweight detection algorithm (namely, a first attention detection neural network model) capable of running on vehicle-mounted equipment, and the other detection algorithm is a complex and more accurate complex detection algorithm (namely, a second attention detection neural network model) capable of running on a detection server.

And configuring a lightweight detection algorithm on the vehicle-mounted equipment, and configuring a complex detection algorithm on the detection server.

(2) Practical application

In the actual use process, the vehicle-mounted equipment can be connected with a vehicle-mounted infrared camera, and the camera is aligned with a vehicle driver and carries out video acquisition on the behavior of the driver. The vehicle-mounted equipment acquires a driver behavior video from the camera and carries out first detection on the driver behavior video.

When the in-vehicle device detects the first type of distraction behavior:

the vehicle-mounted equipment immediately sends out an alarm signal (for example, an acousto-optic alarm signal), collects alarm information and determines candidate alarm videos, wherein the alarm information can comprise related information such as behavior occurrence time, vehicle position and vehicle speed, and the candidate alarm videos can be videos which are captured in the driving behavior videos and are 12s in front of and behind the behavior occurrence time. And then, sending the alarm information and the candidate alarm video to a detection server for further screening through a wireless sending module.

And the detection server performs second detection on the candidate alarm video, and if the detection server also detects the first type of distraction behavior, the candidate alarm video is determined to be the alarm video, and a target image frame is captured to be used as a distraction behavior display picture. And finally, integrating the target image frame, the alarm video and the alarm information into alarm data and sending the alarm data to an alarm platform.

The alarm platform receives alarm data for viewing by an administrator.

When a second type of distraction behavior is detected:

since the detection difficulty of the second type of distraction behavior is high and usually needs to last for a while, the vehicle-mounted device does not immediately send out an alarm signal, but collects alarm information and determines candidate alarm videos, and sends the alarm information and the candidate alarm videos to the detection server for further screening.

And the detection server carries out second detection on the candidate alarm video, if the detection server also detects the second type of distraction behavior, the candidate alarm video is determined to be the alarm video, an alarm message is sent to the vehicle-mounted equipment, and the vehicle-mounted equipment sends an alarm signal after receiving the alarm message. And the detection server captures the target image frame as a display picture of the distraction behavior. And integrating the target image frame, the alarm video and the alarm message into alarm data and sending the alarm data to an alarm platform.

The alarm platform receives alarm data for viewing by an administrator.

Based on the same technical concept, the embodiment of the application also provides a system for monitoring the attention of the driver, which comprises an on-board device and a detection server, wherein,

the vehicle-mounted equipment is used for carrying out first detection on the driver behavior video, determining candidate alarm videos containing the detected distractive behaviors in the driver behavior video if the distractive behaviors are detected in the driver behavior video, and sending the candidate alarm videos to the detection server;

and the detection server is used for carrying out second detection on the candidate alarm video, and if the attention dispersion behavior is detected in the candidate alarm video, determining the candidate alarm video as the alarm video.

In one possible implementation, the false detection rate of the first detection is higher than the false detection rate of the second detection.

In one possible implementation, the distraction behavior includes a first type of distraction behavior and a second type of distraction behavior, the first type of distraction behavior has a lower detection difficulty than the second type of distraction behavior, and the risk coefficient of the first type of distraction behavior is higher than the risk coefficient of the second type of distraction behavior.

In one possible implementation, the vehicle-mounted device is further configured to issue an alarm signal if the first type of distraction behavior is detected in the driver behavior video.

In a possible implementation manner, the detection server is further configured to send an alarm message to the vehicle-mounted device if the second type of distraction behavior is detected in the candidate alarm video;

and the vehicle-mounted equipment is also used for responding to the alarm message and sending out an alarm signal.

In one possible implementation, the first type of distraction includes one or more of yawning, closing eyes, making a call, smoking, not wearing a safety belt, leaving a driver's seat, blocking a camera, disengaging a steering wheel with both hands, and disengaging a steering wheel with one hand;

the second category of distractive activities includes one or more of eating, chatting, dance, and unworn work clothes.

In one possible implementation manner, the vehicle-mounted equipment is further used for acquiring the position, the vehicle speed and the action occurrence time of the vehicle and sending the position, the vehicle speed and the action occurrence time to the detection server if the distraction action is detected in the action video of the driver;

and the detection server is also used for determining the position, the vehicle speed and the action occurrence time as alarm additional information if the distraction action is detected in the candidate alarm video.

In a possible implementation manner, the detection server is further configured to, if the distraction behavior is detected in the candidate alarm video, capture a target image frame in the candidate alarm video, and determine that the target image frame is a distraction behavior display picture.

In a possible implementation manner, the system further comprises an alarm platform and a detection server, and the detection server is further used for sending an alarm video, alarm additional information and a display picture of the distraction behavior to the alarm platform.

With regard to the system in the above-described embodiment, the specific manner in which each device performs the operations has been described in detail in the embodiment related to the method, and will not be elaborated upon here.

It should be noted that: the driver attention monitoring system provided by the above embodiment and the driver attention monitoring method embodiment belong to the same concept, and the specific implementation process thereof is detailed in the method embodiment and is not described herein again.

The embodiment of the application further provides the vehicle-mounted device, which comprises a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to realize the method on the vehicle-mounted device side in the method for monitoring the attention of the driver.

The embodiment of the application also provides a detection server, which comprises a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to realize the method for detecting the server side in the method for monitoring the attention of the driver.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.