Multi-channel cooperative psychophysiological active sensing method and service robot
阅读说明：本技术 多通道协同的心理生理主动感知方法及服务机器人 (Multi-channel cooperative psychophysiological active sensing method and service robot ) 是由 李莹辉 丁帅 杨成佳 李霄剑 曲丽娜 杨善林 张园园 严钰 于 2019-09-02 设计创作，主要内容包括：本申请提供一种多通道协同的心理生理主动感知方法及服务机器人,此机器人通过右眼可见光摄像头采集部件获取监测目标个体的可见光视频,通过左眼红外仪采集部件拍摄监测目标个体的红外热图视频,通过左侧接触式指夹仪器获取监测目标个体的皮电、脉搏、血氧等生理数据；机器人的口袋分析操作台用于分析挖掘上述多通道数据对应的深层特征信息及相关权重,用于确定监测目标个体的心理生理特征值。此机器人具有红外智能捕获对象功能、远程服务功能,可按照用户要求分析得到监测目标个体的心理生理特征值。上述多通道协同的心理生理主动感知服务机器人便于实时操作,采用人机交互的方式提高了心理状态监测效率,自动化、智能化程度大大提高。(The application provides a multi-channel collaborative psycho-physiological active perception method and a service robot, the robot acquires a visible light video of a monitored target individual through a right-eye visible light camera acquisition component, shoots an infrared chart video of the monitored target individual through a left-eye infrared instrument acquisition component, and acquires physiological data of the monitored target individual, such as skin electricity, pulse, blood oxygen and the like through a left-side contact type finger clamp instrument; and the pocket analysis operation console of the robot is used for analyzing and mining deep characteristic information and related weight corresponding to the multi-channel data and determining the psychophysiological characteristic value of the monitoring target individual. The robot has the functions of infrared intelligent object capturing and remote service, and can analyze and obtain psychophysical characteristic values of monitoring target individuals according to user requirements. The multi-channel collaborative psychophysiological active perception service robot is convenient to operate in real time, the psychological state monitoring efficiency is improved in a man-machine interaction mode, and the automation and intelligent degrees are greatly improved.)
1. A multi-channel cooperative psychophysiological active perception method is characterized by comprising the following steps:
acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual;
extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features;
performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features;
extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features;
extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics;
and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
2. The method of claim 1, further comprising:
sensing an object with a distance smaller than a preset distance from a service robot, and when the distance between a person and the service robot is smaller than the preset distance, executing the steps of acquiring a visible light image of a target individual, acquiring an infrared heat map of the target individual, acquiring audio information of the target individual, acquiring physiological information of the target individual, processing the acquired information, and determining a psycho-physiological characteristic value of the target individual; wherein the service robot is an execution subject for executing the method.
3. The method of claim 1, further comprising:
acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
4. The method of claim 1, further comprising, before extracting the first image feature in the visible light image and generating a visible light image feature vector based on the extracted image feature, the steps of:
performing point operation processing, filtering processing and global optimization processing on the visible light image
Preprocessing the infrared thermal image by utilizing a gray level transformation method and a denoising method of a wavelet packet threshold;
and performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
The extracting a first image feature in the visible light image comprises:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
5. The method of claim 1, wherein the performing the dimension reduction on the infrared heat map to extract the second image feature of the information obtained by the dimension reduction comprises:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
6. The method of claim 1, wherein the extracting audio features from the audio information and determining emotional fluctuation features matching the audio features comprises:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
7. The method of claim 1, wherein determining psychophysiological characteristic values of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the mood wave characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the mood wave characteristic vector and the weight of the physiological time sequence characteristic vector comprises:
and determining the psycho-physiological characteristic value of the target individual by utilizing a convolutional neural network based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
8. The method of claim 1, wherein the first image features comprise head pose features of the target individual, micro-expression features of the target individual, eye movement trajectory features of the target individual;
the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual;
the audio features comprise sound wave frequency features and sound intensity features of the target individual;
the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood oxygen characteristics of the target individual.
9. The method of claim 1, wherein determining psychophysiological characteristic values of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the mood wave characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the mood wave characteristic vector and the weight of the physiological time sequence characteristic vector comprises:
determining a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector and a physiological time sequence characteristic vector corresponding to each preset period in a plurality of preset periods;
and determining the psycho-physiological characteristic value of the target individual based on the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector, the weight of the physiological time sequence characteristic vector, the visible light image characteristic vector corresponding to each preset period, the infrared heat map characteristic vector corresponding to each preset period, the emotion fluctuation characteristic vector corresponding to each preset period and the physiological time sequence characteristic vector corresponding to each preset period.
10. A multi-channel collaborative psychophysiological active perception service robot, comprising: the device comprises a first eye acquisition component, a second eye acquisition component, a mouth acquisition component, a pocket analysis operating platform, a physiological information acquisition component, a display screen, a power supply component, a storage component, an analysis processing component, a human-computer interaction component and a remote service component;
the first eye acquisition component is used for acquiring a visible light image of a target individual;
the second eye acquisition component is used for acquiring an infrared heat map of the target individual;
the mouth acquisition component is used for acquiring audio information of a target individual;
the physiological information acquisition component is used for acquiring physiological information of a target individual;
the analysis processing component is used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psycho-physiological characteristic value of a target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector;
the display screen is used for displaying a visible light image of the target individual, an infrared heat map of the target individual, audio information of the target individual, physiological information of the target individual and a psychophysiological characteristic value of the target individual;
the storage component is used for storing visible light images of the target individuals, infrared heat maps of the target individuals, audio information of the target individuals, physiological information of the target individuals and psychophysiological characteristic values of the target individuals;
the power supply component is used for supplying power to the first eye acquisition component, the second eye acquisition component, the mouth acquisition component, the physiological information acquisition component, the display screen and the storage component;
the human-computer interaction component is used for sensing an object with a distance smaller than a preset distance from a service robot, when the distance between a person and the service robot is smaller than the preset distance, an object is captured intelligently through infrared rays, the first eye acquisition component is controlled to acquire a visible light image of a target individual, the second eye acquisition component is controlled to acquire an infrared heat image of the target individual, the mouth acquisition component is controlled to acquire audio information of the target individual, the physiological information acquisition component is used to acquire physiological information of the target individual, and the analysis processing component processes the acquired information to determine a psychophysiological characteristic value of the target individual;
the remote service component is used for acquiring a visible light image, an infrared heat map, audio information and physiological information of a target individual of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psycho-physiological characteristic value of a target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector;
the pocket analysis operation console is used for controlling the content displayed by the display screen, controlling the working state of the service robot, and setting the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector.
Technical Field
The application relates to the field of information and data processing, in particular to a multi-channel collaborative psychophysiological active perception method and a service robot.
Background
Mental state is one of the basic forms of mental activities, and refers to the complete features of mental activities in a certain period of time, such as attention, fatigue, tension, relaxation, worry, joy, etc. It has the characteristics of both psychological process and individual psychological characteristics, and has both temporary and stable properties. The method is a medium link for connecting psychological process and individual psychological characteristics, and forms all backgrounds for developing psychological activities. Physiological characteristics, such as skin power, blood oxygen, etc., can to some extent sense an individual's mental state characteristics. The psychophysiological state characteristics of the individual can reflect the current state of the individual and help to judge whether the individual can do behaviors which harm the individual or threaten the safety of others, so that the psychophysiological state characteristics of the individual need to be monitored.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the application provides a multi-channel collaborative psychophysiological active perception method and a service robot, solves the problems that emotion information types for psychophysiological state analysis acquired in the prior art are single, insufficient and incapable of being analyzed in real time, and has the defects of complex steps, large calculated amount, poor efficiency, low automation degree and low accuracy of analysis results in the psychophysiological state analysis process.
(II) technical scheme
In order to achieve the above purpose, the present application is implemented by the following technical solutions:
in a first aspect, the present application provides a multi-channel collaborative psychophysiological active perception method, including:
acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual;
extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features;
performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features;
extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features;
extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics;
and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
In one possible embodiment, the method further comprises:
sensing an object with a distance smaller than a preset distance from a service robot, and when the distance between a person and the service robot is smaller than the preset distance, executing the steps of acquiring a visible light image of a target individual, acquiring an infrared heat map of the target individual, acquiring audio information of the target individual, acquiring physiological information of the target individual, processing the acquired information, and determining a psycho-physiological characteristic value of the target individual; wherein the service robot is an execution subject for executing the method.
In one possible embodiment, the method further comprises:
acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
In a possible implementation manner, before extracting the first image feature in the visible light image and generating the visible light image feature vector based on the extracted image feature, the method further includes the following steps:
performing point operation processing, filtering processing and global optimization processing on the visible light image
Preprocessing the infrared thermal image by utilizing a gray level transformation method and a denoising method of a wavelet packet threshold;
and performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
In one possible embodiment, the extracting the first image feature in the visible light image includes:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
In a possible implementation manner, the performing the dimension reduction processing on the infrared heat map and extracting the second image feature of the information obtained by the dimension reduction processing includes:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
In a possible implementation manner, the extracting audio features in the audio information and determining emotional fluctuation features matched with the audio features includes:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
In a possible embodiment, the extracting time-frequency features from the physiological information includes:
and performing time-frequency analysis processing on the physiological information based on Fourier transform and depth LSTM to obtain the video characteristics.
In one possible embodiment, the determining psychophysiological feature values of the target individual based on the visible light image feature vector, the infrared heat map feature vector, the emotion fluctuation feature vector and the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector includes:
and determining the psycho-physiological characteristic value of the target individual by utilizing a convolutional neural network based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
In one possible implementation, the first image feature includes a head posture feature of the target individual, a micro-expression feature of the target individual, and an eye movement track feature of the target individual;
the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual;
the audio features comprise sound wave frequency features and sound intensity features of the target individual;
the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood oxygen characteristics of the target individual.
In one possible embodiment, the determining psychophysiological feature values of the target individual based on the visible light image feature vector, the infrared heat map feature vector, the emotion fluctuation feature vector and the physiological time sequence feature vector, the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector includes:
determining a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector and a physiological time sequence characteristic vector corresponding to each preset period in a plurality of preset periods;
and determining the psycho-physiological characteristic value of the target individual based on the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector, the weight of the physiological time sequence characteristic vector, the visible light image characteristic vector corresponding to each preset period, the infrared heat map characteristic vector corresponding to each preset period, the emotion fluctuation characteristic vector corresponding to each preset period and the physiological time sequence characteristic vector corresponding to each preset period.
In a second aspect, the embodiment of the present application provides a multi-channel collaborative psychophysiological active perception service robot, including:
the device comprises a first eye acquisition component, a second eye acquisition component, a mouth acquisition component, a pocket analysis operating platform, a physiological information acquisition component, a display screen, a power supply component, a storage component, an analysis processing component, a human-computer interaction component and a remote service component;
the first eye acquisition component is used for acquiring a visible light image of a target individual;
the second eye acquisition component is used for acquiring an infrared heat map of the target individual;
the mouth acquisition component is used for acquiring audio information of a target individual;
the physiological information acquisition component is used for acquiring physiological information of a target individual;
the analysis processing component is used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psycho-physiological characteristic value of a target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector;
the display screen is used for displaying a visible light image of the target individual, an infrared heat map of the target individual, audio information of the target individual, physiological information of the target individual and a psychophysiological characteristic value of the target individual;
the storage component is used for storing visible light images of the target individuals, infrared heat maps of the target individuals, audio information of the target individuals, physiological information of the target individuals and psychophysiological characteristic values of the target individuals;
the power supply component is used for supplying power to the first eye acquisition component, the second eye acquisition component, the mouth acquisition component, the physiological information acquisition component, the display screen and the storage component;
the human-computer interaction component is used for sensing an object with a distance smaller than a preset distance from a service robot, when the distance between a person and the service robot is smaller than the preset distance, an object is captured intelligently through infrared rays, the first eye acquisition component is controlled to acquire a visible light image of a target individual, the second eye acquisition component is controlled to acquire an infrared heat image of the target individual, the mouth acquisition component is controlled to acquire audio information of the target individual, the physiological information acquisition component is used to acquire physiological information of the target individual, and the analysis processing component processes the acquired information to determine a psychophysiological characteristic value of the target individual;
the remote service component is used for acquiring a visible light image, an infrared heat map, audio information and physiological information of a target individual of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psycho-physiological characteristic value of a target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector;
the pocket analysis operation console is used for controlling the content displayed by the display screen, controlling the working state of the service robot, and setting the weight of the visible light image feature vector, the weight of the infrared heat map feature vector, the weight of the emotion fluctuation feature vector and the weight of the physiological time sequence feature vector.
(III) advantageous effects
The application provides a multi-channel collaborative psychophysiological active perception method and a service robot. The method has the following beneficial effects:
the method comprises the steps of firstly extracting first image features in a visible light image of a target individual, and generating a visible light image feature vector based on the extracted image features; extracting second image features in the infrared heat map, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and then, determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector. According to the technical scheme, the psychophysiological characteristic information of a target individual can be analyzed by extracting and fusing four types of data including visible light images, infrared heat maps, audio information and physiological information, the defects that the type of emotion information for psychophysiological state analysis acquired in the prior art is single and insufficient are overcome, the accuracy of the psychophysiological characteristic information obtained by analysis is improved, and in addition, when the four types of data are processed, a neural network and the like are utilized, the calculation steps can be effectively reduced, the calculation amount is reduced, the calculation efficiency is poor, and the automation degree is improved;
in addition, the service robot acquires visible light videos of the monitoring target individual through the right-eye visible light camera acquisition component, shoots infrared heat image videos of the monitoring target individual through the left-eye infrared instrument acquisition component, and acquires physiological data of the monitoring target individual, such as skin electricity, pulse, blood oxygen and the like through the left-side contact type finger clamp instrument; and the pocket analysis operation console of the service robot is used for analyzing and mining deep feature information and related weight corresponding to the multi-channel data and determining the psycho-physiological feature value of the monitoring target individual. The service robot has the functions of infrared intelligent object capturing and remote service, and can analyze and obtain psychophysical characteristic values of monitored target individuals according to user requirements. The multi-channel collaborative psychophysiological active perception service robot is convenient to operate in real time, the psychological state monitoring efficiency is improved in a man-machine interaction mode, and the automation and intelligent degrees are greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart schematically illustrating a multi-channel collaborative psycho-physiological active perception method according to an embodiment of the present application;
fig. 2 schematically shows a structural diagram of a multi-channel collaborative psychophysiological active perception service robot according to an embodiment of the application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Aiming at the defects of single and insufficient data type, complicated steps in the process of analyzing the psychophysiological state, large calculation amount, poor efficiency, low automation degree and low accuracy of the analysis result in the technical scheme of analyzing the psychophysiological characteristics of individuals in the prior art, the application provides the multi-channel collaborative psychophysiological active sensing method, the method can utilize a neural network model to process the acquired visible light image, infrared chart, audio information and physiological information and four types of data, can overcome the defect of single data type in the prior art for calculating the psychophysiological state, effectively improves the calculation accuracy, reduces the calculation steps, reduces the calculation amount, improves the calculation efficiency and improves the automation degree.
The following describes the multi-channel cooperative psychophysiological active perception method of the present application in detail. The method is performed by a server capable of executing a program.
As shown in fig. 1, the multi-channel cooperative psychophysiological active perception method of the present application includes the following steps:
s110, acquiring a visible light image of the target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual.
The target individual is an individual to be analyzed for psychophysiological state characteristic values. The visible light camera can be used for acquiring a visible light image of the target individual, the infrared camera can be used for acquiring an infrared heat image of the target individual, and the microphone can be used for acquiring audio information of the target individual. The visible light image may be used to determine information such as a head pose of the target individual, the infrared heat map may be used to determine information such as a face temperature of the target individual, and the audio information may be used to determine information such as a sound intensity of the target individual.
The finger-clip type instrument can be used for monitoring the physiological information of the target individual; the physiological information includes a target individual's skin electrical signal, blood oxygen signal, pulse signal, etc.
And S120, extracting first image features in the visible light image, and generating a visible light image feature vector based on the extracted image features.
The visible light image can reflect the psycho-physiological characteristics of the target individual, for example, the visible light image includes distortion of five sense organs, and then the target individual may suffer from the psycho-physiological characteristics. Therefore, in order to improve the accuracy of evaluation when evaluating the psychophysiological characteristics of the target individual, it is necessary to acquire a visible light image of the target individual and determine the first image characteristics based on the visible light image.
Here, extracting the first image feature in the visible light image may include:
and extracting image features of a plurality of preset face feature points and a plurality of preset face motion areas in the visible light image to obtain the first image features.
The plurality of facial feature points may be 68 feature points of a preset human face. The above-mentioned facial motion region may be a specific region of the segmented human face, for example, a forehead region, a cheek region, or the like.
S130, performing dimension reduction processing on the infrared heat map, extracting second image features of the information obtained by the dimension reduction processing, and determining infrared heat map feature vectors based on the extracted second image features.
The infrared heat map may reflect psychophysiological characteristics of the target individual, for example, if the temperature of the face of the target individual is too high in the infrared heat map, the target individual may be excited. Therefore, in order to improve the accuracy of evaluation when evaluating psychophysiological characteristics of a target individual, it is necessary to acquire an infrared heat map of the target individual and determine second image characteristics based on the infrared heat map.
Here, the performing dimension reduction processing on the infrared heat map, and extracting a second image feature of information obtained by the dimension reduction processing includes:
and performing dimension reduction processing on the infrared heat map by using a direct example checking method and a principal component analysis method based on the HSV color model, and extracting image features reduced into processed information to obtain the second image features.
S140, extracting the audio features in the audio information, determining the emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features.
The audio features in the audio information correspond to psychophysiological features of the target individual, for example, the audio information includes audio features with high sound intensity, and the psychophysiological features include features of heart rate acceleration and anger. Therefore, in evaluating the psychophysiological characteristics of the target individual, in order to improve the accuracy of the evaluation, it is necessary to acquire audio information of the target individual and determine the mood swing characteristics based on the audio information.
Here, extracting the audio features in the audio information and determining the emotional fluctuation features matched with the audio features includes:
extracting audio features in the audio information, and searching emotion fluctuation features matched with the audio features from a voice tone library; the voice tone library stores a plurality of voice features and emotion fluctuation features corresponding to each voice feature.
The voice intonation library is preset, and includes voice features and emotion fluctuation features corresponding to each voice feature, and based on the voice features extracted in
S150, extracting the time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics.
The physiological information can directly reflect the physiological change of the target individual, and the physiological change of the target individual is associated with and affects the psychological state of the target individual, so when evaluating the psychophysiological characteristics of the target individual, the physiological information of the target individual needs to be acquired in order to improve the accuracy of evaluation.
Here, extracting the time-frequency features in the physiological information includes:
and performing time-frequency analysis processing on the physiological information based on Fourier transform and depth LSTM to obtain the video characteristics.
And S160, determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector, the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
Here, the psychophysiological characteristic value of the target individual may be specifically determined by using a convolutional neural network based on a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector, and a physiological time series characteristic vector, a weight of the visible light image characteristic vector, a weight of the infrared heat map characteristic vector, a weight of the emotion fluctuation characteristic vector, and a weight of the physiological time series characteristic vector.
The convolutional neural network is obtained by pre-training, can automatically, accurately and quickly determine the psychophysiological characteristic value of a target individual when in use, and has simple and quick steps.
The first image characteristics comprise head posture characteristics of the target individual, micro-expression characteristics of the target individual and eye movement track characteristics of the target individual; the second image characteristics comprise facial temperature characteristics, breathing frequency characteristics and heartbeat frequency characteristics of the target individual; the audio features comprise sound wave frequency features and sound intensity features of the target individual; the time-frequency characteristics comprise the skin electricity characteristics of the target individual, the pulse characteristics of the target individual and the blood oxygen characteristics of the target individual.
In some embodiments, the determining the psychophysiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the mood fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the mood fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector can be further implemented by the following steps:
determining a visible light image characteristic vector, an infrared heat map characteristic vector, an emotion fluctuation characteristic vector and a physiological time sequence characteristic vector corresponding to each preset period in a plurality of preset periods;
and determining the psycho-physiological characteristic value of the target individual based on the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector, the weight of the physiological time sequence characteristic vector, the visible light image characteristic vector corresponding to each preset period, the infrared heat map characteristic vector corresponding to each preset period, the emotion fluctuation characteristic vector corresponding to each preset period and the physiological time sequence characteristic vector corresponding to each preset period.
In the above embodiment, a period with a certain time length is preset, in each period, the visible light image of the target individual, the infrared thermograph of the target individual, the audio information of the target individual, and the physiological information of the target individual are acquired, and based on the acquired data, the visible light image feature vector, the infrared thermograph feature vector, the mood swing feature vector, and the physiological timing feature vector are determined, and then, based on the determined vectors, the psychophysiological feature value of the target individual is determined. Specifically, the psychophysiological characteristic value corresponding to each period may be determined based on each vector corresponding to each period, and then the final psychophysiological characteristic value of the target individual may be determined based on the psychophysiological characteristic value corresponding to each period; or directly determining the final psychophysiological characteristic value of the target individual according to each vector corresponding to all periods.
The above-described embodiment can increase the accuracy of the determined psychophysiological characteristic value by the setting of the period.
In some embodiments, before extracting the first image feature in the visible light image and generating the visible light image feature vector based on the extracted image feature, the method may further include the following steps:
the method comprises the following steps of firstly, carrying out point operation processing, filtering processing and global optimization processing on the visible light image.
And secondly, preprocessing the infrared thermal image by utilizing a gray level transformation method and a wavelet packet threshold denoising method.
And step three, performing pre-weighting processing, low-pass filtering processing and framing processing on the audio information.
The step makes the audio information clearer and facilitates later data analysis.
Through the preprocessing of the data, the error data can be eliminated, and the accuracy of the determined psychophysiological characteristic value is improved.
The method integrates and analyzes the collected non-contact and contact data of the user, monitors the psychophysiological characteristics by using artificial intelligent methods such as deep learning and the like, and finally obtains the accurate analysis result of the target individual based on the long-period psychophysiological characteristics.
In addition, the method further comprises: sensing an object with a distance smaller than a preset distance from a service robot, and when the distance between a person and the service robot is smaller than the preset distance, executing the steps of acquiring a visible light image of a target individual, acquiring an infrared heat map of the target individual, acquiring audio information of the target individual, acquiring physiological information of the target individual, processing the acquired information, and determining a psycho-physiological characteristic value of the target individual; wherein the service robot is an execution subject for executing the method.
In addition, the method further comprises:
acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
Corresponding to the method in the embodiment, the application also discloses a service robot, the service robot can communicate with a user, collect visible light images, infrared heat images, audio information and physiological information, perform real-time analysis operation on the acquired information, and determine the psychophysiological characteristic value of a target individual, and the service robot monitors the psychophysiological state in a multi-channel intelligent cooperation mode, so that the service robot is intelligent, convenient and fast.
Specifically, as shown in fig. 2, the service robot includes: a first
The
the
the
the physiological
the analysis processing component is used for extracting first image features in the visible light image and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; determining a psycho-physiological characteristic value of a target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector;
the
the storage component is used for storing visible light images of the target individuals, infrared heat maps of the target individuals, audio information of the target individuals, physiological information of the target individuals and psychophysiological characteristic values of the target individuals;
the
The service robot further includes a storage device charging
In addition, the service robot further comprises a human-computer interaction component, which is used for sensing an object with a distance smaller than a preset distance from the service robot, and when the distance between a person and the service robot is smaller than the preset distance, the first eye acquisition component is controlled to acquire a visible light image of a target individual, the second eye acquisition component is controlled to acquire an infrared heat image of the target individual, the mouth acquisition component is controlled to acquire audio information of the target individual, the physiological information acquisition component is controlled to acquire physiological information of the target individual, and the analysis processing component processes the acquired information to determine a psychophysiological characteristic value of the target individual.
In addition, the service robot also comprises a remote service component, a remote control component and a remote control component, wherein the remote service component is used for acquiring a visible light image of a target individual, an infrared heat map of the target individual, audio information of the target individual and physiological information of the target individual through remote communication; extracting first image features in the visible light image, and generating visible light image feature vectors based on the extracted image features; performing dimension reduction processing on the infrared heat map, extracting second image features of information obtained by the dimension reduction processing, and determining an infrared heat map feature vector based on the extracted second image features; extracting audio features in the audio information, determining emotion fluctuation features matched with the audio features, and generating emotion fluctuation feature vectors based on the emotion fluctuation features; extracting time-frequency characteristics in the physiological information, and generating a physiological time sequence characteristic vector based on the extracted time-frequency characteristics; and determining the psycho-physiological characteristic value of the target individual based on the visible light image characteristic vector, the infrared heat map characteristic vector, the emotion fluctuation characteristic vector and the physiological time sequence characteristic vector, the weight of the visible light image characteristic vector, the weight of the infrared heat map characteristic vector, the weight of the emotion fluctuation characteristic vector and the weight of the physiological time sequence characteristic vector.
The remote service component acquires the information for analysis by using a camera, a microphone and the like of the remote terminal, and processes the acquired information to determine the psychophysiological characteristic value of the target individual.
The steps executed by the service robot are the same as the method, and are not described again here.
The service robot integrates the contact information acquisition component and the non-contact information acquisition component, can monitor physiological changes (such as galvanic pulse blood oxygen) and psychological changes of a target individual at the same time, and is convenient to operate and high in efficiency. All parts of the service robot are connected into a whole and realize the movable and expandable functions. The cross-media information fusion function can be realized, the intelligent man-machine interaction function with the target individual is realized, and the requirements of the target individual and a user are conveniently met.
In the embodiment, the contact type information acquisition component acquires the physiological signal of the target individual through the finger-clip type galvanic pulse oximeter, and the data such as the heart rate and the respiratory rate of the target individual are monitored through the non-contact type information acquisition component for calibration, so that the analysis result is more accurate. The data acquisition is simple and easy to implement, and the equipment is simple to operate and convenient to use.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.