阅读说明：本技术 基于多模态信息的服刑人员心理健康状态评估方法及系统 (Inmate's psychological health states appraisal procedure and system based on multi-modal information ) 是由 刘治 姚佳 于 2019-08-23 设计创作，主要内容包括：本公开公开了基于多模态信息的服刑人员心理健康状态评估方法及系统,获取已改造好的服刑人员和待测服刑人员,在虚拟现实情景体验后的生理信号、面部表情图像和语音信号,从获取的信号中提取生理信号特征、面部表情图像特征和语音信号特征；将已改造好的服刑人员的生理信号特征、面部表情图像特征和语音信号特征输入到预先训练好的神经网络模型中,输出已改造好的服刑人员的心理状态评估向量；将待测服刑人员的生理信号特征、面部表情特征和语音信号特征输入到预先训练好的神经网络模型中,输出待测服刑人员的心理状态评估向量；计算待测服刑人员与已改造好的服刑人员的心理状态评估向量的距离；根据距离评估服刑人员心理健康状态。(The present disclosure discloses inmate's psychological health states appraisal procedures and system based on multi-modal information, obtain the inmate and inmate to be measured being transformed, physiological signal, facial expression image and voice signal after virtual reality Scenario experiences extract physiological signal feature, facial expression image feature and phonic signal character from the signal of acquisition；The physiological signal feature, facial expression image feature and phonic signal character of the inmate being transformed are input in preparatory trained neural network model, the psychological condition assessment vector for the inmate being transformed is exported；Physiological signal feature, facial expression feature and the phonic signal character of inmate to be measured are input in preparatory trained neural network model, the psychological condition assessment vector of inmate to be measured is exported；Inmate to be measured is calculated at a distance from the psychological condition assessment vector for the inmate being transformed；Inmate's psychological health states are assessed according to distance.)

1. inmate's psychological health states appraisal procedure based on multi-modal information, characterized in that the method is not used in disease The diagnosis of disease；The method, comprising:

Obtain the inmate being transformed, physiological signal, facial expression image and voice after virtual reality Scenario experiences Signal extracts physiological signal feature, facial expression image feature and phonic signal character from the signal of acquisition；

Inmate to be measured is obtained, physiological signal, facial expression image and voice signal after virtual reality Scenario experiences, from Physiological signal feature, facial expression feature and phonic signal character are extracted in the signal of acquisition；

The physiological signal feature, facial expression image feature and phonic signal character of the inmate being transformed are input to pre- First in trained neural network model, the psychological condition assessment vector for the inmate being transformed is exported；

Physiological signal feature, facial expression feature and the phonic signal character of inmate to be measured are input to trained in advance In neural network model, the psychological condition assessment vector of inmate to be measured is exported；

Inmate to be measured is calculated at a distance from the psychological condition assessment vector for the inmate being transformed；

Inmate's psychological health states are assessed according to distance.

2. the method as described in claim 1, characterized in that the training step of trained neural network model includes: in advance

Construct neural network model；

Obtain physiological signal of the inmate after virtual reality Scenario experiences as training sample, facial expression image and Voice signal；

Physiological characteristic, facial expression feature and phonetic feature are extracted from the signal of acquisition；For the people that each serves a sentence in training sample Physiological characteristic, facial expression feature and the phonetic feature mark psychological condition of member assesses vector；

Vector is assessed using physiological characteristic, facial expression feature, phonetic feature and the psychological condition marked is extracted, to nerve net Network model is trained；Obtain preparatory trained neural network model.

3. method according to claim 2, characterized in that the psychological condition assesses vector, be 12 row * 1 column to Amount, the element value that every row includes are the quantized values of every kind of affective state, and the quantized value is integer, the value model of the quantized value Enclose is -4, -3, -2, -1,0,1,2,3,4；12 row elements just include 12 kinds of affective states, 12 kinds of affective states, comprising: anger Anger, it is awake, controlled, friendly, tranquil, dominate, it is painful, interested, humble, excited, overcautious and powerful.

4. method according to claim 2, characterized in that for the physiological characteristic of each inmate, face in training sample Expressive features and phonetic feature mark psychological condition assess vector, are labeled based on PAD Emotion identification scale:

To the same subject after the same virtual reality Scenario experiences, the three dimensional mood identification scale of subject is acquired；One N times are acquired altogether；

To the same subject under the same virtual reality scene, the value of the three dimensional mood identification scale of n times acquisition is carried out It averages processing, obtained psychological condition assessment vector is the heart of the current subject after the same virtual reality Scenario experiences Manage status assessment vector；

To the same subject, replaces next virtual reality scenario and experienced, obtain the heart of next virtual reality scenario Manage status assessment vector；And then the same subject is obtained, the psychological condition under different virtual reality scenarios assesses vector；

Then next subject is replaced, and so on, different subjects can be obtained under the experience of different virtual reality scenarios Psychological condition assess vector；

Then, vector is assessed using psychological condition of the obtained different subjects under the experience of different virtual reality scenarios, to not Physiological characteristic, facial expression feature and the phonetic feature extracted under the experience of different virtual reality scenarios with subject is marked Note.

5. the method as described in claim 1, characterized in that the acquisition modes of physiological signal, comprising:

Pass through the photoelectricity folder acquisition blood volume beat signals or heart rate signal being arranged on subject's thumb；

The electrocardiosignal that electrode by the way that subject's wrist and ankle is arranged in acquires；

The skin conductance signal acquired by the conductivity sensor being arranged on finger；

Pass through the electromyogram for the subject that the electrode being arranged on forearm acquires；

The breath signal that sensor by the way that subject's thorax position is arranged in acquires；Or,

The EEG signals acquired by brain electrical testing electrode.

6. the method as described in claim 1, characterized in that the physiological signal feature refers to:

Blood volume beat signals feature, comprising: the side of the mean value of blood volume beat signals amplitude, blood volume beat signals amplitude Difference, the maximum value of blood volume beat signals amplitude, the minimum value of blood volume beat signals amplitude or blood volume beat signals amplitude Intermediate value；

Heart rate signal feature, comprising: the mean value of heart rate signal amplitude, the variance of heart rate signal amplitude, heart rate signal amplitude are most The intermediate value of big value, the minimum value of heart rate signal amplitude or heart rate signal amplitude；

Electrocardiosignal feature is that 0-10Hz frequency range in ECG signal frequency spectrum is divided into 8 nonoverlapping sub-bands, obtains It takes the Fourier transformation mean value of each sub-band as feature, while 8 sub-bands is merged into two sub-bands, 1-3 frequency Tape merge is low-frequency band, and 4-8 sub-band merges into high frequency band, calculates two sub-bands and is averaged the ratio conduct of Fourier transformation value Feature；

Skin conductance signal feature, comprising: the mean value of skin conductance signal amplitude, the variance of skin conductance signal amplitude, skin The adjacent difference of the first-order difference mean value of conductance signal amplitude, the root mean square of skin conductance signal amplitude or skin conductance signal amplitude Absolute value mean value；

Electromyogram signal feature, comprising: electromyogram signal power spectral density；

Breath signal feature, comprising: on the power spectrum of breath signal choose 0-0.1Hz, 0.1-0.2Hz, 0.2-0.3Hz and Average power spectral density in tetra- frequency bands of 0.3-0.4Hz；

EEG signals feature, comprising: EEG signals power spectral density, i.e. signal power in per unit band；

Alternatively,

The acquisition modes of the facial expression feature are:

Facial expression image of the inmate after virtual reality Scenario experiences is acquired by camera；To facial expression image into Row image converts EDS extended data set, then carries out feature extraction, obtains the textural characteristics of image；

Alternatively,

The acquisition modes of the phonic signal character are:

Voice signal of the inmate after virtual reality Scenario experiences is acquired by microphone；Voice signal is divided into several Frame, carries out Fast Fourier Transform (FFT) to each frame voice signal, obtains frequency domain character, carries out feature extraction to voice signal, mentions Take tonality feature or Sound Speed Characteristics；

Alternatively,

Vector is assessed using physiological characteristic, facial expression feature, phonetic feature and the psychological condition marked is extracted, to nerve net Network model is trained；Obtain the specific steps of preparatory trained neural network model are as follows:

Fusion Features are carried out to physiological characteristic, facial expression feature and phonetic feature, fused feature is input to neural mould In type, output inmate quantify psychological condition assessment vector predicted value, calculate inmate quantify psychological condition assess to The difference between psychological condition assessment vector that the predicted value of amount and inmate have marked, in difference minimum, deconditioning, Obtain trained prediction model.

7. inmate's psychological health states assessment system based on multi-modal information, characterized in that include:

Good inmate's data acquisition module is transformed: the inmate being transformed is obtained, after virtual reality Scenario experiences Physiological signal, facial expression image and voice signal；

Good inmate's data characteristics extraction module is transformed: extracting physiological signal feature, facial expression from the signal of acquisition Characteristics of image and phonic signal character；

Inmate's data acquisition module to be measured: obtaining inmate to be measured, physiological signal after virtual reality Scenario experiences, Facial expression image and voice signal；

Inmate's data characteristics extraction module to be measured: physiological signal feature, facial expression feature are extracted from the signal of acquisition And phonic signal character；

First psychological condition assesses vector output module: by physiological signal feature, the facial expression of the inmate being transformed Characteristics of image and phonic signal character are input in preparatory trained neural network model, export the inmate being transformed Psychological condition assess vector；

Second psychological condition assess vector output module: by the physiological signal feature of inmate to be measured, facial expression feature and Phonic signal character is input in preparatory trained neural network model, export inmate to be measured psychological condition assess to Amount；

Inmate's psychological health states evaluation module to be measured: the heart of inmate to be measured with the inmate being transformed are calculated Manage the Euclidean distance of status assessment vector；Inmate's psychological health states to be measured are assessed according to distance.

8. a kind of electronic equipment, characterized in that on a memory and on a processor including memory and processor and storage The computer instruction of operation when the computer instruction is run by processor, is completed described in any one of claim 1-6 method Step.

9. a kind of computer readable storage medium, characterized in that for storing computer instruction, the computer instruction is processed When device executes, step described in any one of claim 1-6 method is completed.

10. inmate's psychological health states assessment system based on multi-modal information, characterized in that include:

Physiological parameter acquisition device, image collecting device, voice acquisition device and electronic equipment according to any one of claims 8；

The data of acquisition are transferred to electronic equipment by physiological parameter acquisition device, image collecting device and voice acquisition device；

Electronic equipment according to the collected data assesses the psychological health states of inmate.

Technical field

This disclosure relates to inmate's psychological health states appraisal procedure and system based on multi-modal information.

Background technique

The statement of this part is only to refer to background technique relevant to the disclosure, not necessarily constitutes the prior art.

Mental health of convict refers to make criminal keep mental health during serving a sentence, reduces and avoid that psychological disease occurs The measure and method of disease.Mainly have: (1) optimizing environment of serving a sentence.(2) it helps, criminal association is instructed correctly to adjust with self psychology Section mechanism carries out self psychological regulation in time, to avoid psychological unbalance.(3) psychological consultation and treatment mechanism is established, to production Raw psychological disease and the criminal for encountering serious Psychological setback, psychological pressure, carry out psychological counseling and treatment in time.

Countries in the world prison carries out psychological test: (1) general personality inventory to criminal usually using two amounts table, such as Eisenke Personality Questionnaire, Minnesota Multiphasic personality measurement table, Cartel Sixteen Personality Factor Questionnaire etc., by testing Solve the situations such as moral sense, legal system sense, restraint, the adjusting force in the personality characteristics and personality-formation of criminal.(2) it is exclusively used in examining It surveys structure of criminal mentality situation and the scale of possibility is recommitted in prediction, various countries are with spies such as itself society and politics, economy, culture Point is independent to be developed.

In implementing the present disclosure, following technical problem exists in the prior art in inventor:

Existing psychological condition assessment does not account for this specific group of inmate, moreover, not accounting for benefit yet Physiological signal is acquired with some electronic equipments, a variety of physiological signals are handled, to realize to inmate's mental health shape The rapid evaluation of state and accurate assessment, the prior art depend on marriage counselor, and subjectivity is too strong.

Summary of the invention

In order to solve the deficiencies in the prior art, present disclose provides inmate's mental healths based on multi-modal information State evaluating method and system；The disclosure merges the other modes such as the same expression of physiological signal, voice, passes through artificial mind Intelligent recognition through network realizes more accurate psychologic status assessment, is effectively transformed effect assessment while guidance is served a sentence The transformation of personnel.

In a first aspect, present disclose provides inmate's psychological health states appraisal procedures based on multi-modal information；

Inmate's psychological health states appraisal procedure based on multi-modal information, the method are not used in examining for disease It is disconnected；The method, comprising:

Obtain the inmate that has been transformed, physiological signal, facial expression image after virtual reality Scenario experiences and Voice signal extracts physiological signal feature, facial expression image feature and phonic signal character from the signal of acquisition；

Inmate to be measured is obtained, physiological signal, facial expression image and voice letter after virtual reality Scenario experiences Number, physiological signal feature, facial expression feature and phonic signal character are extracted from the signal of acquisition；

The physiological signal feature, facial expression image feature and phonic signal character of the inmate being transformed is defeated Enter into preparatory trained neural network model, exports the psychological condition assessment vector for the inmate being transformed；

Physiological signal feature, facial expression feature and the phonic signal character of inmate to be measured are input to preparatory instruction In the neural network model perfected, the psychological condition assessment vector of inmate to be measured is exported；

Inmate to be measured is calculated at a distance from the psychological condition assessment vector for the inmate being transformed；

Inmate's psychological health states are assessed according to distance.

Second aspect, the disclosure additionally provide inmate's psychological health states assessment system based on multi-modal information；

Inmate's psychological health states assessment system based on multi-modal information, comprising:

Good inmate's data acquisition module is transformed: the inmate being transformed is obtained, in virtual reality scene body Physiological signal, facial expression image and voice signal after testing；

Good inmate's data characteristics extraction module is transformed: extracting physiological signal feature, face from the signal of acquisition Facial expression image feature and phonic signal character；

Inmate's data acquisition module to be measured: obtaining inmate to be measured, the physiology after virtual reality Scenario experiences Signal, facial expression image and voice signal；

Inmate's data characteristics extraction module to be measured: physiological signal feature, facial expression are extracted from the signal of acquisition Feature and phonic signal character；

First psychological condition assesses vector output module: by the physiological signal feature for the inmate being transformed, face Facial expression image feature and phonic signal character are input in preparatory trained neural network model, export the clothes being transformed The psychological condition of punishment personnel assesses vector；

Second psychological condition assesses vector output module: the physiological signal feature of inmate to be measured, facial expression is special Phonic signal character of seeking peace is input in preparatory trained neural network model, exports the psychological condition of inmate to be measured Assess vector；

The people that serves a sentence that inmate's psychological health states evaluation module to be measured: calculating inmate to be measured and has been transformed The Euclidean distance of the psychological condition assessment vector of member；Inmate's psychological health states to be measured are assessed according to distance.

The third aspect, the disclosure additionally provide a kind of electronic equipment, including memory and processor and are stored in storage The computer instruction run on device and on a processor when the computer instruction is run by processor, completes first aspect institute The step of stating method.

Fourth aspect, the disclosure additionally provide a kind of computer readable storage medium, for storing computer instruction, institute When stating computer instruction and being executed by processor, the step of completing first aspect the method.

5th aspect, the disclosure additionally provide inmate's psychological health states assessment system based on multi-modal information；

Inmate's psychological health states assessment system based on multi-modal information, comprising:

Electronics described in physiological parameter acquisition device, image collecting device, voice acquisition device and the third aspect is set It is standby；

Physiological parameter acquisition device, image collecting device and voice acquisition device set the data transmission electron of acquisition It is standby；

Electronic equipment according to the collected data assesses the psychological health states of inmate.

Compared with prior art, the beneficial effect of the disclosure is:

(1) the psychological condition quantitatively evaluating mechanism based on multi-modal information is effectively improved simple anaclisis scale and uses Interrogation reply system carries out the accuracy of mental health scale evaluation, avoids because subject is excited, fitness is low, subjective resistance It is interfered Deng caused by.Physiological signal is acquired using some electronic equipments, a variety of physiological signals are handled, to realize to clothes The rapid evaluation of punishment personnel psychology health status and precisely assessment.

Facial expression (facial muscles change composed mode) and intonation expression (tone, rhythm and speed of speech etc. The variation of aspect), it can reflect the subjective emotional experience of people, while the variation of the mood of people and Mood State can be with certain The fluctuating of physiological characteristic, the disclosure make full use of these information, extract feature using the strategy of fusion and carry out intelligent mood knowledge Not.Good inmate is transformed and has similar psychological condition in face of specific virtual reality scenario, as to sighting target Standard, and the inmate for failing to complete good ideological remoulding can obtain inappropriate emotional experience in face of same scene, be based on Multi-modal information is analyzed and is screened to this species diversity, and the ideological remoulding situation and mental health of inmate can be directed to The evaluation result that degree is quantified.

(2) personalized virtual reality emotion excites experience platform.The disclosure is directed to different types of inmate, opens Send out virtual reality experience environment targeted, the personal experience of programme content combination inmate, while comprehensively considering and serving a sentence The factors such as the age education background of personnel make it obtain the emotion excitation experience that I shall appreciate it as a personal favour.Appropriate emotional arousal experiences band The fluctuation for carrying out affective state can cause the immediate feedback of the information such as facial expression, physiological signal, speech intonation, obtain these letters The feature of breath passes through machine learning algorithm intellectual analysis simultaneously and judges, can be with the psychological health states of ideal quantized subject With ideological remoulding degree.

(3) carrying out effective measurement to psychological condition is the key that realize psychological condition assessment and difficult point, is realized to emotion Accurate measurement need emotion measure theory and measuring tool in psychology.The disclosure is quantified using PAD three-dimensional emotion model Affective state, PAD emotion model were proposed that this dimension observed quantity model can have by Mehrabian and Russell in 1974 The mental state of the mankind is explained on effect ground, it is not limited to the subjective experience of description emotion, while calling out with the external presentation of emotion, physiology Waking up has preferable mapping relations.

(4) the intelligent predicting frame based on deep learning.Nineteen forty-three, psychologist W.McCulloch and mathematician W.Pitts cooperation proposes neuron and the earliest mathematical model of neural network from the angle of mathematical logic.Firstly, having Self-learning function, can be different facial expression, physiologic information, voice signal etc. and corresponding psychological condition quantitatively evaluating Artificial neural network is inputted, it can be learned autonomous classification multi-source information and be carried out psychological condition and sentence by the function of self study Fixed, this is of great significance emotion prediction.Secondly, having the ability that high speed finds optimum solution, one is found again The optimization solution of miscellaneous problem, it usually needs very big operand, and utilize an artificial neural network for particular problem design Network can play the high-speed computation ability of computer, quickly find optimization solution.Based on this advantage, neural network can be in brain The specific regularity of distribution is accurately searched out in the complex informations such as electricity, electrocardio, expression, voice, by the heart of they and human body generation Reason mood sets up specific feedback link.The disclosure connects a large amount of simple process unit of artificial neural network to be formed certainly Adaptive dynamics system analyzes bio signal by concurrency, distributed storage, the functions such as self-organizing of adaptive learning, Accurately objectively carry out Mental health evaluation.

It (5) is to calculate basis to assess subject's mental health degree on PAD emotion model with Euclidean distance.Europe It is a kind of common similarity algorithm that distance is obtained in several, calculates Euclidean distance during human body mental emotion similarity Compare intuitive.Euclidean distance is smaller, and two different affective state similarities are bigger, and otherwise similarity is with regard to smaller.It is logical The Euclidean distance for calculating that subject and the good personnel for keeping fit psychology of transformation show on the scale of the basis PAD is crossed, is sentenced The ideological remoulding situation and mental health degree of disconnected subject, by traditional subjective judgement interacting Question-Answer Psychological Evaluation mode Improve as the objective quantification evaluation criterion based on multi-modal information.

Detailed description of the invention

The accompanying drawings constituting a part of this application is used to provide further understanding of the present application, the application's Illustrative embodiments and their description are not constituted an undue limitation on the present application for explaining the application.

Fig. 1 is the system structure diagram of the present embodiment one；

Fig. 2 (a) and Fig. 2 (b) is the PAD three-dimensional emotion model schematic diagram of the present embodiment one；

Fig. 3 is the overall structure block diagram of the present embodiment one；

Fig. 4 is the multi-source physiological signal emotion recognition schematic diagram based on multilayer neural network of the present embodiment one；

Fig. 5 is that the face based on convolutional neural networks of the present embodiment one shows emotion and recognizes schematic diagram；

Fig. 6 is the voice-based psychological condition identification structure block diagram of the present embodiment one；

Fig. 7 is the sonograph product process figure based on phonetic feature of the present embodiment one；

Fig. 8 is the mental health degree assessment schematic diagram based on Euclidean distance of the present embodiment one.

Specific embodiment

It is noted that described further below be all exemplary, it is intended to provide further instruction to the application.Unless Otherwise indicated, all technical and scientific terms used herein has and the application person of an ordinary skill in the technical field Normally understood identical meanings.

It should be noted that term used herein above is merely to describe specific embodiment, and be not intended to restricted root According to the illustrative embodiments of the application.As used herein, unless the context clearly indicates otherwise, otherwise singular shape Formula be also intended to include plural form, additionally, it should be understood that, when in the present specification use term "comprising" and/or When " comprising ", existing characteristics, step, operation, device, component and/or their combination are indicated.

Last century the eighties, scientist is by psychological research Import computer subject, it is intended to which subjective psychological condition is become For that can calculate, i.e., during human-computer interaction, the emotional reactions of this people are understood by the face picture of people, sound.With face Expression Recognition and speech emotional understand difference, and the psychology based on physiological signal, which calculates, possesses unique advantage, it has really Property, objectivity and significantly can not be subjective handling, the psychologic status of people can be objectively responded, but higher for arousal Psychological condition just have preferable recognition effect.

Carry out mind of convict diagnosis and recommitting psychological calculation by mental test for convict, generally when criminal enters and supervises, serve a sentence It mid-term and carries out before releasing upon completion of a sentence, to determine the personality defect of criminal, verifies rectifying effect and a possibility that prediction is recommitted, And targeted modification scheme is made for inmate based on this.Psychological assessment based on questionnaire is with stronger Subjective factor, assessment result will receive the interference of external environment and tested person's idea, while can encounter convict and mismatch and can not have Situations such as effect is linked up, it is difficult to which objective reality reflects the mental health state of tested person, causes the deviation assessed correctional effect.