阅读说明：本技术 一种基于视觉感知的面部神经麻痹患者康复检测系统 (Facial nerve palsy patient rehabilitation detection system based on visual perception ) 是由 陈如中 于 2021-08-03 设计创作，主要内容包括：本发明涉及机器视觉技术领域,具体涉及一种基于视觉感知的面部神经麻痹患者康复检测系统。该系统通过患者运动过程中的人脸三维信息获得患者嘴巴倾斜程度和嘴巴运动能力。通过热成像图像结合人脸三维信息获得代谢严重程度和康复状态特征。根据嘴巴倾斜程度、嘴巴运动能力、代谢严重程度和康复状态特征全面的对患者面部运动过程中的运动变化和热量变化进行分析,获得康复程度。本发明通过患者的三维信息和热成像图像全面的分析了患者的康复程度,便于康复训练的指导。(The invention relates to the technical field of machine vision, in particular to a rehabilitation detection system for facial nerve palsy patients based on visual perception. The system obtains the degree of inclination of the mouth and the mouth movement capacity of the patient through the three-dimensional information of the face of the patient in the movement process. And the metabolic severity and the rehabilitation state characteristics are obtained by combining the thermal imaging image with the three-dimensional information of the human face. And comprehensively analyzing the motion change and the heat change of the patient in the facial motion process according to the mouth inclination degree, the mouth motion ability, the metabolic severity and the rehabilitation state characteristics to obtain the rehabilitation degree. The invention comprehensively analyzes the rehabilitation degree of the patient through the three-dimensional information and the thermal imaging image of the patient, and is convenient for guidance of rehabilitation training.)

1. A system for visual perception-based rehabilitation testing of a patient with facial paralysis, said system comprising:

the three-dimensional information acquisition module is used for acquiring a facial image of a patient in the facial movement process; obtaining human face three-dimensional information according to the facial image;

the mouth inclination degree acquisition module is used for acquiring the mouth inclination degree through the human face three-dimensional information;

the mouth movement capability acquisition module is used for acquiring a mouth movement information set through the coordinate change of the grid points of the three-dimensional face information in the movement process; acquiring mouth movement capacity according to the mouth movement information set;

a mouth metabolic severity acquisition module for obtaining a thermographic image of the patient's face; segmenting the thermal imaging image to obtain a mouth heat map; respectively rotating the mouth heat map and the face three-dimensional information to be horizontal and fusing to obtain a mouth three-dimensional heat map; obtaining a plurality of groups of symmetrical grid points of the mouth according to the three-dimensional heat diagram of the mouth; obtaining a difference in pixel values between each of the mouth symmetric grid points; constructing heat list data through the average value of each column of pixels of the mouth heat map; obtaining metabolic severity from the caloric list data and the pixel value differences;

the rehabilitation state characteristic acquisition module is used for taking the difference of the heat quantity of two sides of the face of the patient in the thermal imaging image as a first heat quantity difference; taking the difference of the first heat difference before and after the facial movement of the patient as a second heat difference; obtaining a rehabilitation status characteristic through the second caloric difference and the mouth movement information set;

and the rehabilitation degree obtaining module is used for obtaining the rehabilitation degree through the mouth inclination degree, the mouth movement capacity, the metabolic severity and the rehabilitation state characteristics.

2. The rehabilitation detecting system for facial nerve palsy patients based on visual perception according to claim 1, wherein the mouth inclination degree obtaining module comprises:

performing principal component analysis on the mouth grid points in the human face three-dimensional information to obtain a plurality of first principal component directions; taking the first principal component direction with the largest projection variance of the mouth grid point in all the first principal component directions as an optimal first principal component direction;

performing principal component analysis on the face centerline grid points in the face three-dimensional information to obtain a plurality of second principal component directions; taking the second principal component direction with the maximum projection variance of the center line grid points in all the second principal component directions as an optimal second principal component direction;

and representing the degree of mouth inclination by the included angle between the optimal first principal component direction and the optimal second principal component direction.

3. The visual perception-based rehabilitation detection system for facial nerve palsy patients according to claim 1, wherein the mouth movement capability acquisition module further comprises a mouth movement information acquisition module;

the mouth movement information acquisition module is used for taking the difference of the distances from the mouth grid point to the reference grid point in the human face three-dimensional information in time sequence as a movement variation; and reducing the dimension of the set of the motion variable quantity to obtain the mouth motion information set.

4. The visual perception-based rehabilitation detection system for facial nerve palsy patients according to claim 3, wherein the mouth movement information acquisition module further comprises an information dimension reduction module;

and the information dimension reduction module is used for reducing the dimension of the set of motion variable quantities to one dimension by using a principal component analysis dimension reduction method.

5. The visual perception-based rehabilitation detection system for facial nerve palsy patients according to claim 1, wherein the mouth movement ability acquisition module comprises:

obtaining a mouth movement amplitude index according to the maximum value and the minimum value of the mouth movement information set;

obtaining a mouth movement part quantity index according to the number of elements larger than the mean value and the number of elements smaller than the mean value in the mouth movement information set;

and obtaining the mouth movement capacity through the mouth movement amplitude index and the mouth movement part quantity index.

6. The visual perception-based facial paralysis patient rehabilitation detection system of claim 5, wherein said mouth movement capability obtaining module obtains said mouth movement capability through a mouth movement capability formula; the mouth motion ability formula includes:

L＝x_maxexp(x_min-x_max)*n_maxexp(n_min-n_max)

wherein L is the mouth movement ability, x_maxIs that it isMaximum value, x, of the set of mouth movement information_minIs the minimum value of the set of mouth movement information, n_maxNumber of elements greater than mean in the set of mouth movement information, n_minExp () is an exponential function for the number of elements in the mouth movement information set that are smaller than the mean.

7. The rehabilitation detection system for facial nerve palsy patients based on visual perception according to claim 1, wherein the mouth metabolic severity acquisition module is obtained through a mouth metabolic severity formula; the mouth metabolic severity formula comprises:

wherein P is the severity of mouth metabolism, var is the variance of the caloric list data, G is the mean of the caloric list data, d is the pixel value difference, exp () is an exponential function.

8. The visual perception-based facial nerve palsy patient rehabilitation detection system of claim 1, wherein the rehabilitation status feature acquisition module further comprises a detailed rehabilitation status feature acquisition module;

the detailed rehabilitation state acquisition module is used for acquiring the heat change of a corresponding mouth grid point in the mouth motion information in the multi-frame three-dimensional mouth thermal diagram; and acquiring detailed rehabilitation state characteristics according to the mouth movement information, the heat change and the rehabilitation state characteristics.

9. The visual perception-based facial paralysis patient rehabilitation detection system of claim 8, wherein said detailed rehabilitation status feature obtaining module obtains said detailed rehabilitation status feature through a detailed rehabilitation status feature formula; the detailed rehabilitation state feature formula comprises:

wherein H is the detailed rehabilitation status characteristic, N is the mouth movement information quantity, H is the rehabilitation status characteristic, F_iFor the ith detailed rehabilitation status feature, Δ G_iFor the caloric change corresponding to the ith detailed rehabilitation status feature, exp () is an exponential function.

10. The visual perception-based facial nerve palsy patient rehabilitation detection system of claim 1, wherein the rehabilitation degree obtaining module obtains the rehabilitation degree through a rehabilitation degree formula; the rehabilitation degree formula is as follows:

wherein R the rehabilitation degree, P the mouth metabolic severity, h the rehabilitation status characteristic, sin θ the mouth inclination degree, L the mouth motion ability, and a an offset coefficient.

Technical Field

The invention relates to the technical field of machine vision, in particular to a rehabilitation detection system for facial nerve palsy patients based on visual perception.

Background

The facial nerve paralysis disease is a disease which is mainly characterized by facial expression muscle movement dysfunction, is a common disease and frequently-occurring disease, is not limited by age, has symptoms of facial distortion and the like, and can not be completed by serious patients with the most basic actions of lifting eyebrows, closing eyes, bulging mouth and the like.

For patients with mouth paralysis, symptoms of mouth deviation, mouth angle deviation and poor mouth muscle movement ability are shown, for example, only part of the muscles on the mouth can move flexibly when the patient speaks.

Facial nerve paralysis disease patient not only need cooperate the drug therapy, still need do the rehabilitation training for a long time and just can effectual cure the disease, for example do actions such as some more grinning, smile and carry out the rehabilitation training, but the patient can not quantitative recovered condition of direction oneself, reflects recovered degree through the displacement condition of some positions in the machine vision analysis facial image among the prior art, does not consider the metabolic degree of the relevant muscle of mouth, unable comprehensive detection patient's recovered degree.

Disclosure of Invention

In order to solve the above technical problems, the present invention aims to provide a rehabilitation detection system for facial paralysis patients based on visual perception, and the adopted technical scheme is as follows:

the invention provides a rehabilitation detection system for facial nerve palsy patients based on visual perception, which comprises:

the three-dimensional information acquisition module is used for acquiring a facial image of a patient in the facial movement process; obtaining human face three-dimensional information according to the facial image;

the mouth inclination degree acquisition module is used for acquiring the mouth inclination degree through the human face three-dimensional information;

the mouth movement capability acquisition module is used for acquiring a mouth movement information set through the coordinate change of the grid points of the three-dimensional face information in the movement process; acquiring mouth movement capacity according to the mouth movement information set;

a mouth metabolic severity acquisition module for obtaining a thermographic image of the patient's face; segmenting the thermal imaging image to obtain a mouth heat map; respectively rotating the mouth heat map and the face three-dimensional information to be horizontal and fusing to obtain a mouth three-dimensional heat map; obtaining a plurality of groups of symmetrical grid points of the mouth according to the three-dimensional heat diagram of the mouth; obtaining a difference in pixel values between each of the mouth symmetric grid points; constructing heat list data through the average value of each column of pixels of the mouth heat map; obtaining metabolic severity from the caloric list data and the pixel value differences;

the rehabilitation state characteristic acquisition module is used for taking the difference of the heat quantity of two sides of the face of the patient in the thermal imaging image as a first heat quantity difference; taking the difference of the first heat difference before and after the facial movement of the patient as a second heat difference; obtaining a rehabilitation status characteristic through the second caloric difference and the mouth movement information set;

and the rehabilitation degree obtaining module is used for obtaining the rehabilitation degree through the mouth inclination degree, the mouth movement capacity, the metabolic severity and the rehabilitation state characteristics.

Further, the mouth inclination degree acquiring module includes:

performing principal component analysis on the mouth grid points in the human face three-dimensional information to obtain a plurality of first principal component directions; taking the first principal component direction with the largest projection variance of the mouth grid point in all the first principal component directions as an optimal first principal component direction;

performing principal component analysis on the face centerline grid points in the face three-dimensional information to obtain a plurality of second principal component directions; taking the second principal component direction with the maximum projection variance of the center line grid points in all the second principal component directions as an optimal second principal component direction;

and representing the degree of mouth inclination by the included angle between the optimal first principal component direction and the optimal second principal component direction.

Further, the mouth movement ability acquisition module further comprises a mouth movement information acquisition module;

the mouth movement information acquisition module is used for taking the difference of the distances from the mouth grid point to the reference grid point in the human face three-dimensional information in time sequence as a movement variation; and reducing the dimension of the set of the motion variable quantity to obtain the mouth motion information set.

Further, the mouth movement information acquisition module further comprises an information dimension reduction module;

and the information dimension reduction module is used for reducing the dimension of the set of motion variable quantities to one dimension by using a principal component analysis dimension reduction method.

Further, the mouth movement ability acquisition module includes:

obtaining a mouth movement amplitude index according to the maximum value and the minimum value of the mouth movement information set;

obtaining a mouth movement part quantity index according to the number of elements larger than the mean value and the number of elements smaller than the mean value in the mouth movement information set;

and obtaining the mouth movement capacity through the mouth movement amplitude index and the mouth movement part quantity index.

Further, the mouth motion ability obtaining module obtains the mouth motion ability through a mouth motion ability formula; the mouth motion ability formula includes:

L＝x_maxexp(x_min-x_max)*n_maxexp(n_min-n_max)

wherein L is the mouth movement ability, x_maxIs the maximum value, x, of the set of mouth movement information_minIs the minimum value of the set of mouth movement information, n_maxNumber of elements greater than mean in the set of mouth movement information, n_minExp () is an exponential function for the number of elements in the mouth movement information set that are smaller than the mean.

Further, the mouth metabolism severity obtaining module obtains the mouth metabolism severity through a mouth metabolism severity formula; the mouth metabolic severity formula comprises:

wherein P is the severity of mouth metabolism, var is the variance of the caloric list data,g mean value of the heat quantity list data, d is the pixel value difference, e^p() Is an exponential function.

Further, the rehabilitation state feature acquisition module further comprises a detail rehabilitation state feature acquisition module;

the detailed rehabilitation state acquisition module is used for acquiring the heat change of a corresponding mouth grid point in the mouth motion information in the multi-frame three-dimensional mouth thermal diagram; and acquiring detailed rehabilitation state characteristics according to the mouth movement information, the heat change and the rehabilitation state characteristics.

Further, the detailed rehabilitation state feature obtaining module obtains the detailed rehabilitation state features through a detailed rehabilitation state feature formula; the detailed rehabilitation state feature formula comprises:

wherein H is the detailed rehabilitation status characteristic, N is the mouth movement information quantity, H is the rehabilitation status characteristic, F_iFor the ith detailed rehabilitation status feature, Δ G_iFor the caloric change corresponding to the ith detailed rehabilitation status feature, exp () is an exponential function.

Further, the rehabilitation degree obtaining module obtains the rehabilitation degree through a rehabilitation degree formula; the rehabilitation degree formula is as follows:

wherein R the rehabilitation degree, P the mouth metabolic severity, h the rehabilitation status characteristic, sin θ the mouth inclination degree, L the mouth motion ability, and a an offset coefficient.

The invention has the following beneficial effects:

1. in the embodiment of the invention, the degree of inclination of the mouth and the movement capacity of the mouth are considered. The metabolic severity and rehabilitation state characteristics comprehensively analyze the rehabilitation degree of the patient, and all characteristics of the mouth of the patient are included, so that the rehabilitation degree is scientific and accurate, and the rehabilitation training of the patient is facilitated.

2. In the embodiment of the invention, the thermal information after the movement of the muscle of the mouth is analyzed through the thermal imaging image, the metabolic severity of the mouth is evaluated through the thermal information, and the rehabilitation state of a patient is more completely analyzed.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a block diagram of a rehabilitation detection system for a facial nerve palsy patient based on visual perception according to an embodiment of the present invention.

Detailed Description

To further illustrate the technical means and effects of the present invention for achieving the predetermined objects, the following detailed description of the rehabilitation testing system for facial paralysis patients based on visual perception according to the present invention with reference to the accompanying drawings and preferred embodiments shows the following detailed descriptions. In the following description, different "one embodiment" or "another embodiment" refers to not necessarily the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

The following describes a specific scheme of the rehabilitation detection system for facial nerve palsy patients based on visual perception in detail with reference to the accompanying drawings.

Referring to fig. 1, a block diagram of a rehabilitation detection system for a facial nerve palsy patient based on visual perception according to an embodiment of the present invention is shown, the system including: the three-dimensional information acquisition module 101, the mouth inclination degree acquisition module 102, the mouth motion ability acquisition module 103, the mouth metabolic severity acquisition module 104, the rehabilitation state feature acquisition module 105 and the rehabilitation degree acquisition module 106.

The three-dimensional information acquisition module 101 is used for acquiring facial images of a patient during facial movement. And obtaining the three-dimensional information of the human face according to the facial image. The three-dimensional information of the face comprises a plurality of types of grid points, and can represent information characteristics of different parts of the face, so that the three-dimensional information of the face is obtained by three-dimensional reconstruction of a face image. In an embodiment of the invention, a two-dimensional assisted self-supervised learning method (2DASL) is used for three-dimensional reconstruction. In other embodiments, three-dimensional reconstruction may also be performed by using methods such as PRNet, VRNet, and the like.

The mouth inclination degree obtaining module 102 is configured to obtain a mouth inclination degree through the three-dimensional face information. The mouth grid points in the human face three-dimensional model can represent various information characteristics of the mouth, so that the mouth inclination degree can be obtained according to the information of the mouth grid points, and the mouth deflection degree can be accurately measured when a patient makes certain expressions or the facial deformation degree of the patient is large. The mouth inclination degree acquisition module 102 specifically includes:

1) and performing principal component analysis on the mouth grid points in the three-dimensional information of the face to obtain a plurality of first principal component directions. And taking the first principal component direction with the largest projection variance of the mouth grid point in all the first principal component directions as the optimal first principal component direction. In the embodiment of the invention, the principal component directions are three directions in a three-dimensional coordinate system.

2) And performing principal component analysis on the face centerline grid points in the face three-dimensional information to obtain a plurality of second principal component directions. The center line of the face is a straight line connecting the eyebrow center of the face to the nose tip. And taking the second principal component direction with the maximum projection variance of the central line grid points in all the second principal component directions as the optimal second principal component direction.

3) The degree of mouth inclination is expressed as the angle between the optimal first principal component direction and the optimal second principal component direction. In the embodiment of the invention, the sine value of the included angle is used for expressing the degree of mouth inclination.

The mouth motion capability obtaining module 103 is configured to obtain a mouth motion information set through coordinate change of grid points of the three-dimensional face information during the motion process. And obtaining the mouth movement capacity according to the mouth movement information set. The face of the patient is allowed to do some actions, such as speaking, and the mouth movement information set of the patient in the action process can be obtained through the three-dimensional face information on the time sequence.

Preferably, the mouth movement ability acquisition module 103 further includes a mouth movement information acquisition module. The mouth movement information acquisition module is used for taking the difference of the distances from the mouth grid point to the reference grid point in the human face three-dimensional information in time sequence as the movement variation. And reducing the dimension of the motion variable set to obtain a mouth motion information set.

Specifically, the formula for obtaining the motion variation of any one mouth grid point and the reference grid point in the i-frame and the i-1 th frame is as follows:

ΔP＝(P_i-p_i)-(P_i-1-p_i-1)

wherein, Δ P is mouth movement information, P_iIs a mouth grid point, p, in the ith frame of human face three-dimensional information_iAnd the reference grid points are the reference grid points in the face three-dimensional information of the ith frame. P_i-1Is a mouth grid point, p, in the three-dimensional information of the face of the frame i-1_i-1And the reference grid points are the reference grid points in the three-dimensional information of the face of the (i-1) th frame.

The elements in the mouth movement information set represent movement information of the corresponding mouth mesh points. When a certain part of the mouth of a patient moves flexibly, the mouth movement information corresponding to the part is large; when a certain part of the mouth of a patient is slow in movement or does not move, the mouth movement information corresponding to the part is small.

In the embodiment of the present invention, the eyebrow center grid point is used as the reference grid point.

It should be noted that the motion variation set after the dimension reduction includes a motion variation sequence corresponding to a plurality of mouth mesh points in the motion process time period. A motion variance sequence is the variance of motion between different frames of a mouth mesh point. And averaging absolute values of elements in the motion variation quantity sequences to obtain motion information of the mouth grid points, wherein the motion information of all the mouth grid points is used as a mouth motion information set. The mouth movement information set represents a set of position changes corresponding to different mouth mesh points in a time period.

Because the patient can make displacement motion, such as small-amplitude head shaking, when doing some actions, the human face also makes displacement motion. Errors caused by the displacement motion of the human face to the mouth motion information calculation can be eliminated through the reference of the eyebrow center grid points.

Preferably, the mouth movement information obtaining module further comprises an information dimension reduction module. The information dimension reduction module is used for reducing the dimension of the set of motion variable quantities to one dimension by using a Principal Component Analysis (PCA) dimension reduction method.

The principal component analysis dimension reduction method specifically comprises the following steps: and acquiring a motion variation set, and performing centralization processing on the mean value of each element in the motion variation set minus the motion variation set. In the embodiment of the invention, the elements in the motion variable set are three-dimensional vectors, the covariance of all the three-dimensional vectors in the set forms a covariance matrix, three eigenvalues of the covariance matrix are obtained, the eigenvector corresponding to the largest eigenvalue is obtained, the projection length of each element in the motion variable set on the eigenvector is calculated, and the projection length is the final dimension reduction result.

The mouth movement ability acquisition module 103 specifically includes:

and obtaining the mouth movement amplitude index according to the maximum value and the minimum value in the mouth movement information set. And obtaining the quantity index of the mouth movement part according to the number of the elements larger than the mean value and the number of the elements smaller than the mean value in the mouth movement information set. The mouth movement ability is obtained through the mouth movement amplitude index and the mouth movement part quantity index.

The larger the mouth movement amplitude index is, the larger the maximum value of the mouth movement information set is, and meanwhile, the smaller the difference value between the maximum value and the minimum value is, the strong mouth movement capability is shown. The grid points larger than the mean value in the motion information set represent strong muscle motion capability, and the grid points smaller than the mean value represent weak muscle motion capability, so that the larger the number of elements larger than the mean value is, the larger the number of the muscles with strong motion capability is, the smaller the number of the elements smaller than the mean value is, the smaller the number of the muscles with weak motion capability is. Meanwhile, the number of the grid points with small motion amplitude is not greatly different from that of the grid points with large motion amplitude, so that the difference between the muscle area with strong mouth motion capability and the muscle area with weak motion capability is small, and the whole mouth motion capability is strong.

The mouth motion capability acquisition module acquires mouth motion capability through a mouth motion capability formula; the mouth motion ability formula includes:

L＝x_maxexp(x_min-x_max)*n_maxexp(n_min-n_max)

wherein L is the mouth movement ability, x_maxIs the maximum value, x, of the set of mouth movement information_minIs the minimum value of the set of mouth movement information, n_maxNumber of elements greater than mean in the set of mouth movement information, n_minFor the number of elements in the mouth movement information set smaller than the mean value, exp () is an exponential function.

A mouth metabolic severity acquisition module 104 for obtaining a thermographic image of the patient's face. The thermographic images were processed to obtain a mouth calorimeter. And respectively rotating the mouth heat map and the three-dimensional face information to be horizontal and fusing to obtain the three-dimensional mouth heat map. The pixel value size of each grid point in the three-dimensional heat map of the mouth represents the amount of heat at that grid point. For normal persons, the gray values of the mouth calorimetric image should be uniformly distributed along the direction from the mouth angle to the center of the mouth, and the gray value difference at the symmetrical position of the mouth with the vertical axis of symmetry is small. And obtaining a plurality of groups of symmetrical grid points of the mouth according to the three-dimensional heat diagram of the mouth. The difference in pixel values between each set of mouth symmetric grid points is obtained. Heat tabulated data was constructed from the mean of each column of pixels of the mouth heat map. The heat tabulation data represents distribution information of the mouth heat information in the mouth direction. The metabolic severity is obtained from the caloric list data and the pixel value differences.

In the embodiment of the invention, the mouth region in the thermal imaging image is segmented by a DeepLapv3 semantic segmentation network, and the mouth heat map is obtained by normalization processing.

The concrete operation of respectively rotating the mouth heat map and the three-dimensional face information to the horizontal level is as follows: and obtaining two third principal component directions of all pixel positions in the mouth region in the image coordinate system, and taking the third principal component direction with the largest projection variance of the mouth pixel points in all the third principal component directions as the optimal third principal component direction. And rotating the mouth area to be horizontal according to the angle between the optimal third principal component direction and the x axis of the image coordinate system.

In the embodiment of the present invention, the heat list data needs to be preprocessed. The preprocessing operation comprises the following steps: and performing mean filtering on the heat list data by using a one-dimensional filter kernel, wherein the width of the filter kernel is 7, and the step length is 7.

The metabolic severity is obtained through the heat information of the mouth muscles in the thermal imaging image, the metabolic severity fuses the mouth morphological characteristics and the heat distribution characteristics of the mouth, the mouth metabolism condition of the patient in a static state is represented, and the higher the metabolic severity is, the larger the difference of the metabolic capacity of different positions of the mouth of the patient is, and the worse the rehabilitation is.

Specifically, the mouth metabolic severity acquisition module 104 acquires the mouth metabolic severity through a mouth metabolic severity formula, where the mouth metabolic severity formula is:

wherein P is the severity of mouth metabolism, var is the variance of the caloric list data, G is the mean of the caloric list data, d is the difference in pixel values, exp () is an exponential function. var and d describe the heterogeneity of heat distribution on the mouth, the larger var, the smaller G and the larger d indicate the more obvious metabolic difference on both sides of the mouth and the greater the severity of metabolism.

The rehabilitation status feature obtaining module 105 is configured to take the difference in the heat quantity on both sides of the face of the patient in the thermal imaging image as the first heat quantity difference. And taking the difference value of the first heat difference before and after the facial movement of the patient as a second heat difference. The rehabilitation status characteristic is obtained through the second heat difference and the mouth movement information set. The rehabilitation status characteristic is a patient, namely a heat change characteristic before and after facial movement, and the characteristic describes whether the patient has a tendency to recover or not.

In an embodiment of the invention, the left and right facial regions of the patient in the thermographic image are segmented by a semantic segmentation network. The left side face area is composed of a left cheek semantic area and a left mouth corner semantic area, and the right side face area is composed of a right cheek semantic area and a right mouth corner semantic area. The difference of the mean values of the gray values of the left and right side surface regions is used as the first heat difference.

When the rehabilitation status is small, the facial heat of the face of the patient is not changed much after the exercise compared with that before the exercise, namely, the metabolic capacity of facial muscles is not enhanced by mouth movement. When the rehabilitation characteristics are larger, the difference between the heat of the left face and the heat of the right face of the patient before the face moves is far larger than the difference between the left face and the right face of the patient after the face moves, and the metabolic capacity of the left face and the right face of the patient after the face moves is enhanced.

Preferably, the rehabilitation status feature acquisition module 105 further comprises a detailed rehabilitation status feature acquisition module. The detail rehabilitation state acquisition module is used for acquiring the heat change of the corresponding mouth grid point in the mouth motion information in the multi-frame mouth three-dimensional heat map. And acquiring detailed rehabilitation state characteristics according to the mouth movement information, the heat change and the rehabilitation state characteristics. The smaller the mouth movement information, the larger the heat change and the larger the detailed rehabilitation status feature.

Specifically, the detailed rehabilitation state feature acquisition module acquires detailed rehabilitation state features through a detailed rehabilitation state feature formula. The detailed rehabilitation state feature formula comprises:

wherein H is the detail rehabilitation status characteristic, N is the mouth movement information quantity, H is the rehabilitation status characteristic, F_iFor the ith detailed rehabilitation status feature, Δ G_iFor the heat change corresponding to the ith detailed rehabilitation status feature, exp () is an exponential function.

The detailed rehabilitation status feature concerns metabolic capability changes at locations where the amplitude of mouth motion is small. The greater the detailed rehabilitation status features, the smaller the mouth movement, the stronger the metabolic capability of the face, i.e. the greater the rehabilitation status.

The rehabilitation status degree obtaining module 106 is used for obtaining the rehabilitation degree through the degree of inclination of the mouth, the movement capability of the mouth, the metabolic severity and the rehabilitation status characteristics.

Specifically, the rehabilitation degree obtaining module 106 obtains the rehabilitation degree through a rehabilitation degree formula. The rehabilitation degree formula is as follows:

wherein R is the rehabilitation degree, P is the mouth metabolism severity, h is the rehabilitation status characteristic, sin theta is the inclination degree, L is the mouth movement ability, and a is the offset coefficient. The rehabilitation degree formula focuses on the static and dynamic characteristics of the mouth of the patient, comprehensively analyzes the facial characteristics of the patient and obtains the rehabilitation degree of the patient. In the embodiment of the invention, letWhere b is set to 0.1. a represents a gain or response produced by mouth movement that is positively correlated with the patient's degree of rehabilitation.

In summary, the embodiment of the invention obtains the degree of mouth inclination and the mouth movement ability of the patient through the three-dimensional face information in the movement process of the patient. And the metabolic severity and the rehabilitation state characteristics are obtained by combining the thermal imaging image with the three-dimensional information of the human face. And comprehensively analyzing the motion change and the heat change of the patient in the facial motion process according to the mouth inclination degree, the mouth motion ability, the metabolic severity and the rehabilitation state characteristics to obtain the rehabilitation degree.

It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.