阅读说明：本技术 假肢控制方法、装置、假肢设备及计算机可读存储介质 (Prosthesis control method, device, prosthesis equipment and computer readable storage medium ) 是由 宫玉琳 韩成飞 陈晓娟 胡命嘉 温鑫宝 曹建南 于 2021-08-10 设计创作，主要内容包括：本发明公开了一种假肢控制方法,包括获取假肢对应肢体的肌电信号、肌音信号以及肌力信号；识别有效活动时间段；将每个有效活动时间段对应有效肌电信号、有效肌音信号、有效肌力信号通过DS-CNN模型进行特征提取,并将获得肌电特征数据、肌音特征数据以及肌力特征数据通过预先训练获得的3D分层卷积融合模型,进行特征识别分类,确定肢体的控制动作；基于控制动作对假肢驱动装置进行控制驱动。本申请基于肢体的三种不同的生物信号特征之间的融合特征识别肢体动作,提升肢体动作识别分类的准确性,进而提升对假肢控制的准确性和使用者的使用体验。本申请还提供了一种假肢控制装置、假肢设备以及计算机可读存储介质,具有上述有益效果。(The invention discloses a control method of an artificial limb, which comprises the steps of obtaining myoelectric signals, myoelectric signals and muscle strength signals of limbs corresponding to the artificial limb; identifying a valid activity time period; carrying out feature extraction on an effective myoelectric signal, an effective myoelectric signal and an effective myoelectric signal corresponding to each effective activity time period through a DS-CNN model, carrying out feature recognition and classification on the obtained myoelectric feature data, myoelectric feature data and myoelectric feature data through a 3D layered convolution fusion model obtained by pre-training, and determining the control action of the limb; and controlling and driving the artificial limb driving device based on the control action. According to the method and the device, the limb actions are identified based on the fusion characteristics among three different biological signal characteristics of the limb, the accuracy of limb action identification classification is improved, and then the accuracy of artificial limb control and the use experience of a user are improved. The application also provides a prosthesis control device, a prosthesis equipment and a computer readable storage medium, which have the beneficial effects.)

1. A prosthesis control method, comprising:

acquiring myoelectric signals, myoelectric signals and myoelectric signals of limbs corresponding to the artificial limb;

identifying an effective activity time period according to the electromyographic signals, and taking the electromyographic signals, the myoelectric signals and the myoelectric signals in the effective activity time period as effective electromyographic signals, effective myoelectric signals and effective myoelectric signals respectively;

respectively carrying out feature extraction on the effective myoelectric signal, the effective myoelectric signal and the effective myoelectric signal corresponding to each effective activity time period through a DS-CNN (direct sequence-neural network) model to obtain myoelectric feature data, myoelectric feature data and myoelectric feature data;

performing feature recognition and classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by using a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb;

and controlling and driving the artificial limb driving device based on the control brake.

2. A prosthetic control method according to claim 1, wherein identifying an active activity period based on the electromyographic signals comprises:

carrying out summation operation on absolute values of original electromyographic signals acquired by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

utilizing a least square method to sample the first electromyographic signal y corresponding to the ith time in a sliding window with set size_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

substituting the fitting parameter vector into the linear equation to obtain a second electromyographic signal which corresponds to each sampling point and is subjected to smoothing treatment;

and judging whether the number of second electromyographic signals continuously larger than a threshold value in the second electromyographic signals exceeds a preset number, if so, taking the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value as an effective activity time period.

3. A prosthesis control method as claimed in claim 1, further comprising, after the control driving of the prosthesis driving device based on the control brake,:

collecting temperature data and sliding touch data of the artificial limb fingertip;

performing temperature stimulation feedback control on the limb according to the temperature data;

and performing sliding tactile stimulation feedback control on the limb according to the sliding tactile data.

4. A prosthetic control method according to claim 3, wherein performing temperature-stimulated feedback control of the limb based on the temperature data includes:

and performing feedback control on the temperature of the semiconductor refrigeration sheet attached to the limb according to the temperature data.

5. A prosthetic control method according to claim 3, wherein performing haptic stimulus feedback control of the limb based on the haptic data includes:

and carrying out feedback control on the magnitude of the stimulating current of the conducting strip attached to the limb according to the sliding tactile sensation data.

6. A prosthetic control device, comprising:

the signal acquisition module is used for acquiring myoelectric signals, myoelectric signals and muscle strength signals of limbs corresponding to the artificial limb;

the active segment identification module is used for identifying an effective active time segment according to the electromyographic signals, and taking the electromyographic signals, the myoelectric signals and the myoelectric signals in the effective active time segment as effective electromyographic signals, effective myoelectric signals and effective myoelectric signals respectively;

the characteristic extraction module is used for carrying out characteristic extraction on the effective electromyographic signals, the effective muscle tone signals and the effective muscle strength signals corresponding to each effective activity time period through a DS-CNN (digital signal communication network) model to obtain electromyographic characteristic data, muscle tone characteristic data and muscle strength characteristic data;

the action classification module is used for carrying out feature recognition classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by utilizing a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb;

and the control driving module is used for controlling and driving the artificial limb driving device based on the control action.

7. A prosthetic control device according to claim 6, wherein the active segment identification module includes:

the system comprises an absolute value operation unit, a data processing unit and a data processing unit, wherein the absolute value operation unit is used for summing absolute values of original electromyographic signals collected by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

a linear fitting unit for sampling the corresponding first electromyographic signal y for the ith time in a sliding window with a set size by using a least square method_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

the smoothing unit is used for substituting the fitting parameter vector into the linear equation to obtain a smoothed second electromyographic signal corresponding to each sampling point;

the judgment and identification unit is used for judging whether the number of second electromyographic signals continuously larger than a threshold value in the second electromyographic signals exceeds a preset number, and if yes, the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value is an effective activity time period.

8. A prosthetic control device according to claim 6, further comprising a feedback module for collecting temperature data and sliding sensation data of a prosthetic fingertip after control actuation of the prosthetic drive device based on the control action; performing temperature stimulation feedback control on the limb according to the temperature data; and performing sliding tactile stimulation feedback control on the limb according to the sliding tactile data.

9. A prosthetic device, comprising:

a prosthesis body;

the myoelectric sensor, the myosound sensor and the myoforce sensor are used for being attached to the corresponding limb of the artificial limb body;

the main controller is respectively connected with the electromyographic signal sensor, the muscle sound sensor and the muscle force sensor;

the main controller is used for acquiring myoelectric signals acquired by the myoelectric sensor, myoelectric signals acquired by the myoelectric sensor and myoelectric signals acquired by the myoelectric sensor, and executing the steps of the prosthesis control method according to any one of claims 1 to 5 to control the action of the prosthesis body.

10. A computer-readable storage medium, in which a computer program is stored, which computer program is executed by a main controller to implement the steps of the prosthesis control method according to any one of claims 1 to 5.

Technical Field

The present invention relates to the field of prosthetic control technologies, and in particular, to a prosthetic control method, device, prosthetic device, and computer-readable storage medium.

Background

There are still millions of amputees in the world. Currently, the only method of recovery is through the prosthesis. The development history of prostheses spans thousands of years, progressing from decorative prostheses to functional prostheses, and smart prostheses based on the Human Machine Interface (HMI) concept have emerged. The concept of an intelligent prosthesis is generally referred to as "intent control" of the prosthesis, which means that the user controls the prosthesis by brain or healthy body movement intent, so that the user feels the prosthesis, i.e., the use experience that is a part of his own body, and can bring the user a good use experience both physically and psychologically.

At present, the control intention of a user is mainly recognized by detecting the electromyographic signals of limbs of partial non-disabled limbs, which are closest to the disabled limbs, on the body of the user. It is possible for a user to control the motion of his intact limb as well as his normal limb according to his own consciousness when he wants to control the prosthesis. Obviously, the partial limb movement is completed based on the muscle movement of the limb, and the modes of the limb for carrying out the muscle movements corresponding to different movements are different, so that the partial limb movement can be used as a basis for identifying the type of the limb movement, and the artificial limb is correspondingly controlled based on the identified type of the limb movement.

At present, the muscle activities of limb movement are identified and detected mainly by analyzing and identifying myoelectric signals of muscles, but the accuracy of the identification method on the limb movement is limited, so that the control experience of the artificial limb is poor.

Disclosure of Invention

The invention aims to provide a prosthesis control method, a prosthesis control device, a prosthesis device and a computer readable storage medium, which can improve the prosthesis control accuracy to a certain extent and improve the use experience of a user.

In order to solve the above technical problem, the present invention provides a method for controlling a prosthesis, comprising:

acquiring myoelectric signals, myoelectric signals and myoelectric signals of limbs corresponding to the artificial limb;

identifying an effective activity time period according to the electromyographic signals, and taking the electromyographic signals, the myoelectric signals and the myoelectric signals in the effective activity time period as effective electromyographic signals, effective myoelectric signals and effective myoelectric signals respectively;

performing feature extraction on the effective myoelectric signal, the effective myoelectric signal and the effective myoelectric signal corresponding to each effective activity time period through a DS-CNN (direct sequence-neural network) model to obtain myoelectric feature data, myoelectric feature data and myoelectric feature data;

performing feature recognition and classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by using a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb;

and controlling and driving the artificial limb driving device based on the control brake.

In an alternative embodiment of the present application, identifying a valid activity period from the electromyographic signal comprises:

carrying out summation operation on absolute values of original electromyographic signals acquired by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

utilizing a least square method to sample the first electromyographic signal y corresponding to the ith time in a sliding window with set size_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

substituting the fitting parameter vector into the linear equation to obtain a second electromyographic signal which corresponds to each sampling point and is subjected to smoothing treatment;

and judging whether the number of second electromyographic signals continuously larger than a threshold value in the second electromyographic signals exceeds a preset number, if so, taking the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value as an effective activity time period.

In an alternative embodiment of the present application, after the controlling and driving of the prosthesis driving device based on the controlling action, the method further comprises:

collecting temperature data and sliding touch data of the artificial limb fingertip;

performing temperature stimulation feedback control on the limb according to the temperature data;

and performing sliding tactile stimulation feedback control on the limb according to the sliding tactile data.

In an alternative embodiment of the present application, the performing temperature stimulation feedback control on the limb according to the temperature data comprises:

and performing feedback control on the temperature of the semiconductor refrigeration sheet attached to the limb according to the temperature data.

In an alternative embodiment of the present application, performing a tickle stimulus feedback control on the limb according to the tickle data includes:

and carrying out feedback control on the magnitude of the stimulating current of the conducting strip attached to the limb according to the sliding tactile sensation data.

The present application also provides a prosthetic control device comprising:

the signal acquisition module is used for acquiring myoelectric signals, myoelectric signals and muscle strength signals of limbs corresponding to the artificial limb;

the active segment identification module is used for identifying an effective active time segment according to the electromyographic signals, and taking the electromyographic signals, the myoelectric signals and the myoelectric signals in the effective active time segment as effective electromyographic signals, effective myoelectric signals and effective myoelectric signals respectively;

the characteristic extraction module is used for carrying out characteristic extraction on the effective electromyographic signals, the effective muscle tone signals and the effective muscle strength signals corresponding to each effective activity time period through a DS-CNN (digital signal communication network) model to obtain electromyographic characteristic data, muscle tone characteristic data and muscle strength characteristic data;

the action classification module is used for carrying out feature recognition classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by utilizing a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb;

and the control driving module is used for controlling and driving the artificial limb driving device based on the control action.

In an optional embodiment of the present application, the active segment identification module comprises:

the system comprises an absolute value operation unit, a data processing unit and a data processing unit, wherein the absolute value operation unit is used for summing absolute values of original electromyographic signals collected by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

a linear fitting unit for sampling the corresponding first electromyographic signal y for the ith time in a sliding window with a set size by using a least square method_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

the smoothing unit is used for substituting the fitting parameter vector into the linear equation to obtain a smoothed second electromyographic signal corresponding to each sampling point;

the judgment and identification unit is used for judging whether the number of second electromyographic signals continuously larger than a threshold value in the second electromyographic signals exceeds a preset number, and if yes, the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value is an effective activity time period.

In an optional embodiment of the present application, the system further comprises a feedback module, configured to collect temperature data and sliding touch data of the prosthetic fingertip after the control driving of the prosthetic driving device based on the control action; performing temperature stimulation feedback control on the limb according to the temperature data; and performing sliding tactile stimulation feedback control on the limb according to the sliding tactile data.

The present application also provides a prosthetic device, comprising:

a prosthesis body;

the myoelectric sensor, the myosound sensor and the myoforce sensor are used for being attached to the corresponding limb of the artificial limb body;

the main controller is respectively connected with the electromyographic signal sensor, the muscle sound sensor and the muscle force sensor;

the main controller is used for acquiring myoelectric signals acquired by the myoelectric sensor, myoelectric signals acquired by the myoelectric sensor and myoelectric signals acquired by the myoelectric sensor, and executing the steps of the prosthesis control method to control the prosthesis body to act.

The present application also provides a computer readable storage medium having stored therein a computer program for execution by a master controller to implement the steps of the prosthesis control method as claimed in any one of the above.

The invention provides a prosthetic limb control method, which comprises the following steps: acquiring myoelectric signals, myoelectric signals and myoelectric signals of limbs corresponding to the artificial limb; identifying an effective activity time period according to the electromyographic signals, and taking the electromyographic signals, the myographic signals and the muscle strength signals in the effective activity time period as effective electromyographic signals, effective myographic signals and effective muscle strength signals respectively; performing characteristic extraction on the effective myoelectric signal, the effective myoelectric signal and the effective myoelectric signal corresponding to each effective activity time period through a DS-CNN model to obtain myoelectric characteristic data, myoelectric characteristic data and myoelectric characteristic data; carrying out feature recognition and classification on myoelectric feature data, myosound feature data and muscle strength feature data by using a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb; and controlling and driving the artificial limb driving device based on the control action.

When the artificial limb is controlled, three different biological signals, namely an electromyographic signal, a myoelectric signal and a myoelectric signal, are collected from a limb corresponding to the artificial limb, namely relevant information of limb movement is obtained from three different aspects, on the basis, a feature signal extracted based on the three different biological signals is subjected to action recognition by using a 3D layered convolution fusion model, the 3D layered convolution fusion model can realize local interaction between every two modes (between two biological signal features) and global interaction between all modes (between all biological signal features) in the process of carrying out action recognition on the three different biological signals, obtain multi-mode fusion features containing different biological signal features, and realize identification and classification of limb actions based on the multi-mode fusion features; that is to say, when the body action identification classification is carried out on three different biological signal characteristics of the body in the application, a model capable of identifying the action based on the fusion characteristics among various different biological signals is adopted, so that the accuracy of the body action identification classification is improved to a certain extent, the accuracy of the subsequent control of the artificial limb based on the body action identification classification result is also improved, and the use experience of a user is favorably improved.

The application also provides a prosthesis control device, a prosthesis equipment and a computer readable storage medium, which have the beneficial effects.

Drawings

In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.

FIG. 1 is a schematic flow chart of a prosthesis control method according to an embodiment of the present disclosure;

FIG. 2 is a schematic cross-sectional view of a multi-modal sensor apparatus according to an embodiment of the present application;

FIG. 3 is a block diagram of a prosthesis control device provided for an embodiment of the present invention;

FIG. 4 is a schematic view of a right hand prosthetic hand according to an embodiment of the present application;

fig. 5 is a schematic structural view of the prosthetic finger of fig. 4.

Detailed Description

The electromyographic signals, also called Surface Electromyography (sEMG), are weak bioelectric signals formed on the skin Surface after the action electric potential sequence of multiple motion units is conducted through muscles, subcutaneous tissues and skin during muscle contraction, and have information directly related to muscle activation, so that the limb actions can be effectively reflected. However, the sEMG needs a stable signal component in the acquisition process, and is susceptible to noise and poor in anti-interference capability, so that the robustness of the sEMG is poor.

In the application, it is considered that the electromyographic signal is mainly used for obtaining information of muscle contraction and relaxation when the limb is active, and the information can be obtained not only in the form of the electromyographic signal but also from other ways.

Therefore, the technical scheme that the limb action information can be obtained from different ways, the accuracy of limb action identification is improved to a certain extent, and more accurate control of the action of the artificial limb is realized is provided.

In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, fig. 1 is a schematic flow chart of a prosthesis control method provided in an embodiment of the present application, and the method may include:

s11: acquiring myoelectric signals, myoelectric signals and myoelectric signals of the artificial limb corresponding to the limb.

The MMG (Mechanomyography) is a low-frequency vibration signal generated when muscle fibers slide and rub during muscle contraction, the acquisition of the MMG has low requirements on a muscle sound sensor, the MMG can be realized by a common piezoelectric or acceleration sensor, the MMG can be acquired without directly contacting with the skin, and the muscle sound signal is less influenced by the condition of the surface of the skin. But simultaneously, the MMG is easily influenced by limb movement artifacts due to the fact that the overall signal-to-noise ratio is lower than sEMG.

The method is characterized in that when a muscle contracts, a muscle Force signal (FMG) is a pressure signal generated by muscle deformation on the surface of skin, when a limb moves, the pressure signal generated by the muscle contraction on the surface of the skin is detected through a piezoelectric material tightly attached to the skin, wherein a PVDF piezoelectric film sensor and an FSR (Force Sensing resistance pressure sensor) can be used as sensors for collecting the muscle Force signal, and the method has the characteristics of easiness in processing, softness, good toughness and the like, and is excellent piezoelectric signal collecting equipment.

Based on the characteristics of three biological signals, namely an electromyographic signal, a myoelectric signal and the like, the combination of the three biological signals can not only make up the problem of a single biological signal in sensing detection, but also reduce the control fault of the artificial limb caused by sensing errors and improve the overall performance of artificial limb control.

Referring to fig. 2, fig. 2 is a schematic cross-sectional structure diagram of a multi-modal sensor apparatus provided in an embodiment of the present application. In this embodiment, three sensors for detecting an electromyographic signal, a myoelectric signal, and a myoelectric signal may be integrated in the same multimodal sensor apparatus. Wherein, the electrode of the electromyographic sensor corresponding to the electromyographic signal can be selected from a dry electrode 11 made of silver/silver chloride; the muscle sound sensor can select a TD-3 acceleration sensor 12 to collect vibration signals generated during muscle contraction; the muscle force sensor can be selected from the FSR sensor 13 to acquire pressure signals generated when muscles contract.

In addition, the multi-mode sensor device is also provided with a communication component which is connected with the main controller, electromyographic signals, myoelectric signals and myoelectric signals collected by the electromyographic sensor, the myoelectric sensor and the myoelectric sensor are respectively transmitted to the main controller through the communication component, and the main controller analyzes the three biological signals to realize the classification and identification of the limb actions. The communication component in the multimodal sensor apparatus may be a wireless communication module such as bluetooth, WIFI, NFC, or a wired communication module directly connected to the host controller by a wire, which is not limited in this application.

In addition, considering that the muscle extension and contraction at different positions are different when the limbs move, generally, a plurality of groups of multi-mode sensor devices are arranged at the same time instead of only one group of multi-mode sensor devices, each group of multi-mode sensor devices comprises a myoelectric sensor, a myosound sensor and a muscle force sensor, and the sensors of the same type can simultaneously acquire a signal of the same type in each multi-mode sensor device at the same sampling time point. When the limb action is judged subsequently, the myoelectric signals and the muscle strength signals collected by the sensors can be combined for comprehensive analysis.

S12: and identifying an effective activity time period according to the electromyographic signals, and taking the electromyographic signals, the myographic signals and the muscle strength signals in the effective activity time period as effective electromyographic signals, effective myographic signals and effective muscle strength signals respectively.

It can be understood that when the user wants to control the movement of the prosthesis, the user can control the limb to generate the movement, and the three corresponding biological signals can be used as the signals for analyzing the movement of the limb. For the sensors of the three biological signals, the signal acquisition is uninterrupted, that is, when there is no movement of the limb, the sensors can also perform signal acquisition, so that the three biological signals in the corresponding time period during the limb movement are required to be identified and extracted as effective time period signals so as to be used as signal data for subsequent movement identification and analysis; namely the time period of the limb movement is the effective movement time period; the myoelectric signals, the myoelectric signals and the muscle strength signals collected in the effective activity time period are effective myoelectric signals, effective myoelectric signals and effective muscle strength signals.

In general, the effective activity time periods corresponding to the electromyographic signal, the myoelectric signal and the myoelectric signal respectively should be the same time period, so that in the actual identification process, the effective activity time period can be identified according to any one of the signals.

Taking the electromyographic signal as the basis for identifying the valid activity time period, in an optional embodiment of the present application, the identifying process of the valid activity time period may include:

carrying out summation operation on absolute values of original electromyographic signals acquired by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

utilizing a least square method to sample the first electromyographic signal y corresponding to the ith time in a sliding window with set size_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

substituting the fitting parameter vector into a linear equation to obtain a second electromyographic signal which corresponds to each sampling point and is subjected to smoothing treatment;

and judging whether the number of the second electromyographic signals continuously larger than the threshold value in the second electromyographic signals exceeds a preset number, if so, taking the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value as an effective activity time period.

It should be noted that, in the present embodiment, it is considered that the electromyographic signal for the limb fluctuates up and down around the 0 point, and the amplitude of the fluctuation also changes randomly. And because some sampling points are influenced by factors such as noise, the amplitude of some sampling points has overlarge abrupt change, which influences the accuracy of the subsequent effective activity identification period to a certain extent.

Therefore, before effective activity segment recognition is carried out based on the electromyographic signals, absolute value operation is carried out on original electromyographic signals acquired by the electromyographic signals to obtain the electromyographic signals subjected to the absolute value operation, and the electromyographic signals are increased to a certain extent; performing superposition operation on absolute values of original electromyographic signals measured by a plurality of electromyographic sensors corresponding to the same sampling time point, and taking signal data after the superposition operation as a first electromyographic signal corresponding to the sampling time point; and then, the first electromyographic signal is subjected to smoothing treatment, so that the fluctuation range of the amplitude of the first electromyographic signal is narrowed to a certain extent, and the subsequent identification of the effective activity time period is facilitated.

The principle of the electromyogram signal smoothing processing of summing after absolute value operation will be explained below.

First electromyographic signal y set within a sliding window of length L_iThe first myoelectric signal vector is Y ═ Y₁,y₂,......,y_L). It is clear that the first electromyographic signal y is for different sampling time points_iMay be different in magnitude, and thus, the first electromyogram signal may be regarded as a variable signal that changes according to a change in sampling time point.

Therefore, the sampling times corresponding to each sampling in the sliding window represent each sampling point, each sampling time point corresponds to one sampling time, and the later the sampling time point is, the larger the corresponding sampling times are; and setting that the first electromyographic signal and each sampling frequency satisfy a certain linear relation, and performing linear fitting on the linear relation of the first electromyographic signal changing along with the sampling frequency.

Based on the characteristics of linear fitting, the process of linear fitting discrete data points is to discard some points far from the center to obtain a curve equation that is satisfied by most data points or is located at the center of most data points.

The process of smoothing the first electromyographic signal is also a process of discarding part of electromyographic signals with the adjacent electromyographic signals far away, or fitting and converting the electromyographic signals with the adjacent electromyographic signals far away into electromyographic signals with the amplitude closer to the adjacent electromyographic signals.

Therefore, in this embodiment, the smoothing process of the first electromyographic signal is directly implemented by using a linear fitting process. The specific process can comprise the following steps:

the linear fitting is adopted to satisfy the condition that the first electromyographic signal and the sampling times are setWherein, a_j-1Is a linear coefficient; k is a constant parameter and is a positive integer, the specific size of k can be set based on the electromyographic signals, but the size of k should be smaller than L; l is the sliding window length, i.e. the total number of samples within the sliding window.

Based on the linear fitting equation of the first electromyographic signal of the ith sampling point, k element linear equation sets of L k element equation combinations can be obtained:namely: y ═ X. A^T(ii) a Wherein the content of the first and second substances,A＝(a₀,a₁,…,a_k-1) (ii) a Based on the known values of Y ═ Y₁,y₂,......,y_L) And corresponding sampling points are substituted into the linear equation set, linear fitting is carried out through a least square method, and each linear coefficient parameter a in A can be obtained through fitting_j-1。

After determining the fitting parameter vector A ═ (a)₀,a₁,…,a_k-1) Then, it is obvious that the first electromyographic signals corresponding to the sampling points should fluctuate up and down on the curve corresponding to the linear equation, and the size of the electromyographic signals corresponding to the sampling points and located on the curve of the linear equation can be regarded as the corresponding electromyographic signals after the first electromyographic signals are smoothed.

Therefore, only each sampling point i and the fitting parameter vector A are required to be substituted into the linear equation set, and the electromyographic signal obtained through calculation is the second electromyographic signal after smoothing processing.

After the electromyographic signals are smoothed, the size of each second electromyographic signal can be compared with a threshold, and if a plurality of continuous second electromyographic signals are all larger than the threshold, the time period of the sampling time point corresponding to the plurality of second electromyographic signals continuously larger than the threshold is the effective activity time period.

It should be noted that, the identification process of the effective activity time period may be performed based on only the electromyographic signal collected by the electromyographic sensor, or may be performed based on the myographic signal or the muscle strength signal, and the identification manner may be the same as the manner of identifying the effective activity time period by the electromyographic signal, which is not limited in this application.

Furthermore, it is further considered that the myoelectric sensor, the myoelectric sensor and the myoelectric sensor generally have different sampling frequencies, and therefore, the effective activity periods identified based on the myoelectric signal, the myoelectric signal and the myoelectric signal may have a certain deviation at the head and tail time points and may not completely overlap each other. Therefore, in the practical application process, the effective activity time periods corresponding to the myoelectric signals, the myoelectric signals and the myoelectric signals can be respectively identified, and the effective signals of the effective activity time periods are subsequently extracted, wherein the effective activity time periods are also subject to the respective effective activity time periods.

An effective activity time period can be identified respectively according to the electromyographic signals, the myoelectric signals and the myoelectric signals, and finally an average time period or a maximum time period is selected comprehensively as the common effective activity time period of the three signals, which belong to optional embodiments in the application.

S13: and performing characteristic extraction on the effective electromyographic signals, the effective muscle tone signals and the effective muscle force signals corresponding to each effective activity time period through a DS-CNN model to obtain electromyographic characteristic data, muscle tone characteristic data and muscle force characteristic data.

It should be noted that, the model for extracting the characteristics of the effective myoelectric signal, and the effective myoelectric signal also needs to be trained by signal sample data acquired in advance to determine parameters in the characteristic extraction model.

When the characteristic values of multimode biological signals such as effective myoelectric signals, effective myoelectric signals and the like are extracted, the characteristic extraction model is easy to over-fit data characteristics due to the fact that the physiological signal sample magnitude is small, the generalization performance is poor, the difference of the expressions of a training set and a testing set on the model is large, the over-fitting phenomenon is generated, and the accuracy of characteristic data extracted by the finally determined characteristic extraction model in practical application is relatively low.

Therefore, in the embodiment, a DS-CNN (depth separable convolutional network) model is selected to obtain the time sequence characteristics and the spatial characteristics of the effective myoelectric signal, the effective myoelectric signal and the effective myoelectric signal, and the parameters are small, so that the problem of overfitting in the training process is avoided, and the accuracy of extracting the characteristics of the effective myoelectric signal, the effective myoelectric signal and the effective myoelectric signal in the actual application process is ensured.

S14: and performing feature recognition and classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by using a 3D layered convolution fusion model obtained by pre-training to determine the control action of the limb.

After obtaining the myoelectric characteristic data, the myosound characteristic data and the muscle strength characteristic data, the three characteristic data can be subjected to dimension increasing splicing to form multidimensional characteristic data.

The myoelectricity characteristic data, the muscle sound characteristic data and the muscle strength characteristic data are assumed to be three kinds of one-dimensional characteristic data respectively, the three kinds of characteristic data can be spliced to form three-dimensional characteristic data, and if the myoelectricity characteristic data, the muscle sound characteristic data and the muscle strength characteristic data originally belong to two-dimensional or more-dimensional characteristic data, the three kinds of characteristic data can be spliced to form data with higher dimensionality.

After the three kinds of feature data are subjected to dimension increasing splicing, interaction between every two features of the feature data subjected to dimension increasing splicing and global interaction between global data are carried out through a 3D layered convolution fusion model, then fusion features among the feature data can be extracted, the corresponding limb action type can be determined based on the fusion features, and then recognition of limb action classification is achieved, and the limb action is the action required to be controlled on the artificial limb.

S15: and controlling and driving the artificial limb driving device based on the control action.

After the action of controlling and driving the artificial limb is determined, the driving component of the artificial limb is directly controlled to drive the artificial limb to act.

The artificial limb is a bionic structure formed according to the structure of bones and muscles of a human hand, wherein the bones are bionic by a 3D printing material, and the muscles are bionic by a nylon rope and a spring. The prosthesis driving device comprises: steering wheel, steering wheel drive, lithium cell etc.. Taking a hand prosthesis as an example, the finger components of the prosthesis correspond to five fingers of a hand in five groups, and each finger consists of three finger joints, a nylon rope, a spring, a flexible buffer cushion and a pin shaft; wherein the nylon rope drives the finger tips to bend and is arranged on the finger to simulate a ligament of a human hand; the palm supporting plate fixes the finger assembly and is provided with a steering engine drive, the steering wheel is arranged on a transmission shaft of the steering engine drive, and the nylon rope is pulled to control the finger to bend; the steering engine drive drives the steering engine to rotate after receiving the action command, and then the artificial hand makes corresponding action.

To sum up, when the control artificial limb is controlled based on the subjective intention of the user in the application, three different signals, such as myoelectric signals, myoelectric signals and muscle strength signals, of the body corresponding to the artificial limb are collected at the same time, and the characteristics of the three signals are fused to identify the limb action which the user wants to carry out, so that the accuracy of limb action identification can be improved to a certain extent, and the accuracy of artificial limb action control is also improved to a certain extent, and the use experience of the artificial limb is improved.

Based on the above embodiments, in another optional embodiment of the present application, the method may further include:

after the control drive is carried out on the artificial limb driving device, the temperature data and the sliding tactile data between artificial limbs are also collected;

performing temperature stimulation feedback control on the limb according to the temperature data;

and performing sliding touch stimulation feedback control on the limb according to the sliding touch data.

In practical application, a tactile and sliding sensor and a temperature sensor can be arranged at the position of the artificial finger. The tactile sensing module has two functions: firstly, the contact condition of the artificial hand and the object when the artificial hand grabs the object is detected, namely the size of the vertical acting force acting on the object; and secondly, generating acting force for controlling the object according to the sliding trend of the object. The temperature sensing module can adopt a temperature sensor HDC1080 contact temperature and humidity sensor, and can measure temperature and humidity information with high precision at the same time.

In this embodiment, it is considered that, when the artificial limb is controlled by the main controller to act, the user cannot directly contact the object grabbed by the artificial limb, and the force control on the grabbed object is not intuitive, so that the problem of inaccurate force control may exist. For this reason, in this embodiment, the tactile and slip sensor is further disposed on the artificial limb, so as to detect the pressure of the object grabbed by the artificial limb, and feed the detection result back to the controller, so as to appropriately adjust the grabbing strength of the artificial limb.

In order to further improve the user's experience of freely controlling the movement of the prosthesis, in this embodiment, electrode plates attached to the limb are further disposed on the prosthesis, and when the tactile and slip sensation sensors detect pressures of different magnitudes, the main controller can output current stimuli of different magnitudes to the limb through the electrode plates, so that the user can intuitively feel the tightness of the gripped object, and then the user can control the movement of the prosthesis autonomously according to the sensed stimulation intensity of the electrode plates, thereby improving the autonomy of the user in controlling the prosthesis.

For the artificial limb material, the artificial limb is not suitable for contacting an object with too high or too low temperature for a long time, so that a temperature sensor is further arranged on the artificial limb for detecting the surface temperature of the object directly contacting the artificial limb and feeding back the temperature. Furthermore, the principle of the tactile and sliding sensation sensor is similar, so that a user can feel the surface temperature of an object contacted with the artificial limb more intuitively, the artificial limb can be further provided with a semiconductor refrigerating sheet which can be attached to the limb, the main controller receives the temperature measured by the temperature sensor and can control the temperature of the semiconductor refrigerating sheet, and the user can feel the actual temperature of the object to be grabbed and contacted in an incised mode so as to determine whether to continuously contact the object.

The prosthesis control device provided by the embodiment of the invention is described below, and the prosthesis control device described below and the prosthesis control method described above are referred to correspondingly.

Fig. 3 is a block diagram illustrating the construction of a prosthesis control device according to an embodiment of the present invention, and referring to fig. 3, the prosthesis control device may include:

the signal acquisition module 100 is used for acquiring myoelectric signals, myoelectric signals and muscle strength signals of limbs corresponding to the artificial limb;

an active segment identification module 200, configured to identify an effective active time segment according to the electromyographic signal, and take the electromyographic signal, the myographic signal, and the myographic signal in the effective active time segment as an effective electromyographic signal, an effective myographic signal, and an effective myographic signal, respectively;

the feature extraction module 300 is configured to perform feature extraction on the valid myoelectric signal, the valid myoelectric signal and the valid myoelectric signal corresponding to each valid activity time period through a DS-CNN model to obtain myoelectric feature data, myoelectric feature data and myoelectric feature data;

the action classification module 400 is configured to perform feature recognition classification on the myoelectric feature data, the myosound feature data and the muscle strength feature data by using a 3D layered convolution fusion model obtained through pre-training, and determine a control action of the limb;

and the control driving module 500 is used for controlling and driving the artificial limb driving device based on the control action.

In an optional embodiment of the present application, the active segment identification 200 module comprises:

the system comprises an absolute value operation unit, a data processing unit and a data processing unit, wherein the absolute value operation unit is used for summing absolute values of original electromyographic signals collected by a plurality of electromyographic sensors at the same sampling time point to obtain a first electromyographic signal corresponding to each sampling time point;

a linear fitting unit for sampling the first electromyographic signal y corresponding to the ith time in a sliding window with a set size by using a least square method_iTo satisfy the linear equationObtaining a fitting parameter vector A ═ a₀,a₁,…,a_k-1) (ii) a Wherein i ∈ [1, L ]]L is the total sampling times in the sliding window; k is a constant parameter;

the smoothing unit is used for substituting the fitting parameter vector into the linear equation to obtain a smoothed second electromyographic signal corresponding to each sampling point;

the judgment and identification unit is used for judging whether the number of second electromyographic signals continuously larger than a threshold value in the second electromyographic signals exceeds a preset number, and if yes, the sampling time period corresponding to the second electromyographic signals continuously larger than the threshold value is an effective activity time period.

In an optional embodiment of the present application, the system further comprises a feedback module, configured to collect temperature data and sliding touch data of the prosthetic fingertip after the control driving of the prosthetic driving device based on the control action; performing temperature stimulation feedback control on the limb according to the temperature data; and performing sliding tactile stimulation feedback control on the limb according to the sliding tactile data.

In an optional embodiment of the present application, the feedback module is configured to perform feedback control on the temperature of the semiconductor chilling plate attached to the limb according to the temperature data.

In an optional embodiment of the present application, the feedback module is configured to perform feedback control on the magnitude of the stimulation current applied to the conductive patch of the limb according to the trolley sensation data.

The prosthesis control device of this embodiment is used to implement the prosthesis control method, and therefore, specific embodiments of the prosthesis control device can be found in the above embodiments of the prosthesis control method, for example, the signal acquisition module 100, the active segment identification module 200, the feature extraction module 300, the motion classification module 400, and the control driving module 500, and are respectively used to implement steps S11, S12, S13, S14, and S15 in the prosthesis control method, so that specific embodiments thereof may refer to descriptions of corresponding embodiments of each portion, and are not repeated herein.

The present application further provides embodiments of a prosthetic device comprising:

a prosthesis body;

the myoelectric sensor, the myosound sensor and the muscle force sensor are used for being attached to the corresponding limb of the artificial limb body;

the main controller is respectively connected with the electromyographic signal sensor, the muscle sound sensor and the muscle force sensor;

the main controller is used for acquiring myoelectric signals acquired by the myoelectric sensor, myoelectric signals acquired by the myoelectric sensor and myoelectric signals acquired by the myoelectric sensor, and executing the steps of the prosthesis control method to control the prosthesis body to act.

As shown in fig. 4, fig. 4 is a schematic diagram of a right-hand prosthetic hand provided by an embodiment of the present application, the prosthetic hand 20 is a bionic structure composed of a skeleton-muscle structure of a human hand, the skeleton is bionic by a 3D printing material, the muscle is bionic by a nylon rope and a spring rope, five groups of prosthetic fingers 21 correspond to five fingers of the human hand, wherein a thumb component is installed on a side surface of a palm support plate 22, and an index finger component, a middle finger component, a ring finger component and a little finger component are installed at a front end of the palm support plate 22. The artificial hand 20 comprises six steering engines 23 respectively installed at the joints of the finger assembly and the palm support plate 22 for driving the finger assembly to move. As shown in fig. 5, fig. 5 is a schematic structural diagram of the prosthetic finger in fig. 4, the prosthetic finger 21 is provided with a steering engine 23, a control line 24 formed by a nylon rope is controlled, the steering engine 23 rotates forwards to pull the control line 1 to control the bending of the prosthetic finger 21, and a spring 25 controls the prosthetic finger 21 to return to a straightened state.

Taking an artificial limb corresponding to an upper limb as an example, the multi-modal sensor device integrated with the myoelectric sensor, the muscular tone sensor and the muscular strength sensor may include 6 multi-modal sensors respectively distributed on the ulnar flexor carpus, the ulnar extensor carpus, the extensor digitorum, the brachioradialis, the superficial flexor digitorum, and the long flexor pollicis longus.

Each multimodal sensor apparatus combines the functionality of an EMG electrode and vibration detection and force probe; as shown in fig. 2, the three dry electrodes 11 collecting EMG can float in a very small range (about 0.5 mm) along their z-axis (direction perpendicular to the limb surface), the force beam 14 that transmits the skin pressure is connected to the FSR sensor 13 by a self-adhesive silicone rubber pad 15 and then fixed in the bottom housing case 10; since the self-adhesive silicone rubber pad 15 is elastic, it will act as a short "spring", and the electrodes will be kept under pre-tension without external force, and the TD-3 acceleration sensor 12 is located under the beam of the force beam 14.

Optionally, the prosthetic device further includes a temperature sensor and a slip sensation sensor, and may adopt a PVDF sensor formed by a PVDF piezoelectric film element, and since the PVDF sensor generates a charge signal under the external force touch condition, a signal conditioning circuit of the PVDF sensor needs to be designed to collect the slip sensation signal, the signal conditioning circuit may include a charge amplifier, a filter amplifier, a main amplification circuit, and a trap circuit, and the specific circuit may refer to a signal conditioning circuit commonly used by the PVDF sensor in the conventional technology, which is not described in detail in this application.

Optionally, the artificial hand can further comprise an electrode plate, the electrode plate is combined with a slip sensation sensor to generate feedback on a human body, the slip sensation sensor collects the gripping state of the artificial hand in real time, and the pressure collected by the slip sensation sensor is divided into 5 different grades: i, II, III, IV and V respectively correspond to different electrical stimulation modes in 5: i, II, III, IV and V, wherein the electrical stimulation intensity is in positive correlation with the pressure acquired by the touch and slide sensation sensor. Wherein, the stimulation grade V is only the micro current stimulation wearer and has no harm to human body.

Optionally, the prosthetic device further comprises a temperature sensor for detecting the temperature of the surface of the prosthesis contacting the object.

The corresponding semiconductor refrigeration piece that still includes, this semiconductor refrigeration piece combines temperature sensor to produce temperature feedback to the human body, and temperature sensor gathers artificial limb fingertip temperature in real time, for comfort level and the accurate nature of improvement feedback artificial limb temperature to limbs, the semiconductor refrigeration piece has adopted closed-loop control, and both ends can one section heat absorption cooling, and the other end can release heat. In order to realize the real-time feedback capability of the temperature control system, a common PID algorithm is adopted to realize accurate feedback of the temperature. In practical situations, the damage of low temperature to the artificial limb is far lower than high temperature, the temperature tolerance of the PLA material adopted by the artificial limb is better, the temperature contacted in daily life cannot cause overlarge physical damage to the artificial limb, and when the artificial limb is in overhigh temperature (more than 80 ℃), a user is reminded to control the artificial limb to leave a temperature source by arranging an alarm device, so that the protection and daily temperature feedback perception functions in high temperature are achieved.

Furthermore, a power supply module is also configured in the prosthetic device, the power supply adaptation voltage of the main controller is generally 5V, and a 5V/2A lithium battery can be adopted for supplying power.

The present application further provides an embodiment of a computer-readable storage medium having a computer program stored therein for execution by a master controller to implement the steps of the prosthesis control method as described in any one of the above.

The computer-readable storage medium may include Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.

The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.