Electroencephalogram signal classification method and device, computer equipment and storage medium

文档序号：293392 发布日期：2021-11-26 浏览：12次中文

阅读说明：本技术 脑电信号分类方法、装置、计算机设备及存储介质 (Electroencephalogram signal classification method and device, computer equipment and storage medium ) 是由柳露艳马锴郑冶枫于 2021-02-26 设计创作，主要内容包括：本申请是脑电信号分类方法、装置、计算机设备及存储介质,涉及信号处理技术领域。所述方法包括：获取第一脑电信号；基于至少两个电极信号,获取至少两个电极信号分别对应的时频特征图；基于至少两个电极信号分别对应的时频特征图,进行特征提取,获得第一提取特征图；对第一提取特征图,进行基于注意力机制的加权处理,获取注意力特征图；基于注意力特征图,获取第一脑电信号对应的运动想象类型。上述方案,通过融合脑电信号的时域、频域以及空间特征的第一提取特征图经过基于注意力机制的加权处理后得到注意力特征图,通过该注意力特征图确定第一脑电信号对应的运动想象类型,提高了对脑电信号对应的运动想象类型预测的准确性。(The application relates to an electroencephalogram signal classification method, an electroencephalogram signal classification device, computer equipment and a storage medium, and relates to the technical field of signal processing. The method comprises the following steps: acquiring a first electroencephalogram signal; acquiring time-frequency characteristic graphs respectively corresponding to the at least two electrode signals based on the at least two electrode signals; extracting features based on time-frequency feature maps respectively corresponding to at least two electrode signals to obtain a first extracted feature map; performing weighting processing based on an attention mechanism on the first extracted feature map to obtain an attention feature map; and acquiring a motor imagery type corresponding to the first brain electrical signal based on the attention feature map. According to the scheme, the attention feature map is obtained by fusing the first extracted feature map of the time domain, the frequency domain and the spatial features of the electroencephalogram signal and performing weighting processing based on the attention mechanism, the motor imagery type corresponding to the first electroencephalogram signal is determined through the attention feature map, and the accuracy of predicting the motor imagery type corresponding to the electroencephalogram signal is improved.)

1. An electroencephalogram signal classification method, characterized in that the method comprises:

acquiring a first electroencephalogram signal; the first electroencephalogram signal comprises at least two electrode signals; the electrode signals are used for indicating brain wave signals generated by a target object in a space region corresponding to the electrode signals;

acquiring time-frequency characteristic graphs corresponding to the at least two electrode signals respectively based on the at least two electrode signals; the time-frequency feature map is used for indicating time-domain features and frequency-domain features corresponding to the electrode signals;

extracting features based on time-frequency feature maps corresponding to at least two electrode signals respectively to obtain a first extracted feature map; the first extraction feature map is fused with the spatial features of at least two electrode signals; spatial features of at least two of the electrode signals are associated with spatial regions to which the at least two of the electrode signals correspond;

performing weighting processing based on an attention mechanism on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal;

and acquiring a motor imagery type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal.

2. The method according to claim 1, wherein performing feature extraction based on time-frequency feature maps corresponding to at least two of the electrode signals, respectively, to obtain a first extracted feature map comprises:

based on time-frequency characteristic graphs corresponding to at least two electrode signals respectively, performing characteristic extraction through a first convolution layer in an electroencephalogram classification model to obtain a first extracted characteristic graph;

the performing attention-based weighting processing on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal includes:

based on a first attention weighting module in the electroencephalogram signal classification model, carrying out weighting processing based on an attention mechanism on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal;

the electroencephalogram signal classification model is a machine learning model trained by taking a first sample electroencephalogram signal as a sample and taking a motor imagery type corresponding to the first sample electroencephalogram signal as a label.

3. The method of claim 2, wherein the attention mechanism comprises at least one of a spatial attention mechanism and a channel attention mechanism.

4. The method of claim 3, wherein the first attention weighting module comprises a first spatial attention weighting module, a second convolutional layer, a first channel attention module, and a third convolutional layer;

the attention-based weighting module in the electroencephalogram signal classification model performs attention-based weighting processing on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal, and the attention feature map comprises:

based on the first spatial attention weighting module, carrying out weighting processing based on a spatial attention mechanism on the first extracted feature map to obtain a first spatial feature map;

performing feature extraction on the first spatial feature map based on the second convolutional layer to obtain a second extracted feature map;

based on the first channel attention weighting module, carrying out weighting processing based on a channel attention mechanism on the second extracted feature map to obtain a first channel feature map;

performing feature extraction on the first channel feature map based on a third convolutional layer to obtain a third extracted feature map;

and acquiring the attention feature map based on the first spatial feature map, the first channel feature map and the third extraction feature map.

5. The method of claim 4, further comprising a second attention weighting module in the first attention weighting module;

the obtaining the attention feature map based on the first spatial feature map, the first channel feature map and the third extracted feature map includes:

fusing the first spatial feature map, the first channel feature map and the third extraction feature map to obtain a first fused feature map;

and performing weighting processing based on an attention mechanism through the second attention weighting module based on the first fusion feature map to obtain the attention feature map.

6. The method of claim 5, wherein obtaining the attention feature map by performing an attention-based weighting process by a second attention weighting module based on the first fused feature map in response to the second attention weighting module comprising a second spatial attention weighting module and a second channel attention weighting module comprises:

performing weighting processing based on a channel attention mechanism on the first fusion feature map through the second channel attention weighting module to obtain a second channel feature map;

and performing weighting processing based on a spatial attention mechanism on the second channel feature map through the second spatial attention weighting module to obtain the attention feature map.

7. The method according to any one of claims 1 to 6, wherein the obtaining a time-frequency feature map corresponding to each of the at least two electrode signals based on the at least two electrode signals comprises:

and performing continuous wavelet transformation based on the at least two electrode signals to obtain time-frequency characteristic graphs respectively corresponding to the at least two electrode signals.

8. The method of claim 7, wherein the brain electrical signal classification model further comprises a first fully connected layer;

the acquiring of the motor imagery type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal comprises:

based on the attention feature map corresponding to the first electroencephalogram signal, data processing is carried out through the first full-connection layer, and a feature vector corresponding to the first electroencephalogram signal is obtained;

acquiring probability distribution corresponding to the first electroencephalogram signal based on the feature vector corresponding to the first electroencephalogram signal; the probability distribution is used for indicating the probability that the first electroencephalogram signal is of various motor imagery types respectively;

and determining the motor imagery type corresponding to the first electroencephalogram signal based on the probability distribution corresponding to the first electroencephalogram signal.

9. An electroencephalogram signal classification method, characterized in that the method comprises:

acquiring a first sample brain electrical signal; the first sample brain electrical signal comprises at least two first sample electrode signals; the first sample electrode signal is used for indicating a brain wave signal generated in a space region corresponding to the first sample electrode signal when a target object is in motor imagery;

acquiring first sample time-frequency characteristic graphs respectively corresponding to at least two first sample electrode signals based on the at least two first sample electrode signals; the sample time-frequency characteristic graph is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the first sample electrode signal;

based on first sample time-frequency feature maps corresponding to at least two first sample electrode signals respectively, performing feature extraction through a first convolution layer in the electroencephalogram signal classification model to obtain a first sample extraction feature map; the first sample extraction feature map is fused with the spatial features of at least two first sample electrode signals; spatial features of at least two of the first sample electrode signals are associated with spatial regions to which at least two of the first sample electrode signals correspond;

extracting a feature map from the first sample based on an attention weighting module in the electroencephalogram signal classification model, and performing weighting processing based on an attention mechanism to obtain an attention feature map corresponding to the electroencephalogram signal of the first sample;

acquiring sample probability distribution corresponding to the first sample electroencephalogram signal based on the attention feature map corresponding to the first sample electroencephalogram signal; the sample probability distribution is used for indicating the probability that the first sample electroencephalogram signal is of various motor imagery types respectively;

training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal;

the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

10. The method of claim 9, further comprising:

acquiring a second sample electroencephalogram signal; the second sample brain electrical signal comprises at least two second sample electrode signals; the second sample electrode signal is used for indicating a brain wave signal generated in a space region corresponding to the second sample electrode signal when the target object is in motor imagery;

acquiring second sample time-frequency characteristic graphs respectively corresponding to the at least two second sample electrode signals on the basis of the at least two second sample electrode signals; the second sample time-frequency feature map is used for indicating time-domain features and frequency-domain features corresponding to the second sample electrode signals;

performing feature extraction through a first convolution layer in the electroencephalogram signal classification model based on second sample time-frequency feature maps corresponding to at least two second sample electrode signals respectively to obtain a second sample extraction feature map; the second sample extraction feature map is fused with the spatial features of at least two second sample electrode signals; spatial features of at least two of the second sample electrode signals are correlated with spatial regions to which at least two of the second sample electrode signals correspond;

extracting a feature map from the second sample based on an attention weighting module in the electroencephalogram signal classification model, and performing weighting processing based on an attention mechanism to obtain an attention feature map corresponding to the electroencephalogram signal of the second sample;

the training of the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal further comprises:

training the electroencephalogram signal classification model based on the motor imagery type corresponding to the sample probability distribution and the first sample electroencephalogram signal, the attention feature map corresponding to the first sample electroencephalogram signal and the attention feature map corresponding to the second sample electroencephalogram signal.

11. The method of claim 10, wherein the second sample brain electrical signal is indicative of brain wave signals generated by a target subject corresponding to the first sample brain electrical signal at different times;

or the second sample electroencephalogram signal is used for indicating electroencephalogram signals generated by other objects except the target object corresponding to the first sample electroencephalogram signal.

12. An electroencephalogram signal classification device, characterized in that said device comprises:

the first signal acquisition module is used for acquiring a first electroencephalogram signal; the first electroencephalogram signal comprises at least two electrode signals; the electrode signals are used for indicating brain wave signals generated by a target object in a space region corresponding to the electrode signals;

the first time-frequency characteristic acquisition module is used for acquiring time-frequency characteristic graphs corresponding to at least two electrode signals respectively based on the at least two electrode signals; the time-frequency feature map is used for indicating time-domain features and frequency-domain features corresponding to the electrode signals;

the first extraction feature acquisition module is used for extracting features based on time-frequency feature maps respectively corresponding to at least two electrode signals to obtain a first extraction feature map; the first extraction feature map is fused with the spatial features of at least two electrode signals; spatial features of at least two of the electrode signals are associated with spatial regions to which the at least two of the electrode signals correspond;

the first attention feature acquisition module is used for performing weighting processing based on an attention mechanism on the first extracted feature map to acquire an attention feature map corresponding to the first electroencephalogram signal;

the imagination type acquisition module is used for acquiring a motor imagination type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal;

13. An electroencephalogram signal classification device, characterized in that said device comprises:

the first sample acquisition module is used for acquiring a first sample electroencephalogram signal; the first sample brain electrical signal comprises at least two first sample electrode signals; the first sample electrode signal is used for indicating a brain wave signal generated in a space region corresponding to the first sample electrode signal when a target object is in motor imagery;

the first sample time-frequency obtaining module is used for obtaining first sample time-frequency characteristic graphs corresponding to at least two first sample electrode signals respectively based on the at least two first sample electrode signals; the sample time-frequency characteristic graph is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the first sample electrode signal;

the first sample extraction and acquisition module is used for performing feature extraction through a first convolution layer in the electroencephalogram classification model based on first sample time-frequency feature maps corresponding to at least two first sample electrode signals respectively to obtain a first sample extraction feature map; the first sample extraction feature map is fused with the spatial features of at least two first sample electrode signals; spatial features of at least two of the first sample electrode signals are associated with spatial regions to which at least two of the first sample electrode signals correspond;

the first sample attention acquisition module is used for extracting a feature map from the first sample based on an attention weighting module in the electroencephalogram signal classification model, carrying out weighting processing based on an attention mechanism and acquiring an attention feature map corresponding to the electroencephalogram signal of the first sample;

the first sample probability acquisition module is used for acquiring sample probability distribution corresponding to the first sample electroencephalogram signal based on the attention feature map corresponding to the first sample electroencephalogram signal; the sample probability distribution is used for indicating the probability that the first sample electroencephalogram signal is of various motor imagery types respectively;

the first training module is used for training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal;

the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

14. A computer device comprising a processor and a memory, said memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, said at least one instruction, said at least one program, said set of codes, or set of instructions being loaded and executed by said processor to implement the electroencephalogram signal classification method according to any one of claims 1 to 11.

15. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the brain electrical signal classification method of any one of claims 1 to 11.

Technical Field

The present application relates to the field of signal processing, and in particular, to a method and an apparatus for classifying electroencephalograms, a computer device, and a storage medium.

Background

Electroencephalography (EEG) is a method of recording brain activity using electrophysiological markers, in which post-synaptic potentials generated in synchronization with a large number of neurons are summed up during brain activity. It records the electrical wave changes during brain activity, which is a general reflection of the electrophysiological activity of brain neurons on the surface of the cerebral cortex or scalp.

In the related art, an MI-BCI (Motor image-Brain Computer Interface) system has a wide application prospect in many fields, and can control external devices through Brain wave signals generated by the Brain in imagination of limb movements without any limb movements. The intelligent wheelchair can help patients with limb inconvenience such as cerebral apoplexy, hemiplegia and the like to perform rehabilitation training or control wheelchair to go out, and can also be used for educational entertainment of common users, such as brain-controlled VR (Virtual Reality) games and the like, MI (Motor image) signal classification identification is a key link in an MI-BCI system, and the decoding accuracy directly influences the performance and user experience of the system.

In the technical scheme, when the MI signals are classified and identified, the classification accuracy is low.

Disclosure of Invention

The embodiment of the application provides an electroencephalogram signal classification method, an electroencephalogram signal classification device, computer equipment and a storage medium, which can improve the accuracy of prediction of motor imagery types corresponding to electroencephalogram signals, and the technical scheme is as follows:

in one aspect, a method for classifying electroencephalogram signals is provided, the method comprising:

performing weighting processing based on an attention mechanism on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal;

and acquiring a motor imagery type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal.

In another aspect, a method for classifying brain electrical signals is provided, the method comprising:

training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal;

the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

In still another aspect, an electroencephalogram signal classification apparatus is provided, the apparatus including:

In a possible implementation manner, the first extracted feature obtaining module includes:

the first extracted feature map acquisition unit is used for extracting features through a first convolution layer in an electroencephalogram classification model based on time-frequency feature maps corresponding to at least two electrode signals respectively to obtain a first extracted feature map;

the first attention feature acquisition module comprising:

an attention feature obtaining unit, configured to perform attention-based weighting processing on the first extracted feature map based on a first attention weighting module in the electroencephalogram classification model, so as to obtain an attention feature map corresponding to the first electroencephalogram;

In one possible implementation, the attention mechanism includes at least one of a spatial attention mechanism and a channel attention mechanism.

the first attention feature map acquisition unit includes:

the first spatial weighting subunit is configured to perform, based on the first spatial attention weighting module, weighting processing based on a spatial attention mechanism on the first extracted feature map to obtain a first spatial feature map;

a second feature obtaining subunit, configured to perform feature extraction on the first spatial feature map based on the second convolutional layer, so as to obtain a second extracted feature map;

the first channel weighting subunit is configured to perform, based on the first channel attention weighting module, weighting processing based on a channel attention mechanism on the second extracted feature map to obtain a first channel feature map;

a third feature obtaining subunit, configured to perform feature extraction on the first channel feature map based on the third convolutional layer, so as to obtain a third extracted feature map;

an attention feature obtaining subunit, configured to obtain the attention feature map based on the first spatial feature map, the first channel feature map, and the third extracted feature map.

In a possible implementation manner, the first attention weighting module further includes a second attention weighting module;

the attention feature acquisition subunit further includes:

a first fusion subunit, configured to fuse the first spatial feature map, the first channel feature map, and the third extracted feature map to obtain a first fused feature map;

and the first attention weighting subunit is used for performing weighting processing based on an attention mechanism through the second attention weighting module based on the first fusion feature map to obtain the attention feature map.

In one possible implementation manner, in response to the second attention weighting module including a second spatial attention weighting module and a second channel attention weighting module, the first attention weighting subunit further includes:

the second channel attention weighting subunit is configured to perform, by the second channel attention weighting module, weighting processing based on a channel attention mechanism on the first fusion feature map to obtain a second channel feature map;

and the second spatial attention weighting subunit is configured to perform, by the second spatial attention weighting module, weighting processing based on a spatial attention mechanism on the second channel feature map to obtain the attention feature map.

In a possible implementation manner, the first time-frequency characteristic obtaining module includes:

and the electrode time-frequency signal acquisition unit is used for carrying out continuous wavelet transformation on the basis of at least two electrode signals to acquire time-frequency characteristic graphs corresponding to the at least two electrode signals respectively.

In one possible implementation, the electroencephalogram classification model further includes a first fully connected layer;

the imagination type obtaining module is also used for,

and determining the motor imagery type corresponding to the first electroencephalogram signal based on the probability distribution corresponding to the first electroencephalogram signal.

In still another aspect, an electroencephalogram signal classification apparatus is provided, the apparatus including:

the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

In one possible implementation, the apparatus further includes:

the second electroencephalogram signal acquisition module is used for acquiring a second sample electroencephalogram signal; the second sample brain electrical signal comprises at least two second sample electrode signals; the second sample electrode signal is used for indicating a brain wave signal generated in a space region corresponding to the second sample electrode signal when the target object is in motor imagery;

a second sample time-frequency characteristic diagram obtaining module, configured to obtain, based on at least two second sample electrode signals, second sample time-frequency characteristic diagrams corresponding to the at least two second sample electrode signals, respectively; the second sample time-frequency feature map is used for indicating time-domain features and frequency-domain features corresponding to the second sample electrode signals;

the second sample extraction feature map acquisition module is used for performing feature extraction through a first convolution layer in the electroencephalogram signal classification model based on second sample time-frequency feature maps corresponding to at least two second sample electrode signals respectively to obtain a second sample extraction feature map; the second sample extraction feature map is fused with the spatial features of at least two second sample electrode signals; spatial features of at least two of the second sample electrode signals are correlated with spatial regions to which at least two of the second sample electrode signals correspond;

a second attention feature obtaining module, configured to extract a feature map from the second sample based on an attention weighting module in the electroencephalogram classification model, perform weighting processing based on an attention mechanism, and obtain an attention feature map corresponding to the electroencephalogram of the second sample;

the first model training module is further configured to,

In one possible implementation manner, the second sample electroencephalogram signal is used for indicating electroencephalogram signals generated by a target object corresponding to the first sample electroencephalogram signal at different moments;

In yet another aspect, a computer-readable storage medium is provided, having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement the brain electrical signal classification method described above.

In yet another aspect, a computer program product or computer program is provided that includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the electroencephalogram signal classification method.

The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:

acquiring an electroencephalogram signal containing at least two electrode signals, acquiring a time-frequency characteristic diagram according to the at least two electroencephalogram signals, wherein the time-frequency characteristic diagram is used for indicating time-domain characteristics and frequency-domain characteristics of the first electroencephalogram signal, performing characteristic extraction on the time-frequency characteristic diagram to obtain a first extracted characteristic diagram, weighting characteristics of different levels of the extracted first extracted characteristic diagram based on an attention mechanism to obtain a weighted attention characteristic diagram, and finally determining a motor imagery type corresponding to the electroencephalogram signal through the weighted attention characteristic diagram. In the scheme, the time-frequency characteristic diagram is a time-frequency characteristic diagram corresponding to brain wave signals generated by a target object in regions corresponding to different electrode signals, namely the time-frequency characteristic diagram also contains the spatial relationship among different electrode signals, so that the time-frequency characteristic diagram is subjected to characteristic extraction through an electroencephalogram signal classification model, the time-domain characteristics and the frequency-domain characteristics of the electroencephalogram signals can be simultaneously considered, the characteristic diagram extracted from the time-frequency characteristic diagram is subjected to weighting processing through an attention mechanism, the spatial relationship among at least two electrode signals of the electroencephalogram signals can be considered, therefore, the finally obtained attention characteristic diagram is extracted after the time-domain characteristics, the frequency-domain characteristics and the space-domain characteristics of the electroencephalogram signals are simultaneously fused, and on the basis of ensuring the level diversity of image characteristics, the attention mechanism ensures that the image characteristics pay more attention to rich characteristic positions in the characteristic diagram, the motor imagery type corresponding to the first electroencephalogram signal is determined through the attention feature map, and accuracy of prediction of the motor imagery type corresponding to the electroencephalogram signal is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 illustrates a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

FIG. 2 is a schematic flow diagram illustrating a method of classification of brain electrical signals according to an exemplary embodiment;

FIG. 3 is a schematic flow diagram illustrating a method of classification of brain electrical signals according to an exemplary embodiment;

FIG. 4 is a method flow diagram illustrating a method of classification of brain electrical signals according to an exemplary embodiment;

FIG. 5 is a channel scale stitching diagram of the time-frequency feature map according to the embodiment shown in FIG. 4;

FIG. 6 is a schematic diagram of a feature hierarchy according to the embodiment shown in FIG. 4;

FIG. 7 is a schematic diagram of a spatial attention mechanism associated with the embodiment of FIG. 4;

FIG. 8 is a schematic view of a channel attention mechanism according to the embodiment of FIG. 4;

FIG. 9 is a schematic diagram illustrating a principle of an electroencephalogram classification model according to the embodiment shown in FIG. 4;

FIG. 10 is a diagram illustrating an application of a classification model for electroencephalogram signals according to the embodiment shown in FIG. 4;

FIG. 11 is a block diagram illustrating model training and model application flow, according to an exemplary embodiment;

FIG. 12 is a block diagram illustrating the structure of an electroencephalogram signal classification apparatus according to an exemplary embodiment;

FIG. 13 is a block diagram illustrating the structure of an electroencephalogram signal classification apparatus according to an exemplary embodiment;

FIG. 14 is a schematic diagram illustrating a configuration of a computer device, according to an example embodiment.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

First, terms related to embodiments of the present application will be described.

1) Artificial Intelligence (AI)

Artificial intelligence is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

2) Machine Learning (Machine Learning, ML)

Machine learning is a multi-field cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

3) Transfer learning (Transfer learning)

The transfer learning is a machine learning method, namely, a model developed for a task A is taken as an initial point and is reused in the process of developing the model for a task B. Transfer learning is a method of machine learning, meaning that a pre-trained model is reused in another task. However, migration learning is very popular in some deep learning problems, such as where there are a large number of resources required to train a deep model or where there are a large number of data sets used to pre-train a model. The migration learning only works if the depth model features in the first task are generalization features. This migration in deep learning is referred to as inductive migration. It is an advantageous way to narrow the search range of possible models by using a model that is suitable for different but related tasks.

4) Continuous Wavelet Transform (Continuous Wavelet Transform, CWT)

Continuous wavelet transform is an operation that decomposes a signal into different frequency components that vary with time. Although the Fourier Transform and its Discrete Fourier Transform (DFT) have become the most common tools in signal processing, especially time-frequency analysis, the Fourier Transform has a problem that the time-domain and frequency-domain information of the signal cannot be localized at the same time. Continuous wavelets are the convolution of a function (from negative infinite to positive infinite integration to zero), which may be referred to as a wavelet, with the signal to be processed at some scale. The scale of the wavelet function is changed, so that the band-pass range of the filter is changed, and the wavelet coefficient under each corresponding scale reflects the information of the corresponding band-pass. In essence, a continuous wavelet is a set of multi-scale filters that can control the range of the pass band.

5) Channel (channel)

In a convolutional neural network, a channel can be used to indicate a feature map (featuremap), the strength of a certain point in the channel can represent the numerical value of the feature map at the point, different channels are used to indicate feature maps of different dimensions, and for a feature map with multiple channels, the meaning is that the feature map has image features of multiple dimensions. There are two main operations in a convolutional network, one Convolution (Convolution) and one Pooling (Pooling). Wherein, the pooling layer does not affect the interaction between the channels, but only operates in each channel; the convolutional layer can interact between channels, and then a new channel is generated in the next layer.

The electroencephalogram signal classification method provided by the embodiment of the application can be applied to computer equipment with high data processing capacity. In a possible implementation manner, the electroencephalogram classification method provided by the embodiment of the present application can be applied to a personal computer, a workstation, or a server, that is, training of an electroencephalogram classification model can be performed through the personal computer, the workstation, or the server. In a possible implementation manner, the electroencephalogram classification model trained by the electroencephalogram classification method provided by the embodiment of the application can be applied to classification of electroencephalograms, that is, data processing is performed on the acquired electroencephalograms generated by the head during human motor imagery, so that motor imagery types corresponding to the electroencephalograms are obtained.

Referring to FIG. 1, a schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 200 includes a terminal 110 and a server 120, wherein the terminal 110 and the server 120 perform data communication through a communication network, optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

The terminal 110 is installed with an application program having an electroencephalogram signal processing function, and the application program may be a virtual reality application program, a game application program, or an Artificial Intelligence (AI) application program having an electroencephalogram signal processing function, which is not limited in this embodiment of the present application.

Optionally, the terminal 110 may be a terminal device having a brain-computer interface, and the brain-computer interface may obtain an electroencephalogram signal of the head of the target object through the electrode; or the computer equipment is provided with a data transmission interface which is used for receiving the electroencephalogram signals collected by the data collection equipment with the brain-computer interface.

Optionally, the terminal 110 may be a mobile terminal such as a smart phone, a tablet computer, a laptop portable notebook computer, or the like, or a terminal such as a desktop computer, a projection computer, or the like, or an intelligent terminal having a data processing component, which is not limited in this embodiment of the application.

The server 120 may be implemented as one server, or may be implemented as a server cluster formed by a group of servers, which may be physical servers, or may be implemented as a cloud server. In one possible implementation, the server 120 is a backend server for applications in the terminal 110.

In a possible implementation manner of this embodiment, the server 120 trains the electroencephalogram classification model through a preset training sample set (i.e., a sample electroencephalogram signal), where the training sample set may include sample electroencephalograms corresponding to a plurality of motor imagery types. After the training process of the electroencephalogram signal classification model by the server 120 is completed, the trained electroencephalogram signal classification model is sent to the terminal 110 through wired or wireless connection. The terminal 110 receives the trained electroencephalogram signal classification model and inputs data information corresponding to the electroencephalogram signal classification model into an application program with an electroencephalogram signal processing function, so that when a user uses the application program to process an electroencephalogram signal, the electroencephalogram signal can be processed according to the trained electroencephalogram signal classification model, and all or part of steps of the electroencephalogram signal classification method can be realized.

FIG. 2 is a flow diagram illustrating a method of classification of brain electrical signals according to an exemplary embodiment. The method may be performed by a computer device, which may be the terminal 120 in the embodiment shown in fig. 1 described above. As shown in fig. 2, the flow of the electroencephalogram signal classification method may include the following steps:

step 201, acquiring a first electroencephalogram signal; the first brain electrical signal comprises at least two electrode signals; the electrode signal is used for indicating brain wave signals generated by the target object in a space region corresponding to the electrode signal.

In one possible implementation, the first brain electrical signal is a brain wave signal of a target object acquired through a device having a brain-computer interface, the brain-computer interface has at least two electrodes, and the two electrodes are located in different spatial regions of the head of the target object during signal acquisition of the target object through the brain-computer interface so as to acquire brain wave signals generated by the different spatial regions of the target object.

Step 202, based on at least two electrode signals, obtaining time-frequency characteristic graphs corresponding to the at least two electrode signals respectively.

The time-frequency characteristic diagram is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the electrode signals.

Step 203, extracting features based on the time-frequency feature maps corresponding to the at least two electrode signals, respectively, to obtain a first extracted feature map.

The first extraction feature map is fused with at least two spatial features of the electrode signals; the spatial characteristics of at least two of the electrode signals are associated with at least two spatial regions to which the electrode signals correspond.

The first extracted feature map is obtained by extracting features based on time-frequency feature maps corresponding to the at least two electrode signals respectively, and spatial regions corresponding to the at least two electrode signals are different, so that the spatial regions corresponding to the at least two electrode signals are considered while extracting features based on the time-frequency feature maps corresponding to the at least two electrode signals respectively. And the fused first extracted feature map is obtained based on the time-frequency feature maps respectively corresponding to the at least two electrode signals, that is, the first extracted feature map is fused with information related to the spatial region in the at least two electrode signals, that is, the spatial features of the at least two electrode signals.

And 204, performing weighting processing based on an attention mechanism on the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal.

In a possible implementation manner, the features of different levels in the first extracted feature map are subjected to weighting processing based on an attention mechanism, and an attention feature map corresponding to the first electroencephalogram signal is obtained.

The features of different levels in the first extracted feature map are used for indicating the features obtained by different feature extraction modes in the first extracted feature map.

For example, feature extraction is performed on the first extracted feature map through a first feature extraction mode to obtain a first-level feature, and information in the first-level feature corresponds to the first feature extraction mode; and performing feature extraction on the first-level features by using a second feature extraction mode to obtain second-level features, wherein the second-level features are obtained by performing feature extraction on the first-level features and the second-level features by using the first feature extraction mode and the second feature extraction mode based on the first extraction feature map, so that the information in the second-level features simultaneously contains the characteristics of the first feature extraction mode and the second feature extraction mode.

Step 205, acquiring a motor imagery type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal.

In this embodiment of the application, the attention feature map corresponding to the first electroencephalogram signal is obtained based on time-frequency feature maps corresponding to the at least two electrode signals, and the at least two electrode signals are used to indicate electroencephalograms generated in different spatial regions of the target object, so that the attention feature map has certain spatial features, that is, the extracted attention feature map considers time-domain features, frequency-domain features, and spatial-domain features at the same time.

In summary, in the scheme shown in the embodiment of the application, an electroencephalogram including at least two electrode signals is obtained, a time-frequency feature map is obtained according to the at least two electroencephalograms, the time-frequency feature map is used for indicating time-domain features and frequency-domain features of the first electroencephalogram, feature extraction is performed on the time-frequency feature map to obtain a first extracted feature map, features of different levels of the extracted first extracted feature map are weighted based on an attention mechanism to obtain a weighted attention feature map, and finally, a motor imagery type corresponding to the electroencephalogram is determined through the weighted attention feature map. In the scheme, the time-frequency characteristic diagram is a time-frequency characteristic diagram corresponding to brain wave signals generated by a target object in regions corresponding to different electrode signals, namely the time-frequency characteristic diagram also contains the spatial relationship among different electrode signals, so that the time-frequency characteristic diagram is subjected to characteristic extraction through an electroencephalogram signal classification model, the time-domain characteristics and the frequency-domain characteristics of the electroencephalogram signals can be simultaneously considered, the characteristic diagram extracted from the time-frequency characteristic diagram is subjected to weighting processing through an attention mechanism, the spatial relationship among at least two electrode signals of the electroencephalogram signals can be considered, therefore, the finally obtained attention characteristic diagram is extracted after the time-domain characteristics, the frequency-domain characteristics and the space-domain characteristics of the electroencephalogram signals are simultaneously fused, and on the basis of ensuring the level diversity of image characteristics, the attention mechanism ensures that the image characteristics pay more attention to rich characteristic positions in the characteristic diagram, the motor imagery type corresponding to the first electroencephalogram signal is determined through the attention feature map, and accuracy of prediction of the motor imagery type corresponding to the electroencephalogram signal is improved.

FIG. 3 is a flow diagram illustrating a method of classification of brain electrical signals according to an exemplary embodiment. The method may be performed by a computer device, which may be the server 120 in the embodiment illustrated in fig. 1 described above. As shown in fig. 3, the flow of the electroencephalogram signal classification method may include the following steps:

step 301, acquiring a first sample electroencephalogram signal; the first sample brain electrical signal comprises at least two first sample electrode signals; the first sample electrode signal is used for indicating brain wave signals generated in a space region corresponding to the first sample electrode signal when the target object is in motor imagery.

The first sample electroencephalogram signal is used for training the electroencephalogram signal classification model, and the first sample electroencephalogram signal is used for indicating an electroencephalogram signal generated in an area corresponding to the sample electrode signal when the sample target object is in motor imagery, so that the electroencephalogram signal classification model at the position where the first sample electroencephalogram signal is trained can be used for analyzing the motor imagery type of the electroencephalogram signal.

Step 302, based on at least two first sample electrode signals, obtaining first sample time-frequency characteristic diagrams corresponding to the at least two first sample electrode signals respectively.

The sample time-frequency characteristic graph is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the first sample electrode signal.

Step 303, performing feature extraction through a first convolution layer in the electroencephalogram classification model based on first sample time-frequency feature maps corresponding to at least two first sample electrode signals, respectively, to obtain a first sample extracted feature map.

Wherein, the first sample extraction feature map is fused with at least two spatial features of the first sample electrode signals; spatial features of at least two of the first sample electrode signals are associated with spatial regions to which at least two of the first sample electrode signals correspond.

And 304, extracting a characteristic diagram of the first sample based on an attention weighting module in the electroencephalogram signal classification model, and performing weighting processing based on an attention mechanism to obtain an attention characteristic diagram corresponding to the electroencephalogram signal of the first sample.

305, acquiring sample probability distribution corresponding to the first sample electroencephalogram signal based on the attention feature map corresponding to the first sample electroencephalogram signal; the sample probability distribution is used for indicating the probability that the first sample electroencephalogram signal is of various motor imagery types respectively.

And step 306, training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal.

The electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

FIG. 4 is a method flow diagram illustrating a method of electroencephalogram classification, according to an exemplary embodiment. The method may be performed by a model training device, which may be the server 120 in the embodiment shown in fig. 1 and a signal processing device, which may be the terminal 120 in the embodiment shown in fig. 1. As shown in fig. 4, the flow of the electroencephalogram signal classification method may include the following steps:

step 401, a first sample electroencephalogram signal is acquired.

Wherein the first sample brain electrical signal comprises at least two first sample electrode signals.

In one possible implementation, the at least two first sample electrode signals of the first sample brain electrical signals may be brain wave signals generated by the head of the sample target subject during motor imagery, acquired through the electrodes of the brain-computer interface by a terminal device having the brain-computer interface. The number of the first sample electrode signals is the same as that of the electrodes corresponding to the brain-computer interface, that is, the brain-computer interface can simultaneously acquire electroencephalogram signals generated in different spatial regions of the head of the same sample target object through different electrodes during motor imagery.

In a possible implementation manner, the brain-computer interface obtains electroencephalogram signals generated by different areas of the head of the sample target object through electrodes connected to the sample target object, and the electrodes connected to the sample target object transmit the electroencephalogram signals corresponding to the electrodes to terminal equipment corresponding to the brain-computer interface through transmission lines.

In one possible implementation mode, based on each electrode of the brain-computer interface, acquiring an original sample electroencephalogram signal generated by the head of the sample target object during motor imagery; based on the original sample electroencephalogram signal, filtering processing is carried out through a band-pass filter, and the first sample electroencephalogram signal is obtained.

Because there is more noise interference in the original sample electroencephalogram signals obtained through the electrodes of the brain-computer interface, the original sample electroencephalogram signals need to be filtered through a band-pass filter, and the influence of irrelevant noise on the electroencephalogram signals is reduced.

In one possible implementation, each original sample EEG signal is subjected to 3-38Hz band-pass filtering to remove the influence of irrelevant physiological noise such as eye movement and power frequency interference (i.e. interference caused by the power system, usually 50Hz) on the EEG signal.

Step 402, obtaining first sample time-frequency characteristic diagrams corresponding to at least two first sample electrode signals respectively based on the at least two first sample electrode signals.

In a possible implementation manner, at least two first sample electrode signals of the first sample brain electrical signals are respectively subjected to a standardization operation, and at least two sample standard signals are obtained; and acquiring the first sample time-frequency characteristic diagram based on the at least two sample standard signals.

After the band-pass filtering processing of 3-38HZ is carried out on each original sample electroencephalogram signal, unrelated physiological noise and power frequency interference are filtered out to obtain a first sample electroencephalogram signal, but the first sample electroencephalogram signal still has noise which cannot be removed through the band-pass filtering, so that at least two sample electrode signals of the first sample electroencephalogram signal can be subjected to standardization operation in order to reduce signal disturbance caused by the noise. Wherein the normalization operation may be any one of an exponential weighted moving average operation, mean variance normalization, and co-spatial mode algorithm.

In a possible implementation manner, continuous wavelet transformation is performed based on the at least two first sample electrode signals, and the at least two first sample electrode signals respectively correspond to first sample time-frequency feature maps.

When the first sample electroencephalogram signal is subjected to data processing through continuous wavelet transformation, the first sample electroencephalogram signal can be fitted through a basis function corresponding to the continuous wavelet transformation, and different from Fourier transformation, a wavelet basis corresponding to the continuous wavelet transformation is influenced by time and frequency, so that a first sample time-frequency characteristic diagram obtained by performing continuous wavelet transformation on two first sample electrode signals of the first sample electroencephalogram signal contains time domain characteristics of the first sample electrode signal in the first sample electroencephalogram signal and frequency domain characteristics of the first sample electrode signal in the first sample electroencephalogram signal.

In the embodiment of the present application, the wavelet basis function corresponding to the continuous wavelet transform may be cmor3.0-3.0, and the wavelet basis function may also be any one of haar wavelet, db wavelet, sym wavelet, and coif series wavelet.

In a possible implementation manner, continuous wavelet transformation is performed based on the at least two first sample electrode signals, and time-frequency characteristic graphs respectively corresponding to the at least two first sample electrode signals are obtained; the time-frequency characteristic map is used for indicating the time-domain characteristics and the frequency-domain characteristics of the sample electrode signals; and acquiring a time-frequency characteristic diagram corresponding to the first sample brain electrical signal based on the first sample time-frequency characteristic diagrams respectively corresponding to the at least two first sample electrode signals.

When the first sample electroencephalogram signal contains at least two sample electrode signals, namely the first sample electroencephalogram signal comprises electroencephalogram signals generated by at least two areas of the head when a brain-computer interface obtains a sample target object motor imagery through at least two electrodes. And then, splicing the time-frequency characteristic graphs respectively corresponding to the at least two sample electrode signals according to channels to obtain the time-frequency characteristic graph corresponding to the first sample electroencephalogram signal.

The time-frequency characteristic diagram corresponding to the first sample electroencephalogram signal is provided with image characteristics of at least two channels, the image characteristics of the at least two channels respectively correspond to first sample time-frequency characteristic diagrams of at least two electrodes, that is, the time-frequency characteristic diagram corresponding to the first sample electroencephalogram signal is formed based on the first sample time-frequency characteristic diagrams corresponding to the at least two electrodes, and each channel in the time-frequency characteristic diagram corresponding to the first sample electroencephalogram signal respectively corresponds to the first sample time-frequency characteristic diagram corresponding to each electrode.

In a possible implementation manner, the image features of each channel in the time-frequency feature map corresponding to the first sample electroencephalogram signal are respectively determined according to the signal images of the time-frequency feature maps of the at least two electrodes. That is, after the time-frequency feature maps corresponding to the at least two sample electrode signals are obtained, the time-frequency feature maps corresponding to the at least two sample electrode signals can be obtained according to the time-frequency feature maps corresponding to the at least two sample electrode signals, and the time-frequency feature maps corresponding to the at least two first sample electrode signals are spliced according to the channels, so as to obtain the time-frequency feature map corresponding to the first sample brain electrical signal. At this time, the time-frequency characteristic diagram corresponding to the first sample electroencephalogram signal contains time-domain characteristics and frequency-domain characteristics of electroencephalogram signals generated by different areas of the head when the sample target object performs motor imagery.

Please refer to fig. 5, which illustrates a channel scale stitching diagram of a time-frequency feature map according to an embodiment of the present application. As shown in fig. 5, taking an example that the first electroencephalogram signal includes two electrodes to acquire electroencephalograms generated by two regions of a head when a target object is in motion imagery, a first electrode signal 501 corresponding to the first electroencephalogram signal is subjected to continuous wavelet transformation to obtain a first time-frequency feature map 503 corresponding to the first electrode signal; and performing continuous wavelet transformation on the second electrode signal 502 corresponding to the first electroencephalogram signal to obtain a second time-frequency characteristic diagram 504 corresponding to the second electrode signal 502. The first time-frequency feature map 503 and the second time-frequency feature map 504 are further spliced according to the channel scale, so as to obtain a time-frequency feature map 505 corresponding to the first electroencephalogram signal, which has two channels simultaneously to store all the features of the first time-frequency feature map 503 and the second time-frequency feature map 504.

Step 403, performing feature extraction through a first convolution layer in the electroencephalogram classification model based on first sample time-frequency feature maps corresponding to at least two first sample electrode signals, respectively, to obtain a first sample extracted feature map.

The first sample time-frequency characteristic diagrams corresponding to the at least two first sample electrode signals respectively can be subjected to feature extraction through a first convolution layer in the electroencephalogram classification model, and a first sample extraction characteristic diagram is obtained.

In a possible implementation manner, the image features of each channel in the first sample extracted feature map all include image features of the first sample time-frequency feature map corresponding to the at least two first sample electrode signals, that is, include image features of each channel in the first sample time-frequency feature map.

For example, when the first convolution layer includes convolution kernels of 3 × 3 and the number of the convolution kernels is 5, each convolution kernel in the first convolution layer performs convolution operation with each channel in the first sample time-frequency feature map and then sums up to obtain the image feature corresponding to the convolution kernel, so that when 5 convolution kernels perform convolution operation with the first sample time-frequency feature map, the image features of 5 channels, that is, the first sample extracted feature map with the channel of 5, can be obtained, and since the image features in each channel in the first sample extracted feature map are obtained by summation after convolution operation according to each channel, the image features in each channel all include the image features of each channel in the first sample time-frequency feature map, that is, the first convolution layer performs time-frequency feature extraction on the first sample time-frequency feature map, and the time-frequency features of each channel in the first sample time-frequency feature map are fused together, and each channel in the time-frequency characteristic diagram corresponding to the first electroencephalogram signal is an electroencephalogram signal of a sample object acquired by electrodes at different positions, so that the fused first sample extraction characteristic diagram is fused with the characteristics of at least two electrode signals, and the spatial characteristics of the at least two sample electrode signals are related to the spatial regions corresponding to the at least two sample electrode signals, so that the fused first sample extraction characteristic diagram is a characteristic diagram simultaneously having time domain characteristics, frequency domain characteristics and spatial characteristics.

And step 404, extracting a feature map from the first sample based on an attention weighting module in the electroencephalogram signal classification model, and performing weighting processing based on an attention mechanism to obtain an attention feature map corresponding to the electroencephalogram signal of the first sample.

In a possible implementation manner, features of different levels of the feature map are extracted from the first sample based on an attention weighting module in the electroencephalogram classification model, and weighting processing based on an attention mechanism is performed to obtain an attention feature map corresponding to the first sample electroencephalogram signal.

And the features of different layers in the first sample extraction feature map are used for indicating the features obtained after feature extraction is carried out on different convolution layers in the first sample extraction feature map.

Please refer to fig. 6, which illustrates a feature hierarchy diagram according to an embodiment of the present application. As shown in fig. 6, a feature map 601 is obtained by extracting features from a first convolutional layer 602 to obtain a first hierarchical feature map 603, and the first hierarchical feature map 603 can be used to indicate features of the feature map 601 corresponding to the first convolutional layer 602. The first hierarchical feature map 603 is subjected to feature extraction by a second convolutional layer 604 to obtain a second hierarchical feature map 605, and the second hierarchical feature map 605 is obtained by sequentially performing feature extraction on the feature map 601 by the first convolutional layer 602 and the second convolutional layer 604, so that the second hierarchical feature map indicates features corresponding to the first convolutional layer 602 and the second convolutional layer 604 at the same time, and thus the second hierarchical feature map and the first hierarchical feature map are features of different layers.

In one possible implementation, the attention mechanism includes at least one of a spatial attention mechanism and a channel attention mechanism.

Please refer to fig. 7, which illustrates a schematic diagram of a spatial attention mechanism according to an embodiment of the present application. As shown in fig. 7, for a feature map 701 with a size W × H and a channel C, feature maps of all channels are averaged to obtain an average feature map, the average feature map is transformed by a learnable convolutional layer to form a spatial attention value, and finally the spatial attention value is multiplied by feature maps of all channels to form a spatial attention feature map 702.

Since the spatial attention mechanism is based on the feature maps of all channels, averaging is performed according to each region in the feature maps, and finally an average feature map integrating the features of the feature maps of all channels is obtained, where the average feature map is used to indicate the region with the most features in the feature maps of different channels with the same size, the spatial attention value formed by the average feature map is weighted to obtain a spatial attention feature map, which focuses more on the region with the most features in each feature map.

Please refer to fig. 8, which illustrates a schematic diagram of a channel attention mechanism according to an embodiment of the present application. As shown in fig. 8. For a feature map 801 with a channel C and a size of W × H, mean values of each feature map are pooled to obtain a feature map mean value corresponding to each of the C channels, the mean value corresponding to each channel is mapped through a full connection layer to form a channel attention value, an activation function of the full connection layer is a sigmoid function, and finally the channel attention value corresponding to each channel is multiplied (i.e., weighted) with the corresponding channel feature map to form a channel attention feature map 802.

Because the channel attention mechanism is mapped into a channel attention value through a full connection layer according to the respective corresponding mean values of the channels, the attention feature map obtained after weighting according to the channel attention value will pay more attention to the channel with a larger mean value (i.e., the image feature of the channel with a larger mean value corresponds to a larger weight).

In one possible implementation, the first attention weighting module includes a first spatial attention weighting module, a second convolutional layer, a first channel attention module, and a third convolutional layer; based on the first spatial attention weighting module, carrying out weighting processing based on a spatial attention mechanism on the first sample extraction feature map to obtain a first sample spatial feature map; performing feature extraction on the first sample space feature map based on the second convolution layer to obtain a second sample extraction feature map; based on the first channel attention weighting module, carrying out weighting processing based on a channel attention mechanism on the second sample extraction feature map to obtain a first sample channel feature map; performing feature extraction on the first sample channel feature map based on the third convolutional layer to obtain a third sample extraction feature map; and acquiring an attention feature map corresponding to the first sample electroencephalogram signal based on the first sample spatial feature map, the first sample channel feature map and the third sample extraction feature map.

The first sample spatial feature map is obtained by performing weighting processing based on the spatial attention mechanism on the first sample extraction feature map, so that the features in the first sample spatial feature map pay more attention to the position where the image features are maximum in each image channel; on the basis of the second convolution layer, a second sample extraction feature map obtained by performing feature extraction on the first sample spatial feature map is a feature of a different level from the first sample spatial feature map, so that the first sample channel feature map obtained by weighting the second sample extraction feature map through a channel attention mechanism is a feature of a different level as the first sample spatial feature map; and a third sample extraction feature map obtained by extracting features of the first sample channel feature map based on the third convolution layer is a feature map with different levels from the first sample channel feature map and the first sample spatial feature map, namely the first sample spatial feature map, the first sample channel feature map and the third sample extraction feature map are image features with different levels obtained by different feature extraction methods based on the first sample extraction feature map.

Therefore, the attention feature map is obtained by weighting the image features of the different layers by a spatial attention weighting mechanism and a channel attention weighting mechanism, respectively. The attention feature map corresponding to the first sample electroencephalogram signal simultaneously contains image features of different levels, and the image features of different levels are weighted through an attention mechanism, so that the attention feature map corresponding to the first sample electroencephalogram signal simultaneously contains features of different levels extracted through different convolution kernels which are weighted through the attention mechanism on the basis of time domain features, frequency domain features and space domain features, the attention feature map has image features of more levels, and on the basis of ensuring the diversity of image feature levels, the attention mechanism can ensure that the image features more focus on rich feature positions in the feature map, and the feature extraction effect is improved.

In a possible implementation manner, the first attention weighting module further includes a second attention weighting module; fusing the first sample spatial feature map, the first sample channel feature map and the third sample extraction feature map to obtain a first sample fused feature map; based on the first sample fusion feature map, the second attention weighting module carries out weighting processing based on an attention mechanism, and an attention feature map corresponding to the first sample electroencephalogram signal is obtained.

The first sample fusion feature map is obtained by fusing the first sample spatial feature map, the first sample channel feature map and the third sample extraction feature map, so that the first sample fusion feature map has the three image features of different levels at the same time. Based on the first sample fusion feature map, the second attention weighting module carries out weighting processing on the first sample fusion feature map, and the obtained attention feature map is further carried out weighting processing through an attention mechanism on the basis of the first sample fusion feature map, so that important features in the fused first sample fusion feature map are further enhanced, and the feature extraction effect is improved.

In one possible implementation, the second attention weighting module includes at least one of a second spatial attention weighting module and a second channel attention weighting module.

In one possible implementation manner, in response to that the second attention weighting module includes a second spatial attention weighting module and a second channel attention weighting module, the second channel attention weighting module performs weighting processing based on a channel attention mechanism on the first sample fusion feature map to obtain a second sample channel feature map; and performing weighting processing based on a spatial attention mechanism on the second sample channel characteristic diagram through the second sample spatial attention weighting module to obtain an attention characteristic diagram corresponding to the first sample electroencephalogram signal.

When the second attention weighting module comprises a second spatial attention weighting module, the attention feature map obtained by weighting the first sample extraction feature map according to the spatial attention weighting module is more focused on the area with the most abundant features in each feature map; when the attention weighting module includes the second channel attention weighting module, the attention feature map obtained by weighting the first sample extraction feature map by the channel attention weighting module is more focused on the channel with the most abundant features in the feature map, so that the first sample fusion feature map is processed by the second attention weighting module, and the first sample fusion feature map can be further weighted by a channel attention mechanism and a spatial attention mechanism, so that the feature map can more focus on the information-rich part on the channel scale and the spatial scale.

Step 405, obtaining a sample probability distribution corresponding to the first sample electroencephalogram signal based on the attention feature map corresponding to the first sample electroencephalogram signal.

In a possible implementation manner, acquiring a feature vector corresponding to the first sample electroencephalogram signal based on an attention feature map corresponding to the first sample electroencephalogram signal; and acquiring the sample probability distribution corresponding to the first sample electroencephalogram signal based on the feature vector corresponding to the first sample electroencephalogram signal.

In one possible implementation, the electroencephalogram signal classification model further includes a first fully connected layer; and based on the attention feature map corresponding to the first sample electroencephalogram signal, performing data processing through the first full-connection layer to obtain a feature vector corresponding to the first sample electroencephalogram signal.

Inputting the attention feature map corresponding to the first sample electroencephalogram signal into the first full-connection layer, so as to obtain a feature vector corresponding to the first sample electroencephalogram signal, wherein the size of the value of different dimensionalities in the feature vector indicates the possibility that the first sample electroencephalogram signal corresponds to different motor imagery types.

In a possible implementation manner, the feature vector corresponding to the first sample electroencephalogram signal is input into the softmax activation layer of the electroencephalogram signal classification model, and the sample probability distribution corresponding to the first sample electroencephalogram signal is obtained.

Step 406, training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal.

In one possible implementation, a second sample brain electrical signal is obtained; the second sample brain electrical signal comprises at least two second sample electrode signals; the second sample electrode signal is used for indicating brain wave signals generated in a space area corresponding to the second sample electrode signal when the target object is in motor imagery; acquiring second sample time-frequency characteristic graphs respectively corresponding to at least two second sample electrode signals based on the at least two second sample electrode signals; the second sample time-frequency characteristic map is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the second sample electrode signals; based on second sample time-frequency characteristic graphs respectively corresponding to at least two second sample electrode signals, performing characteristic extraction through a first convolution layer in the electroencephalogram signal classification model to obtain a second sample extraction characteristic graph; the second sample extraction feature map is fused with at least two spatial features of the second sample electrode signals; the spatial characteristics of at least two second sample electrode signals are related to the spatial areas corresponding to the at least two second sample electrode signals; extracting a feature map from the second sample based on an attention weighting module in the electroencephalogram signal classification model, and performing weighting processing based on an attention mechanism to obtain an attention feature map corresponding to the electroencephalogram signal of the second sample; training the electroencephalogram classification model based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal; the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

The process of extracting the features of the electroencephalogram signal of the second sample through the electroencephalogram signal classification model to obtain the feature vector corresponding to the electroencephalogram signal of the second sample is similar to the process of extracting the features of the electroencephalogram signal of the first sample through the electroencephalogram signal classification model to obtain the feature vector corresponding to the electroencephalogram signal of the first sample, and the process is not repeated here.

In one possible implementation manner, a first loss function value is obtained based on the sample probability distribution and the motor imagery type corresponding to the first sample electroencephalogram signal; obtaining a second loss function value based on the feature vector corresponding to the first sample electroencephalogram signal and the feature vector corresponding to the second sample signal; and training the electroencephalogram signal classification model based on the first loss function value and the second loss function value.

In a possible implementation manner, the feature vector corresponding to the first sample electroencephalogram signal and the feature vector corresponding to the second sample signal are input into a domain discriminator in the electroencephalogram classification model, so as to obtain a second loss function value.

The domain discriminator is a convolution layer structure in transfer learning and is used for obtaining an output matrix according to input and determining the positive and negative of a sample according to the mean value of the output matrix. In an embodiment of the present application, the domain discriminator is configured to determine whether the input second sample electroencephalogram signal is the same type of electroencephalogram signal as the first sample electroencephalogram signal.

In a possible implementation manner of the embodiment of the present application, the model training process includes three parts of loss functions: classifier loss function, domain discriminator loss function.

Wherein the classifier penalty function may be as follows:

wherein the content of the first and second substances,andrespectively, the true label and the prediction probability of the source domain data (i.e., the first sample electroencephalogram signal), L represents the cross-entropy loss function, θ_fAnd theta_cRepresenting feature extractor parameters and classifier parameters, respectively. Wherein E represents the source domain dataRespectively for each type of motor imageryProbability distribution of (2).

The domain discriminator loss function is as follows:

wherein the content of the first and second substances,andrespectively a source domain feature and a target domain feature. The domain discriminator loss function is based on the value output by the discriminatorThe domain discriminator is used for respectively judging the probability that the input first sample brain electric signal and the second sample brain electric signal are a source domain and a target domain, wherein,representing a probability distribution that the first sample brain electrical signal is in the source domain,and D is the output of the discriminator, and the model can simultaneously learn the characteristics of the source domain and the target domain through the discriminator in the field.

The overall loss function of the above model is:

L_d(θ_f，θ_c，θ_d)＝L_c-αL_d

wherein, α is the balance classification loss L_cSum discriminator loss L_dIs determined. Through the overall loss function of the model, the model learns the characteristics of the source domain and the target domain while improving the classification performance through the classification loss function value, so that the model can have certain classification and recognition capabilities on the input second sample electroencephalogram signals under the condition that the target domain (namely the second sample electroencephalogram signals) does not have labels, and the generalization of model training is improved.

or the second sample electroencephalogram signal is used for indicating electroencephalogram signals generated by other human bodies except the target object corresponding to the first sample electroencephalogram signal.

When the second sample electroencephalogram signal is an electroencephalogram signal generated when a sample target object of the first sample electroencephalogram signal executes motor imagery at different time, the first sample electroencephalogram signal is used as source domain data in countermeasure learning, the second sample electroencephalogram signal is used as target domain data in countermeasure learning, and a model is trained through a motor imagery type corresponding to the first sample electroencephalogram signal and a sample probability distribution corresponding to the first sample electroencephalogram signal, so that the classification capability of the model on the motor imagery type can be improved; when the model is trained through the loss function, the second sample electroencephalogram signal is electroencephalogram data generated when the target object executes motor imagery at different time, so that the trained model has good recognition degree on electroencephalograms triggered by the same human body at different time, and the time invariance of the trained model on electroencephalogram recognition is improved.

When the second sample electroencephalogram signal and the first sample electroencephalogram signal are electroencephalogram signals generated when different sample target objects execute motor imagery, the first sample electroencephalogram signal is used as source domain data in countermeasure learning, the second sample electroencephalogram signal is used as target domain data in countermeasure learning, and a model is trained through a motor imagery type corresponding to the first sample electroencephalogram signal and a sample probability distribution corresponding to the first sample electroencephalogram signal, so that the classification capability of the model on the motor imagery type can be improved; when the model is trained through the loss function, the second sample electroencephalogram signal is electroencephalogram data generated when different human bodies corresponding to the first sample electroencephalogram signal execute motor imagery, so that the trained model has good recognition degree on electroencephalograms triggered by different human bodies, and the generalization capability of the trained model on electroencephalogram recognition is improved.

In a possible implementation manner, the electroencephalogram classification model can analyze the feature vector corresponding to the first sample electroencephalogram signal and the feature vector corresponding to the second sample electroencephalogram signal through the domain discriminator in the training process, and update the electroencephalogram classification model according to the loss function, and in the application process of the electroencephalogram classification model, the domain discriminator can be removed, and only the classifier (namely, the first full-connection layer) is reserved so as to classify the input electroencephalogram signal.

In a possible implementation manner, the electroencephalogram classification model further comprises a discarding layer, the discarding layer is used for discarding image features in a specified proportion, the discarding layer can be located at each position in the electroencephalogram classification model, and the discarding layer is added in the electroencephalogram classification model, so that the probability of overfitting of the model in the training process can be reduced.

Step 407, acquiring a first electroencephalogram signal.

In one possible implementation, the at least two electrode signals of the first brain electrical signal may be brain wave signals generated on the head of the target object acquired through the electrodes of the brain-computer interface by a terminal device having the brain-computer interface. The number of the electrode signals is the same as that of the electrodes corresponding to the brain-computer interface, that is, the brain-computer interface can simultaneously acquire the electroencephalogram signals generated by the same target object in different spatial regions of the head through different electrodes.

The first brain electrical signal can be a brain electrical signal acquired by a brain-computer interface through an electrode when a target object is in motor imagery; or the first brain electrical signal can also be a brain electrical signal acquired by the brain-computer interface through the electrodes when the target object is in a state other than the motor imagery.

In a possible implementation manner, the brain-computer interface obtains electroencephalogram signals generated by different areas of the head of the target object through electrodes connected to the target object, and the electrodes connected to the target object transmit the electroencephalogram signals corresponding to the electrodes to terminal equipment corresponding to the brain-computer interface through different electroencephalogram channels.

In one possible implementation, based on each electrode of the brain-computer interface, acquiring an original electroencephalogram signal generated by the target object; based on the original electroencephalogram signal, filtering processing is carried out through a band-pass filter, and the first sample electroencephalogram signal is obtained.

Similar to the above processing steps of the original sample electroencephalogram signal, because there is more noise interference in the original electroencephalogram signal obtained through the electrodes of the brain-computer interface, the original sample electroencephalogram signal needs to be filtered through a band-pass filter first, and the influence of irrelevant noise on the electroencephalogram signal is reduced.

Step 408, based on the at least two electrode signals, obtaining time-frequency feature maps corresponding to the at least two electrode signals respectively.

In one possible implementation manner, at least two electrode signals of the first sample brain electrical signal are respectively subjected to standardization operation, and at least two standard signals are obtained; and acquiring time-frequency characteristic graphs respectively corresponding to the at least two electrode signals based on the at least two standard signals.

After each original electroencephalogram signal is subjected to 3-38HZ band-pass filtering, unrelated physiological noise and power frequency interference are filtered out to obtain a first electroencephalogram signal, but the first electroencephalogram signal still has noise which cannot be removed through band-pass filtering, so that at least two electrode signals of the first electroencephalogram signal can be subjected to standardization operation in order to reduce signal disturbance caused by the noise. Wherein the normalization operation may be any one of an exponential weighted moving average operation, mean variance normalization, and co-spatial mode algorithm.

In a possible implementation manner, based on at least two electrode signals, continuous wavelet transformation is performed to obtain time-frequency characteristic graphs corresponding to the at least two electrode signals respectively.

When the data processing is performed on the at least two electrode signals of the first electroencephalogram signal through the continuous wavelet transform, the at least two electrode signals in the first electroencephalogram signal can be fitted through the basis functions corresponding to the continuous wavelet transform, and different from the fourier transform, the basis through the continuous wavelet transform is affected by both time and frequency, so that the time-frequency characteristic diagram respectively corresponding to the at least two electrode signals obtained by performing the continuous wavelet transform on the at least two electrode signals of the first electroencephalogram signal contains the time-domain characteristics of the electrode signals and also contains the frequency-domain characteristics of the electrode signals.

In a possible implementation manner, the time-frequency feature map corresponding to the first electroencephalogram signal is obtained based on the time-frequency feature maps corresponding to the at least two electrode signals, respectively.

When the first electroencephalogram signal contains at least two electrode signals, namely the first electroencephalogram signal comprises electroencephalogram signals generated by at least two areas of the head when a brain-computer interface acquires motor imagery of a target object through at least two electrodes. And then, splicing the time-frequency characteristic diagrams respectively corresponding to the at least two electrode signals according to channels to obtain the time-frequency characteristic diagram corresponding to the first electroencephalogram signal.

The time-frequency characteristic graph corresponding to the first electroencephalogram signal is provided with image characteristics of at least two channels, and the image characteristics of the at least two channels respectively correspond to the time-frequency characteristic graphs corresponding to the at least two electrode signals.

In a possible implementation manner, the image features of each channel in the time-frequency feature map corresponding to the first electroencephalogram signal are respectively determined according to the signal images of the time-frequency feature maps of the at least two electrodes. That is, after the time-frequency feature maps corresponding to the at least two electrode signals are obtained, the time-frequency feature maps corresponding to the at least two electrode signals can be spliced according to the channels, so as to obtain the time-frequency feature map corresponding to the first electroencephalogram signal. At this time, the time-frequency characteristic diagram corresponding to the first electroencephalogram signal contains time-domain characteristics and frequency-domain characteristics of the electroencephalogram signals generated by different spatial regions of the head of the target object.

Step 409, based on the time-frequency characteristic diagrams corresponding to at least two electrode signals respectively, performing characteristic extraction through a first convolution layer in the electroencephalogram classification model to obtain a first extracted characteristic diagram.

The time-frequency characteristic diagram corresponding to the first electroencephalogram signal can be subjected to characteristic extraction through the first convolution layer in the electroencephalogram signal classification model, and the first extracted characteristic diagram is obtained.

In a possible implementation manner, the image feature of each channel in the first extracted feature map includes an image feature of each channel in a time-frequency feature map corresponding to the first electroencephalogram signal.

For example, when the first convolution layer is convolution kernels of 3 × 3 and the number of convolution kernels is 5, each convolution kernel in the first convolution layer performs convolution operation on each channel in the time-frequency feature map corresponding to the first electroencephalogram signal and then sums up to obtain the image feature corresponding to the convolution kernel, so that when 5 convolution kernels perform convolution operation on the time-frequency feature maps corresponding to the electrode signals in the time-frequency feature map corresponding to the first electroencephalogram signal, the image features of 5 channels, that is, the first extracted feature map with the channel being 5, can be obtained, and since the image features in each channel in the first extracted feature map are obtained by summation after convolution operation on each channel, the image features in each channel all include the image features of each channel (that is, each electrode signal) in the time-frequency feature map corresponding to the first electroencephalogram signal, the time-frequency characteristic diagram corresponding to the first electroencephalogram signal is subjected to characteristic extraction through the first convolution layer, the time-frequency characteristics of all channels in the time-frequency characteristic diagram corresponding to the first electroencephalogram signal are fused, and all the channels in the first time-frequency characteristic diagram are electroencephalograms of objects acquired by electrodes in different spatial positions, so that the fused first extracted characteristic diagram is a characteristic diagram simultaneously having time-domain characteristics, frequency-domain characteristics and space-domain characteristics.

And step 410, based on an attention weighting module in the electroencephalogram signal classification model, performing attention mechanism-based weighting processing on the features of different levels of the first extracted feature map to obtain an attention feature map corresponding to the first electroencephalogram signal.

In a possible implementation manner, the features of different levels in the first extracted feature map are used for indicating the features obtained after feature extraction is performed on different convolutional layers in the first extracted feature map.

In one possible implementation, the first attention weighting module includes a first spatial attention weighting module, a second convolutional layer, a first channel attention module, and a third convolutional layer; based on the first spatial attention weighting module, carrying out weighting processing based on a spatial attention mechanism on the first extracted feature map to obtain a first spatial feature map; performing feature extraction on the first spatial feature map based on the second convolutional layer to obtain a second extracted feature map; based on the first channel attention weighting module, carrying out weighting processing based on a channel attention mechanism on the second extracted feature map to obtain a first channel feature map; performing feature extraction on the first channel feature map based on the third convolutional layer to obtain a third extracted feature map; and acquiring an attention feature map corresponding to the first electroencephalogram signal based on the first spatial feature map, the first channel feature map and the third extracted feature map.

The first spatial feature map is obtained by performing weighting processing based on the spatial attention mechanism on the first extracted feature map, so that the features in the first spatial feature map pay more attention to the position where the image features are the largest in each image channel; on the basis of the second convolutional layer, a second extracted feature map obtained by extracting features of the first spatial feature map is features of a different level from the first spatial feature map, so that the first channel feature map obtained by weighting the second extracted feature map through a channel attention mechanism is features of a different level as the first spatial feature map; the third extracted feature map obtained by extracting the features of the first channel feature map based on the third convolutional layer is a feature map having different levels from the first channel feature map and the first spatial feature map, that is, the first spatial feature map, the first channel feature map and the third extracted feature map are image features of different levels obtained by different feature extraction methods based on the first extracted feature map.

Therefore, the attention feature map is obtained by weighting the image features of the different layers by a spatial attention weighting mechanism and a channel attention weighting mechanism, respectively. The attention feature map corresponding to the first electroencephalogram signal simultaneously contains image features of different levels, and the image features of the different levels are weighted through an attention mechanism, so that the attention feature map corresponding to the first electroencephalogram signal simultaneously contains the features of the different levels extracted by different convolution kernels which are weighted through the attention mechanism on the basis of having time domain features, frequency domain features and space domain features, the attention feature map has image features of more levels, and on the basis of ensuring the diversity of image feature levels, the attention mechanism can ensure that the image features more emphasize abundant feature positions in the feature map, and the feature extraction effect is improved.

In a possible implementation manner, the first attention weighting module further includes a second attention weighting module; fusing the first spatial feature map, the first channel feature map and the third extraction feature map to obtain a first fused feature map; and based on the first fusion feature map, performing weighting processing based on an attention mechanism through the second attention weighting module to obtain an attention feature map corresponding to the first electroencephalogram signal.

In one possible implementation, the second attention weighting module includes at least one of a second spatial attention weighting module and a second channel attention weighting module.

In a possible implementation manner, in response to that the second attention weighting module includes a second spatial attention weighting module and a second channel attention weighting module, the second channel attention weighting module performs weighting processing based on a channel attention mechanism on the first fusion feature map to obtain a second channel feature map; and performing weighting processing based on a spatial attention mechanism on the second channel characteristic diagram through the second spatial attention weighting module to obtain an attention characteristic diagram corresponding to the first electroencephalogram signal.

When the second attention weighting module comprises a second spatial attention weighting module, the attention feature map obtained by weighting the first extracted feature map according to the spatial attention weighting module is more focused on the area with the most abundant features in each feature map; when the attention weighting module includes the second channel attention weighting module, the attention feature map obtained by weighting the first extracted feature map by the channel attention weighting module is more focused on the channel with the most abundant features in the feature map, so that the first fused feature map is processed by the second attention weighting module, and the first fused feature map can be further weighted by a channel attention mechanism and a spatial attention mechanism, so that the feature map can more focus on the part with abundant information on a channel scale and a spatial scale.

Step 411, obtaining a motor imagery type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal.

In a possible implementation manner, based on an attention feature map corresponding to the first electroencephalogram signal, obtaining a probability distribution corresponding to the first electroencephalogram signal; the probability distribution is used for indicating the probability that the first electroencephalogram signal is of various motor imagery types respectively; and acquiring the motor imagery type corresponding to the first electroencephalogram signal based on the probability distribution corresponding to the first electroencephalogram signal.

In a possible implementation manner, a feature vector corresponding to the first electroencephalogram signal is obtained based on an attention feature map corresponding to the first electroencephalogram signal; and acquiring probability distribution corresponding to the first electroencephalogram signal based on the feature vector corresponding to the first electroencephalogram signal.

In one possible implementation, the electroencephalogram signal classification model further includes a first fully connected layer; and based on the attention feature map corresponding to the first electroencephalogram signal, performing data processing through the first full-connection layer to obtain a feature vector corresponding to the first electroencephalogram signal.

Inputting the attention feature map corresponding to the first electroencephalogram signal into the first full-connection layer, and obtaining a feature vector corresponding to the first electroencephalogram signal, wherein the size of the value of different dimensionalities in the feature vector indicates the possibility that the first electroencephalogram signal corresponds to different motor imagery types.

In a possible implementation manner, the feature vector corresponding to the first electroencephalogram signal is input into the softmax activation layer of the electroencephalogram signal classification model, and the probability distribution corresponding to the first electroencephalogram signal is obtained.

In a possible implementation manner, the motor imagery type with the highest probability in the probability distribution corresponding to the first electroencephalogram signal is obtained as the motor imagery type corresponding to the first electroencephalogram signal.

In a possible implementation manner, when the probability of the motor imagery type with the highest probability in the probability distribution corresponding to the first electroencephalogram signal is greater than a threshold, the motor imagery type with the highest probability is acquired as the motor imagery type corresponding to the first electroencephalogram signal.

In another possible implementation manner, when the probability corresponding to the motor imagery type with the highest probability in the probability distribution corresponding to the first electroencephalogram signal does not exceed a threshold, the first electroencephalogram signal is determined to be an electroencephalogram signal which cannot be identified.

Please refer to fig. 9, which shows a schematic diagram of a classification model of electroencephalogram signals according to an embodiment of the present application. As shown in fig. 9, in the training process of the electroencephalogram signal classification model, source domain data 901 (i.e., a first sample electroencephalogram signal) and target domain data 902 (i.e., a second sample electroencephalogram signal) may be input at the same time, the source domain data may be subjected to a data preprocessing process (wavelet transform) to obtain source time-frequency data 903, the target domain data may be subjected to a data preprocessing process to obtain target time-frequency data 904, the source time-frequency data and the target time-frequency data may be input to the model respectively for feature extraction to obtain a source feature vector 905 and a target feature vector 906, the source feature vector 905 is input to the classifier 910 for classification to obtain probability distribution corresponding to the source domain data, and the source feature vector 905 and the target feature vector 906 obtain a loss function value according to a domain adaptivity mechanism, and then the model is updated according to the loss function value.

The MI-BCI system has wide application prospect in many fields, and can control external equipment by imagining limb movement through the brain under the condition of no limb movement. The wheelchair can help patients with limb inconvenience such as cerebral apoplexy, hemiplegia and the like to perform rehabilitation training or control the wheelchair to go out, and can also be used for education and entertainment of common users, such as brain control VR games and the like. The MI signal classification identification is a key link in the MI-BCI system, and the decoding accuracy directly affects the performance and user experience of the system. Because different electroencephalograms to be tested have large difference, the existing method needs to train a model for each tested object independently and needs to adjust model hyper-parameters elaborately, so that the training process is time-consuming and tedious, the classification performance is poor, and the problems are limited in the application scene of the BCI interaction technology. In order to meet the requirements of performance and structural universality of classification of electroencephalogram signals, the scheme shown in the embodiment of the application transforms input signals into multi-domain representation and processes the input signals by using an attention adaptive model, automatically extracts space-time characteristics in an EEG signal sequence through deep learning, enables the extracted electroencephalogram space-time characteristics to have time invariance by using field adaptation, reduces the difference between different tested EEG signals, solves the problems that each tested EEG needs to be independently and finely tuned in a model and needs to be calibrated before the model is applied due to individual difference, and effectively improves the accuracy of MI classification.

The embodiment of the application provides an attention self-adaptive electroencephalogram classification model based on multi-domain representation aiming at the time difference of motor imagery electroencephalogram signals. Firstly, carrying out 3-38Hz band-pass filtering processing on each EEG signal sample to remove the influence on the EEG signal caused by irrelevant physiological noise such as eye movement and power frequency interference; then, the filtered signal is subjected to an exponential weighted moving average operation to reduce signal disturbance caused by noise; performing continuous wavelet transformation on each channel to obtain channel time-frequency representations, splicing the time-frequency representations of all the channels to be used as input of an attention self-adaptive electroencephalogram decoding model, automatically learning key time-domain and space-domain features through an attention mechanism, and aligning distribution of electroencephalogram data of a source domain and a target domain, so that the extracted electroencephalogram features have time invariance, and the generalization capability of the decoding model is improved; finally, the model predicts the motor imagery class to which the input EEG signal corresponds. The scheme shown in the embodiment of the application provides an attention adaptive electroencephalogram decoding model based on multi-domain representation of a space domain, a time domain, a frequency domain and the like, and spatial and temporal features with identifiability and individual invariance can be extracted from electroencephalogram data of multiple testees, so that the decoding capability and accuracy of the model are effectively improved. According to the scheme shown in the embodiment of the application, an attention mechanism and a field self-adaptive mechanism are introduced, an attention feature map can be generated according to an input sample, a key channel, time information, frequency information and the like related to a classification task are positioned, a feature extractor can extract more separable features, and meanwhile, the generalization capability of the model in a target domain is enhanced and the performance of the model is improved by aligning the condition distribution of source domain data and target domain data.

The technical scheme can realize idea transmission and control for the subject by being embedded into different hardware systems or software systems. For example, the BCI system combined with the exoskeleton robot can be used for active rehabilitation of the motion function of patients with hemiplegia and cerebral apoplexy; the BCI system combined with the electric wheelchair can help users with limb mobility inconvenience to freely move out; the brain-controlled VR game system combined with the game can realize the action of controlling the virtual world object by the human body through the idea imagination.

According to the scheme, channel importance and time-frequency characteristics in the electroencephalogram signals are fully considered, other tested data are used for training the network model, the available model can be trained for classification under the condition that target tested data are not labeled, the acquisition time and model calibration time of labeled data are reduced, the identification capability and training efficiency of the MI-BCI system can be improved while manpower and material resources are saved, and better user experience is provided.

The technical scheme uses The public Competition data The BCI Competition IV Dataset 2a to kinematically visualize a public data set. This data set contains 9 subjects, in which brain electrical data of each subject was recorded by 22 brain electrical electrodes and 3 eye electrical electrodes, with a signal sampling rate of 250Hz, including 4 motor imagery categories (left hand, right hand, feet and tongue). The experiment comprises two stages, wherein the data file acquired in the training stage of each subject is used as a training set, and the data file acquired in the testing stage is used as a testing set.

The scheme intercepts signals in a motor imagery interval for each sample, namely 4s of data in 2s-6s, and the time dimension of each sample is 1000 because the sampling frequency of the signals is 250 Hz. According to the scheme, 3 electro-ocular channels are directly removed, and only 22 electroencephalogram channels related to the motor imagery task are considered; band-pass filtering selects a Butterworth filter of 3 orders, and the band-pass range is [3-38Hz ]; the signal normalization is performed by an exponential weighted moving average method with a weight parameter set to 0.999, but other normalization operations such as mean variance normalization and CSP algorithm may be used.

According to the scheme, the spatial-time and frequency domain representation of the electroencephalogram is obtained from the original electroencephalogram signals by adopting continuous wavelet transformation, firstly, the signals of each electroencephalogram channel are subjected to continuous wavelet transformation, and time-frequency characteristic graphs corresponding to all electroencephalogram channels are spliced to form the multi-domain representation which integrates spatial information, time information and frequency domain information. The scheme adopts 'cmor3.0-3.0' as a wavelet basis function, the resolution is set to be 256, and other wavelet basis functions such as haar wavelet, db wavelet, sym wavelet, coif series wavelet and the like can be selected.

Please refer to fig. 10, which shows an application diagram of an electroencephalogram classification model according to an embodiment of the present application. As shown in fig. 10, the embodiment of the present application designs an attention adaptive electroencephalogram classification model based on multi-domain characterization according to the time characteristics, the spatial characteristics and the frequency characteristics of the input EEG signals, and the basic model parameters thereof are shown in table 1. The decoding model includes three parts: a feature extractor, a classifier and a domain discriminator. The input signals 1001 to the network include a source domain signal (i.e., a first sample brain electrical signal) and a target domain signal (i.e., a second sample brain electrical signal), each having a size of N × 61 × 160(N is the number of electrodes). The first layer of the feature extractor is convolution layer Conv _1-Batch Normalization-ReLu (i.e. first convolution layer 1002), wherein the convolution kernel size is 3 × 15, the step sizes stride are all 1, and the number of convolution channels is 8; the second layer is a Spatial Attention layer Spatial Attention (i.e. the first Spatial Attention weighting module 1003), and generates a Spatial Attention map according to the input signal, wherein the size of a convolution kernel is 3 × 3, the step lengths stride are all 1, and the number of convolution channels is 1; the third layer is convolution layer Conv _2-Batch Normalization-ReLu (i.e. the second convolution layer 1004), wherein the convolution kernel size is 3 × 15, the step sizes stride are all 1, and the number of convolution channels is 16; then, performing size compression on the feature graph through an average pooling layer (the kernel size is 2 × 2, and the step size stride is 2 × 2), weighting each convolution Channel through a Channel Attention Channel assignment (namely, a first Channel Attention weighting module 1005), wherein a Channel Attention weight is generated by a fully-connected layer with a hidden node of 4, and then connecting a discarding layer Dropout to suppress overfitting, and the discarding rate is set to be 0.5; the seventh layer is a convolution layer Conv _3-Batch Normalization-ReLu (i.e., the third convolution layer 1006), in which the convolution kernel size is 3 × 15, the step sizes stride are all 1, and the number of convolution channels is 32; the feature size is then compressed by one average pooling layer (kernel size 2 x 2, step size stride 2 x 2). In order to fuse information of different layers and enhance the flow of network information, the output of the first discarded layer and the output of the first Spatial Attention layer are formed into a 13 × 26 feature map through adaptive mean pooling, and the feature map is spliced with the output of the third convolutional layer according to convolution channels, and each convolution Channel is weighted through Channel Attention weighting (i.e., the second Channel Attention weighting module 1007), wherein the Attention weight is generated by a fully-connected layer with a hidden node of 8, and then a Spatial Attention map is generated through Spatial Attention weighting (i.e., the second Spatial Attention weighting module 1008), wherein the convolution kernel size is 3 × 3, stride is 1, and the number of convolution channels is 1.

And finally, flattening the space attention diagram as the depth electroencephalogram feature, tiling the space attention diagram of the first sample electroencephalogram signal into a feature vector corresponding to the first sample electroencephalogram signal, tiling the space attention diagram of the second sample electroencephalogram signal into a feature vector corresponding to the second sample electroencephalogram signal, and respectively sending the feature vectors into a Classifier1009 and a Domain Discriminator 1010. The classifier is responsible for completing the task of classifying the electroencephalogram signals, and the domain discriminator is responsible for discriminating whether the electroencephalogram signals belong to source domain signals or target domain signals. Both the two are output final classification probabilities through a full connection layer and a Softmax activation layer in network structure design. In particular, the classifier outputs four types of prediction probabilities (corresponding to left, right, feet, and tongue), and the domain discriminator outputs two types of prediction probabilities (corresponding to source and target domains).

TABLE 1 attention adaptive EEG decoding model parameter table

In the embodiment of the application, parameters of the neural network model can be solved by adopting an Adam-based gradient descent method, and model parameters are initialized by adopting an Xavier initialization method. In the solving process, each tested EEG multi-domain representation and corresponding labels are sent into a network for learning, and model optimization is completed through error back propagation.

In the scheme shown in the embodiment of the application, the motor imagery spatial domain-time domain-frequency domain representation is decoded and classified end to end through a deep learning technology, and a complex characteristic extraction process through priori knowledge is not needed, so that the model has universality. Meanwhile, a multi-domain feature map is formed by utilizing the spatial information, the time information and the frequency information of the electroencephalogram signals, and electroencephalogram features related to tasks can be completely reserved; an attention mechanism is introduced to learn the electroencephalogram characteristic diagram, and the characteristics of a specific channel, frequency and time can be paid more attention to through a supervised training guide network, so that the model learning has more separable characteristics; the introduction of a counterstudy mechanism forces a feature extractor to extract the electroencephalogram features common to all tested individuals from the electroencephalogram feature map, so that the depth model has better generalization capability.

Reference is now made to FIG. 11, which is a block diagram illustrating a model training and model application flow, according to an exemplary embodiment. The model training and model application process may be performed by a model training device 1100 and a model application device (i.e., a signal processing device) 1110 together, as shown in fig. 11, and is as follows:

in the training device 1110, source domain data (a time-frequency feature map corresponding to the first sample electroencephalogram signal) 1111 and target domain data 1112 (a time-frequency feature map corresponding to the second sample electroencephalogram signal) are respectively input into the electroencephalogram classification model, where the electroencephalogram classification model in the embodiment of the present application may be the electroencephalogram classification model shown in fig. 10, and a specific structure thereof is not described herein again. After the time-frequency feature map 1111 corresponding to the first sample electroencephalogram signal is processed by the electroencephalogram signal classification model, the obtained feature map corresponding to the first sample electroencephalogram signal can be input into the first full connection layer 1113 in the electroencephalogram signal classification model to obtain probability distribution 1115 corresponding to the first sample electroencephalogram signal, and a first loss function value can be obtained according to the probability distribution corresponding to the first sample electroencephalogram signal and the motor imagery type corresponding to the first sample electroencephalogram signal.

In the training device 1110, after the time-frequency feature map 1112 corresponding to the second sample electroencephalogram signal is processed by the electroencephalogram signal classification model, the obtained feature map corresponding to the second sample electroencephalogram signal and the feature map corresponding to the first sample electroencephalogram signal can be input into the discriminator 1114 in the electroencephalogram signal classification model to respectively obtain the domain classification probability corresponding to the first sample electroencephalogram signal and the domain classification probability corresponding to the second sample electroencephalogram signal, and the domain classification probability is used for indicating the probability that the first sample electroencephalogram signal and the second sample electroencephalogram signal belong to the training set corresponding to the first sample electroencephalogram signal. And obtaining a second loss function value according to the domain classification probability corresponding to the first sample electroencephalogram signal and the domain classification probability corresponding to the second sample electroencephalogram signal.

In the training device 1110, the electroencephalogram classification model can perform parameter updating according to the first loss function value and the second loss function value, and the first loss function value part can ensure the motor imagery classification capability of the updated electroencephalogram classification model on electroencephalogram signals; the second loss function value part can enable the model to have certain recognition capability for the second electroencephalogram signal sample, and the generalization of the model after training is improved.

After the EEG classification model is trained, the EEG classification model may be transmitted to the application device 1120. In this embodiment, since the domain discriminator part increases the generalization of the model based on the transfer learning mechanism, and does not work in the model application process, the domain discriminator part may be omitted from the electroencephalogram classification model, that is, the application device 1120 may only load a part of the model of the electroencephalogram classification model, so as to perform feature extraction on the input first electroencephalogram 1121, obtain an extracted feature map, and input the extracted feature map into the first fully-connected layer 1122 in the application device, so as to obtain the probability distribution 1123 corresponding to the first electroencephalogram.

Fig. 12 is a block diagram illustrating a structure of an electroencephalogram signal classification apparatus according to an exemplary embodiment. The electroencephalogram signal classification device can realize all or part of the steps in the method provided by the embodiment shown in fig. 2 or fig. 4, and comprises the following steps:

a first signal obtaining module 1201, configured to obtain a first electroencephalogram signal; the first electroencephalogram signal comprises at least two electrode signals; the electrode signals are used for indicating brain wave signals generated by a target object in a space region corresponding to the electrode signals;

a first time-frequency feature obtaining module 1202, configured to obtain, based on at least two of the electrode signals, time-frequency feature maps corresponding to the at least two of the electrode signals, respectively; the time-frequency feature map is used for indicating time-domain features and frequency-domain features corresponding to the electrode signals;

a first extracted feature obtaining module 1203, configured to perform feature extraction based on time-frequency feature maps corresponding to at least two of the electrode signals, respectively, to obtain a first extracted feature map; the first extraction feature map is fused with the spatial features of at least two electrode signals; spatial features of at least two of the electrode signals are associated with spatial regions to which the at least two of the electrode signals correspond;

a first attention feature obtaining module 1204, configured to perform weighting processing based on an attention mechanism on the first extracted feature map, and obtain an attention feature map corresponding to the first electroencephalogram signal;

a imagination type obtaining module 1205 for obtaining a motor imagination type corresponding to the first electroencephalogram signal based on the attention feature map corresponding to the first electroencephalogram signal;

In a possible implementation manner, the first extracted feature obtaining module 1203 includes:

the first attention feature acquisition module 1203, including:

In one possible implementation, the attention mechanism includes at least one of a spatial attention mechanism and a channel attention mechanism.

the first attention feature map acquisition unit includes:

a second feature obtaining subunit, configured to perform feature extraction on the first spatial feature map based on the second convolutional layer, so as to obtain a second extracted feature map;

a third feature obtaining subunit, configured to perform feature extraction on the first channel feature map based on the third convolutional layer, so as to obtain a third extracted feature map;

an attention feature obtaining subunit, configured to obtain the attention feature map based on the first spatial feature map, the first channel feature map, and the third extracted feature map.

In a possible implementation manner, the first attention weighting module further includes a second attention weighting module;

the attention feature acquisition subunit further includes:

a first fusion subunit, configured to fuse the first spatial feature map, the first channel feature map, and the third extracted feature map to obtain a first fused feature map;

In a possible implementation manner, the first time-frequency characteristic obtaining module 1202 includes:

In one possible implementation, the electroencephalogram classification model further includes a first fully connected layer;

the imagination type acquisition module 1205 is further operable to,

and determining the motor imagery type corresponding to the first electroencephalogram signal based on the probability distribution corresponding to the first electroencephalogram signal.

In summary, according to the scheme shown in the embodiment of the present application, an electroencephalogram including at least two electrode signals is obtained, a time-frequency feature map is obtained according to the at least two electroencephalograms, the time-frequency feature map is used to indicate a time-domain feature and a frequency-domain feature of the first electroencephalogram, feature extraction is performed on the time-frequency feature map to obtain a first extracted feature map, features of different levels of the extracted first extracted feature map are weighted based on an attention mechanism to obtain a weighted attention feature map, and finally, a motor imagery type corresponding to the electroencephalogram is determined through the weighted attention feature map. In the scheme, the time-frequency characteristic diagram is a time-frequency characteristic diagram corresponding to brain wave signals generated by a target object in regions corresponding to different electrode signals, namely the time-frequency characteristic diagram also contains the spatial relationship among different electrode signals, so that the time-frequency characteristic diagram is subjected to characteristic extraction through an electroencephalogram signal classification model, the time-domain characteristics and the frequency-domain characteristics of the electroencephalogram signals can be simultaneously considered, the characteristic diagram extracted from the time-frequency characteristic diagram is subjected to weighting processing through an attention mechanism, the spatial relationship among at least two electrode signals of the electroencephalogram signals can be considered, therefore, the finally obtained attention characteristic diagram is extracted after the time-domain characteristics, the frequency-domain characteristics and the space-domain characteristics of the electroencephalogram signals are simultaneously fused, and on the basis of ensuring the level diversity of image characteristics, the attention mechanism ensures that the image characteristics pay more attention to rich characteristic positions in the characteristic diagram, the motor imagery type corresponding to the first electroencephalogram signal is determined through the attention feature map, and accuracy of prediction of the motor imagery type corresponding to the electroencephalogram signal is improved.

Fig. 13 is a block diagram illustrating a structure of an electroencephalogram signal classification apparatus according to an exemplary embodiment. The electroencephalogram signal classification device can realize all or part of the steps in the method provided by the embodiment shown in fig. 3 or fig. 4, and comprises the following steps:

a first sample acquisition module 1301, configured to acquire a first sample electroencephalogram signal; the first sample brain electrical signal comprises at least two first sample electrode signals; the first sample electrode signal is used for indicating a brain wave signal generated in a space region corresponding to the first sample electrode signal when a target object is in motor imagery;

a first sample time-frequency obtaining module 1302, configured to obtain, based on at least two first sample electrode signals, first sample time-frequency feature maps corresponding to the at least two first sample electrode signals, respectively; the sample time-frequency characteristic graph is used for indicating time-domain characteristics and frequency-domain characteristics corresponding to the first sample electrode signal;

a first sample extracting and obtaining module 1303, configured to perform feature extraction through a first convolution layer in the electroencephalogram classification model based on first sample time-frequency feature maps corresponding to at least two first sample electrode signals, respectively, to obtain a first sample extracted feature map; the first sample extraction feature map is fused with the spatial features of at least two first sample electrode signals; spatial features of at least two of the first sample electrode signals are associated with spatial regions to which at least two of the first sample electrode signals correspond;

a first sample attention obtaining module 1304, configured to extract a feature map from the first sample based on an attention weighting module in the electroencephalogram signal classification model, perform weighting processing based on an attention mechanism, and obtain an attention feature map corresponding to the first sample electroencephalogram signal;

a first sample probability obtaining module 1305, configured to obtain a sample probability distribution corresponding to the first sample electroencephalogram signal based on the attention feature map corresponding to the first sample electroencephalogram signal; the sample probability distribution is used for indicating the probability that the first sample electroencephalogram signal is of various motor imagery types respectively;

a first training module 1306, configured to train the electroencephalogram classification model based on the sample probability distribution and a motor imagery type corresponding to the first sample electroencephalogram signal;

the electroencephalogram signal classification model is used for predicting a motor imagery type corresponding to a first electroencephalogram signal based on the first electroencephalogram signal.

In one possible implementation, the apparatus further includes:

the first model training module is further configured to,

FIG. 14 is a schematic diagram illustrating a configuration of a computer device, according to an example embodiment. The computer device may be implemented as the model training device and/or the signal processing device in the various method embodiments described above. The computer apparatus 1400 includes a Central Processing Unit (CPU) 1401, a system Memory 1404 including a Random Access Memory (RAM) 1402 and a Read-Only Memory (ROM) 1403, and a system bus 1405 connecting the system Memory 1404 and the Central Processing Unit 1401. The computer device 1400 also includes a basic input/output system 1406 that facilitates transfer of information between devices within the computer, and a mass storage device 1407 for storing an operating system 1413, application programs 1414, and other program modules 1415.

The mass storage device 1407 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include a computer readable medium (not shown) such as a hard disk or Compact disk Read-Only Memory (CD-ROM) drive.

Without loss of generality, the computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, flash memory or other solid state storage technology, CD-ROM, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the foregoing. The system memory 1404 and mass storage device 1407 described above may collectively be referred to as memory.

The computer device 1400 may connect to the internet or other network devices through the network interface unit 1411 connected to the system bus 1405.

The memory further includes one or more programs, which are stored in the memory, and the central processing unit 1401 implements all or part of the steps of the method shown in fig. 2, 4, or 5 by executing the one or more programs.

In an exemplary embodiment, a non-transitory computer readable storage medium comprising instructions, such as a memory comprising computer programs (instructions), executable by a processor of a computer device to perform the methods shown in the various embodiments of the present application, is also provided. For example, the non-transitory computer readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods shown in the various embodiments described above.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

42页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：PETCT系统对准装置及其对准方法

Electroencephalogram signal classification method and device, computer equipment and storage medium

相关技术

网友询问留言