阅读说明：本技术 基于多域自适应图卷积神经网络的情感脑电特征表示方法 (Emotional electroencephalogram feature representation method based on multi-domain self-adaptive graph convolution neural network ) 是由 吕宝粮 李芮 于 2021-09-30 设计创作，主要内容包括：一种基于多域自适应图卷积神经网络的情感脑电特征表示方法,通过对脑电数据集进行特征类别的标记,再将其中的脑电数据通过时域和频域预处理和特征提取后将两个域特征训练多域自适应图卷积网络,最后采用训练后的训练多域自适应图卷积网络进行在线特征识别。本发明融合脑电的频域和时域的信息以及信号模式相关的脑功能连接,通过多域自适应图卷积神经网络充分利用多个域的互补信息,同时考虑脑电通道的拓扑结构,自适应地学习脑功能连接实现识别。(A method for expressing emotion electroencephalogram features based on a multi-domain self-adaptive graph convolution neural network includes the steps of marking feature types of electroencephalogram data sets, conducting time domain and frequency domain preprocessing and feature extraction on electroencephalogram data, then training the multi-domain self-adaptive graph convolution network through two domain features, and finally conducting on-line feature recognition through the trained multi-domain self-adaptive graph convolution network. The invention integrates the information of frequency domain and time domain of the brain electricity and the brain function connection related to the signal mode, fully utilizes the complementary information of a plurality of domains through a multi-domain self-adaptive graph convolution neural network, and self-adaptively learns the brain function connection by considering the topological structure of the brain electricity channel to realize the identification.)

1. A brain electrical characteristic recognition method based on a multi-domain self-adaptive graph convolution neural network is characterized in that a brain electrical data set is marked according to characteristic categories, two domain characteristics are used for training the multi-domain self-adaptive graph convolution network after the brain electrical data in the brain electrical data set is preprocessed through a time domain and a frequency domain and extracted according to characteristics, and finally the trained multi-domain self-adaptive graph convolution network is used for carrying out online characteristic recognition;

the time domain and frequency domain preprocessing and feature extraction specifically comprise: time domain preprocessing, time domain feature extraction, time domain preprocessing and frequency domain feature extraction are sequentially carried out;

the frequency domain preprocessing refers to: and performing baseline correction and artifact removal processing on the acquired electroencephalogram data, and performing filtering processing on the electroencephalogram signals.

2. The method for recognizing the electroencephalogram characteristic based on the multi-domain adaptive graph-convolution neural network according to claim 1, wherein the time domain preprocessing refers to: performing baseline correction and artifact removal processing on the acquired electroencephalogram data, performing filtering processing on electroencephalogram signals, and dividing the electroencephalogram signals into 5 frequency bands through a filter; then, the electroencephalogram signals are divided according to the S second non-overlapping time window, and each sample obtained by the method can be composed of 5V electroencephalogram fragments with the duration of S seconds and respectively corresponds to 5 frequency bands.

3. The electroencephalogram feature identification method based on the multi-domain adaptive graph-convolution neural network as claimed in claim 1, wherein the time domain feature extraction is as follows: extracting time-domain signal mode related functional brain connection A^TeThe method comprises the following specific steps:

1) for each sample, calculating the Pearson correlation coefficient between the EEG signals with the time length of S seconds of each channel under each frequency band, expressing the obtained relation between the pair of EEG signals under each frequency band of each sample by a V multiplied by V symmetrical connection matrix A, expressing the connection weight between the EEG signals of every two channels by elements in the matrix, namely the information of edges in the brain function connection network, and finally obtaining a function connection matrixWherein N is pretreatmentThe number of processed samples, wherein F represents 5 frequency bands, and V multiplied by V is the dimension of a time domain connection matrix obtained by calculation according to the Pearson correlation coefficient;

2) all tested samples in a training set are used together to select functional brain connections related to signal pattern recognition, L is a signal pattern class set, all samples and all tested samples are averaged by all functional connection matrixes in the training set on each frequency band F e F and each type of signal pattern L e L, namely all connection matrixes with labels L are averaged, and the expression form is as follows:whereinRepresenting the connection matrix corresponding to the i-th sample in the f-band, yⁱRepresenting the signal mode category corresponding to the ith sample;

3) will matrixThe upper right corner elements are sorted from large to small according to the absolute value of the connection weight;

4) and (3) reserving the strongest connection by using the obtained F x L average brain networks with the same proportion threshold value t to obtain the key connection under each type of signal mode:

5) merging the key connections reserved in the average brain network of L signal pattern classes, wherein the expression form is as follows:

6) performing key connection reserved in average brain network of F frequency bandsPooled, expressed in the form: a. the_critical＝union_f∈F(A^f)；

7) Normalizing the obtained key connection, and calculating the time domain signal mode related functional brain connection A^Te＝normalizing(A_critical)。

4. The EEG feature recognition method based on multi-domain adaptive graph-convolution neural network as claimed in claim 1, wherein said frequency domain feature extraction is to divide EEG signal into 5 frequency bands by short time Fourier transform, extract differential entropy frequency domain feature of each frequency band, and extract frequency domain feature for obtaining EEG time sequence Using time windows T to convert to

5. The method for electroencephalogram feature recognition based on the multi-domain adaptive atlas neural network of claim 1, wherein the multi-domain adaptive atlas network comprises: a plurality of multi-domain adaptive graph volume blocks, a global average pooling layer, a softmax layer, wherein: the multi-domain self-adaptive volume block is fused with electroencephalogram time domain and frequency domain features, features related to signal mode identification are extracted, a pooling layer carries out global average pooling on output features of the last layer of volume block, and a softmax layer classifies the features output after pooling to obtain signal mode categories.

6. The electroencephalogram feature recognition method based on the multi-domain adaptive graph-convolution neural network according to claim 1, wherein the online feature recognition specifically comprises:

step 1) adaptation in multiple domainsIn a graph convolution network, frequency domain features are combinedSet as input to the first graph volume block:

step 2) the volume Block of the b-th graph is Block^bAnd for each B e B,

step 3), performing global average pooling on the output of the last layer of the graph volume block:

step 4), inputting a softmax layer to classify the categories: y is_pre＝softmax(f_g):

Step 5), the frequency domain obtained by modeling and the brain function connection related to the signal mode are as follows:

for each multi-domain map volume Block^bIt is composed of space map convolutional layer and time convolutional layer.

7. The electroencephalogram feature recognition method based on the multi-domain adaptive graph convolution neural network as claimed in claim 1, wherein the spatial graph convolution refers to:wherein: w is C_out×C_inX 1 convolution weight vector, f_inTo input the frequency domain features, convolution is performed for the first layer spatial map,α₁,α₁,α₁a trainable weight magnitude parameter for each connection matrix; a. the^TeFor the above-mentioned temporally extracted signal pattern related brain connections,is a common weighted adjacency matrix shared by all samples and is set as a trainable parameter, an adjacency matrix private to each sample And W_τIs a weight vector of 1 × 1 convolution operation for converting the input frequency domain feature f_inMapping into the embedding space, and measuring the connection strength between channels of each feature through dot product.

8. The method for recognizing the electroencephalogram characteristic based on the multi-domain adaptive graph convolution neural network according to claim 1, wherein the time convolution refers to: performing convolution kernel on time dimension T of input feature to obtain K_tConvolution operation of x 1.

Technical Field

The invention relates to a technology in the field of electroencephalogram signal processing, in particular to an emotion electroencephalogram feature representation method based on a multi-domain self-adaptive graph convolution neural network.

Background

Electroencephalogram features can be classified into three categories, time domain, frequency domain, and time-frequency domain. Electroencephalogram signals are discrete time sequences, which indicates that the time domain may contain important information for signal pattern recognition. In the time domain, the most widely used electroencephalograms feature fractal dimension, high-order cross. Due to the unsteady characteristics of electroencephalogram signals and the interference of noise and artifacts of the original electroencephalogram data, frequency domain characteristics of power spectral density and differential entropy, as well as some time-frequency domain characteristics, are widely adopted by researchers, wherein the frequency domain differential entropy characteristics are well represented in the task of electroencephalogram-based signal pattern recognition.

Disclosure of Invention

Aiming at the problem that the prior art does not fully utilize information of electroencephalogram in time domain, frequency domain and functional brain connection at the same time, the invention provides an emotional electroencephalogram feature representation method based on a multi-domain self-adaptive graph convolution neural network, which fuses information of the frequency domain and the time domain of electroencephalogram and brain function connection related to signal mode, fully utilizes complementary information of a plurality of domains through the multi-domain self-adaptive graph convolution neural network, considers the topological structure of electroencephalogram channels at the same time, and learns brain function connection in a self-adaptive manner to realize identification.

The invention is realized by the following technical scheme:

the invention relates to an electroencephalogram feature recognition method based on a multi-domain self-adaptive graph convolution neural network.

The time domain and frequency domain preprocessing and feature extraction specifically comprise: the method comprises the steps of time domain preprocessing, time domain feature extraction, time domain preprocessing and frequency domain feature extraction which are sequentially carried out.

The time domain preprocessing refers to: the method comprises the steps of performing baseline correction and artifact removal processing on acquired electroencephalogram data, performing filtering processing on electroencephalogram signals, and dividing the electroencephalogram signals into 5 frequency bands through a filter. Then, the electroencephalogram signals are divided according to the S second non-overlapping time window, and each sample obtained by the method can be composed of 5V electroencephalogram fragments with the duration of S seconds and respectively corresponds to 5 frequency bands.

The time domain feature extraction is as follows: extracting time-domain signal mode related functional brain connection A^TeThe method comprises the following specific steps:

1) for each sample, calculating the Pearson correlation coefficient between electroencephalogram signals with the time length of each channel being S seconds under each frequency band, representing the obtained relation between the pair of electroencephalogram signals under each frequency band of each sample by using a V multiplied by V symmetrical connection matrix A, representing the connection weight between the electroencephalogram signals of every two channels by elements in the matrix, namely the information of edges in the brain function connection network, and finally obtaining a function connection matrixWherein: n is the number of samples after pretreatment, F represents 5 frequency bands, and V multiplied by V is the dimension of a time domain connection matrix obtained by calculation according to the Pearson correlation coefficient.

2) All tested samples in a training set are used together to select functional brain connections related to signal pattern recognition, L is a signal pattern class set, all samples and all tested samples are averaged by all functional connection matrixes in the training set on each frequency band F e F and each type of signal pattern L e L, namely all connection matrixes with labels L are averaged, and the expression form is as follows:wherein:representing the connection matrix corresponding to the i-th sample in the f-band, yⁱIndicating the signal mode class corresponding to the ith sample.

3) Will matrixThe top right-hand elements of (1) are sorted from large to small according to the absolute value of the connection weight.

4) Using the obtained F x L average brain networks to reserve the strongest connection by using the same proportion threshold value t to obtainKey connections in each type of signal mode:

5) merging the key connections reserved in the average brain network of L signal pattern classes, wherein the expression form is as follows:

6) merging key connections reserved in the average brain network of the F frequency bands, wherein the expression form is as follows: a. the_critical＝

union_f∈F(A^f)。

7) Normalizing the obtained key connection, and calculating the time domain signal mode related functional brain connection A^Te＝normalizing(A_critical)。

The frequency domain preprocessing refers to: and performing baseline correction and artifact removal processing on the acquired electroencephalogram data, and performing filtering processing on the electroencephalogram signals.

The frequency domain feature extraction is to divide the electroencephalogram signal into 5 frequency bands through short-time Fourier transform, extract the differential entropy frequency domain feature of each frequency band, and extract the extracted frequency domain feature for acquiring an electroencephalogram time sequence Using time windows T to convert to

The multi-domain adaptive graph convolution network comprises: a plurality of multi-domain adaptive graph volume blocks, a global average pooling layer, a softmax layer, wherein: the multi-domain self-adaptive volume block is fused with electroencephalogram time domain and frequency domain features, features related to signal mode identification are extracted, a pooling layer carries out global average pooling on output features of the last layer of volume block, and a softmax layer classifies the features output after pooling to obtain signal mode categories.

The online feature recognition specifically includes:

step 1) frequency domain characteristics are combined in a multi-domain self-adaptive graph convolution networkSet as input to the first graph volume block:

step 2) the volume Block of the b-th graph is Block^bAnd for each B e B,

step 3), performing global average pooling on the output of the last layer of the graph volume block:

step 4), inputting a softmax layer to classify the categories: y is_pre＝softmax(f_g)。

Step 5), the frequency domain obtained by modeling and the brain function connection related to the signal mode are as follows:

for each multi-domain map volume Block^bIt is composed of space map convolutional layer and time convolutional layer.

The convolution of the space map refers to:wherein: w is C_out×C_inX 1 convolution weight vector, f_inTo input the frequency domain features, convolution is performed for the first layer spatial map,α₁,α₁,α₁a trainable weight magnitude parameter for each connection matrix; a. the^TeFor the above-mentioned temporally extracted signal pattern related brain connections,is a common weighted adjacency matrix shared by all samples and is set as a trainable parameter, an adjacency matrix private to each sample And W_τIs a weight vector of 1 × 1 convolution operation for converting the input frequency domain feature f_inMapping into the embedding space, and measuring the connection strength between channels of each feature through dot product.

The time convolution refers to: performing convolution kernel on time dimension T of input feature to obtain K_tConvolution operation of x 1.

Technical effects

The invention simultaneously utilizes the information of the connection between the electroencephalogram time domain and the functional brain and the frequency domain to carry out signal mode identification, introduces the contribution of weight parameter balance time domain and frequency domain, fuses the functional connection between the single channel characteristic of the electroencephalogram and the channel, and learns the functional brain connection related to the signal mode in a self-adaptive manner, thereby integrally solving the defect that the prior art is lack of information of all domains of the electroencephalogram.

Compared with the prior art, the method combines the characteristics of the time domain and the frequency domain of the electroencephalogram signal with the brain topological structure information, adaptively learns the brain neural mode related to the signal mode, and improves the accuracy of electroencephalogram-based signal mode identification.

Drawings

FIG. 1 is a schematic diagram of the electroencephalogram signal channel topology of the present invention;

FIG. 2 is a multi-domain adaptive graph convolution neural network of the present invention;

FIG. 3 is a flow chart of the present invention;

FIGS. 4a, b, c are schematic diagrams of confusion matrices for distinguishing three, four, and five types of signal patterns according to the present invention, respectively;

FIG. 5 is a visualization of key brain functional connections associated with signal patterns in the present invention;

in the figure: SEED means the emotion electroencephalogram data set of Shanghai transportation university, and Block means each multi-domain map volume Block.

Detailed Description

As shown in fig. 1, the present embodiment relates to an electroencephalogram signal channel topology, which is G ═ V (V)^*,E^*) Wherein the top point V of the diagram represented by the electroencephalogram channel^*Through the edge E between the channels^*Connected together, as shown in FIG. 1, in which v₁,v₂,v₃Is a V^*Three brain electrical channel examples in E₁₂,e₁₃Each represents E^*Middle v₁v₂，v₁v₃The edge in between. Defining a weighted adjacency matrix A ∈ R^V×VTo represent edge E in the diagram^*The brain function connection is realized as a topological structure of the graph. Furthermore, the model of (a) can learn the functional connections of the brain in relation to the signal patterns in an adaptive manner.

As shown in fig. 3, this embodiment relates to a method for extracting emotion electroencephalogram features based on a multi-domain adaptive graph-convolution neural network, which includes the following steps:

the method comprises the following steps: according to the participation of a healthy subject in a signal mode experiment, watching a signal mode stimulation material to induce a corresponding signal mode of the subject, and acquiring the electroencephalogram data of the subject according to 10-20 international standard potential distribution through a 62-electroencephalogram cap.

Step two: and preprocessing the data and extracting time domain characteristics.

Step three: preprocessing the data and extracting frequency domain characteristics.

Step four: combining the electroencephalogram data of the time domain and the frequency domain with a brain topological structure, and fusing through a multi-domain self-adaptive graph convolution neural network to perform model training.

Step five: and inputting the tested test data into the trained neural network for prediction, and outputting a prediction signal mode label.

As shown in fig. 4, the model achieves better prediction effects on three-class, four-class and five-class signal modes.

As shown in fig. 5, the functional connections of the brain involved in signal pattern recognition are distributed primarily in the forehead area.

In the method, the time domain and frequency domain information of the electroencephalogram signals and the brain function connection related to the signal mode are combined into the brain function connection which is originally created and never disclosed, and the working mode of the brain function connection is different from that recorded in any existing literature: the existing literature does not fully utilize the time domain and frequency domain of electroencephalogram and brain topological structure information, and compared with the existing conventional technical means, the technical details which are obviously improved are as follows: and fusing the time domain and frequency domain information of the brain electricity with the brain topological structure information. And adaptively learns functional brain connections associated with signal patterns.

Through a specific practical experiment, the method is operated by using a PyTorch neural network framework and setting the learning rate parameter as 0.01, and experimental data can be obtained as follows: the accuracy rates of 94.81%, 87.63% and 80.77% are respectively achieved for the three-classification, the four-classification and the five-classification of the signal patterns.

The foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.