阅读说明：本技术 一种基于深度神经网络的可穿戴心电图实时诊断系统 (Wearable electrocardiogram real-time diagnosis system based on deep neural network ) 是由 袁烨 冉少林 周子恒 程骋 于 2021-08-23 设计创作，主要内容包括：本发明公开了一种基于深度神经网络的可穿戴心电图实时诊断系统,属于医疗器械技术领域,包括：信号采集模块和主控模块,所述信号采集模块,用于采集12导联心电图；所述主控模块包括嵌入FPGA的诊断模型,所述诊断模型为训练好的深度神经网络,所述主控模块,用于对12导联心电图进行实时诊断,在诊断过程中FPGA中的卷积加速器采用分时复用的方式为诊断模型中的每个卷积层配置参数完成卷积运算,得到诊断结果。本发明将诊断模型嵌入FPGA,减少对网络传输的依赖,实现了实时诊断,采用12导联心电图作为输入数据进行预测,提高诊断结果准确率,设计可重构卷积加速器,从而大大提高计算性能,满足可穿戴心电设备进行实时检测和准确诊断的要求。(The invention discloses a wearable electrocardiogram real-time diagnosis system based on a deep neural network, which belongs to the technical field of medical appliances and comprises the following components: the system comprises a signal acquisition module and a main control module, wherein the signal acquisition module is used for acquiring a 12-lead electrocardiogram; the main control module comprises a diagnosis model embedded in the FPGA, the diagnosis model is a trained deep neural network, the main control module is used for diagnosing 12-lead electrocardiograms in real time, and a convolution accelerator in the FPGA completes convolution operation for each convolution layer configuration parameter in the diagnosis model in a time-sharing multiplexing mode in the diagnosis process to obtain a diagnosis result. According to the invention, the diagnosis model is embedded into the FPGA, dependence on network transmission is reduced, real-time diagnosis is realized, a 12-lead electrocardiogram is used as input data for prediction, the accuracy of a diagnosis result is improved, and the reconfigurable convolution accelerator is designed, so that the calculation performance is greatly improved, and the requirements of real-time detection and accurate diagnosis of wearable electrocardiogram equipment are met.)

1. A wearable electrocardiogram real-time diagnosis system based on a deep neural network is characterized by comprising: a signal acquisition module and a main control module,

the signal acquisition module is used for acquiring a 12-lead electrocardiogram;

the main control module comprises a diagnosis model embedded in the FPGA, the diagnosis model is a trained deep neural network, the main control module is used for diagnosing 12-lead electrocardiograms in real time, and a convolution accelerator in the FPGA completes convolution operation for each convolution layer configuration parameter in the diagnosis model in a time-sharing multiplexing mode in the diagnosis process to obtain a diagnosis result.

2. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in claim 1, wherein the deep neural network comprises in sequence:

an input layer, a pre-convolution module, a maximum pooling layer, a first volume block, a second volume block, an average pooling layer, a full-link layer and a classifier,

the convolution kernel of the pre-convolution module is larger than that of the first convolution block, and the convolution kernel of the first convolution block is equal to that of the second convolution block; adding a batch normalization layer after each convolution layer in the pre-convolution module, the first convolution block and the second convolution block; and two dropout layers are arranged in the full connection layer.

3. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in claim 2, wherein the system further comprises:

a training module for training the deep neural network by using the 12-lead electrocardiogram sample with class label as the training set, wherein the training output is M class labels, M is more than or equal to 1, and the loss function L of the classifier in the training process_ωComprises the following steps:

wherein, y_j，cA value representing the true category label is shown,indicates a predicted class label value, ω_cIs a weight factor, and trainingThe number of samples of each category in the set is proportional, C is the category C, C is the total number of categories in the training set, j is the jth sample, and N is the total number of samples in the training set.

4. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in claim 3, wherein the weighting factors are:

wherein n is_cTotal number of samples, n, representing class c in the training set_c≤N。

5. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in claim 3 or 4, wherein the diagnosis model is embedded with FPGA after channel pruning is performed, and the specific implementation manner of the channel pruning is as follows:

taking a parameter gamma which needs to be learned in the batch normalization layer as a scale factor of network compression, and modifying a loss function into: l ═ L_ω+ λ Σ g (γ), where g (γ) is the sparsely induced penalty for the scale factor and λ is the equilibrium parameter;

and after the modified loss function is used for training the deep neural network, removing the channel of the batch normalization layer with smaller gamma in the deep neural network and the weight in the convolution kernel connected with the channel.

6. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in claim 5, wherein the deep neural network is trained by using the modified loss function, and the sub-gradient descent optimization is performed on the non-differentiable sparse induced penalty term in the loss function.

7. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in any one of claims 1-4, wherein the parameters, inputs and intermediate data of the diagnosis model are quantized to unsigned integers.

8. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in any one of claims 2-4, wherein the main control module further comprises a processor for performing floating point operation on the output data in the classifier to obtain the prediction result.

9. The wearable electrocardiogram real-time diagnosis system based on the deep neural network as claimed in any one of claims 1 to 4, wherein the master control module further comprises an internal memory, the internal memory comprises an input buffer, a weight buffer and an output buffer, which are respectively used for buffering the input, weight and output data of the diagnosis model.

10. The wearable electrocardiogram real-time diagnosis system based on the deep neural network of claim 9, wherein the input buffer, the weight buffer and the output buffer are buffered by using ping-pong strategy to cover the communication delay and the calculation delay.

Technical Field

The invention belongs to the technical field of medical instruments, and particularly relates to a wearable electrocardiogram real-time diagnosis system based on a deep neural network.

Background

The electrocardiogram is one of the basic tools widely used in clinic for diagnosing the heart health condition of a patient, and can record abnormal electrocardiosignals when heart diseases occur. Therefore, the timely and accurate electrocardiogram diagnosis can strive for valuable treatment time for the patient.

With the rise of deep learning, the electrocardiogram classification diagnosis method based on the deep neural network occupies an increasingly critical position. However, the hardware of the electrocardiogram classification device based on deep learning is often large and complicated in calculation, and is not suitable for use in wearable devices and the like. At present, the wearable device mainly adopts a single-lead detection mode, but the accuracy is not enough, and the types of diagnosable diseases are limited; and many models are limited by large calculated amount, and usually a cloud computing mode is adopted for real-time diagnosis, so that the dependence on a network is increased, and when the network is disconnected, the wearable equipment cannot work normally, so that the health monitoring of a patient is seriously influenced.

Therefore, the prior art has the technical problems of low accuracy, strong dependence on a network and incapability of real-time diagnosis.

Disclosure of Invention

Aiming at the defects or the improvement requirements of the prior art, the invention provides a wearable electrocardiogram real-time diagnosis system based on a deep neural network, so that the technical problems of low accuracy, strong dependence on the network and incapability of real-time diagnosis in the prior art are solved.

To achieve the above object, according to one aspect of the present invention, there is provided a wearable electrocardiogram real-time diagnosis system based on a deep neural network, comprising: a signal acquisition module and a main control module,

the signal acquisition module is used for acquiring a 12-lead electrocardiogram;

the main control module comprises a diagnosis model embedded in the FPGA, the diagnosis model is a trained deep neural network, the main control module is used for diagnosing 12-lead electrocardiograms in real time, and a convolution accelerator in the FPGA completes convolution operation for each convolution layer configuration parameter in the diagnosis model in a time-sharing multiplexing mode in the diagnosis process to obtain a diagnosis result.

Further, the deep neural network sequentially comprises:

an input layer, a pre-convolution module, a maximum pooling layer, a first volume block, a second volume block, an average pooling layer, a full-link layer and a classifier,

the convolution kernel of the pre-convolution module is larger than that of the first convolution block, and the convolution kernel of the first convolution block is equal to that of the second convolution block; adding a batch normalization layer after each convolution layer in the pre-convolution module, the first convolution block and the second convolution block; and two dropout layers are arranged in the full connection layer.

Further, the system further comprises:

a training module for training the deep neural network by using the 12-lead electrocardiogram sample with class label as the training set, wherein the training output is M class labels, M is more than or equal to 1, and the loss function L of the classifier in the training process_ωComprises the following steps:

wherein, y_j，cA value representing the true category label is shown,indicates a predicted class label value, ω_cIs a weight factor which is proportional to the number of samples of each class in the training set, C is the class C, C is the total number of classes in the training set, j is the jth sample, and N is the total number of samples in the training set.

Further, the weighting factor is:

wherein n is_cTotal number of samples, n, representing class c in the training set_c≤N。

Furthermore, the diagnostic model is embedded into the FPGA after channel pruning is performed, and the specific implementation manner of the channel pruning is as follows:

taking a parameter gamma which needs to be learned in the batch normalization layer as a scale factor of network compression, and modifying a loss function into: l ═ L_ω+ λ Σ g (γ), where g (γ) is the sparsely induced penalty for the scale factor and λ is the equilibrium parameter;

and after the modified loss function is used for training the deep neural network, removing the channel of the batch normalization layer with smaller gamma in the deep neural network and the weight in the convolution kernel connected with the channel.

Further, when the modified loss function is used for training the deep neural network, sub-gradient descent optimization is performed on the non-differentiable sparse induction penalty term in the loss function.

Further, the parameters, inputs and intermediate data of the diagnostic model are quantized to unsigned integers.

Furthermore, the main control module further comprises a processor, and the processor is used for operating the floating point operation of the output data in the classifier to obtain the prediction result.

Further, the main control module further comprises an internal memory, wherein the internal memory comprises an input cache, a weight cache and an output cache, and the internal memory is respectively used for caching input data, weight data and output data of the diagnosis model.

Furthermore, ping-pong strategies are adopted for the buffering modes of the input buffer, the weight buffer and the output buffer to cover communication delay and calculation delay.

In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:

(1) the main control module comprises a diagnosis model embedded in the FPGA, dependence on network transmission is reduced, real-time diagnosis is realized, a 12-lead electrocardiogram is used as input data for prediction, the accuracy of a diagnosis result is improved, a reconfigurable convolution accelerator is designed, and the forward reasoning process of the model is realized through time-sharing multiplexing. The convolution accelerator completes convolution operation for each convolution layer configuration parameter in the diagnosis model, and makes full use of the logic resource of each convolution layer, thereby greatly improving the calculation performance. The invention has high disease diagnosis accuracy and strong real-time property, and meets the requirements of real-time detection and diagnosis of wearable electrocardio equipment.

(2) Two dropout layers are arranged in the full connection layer in the network, so that the problem of over-fitting is solved. Designing a pre-convolution module with a convolution kernel with a larger size, wherein the pre-convolution module is used for obtaining a wide receptive field and global characteristics (the global waveform state of an electrocardiogram), and the pre-convolution module with a plurality of convolution neural networks with a smaller size is used for learning high-level characteristics (abstract characteristics for subsequent classification); the method is a process for compressing data volume and simplifying operation, a batch normalization layer (BN) is added after each convolution layer to reduce internal covariate offset, training output is M class labels, and the final result can be one or a multi-label classification result.

(3) To solve the data imbalance problem, a larger number of samples corresponds to a smaller weight, and a smaller number of samples corresponds to a larger weight. For multi-class tasks, a traditional binary cross-entropy loss is extended by introducing a weighting factor proportional to the number of class samples.

(4) The deep neural network diagnosis model has huge calculation amount and strict requirements on hardware resources, and the invention aims to carry out real-time diagnosis on heart diseases by using limited resources on an embedded platform, so that a software and hardware collaborative optimization method is provided. By compressing the model by removing channels that contribute less to the classification result, the compressed model does not require a dedicated sparse matrix operation library and reduces the consumption of on-chip memory cells. The floating-point arithmetic operation is converted into the integer operation, so that a large amount of on-chip hardware resources and external storage bandwidth are saved, and meanwhile, the power consumption in the calculation and data transmission processes is reduced. The reconfigurable convolution accelerator is designed, so that the calculation performance is greatly improved. Ping-pong strategies are employed to cover communication delays and computational delays. In order to alleviate the deficiency of on-chip BRAM, a cyclic block optimization technology is adopted to divide data extracted from a memory into data blocks suitable for caching so as to ensure that data accessed in a computing area stored in the cache can be reused.

Drawings

Fig. 1 is a block diagram of a wearable electrocardiogram real-time diagnosis system based on a deep neural network according to an embodiment of the present invention;

FIG. 2 is a diagram of a model architecture of a deep convolutional neural network based on multi-scale according to an embodiment of the present invention;

fig. 3 is a system architecture diagram of a hardware platform provided by an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

As shown in fig. 1, a wearable electrocardiogram real-time diagnosis system based on a deep neural network comprises: a signal acquisition module and a main control module,

the signal acquisition module is used for acquiring a 12-lead electrocardiogram;

the main control module comprises a diagnosis model embedded in the FPGA, the diagnosis model is a trained deep neural network, the main control module is used for diagnosing 12-lead electrocardiograms in real time, and a convolution accelerator in the FPGA completes convolution operation for each convolution layer configuration parameter in the diagnosis model in a time-sharing multiplexing mode in the diagnosis process to obtain a diagnosis result.

As shown in fig. 2, Input in the figure represents Input electrocardiographic data, PreConv represents a pre-Convolution module, Conv Block1 represents a first volume Block, Conv Block2 represents a second volume Block, Max power represents a maximum Pooling layer, Average power represents an Average Pooling layer, contribution represents a volume layer, BatchNorm represents a batch normalization layer, ReLU represents a rectifying linear unit, density represents a Dense Block, Ch represents a channel, and Stride represents a Stride.

The large-scale deep neural network model constructed by the invention sequentially comprises the following steps:

the system comprises an input layer, a pre-convolution module, a maximum pooling layer, a first volume block, a second volume block, an average pooling layer, a full-link layer and a classifier.

Specifically, in the present embodiment, convolution kernels of two sizes, 21 and 5, are selected. Since the input data is long, a large convolution kernel and 32 filters are applied to the original input in the first three convolutional layers with 2 steps to learn the global features. Subsequent layers are stacked by convolutional layers with a small kernel of kernel size 5 and 1 step.

Referring to fig. 2, in the first convolution block, the number of convolution filters in the first group is 64 and then doubled until 256 is reached. The last convolution block (second convolution block) is fast downsampled using three maximum pool operations.

Two dropout layers are placed in the full-connection layer to avoid the problem of overfitting.

The pre-convolution module is provided with a convolution kernel with a larger size and is used for obtaining wide receptive field and global characteristics, and the pre-convolution module is provided with a plurality of convolution neural networks with a smaller size and is used for learning high-level characteristics; adding a bulk normalization layer (BN) after each convolutional layer to reduce internal covariate offset, wherein an activation function adopts ReLU; and a multi-label classifier with a sigmoid function is used for the final multi-label classification result. The classifier outputs the probability of the corresponding label on an output layer by adopting a binary cross entropy loss function.

Loss function L of classifier in training process_ωComprises the following steps:

wherein, y_j，cA value representing the true category label is shown,indicates a predicted class label value, ω_cIs a weight factor proportional to the number of samples in each class in the training set, and c is the c-th classIn other words, C is the total number of classes in the training set, j is the jth sample, and N is the total number of samples in the training set.

Further, the weighting factor is:

wherein n is_cTotal number of samples, n, representing class c in the training set_c≤N。

As shown in FIG. 3, PS and PL denote the processing system portion and the programmable logic portion, respectively; CPU represents a general purpose processor; DRAM represents external storage; PE0, PE1, PE2, PE3 represent 4 Processing Elements; the Input buffer, Weight buffer and Output buffer respectively represent an Input buffer, a Weight buffer and an Output buffer. BRAM is the on-chip memory realized by using FPGA unit, control unit represents the control unit, and on the fly logic represents the dynamic logic. And designing a reconfigurable convolution accelerator, and realizing the forward reasoning process of the model through time-sharing multiplexing. The reconfigurable accelerator can adapt to convolution operation of different specifications by configuring parameters (including convolution kernel size, channel number and step length) of a certain layer of convolution under the condition of giving hardware resources, and fully utilizes logic resources of each convolution layer on a chip, thereby greatly improving the calculation performance.

The PS part is mainly used for prediction of data and is responsible for calculation of sigmoid functions. PL is responsible for convolution operation, and the cache part on the chip comprises an input cache, an output cache and a weight cache and is responsible for data preparation and calculation result storage. Each type of buffer employs a ping-pong strategy to cover communication delays and computation delays. The ping-pong strategy comprises: the processing result of the previous stage is stored in the cache of the pong path, when the data of the pong path is ready, the data of the ping path (the next stage) is processed, the next stage can directly process the data of the pong path without waiting, and the previous stage can store the result in the ping path. The situation that the upper level can send new data only after the lower level finishes processing the data is avoided, and therefore efficiency is improved.

The processing element is a convolution calculation engine. The PS part includes a general purpose processor (CPU) and external memory. The Sigmoid function is realized in the CPU, and the floating point operation occupies more resource overhead in the FPGA and has less performance improvement, so that the method is more suitable for the CPU. All model parameters and instructions are stored in the external memory DRAM.

In order to alleviate the deficiency of on-chip BRAM, a cyclic block optimization technology is adopted to divide data extracted from a memory into data blocks suitable for caching so as to ensure that data accessed in a computing area stored in the cache can be reused. Meanwhile, a cyclic expansion strategy is adopted, the inherent parallelism of the FPGA is fully utilized, and the convolution operation is accelerated.

The diagnosis model is embedded into the FPGA after channel pruning is carried out, and the specific implementation mode of the channel pruning is as follows:

taking a parameter gamma which needs to be learned in the batch normalization layer as a scale factor of network compression, and modifying a loss function into: l ═ L_ω+ λ Σ g (γ), where g (γ) is the sparsely induced penalty for the scale factor and λ is the equilibrium parameter;

and after the modified loss function is used for training the deep neural network, removing the channel of the batch normalization layer with smaller gamma in the deep neural network and the weight in the convolution kernel connected with the channel.

Further, when the modified loss function is used for training the deep neural network, sub-gradient descent optimization is performed on the non-differentiable sparse induction penalty term in the loss function.

Using L1 norm as sparsity penalty function, L ═ L_ω+λ||γ||₁λ is 5 x 10^-4. After sparse induction training, fine tuning is required to retrain the pruned model to restore model performance.

Further, the BN operation and the convolution operation in the channel pruning are fused to reduce consumption of computing resources.

Converting floating point arithmetic operation into integer operation, namely quantizing model parameters, weights and intermediate characteristic diagrams in the model so as to save a large amount of on-chip hardware resources and external storage bandwidth and reduce power consumption in the processes of calculation and data transmission; the quantization scheme maps the model parameters to corresponding unsigned integers. Where the input to the model, the model parameters and the intermediate values are all quantized to 8-bit unsigned integers.

Example 1

The network training comprises the following steps:

step 1, collecting standard 12-lead electrocardiosignals of a patient.

In this embodiment, standard 12-lead electrocardiographic data is acquired by a patch electrode, then the data is preprocessed, including low-pass filtering, high-pass filtering, notch processing, and an AD conversion module, so as to obtain high-quality discrete data of electrocardiographic signals.

And 2, training the constructed large-scale electrocardiogram data set to obtain a multi-label electrocardiogram diagnosis model.

Specifically, the electrocardiographic signals in the data set used were collected from three branches of the Tongji Hospital of Huazhong university of science and technology. All cardiac signals were acquired at 500Hz for 10s using a standard 12 lead instrument. Contains 26 central cardiac rhythms: normal (N), Sinus Tachycardia (ST), Sinus Bradycardia (SB), atrial premature beats (PAC), Atrial Rhythm (AR), Accelerated Atrial Rhythm (AAR), Atrial Tachycardia (AT), atrial flutter (AFlutter), atrial fibrillation (AFib), junctional premature beats (PJC), Junctional Rhythm (JR), Accelerated Junctional Rhythm (AJR), paroxysmal supraventricular tachycardia (PSVT), ventricular premature beats (PVC), unique ventricular rhythm (IVR), accelerated unique ventricular rhythm (AIVR), Ventricular Tachycardia (VT), Artificial Atrial Pacing (AAPR), Artificial Ventricular Pacing (AVPR) right bundle branch conduction block (RBBB), primary atrioventricular conduction block (first AVB), mobbetz secondary atrioventricular conduction block (mobz primary AVB), mobbetz secondary ventricular conduction block (mobz atrioventricular AVB), waltz-bev syndrome (wpa-w), and wpa-w. The total number of samples is 206468.

Duplicate patients were removed from the dataset so that there was no overlap between the groups. The data set was randomly divided into a training set, a validation set, and a test set, accounting for 80%, 10%, and 10% of the data set, respectively.

Further, independent external public data provided by the Chinese physiological Signal challenge service (SPSC 2018) is used for external verification. This data set has different abnormalities in the electrocardiographic signals and longer recording time than the data set of the hospital of the same institution, and therefore the first 10 electrocardiograms in this data set are trimmed to ensure that the data pattern is consistent with the input format of the proposed model of the present invention. 962 records of six types of rhythms (normal and five abnormal rhythms) in the data set were finally used for external verification.

And 3, constructing a large-scale deep neural network model.

Specifically, in the present embodiment, convolution kernels of two sizes, 21 and 5, are selected. Since the input data is long, a large convolution kernel and 32 filters are applied to the original input in the first three convolutional layers with 2 steps to learn the global features. Subsequent layers are stacked by convolutional layers with a small kernel of kernel size 5 and 1 step.

In the first convolution block, the number of convolution filters in the first set is 64, and then doubled until 256 is reached. The last convolution block (second convolution block) is fast downsampled using three maximum pool operations.

Two dropout layers are placed in the full-connection layer to avoid the problem of overfitting.

In this example, the model was initialized with random weights using the xavier initialization technique and optimized using the Adam optimizer.

In this example, the performance of the model was evaluated using evaluation indexes such as sensitivity, specificity, F1 score, and accuracy.

And then embedding the trained diagnostic model into the FPGA to perform software and hardware collaborative optimization.

Specifically, the method comprises channel pruning, model parameter quantification, design of a reconfigurable convolution accelerator and optimization of a hardware architecture.

In this example, the method of structured pruning is used to remove channels that contribute less to the classification result.

Based on the compromise analysis of the number of pruning channels and the model, in this embodiment, the model of the final pruning is generated with a pruning rate of 50%. The pruned model is retrained to restore performance after channel pruning.

In this embodiment, on the basis of the pruning model, a quantization model is obtained by an integer quantization strategy.

Specifically, it is considered that the serial operation mode of the CPU is not suitable for parallel computation of the CNN model; the high utilization of GPUs relies on large volumes, making real-time reasoning impractical. Thus, in an embodiment, a Xilinx Zynq XC-7Z020 FPGA chip is used.

In the reasoning process of the diagnosis model, the designed reconfigurable convolution accelerator is subjected to time-sharing multiplexing, and hardware acceleration calculation of different convolutions is performed by configuring different parameters. The Vivado HLS (2019.2) is used for realizing the design of an accelerator, the size of operation is optimized, data can be reused to the maximum extent, data transmission is reduced, and the BRAM is fully utilized.

The peak power consumption of the embedded computing platform constructed in the embodiment is 2.81W. Meanwhile, the deducing time of one electrocardiogram sample classification is 2.895 seconds in the embodiment.

In conclusion, the model constructed by the embodiment of the invention can accurately diagnose the multi-label patient electrocardiogram data; the hardware platform obtained by optimization in the embodiment of the invention can diagnose the electrocardio-diseases in real time; the embodiment of the invention realizes the wearable electrocardiogram real-time diagnosis system based on the deep neural network.

It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.