阅读说明：本技术 卷积神经网络判断根据提取方法及装置 (Convolutional neural network judgment basis extraction method and device ) 是由 福原诚史 藤原一彦 丸山芳弘 于 2020-02-25 设计创作，主要内容包括：本发明的卷积神经网络判断根据提取装置(1)具备贡献率计算部(2)及根据提取部(3)。贡献率计算部(2)求得全结合层的权重(36)相对于输出层的任一输出标签的贡献率(β-(c,m))。根据提取部(3)基于输入至全结合层的特征量图(25)、全结合层的权重(36)及上述贡献率(β-(c,m)),提取CNN的判断的根据。由此,可实现即使在CNN的隐藏层较少的情况、或卷积层中所使用的滤波器的尺寸较小的情况下,也可提取输入的数据中成为基于CNN的判断的根据的特征区域的方法及装置。(A convolutional neural network judgment device (1) is provided with a contribution rate calculation unit (2) and an extraction unit (3). A contribution rate calculation unit (2) determines the contribution rate (beta) of the weight (36) of the full-binding layer to any output label of the output layer c,m ). The extraction unit (3) extracts a feature quantity map (25) based on the input to the full-bonding layer, the weight (36) of the full-bonding layer, and the contribution ratio (beta) c,m ) The basis of the judgment of the CNN is extracted. Thus, it is possible to realize a method and an apparatus for extracting a feature region that is a basis for the determination by the CNN from the input data even when the hidden layer of the CNN is small or the size of the filter used in the convolutional layer is small.)

1. A convolutional neural network decision basis extraction method is characterized in that,

is a method for extracting the basis of judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer,

the disclosed device is provided with:

a contribution ratio calculation step of obtaining a contribution ratio of the weight of the full-binding layer to any output label of the output layer; and

and extracting the basis based on the feature quantity map input to the full-bonding layer, the weight of the full-bonding layer and the contribution rate.

2. A convolutional neural network decision basis extraction method is characterized in that,

is a method for extracting the basis of judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer,

the disclosed device is provided with:

a contribution ratio calculation step of obtaining a contribution ratio of a feature vector generated by the full-bonding layer with respect to any output label of the output layer; and

and extracting the basis based on the feature quantity map input to the full-bonding layer, the weight of the full-bonding layer and the contribution rate.

3. The convolutional neural network decision basis extraction method as claimed in claim 1 or 2,

further provided with: and a display step of establishing correspondence with the input data input to the input layer and displaying the basis.

4. A convolutional neural network decision basis extraction device is characterized in that,

is a device for extracting the basis of judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer,

the disclosed device is provided with:

a contribution ratio calculation unit that obtains a contribution ratio of the weight of the full-binding layer to any output label of the output layer; and

and a feature value extracting unit that extracts the feature value based on the feature value map input to the full-bonding layer, the weight of the full-bonding layer, and the contribution ratio.

5. A convolutional neural network decision basis extraction device is characterized in that,

is a device for extracting the basis of judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer,

the disclosed device is provided with:

a contribution ratio calculation unit that obtains a contribution ratio of the feature vector generated by the full-bonding layer to any output label of the output layer; and

an extraction unit extracts the basis on the feature quantity map input to the full-bonding layer, the weight of the full-bonding layer, and the contribution ratio.

6. The convolutional neural network decision basis extraction means as claimed in claim 4 or 5,

further provided with: and a display unit which corresponds to the input data input to the input layer and displays the basis.

Technical Field

The invention relates to a method and a device for extracting judgment basis of a convolutional neural network.

Background

In general, Deep Neural Network (DNN) based classification can achieve a high correct answer rate. On the other hand, however, the calculation process at the time of DNN-based classification is difficult for human beings to judge. Therefore, it is desirable to visualize the calculation process or the judgment reference of the learning pattern in a human-understandable manner for the entire DNN-based learning pattern to evaluate whether or not the learning pattern is appropriate.

A Convolutional Neural Network (hereinafter, referred to as "CNN"), which is one of DNNs, is used in the field of image recognition and the like, and recently, application examples have been reported in the field of spectrum analysis (see patent document 1 and non-patent documents 1 and 2). In the field of spectrum analysis, high results have been obtained over the years using principal component analysis for extracting feature amounts, a classifier such as a Support vector machine (svm), and the like. In recent years, CNN is also being used in the field of spectrum analysis, and results are reported.

In the field of image recognition based on CNN, a technique is known in which a feature region that is a basis for classification based on CNN in an input image is displayed on the input image (see non-patent document 3). According to this technique, it is possible to evaluate whether the CNN-based learning mode is appropriate. However, in the field of spectrum analysis of CNN, a technique for obtaining a feature region that is a basis for classification by CNN in an input spectrum has not been known.

Documents of the prior art

Patent document

Patent document 1: japanese patent No. 643549

Non-patent document

Non-patent document 1: liu et al, "Deep connected neural networks for Raman scattering reception," analysis, 142,21, pp.4067-4074(2017)

Non-patent document 2: j.acquirelli et al, "structural neural networks for visual spectroscopic data analysis", anal.chim.acta,954, pp.22-31(2017)

Non-patent document 3: R.R. Selvaraju et al, "Grad-CAM: Visual extensions from Deep nets via Gradient-based Localization", arXiv:1610.02391v3(2017)

Disclosure of Invention

Problems to be solved by the invention

According to the study of the present inventors, the technique described in non-patent document 3 is applied to the spectrum analysis by CNN, and as a result, it is difficult to find a feature region that becomes a basis for classification by CNN. This is considered to be due to the following reason.

In the case of image recognition by CNN, CNN needs to have a deep network structure in which a hidden layer also reaches 16 layers or more. The technique described in non-patent document 3 is to calculate a feature amount map obtained by calculating a CNN convolutional layer or pooling layer, and to display a feature region, which is a classification basis in an input image, on the input image.

In contrast, when spectrum analysis is performed by CNN, CNN is sufficient in a network structure having a small number of hidden layers (several layers). In such a network structure, as described in non-patent document 3, in the calculation based on the feature quantity map obtained by the calculation of the convolutional layer or the pooling layer, it is difficult to obtain a feature region which becomes a classification basis based on the CNN in the input spectrum. Further, since the size of the filter used in the convolutional layer is considered to be the degree of the line width of the spectrum, it is conceivable that only the shape information can be obtained in the calculation based on the feature quantity map rather than the position information.

Such a problem is thought to exist not only in the case where CNN is applied to the field of spectrum analysis, but also in the case where the hidden layer of CNN is small or in the case where the size of the filter used in the convolutional layer of CNN is small.

An object of the present invention is to provide a method and an apparatus for extracting a feature region that is a basis for a CNN determination from input data even when a hidden layer of a CNN is small or a filter used for a convolutional layer is small in size.

Means for solving the problems

The embodiment of the invention is that the convolutional neural network judgment is based on an extraction method. The judgment basis extraction method is a method for extracting a basis for judging a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combination layer, and an output layer, and includes the steps of: calculating a contribution ratio of the weight of the full-binding layer to the contribution ratio of any output label of the output layer; and extracting the basis of the feature quantity graph input into the full-bonding layer, the weighting and the contribution rate of the full-bonding layer.

The embodiment of the invention is an extraction method based on the judgment of a convolutional neural network. The method for extracting a judgment basis is a method for extracting a judgment basis of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer, and includes the steps of: a contribution ratio calculation step of calculating a contribution ratio of the feature quantity vector generated by the full-combined layer with respect to any output label of the output layer; and extracting the basis of the feature quantity graph input into the full-bonding layer, the weighting and the contribution rate of the full-bonding layer. The feature vector is generated based on the feature map input to the full-join layer and the weighting of the full-join layer.

An embodiment of the present invention is an extraction device based on the judgment of a convolutional neural network. The judgment basis extracting device extracts a basis for judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer, and includes: a contribution ratio calculation unit that obtains a contribution ratio of the weight of the full-binding layer to any output label of the output layer; and a basis extraction unit that extracts a basis on the feature quantity map input to the full-bonding layer, the weighting and the contribution rate of the full-bonding layer.

An embodiment of the present invention is a convolutional neural network decision basis extraction device. The judgment basis extracting device extracts a basis for judgment of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer, and includes: a contribution ratio calculation unit that obtains a contribution ratio of the feature vector generated by the full-boding layer with respect to any output label of the output layer; and a basis extraction unit that extracts a basis on the feature quantity map input to the full-bonding layer, the weighting and the contribution rate of the full-bonding layer.

ADVANTAGEOUS EFFECTS OF INVENTION

According to the embodiment of the present invention, even when the number of hidden layers of CNN is small or the size of the filter used in the convolutional layer is small, it is possible to extract a feature region that is a basis for the CNN determination from the input data.

Drawings

Fig. 1 is a diagram showing an example of the configuration of a convolutional neural network.

Fig. 2 is a diagram showing a configuration of the convolutional neural network determination basis extraction device.

Fig. 3 is a diagram showing another configuration example of the convolutional neural network.

Fig. 4 is a diagram showing the feature regions obtained in example 1 as the basis of classification.

Fig. 5 is a diagram showing the feature regions obtained in example 1, which are the basis of classification.

Fig. 6 is a diagram showing the characteristic regions obtained in example 1, which are the basis of classification.

Fig. 7 (a) is a diagram showing a feature region that is a basis of classification obtained in example 2, and fig. 7 (b) is an enlarged view showing a part of fig. 7 (a).

Fig. 8 (a) is a diagram showing a feature region that is a basis of classification obtained in example 2, and fig. 8 (b) is an enlarged view showing a part of fig. 8 (a).

Fig. 9 is a diagram showing an example of the frequency spectrum of each of the 9 drugs used in example 3.

Fig. 10 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug a).

Fig. 11 is a diagram showing characteristic regions obtained in example 3 (drug B) which are the basis of classification.

Fig. 12 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug C).

Fig. 13 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (medicine D).

Fig. 14 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug E).

Fig. 15 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug F).

Fig. 16 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug G).

Fig. 17 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug H).

Fig. 18 is a diagram showing characteristic regions that are the basis of classification obtained in example 3 (drug I).

FIG. 19 is a diagram showing an example of the frequency spectrum of each of the 20 kinds of amino acids used in example 4.

FIG. 20 is a graph showing a pure spectrum of alanine (Ala) used in example 4.

Fig. 21 is a diagram showing the feature regions obtained in example 4, which are the basis of classification.

Fig. 22 is a diagram showing the feature regions obtained in example 4, which are the basis of classification.

Fig. 23 is a diagram showing a configuration of the convolutional neural network determination basis extraction device.

Fig. 24 is a diagram showing the feature regions obtained in example 3, which are the basis of classification.

Fig. 25 is a diagram showing the feature regions obtained in example 5, which are the basis of classification.

Fig. 26 is a diagram showing the feature regions obtained in example 4, which are the basis of classification.

Fig. 27 is a diagram showing the feature regions obtained in example 6 as the basis of classification.

Detailed Description

Hereinafter, embodiments of the convolutional neural network decision method and device will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. The present invention is not limited to these examples.

Fig. 1 is a diagram showing an example of the configuration of a convolutional neural network. A Convolutional Neural Network (CNN)10 of the configuration example shown in the figure includes: input layer 11, convolutional layer 12, pooling layer 13, convolutional layer 14, pooling layer 15, full-bonding layer 16, and output layer 17. CNN10 may be implemented by a CPU (Central Processing Unit), or may be implemented by a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) that can perform higher-speed Processing. CNN10 has a memory for storing various data and parameters.

The convolutional layer 12 generates the feature quantity map 22 by applying the filter 32 to the input data sequence 21 input to the input layer 11. The convolutional layer 12 moves the filter 32 relative to the input data sequence 21, and performs convolution operation of the input data sequence 21 and the filter 32 at each position to generate the feature quantity map 22. Generally, the convolutional layer 12 uses a plurality of filters 32, and generates the same number of feature quantity maps 22 as the filters 32.

The pooling layer 13 reduces the feature amount map 22 generated by the convolutional layer 12 to generate a feature amount map 23. The pooling layer 13 extracts two pieces of data from the feature amount map 22, for example, and obtains the maximum value or the average value of the two pieces of data to generate a one-half size feature amount map 23 of the feature amount map 22.

The convolutional layer 14 generates a feature amount map 24 by applying the filter 34 to the feature amount map 23 generated by the pooling layer 13. The convolutional layer 14 moves the filter 34 relative to the feature quantity map 23, and generates the feature quantity map 24 by performing convolution operation of the feature quantity map 23 and the filter 34 at each position.

The pooling layer 15 reduces the feature amount map 24 generated by the convolutional layer 14 to generate a feature amount map 25. The pooling layer 15 extracts two pieces of data from the feature amount map 24, for example, and obtains the maximum value or the average value of the two pieces of data to generate a one-half size feature amount map 25 of the feature amount map 24.

The full join layer 16 generates the feature vector 26 by applying the weight 36 to the feature map 25 generated by the pooling layer 15. The output layer 17 generates an output label 27 by applying the weight 37 to the feature vector 26 generated by the full-key layer 16.

Let the size of the feature quantity map 25 be I, the number of feature quantity maps be K, and the value of the position I of the kth feature quantity map be A_i,k. The weight 36 of the full-joining layer is represented by I × K, the number of the full-joining layers is represented by M, and the value of the position (I, K) in the weight of the mth full-joining layer is represented by Fw_i,k,m. The feature quantity vector 26 has a size M. The size of the weight 37 of the output layer is M, the number of the weights of the output layer is C, and the value of the position M in the weight of the C-th output layer is G_c,m. Outputting the value y of tag c in tag 27_cRepresented by the following formula (1).

[ number 1]

CNN10 is learned based on a comparison between the output label 27 of the output layer when the data for learning is input to the input layer 11 of CNN10 and the teacher label corresponding to the data for learning. The filter 32, the filter 34, the weight 36 of the full-combined layer, and the weight 37 of the output layer are optimized by learning using a plurality of pieces of data for learning and a teacher label.

When the data for evaluation is input to the input layer 11 of the learned CNN10, the data for evaluation is classified by the CNN10, and the classification result appears on the output label 27 of the output layer. The extraction device 1 and method of the convolutional neural network judgment basis according to the present embodiment extract a feature region that is a basis of the judgment by the CNN10 in the input evaluation data.

Fig. 2 is a diagram showing a configuration of the convolutional neural network determining apparatus 1. In this figure, in addition to the convolutional neural network determination basis extraction device (hereinafter, referred to as "CNN determination basis extraction device") 1, a feature amount map 25 of CNN10, a feature amount vector 26, an output label 27 of an output layer, a weight 36 of a full-anchor layer, and a weight 37 of an output layer are also shown.

The CNN determination can be realized by the extraction device 1 using a computer including a CPU, a memory, and the like, and further including a display unit such as a liquid crystal display that displays input data and output data. The CNN determination may be implemented by the extraction apparatus 1 together with the CNN10 by a computer.

The CNN determination is preferably performed by the extraction device 1 including the contribution ratio calculation unit 2 and the extraction unit 3, and further including the display unit 4.

The contribution ratio calculating section 2 obtains the contribution ratio of the weight 36 of the all-bond layer to any output label of the output layer 17. The weighting 36 of the mth full-binding layer with respect to the value y of the label c in the output label 27_cContribution ratio of (1)_c,mAs y_cRelative to Fw_i,k,mThe ratio of the amount of change of (c) is represented by the following formula (2).

[ number 2]

The extraction unit 3 extracts the feature quantity map 25 based on the input to the full bond layer 16, the weighting 36 of the full bond layer, and the contribution ratio β_c,mThe judgment basis of CNN10 is extracted. Data sequence Q indicating the basis of the judgment of CNN10_cIth value of (2)_c,iAs a_i,k、β_c,mAnd Fw_i,k,mThe sum of the k and m is expressed by the following formula (3). Data column Q_cIs I.

[ number 3]

The display unit 4 displays a data sequence Q indicating the basis of the determination of CNN10 in association with the input data input to the input layer 11_c。

The convolutional neural network determination preferably further includes a contribution ratio calculation step and a display step according to an extraction method (hereinafter, referred to as a "CNN determination based extraction method"). In the contribution ratio calculating step, the method calculatesThe contribution beta of the weight 36 of the full combined layer to any output label of the output layer 17_c,m(formula (2)). In the extraction step, the feature quantity map 25, the weighting 36 and the contribution ratio β of the full-joining layer are based on the input to the full-joining layer 16_c,mThe judgment of CNN10 is based on (equation (3)). In the display step, a data sequence Q indicating the basis of the judgment of CNN10 is displayed in association with the input data input to the input layer 11_c。

Fig. 3 is a diagram showing another configuration example of the convolutional neural network. A Convolutional Neural Network (CNN)10A of the configuration example shown in the figure includes: an input layer 11, a convolutional layer 12, a pooling layer 13, a full bond layer 16, and an output layer 17. The CNN10 shown in fig. 1 includes 2 sets of convolutional layers and pooling layers, whereas the CNN10A shown in fig. 3 includes 1 set of convolutional layers and pooling layers. The extraction device 1 according to the CNN determination shown in fig. 2 can be applied to the CNN having the configuration shown in fig. 3.

Next, the following description will be made of embodiments 1 to 4. In examples 1 and 2, CNN having the configuration shown in fig. 3 was used. In examples 3 and 4, CNN having the configuration shown in fig. 1 was used.

Embodiment 1 is as follows. In example 1, a spectrum having a simple shape produced by simulation was used as data for learning and data for evaluation. In each of the learning spectrum and the evaluation spectrum, the channel number is 1024, and the peak has a maximum peak at any position of 100ch, 500ch, and 1000 ch. In addition, in any of the learning spectrum and the evaluation spectrum, a noise peak is present at each of three positions different from any of 100ch, 500ch, and 1000ch, and white noise is further added.

The maximum peak and the noise peak are both in a lorentz function shape, normalized with the maximum peak value being 1, and the noise peak value being a random value within a range of 0.1 or more and less than 1. The teacher label corresponding to the learning spectrum is given as a one-hot vector (an array in which the correct learning label is 1 and the other labels are 0) as the maximum peak position (any of 100ch, 500ch, and 1000 ch) of the learning spectrum.

In embodiment 1, CNN having the configuration shown in fig. 3 is used. The size of the filter 32 is set to 8, and the number thereof is set to 64. The weight 36 of the full-combined layer was set to 512 × 64, and the number thereof was set to 128. The size of the weight 37 of the output layer is 128, and the number thereof is 3. CNN is learned using the learning spectrum and teacher label.

The learned CNN is input with an evaluation spectrum, and the CNN is classified into the evaluation spectrum. According to the present embodiment, the feature region to be the basis of the classification is obtained from the full-bodied layer (example), and the feature region to be the basis of the classification is obtained from the pooling layer according to the technique described in non-patent document 3 (comparative example).

Fig. 4 to 6 are diagrams showing the feature regions obtained in example 1, which are the basis of classification. Each graph sequentially displays an inputted evaluation spectrum, a data sequence showing feature regions serving as basis of classification obtained in the comparative example, and a data sequence Q showing feature regions serving as basis of classification obtained in the example_c(formula (3)).

Fig. 4 shows an example in which the maximum peak position of the evaluation spectrum is 100 ch. Fig. 5 shows an example in which the maximum peak position of the evaluation spectrum is 500 ch. Fig. 6 shows an example in which the maximum peak position of the evaluation spectrum is 1000 ch.

In any of fig. 4 to 6, in the comparative example, the feature region that is the basis of the classification exists not only at the maximum peak position but also at the noise peak position. In contrast, in the embodiment, the feature region that becomes the basis of the classification exists only at the maximum peak position. In the embodiment, the feature region that becomes the basis of the classification is displayed more accurately than in the comparative example.

The embodiment 2 is described below. In example 2, the same data as the learning spectrum and the evaluation spectrum used in example 1 are used as the learning data and the evaluation data. However, the evaluation spectrum does not have a noise peak in the shape of a lorentzian function.

In embodiment 2, using CNN as used in embodiment 1Constitute the same data. However, the size and number of the filters 32 are set to various values, CNN is learned and classified, and a data sequence Q showing a feature region that is a basis of the classification is obtained_c(formula (3)).

Fig. 7 and 8 are diagrams showing the feature regions obtained in example 2, which are the basis of classification. Each figure is a data sequence Q showing an inputted evaluation spectrum and showing a feature region which is a basis of classification obtained by the example_c(formula (3)).

Fig. 7 (a) and (b) show examples in which the number of filters is fixed to 64, and the filter sizes are set to values of 8, 16, 128, and 1024. Fig. 7 (b) is an enlarged view of a part of fig. 7 (a). As is clear from this figure, as the filter size approaches the spectrum width, the CNN pays attention to the vicinity of the maximum peak position of the input evaluation spectrum as a classification basis.

Fig. 8 (a) and (b) show examples in which the filter size is fixed to 16, and the number of filters is set to 8, 64, and 256. Fig. 8 (b) is an enlarged view of a part of fig. 8 (a). As is clear from this figure, as the number of filters increases, the CNN is captured as a classification basis at a position close to the maximum peak position of the input evaluation spectrum.

This embodiment shows that optimization of the size and number of filters can be performed.

Embodiment 3 is described below. In example 3, the raman spectra of 9 commercially available drugs a to I were used as the spectrum for learning and the spectrum for evaluation. Interpolation processing is performed on the Raman spectrum measured for each drug at a wave number of 350cm^-1～1800cm^-1In the range of (1) to (1 cm)^-1Spectrum of the scale.

In either of the learning spectrum and the evaluation spectrum, the channel number is 1451, and the maximum peak is normalized to 1. In addition, 4 spectra having different SN ratios were used as the spectrum for learning for each of the 9 drugs. Fig. 9 is a diagram showing an example of the frequency spectrum of each of the 9 types of medicines used in example 3.

In embodiment 3, CNN of the configuration shown in fig. 1 is used. The size of the filter 32 is set to 8, and the number thereof is set to 64. The size of the filter 34 is set to 8, and the number thereof is set to 64. The size of the weight 36 of the full-combined layer was 363 × 64, and the number thereof was 128. The size of the weight 37 of the output layer is 128, and the number thereof is 3. CNN is learned using the learning spectrum and teacher label.

The spectrum other than the learning spectrum is input to the CNN as an evaluation spectrum, and the CNN classifies the evaluation spectrum. The feature region that is the basis of the classification is obtained from the full-binding layer.

Fig. 10 to 18 are diagrams showing the feature regions obtained in example 3, which are the basis of classification. Each figure is a data sequence Q showing an inputted evaluation spectrum and showing a feature region which is a basis of classification obtained by the example_c(formula (3)).

Fig. 10 shows an example of the case of the drug a. Fig. 11 shows an example of the case of the drug B. Fig. 12 shows an example of the case of the medicine C. Fig. 13 shows an example of the case of the medicine D. Fig. 14 shows an example of the case of the drug E. Fig. 15 shows an example of the case of the medicine F. Fig. 16 shows an example of the case of the medicine G. Fig. 17 shows an example of the case of the drug H. Fig. 18 shows an example of the case of the drug I.

For any drug, a peak position where the evaluation spectrum is strong is displayed, and there is a characteristic region that becomes a basis of classification. On the other hand, Q is a weak peak position of the evaluation spectrum or a position where the background intensity of the evaluation spectrum is visible_c,iThe value is small. When observing the drug D (FIG. 13), the wave number of the drug D was 360cm, which was capable of separating the drug D from the other 8 drugs^-1Near, Q_c,iBecomes a larger value. Therefore, according to the present embodiment, it is possible to confirm the feature region that can be extracted as a basis for classification by CNN.

Embodiment 4 is described below. In example 4, spectra prepared from raman spectra of the following 20 amino acids were used as the learning spectrum and the evaluation spectrum. FIG. 19 is a diagram showing examples of the frequency spectra of 20 kinds of amino acids used in example 4.

Alanine (Ala), arginine (Arg), asparagine (Asn), aspartic acid (Asp), cysteine (Cys), glutamine (Gln), glutamic acid (Glu), glycine (Gly), histidine (His), isoleucine (Ile), leucine (Leu), lysine (Lys), methionine (Met), phenylalanine (Phe), proline (Pro), serine (Ser), threonine (Thr), tryptophan (Trp), tyrosine (Tyr), valine (Val)

For each amino acid, the measured Raman spectrum was interpolated at a wave number of 350cm^-1～1800cm^-1In the range of (1) to (1 cm)^-1Spectrum of the scale. These spectra were synthesized using 1 arbitrary amino acid among the 20 amino acids as a host and any other amino acid as a guest. For each subject, 5 spectra were created, and normalized with the maximum peak at 1. A total of 1900 (20 × 19 × 5) spectra were created.

The mixing ratio of the spectrum of the amino acid of the host to the spectrum of the amino acid of the guest is set to be random within a range of 1:0.1 to 1:0.5 with respect to the spectrum for learning. The teacher label is given as a one-hot vector of the main body's amino acids. The mixing ratio of the spectrum of the host amino acid to the spectrum of the guest amino acid was set to 1:0.45 for the spectrum for evaluation.

In the 4 th embodiment, the same CNN as that used in the 3 rd embodiment is used. CNN is learned using the learning spectrum and teacher label. A spectrum for evaluation different from a spectrum for learning is inputted to the CNN, and the CNN is classified into the spectrum for evaluation. The feature region that is the basis of the classification is obtained from the full-binding layer.

FIG. 20 is a graph showing a pure spectrum of alanine (Ala) used in example 4. Fig. 21 and 22 are diagrams showing the feature regions obtained in example 4, which are the basis of classification.

FIG. 21 is a data sequence Q showing characteristic regions that are the basis of classification obtained when a spectrum for evaluation in which a subject is histidine (His) and a guest is alanine (Ala) is inputted to CNN_c(formula 3) and histidine (His). The pure spectrum ofThe strong peak position of the spectrum of histidine (His) shown in the subject has a characteristic region that is a basis for classification.

On the other hand, the stronger peak position (wave number 850 cm) of the pure spectrum of alanine (Ala), the host^-1Nearby), Q_c,iIs negative. That is, if the wave number observed in the spectrum for evaluation is 850cm^-1The nearby peaks are regions unnecessary for histidine (His) classification, and it can be understood that CNN is learned.

FIG. 22 is a data sequence Q showing characteristic regions that are the basis of classification obtained when a spectrum for evaluation in which a host is leucine (Leu) and a guest is alanine (Ala) is input to CNN_cThe SN ratio of the pure spectrum of leucine (Leu) is relatively poor (formula 3), but in this case, similarly, a characteristic region exists which is displayed at a strong peak position of the spectrum of leucine (Leu), which is the main body, and which is the basis of classification.

It is conceivable that the position of the stronger peak of the pure spectrum of the leucine (Leu) as the host is the wave number of 850cm at the position of the stronger peak of the pure spectrum of the alanine (Ala) as the guest is close to^-1Nearby, but this peak position seen in the evaluation spectra did not contribute to the classification of leucine (Leu). It is conceivable to evaluate the other wavenumber 475cm seen in the spectrum^-1Near and 545cm^-1The nearby peaks contribute to the classification of leucine (Leu).

Good results were also obtained for other combinations of host and guest. Therefore, according to the present embodiment, it is possible to confirm the feature region that can be extracted as a basis for classification by CNN.

The CNN determination-based extraction device and the CNN determination-based extraction method according to the present embodiment are not limited to the case where the input data is a spectrum, and other input data (e.g., image data) may be applied. According to the present embodiment, even when the hidden layer of the CNN is small or the size of the filter used for the convolutional layer is small, the feature region that is the basis of the determination by the CNN in the input data can be extracted. The extraction device and the extraction method according to the CNN determination basis of the present embodiment can facilitate the design and verification of the CNN model, ensure reliability, and provide a CNN model that is easy for a user to understand.

Further, it is conceivable that the CNN determination based extraction device and the CNN determination based extraction method according to the present embodiment can extract common parts if CNN is learned by attaching the same teacher label to samples including the same type in classification of mixed spectrum, and can be used for specification of unnecessary content in not only visualization of common components but also true/false determination because negative values are obtained in parts that are considered to reduce the classification probability (example 4).

The embodiment and the 1 st to 4 th examples described so far are examples of determining the contribution ratio of the weight 36 of the full-range layer to any output label of the output layer 17, and extracting the basis of the determination of CNN10 using the contribution ratio. As in the embodiment and the 5 th and 6 th examples described below, the contribution ratio of the feature spectrum 26 to any output label of the output layer 17 may be obtained, and the determination basis of the CNN10 may be extracted using the contribution ratio.

Fig. 23 is a diagram showing a configuration of the convolutional neural network determining apparatus 1A. This figure also shows a feature quantity map 25 of CNN10, a feature quantity vector 26, an output label 27 of an output layer, a weight 36 of a full-boding layer, and a weight 37 of an output layer, in addition to the convolutional neural network determination basis extracting means (CNN determination basis extracting means) 1A.

The CNN determination may be performed by the extraction device 1A using a computer including a CPU, a memory, and the like, and may include a display unit such as a liquid crystal display that displays input data and output data. The CNN determination according to the extraction apparatus 1A may also be realized by a computer together with the CNN 10.

The CNN determination is performed by the extraction device 1A, the contribution ratio calculation unit 2A and the extraction unit 3, and preferably further includes a display unit 4. As compared with the configuration shown in fig. 2, the CNN shown in fig. 23 differs depending on the extraction device 1A in that a contribution ratio calculator 2A is provided instead of the contribution ratio calculator 2.

The contribution ratio calculating unit 2A calculatesThe contribution rate of the feature quantity vector 26 with respect to any output label of the output layer 17 is found. The feature vector 26 is based on the feature map (a) input to the full-join layer_i,k) And weight of full binding layer (Fw)_i,k,m) And then generated. M-th element F of feature vector 26_mWith respect to the value y of tag c in output tag 27_cContribution ratio of (1)_c,mIs as y_cRelative to the amount of change of F_mThe ratio of the amount of change of (a) is represented by the following formula (4).

[ number 4]

The extraction unit 3 extracts the feature quantity map 25 based on the input to the full bond layer 16, the weighting 36 of the full bond layer, and the contribution ratio β_c,mThe judgment basis of CNN10 is extracted. Data sequence Q from which the CNN10 is judged is displayed_cIth value of (2)_c,iAs a_i,k、β_c,mAnd Fw_i,k,mThe sum of the k and m is expressed by the above formula (3).

The convolutional neural network determination preferably further includes a display step, based on the extraction method (the extraction method according to which the CNN is determined) and the contribution ratio calculation step. In the contribution ratio calculation step, the contribution ratio β of the feature vector 26 to any output label of the output layer 17 is obtained_c,m(formula (4)). In the extraction device, the feature quantity map 25, the weighting 36 and the contribution ratio beta of the full-bonding layer 16 are based on the input feature quantity map_c,mThe judgment of CNN10 is based on (equation (3)). In the display step, a data sequence Q indicating the basis of the judgment of CNN10 is displayed in association with the input data input to the input layer 11_c。

Next, embodiment 5 and embodiment 6 will be described. In example 5, only the contribution ratio calculation is different from that in example 3, and the other points are set to the same conditions. In example 6, only the contribution ratio calculation is different from that in example 4, and the other points are set to the same conditions. However, when alanine (Ala) is used as the host, arginine (Arg) is used as the guest, and when an amino acid other than alanine (Ala) is used as the host, alanine is used as the guest.

In examples 3 and 4, the contribution ratio of the weight 36 of the full-bond layer to any output label of the output layer 17 is obtained (formula (2) above), whereas in examples 5 and 6, the contribution ratio of the feature quantity vector 26 to any output label of the output layer 17 is obtained (formula (4) above).

Fig. 24 is a diagram showing the feature regions obtained in example 3, which are the basis of classification. Fig. 25 is a diagram showing the feature regions obtained in example 5, which are the basis of classification. The 3 rd embodiment (fig. 24) differs from the 5 th embodiment (fig. 25) only in the aspect of contribution ratio calculation, but the same feature region is extracted as a basis for classification based on CNN.

Fig. 26 is a diagram showing the feature regions obtained in example 4, which are the basis of classification. Fig. 27 is a diagram showing the feature regions obtained in example 6 as the basis of classification. The 4 th embodiment (fig. 26) differs from the 6 th embodiment (fig. 27) only in the aspect of contribution ratio calculation, but the same feature region is extracted as a basis for classification based on CNN.

In each of embodiment 1 and embodiment 2, when the contribution ratio of the feature vector 26 (the above expression (4)) is used instead of the contribution ratio of the weighting 36 of the full-binder layer (the above expression (2)), the same feature region is extracted as a basis for classification by CNN.

As described above, similarly to the case where the contribution ratio of the weight 36 of the full-combined layer to any output label of the output layer 17 (the above expression (2)) is used, in the case where the contribution ratio of the feature vector 26 to any output label of the output layer 17 (the above expression (4)) is used, in the case where the hidden layer of the CNN is small or the size of the filter used in the convolutional layer is small, the feature region that is the basis of the determination by the CNN in the input data may be extracted.

The convolutional neural network determination method and apparatus according to the present invention are not limited to the above embodiments and configuration examples, and various changes may be made.

The convolutional neural network judgment basis extracting method according to the above embodiment is a method of extracting a judgment basis of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-combining layer, and an output layer, and includes the steps of: a contribution ratio calculation step of calculating a contribution ratio of the weight of the full-binding layer to the contribution ratio of any output label of the output layer; and extracting the basis of the feature quantity graph input into the full-bonding layer, the weighting and the contribution rate of the full-bonding layer.

In the convolutional neural network determination based on the above-described configuration, in the contribution ratio calculation step, the contribution ratio of the feature vector generated by the full-binding layer may be calculated instead of the weighted contribution ratio of the full-binding layer. The feature vector is generated based on the feature map input to the full-join layer and the weighting of the full-join layer.

In the convolutional neural network decision basis extraction method configured as described above, the method may further include a display step of displaying the basis in association with the input data input to the input layer.

The convolutional neural network decision basis extraction device according to the above embodiment is a device for extracting a decision basis of a convolutional neural network having an input layer, a convolutional layer, a pooling layer, a full-binding layer, and an output layer, and includes: a contribution ratio calculation unit that obtains a contribution ratio of the weight of the all-binding layer to any one of the output labels of the output layers; and a basis extraction unit that extracts a basis on the feature quantity map input to the full-bonding layer, the weighting and the contribution rate of the full-bonding layer.

In the convolutional neural network determining device configured as described above, the contribution ratio calculating unit may be configured to calculate the contribution ratio of the feature vector generated by the full-joining layer, instead of the weighted contribution ratio of the full-joining layer.

The convolutional neural network determination device configured as described above may further include a display unit configured to display the basis in association with the input data input to the input layer.

Industrial applicability

The present invention can use a method and an apparatus for extracting a feature region that is a basis for a CNN determination from input data even when there are few hidden layers as CNNs or when the size of a filter used for a convolutional layer is small.

Description of the symbols

1. 1a … … convolutional neural network judgment based on extraction means (CNN judgment based on extraction means); 2. a 2a … … contribution ratio calculation unit; 3 … … according to the extraction section; a 4 … … display section; 10. 10a … … Convolutional Neural Network (CNN); 11 … … input layer; 12 … … convolutional layers; 13 … … pooling layer; 14 … … convolutional layers; 15 … … pooling layer; 16 … … full bond layer; 17 … … output layer; 21 … … inputting data columns; 22-25 … … characteristic quantity diagram; 26 … … feature vector; 27 … … output label of the output layer; 32. a 34 … … filter; 36 … … weighting of full join layers; 37 … … outputs the weighting of the layers.