阅读说明：本技术 基于双分支深度融合卷积神经网络的红枣品质分类方法 (Red date quality classification method based on double-branch deep fusion convolutional neural network ) 是由 耿磊 徐文龙 肖志涛 张芳 吴骏 刘彦北 于 2018-07-02 设计创作，主要内容包括：本发明公开了一种基于双分支深度融合卷积神经网络的红枣品质分类方法,该方法包括：该方法首先对采集到的不同品质的红枣图像进行预处理,并将其归一化为相同尺寸,预处理后的红枣图像(丰满枣、干条枣、裂口枣和瑕疵枣)被随机分为训练集和测试集。然后,将卷积神经网络设计为双分支结构,其中第1条分支网络结合迁移学习策略,利用在大型数据集Imagenet上训练生成的模型对其进行预训练。第2条分支网络,增加了特征图流动的分支数目和融合次数。最后利用设计的双分支深度融合卷积神经网络对预处理过的红枣图像数据进行训练,提取红枣图像特征,生成训练模型,并完成不同红枣品质的多分类任务。该方法极大地提高了红枣品质分类的准确率和效率。(The invention discloses a red date quality classification method based on a double-branch deep fusion convolutional neural network, which comprises the following steps: the method comprises the steps of firstly preprocessing collected red date images with different qualities, normalizing the red date images into the same size, and randomly dividing the preprocessed red date images (full dates, dry dates, cracked dates and defective dates) into a training set and a testing set. Then, the convolutional neural network is designed into a double-branch structure, wherein the 1 st branch network is combined with a migration learning strategy and is pre-trained by using a model generated by training on a large data set Imagenet. And 2, the branch network increases the branch number and the fusion times of the feature diagram flow. And finally, training the preprocessed red date image data by using the designed double-branch depth fusion convolutional neural network, extracting the red date image characteristics, generating a training model, and completing a multi-classification task of different red date qualities. The method greatly improves the accuracy and efficiency of red date quality classification.)

1. The red date quality multi-classification method based on the double-branch deep fusion convolutional neural network comprises the following steps:

a, collecting red date sample images with different qualities, and dividing the red date sample images into four types of full dates, dry strip dates, split dates and defective dates according to the national standard of appearance quality of the dried red dates and the requirement of enterprises on red date quality grading;

and B, preprocessing the data of the red dates, and normalizing the data to be 227 x 227 in the same size as the data of subsequent network training.

And C, designing the structure of the convolutional neural network into a double-branch network, combining a transfer learning strategy and a fusion module, carrying out network training and feature extraction on the red date image data, and generating and storing a trained network model file.

And D, performing multi-classification recognition on the red date image samples of the test set by using the trained network model.

2. The red date quality multi-classification method based on the double-branch deep fusion convolutional neural network as claimed in claim 1, wherein in the step C, the double-branch deep fusion convolutional neural network is designed as follows:

the convolutional neural network is designed into a double-branch structure, the first part of the double-branch deep fusion convolutional neural network is a data input layer, and a preprocessed red date image is used as the input of the layer. The second part is a feature extraction layer, which consists of two branches, the number of convolution layers and pooling layers of the two branches being the same. The third part is a classification layer, which comprises a splicing fusion layer, a Dropout layer and a classification layer. Wherein, the Concat layer is spliced and fused, and the convolution layer with convolution kernel of 1 is classified. The specific structure of the double-branch deep fusion convolutional neural network is as follows:

c1, the first part of the double-branch deep fusion convolutional neural network is used as data input, and the data input layer of the first part is used for inputting the data of the four red dates with different qualities as the data of the following double-branch network.

C2, the second part of the double-branch depth fusion convolutional neural network is an image feature extraction layer, the part mainly comprises two branch networks, wherein the first branch network mainly comprises 1 convolutional layer, 8 Fire modules, 3 pooling layers and a plurality of Concat layers. The second branch network is mainly composed of 1 convolutional layer, 8 Fusion modules, 3 pooling layers and several Concat layers.

C3, the third part of the two-branch depth fusion convolutional neural network is an image classification layer, and the part consists of a Dropout layer, a convolutional layer with a convolutional kernel of 1, a global average pooling layer and a loss layer.

3. The method for multi-classification of red dates based on the double-branch deep fusion convolutional neural network as claimed in claim 2, wherein in step C2, the first branch is a feature extraction part of the lightweight network Squeezenet, and the branch is composed of 1 convolutional layer with convolution kernel of 3 × 3, 8 fire modules, 2 max pooling layers, and several Relu activation functions and Concat layers. Wherein the fire module consists of 3 convolutional layers (squeeze1 × 1, expanded 1 × 1, and expanded 3 × 3), 3 Relu active layers, and1 Consat layer. The first branch network is combined with a migration learning strategy, and the model generated by training on the large data set Imagenet is used for pre-training the red date data.

4. The method of claim 2, wherein in step C2, the second branch network extracts feature information of the red date pattern using a Fusion module, and the branch comprises 1 convolution layer with a convolution kernel of 3 × 3, 8 Fusion modules, 2 maximum pooling layers, BN lot specification operation layers, pcelu activation functions, and concat layers. Wherein the Fusion module mainly comprises 3 convolution layers (squeeze1 × 1, expanded 1 × 1 and expanded 3 × 3), 3 PRelu activations, 3 BN layers and1 Concat layer, and the output feature maps between the Fusion modules are fused by fusing the output of the convolution layer squeeze1 × 1 in the Fusion module with the output feature maps of the convolution layers expanded 1 × 1 and expanded 3 × 3, so that the number of branches and the Fusion times of the feature map flow are increased.

5. The red date quality multi-classification method based on the double-branch depth fusion convolutional neural network as claimed in claim 1, wherein in step B, firstly, the image is subjected to binarization processing to highlight the contour of the target of interest. 252 is selected as the threshold value for the binarization processing. And then searching the minimum circumscribed rectangle outline of the binary image, searching the boundary point of the square outline as the boundary point of the ROI area, thereby extracting the ROI area of the image, and then normalizing the red date image data into the same size 227 x 227.

Technical Field

The invention relates to a red date quality classification method based on a double-branch deep fusion convolutional neural network, and belongs to the field of image processing, machine vision and agricultural product quality detection.

Background

China is the country of origin of jujube, the biggest jujube producing country and the only export country in the world, has more than 95% of jujube tree resources all over the world, occupies a leading position in the world red jujube industry, and red jujube is the agricultural product with the greatest competitiveness after being added into world trade organizations in China. In recent years, the output of Chinese red dates keeps overall increase, and the market scale is over one billion yuan.

The quality sorting of red dates is a key technical link of red date storage, processing, circulation and the like, and directly influences the economic benefit of fruit growers. However, the current red dates are mainly sorted manually, and the defects of high labor intensity, low efficiency, high cost, difficulty in guaranteeing sorting precision and sanitary quality and the like exist. Therefore, the red date quality sorting system based on machine vision has wide market application prospect.

Disclosure of Invention

In order to achieve the aim of improving the efficiency and the precision of red date quality classification, the invention provides a red date quality classification method based on a double-branch deep fusion convolutional neural network, which adopts the following technical scheme:

a, collecting red date sample images with different qualities, and dividing the red date sample images into four types of full dates, dry strip dates, split dates and defective dates according to the national standard of appearance quality of the dried red dates and the requirement of enterprises on red date quality grading;

and B, preprocessing the data of the red dates, and normalizing the data to be 227 x 227 in the same size as the data of subsequent network training.

And C, designing the structure of the convolutional neural network into a double-branch network, combining a transfer learning strategy and a fusion module, carrying out network training and feature extraction on the red date image data, and generating and storing a trained network model file.

And D, performing multi-classification recognition on the red date image samples of the test set by using the trained network model.

Drawings

FIG. 1 shows some data of different quality red date images.

Fig. 2 is red date image preprocessing.

FIG. 3 is a diagram showing the structure of Fire module and Fusion module

Fig. 4 is a diagram of a dual-branch depth fusion convolutional network.

Detailed Description

The red date quality classification method based on the double-branch deep fusion convolutional neural network comprises the following steps:

a, collecting red date sample images with different qualities, and dividing the red date sample images into four types of full dates, dry strip dates, split dates and defective dates according to the national standard of appearance quality of the dried red dates and the requirement of enterprises on red date quality grading;

and B, preprocessing the data of the red dates, and normalizing the data to be 227 x 227 in the same size as the data of subsequent network training.

And C, designing the structure of the convolutional neural network into a double-branch network, combining a transfer learning strategy and a fusion module, carrying out network training and feature extraction on the red date image data, and generating and storing a trained network model file.

And D, performing multi-classification recognition on the red date image samples of the test set by using the trained network model.

Preprocessing images with different red date qualities in an early stage:

the red date data set sample is acquired and stored on a white background through a CCD industrial camera with 30 ten thousand pixels and an annular light source with adjustable illumination intensity, and totally comprises 20000 pieces of red date image data. The red date data set is divided into a training set and a testing set according to the proportion of 4: 1. Red date sample data contains four types: filled dates, dry-striped dates, cracked dates, and defective dates.

When the convolutional neural network is trained, although training data is fixed, a training mechanism of random batch processing (mini-batch) is adopted, so that a training data set is randomly disturbed before model training is carried out, the training data of different rounds of training of the model with the same batch number are different, and the prediction result of the model on a test set is improved.

The acquired original red date image data has a large white background and is meaningless to the training of a convolutional neural network, the original red date image data needs to be preprocessed, and a basic image processing method is adopted to extract the region of interest of the original red date image data. The image is first binarized to highlight the contours of the object of interest. 252 is selected as the threshold value for the binarization processing. Then, the minimum square external contour of the binary image is searched, the boundary point of the square contour is searched to be used as the boundary point of the ROI area, so that the ROI area of the image is extracted, and then the red date image data is normalized to be 227 x 227 in the same size.

The Fire module and Fusion module structures are as follows:

the lightweight network Squeezenet is proposed to simplify the model complexity, reduce the parameter number of the convolutional neural network and achieve the optimal CNN identification precision. The Fire module is the core of the lightweight network squeezet, as shown in fig. 1, it is composed of three convolution layers, 3 active layers and1 Contact layer, the convolution kernel sizes of convolution squeeze1 × 1 layer, convolution extended 1 × 1 layer and convolution extended 3 × 3 layer are 1 × 1, 1 × 1 and3 × 3 respectively, and the output feature maps of convolution extended 1 × 1 layer and convolution extended 3 × 3 layer are spliced through the concat layer and used as the input of the next Fire module. The design of the Fire module is to reduce the size of a convolution kernel, reduce the number of characteristic graphs input to convolution layers with the convolution kernel size of 3 x 3 and delay the down-sampling strategy, thereby reducing the number of parameters of a convolution neural network, obtaining a larger characteristic graph and improving the training precision of a network model.

In order to further improve the training precision of the network on red date classification and identification, the Squeezenet network is further improved. As shown in fig. 2, the modified Fusion module is configured to fuse the convolution layer squeeze1 × 1 with the output feature maps of the convolution layers expanded 1 × 1 and expanded 3 × 3 by changing the flow direction of the feature map inside the Fire module to form a feature fused Fusion module, and to fuse the feature information extracted by the adjacent Fusion modules to increase the number of branches of feature flow in the network, thereby extracting more abundant feature information. The improved squaezenet network is called as a deep fusion convolutional neural network because the network structure is deeper and the feature fusion times among different convolution extraction layers are increased. The deep Fusion convolutional neural network structure consists of 2 common convolutional layers, 8 Fusion modules, 4 pooling layers and1 Dropout layer, a BN batch specification operation layer and a Prelu activation layer are added after each convolution, and the output image characteristics of the Fusion modules of 2 and3, 4 and 5, 6 and 7, 8 and 9 are fused. The feature graphs of more branches are fused in the middle layer of the network, so that the flow of feature information can be optimized, and the training precision of the model is improved.

The design principle of the deep fusion convolutional neural network is to fuse the extracted feature maps on the basis of keeping the design advantages of the Squeezenet convolutional neural network mechanism. The lightweight network Squeezenet convolution layer reduces 9 times of parameters by using 1 x 1 convolution kernel to replace 3 x 3 convolution kernel, reduces the number of characteristic graphs input into the convolution kernel of the 3 x 3 convolution layer, reduces model parameters, obtains larger characteristic graphs by delaying the pooling layer, and is greatly helpful for improving the accuracy of model classification. The deep fusion convolutional neural network increases the number of branches of the flowing of the feature map, fully utilizes the extracted image feature information, ensures the richness of the image features and improves the classification precision of the network model.

The structure of the double-branch deep fusion convolutional neural network is as follows:

on the basis of a deep fusion convolutional neural network (deep fusion net), a double-branch deep fusion convolutional neural network structure (DDFnet) is provided for the quality classification and identification problem of red dates. As shown in fig. 3, the 1 st branch is combined with the strategy of migration learning, and the 1 st convolutional layer and 8 fire modules are pre-trained by using a model generated by a lightweight network Squeezenet on an ImageNet data set, where the branch is composed of 1 convolutional layer with convolution kernel of 3 × 3, 8 fire modules, 2 max pooling layers, and several Relu activation functions and concat layers. And the 2 nd branch adopts a Fusion module to extract the characteristic information of the red date graph, and comprises 1 convolution layer with convolution kernel of 3 x 3, 8 Fusion modules, 2 maximum pooling layers, a plurality of BN batch specification operation layers, a plurality of PRelu activation functions and a concat layer. And splicing the feature graphs output by the last Fire module and the Fusion module of the two branches through a concat layer, and classifying by using a convolution layer conv10_1 and a global average pooling layer to replace a full connection layer, so that the number of parameters is reduced, and the overfitting phenomenon is reduced.

The design of the double-branch deep fusion convolutional neural network is to pre-train the double-branch deep fusion convolutional neural network by utilizing a model obtained by training the Squeezenet on a large data set ImageNet in a convolutional neural network structure applied to migration learning, accelerate the convergence speed of the whole network training, increase the width of the network structure and improve the classification accuracy and generalization capability of a network training generation model.

Network training and tuning:

in the two-branch deep fusion convolutional neural network structure, the 1 st branch adopts a transfer learning strategy, and a pre-training model generated on a large data set ImageNet is utilized to directly apply the corresponding structure and weight to the red date quality classification and identification problem, so that the network weight updating time is shortened, the network training convergence speed is accelerated, the red date quality classification and identification precision is improved, the trained model shows better generalization capability, and the significance to network optimization is great.

The traditional Relu activation layer is easy to have the phenomenon that neurons can not be activated under the condition of high learning rate, and in order to accelerate the convergence rate of network training and prevent the gradient disappearance phenomenon in the training process, a PRelu activation function layer is introduced after each convolution layer of the 2 nd branch network. Only few parameters are added to the PRelu activation function, the calculation amount of network training and the appearance of an overfitting phenomenon are hardly influenced, however, compared with Relu, the PRelu corrects data distribution, a negative value is reserved, and the updating parameter a is propagated reversely_iThe PRelu formula is as follows:

since the red date image data amount is small, in order to prevent the over-fitting phenomenon, a Dropout layer is added before the convolution layer conv10_1 of the two-branch depth fusion convolution neural network. Dropout refers to the temporary discarding of neurons from the convolutional neural network with a certain probability during the training of the network. In the network training process, the size of drop _ ratio of a Dropout layer is usually set, and in the two-branch deep-fusion convolutional neural network, the drop _ ratio is 0.5.

The network parameter initialization can affect the convergence speed of the network training in the model training of the convolutional neural network, and a proper initialization scheme is selected, so that the gradient disappearance phenomenon can be prevented. In the two-branch deep fusion convolutional neural network, the 1 st branch adopts a pre-training mode to initialize network parameters, the 2 nd branch adopts a random initialization scheme, and except that the convolutional layer conv10_1 adopts a Gaussian initialization method, the other convolutional layers adopt a xavier parameter initialization method.

And introducing a BN layer into the second branch of the convolutional neural network, calculating the mean value and variance of batch processed data, normalizing the input batch processed data to enable the mean value of each dimension of an output signal to be 0 and the variance to be 1, and finally enabling the output of the BN layer to restore the initial input by using scale change and offset operation. The introduction of batch standardization operation can accelerate the convergence speed of network training, so that the deep network model can be trained more easily and stably, and positive influence can be generated on the improvement of the generalization capability of the network.