阅读说明：本技术 一种基于引导信息的立体匹配方法 (Stereo matching method based on guide information ) 是由 魏东 何雪 刘涵 于璟玮 于 2021-08-17 设计创作，主要内容包括：一种基于引导信息的立体匹配方法,包括：图像输入：输入左右相机两幅图像,将左图像和右图像分别作为参考图像和目标图像；特征提取：使用卷积网络VGG对输入的两幅图像分别进行特征提取,获取不同尺度图像特征；提取边缘信息：使用HED网络对输入的两幅图像分别提取边缘特征图,获取边缘细节信息；视差注意力计算初始视差：提取到的不同尺度的特征与边缘信息融合后输入到视差注意力模块,生成视差注意力图,再回归到初始视差图；提取语义信息：特征提取的最后池化层得到的特征送入到DenseASPP网络提取语义特征图,获取语义信息；视差优化：语义特征图与初始视差图融合,利用语义信息进行视差优化得到最终视差图。(A stereo matching method based on guide information comprises the following steps: image input: inputting two images of a left camera and a right camera, and respectively taking the left image and the right image as a reference image and a target image; feature extraction: respectively extracting features of the two input images by using a convolution network VGG to obtain the features of the images with different scales; extracting edge information: respectively extracting edge characteristic graphs from the two input images by using an HED network to obtain edge detail information; parallax attention initial parallax is calculated: the extracted features with different scales and the edge information are fused and then input to a parallax attention module to generate a parallax attention diagram, and then the parallax attention diagram is returned to the initial parallax diagram; extracting semantic information: the features obtained by the final pooling layer of feature extraction are sent to a DenseASPP network to extract a semantic feature map, and semantic information is obtained; parallax optimization: and fusing the semantic feature map and the initial disparity map, and performing disparity optimization by using semantic information to obtain a final disparity map.)

1. A stereo matching method based on guide information is characterized in that: the method comprises the following steps:

the method comprises the following steps: image input: inputting left and right images, wherein the left image is used as a reference image, and the right image is used as a target image;

step two: feature extraction: respectively extracting features of the input left image and the input right image by using a convolution network VGG to obtain different scale image features smaller than the image in the first step;

step three: extracting edge information: respectively extracting edge characteristic graphs from the input left image and the input right image by using an integral nested edge detection HED network to obtain edge detail information;

step four: parallax attention initial parallax is calculated: fusing the image features of different scales obtained in the step two with edge information, inputting the fused image features and the edge information into a cascade parallax attention module to generate a parallax attention diagram, and regressing the parallax attention to obtain an initial parallax diagram;

step five: extracting semantic information: performing pooling layer processing at the end of feature extraction, and sending the obtained pooling layer features into a densely connected void space convolution pooling pyramid DenseASPP network to extract a semantic feature map and obtain semantic information;

step six: parallax optimization: and fusing the semantic feature graph and the initial disparity map, sending the fused image into an hourglass-shaped network, performing disparity optimization on the initial disparity map by utilizing semantic information to obtain a final disparity map, and finishing stereo matching.

2. The guide information-based stereo matching method according to claim 1, wherein: and cutting the image in the first step, and sending the cut image into a VGG network for feature extraction.

3. The guide information-based stereo matching method according to claim 1, wherein: inputting left and right camera images, sending the images into an HED network to extract an edge feature map, performing up-sampling operation on the image features with different scales obtained in the step two, overlapping the image features with the edge feature map on channel dimension, and fusing edge information.

4. The guide information-based stereo matching method according to claim 1, wherein: and step four, the features after the edge information is fused are sent to a cascade parallax attention module to calculate initial parallax, the parallax attention calculates the feature similarity of each pixel in the reference image and all possible differences in the target image, so that a parallax attention diagram with the channel number of 1 is generated, and regression calculation is performed on the parallax attention diagram to obtain the initial parallax.

5. The guide information-based stereo matching method according to claim 1, wherein: and fifthly, performing up-sampling operation on the semantic feature graph obtained through the semantic information extraction network, then overlapping the semantic feature graph with the initial disparity map in channel dimension, and sending the fused features into a hourglass type network for disparity optimization to obtain a final disparity map.

6. The guide information-based stereo matching method according to claim 1, wherein: step six, the disparity map generated by the network passes through smooth_L1Error between loss function calculation and true value; the error is subjected to gradient updating through an Adam optimization method so as to guide the training of the whole network model.

Technical Field

The invention belongs to the field of computer vision and the technical field of deep learning, and particularly relates to a three-dimensional matching method based on guide information.

Background

Most of the information acquired in daily life comes from vision. The human beings respectively obtain image information through the binocular that the position is different, and the brain can fuse two monocular images that obtain, judges the distance of object in the field of vision, and the degree of depth information to form the stereoscopic scene. Depth information is a very important part of many computer vision applications, such as autopilot, robotics, augmented reality, drone, 3D model reconstruction, object detection and recognition, etc.

Binocular stereopsis is the process of real scenes by simulating the human visual system. The binocular stereo matching is to find corresponding matched pixel points from two same scene images shot at different positions, so as to obtain the parallax value of the corresponding pixels, form a parallax map, and obtain the depth information of objects in the scene by calculating the parallax.

However, the current binocular stereo matching algorithm still cannot keep good real-time performance while keeping accuracy, and the high-accuracy algorithm is usually replaced by the high-accuracy algorithm at the expense of computational complexity. Therefore, the research object of the current binocular stereo matching technology is to explore a stereo matching algorithm with high precision and low computational complexity. Although experts and scholars in many fields have long studied and advanced the stereo matching technology, there are still some problems and difficulties in actual matching. For example, occlusion, parallax discontinuous regions, weak texture regions, and the like in a stereoscopic scene make it difficult to obtain matching results with high accuracy.

In recent years, the convolutional neural network technology is widely applied to the field of pre-computer vision, and a binocular stereo matching algorithm based on the convolutional neural network also achieves good performance. The convolutional neural network has higher robustness in extracting and processing image features, and can effectively overcome the influence of factors such as inconsistent scales, shielding and rotation in an image scene.

The traditional stereo matching algorithm is summarized into four steps: matching cost calculation, cost aggregation, parallax calculation and parallax optimization. Most conventional algorithms are manually carefully selected features, and a multi-stage regularization function is required, so that the effect and the progress of the conventional methods are limited finally. With the rapid development of the deep learning technology, the deep learning technology shows the powerful capability of the deep learning technology in feature learning, and makes a very important breakthrough and progress in various computer vision tasks. Similarly, the binocular stereo matching algorithm based on deep learning also achieves a very good effect compared with some traditional algorithms. According to the development process of deep learning, the method is mainly divided into a non-end-to-end deep learning algorithm and an end-to-end deep learning algorithm.

The non-end-to-end deep learning combines the convolutional neural network with the traditional method, and applies the convolutional neural network to a certain step in the stereo matching process. The end-to-end network approach to stereo matching can be divided into two main approaches. One is that earlier methods describe depth estimation as a regression problem, using a single encoder-decoder, superimposing the left and right images, and regressing the disparity map. The methods do not need an explicit feature matching module and run quickly. They require a large amount of training data and are difficult to obtain. Another approach mimics the traditional stereo matching step, breaking the problem into stages that can be composed of micro-blocks, allowing end-to-end training.

Although the effect of the stereo matching method based on deep learning is greatly improved, the local blurring problem is still difficult to overcome, which is a common problem in disparity estimation. This is because the matching features for disparity estimation on these blurred regions are not sufficient to guide the model to seek the correct convergence direction. Feature matching can be better achieved in fuzzy regions by taking advantage of more features such as global perception of foreground and background, scaling to known sizes relative to familiar objects, and semantic consistency of individuals. Such blurred regions in disparity estimation, mostly located in the center region of a given large object, can be handled by semantic segmentation. The correct correspondence is found in the non-textured areas, complex structures, small objects and near the boundaries, and these problems can be alleviated by information such as edge contours.

Disclosure of Invention

Object of the Invention

The invention provides a stereo matching method based on guide information, aiming at the problems of large error, shielding, discontinuous parallax and fuzzy areas of weak texture at the stereo matching edge and combining the problems of fixed maximum parallax over-parameter in a plurality of algorithms.

Technical scheme

The method comprises the following steps: image input: inputting left and right images, wherein the left image is used as a reference image, and the right image is used as a target image; and after the image is cut, sending the image to a VGG network for feature extraction.

Step two: feature extraction: respectively extracting features of the input left image and the input right image by using a convolution network VGG to obtain different scale image features smaller than the image in the first step;

step three: extracting edge information: respectively extracting edge characteristic graphs from the input left image and the input right image by using an integral nested edge detection HED network to obtain edge detail information; inputting the left camera image and the right camera image, sending the images into an HED network to extract an edge feature graph, performing up-sampling operation on the image features with different scales obtained in the step two, and then overlapping the image features with the edge feature graph on channel dimension to fuse edge information.

Step four: parallax attention initial parallax is calculated: fusing the image features of different scales obtained in the step two with edge information, inputting the fused image features and the edge information into a cascade parallax attention module to generate a parallax attention diagram, and regressing the parallax attention to obtain an initial parallax diagram; and sending the characteristics fused with the edge information into a cascade parallax attention module to calculate initial parallax, calculating the characteristic similarity of each pixel in the reference image and all possible differences in the right image by using the parallax attention module so as to generate a parallax attention diagram with the channel number of 1, and performing regression calculation on the parallax attention diagram to obtain the initial parallax.

Step five: extracting semantic information: performing pooling layer processing at the end of feature extraction, and sending the obtained pooling layer features into a densely connected void space convolution pooling pyramid DenseASPP network to extract a semantic feature map and obtain semantic information; and performing up-sampling operation on the semantic feature graph obtained by the semantic information extraction network, then overlapping the semantic feature graph with the initial disparity map in channel dimension, and sending the fused features into an hourglass type network for disparity optimization to obtain a final disparity map.

Step six: parallax optimization: and fusing the semantic feature graph and the initial disparity map, sending the fused image into an hourglass-shaped network, performing disparity optimization on the initial disparity map by utilizing semantic information to obtain a final disparity map, and finishing stereo matching. Network generated disparity map passes smooth_L1Error between loss function calculation and true value; the error is subjected to gradient updating through an Adam optimization method so as to guide the training of the whole network model.

Advantages and effects

The invention introduces an end-to-end thought in deep learning to combine four steps in the traditional stereo matching method into one step, uses parallax attention when calculating the matching cost, and compared with the traditional algorithm which generates a 3D/4D cost volume method end to end, the invention has the advantages that different stereo image pairs with larger parallax variation are prevented from being processed by fixed maximum parallax, and unreasonable cost distribution is possibly caused by the fuzziness of difference regression. The parallax attention mechanism can learn stereo correspondence under large parallax change, combine epipolar constraint with the attention mechanism, calculate feature similarity along the epipolar line, and specifically calculate the feature similarity of each pixel in the reference image and all possible differences in the target image to generate the parallax attention. Edge and semantic information is introduced into the parallax attention map, and the fuzzy region effects of large errors, occlusion, discontinuous parallax and weak texture at the stereo matching edge are improved.

The method of the invention adopts the addition of the edge and the semantic guidance information to the parallax attention to generate the parallax map, solves the problem that the occlusion, the parallax discontinuous area, the weak texture area and the like in the stereo scene are difficult to obtain the matching result with higher precision, and reduces the error matching and improves the precision.

Drawings

Fig. 1 is a processing flow of a stereo matching method based on guiding information according to the present invention;

fig. 2 is a schematic network structure diagram of a stereo matching method based on guiding information according to the present invention;

FIG. 3 is a schematic diagram of an HED network structure provided by the present invention;

fig. 4 is a schematic structural diagram of a parallax attention module according to the present invention;

FIG. 5 is a schematic structural diagram of an output module according to the present invention;

fig. 6 is a schematic diagram of a denseas spp network structure provided by the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings:

examples

Performing edge information extraction and feature extraction on left and right camera images, namely a reference image and a target image, and obtaining feature maps and edge features with different scales; then, the edge information and the features of different scales are fused and sent to a cascade observation attention module for matching cost calculation to obtain a parallax attention diagram; performing regression calculation on the visual attention diagram to obtain an initial disparity map; and extracting semantic information from the final features obtained from the feature extraction, and then fusing the semantic information and the initial disparity map for disparity optimization to obtain a final disparity map. The process flow of the method is shown in fig. 1.

The method comprises the following steps: image input: inputting left and right images, wherein the left image is used as a reference image, and the right image is used as a target image; and a cropping operation of size 256 x 512 is performed.

Step two: feature extraction: respectively extracting the features of the input left and right images by using 13 layers of convolution layers of the VGG network;

as shown in fig. 2, the left and right camera images are convolved by a 13-layer convolution layer network to extract features, the input image in the first step is cut into 256 × 512 images, and the two input images are respectively passed through 5 modules to obtain 5 different scale features of 128 × 256, 64 × 128, 32 × 64, 16 × 32, and 8 × 16 sizes. Each of the 5 modules consists of a plurality of convolutional layers, 1 BN layer, 1 ReLU activation function layer, and 1 max pooling layer. The convolution layer part of the 1 st module is 2 convolution layers with 64 output channels; the convolution layer part of the 2 nd module is 2 convolution layers with 96 output channels; the convolution layer part of the 3 rd module is a convolution layer with 3 output channels of 128; the convolution layer part of the 4 th module is a convolution layer with 3 output channels with 256 output channels; the convolution layer part of the 5 th module is a convolution layer with 3 output channels of 512; the convolutional layer uses a convolution with a size of 3 × 3, step size of 1 and padding of 1.

The BN layer calls a batch normalization (Batchnormalization) regularization method, and the BN layer is used for accelerating the training of the model and preventing the model from being over-fitted. Because the distribution rules of the processed pictures at each time are different in the expression mode of the numerical values, the learning of the network model is not facilitated, and the value range of the input data can be unified to the range of [ -1,1] by using a common batch normalization method in deep learning. In addition to solving the problem of difficulty in learning the network model, the method is also beneficial to the gradient updating of the back propagation.

The nonlinear utilization of the ReLU activation function can accelerate the network convergence speed and reduce the sensitivity of the network to the adjustment of the hyper-parameters. The specific processing when using batch normalization is to subtract the channel-by-channel calculated mean value from the batch size (batch size) after the convolution layer and divide by the standard deviation, and when dividing the image by the standard deviation in training, the divisor may be directly replaced by a value of 255, i.e. the maximum value of 8-bit unsigned integer representing the maximum number of RGB channels, in order to reduce the calculation amount. The mathematical expression for the ReLU activation function is as follows:

wherein x is_iRepresenting the input value, y_iRepresenting the output value.

Step three: extracting edge information: and sending the left camera image and the right camera image into an HED network to extract edge features to obtain an edge feature graph.

As shown in fig. 3, the HED network is a multi-scale and multi-fusion network structure, so-called multi-scale, that is, the output of the last convolution layer of each group of VGG16 is taken out, and because the length and width dimensions of the feature map obtained by each group are different, the feature map obtained by each group needs to be further operated by transposition convolution/deconvolution, and in effect, the length and width dimensions of the feature maps obtained by the second to fifth groups are respectively enlarged by 2 to 16 times, so that the feature maps obtained by each scale are all the same size, and 5-layer outputs are cascaded and converted into an edge map of 1 channel.

Step four: the cascade parallax attention mechanism performs cost calculation: the features after the edge information is fused are sent to a cascade parallax attention module to obtain a parallax attention diagram, and the parallax attention diagram is regressed to obtain an initial parallax;

the input of the parallax attention module is a feature obtained by the following processing: and (3) respectively performing bilinear interpolation upsampling operation on the first three scale features obtained in the feature extraction in the step two, namely the features of 128 × 256, 64 × 128 and 32 × 64 scales, and then cascading with edge information to obtain the feature.

The disparity attention module obtains different differences by matrix multiplication instead of a shift operation. Thus, the parallax attention module does not need to manually set a fixed maximum parallax, and can handle large parallax variations. Methods based on calculating cost volumes (cost volumes) typically regress differences based on matching costs and then calculate losses from these differences. However, this may lead to unreasonable costs due to the ambiguity of the difference regression. In contrast, performing direct regularization on the disparity attention map can achieve better performance. By using the disparity attention module, costs from the reference image and the target image can be aggregated without explicit disparity calculation.

The parallax attention module combines epipolar constraint with an attention mechanism to compute feature similarities along the epipolar lines. In the attention mechanism, the dimension R is first set^H×W×CIs reshaped into R^HW×CThen, using matrix multiplication (R)^HW×C×R^C×HW) The correlation of any two locations in the image is calculated. For the reference image and the target image, one of the reference imagesThe corresponding pixels of the pixels are located only on the polar lines in the target image. Given this polar constraint, parallax attention uses matrix multiplication to compute the correlation between pixels in the reference image and all locations along the epipolar line in the target image. As shown in FIG. 4, the feature mapping would be A, B ∈ R^H×W×CFirst fed to a 1 x 1 convolution for feature adaptation. Specifically, A is the input 1 × 1 convolution to produce the query feature map Q ∈ R^H×W×C. At the same time, B is fed into another 1 x 1 convolution, generating a key feature map K ∈ R^H×W×CThen remodeled into R^H×C×W. Then, matrix multiplication is performed between Q and K, and a Softmax layer is applied, resulting in a disparity-to-attention map M_B→A∈R^H×W×W. By matrix multiplication, the characteristic correlation between any two positions along the epipolar line can be efficiently encoded into the disparity attention map. It should be noted that the parallax attention mechanism takes into account all differences, that is, there is no need to manually set a fixed maximum parallax, and larger parallax variations can be handled. Since the parallax attention can learn to focus on the feature of the exact difference using the feature similarity, the correspondence can be captured.

The matching cost of the last block in the cascaded disparity attention module is fed to the output module as shown in fig. 5. In the output module, the output module is provided with a plurality of output modules,andis first fed to a Softmax layer to produce a parallax attention map of channel number 1, respectivelyAndand performing regression calculation on the obtained parallax attention diagram to obtain the initial parallax, wherein the formula is as follows:

wherein the content of the first and second substances,which represents the initial disparity, is the difference between the original disparity,is the parallax attention of the target image to the reference image calculated by the third parallax attention module, and w and k are feature dimensions.

Step five: extracting semantic information: the last layer of features obtained by feature extraction is sent to a DenseASPP network to obtain semantic information;

the denseas ASPP network shown in fig. 6 combines the ASPP and the dense connection in DenseNet to form denseas pp, which has a larger receptive field and denser sampling points, connects a set of dilation convolutions in a denser manner, obtains a larger range of dilation rates, and is denser. However, because of dense connection, the number of channels of the feature rises rapidly, 1 × 1 convolution is used for reducing the number of channels, the size and the calculation amount of the model are limited, the DenseASPP is composed of expansion convolutions with expansion rates of 3, 6, 12, 18 and 24, the 1 × 1 convolution before the expansion convolution reduces the number of channels to c/8 channels for reducing the number of feature maps, and finally the semantic feature maps with the channels of 1 are obtained through convolution. And (4) connecting and fusing the initial parallax with the obtained semantic feature map, and feeding the semantic feature map to a hourglass network for parallax optimization.

Step six: parallax optimization: the extracted semantic feature graph is subjected to up-sampling operation and then is sent to an hourglass type network with a three-layer down-sampling structure and a four-layer up-sampling structure in a cascading mode with the initial parallax graph, parallax optimization is carried out on the initial parallax graph by utilizing semantic information to obtain a final parallax graph, and the obtained semantic information and the initial parallax graph are fused and sent to the hourglass type network for parallax optimization.

As shown in fig. 2, the parallax optimized partial hourglass network consists of three layers of downsampling and four layers of upsampling, the three layers of downsampling are convolutional layers with the number of output channels of 96, 128 and 160; four-layer upsampling is a convolutional layer with output channels of 160, 128, 96 and 32.

Network generated disparity map passes smooth_L1The error between the loss function calculation and the true value. The loss function is as follows:

wherein L is_SRepresenting the loss function, N representing the number of pixels with true values, D_iAndrepresenting true and estimated disparities, smooth, respectively_L1The definition is as follows:

the error is subjected to gradient updating through an Adam optimization method so as to guide the training of the whole network model.

The training process is to guide the training of the whole network model through a loss function, namely, the gradient is updated by an optimization method, so that the gradient is continuously reduced to approach an optimal solution to update the weight parameters, and the weight parameters need to consider two aspects of weight initialization and optimization methods.

The weight initialization is to enable the network model to have a better initial position when seeking a global optimal solution in a numerical space, so that better and faster convergence is facilitated during network model learning. HED and DenseASPP networks are introduced into the network and used for extracting edge and semantic information, trained model parameters are introduced when weights are initialized, training time is saved, and random normal distribution with the mean value of 0 and the variance of 0.02 is adopted when weights of other convolutional layers are initialized.

The process by which the network model searches for the optimal solution may be referred to as optimization. The method adopted during optimization is an Adam method improved on a gradient descent method, and the Adam method is used for automatically adjusting the learning rate to help the network model to better and faster converge during learning as long as initial values of some related hyper-parameters are set, so that the matching training of the whole network model is carried out, wherein the hyper-parameter values are set according to default parameters in the Adam method.