阅读说明：本技术 一种基于深度学习的多模式分像素插值方法 (Multi-mode sub-pixel interpolation method based on deep learning ) 是由 刘家瑛 夏思烽 胡越予 郭宗明 于 2019-03-12 设计创作，主要内容包括：本发明公开了一种基于深度学习的多模式分像素插值方法,其步骤包括：1)差分预测网络对输入的已编码的整像素参考块采用两种模式进行预测,生成两种模式下的分像素预测值；模式一、差分预测网络预测所有目标亚像素到该整像素参考块的左上角整像素的残差,并将左上角整像素与预测残差相加,得到一组目标亚像素值；模式二、差分预测网络预测目标亚像素与该整像素参考块的左上角之外的一整像素的残差,计算对应的目标亚像素预测值；2)编码器分别使用已有插值方法生成的亚像素级参考块、模式一和模式二生成的亚像素级参考块,对待编码块进行帧间编码,然后基于三种编码的编码效果选择最佳亚像素级参考块,并将供解码器端使用的相应信息存储到码流。(The invention discloses a multi-mode sub-pixel interpolation method based on deep learning, which comprises the following steps: 1) the difference prediction network adopts two modes to predict the input coded integer pixel reference block to generate sub-pixel prediction values under the two modes; in the mode I, the differential prediction network predicts the residual errors from all target sub-pixels to the upper left corner whole pixel of the whole pixel reference block, and adds the upper left corner whole pixel and the predicted residual errors to obtain a group of target sub-pixel values; in the second mode, the difference prediction network predicts the residual error between the target sub-pixel and an integral pixel outside the upper left corner of the integral pixel reference block and calculates the corresponding target sub-pixel predicted value; 2) the encoder respectively uses the sub-pixel level reference block generated by the existing interpolation method and the sub-pixel level reference block generated by the mode I and the mode II to perform interframe coding on a block to be coded, then selects the optimal sub-pixel level reference block based on the coding effects of the three kinds of coding, and stores corresponding information used by a decoder end into a code stream.)

1. A multi-mode pixel-division interpolation method based on deep learning comprises the following steps:

1) the difference prediction network adopts two modes to predict the input coded integer pixel reference block to generate sub-pixel prediction values under the two modes; in the first mode, the differential prediction network predicts the residual error from all target sub-pixels to the upper left corner whole pixel of the whole pixel reference block, and adds the upper left corner whole pixel and the predicted residual error to obtain a group of target sub-pixel values; in the second mode, the difference prediction network predicts the residual error between the target sub-pixel and one whole pixel except the upper left corner of the whole pixel reference block, and obtains the corresponding target sub-pixel prediction value based on the obtained residual error;

2) the encoder respectively uses a sub-pixel level reference block generated by the existing interpolation method, a sub-pixel level reference block generated by the mode one and a sub-pixel level reference block generated by the mode two to perform interframe coding on a block to be coded, then selects the best sub-pixel level reference block based on the coding effects of the three kinds of coding, and stores corresponding information for the decoder end to a code stream.

2. The method of claim 1, wherein the method of training the differential prediction network is:

11) acquiring a plurality of sample pictures, performing alternate sampling and coding reconstruction on each sample picture to obtain a whole pixel block, performing alternate sampling and blurring on the sample pictures to obtain sub-pixel blocks, and obtaining training data corresponding to the sample pictures;

12) sending the whole pixel block into a differential prediction network, carrying out forward calculation of the differential prediction network to obtain a residual prediction value, and then adding the predicted residual value and the whole pixel at the corresponding position of the prediction mode to obtain a target sub-pixel;

13) calculating the mean square error of the calculation result obtained in the step 12) and the target sub-pixel block in the training data;

14) reversely propagating the calculated mean square error to each layer of the differential prediction network so as to update the weight of each layer;

15) and repeating the steps 11) to 14) until the mean square error of the differential prediction network is converged.

3. The method of claim 1 or 2, wherein the mode one comprises a fractional pixel interpolation model of 1/2 pixel bits and a fractional pixel interpolation model of 1/4 pixel bits; the second pattern includes 1/2 pixel bit sub-pixel interpolation models and 1/4 pixel bit sub-pixel interpolation models.

4. The method of claim 3, wherein for 1/2 fractional-pel interpolation, the training data for each sample image is a whole block of pixels and a corresponding 3 blocks of 1/2 fractional-pel blocks; for 1/4 fractional-pel interpolation, the training data for each sample image is a whole block of pixels and 12 1/4 fractional-pel blocks.

5. The method of claim 1, wherein mode two is: and predicting residual errors of the target sub-pixels and integral pixels at the upper right corner, the lower left corner or the lower right corner by using the differential prediction network, and obtaining corresponding target sub-pixel predicted values based on the obtained residual errors.

6. The method of claim 1, wherein mode two is: and predicting the residual error between the target sub-pixel and the nearest whole pixel by the difference prediction network, and obtaining a corresponding target sub-pixel predicted value based on the obtained residual error.

Technical Field

The invention belongs to the field of video coding, and mainly relates to a pixel-by-pixel interpolation method for interframe motion compensation. Can be used to improve video compression.

Background

In the use and transmission process of digital video, video coding and decoding are indispensable key technologies. The video coding and decoding technology greatly reduces the cost of the digital video in the storage and transmission processes by carrying out coding compression on the video at the coding end and decoding recovery on the video at the decoding end, so that the digital video can be widely used in daily life. Motion compensation is a key method for improving video compression ratio by using inter-frame redundant information in video coding and decoding technology.

In the motion compensation process, the encoder searches for an encoded reference block similar to a current video frame block to be encoded in an encoded and compressed video frame, and based on the similar encoded reference block, the encoder can only encode and record a residual error between the block to be encoded and the reference block and index information of the reference block without encoding complete information of the block to be encoded, so that the storage space required by encoding is reduced, and the compression ratio is improved. However, due to the discretization property of video samples, in motion compensation finding an encoded reference block, when the motion offset between the block to be encoded and the reference block is at sub-pixel precision, it will be difficult to find a reference block in the reference frame that is sufficiently similar to the block to be encoded.

For this reason, in the motion compensation technique, a sub-pixel interpolation algorithm is used to perform sub-pixel level interpolation on reference blocks in encoded adjacent frames, and generate sub-pixel blocks with different sub-pixel precisions, so as to obtain more reference information and obtain more similar reference blocks to further assist in encoding compression. The current coding technology generally interpolates 15 sub-pixel levels reaching 1/4 pixel precisionThe reference block is used as an additional inter-frame reference, and the relative position relationship between integer pixels and sub-pixels is schematically shown in FIG. 1, and for the integer pixel block I^AEach integer pixel in (1)There are 3 corresponding 1/2 bit pixelsAnd 12 1/4 bit pixels

The current coding technology generally adopts a manually designed simple and fixed interpolation filter for interpolation, and the interpolation method of the type often cannot well process various video signals due to the fact that the adopted interpolation filter is simple and fixed.

Inspired by the successful application of the deep neural network technology in the image processing problem, some methods introduce the deep neural network into the sub-pixel interpolation algorithm and obtain certain performance improvement. However, the existing method still models the sub-pixel interpolation problem into the conventional interpolation problem when a deep neural network is constructed, and predicts all sub-pixels by using integer pixels at a single position. Namely, the existing method only predicts the residual error between the sub-pixel and the upper left corner integer pixel to realize the prediction of the sub-pixel, and the prediction result is not accurate enough.

Disclosure of Invention

Aiming at the technical problems in the prior art, the invention aims to provide a multi-mode pixel-division interpolation method based on deep learning, which realizes the prediction of multiple modes of pixel division by predicting the residual error between a sub-pixel and integer pixels at different positions, and provides the multi-mode prediction result for an encoder to select, thereby obtaining better encoding performance.

The technical scheme of the invention is as follows:

a multi-mode pixel-division interpolation method based on deep learning comprises the following steps:

1) the difference prediction network adopts two modes to predict the input coded integer pixel reference block to generate sub-pixel prediction values under the two modes; in the first mode, the differential prediction network predicts the residual error from all target sub-pixels to the upper left corner whole pixel of the whole pixel reference block, and adds the upper left corner whole pixel and the predicted residual error to obtain a group of target sub-pixel values; in the second mode, the difference prediction network predicts the residual error between the target sub-pixel and one whole pixel except the upper left corner of the whole pixel reference block, and obtains the corresponding target sub-pixel prediction value based on the obtained residual error;

2) the encoder respectively uses a sub-pixel level reference block generated by the existing interpolation method, a sub-pixel level reference block generated by the mode one and a sub-pixel level reference block generated by the mode two to perform interframe coding on a block to be coded, then selects the best sub-pixel level reference block based on the coding effects of the three kinds of coding, and stores corresponding information for the decoder end to a code stream.

Further, the method for training the differential prediction network comprises the following steps:

11) acquiring a plurality of sample pictures, performing alternate sampling and coding reconstruction on each sample picture to obtain a whole pixel block, performing alternate sampling and blurring on the sample pictures to obtain sub-pixel blocks, and obtaining training data corresponding to the sample pictures;

12) sending the whole pixel block into a differential prediction network, carrying out forward calculation of the differential prediction network to obtain a residual prediction value, and then adding the predicted residual value and the whole pixel at the corresponding position of the prediction mode to obtain a target sub-pixel;

13) calculating the mean square error of the calculation result obtained in the step 12) and the target sub-pixel block in the training data;

14) reversely propagating the calculated mean square error to each layer of the differential prediction network so as to update the weight of each layer;

15) and repeating the steps 11) to 14) until the mean square error of the differential prediction network is converged.

Further, the mode one includes 1/2 pixel bit sub-pixel interpolation model and 1/4 pixel bit sub-pixel interpolation model; the second pattern includes 1/2 pixel bit sub-pixel interpolation models and 1/4 pixel bit sub-pixel interpolation models.

Further, for 1/2 fractional-pel interpolation, the training data corresponding to each sample image is a whole pixel block and 3 corresponding 1/2 fractional-pel blocks; for 1/4 fractional-pel interpolation, the training data for each sample image is a whole block of pixels and 12 1/4 fractional-pel blocks.

Further, the second mode is: and predicting residual errors of the target sub-pixels and integral pixels at the upper right corner, the lower left corner or the lower right corner by using the differential prediction network, and obtaining corresponding target sub-pixel predicted values based on the obtained residual errors.

Further, the second mode is: and predicting the residual error between the target sub-pixel and the nearest whole pixel by the difference prediction network, and obtaining a corresponding target sub-pixel predicted value based on the obtained residual error.

The invention finds that for part of sub-pixels, the whole pixels are closer to the part of sub-pixels, such as the sub-pixels shown in the figure 1Its distance is whole pixelIntegral pixelIn the more recent past, the more recent is,toThe residual of (a) can be predicted more accurately; therefore, on the basis of predicting all sub-pixels based on the upper left corner integer pixel in the prior art, the invention additionally selects the integer pixel at another position for each sub-pixel and provides the prediction result of another group of the sub-pixels. The two sets of prediction results are used together as an inter-frame reference at a sub-pixel level for the encoder.

The prediction process is mainly realized based on a differential prediction network, the network takes a coded reference block, namely a whole pixel block, as input, learns a group of nonlinear mapping parameters based on pre-prepared training data, and obtains the sub-pixel level prediction of the whole pixel block based on the parameters to obtain the required sub-pixel block.

On the basis of the prediction mode, the invention designs two prediction modes, and the two prediction modes are shown in the attached figures 2 and 3. In both prediction modes, the network structure and the input are kept the same, the input being the encoded integer-pixel reference block, but both modes will predict the residue of the corresponding target sub-pixel of the input integer-pixel reference block to the integer-pixel at different positions of the integer-pixel reference block. In the first mode, as shown in fig. 2, the network predicts the residual error from all target sub-pixels to the upper left integer pixel of the integer pixel reference block, and adds the upper left integer pixel to the predicted residual error to obtain 15 target sub-pixel values; and in the second mode, for each target sub-pixel to be predicted, selecting an integer pixel at the nearest neighbor position from integer pixels at the three positions of the upper right corner, the lower left corner and the lower right corner of the target sub-pixel, and predicting the residual error between the target sub-pixel and the selected integer pixel to obtain a target sub-pixel predicted value in the second mode. The specific selection is shown in figure 3. On the basis of the obtained sub-pixel predicted values of the two modes, the encoder uses a sub-pixel level reference block generated by the existing interpolation method and two groups of sub-pixel level reference blocks generated by the invention to carry out inter-frame coding on a block to be coded, and based on the actual coding effect, the encoder selects the best reference block and stores corresponding information (namely the method for obtaining the best reference block, including the method for obtaining the original interpolation method of the encoder, the method for using the interpolation method of the invention and which interpolation mode is selected) into a code stream for a decoder end to use.

The main steps of the method of the invention are described next.

In the invention, four sub-pixel interpolation models of 1/2 pixel bits and 1/4 pixel bits in two modes are trained. Assuming that a sub-pixel interpolation algorithm of a 1/N pixel bit corresponding mode is to be realized, enough various pictures need to be collected first, training data is generated through steps of alternate point sampling, blurring, coding simulation and the like, and a batch of picture pairs of a whole pixel block and a sub-pixel block are obtained. And enabling the generated whole pixel block to flow through the network to obtain a final predicted sub-pixel bit block, performing mean square error calculation on the predicted sub-pixel bit block and the sub-pixel block in the generated training data, using the calculation result as an error value, reversely transmitting the error value to each layer of the network, and updating the weight in the network. Iterating until the neural network model converges, as follows:

step 1: collecting a batch of pictures, carrying out alternate sampling and coding reconstruction on the pictures to obtain an entire pixel block, carrying out alternate sampling and blurring on the pictures to obtain sub-pixel blocks, and constructing a training data pair.

Step 2: and sending the whole pixel block into a network to perform forward calculation of the network. And after the network acquires the corresponding residual prediction value, based on the prediction mode to which the training model belongs, adding the integer pixel at the corresponding position with the predicted residual value to acquire the target sub-pixel.

And step 3: and 2, obtaining a calculation result, and calculating the mean square error with the target sub-pixel block in the training data.

And 4, step 4: and reversely transmitting the calculated mean square error to each layer of the neural network so as to update the weight of each layer, and enabling the result to be closer to the target effect in the next iteration.

And 5: and repeating the steps 1-4 until the mean square error of the neural network is converged.

After the trained network model is obtained, the model is applied to an interframe motion compensation algorithm of an encoder to generate sub-pixel prediction values under two modes, the encoder is enabled to respectively use sub-pixel blocks generated by the original interpolation algorithm to encode through 3 times of encoding, the sub-pixel block predicted in the mode 1 is encoded, the sub-pixel block predicted in the mode 2 is encoded, the optimal sub-pixel block is selected according to the three encoding results, and the corresponding selection result is recorded and transmitted to a decoder to ensure the matching of encoding and decoding.

Compared with the prior art, the invention has the following positive effects:

the method and the device realize the multi-mode prediction of the split pixels by predicting the residual errors between the split pixels and the integer pixels at different positions, and provide the multi-mode prediction result for an encoder to select, thereby obtaining better encoding performance. By using the sub-pixels generated by the invention, the test is carried out on a sequencing-by-sequence, and compared with the encoding result of the original encoder, the compression rate of 2.8 percent can be averagely improved on the brightness component occupying the video data main body, and the specific improvement effect is as follows:

drawings

FIG. 1 is a diagram illustrating relative positions of integer pixels and sub-pixels;

FIG. 2 is a diagram of a sub-pixel prediction mode 1 according to the present invention;

FIG. 3 is a diagram illustrating a sub-pixel prediction mode 2 according to the present invention;

FIG. 4 is a schematic flow chart of generating training data according to the present invention.

Detailed Description

In order to further explain the technical method of the present invention, the following describes the sub-pixel interpolation method of the present invention in detail with reference to the drawings and specific examples.

This example will focus on a detailed description of the training process of the neural network in the technical approach. Suppose we have now constructed the required convolutional neural network model and have N training images { I }₁,I₂,...,I_NAs a training set, a sub-pixel block interpolation network with precision in mode 2 is trained 1/4.

The method of the example is as follows with reference to the attached drawings:

firstly, a training process:

step 1: will train set { I₁,I₂,...,I_NEach image I in_kTraining data generation as in fig. 4 is performed. For the generation of the whole pixel block, firstly, alternate point downsampling is carried out to obtain a preliminary whole pixel block, then, an encoder is used for encoding to obtain the result of encoding reconstructionFor the generation of the sub-pixel block, firstly, the standard deviation value of the picture is the interval [0.5,0.6 ]]Internal randomSeveral gaussian blurred pictures (1/2 blocks are generated 0.4 to 0.5). Performing alternate sampling on the Gaussian blur picture to obtain 1/4 sub-pixel blocks

Thus, the required training data set can be obtainedAnd in the subsequent training process, the network weight is updated based on the training data iteration, and the generated training data pairs are randomly selected each time for training the network.

Step 2: taking an iteration as an example, assume that this iteration will be

As input, the network performs forward propagation based on the existing parameters to obtain the residual between the target sub-pixel block and the whole pixel block

In mode 2, with reference to fig. 1 and fig. 3, the value of each sub-pixel position (i, j) in the target sub-pixel block is finally obtained based on the following formula:

a target sub-block of pixels is predicted for the net.

And step 3: sub-pixel block with step 2 predictionAnd label sub-pixel blocks in training dataAnd calculating the mean square error of the network prediction.

And 4, step 4: after the mean square error value is obtained, the error value gradient is propagated reversely to train the network to update the network parameters so as to reduce the prediction error of the network.

And 5, repeating the steps 2-4 until the neural network converges, namely the mean square error of the network predicted value and the original target value is not reduced.

Secondly, an encoding process:

after 1/2 and 1/4 sub-pixel interpolation networks of two modes are trained, in the actual test of an encoder, a searched coded reference block is input into 1/2 and 1/4 sub-pixel interpolation networks of the two modes which are trained, the interpolation prediction results of 30 sub-pixel blocks of the two modes are obtained, the interpolation prediction results and the original sub-pixel interpolation results of the encoder are subjected to an attempt of three-pass motion compensation by the encoder, and a corresponding sub-pixel interpolation method is selected based on the actual motion compensation results.

Fig. 1 depicts the relative positions of integer pixels and sub-pixels, fig. 2 and 3 summarize two sub-pixel prediction modes of the present invention, and fig. 4 summarizes the training data generation method of the present invention. It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.