阅读说明：本技术 一种视频帧预测方法、装置及终端设备 (Video frame prediction method and device and terminal equipment ) 是由 李东阳 于 2019-11-29 设计创作，主要内容包括：本发明适用于视频压缩技术领域，提供了一种视频帧预测方法、装置及终端设备，所述方法包括：计算当前帧和参考帧之间的光流信息；将所述光流信息、参考帧和当前帧输入运动补偿网络得到运动补偿特征信息；对所述运动补偿特征信息进行熵编码和熵解码后输入所述运动补偿网络得到重构光流信息、分离卷积核和mask(掩膜)；基于所述重构光流信息、分离卷积核和mask得到当前帧的预测帧。本发明通过结合光流和分离卷积来使得分离卷积核具有自适应位置映射的效果，从而可减小分离卷积核的大小，提高了视频预测的性能。(The invention is suitable for the technical field of video compression, and provides a video frame prediction method, a video frame prediction device and terminal equipment, wherein the method comprises the following steps: calculating optical flow information between the current frame and the reference frame; inputting the optical flow information, the reference frame and the current frame into a motion compensation network to obtain motion compensation characteristic information; entropy coding and decoding the motion compensation characteristic information, and inputting the motion compensation characteristic information into the motion compensation network to obtain reconstructed optical flow information, a separation convolution kernel and a mask; and obtaining a prediction frame of the current frame based on the reconstructed optical flow information, the separation convolution kernel and the mask. The invention combines the optical flow and the separation convolution to enable the separation convolution kernel to have the effect of self-adaptive position mapping, thereby reducing the size of the separation convolution kernel and improving the performance of video prediction.)

1. A method for video frame prediction, comprising:

calculating optical flow information between the current frame and the reference frame;

inputting the optical flow information, the reference frame and the current frame into a motion compensation network to obtain motion compensation characteristic information;

entropy coding and decoding the motion compensation characteristic information, and inputting the motion compensation characteristic information into the motion compensation network to obtain reconstructed optical flow information, a separation convolution kernel and a mask;

and obtaining a prediction frame of the current frame based on the reconstructed optical flow information, the separation convolution kernel and the mask.

2. The video frame prediction method of claim 1, wherein said calculating optical flow information between the current frame and the reference frame comprises:

and calculating the spatial position mapping relation between the pixels of the current frame image and the pixels of the reference frame image to obtain optical flow information.

3. The video frame prediction method of claim 1, wherein said deriving the predicted frame of the current frame based on the reconstructed optical flow information, the separate convolution kernel, and the mask comprises:

performing warp operation on the reference frame according to the reconstructed optical flow information to obtain a warp prediction frame;

performing separation convolution operation on the reference frame and the separation convolution kernel to obtain a separation convolution prediction frame;

and fusing the warp predicted frame and the separated convolution predicted frame according to the mask to obtain a predicted frame of the current frame.

4. The method of claim 3, wherein said fusing the warp predicted frame and the separate convolution predicted frame to obtain the predicted frame of the current frame according to the mask comprises:

the mask, warp predicted frame, separate convolution predicted frame and predicted frame of the current frame satisfy the relation:

the Warp prediction frame × mask + split convolution prediction frame × (1-mask) — the prediction frame of the current frame.

5. The video frame prediction method of claim 1, further comprising, after said deriving a predicted frame for a current frame based on said reconstructed optical flow information and a separate convolution kernel:

subtracting the predicted frame of the current frame from the current frame to obtain a residual error;

inputting the residual error into the residual error compression network to obtain a decompressed residual error;

and adding the predicted frame of the current frame and the decompressed residual error to obtain a reconstructed frame of the current frame.

6. A video frame prediction apparatus, comprising:

the optical flow module is used for calculating optical flow information between the current frame and the reference frame;

the motion compensation module is used for inputting the optical flow information, the reference frame and the current frame into a motion compensation network to obtain motion compensation characteristic information;

the entropy coding and decoding module is used for performing entropy coding and entropy decoding on the motion compensation characteristic information and then inputting the motion compensation characteristic information into the motion compensation network to obtain reconstructed optical flow information, a separation convolution kernel and a mask;

and the prediction frame module is used for obtaining a prediction frame of the current frame based on the reconstructed optical flow information, the separation convolution kernel and the mask.

7. The video frame prediction apparatus of claim 6, wherein the optical flow module comprises:

and the optical flow information calculation unit is used for calculating the spatial position mapping relation between the pixels of the current frame image and the pixels of the reference frame image to obtain optical flow information.

8. The video frame prediction device of claim 6, wherein the predicted frame module comprises:

a warp unit, configured to perform a warp operation on the reference frame according to the reconstructed optical flow information to obtain a warp prediction frame;

the separation convolution unit is used for carrying out separation convolution operation on the reference frame and the separation convolution kernel to obtain a separation convolution prediction frame;

and the Mask unit is used for fusing the warp predicted frame and the separated convolution predicted frame according to the Mask to obtain a predicted frame of the current frame.

9. Video frame prediction terminal device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor realizes the steps of the method according to any of the claims 1 to 5 when executing said computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.

Technical Field

The invention belongs to the technical field of video compression, and particularly relates to a video frame prediction method, a video frame prediction device and terminal equipment.

Background

The video compression technology disclosed at present mainly includes prediction by optical flow and separation convolution, however, the optical flow method is suitable for predicting rigid motion such as translation, the separation convolution is suitable for predicting non-rigid motion such as rotation and scaling, and when the video includes both rigid motion and non-rigid motion, neither method can be used for prediction, and the video prediction performance is low.

Therefore, a new technical solution is needed to solve the above problems.

Disclosure of Invention

In view of this, embodiments of the present invention provide a video frame prediction method and a terminal device, so as to solve the problem of low video prediction performance in the prior art.

A first aspect of an embodiment of the present invention provides a video frame prediction method, including:

calculating optical flow information between the current frame and the reference frame;

inputting the optical flow information, the reference frame and the current frame into a motion compensation network to obtain motion compensation characteristic information;

entropy coding and decoding the motion compensation characteristic information, and inputting the motion compensation characteristic information into the motion compensation network to obtain reconstructed optical flow information, a separation convolution kernel and a mask;

and obtaining a prediction frame of the current frame based on the reconstructed optical flow information, the separation convolution kernel and the mask.

A second aspect of the embodiments of the present invention provides a video frame prediction apparatus, including:

the optical flow module is used for calculating optical flow information between the current frame and the reference frame;

the motion compensation module is used for inputting the optical flow information, the reference frame and the current frame into a motion compensation network to obtain motion compensation characteristic information;

the entropy coding and decoding module is used for performing entropy coding and entropy decoding on the motion compensation characteristic information and then inputting the motion compensation characteristic information into the motion compensation network to obtain reconstructed optical flow information, a separation convolution kernel and a mask;

and the prediction frame module is used for obtaining a prediction frame of the current frame based on the reconstructed optical flow information, the separation convolution kernel and the mask.

A third aspect of embodiments of the present invention provides a video frame prediction terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method provided in the first aspect when executing the computer program.

A fourth aspect of embodiments of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method as provided in the first aspect above.

Compared with the prior art, the embodiment of the invention has the following beneficial effects:

the invention combines the optical flow and the separation convolution to enable the separation convolution kernel to have the effect of self-adaptive position mapping, thereby reducing the size of the separation convolution kernel and improving the performance of video prediction.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic flow chart illustrating an implementation of a video frame prediction method according to an embodiment of the present invention;

FIG. 2 is a diagram of an apparatus for predicting video frames according to an embodiment of the present invention;

fig. 3 is a schematic diagram of a video frame prediction terminal device according to an embodiment of the present invention.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

In order to explain the technical means of the present invention, the following description will be given by way of specific examples.