阅读说明：本技术 丢帧处理方法及装置 (Frame loss processing method and device ) 是由 唐春平 范志刚 于 2019-08-19 设计创作，主要内容包括：本公开提供一种丢帧处理方法及装置,涉及计算机图像领域,所述方法包括接收编码数据,并对所述编码数据进行解码得到至少一帧原始图像,并存储所述至少一帧原始图像；获取丢帧通知消息,其中,所述丢帧通知消息携带有指示发生丢帧的标识信息；获取与所述丢帧通知消息对应的丢帧接收设备标识；根据所述丢帧接收设备标识,对存储的最后一帧原始图像进行参考帧编码并发送给对应的丢帧设备。本公开能够解决现有接收端出现丢帧,如果对接收到丢失掉的图像帧进行重传,则会导致接收端的实时画面播放发生延时,如果立刻切换参考帧,则会使得VGR侧的并发数据量瞬间增大的问题。(The invention provides a frame loss processing method and a device, which relate to the field of computer images, wherein the method comprises the steps of receiving coded data, decoding the coded data to obtain at least one original frame image, and storing the at least one original frame image; acquiring a frame loss notification message, wherein the frame loss notification message carries identification information indicating that frame loss occurs; acquiring a lost frame receiving equipment identifier corresponding to the lost frame notification message; and according to the frame loss receiving equipment identification, carrying out reference frame coding on the stored original image of the last frame and sending the coded original image of the last frame to corresponding frame loss equipment. The method and the device can solve the problems that the frame loss occurs in the existing receiving end, if the lost image frame is retransmitted, the real-time image playing of the receiving end is delayed, and if the reference frame is switched at once, the concurrent data volume of the VGR side is increased instantly.)

1. A method for handling frame loss, the method comprising:

receiving encoded data, decoding the encoded data to obtain at least one frame of original image, and storing the at least one frame of original image;

acquiring a frame loss notification message, wherein the frame loss notification message carries identification information indicating that frame loss occurs;

acquiring a lost frame receiving equipment identifier corresponding to the lost frame notification message;

and according to the frame loss receiving equipment identification, carrying out reference frame coding on the stored original image of the last frame and sending the coded original image of the last frame to corresponding frame loss equipment.

2. The frame loss processing method of claim 1, wherein the frame loss notification message further carries a frame number of a frame loss, the method further comprising:

judging whether the corresponding frame loss is a reference frame according to the frame number of the frame loss;

the step of encoding the reference frame of the stored original image of the last frame according to the identifier of the frame loss receiving device and sending the encoded reference frame to the corresponding frame loss device comprises the following steps:

and if the frame loss is a reference frame, according to the identification of the frame loss receiving equipment, carrying out reference frame coding on the stored original image of the last frame and sending the coded original image of the last frame to the corresponding frame loss equipment.

3. The frame loss processing method of claim 2, further comprising:

and if the lost frame is a non-reference frame, transmitting the received coded data to all receiving devices.

4. The frame loss processing method according to claim 1, wherein said decoding the encoded data to obtain at least one original image, and storing the at least one original image comprises:

and decoding the coded data to obtain YUV data of the at least one frame of original image, and storing the YUV data of the at least one frame of original image.

5. The frame loss processing method of claim 2, wherein the determining whether the corresponding frame loss is a reference frame according to the frame number of the frame loss comprises:

inquiring whether the frame number of the lost frame exists in a preset reference frame list, and if so, judging the lost frame to be a reference frame;

if not, judging the frame loss as a non-reference frame; the preset reference frame list stores frame numbers of all reference frames.

6. A frame loss processing apparatus, the apparatus comprising:

the receiving module is used for receiving the coded data, decoding the coded data to obtain at least one frame of original image and storing the at least one frame of original image;

a notification obtaining module, configured to obtain a frame loss notification message, where the frame loss notification message carries identification information indicating that a frame loss occurs;

an obtaining module, configured to obtain a frame loss receiving device identifier corresponding to the frame loss notification message;

and the first sending module is used for coding the reference frame of the stored original image of the last frame according to the identification of the frame loss receiving equipment and sending the coded reference frame to the corresponding frame loss equipment.

7. The frame loss processing apparatus of claim 6, wherein the frame loss notification message further carries a frame number of a frame loss, and the apparatus further comprises a determining module, configured to determine whether a corresponding frame loss is a reference frame according to the frame number of the frame loss;

and the first sending module is specifically used for coding the reference frame of the stored original image of the last frame according to the identifier of the frame loss receiving device and sending the coded reference frame to the corresponding frame loss device if the frame loss is the reference frame.

8. The frame loss processing apparatus of claim 6, wherein the apparatus further comprises:

and the second sending module is used for sending the received coded data to all receiving devices if the lost frame is a non-reference frame.

9. The device of claim 6, wherein the receiving module is specifically configured to:

receiving coded data, decoding the coded data to obtain YUV data of the at least one frame of original image, and storing the YUV data of the at least one frame of original image.

10. The device for processing frame loss according to claim 7, wherein the determining module is specifically configured to:

inquiring whether the frame number of the lost frame exists in a preset reference frame list, and if so, judging the lost frame to be a reference frame;

if not, judging the frame loss as a non-reference frame; the preset reference frame list stores frame numbers of all reference frames.

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a frame loss processing method and apparatus.

Background

The one-to-many transmission of the real-time pictures refers to: and distributing the same real-time picture source to a plurality of accessed receiving terminals. In this scenario, a sending end (S end) encodes a captured image, an image router (VGR) distributes the encoded image, and a receiving end (R end) decodes and displays the encoded image received by each. The basic architecture can be seen in fig. 1. Three R terminals (R1, R2, and R3) are schematically shown in fig. 1, and those skilled in the art will appreciate that one or more of the R terminals may be actually provided.

In the above framework, when the S-side performs inter-frame prediction coding on the image, the coded non-reference frame needs to be coded according to the content of the reference frame, and in this case, once the receiving end loses a frame, the non-reference frame may not be decoded normally, and the problem of screen skip or frame skip may occur. If the lost image frame is retransmitted, the real-time image playing of the receiving end is delayed. If the S-side switches the I frame immediately, the amount of concurrent data on the VGR side is increased instantaneously, and especially, if the S-side switches the I frame continuously, the bandwidth on the VGR side is greatly affected if the S-side frequently loses frames when the network status is not good.

Disclosure of Invention

The embodiment of the disclosure provides a frame loss processing method and device, which can solve the problems that the frame loss occurs in the conventional receiving end, if the lost image frame is retransmitted, the real-time image playing of the receiving end is delayed, and if an S end switches an I frame immediately, the concurrent data volume of a VGR side is increased instantly. The technical scheme is as follows:

according to a first aspect of the embodiments of the present disclosure, a frame loss processing method is provided, the method including:

receiving encoded data, decoding the encoded data to obtain at least one frame of original image, and storing the at least one frame of original image;

acquiring a frame loss notification message, wherein the frame loss notification message carries identification information indicating that frame loss occurs;

acquiring a lost frame receiving equipment identifier corresponding to the lost frame notification message;

and according to the frame loss receiving equipment identification, carrying out reference frame coding on the stored original image of the last frame and sending the coded original image of the last frame to corresponding frame loss equipment.

In one embodiment, the frame loss notification message further carries a frame number of the frame loss, and the method further includes:

judging whether the corresponding frame loss is a reference frame according to the frame number of the frame loss;

the step of encoding the reference frame of the stored original image of the last frame according to the identifier of the frame loss receiving device and sending the encoded reference frame to the corresponding frame loss device comprises the following steps:

and if the frame loss is a reference frame, the reference frame coding is carried out on the stored original image of the last frame according to the identification of the frame loss receiving equipment, and the reference frame coding is sent to the corresponding frame loss equipment.

In one embodiment, the processing of frame loss according to the at least one original image comprises:

acquiring a lost frame receiving equipment identifier corresponding to the lost frame notification message;

and according to the frame loss receiving equipment identification, carrying out reference frame coding on the stored original image of the last frame and sending the coded original image of the last frame to corresponding frame loss equipment.

In one embodiment, the method further comprises:

and if the lost frame is a non-reference frame, transmitting the received coded data to all receiving devices.

In one embodiment, decoding the encoded data to obtain at least one original image, and storing the at least one original image includes:

and decoding the coded data to obtain YUV data of the at least one frame of original image, and storing the YUV data of the at least one frame of original image.

In one embodiment, the determining whether the corresponding frame loss is a reference frame according to the frame number of the frame loss comprises:

inquiring whether the frame number of the lost frame exists in a preset reference frame list or not, and if so, judging that the lost frame is a reference frame;

if the frame loss does not exist, judging that the frame loss is a reference frame; the preset reference frame list stores frame numbers of all reference frames.

According to a second aspect of the embodiments of the present disclosure, there is provided a frame loss processing apparatus, including:

the receiving module is used for receiving the coded data, decoding the coded data to obtain at least one frame of original image and storing the at least one frame of original image;

a notification obtaining module, configured to obtain a frame loss notification message, where the frame loss notification message carries identification information indicating that a frame loss occurs;

an obtaining module, configured to obtain a frame loss receiving device identifier corresponding to the frame loss notification message;

and the first sending module is used for coding the reference frame of the stored original image of the last frame according to the identification of the frame loss receiving equipment and sending the coded reference frame to the corresponding frame loss equipment.

In one embodiment, the apparatus further includes a determining module, configured to determine whether a corresponding frame loss is a reference frame according to the frame number of the frame loss;

and the first sending module is specifically used for coding the reference frame of the stored original image of the last frame according to the identifier of the frame loss receiving device and sending the coded reference frame to the corresponding frame loss device if the frame loss is the reference frame.

In one embodiment, the above apparatus further comprises:

and the second sending module is used for sending the received coded data to all receiving devices if the lost frame is a non-reference frame.

In one embodiment, the frame loss notification message further carries a frame number of a frame loss, and the device further comprises a judging module for judging whether the corresponding frame loss is a reference frame according to the frame number of the frame loss;

and the processing module is specifically used for performing frame loss processing according to the at least one frame original image if the frame loss is a reference frame.

In one embodiment, the processing module comprises:

the identification obtaining submodule is used for obtaining the identification of the frame loss receiving equipment corresponding to the frame loss notification message;

and the sending submodule is used for coding the reference frame of the stored original image of the last frame according to the identification of the frame loss receiving equipment and sending the coded reference frame to the corresponding frame loss equipment.

In one embodiment, the receiving module is specifically configured to:

receiving coded data, decoding the coded data to obtain YUV data of the at least one frame of original image, and storing the YUV data of the at least one frame of original image.

In one embodiment, the determining module is specifically configured to:

inquiring whether the frame number of the lost frame exists in a preset reference frame list or not, and if so, judging that the lost frame is a reference frame;

if not, judging the frame loss as a non-reference frame; the preset reference frame list stores frame numbers of all reference frames.

The scheme is applied to a scene of one-to-many transmission of real-time pictures, and mainly comprises the following technical scheme: the R end reports frame loss notification information to the VGR; and the VGR performs frame loss processing according to the frame loss notification information, wherein the specific frame loss processing is to encode the YUV data of the stored last frame image into an I frame and send the I frame to the R end for sending the frame loss. Therefore, the instant bandwidth at the VGR is not suddenly and remarkably increased, the images of the R end with frame loss and other R ends can be normally decoded, and the decoding and playing of subsequent images are not influenced; and the processing overhead of the S end does not need to be increased.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a scene architecture diagram used in the frame loss processing method provided in the embodiment of the present disclosure;

fig. 2 is a flowchart of a frame loss processing method according to an embodiment of the present disclosure;

fig. 3 is a flowchart of a frame loss processing method according to an embodiment of the present disclosure;

fig. 4 is a schematic view of a usage scenario of a frame loss processing method according to an embodiment of the present disclosure;

fig. 5 is a block diagram of a frame loss processing apparatus according to an embodiment of the present disclosure;

fig. 6 is a block diagram of a frame loss processing apparatus according to an embodiment of the present disclosure;

fig. 7 is a block diagram of a frame loss processing apparatus according to an embodiment of the present disclosure.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Interframe predictive coding is a coding method for achieving the purpose of image compression by using the correlation between video image frames, i.e. temporal correlation. In the present invention, we can roughly divide all image frames into: reference frames (I-frames) and non-reference frames (P-frames). The reference frame refers to an image frame which contains complete information and does not need to rely on other image frames for decoding; the reference frame refers to an image frame that has a reference function for at least one image frame, that is, when one image frame is a reference frame, at least one image frame needs to be encoded depending on the image.

It is apparent that the loss of a reference frame will cause the subsequent at least one, multiple, and even all subsequent frames to fail to decode properly.

In real-time video transmission, at least the following common interframe coding methods are used:

the first method comprises the following steps: except the first frame, other frames in the video are coded by referring to the previous frame until the whole video is coded;

and the second method comprises the following steps: dividing a video into groups of pictures (GOPs), wherein each frame in each GOP is formed by the following steps: IPPPPPPPPP, all P frames are encoded with reference to the first I frame; the method has application in H264 coding;

and the third is that: dividing a video into a plurality of GOPs, wherein each frame in each GOP is formed by the following steps: IPPPPPPPPP, each P frame is encoded with reference to its previous frame (I or P possible);

the fourth encoding method: each image frame is compared with a predetermined number (e.g., 16) of previous image frames, and the image with the highest similarity is used as its reference frame.

Except the second mode, most image frames in other coding modes are reference frames, so that frame loss can be directly carried out once frame loss occurs in the video transmission process; in the second mode, the number of the non-reference frames is greater than that of the reference frames, and when the non-reference frames are lost, the subsequent image decoding is not affected, so that whether frame loss processing is performed or not can be determined according to a specific frame loss type in the video transmission process.

Based on the above analysis, the following is a detailed description by two specific examples.