阅读说明：本技术 视频传输方法、装置、飞行器、播放设备及存储介质 (Video transmission method and device, aircraft, playing equipment and storage medium ) 是由 赵亮 周士贞 陈颖 于 2019-05-27 设计创作，主要内容包括：本申请公开了一种视频传输方法、装置、飞行器、播放设备及存储介质,包括:获取当前图像帧,对当前图像帧进行编码后向接收端发送(S110)；若接收到接收端发送的错误反馈信息,根据当前图像帧确定纠错刷新帧(S120)；对纠错刷新帧的部分区域进行帧内编码,将编码后的纠错刷新帧向接收端发送(S130),以使传输错误较快恢复。(The application discloses a video transmission method, a device, an aircraft, a playing device and a storage medium, which comprises the steps of obtaining a current image frame, coding the current image frame and then sending the current image frame to a receiving end (S110); if receiving the error feedback information sent by the receiving end, determining an error correction refresh frame according to the current image frame (S120); the partial region of the error correction refresh frame is intra-coded, and the coded error correction refresh frame is transmitted to the receiving end (S130) so that the transmission error is recovered quickly.)

1. A video sending method is applied to a sending end and is characterized by comprising the following steps:

acquiring a current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end;

if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame;

and carrying out intra-frame coding on a partial area of the error correction refresh frame, and sending the coded error correction refresh frame to the receiving end.

2. The video transmission method according to claim 1, wherein said encoding the current image frame comprises:

judging whether the current image frame is an interval refresh frame or not according to an interval refresh condition;

and if the current image frame is not the interval refresh frame, performing interframe coding on the current image frame.

3. The video sending method according to claim 2, wherein said determining whether the current image frame is an interval refresh frame according to an interval refresh condition comprises:

and if the current image frame is an interval refresh frame, carrying out intra-frame coding on the current image frame.

4. The video sending method according to claim 2, wherein said determining whether the current image frame is an interval refresh frame according to an interval refresh condition comprises:

and if the current image frame is an interval refresh frame, carrying out intra-frame coding on a partial area of the current image frame.

5. The method of claim 4, wherein the intra-coding the partial region of the current image frame if the current image frame is an interval refresh frame comprises:

if the current image frame is an interval refresh frame, intra-frame coding is carried out on partial area of the current image frame, and inter-frame coding is carried out on the rest area.

6. The video transmission method according to any one of claims 2 to 5, wherein said determining whether the current image frame is an interval refresh frame according to an interval refresh condition comprises:

if the serial number of the current image frame is positioned in the refresh range of any refresh cycle, judging that the current image frame is an interval refresh frame, wherein the refresh cycle comprises an adjacent refresh range and a prediction range;

and if the serial number of the current image frame is in the prediction range of any refresh period, judging that the current image frame is not an interval refresh frame.

7. The video transmission method according to any one of claims 2 to 5, wherein said determining whether the current image frame is an interval refresh frame according to an interval refresh condition comprises:

if the serial number of the current image frame is positioned in a refreshing subsequence of any image sequence, judging that the current image frame is an interval refreshing frame, wherein the image sequence comprises an adjacent refreshing subsequence and a prediction subsequence, and the refreshing subsequence comprises at least one image frame at the beginning of the image sequence;

and if the serial number of the current image frame is positioned in a prediction subsequence of any image sequence, judging that the current image frame is not an interval refresh frame.

8. The video transmission method according to any one of claims 1 to 5, wherein, after intra-coding a partial region of the error correction refresh frame and transmitting the encoded error correction refresh frame to the receiving end, the method comprises:

and returning to the step of acquiring the current image frame, encoding the current image frame, and continuously executing the step of sending the encoded current image frame to a receiving end until error feedback information sent by the receiving end is received.

9. The video sending method according to any one of claims 1 to 5, wherein before the obtaining the current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end, the method comprises:

carrying out intra-frame coding on an image frame, and sending the coded image frame to a receiving end;

and performing interframe coding on a plurality of image frames behind the image frame according to the intraframe coded image frame, and sending the coded image frame to a receiving end.

10. The method according to any one of claims 1-5, wherein said determining an error correction refresh frame from the current image frame comprises:

and determining a preset number of error correction refresh frames in the current image frame and/or the image frame after the current image frame.

11. The video transmission method according to claim 10, wherein the determining a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame comprises:

and taking the preset number of image frames starting from the current image frame as error correction refreshing frames.

12. The method according to claim 11, wherein said intra-coding the partial region of the error correction refresh frame and transmitting the coded error correction refresh frame to the receiving end comprises:

taking the current image frame as a first error correction refresh frame, carrying out intra-frame coding on a partial area of the first error correction refresh frame, and sending the coded first error correction refresh frame to the receiving end;

acquiring a next error correction refresh frame;

intra-frame coding is carried out on partial area of the error correction refresh frame, and the coded error correction refresh frame is sent to the receiving end;

and returning to the step of acquiring the next error correction refresh frame, carrying out intra-frame coding on a partial area of the error correction refresh frame, and continuously executing the step of sending the coded error correction refresh frame to the receiving end until the last error correction refresh frame is coded and then sent to the receiving end.

13. The method of claim 12, wherein a union of regions intra-coded by a predetermined number of error correction refresh frames is equal to a range of any error correction refresh frame.

14. The method according to claim 13, wherein said intra-coding the partial area of the error correction refresh frame comprises:

and carrying out intra-frame coding on partial areas of the error correction refreshing frame, and carrying out inter-frame coding on the rest areas.

15. The method according to claim 14, wherein said intra-coding the partial area of the first error correction refresh frame comprises:

and carrying out intra-frame coding on the head area of the first error correction refresh frame.

16. The method according to claim 15, wherein said intra-coding a partial region of the error correction refresh frame and transmitting the encoded error correction refresh frame to the receiving end comprises:

and moving the intra-frame coding region of the previous error correction refresh frame of the error correction refresh frame backward to obtain the intra-frame coding region of the error correction refresh frame, and performing intra-frame coding on the intra-frame coding region of the error correction refresh frame.

17. The video transmission method according to claim 16, wherein the determining a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame comprises:

determining the same number of error correction refresh frames in the current image frame and/or image frames subsequent to the current image frame according to the number of stripes in the current image frame.

18. The method according to claim 17, wherein said intra-coding the header region of the first error correction refresh frame comprises:

intra-coding a first slice of the first error correction refresh frame.

19. The method of claim 18, wherein the moving back the intra-coded region of the error correction refresh frame from the previous error correction refresh frame of the error correction refresh frame to obtain the intra-coded region of the error correction refresh frame, and the intra-coding the intra-coded region of the error correction refresh frame comprises:

and adding one to the strip serial number of the previous error correction refresh frame of the error correction refresh frame for intra-frame coding to obtain the strip serial number of the error correction refresh frame for intra-frame coding, and performing intra-frame coding on the strip corresponding to the strip serial number in the error correction refresh frame.

20. The video transmission method according to claim 14, wherein the error feedback information includes area index information;

the intra-frame encoding the partial area of the first error correction refresh frame includes:

and determining the area of the first error correction refresh frame for intra-frame coding according to the area index information in the acquired error feedback information, and performing intra-frame coding on the area of the first error correction refresh frame for intra-frame coding.

21. The method according to claim 20, wherein said intra-coding the partial area of the error correction refresh frame comprises:

and moving the intra-frame coding region of the previous error correction refresh frame of the error correction refresh frame backward to obtain the intra-frame coding region of the error correction refresh frame, and performing intra-frame coding on the intra-frame coding region of the error correction refresh frame.

22. The method of claim 21, wherein the moving back the intra-coded region of the error correction refresh frame from the previous error correction refresh frame of the error correction refresh frame to obtain the intra-coded region of the error correction refresh frame comprises:

if the area of the previous error correction refresh frame for intra-frame coding is not at the tail of the previous error correction refresh frame, the area of the previous error correction refresh frame for intra-frame coding is moved backwards to obtain the area of the error correction refresh frame for intra-frame coding;

and if the area of the previous error correction refresh frame subjected to intra-frame coding is at the tail of the previous error correction refresh frame, determining the head area of the error correction refresh frame as the area of the error correction refresh frame subjected to intra-frame coding.

23. The video transmission method according to claim 22, wherein the determining a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame comprises:

determining the same number of error correction refresh frames in the current image frame and/or image frames subsequent to the current image frame according to the number of stripes in the current image frame.

24. The method of claim 23, wherein the determining the area of the first error correction refresh frame for intra-frame coding according to the area index information in the obtained error feedback information comprises:

and determining the strip corresponding to the strip serial number in the error feedback information in the first error correction refreshing frame as an intra-frame coding area.

25. The method of claim 24, wherein the moving back the intra-coded region of the previous error correction refresh frame to obtain the intra-coded region of the error correction refresh frame comprises:

and adding one to the strip serial number of the previous error correction refresh frame subjected to intra-frame coding to obtain the strip serial number of the error correction refresh frame subjected to intra-frame coding.

26. The method according to claim 25, wherein said determining the header area of the error correction refresh frame as the area in which the error correction refresh frame is intra-coded comprises:

and determining the first stripe of the error correction refresh frame as the stripe which is subjected to intra-frame coding by the error correction refresh frame.

27. The method of claim 11, wherein the determining an error correction refresh frame according to the current image frame if receiving the error feedback information sent by the receiving end comprises:

and if error feedback information sent by the receiving end is received, determining an error correction sequence according to the current image frame, wherein the error correction sequence comprises a preset number of error correction refresh frames and prediction frames after the preset number of error correction refresh frames.

28. The method of claim 27, wherein said intra-coding the partial region of the error correction refresh frame and transmitting the coded error correction refresh frame to the receiving end comprises:

and if correct feedback information sent by the receiving end when the error correction refresh frame is decoded is received, inter-frame coding is carried out on the prediction frame in the error correction sequence, and the coded prediction frame is sent to the receiving end.

29. The method of claim 28, wherein the inter-coding the predicted frame in the error correction sequence and transmitting the coded predicted frame to the receiving end comprises:

and returning to the step of acquiring the current image frame, encoding the current image frame, and continuously executing the step of sending the encoded current image frame to a receiving end until error feedback information sent by the receiving end is received.

30. The video transmission method according to any one of claims 1 to 5, wherein the error feedback information comprises feedback information of the receiving end in response to a packet loss event or feedback information of the receiving end in response to an integrity check error.

31. The video transmission method according to any one of claims 1 to 5, wherein the current image frame is an image frame acquired by a camera provided on the transmitting end in real time.

32. A video receiving method applied to a receiving end, comprising:

receiving the coded current image frame from a sending end, and decoding the current image frame;

if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refresh frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area;

and decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

33. The video receiving method of claim 32, wherein said decoding the current image frame comprises:

and if the coding format corresponding to the current image frame is interframe coding, decoding the current image frame according to the reference data for decoding.

34. The video receiving method of claim 33, wherein said decoding the current image frame comprises:

if the coding format corresponding to the current image frame is intra-frame coding, performing intra-frame decoding on the current image frame;

and refreshing the reference data according to the result of the intra-frame decoding.

35. The video receiving method of claim 34, wherein said refreshing the reference data according to the result of intra decoding comprises:

and sending correct feedback information to the sending end.

36. The video receiving method of claim 33, wherein said decoding the current image frame comprises:

if the coding format corresponding to the current image frame is intra-frame coding of a partial area, performing intra-frame decoding on the intra-frame coded area of the current image frame to obtain refreshing fragment data corresponding to the current image frame;

updating a portion of the reference data corresponding to the intra-coded region to the refresh segment data.

37. The video receiving method of claim 36, wherein said decoding the current image frame further comprises:

and if the coding format corresponding to the current image frame is intra-frame coding of a partial area, performing inter-frame decoding on the rest areas of the current image frame according to the reference data.

38. The video receiving method according to claim 36, wherein said updating the partial data corresponding to the intra-coded region in the reference data to the refresh segment data comprises:

and if all the data in the reference data are updated once, sending correct feedback information to the sending end.

39. The video receiving method according to any of claims 32-38, wherein said sending error feedback information to said sending end if said receiving said current image frame is erroneous or said decoding said current image frame is erroneous comprises:

and if a packet loss event occurs when the coded current image frame is received, sending error feedback information to the sending end.

40. The video receiving method of claim 39, wherein said receiving the encoded current image frame from the transmitting end, and decoding the current image frame comprises:

receiving image area data obtained by encoding each area of the current image frame from a sending end;

and decoding each image area data of the current image frame.

41. The video receiving method of claim 40, wherein said sending error feedback information to the sending end if a packet loss event occurs while receiving the encoded current image frame comprises:

and if the coded image area data is not correctly received from the sending end, generating error feedback information according to the image area data, and sending the error feedback information to the sending end.

42. The video receiving method according to claim 41, wherein the image area data includes a packet sequence number;

if the encoded image area data is not correctly received from the sending end, generating error feedback information according to the image area data, including:

if the packet sequence number of the image area data is not continuous with the packet sequence number of the previous image area data, calculating area index information corresponding to the image area data according to the packet sequence number of the previous image area data and the data packet sequence number of the first image area data corresponding to the current image frame;

generating error feedback information including the region index information.

43. The video receiving method according to claim 41, wherein the image area data includes area index information;

the generating of the error feedback information according to the image area data includes:

generating error feedback information including the region index information.

44. The video receiving method according to claim 40, wherein the image area data includes encoding format information;

the decoding each image region data of the current image frame includes:

and decoding the image area data according to a decoding mode corresponding to the coding format information.

45. The video receiving method according to any of claims 32-38, wherein said sending error feedback information to said sending end if said receiving said current image frame is erroneous or said decoding said current image frame is erroneous comprises:

and if the received current image frame does not pass the integrity check, sending error feedback information to the sending end, wherein the error feedback information comprises area index information corresponding to the head area of the current image frame.

46. The video receiving method according to any of claims 32-38, wherein said sending error feedback information to said sending end if said receiving said current image frame is erroneous or said decoding said current image frame is erroneous comprises:

and if the current image frame fails to be decoded, sending error feedback information to the sending end.

47. The video receiving method as claimed in any of claims 32 to 38, wherein said receiving an error correction refresh frame intra-coded by said transmitting end in response to a partial region transmitted by said error feedback information comprises:

and receiving a preset number of encoded error correction refresh frames from the sending end, wherein the error correction refresh frames are obtained by carrying out intra-frame encoding on partial areas of the error correction refresh frames determined according to the error feedback information by the sending end.

48. The video receiving method according to claim 47, wherein said decoding said error correction refresh frame, refreshing reference data for decoding according to a decoding result, comprises:

carrying out intra-frame decoding on the intra-frame coded region of the error correction refresh frame to obtain refresh segment data corresponding to the error correction refresh frame;

and refreshing the reference data according to the refresh segment data corresponding to the error correction refresh frame.

49. The video receiving method of claim 48, wherein said intra-decoding the error correction refresh frame intra-coded region comprises:

and carrying out intra-frame decoding on the intra-frame coded area of the error correction refreshing frame, and carrying out inter-frame decoding on the rest areas according to the reference data for decoding.

50. The video receiving method of claim 49, wherein said receiving the error correction refresh frame intra-coded by the transmitting end in response to the partial region sent by the error feedback information comprises:

and receiving encoded image area data from the sending end, wherein the image area data comprises data obtained by encoding a partial area of the error correction refresh frame by the sending end and encoding format information.

51. The video receiving method of claim 50, wherein said intra-decoding the intra-coded region of the error correction refresh frame to obtain refresh segment data corresponding to the error correction refresh frame comprises:

and if the coding format information in the image area data is intra-frame coding, performing intra-frame decoding on the data after the partial area coding to obtain refresh segment data corresponding to the image area data.

52. The video receiving method of claim 51, wherein said refreshing the reference data according to the refresh segment data corresponding to the error correction refresh frame comprises:

updating partial data corresponding to the partial region in the reference data to the refresh segment data.

53. The video receiving method according to claim 52, wherein said image area data further includes area index information of said partial area;

the intra-frame decoding the data after the partial region coding to obtain the refresh segment data corresponding to the partial region includes:

and carrying out intra-frame decoding on the data after the partial region coding to obtain refresh segment data corresponding to the region index information.

54. The video receiving method according to claim 53, wherein said updating partial data corresponding to said partial region in said reference data to said refresh segment data comprises:

updating a portion of the reference data corresponding to the region index information to the refresh segment data.

55. The video receiving method according to claim 48, wherein after the reference data is refreshed according to the refresh segment data corresponding to the preset number of error correction refresh frames, correct feedback information is sent to the sending end.

56. The video receiving method according to claim 48, wherein after the reference data is refreshed according to the refresh segment data corresponding to the preset number of error correction refresh frames, returning to the current image frame after receiving the encoding from the transmitting end, and the step of decoding the current image frame is continuously performed until the current image frame is received with errors or the current image frame is decoded with errors.

57. A video transmission method, comprising:

the sending end encodes the acquired current image frame and sends the encoded current image frame to the receiving end;

the receiving end receives and decodes the current image frame, and if the current image frame is received to be wrong or the current image frame is decoded to be wrong, error feedback information is sent to the sending end;

the sending end determines an error correction refresh frame according to the error feedback information, performs intra-frame coding on partial area of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end;

and the receiving end decodes the error correction refreshing frame and refreshes the reference data for decoding according to the decoding result.

58. The video transmission method according to claim 57, wherein the sending end determines whether the current image frame is an interval refresh frame according to an interval refresh condition;

if the current image frame is not an interval refresh frame, the sending end carries out interframe coding on the current image frame;

and if the coding format corresponding to the current image frame is interframe coding, the receiving end decodes the current image frame according to the reference data for decoding.

59. The video transmission method according to claim 58, wherein if said current image frame is an interval refresh frame, said transmitting end intra-codes said current image frame;

if the coding format corresponding to the current image frame is intra-frame coding, the receiving end performs intra-frame decoding on the current image frame;

and the receiving end refreshes the reference data according to the result of the intra-frame decoding.

60. The video transmission method according to claim 59, wherein said receiving end sends correct feedback information to said transmitting end after refreshing said reference data according to the result of intra-frame decoding.

61. The video transmission method according to claim 58, wherein if said current image frame is an interval refresh frame, said transmitting end intra-codes a partial area of said current image frame;

if the coding format corresponding to the current image frame is intra-frame coding of a partial region, the receiving end performs intra-frame decoding on the intra-frame coded region of the current image frame to obtain refreshing segment data corresponding to the current image frame;

and the receiving end updates partial data corresponding to the intra-coded region in the reference data into the refresh segment data.

62. The video transmission method according to claim 61, wherein if the current image frame is an interval refresh frame, the sending end performs intra-frame coding on a part of the area of the current image frame, and performs inter-frame coding on the rest of the area;

and if the coding format corresponding to the current image frame is intra-frame coding of a partial area, the receiving end performs inter-frame decoding on the rest areas of the current image frame according to the reference data.

63. The video transmission method according to claim 62, wherein if all the data in the reference data are updated, the receiving end sends correct feedback information to the transmitting end.

64. The video transmission method according to any of claims 58 to 63, wherein if the number of the current image frame is within a refresh range of any refresh cycle, the sending end determines that the current image frame is an alternate refresh frame, and the refresh cycle includes an adjacent refresh range and a prediction range;

and if the serial number of the current image frame is in the prediction range of any refresh period, the sending end judges that the current image frame is not an interval refresh frame.

65. The video transmission method according to any of claims 58 to 63, wherein if the number of the current image frame is located in a refresh sub-sequence of any image sequence, the sending end determines that the current image frame is an interval refresh frame, the image sequence comprises an adjacent refresh sub-sequence and a prediction sub-sequence, and the refresh sub-sequence comprises at least one image frame at the beginning of the image sequence;

and if the serial number of the current image frame is positioned in a prediction subsequence of any image sequence, the sending end judges that the current image frame is not an interval refresh frame.

66. The video transmission method according to any one of claims 57 to 63, wherein the sending end performs intra-frame coding on a partial region of the error correction refresh frame, returns to the coding of the obtained current image frame after sending the coded error correction refresh frame to the receiving end, and continues the step of sending the coded current image frame to the receiving end until receiving the error feedback information sent by the receiving end.

67. The video transmission method according to any one of claims 57 to 63, wherein the sending end encodes the acquired current image frame, and before sending the encoded current image frame to the receiving end, the method comprises:

the sending end carries out intra-frame coding on an image frame and sends the coded image frame to a receiving end;

the sending end carries out interframe coding on a plurality of image frames behind the image frames according to the image frames coded in the frames and sends the coded image frames to the receiving end.

68. The video transmission method according to any of claims 57-63, wherein the error feedback information comprises feedback information of the receiver in response to a packet loss event or feedback information of the receiver in response to an integrity check error.

69. The video transmission method according to claim 68, wherein the sending end determines a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame, and performs intra-frame coding on a partial area of each error correction refresh frame;

and the receiving end receives a preset number of encoded error correction refresh frames from the transmitting end.

70. The video transmission method according to claim 69, wherein the sending end performs intra-frame coding on a partial region of the error correction refresh frame, and performs inter-frame coding on the remaining region;

and the receiving end performs intra-frame decoding on the intra-frame coded area of the error correction refreshing frame and performs inter-frame decoding on the rest areas according to the reference data for decoding.

71. The video transmission method according to claim 70, wherein the union of the regions intra-coded by the predetermined number of error correction refresh frames is equal to the range of any error correction refresh frame.

72. The video transmission method according to claim 70, wherein said receiving end performs intra-frame decoding on said intra-frame coded region of said error correction refresh frame to obtain refresh segment data corresponding to said error correction refresh frame;

and the receiving end refreshes the reference data according to the refresh segment data corresponding to the error correction refresh frame.

73. The video transmission method according to claim 69, wherein said transmitting end uses a preset number of image frames starting with said current image frame as an error correction refresh frame.

74. The video transmission method according to claim 69, wherein the sending end determines an error correction refresh frame according to the error feedback information, intra-codes a partial region of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end, comprising:

the sending end takes the current image frame as a first error correction refresh frame, intra-frame coding is carried out on a partial area of the first error correction refresh frame, and the coded first error correction refresh frame is sent to the receiving end;

the sending end acquires a next error correction refresh frame;

the sending end carries out intra-frame coding on a partial area of the error correction refresh frame and sends the coded error correction refresh frame to the receiving end;

and returning to the sending end to obtain the next error correction refresh frame, carrying out intra-frame coding on a partial area of the error correction refresh frame by the sending end, and continuously executing the step of sending the coded error correction refresh frame to the receiving end until the last error correction refresh frame is coded and then sent to the receiving end.

75. The video transmission method according to claim 74, wherein said transmitting end intra-codes a header region of said first error correction refresh frame;

and the sending end moves the area of the previous error correction refresh frame of the error correction refresh frame subjected to intra-frame coding backward to obtain the area of the error correction refresh frame subjected to intra-frame coding, and performs intra-frame coding on the area of the error correction refresh frame subjected to intra-frame coding.

76. The video transmission method according to claim 75, wherein the sending end determines the same number of error correction refresh frames in the current image frame and/or image frames following the current image frame according to the number of stripes in the current image frame;

the sending end performs intra-frame coding on a first stripe of the first error correction refresh frame;

and the sending end adds one to the strip serial number of the intra-frame coding of the previous error correction refresh frame of the error correction refresh frame to obtain the strip serial number of the intra-frame coding of the error correction refresh frame, and performs the intra-frame coding on the strip corresponding to the strip serial number in the error correction refresh frame.

77. The video transmission method according to claim 74, wherein the error feedback information includes area index information;

the sending end determines the area of the first error correction refresh frame for intra-frame coding according to the area index information in the obtained error feedback information, and performs intra-frame coding on the area of the first error correction refresh frame for intra-frame coding;

and the sending end moves the area of the previous error correction refresh frame of the error correction refresh frame subjected to intra-frame coding backward to obtain the area of the error correction refresh frame subjected to intra-frame coding, and performs intra-frame coding on the area of the error correction refresh frame subjected to intra-frame coding.

78. The video transmission method according to claim 77, wherein if the intra-coded region of the previous error correction refresh frame is not at the end of the previous error correction refresh frame, said sender moves back the intra-coded region of the previous error correction refresh frame to obtain the intra-coded region of the error correction refresh frame;

if the area of the previous error correction refresh frame for intra-frame coding is at the tail of the previous error correction refresh frame, the sending end determines the head area of the error correction refresh frame as the area of the error correction refresh frame for intra-frame coding.

79. The video transmission method according to claim 77, wherein the transmitting end determines the same number of error correction refresh frames in the current image frame and/or image frames subsequent to the current image frame according to the number of stripes in the current image frame;

the sending end determines a strip corresponding to a strip serial number in the error feedback information in the first error correction refresh frame as an intra-frame coding area;

if the area of the previous error correction refresh frame for intra-frame coding is not at the tail of the previous error correction refresh frame, the sending end adds one to the strip serial number of the previous error correction refresh frame for intra-frame coding to obtain the strip serial number of the error correction refresh frame for intra-frame coding;

and if the area of the previous error correction refresh frame for intra-frame coding is at the tail of the previous error correction refresh frame, the sending end determines the first stripe of the error correction refresh frame as the stripe of the error correction refresh frame for intra-frame coding.

80. The video transmission method according to any of claims 57 to 63, wherein if a packet loss event occurs when the receiving end receives the encoded current image frame, an error feedback message is sent to the sending end.

81. The video transmission method according to claim 80, wherein said receiving end receives image area data obtained by encoding each area of said current image frame from a transmitting end, and decodes each image area data of said current image frame.

82. The video transmission method according to claim 81, wherein if the receiving end does not correctly receive the encoded image area data from the transmitting end, generating error feedback information according to the image area data, and transmitting the error feedback information to the transmitting end.

83. The video transmission method of claim 82, wherein said image area data comprises a packet sequence number;

if the packet sequence number of the image area data is not continuous with the packet sequence number of the previous image area data, the receiving end calculates area index information corresponding to the image area data according to the packet sequence number of the previous image area data and the data packet sequence number of the first image area data corresponding to the current image frame;

and the receiving end generates error feedback information comprising the area index information.

84. The video transmission method according to claim 82, wherein the image area data includes area index information;

and if the receiving end does not correctly receive the encoded image area data from the transmitting end, generating error feedback information comprising the area index information.

85. The video transmission method according to claim 80, wherein the image area data includes encoding format information;

and the receiving end decodes the image area data according to the decoding mode corresponding to the coding format information.

86. The video transmission method according to any of claims 57 to 63, wherein if a current image frame received by the receiving end fails to pass integrity check, the receiving end sends error feedback information to the sending end, where the error feedback information includes area index information corresponding to a header area of the current image frame.

87. The video transmission method according to any of claims 57-63, wherein if said receiving end fails to decode said current image frame, said receiving end sends error feedback information to said transmitting end.

88. The video transmission method according to claim 69, wherein the receiving end receives a preset number of encoded error correction refresh frames from the transmitting end, comprising:

and the receiving end receives the encoded image area data from the transmitting end, wherein the image area data comprises data obtained by encoding a partial area of the error correction refresh frame by the transmitting end and encoding format information.

89. The video transmission method of claim 88, wherein: if the coding format information in the image area data is intra-frame coding, the receiving end performs intra-frame decoding on the data after the partial area coding to obtain refreshing segment data corresponding to the image area data;

and the receiving end updates partial data corresponding to the partial region in the reference data into the refresh segment data.

90. The video transmission method according to claim 89, wherein: the image area data further includes area index information of the partial area;

the receiving end performs intra-frame decoding on the data after the partial region coding to obtain refreshing segment data corresponding to the region index information;

and the receiving end updates partial data corresponding to the region index information in the reference data into the refresh segment data.

91. The video transmission method according to claim 89, wherein said receiving end sends correct feedback information to said sending end after refreshing said reference data according to the refresh segment data corresponding to said preset number of error correction refresh frames.

92. The video transmission method of claim 91, wherein: if the sending end receives error feedback information sent by the receiving end, the sending end determines an error correction sequence according to the current image frame, wherein the error correction sequence comprises a preset number of error correction refresh frames and prediction frames behind the preset number of error correction refresh frames;

and the sending end performs intra-frame coding on a partial area of the error correction refresh frame, after the coded error correction refresh frame is sent to the receiving end, if correct feedback information sent by the receiving end when the error correction refresh frame is decoded is received, the inter-frame coding is performed on the prediction frame in the error correction sequence, and the coded prediction frame is sent to the receiving end.

93. The video transmission method according to claim 92, wherein: the sending end performs interframe coding on the prediction frame in the error correction sequence, returns the sending end to code the acquired current image frame after the coded prediction frame is sent to the receiving end, sends the coded current image frame to the receiving end, and continues to execute the steps of receiving and decoding the current image frame by the receiving end until the receiving end receives the current image frame or decodes the current image frame to be in error, and sends error feedback information to the sending end.

94. A video transmission apparatus, characterized in that the video transmission apparatus comprises a memory and a processor;

the memory is used for storing a computer program;

the processor is configured to execute the computer program and, when executing the computer program, implement the following steps:

acquiring a current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end;

if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame;

and carrying out intra-frame coding on a partial area of the error correction refresh frame, and sending the coded error correction refresh frame to the receiving end.

95. An aircraft, characterized in that it comprises:

the image acquisition assembly is used for acquiring images;

the video transmitting apparatus according to claim 94, configured to obtain the image captured by the image capturing component, encode the image and transmit the encoded image to the receiving end;

the flying assembly is used for flying.

96. A video receiving apparatus, wherein the video transmitting apparatus comprises a memory and a processor;

the memory is used for storing a computer program;

the processor is configured to execute the computer program and, when executing the computer program, implement the following steps:

receiving the coded current image frame from a sending end, and decoding the current image frame;

if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refresh frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area;

and decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

97. A video playback device, comprising:

the video receiving apparatus of claim 96, configured to receive and decode an image from a transmitting end;

and the display component is used for displaying the data decoded by the video receiving device.

98. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to implement the video transmission method according to any one of claims 1-31; or

Implementing the video receiving method according to any of claims 32-56.

Technical Field

The present application relates to the field of video transmission technologies, and in particular, to a video transmission method and apparatus, an aircraft, a playback device, and a storage medium.

Background

At present, in video transmission with a high real-time requirement, a large number of video frames are inter-coded into predicted frames (P frames), so that the utilization rate of a channel is improved, and the quality of the video can be clearer under the same channel condition.

The encoding and decoding of the predicted frame need to refer to the information of other video frames, in the prior art, when the decoding of the predicted frame of the receiving end fails, the transmitting end re-encodes a complete key frame (I frame) and transmits the complete key frame to the receiving end, and the receiving end decodes the key frame to obtain new reference information so as to enable the decoding of the subsequent predicted frame to be correct; however, since the compression efficiency of the key frame is low, the code rate burst is easy to occur, more time is needed to transmit and receive the key frame, the transmission and reception of the subsequent video frame are affected, and the situations of delay increase and blocking often occur near the key frame; therefore, this method is not very friendly to the wireless transmission physical channel, especially in the scenario that the channel condition is not very good or the channel changes rapidly, the graph transmission delay and the click feeling are obvious.

Although there is a technology that reference information required for decoding is periodically refreshed by a video decoder (GDR), if the reference information is updated in time to ensure correct decoding of a predicted frame, the refresh period needs to be shortened, so that redundant information of coding is increased, the utilization rate of a channel is reduced, and the picture quality is reduced under the same code rate; when the refresh period is increased, the reference information can not be updated in time, and the error recovery speed is slow after the error decoding of the predicted frame.

Disclosure of Invention

Based on this, the application provides a video transmission method, a video transmission device, an aircraft, a playing device and a storage medium, and aims to solve the technical problems that in the existing video transmission process, when a video frame is received and decoded incorrectly, the encoding code rate is easy to burst, delay is increased, and jamming and blocking are easy to occur, so that the transmission of the video frame is influenced.

In a first aspect, the present application provides a video sending method, applied to a sending end, including:

acquiring a current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end;

if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame;

and carrying out intra-frame coding on a partial area of the error correction refresh frame, and sending the coded error correction refresh frame to the receiving end.

In a second aspect, the present application provides a video receiving method, applied to a receiving end, including:

receiving the coded current image frame from a sending end, and decoding the current image frame;

if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refresh frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area;

and decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

In a third aspect, the present application provides a video transmission method, including:

the sending end encodes the acquired current image frame and sends the encoded current image frame to the receiving end;

the receiving end receives and decodes the current image frame, and if the current image frame is received to be wrong or the current image frame is decoded to be wrong, error feedback information is sent to the sending end;

the sending end determines an error correction refresh frame according to the error feedback information, performs intra-frame coding on partial area of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end;

and the receiving end decodes the error correction refreshing frame and refreshes the reference data for decoding according to the decoding result.

In a fourth aspect, the present application provides a video transmitting apparatus comprising a memory and a processor;

the memory is used for storing a computer program;

the processor is configured to execute the computer program and, when executing the computer program, implement the following steps:

acquiring a current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end;

if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame;

and carrying out intra-frame coding on a partial area of the error correction refresh frame, and sending the coded error correction refresh frame to the receiving end.

In a fifth aspect, the present application provides an aircraft comprising:

the image acquisition assembly is used for acquiring images;

the video transmitting device is used for acquiring the image acquired by the image acquisition assembly, and transmitting the encoded image to the receiving end;

the flying assembly is used for flying.

In a sixth aspect, the present application provides a video receiving apparatus comprising a memory and a processor;

the memory is used for storing a computer program;

the processor is configured to execute the computer program and, when executing the computer program, implement the following steps:

receiving the coded current image frame from a sending end, and decoding the current image frame;

if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refresh frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area;

and decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

In a seventh aspect, the present application provides a video playback device, including:

the video receiving apparatus is configured to receive an image from a sending end and perform decoding;

and the display component is used for displaying the data decoded by the video receiving device.

In an eighth aspect, the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to implement the aforementioned video transmission method; or

The video receiving method is realized.

The embodiment of the application provides a video transmission method, a video transmission device, an aircraft, a playing device and a storage medium, when an error occurs in transmission of a certain image frame, intra-frame coding is performed on a partial region of an error correction refresh frame, so that not only can the resource occupation of network resources be reduced, but also delay and jitter increase caused by code rate burst due to intra-frame coding of the complete image frame can be avoided, and reference data required by decoding can be timely provided for a receiving end, so that the transmission error can be quickly recovered.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flowchart of a video transmitting method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a scene in which a transmitting end transmits an image frame to a receiving end;

fig. 3 to fig. 5 are schematic flow charts of other embodiments of the video transmission method of the present application;

fig. 6 and 7 are schematic flowcharts of intra-coding a partial region of the error correction refresh frame in fig. 1;

fig. 8 and 9 are schematic flow charts of other embodiments of the video transmission method according to the present application;

fig. 10 and fig. 11 are schematic flowcharts of intra-coding a partial region of the error correction refresh frame in fig. 1;

FIG. 12 is a flowchart illustrating intra-coding a partial area of the error correction refresh frame in FIG. 1;

FIG. 13 is a diagram illustrating the determination of intra-coded regions based on region index information in error feedback information;

fig. 14 and 15 are schematic flowcharts of intra-coding a partial region of the error correction refresh frame in fig. 1;

fig. 16 and 17 are schematic flow charts of encoding the current image frame in fig. 1;

FIG. 18 is a schematic diagram of intra-coding an interval refresh frame in a refresh period;

fig. 19 is a diagram for intra-coding a partial area of an interval refresh frame in a refresh period;

fig. 20 is a flowchart illustrating another embodiment of a video transmission method according to the present application;

FIG. 21 is a diagram of FIG. 20 in which an image frame is intra-coded and a number of subsequent image frames are inter-coded;

fig. 22 is a schematic flowchart of a video receiving method according to an embodiment of the present application;

fig. 23 and 24 are schematic diagrams of a flow chart of decoding the current image frame in fig. 22;

FIG. 25 is a schematic flow chart of decoding the error correction refresh frame and the refresh reference data in FIG. 22;

fig. 26 is a flowchart illustrating a video transmission method according to an embodiment of the present application;

fig. 27 is a diagram illustrating image frame transmission according to an embodiment of a video transmission method.

Fig. 28 is a schematic block diagram of a video transmitting apparatus according to an embodiment of the present application;

FIG. 29 is a schematic block diagram of an aircraft provided by an embodiment of the present application;

fig. 30 is a schematic block diagram of a video receiving apparatus according to an embodiment of the present application;

fig. 31 is a schematic block diagram of a video playback device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.

Referring to fig. 1, fig. 1 is a schematic flowchart of a video sending method for a sending end according to an embodiment of the present application, where the video sending method may be applied in a terminal device or a server, and is used for sending a video to a receiving end; the terminal device can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device or an unmanned aerial vehicle, and the like, and the server can be an independent server or a server cluster. Further, unmanned aerial vehicle can be for rotor type unmanned aerial vehicle, for example four rotor unmanned aerial vehicle, six rotor unmanned aerial vehicle, eight rotor unmanned aerial vehicle, also can be fixed wing unmanned aerial vehicle, and has camera equipment on this unmanned aerial vehicle.

As shown in fig. 1, the video transmission method of the present embodiment includes steps S110 to S130.

S110, obtaining a current image frame, coding the current image frame, and sending the coded current image frame to a receiving end.

In this embodiment, the transmitting end transmits image frames to the receiving end frame by frame, as shown in fig. 2; illustratively, a camera on a sending end acquires image frames in real time, encodes the acquired image frames frame by frame and sends the encoded image frames to a receiving end.

For example, the current image frame is an image frame obtained by a camera arranged on the sending end in real time, and represents an image frame obtained and to be sent by the sending end at the current moment.

Illustratively, the sending end performs inter-frame (Interframe) coding on the current image frame to obtain a Predictive-coded image frame (P frame); and performing inter-frame prediction coding by referring to the previous image frame in a motion prediction mode.

As shown in fig. 2, the sending end sequentially uses the 2 nd image frame to the (n-1) th image frame as the current image frame, performs inter-frame coding to obtain a P frame, and sends the P frame to the receiving end.

For example, referring to the 1 st image frame, the 2 nd image frame is inter-coded; the 3 rd image frame is inter-coded with reference to the 1 st image frame or the 2 nd image frame.

And S120, if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame.

In this embodiment, the sending end sends the encoded image frames to the receiving end, and if the receiving end receives an error in a certain image frame or decodes an error in an image frame, sends error feedback information to the sending end.

Exemplarily, the receiving end receives and decodes any image frame and then sends successful feedback information to the sending end, and decodes the image frame; the sending end acquires a new image frame after receiving the successful feedback information, encodes the new image frame and sends the encoded new image frame to the receiving end; and if the receiving end receives a certain image frame or a decoded image frame is in error, sending error feedback information to the sending end.

Illustratively, the receiving end sends the successful feedback information and the error feedback information to the sending end through an uplink channel between the receiving end and the sending end. For example, the uplink channel may also be used to transmit flight control instructions to a transmitting end, such as an unmanned aerial vehicle, and various control instructions, such as instructions for taking a picture, recording a video, returning a flight, and the like.

Illustratively, the error feedback information includes feedback information of the receiving end responding to a packet loss event, or includes feedback information of the receiving end responding to an integrity check error.

When a sending end sends a certain image frame to a receiving end, error feedback information sent by the receiving end is received, the sending or decoding of the image frame is failed, and an error correction refreshing frame is determined according to the image frame corresponding to the error feedback information.

S130, intra-frame coding is carried out on partial area of the error correction refresh frame, and the coded error correction refresh frame is sent to the receiving end.

Unlike the prior art that intra-frame (Intraframe) encoding is performed on the entire image frame, in this embodiment, intra-frame encoding is performed on a partial area of the error correction refresh frame, and the encoded error correction refresh frame is sent to the receiving end. As shown in fig. 2, partial areas of the nth to n +2 th image frames are intra-coded.

When the intra-frame coding is carried out, other image frames do not need to be referred to, the information of the image background and the motion body of the image frame or the partial area of the image frame can be completely described, the data volume of the intra-frame coded data is large, and the network resources occupied when the intra-frame coded data is transmitted to a receiving end are large; when decoding the image frame or the image frame part area coded in the frame, the receiving end can decode the part of data without referring to other image frames, and the decoded data is used as reference data required by decoding other image frames.

The video sending method provided by the embodiment performs intra-frame coding on a partial area of an error correction refresh frame when an error occurs in transmission of a certain image frame, so that not only can resource occupation of network resources be reduced, and delay and jitter increase caused by code rate burst due to intra-frame coding on a complete image frame be avoided, but also reference data required by decoding can be provided for a receiving end in time, and transmission errors can be recovered quickly.

In some embodiments, as shown in fig. 3, the intra-coding the partial region of the error correction refresh frame in step S130, and transmitting the encoded error correction refresh frame to the receiving end includes:

s1301, intra-frame coding is carried out on partial area of the error correction refresh frame, inter-frame coding is carried out on the rest area of the error correction refresh frame, and the coded error correction refresh frame is sent to the receiving end.

The data size of the data after interframe coding is smaller, and less network resources are occupied when the data are transmitted to a receiving end; therefore, the data size of the error correction refresh frame after coding is smaller, and the intra-coded area is included, so that the reference data can be refreshed at the receiving end.

In some embodiments, as shown in fig. 4, if the error feedback information sent by the receiving end is received, the determining, in step S120, an error correction refresh frame according to the current image frame includes:

and S121, if error feedback information sent by the receiving end is received, determining a preset number of error correction refresh frames in the current image frame and/or the image frame after the current image frame.

And if the error feedback information sent by the receiving end is received, determining a preset number of error correction refresh frames in the image frame corresponding to the error feedback information, namely the current image frame and/or the image frame after the current image frame.

For example, the sending end acquires the nth image frame in S110, encodes the image frame, and sends the encoded image frame to the receiving end; however, if the error feedback information sent by the receiving end is received in S120, a preset number, such as M error correction refresh frames, may be determined in the nth image frame and/or the image frames following the nth image frame, where n and M are natural numbers other than 0. As shown in fig. 2, M equals 3.

For example, as shown in fig. 5, the step S121 determines a preset number of error correction refresh frames in the current image frame and/or an image frame after the current image frame, including:

and S1201, taking the preset number of image frames starting from the current image frame as error correction refreshing frames.

For example, as shown in fig. 2, M image frames starting from the nth image frame, i.e., the nth image frame through the (n + M-1) th image frame, are determined as M error correction refresh frames. And transmitting the error correction refresh frame to the receiving end by more rapidly responding to the error feedback information of the receiving end.

In some embodiments, as shown in fig. 6, step S130 performs intra-frame encoding on a partial region of the error correction refresh frame, and transmits the encoded error correction refresh frame to the receiving end, including steps S131 to S134.

S131, taking the current image frame as a first error correction refresh frame, carrying out intra-frame coding on a partial area of the first error correction refresh frame, and sending the coded first error correction refresh frame to the receiving end.

In the present embodiment, a predetermined number of image frames starting with a current image frame corresponding to error feedback information are used as error correction refresh frames.

Illustratively, a receiving end sends error feedback information to a sending end when receiving and decoding an nth image frame, and the sending end determines the nth image frame to an (n + M-1) th image frame as M error correction refresh frames; the first error correction refresh frame is the current image frame corresponding to the error feedback information, i.e. the nth image frame.

Illustratively, a partial region of the nth image frame is intra-coded, and the rest regions are inter-coded, and then the coded nth image frame is transmitted to the receiving end.

And S132, acquiring a next error correction refresh frame.

Illustratively, a camera on the unmanned aerial vehicle acquires image frames frame by frame, and the sending end can acquire next image frames in real time while or after a certain image frame is coded and transmitted by the sending end. Because the speed of the camera acquiring the image frames is approximately constant, the time length of the transmission of the image frames by the sending end needs to be limited within a small range, and the video transmission is not blocked. In the embodiment, intra-frame coding is performed on a partial area of an error correction refresh frame, so that the transmission code rate is low and the time is short; the error correction refreshing frame can be transmitted at least before the next image frame is shot, the real-time performance of video transmission is guaranteed, and delay is reduced.

Illustratively, an image frame newly captured by the camera, for example, an n + s th image frame, is acquired as the next error correction refresh frame, where s is smaller than M, and s is a natural number other than 0.

S133, intra-frame coding is carried out on partial area of the error correction refresh frame, and the coded error correction refresh frame is sent to the receiving end.

Illustratively, a partial region of the (n + s) th image frame is intra-coded, and the rest regions are inter-coded, and then the coded (n + s) th image frame is sent to the receiving end.

In some embodiments, different error correction refresh frames corresponding to the same error feedback information have different intra-coded regions. Thus, the receiving end can update different portions of the reference data according to decoding different error correction refresh frames.

S134, returning to the step of acquiring the next error correction refresh frame, performing intra-frame coding on a partial area of the error correction refresh frame, and continuing to execute the step of sending the coded error correction refresh frame to the receiving end until the last error correction refresh frame is sent to the receiving end after being coded.

For example, as shown in fig. 7, the step S134 of returning to obtain the next error correction refresh frame, performing intra-frame coding on a partial region of the error correction refresh frame, and continuing to perform the step of sending the encoded error correction refresh frame to the receiving end until the last error correction refresh frame is sent to the receiving end after being encoded, includes:

s1341, judging whether the last error correction refresh frame is coded and then sending to the receiving end.

Illustratively, if the last error correction refresh frame is not coded and then is sent to the receiving end, returning to the step of obtaining the next error correction refresh frame, and intra-coding a partial area of the error correction refresh frame, and continuing to execute the step of sending the coded error correction refresh frame to the receiving end; if the last error correction refresh frame is already encoded and then sent to the receiving end, the process may return to step S110 to continue execution.

For example, if S is smaller than M-1, the step S132 is returned to obtain the (n + S + 1) th image frame, intra-frame coding is performed on a partial area of the image frame, and the coded image frame is sent to the receiving end; and if s is equal to M-1, the (n + M-1) th image frame, namely the last error correction refresh frame is sent to the receiving end after partial intra-frame coding, and a new error correction refresh frame does not need to be obtained in a return mode.

For example, the partial area intra-frame coding is sequentially performed on the preset number of error correction refresh frames determined in step S121, and the error correction refresh frames after being coded are sent to the receiving end.

In some embodiments, for any error feedback information, the union of the regions in which the corresponding predetermined number of error correction refresh frames are intra-coded is equal to the range of any error correction refresh frame. Performing intra-frame coding on partial areas of a preset number of error correction refresh frames, wherein the intra-frame coding is equivalently performed on all areas of a whole image frame; therefore, the receiving end can completely refresh the reference data once by decoding the preset number of error correction refresh frames, so that the accuracy of decoding the subsequent image frames by the receiving end is improved, and the phenomenon of screen splash of the received video is prevented.

For example, in step S130, intra-frame coding is performed on a partial region of the error correction refresh frame, and after the coded error correction refresh frame is sent to the receiving end, if error feedback information sent by the receiving end is received, it indicates that an error occurs in the error correction refresh frame currently received and decoded by the receiving end; the transmitting end returns to step S120 and step S130 to continue execution, for example, determining an error correction refresh frame according to the error correction refresh frame, that is, the current image frame, and performing a round of determination of the error correction refresh frame and partial area intra-frame coding again.

In some embodiments, as shown in fig. 8, after the step S130 intra-coding the partial area of the error correction refresh frame and transmitting the coded error correction refresh frame to the receiving end, the method includes:

s140, returning to the step of acquiring the current image frame, encoding the current image frame, and continuously executing the step of sending the encoded current image frame to a receiving end until error feedback information sent by the receiving end is received.

For example, after the error correction refresh frames with the preset number determined in step S121 are subjected to intra-coding of the partial area and transmitted to the receiving end, the reference data update of the receiving end is completed, and normal decoding, such as inter-frame decoding, may be performed. The sending end may return to step S110 to obtain, for example, an image frame newly captured by the camera, that is, a current image frame, encode the current image frame, and send the encoded current image frame to the receiving end. If the error feedback information sent by the receiving end is received later, steps S120 to S140 are executed to update the reference data by the receiving end.

In other embodiments, as shown in fig. 9, if the error feedback information sent by the receiving end is received, the determining, in step S120, an error correction refresh frame according to the current image frame includes:

and S122, determining an error correction sequence according to the current image frame if error feedback information sent by the receiving end is received.

Specifically, the error correction sequence includes a preset number of error correction refresh frames and a prediction frame after the preset number of error correction refresh frames.

Illustratively, the receiving end receives an error correction refresh frame which is partially intra-coded by the sending end, and the reference data can be updated by decoding the error correction refresh frame; for the sending end, the error correction refresh frame can be referred to perform interframe coding on a plurality of image frames after the error correction refresh frame; for the receiving end, several image frames following the error correction refresh frame may be inter-decoded based on the reference data.

Illustratively, a receiving end sends error feedback information to a sending end when receiving and decoding an nth image frame, the sending end determines the nth image frame to the (n + M-1) th image frame as M error correction refresh frames, and determines the (n + M) th image frame to the (n + M + p-1) th image frame as p prediction frames; and the error correction refreshing frame and the prediction frame behind the error correction refreshing frame form an error correction sequence corresponding to the error feedback information.

In some embodiments, as shown in fig. 9, the step S140 returns to the acquiring of the current image frame, encodes the current image frame, and continues to perform the step of sending the encoded current image frame to the receiving end until the error feedback information sent by the receiving end is received, where the step S includes:

s150, if correct feedback information sent by the receiving end when the error correction refresh frame is decoded is received, inter-frame coding is carried out on the prediction frame in the error correction sequence, and the coded prediction frame is sent to the receiving end.

Illustratively, step S150 is performed after intra-coding a partial area of the error correction refresh frame in step S130 and transmitting the coded error correction refresh frame to the receiving end.

Illustratively, if the receiving end receives each error correction refresh frame and decodes each error correction refresh frame correctly, the reference data is updated, and correct feedback information can be sent to the sending end. And if the sending end receives the correct feedback information, the inter-frame coding is carried out on the prediction frame in the error correction sequence, and the coded prediction frame is sent to the receiving end.

For example, when inter-coding a certain predicted frame, any predicted frame preceding the predicted frame in the error correction sequence may be referred to. For example, the (n + M-1) th image frame may be referred to when inter-coding the (n + M) th image frame; any one of the nth to n + M + p-2 image frames may be referred to in inter-coding the n + M + p-1 image frame.

For example, in step S130, intra-frame coding is performed on a partial region of the error correction refresh frame, and after the coded error correction refresh frame is sent to the receiving end, if error feedback information sent by the receiving end is received, it indicates that an error occurs in the error correction refresh frame currently received and decoded by the receiving end; the transmitting end returns to step S120 and step S130 to continue execution, for example, determining an error correction refresh frame according to the error correction refresh frame, i.e. the current image frame, and performing once more determination of the error correction refresh frame and partial area intra-frame coding.

In some embodiments, as shown in fig. 10, intra-coding the partial area of the first error correction refresh frame in step S131 includes:

s1311, intra-frame coding the header region of the first error correction refresh frame.

As shown in fig. 2, the transmitting end determines the nth image frame to the (n + M-1) th image frame as M error correction refresh frames according to the error feedback information sent by the receiving end, where the nth image frame is the first error correction refresh frame. The head area of the first error correction refresh frame is used as the area for intra-frame coding, and other areas can be inter-frame coded.

For example, for a certain image frame, the image of the first acquired partial area is the image of the head area, for example, the first several rows of pixels of the image frame. For example, each image frame acquired by the transmitting end has a resolution of 1920 × 1080, and in the horizontal direction, that is, each row has 1920 pixels, and in the vertical direction, that is, each column has 1080 pixels; the first 200 rows, the area where the total of 200 × 1920 pixels are located, is the head area of the image frame.

In this embodiment, the intra-coding the partial area of the error correction refresh frame in step S133 includes:

s1331, moving back the intra-frame coding region of the previous error correction refresh frame of the error correction refresh frame to obtain the intra-frame coding region of the error correction refresh frame, and performing intra-frame coding on the intra-frame coding region of the error correction refresh frame.

For example, as shown in fig. 2, the region of the nth image frame that is intra-coded is moved away from the head region, i.e., moved backward, to obtain the region of the (n + 1) th image frame that is intra-coded; and moving the area of the (n + 1) th image frame subjected to intra-frame coding to a direction far away from the head area, namely moving backwards to obtain the area of the (n + 2) th image frame subjected to intra-frame coding.

In fig. 2, the regions of the error correction refresh frames in the same error correction sequence that are intra-coded do not overlap; for example, the area in which the nth image frame is intra-coded is the pixel area in the 0 th to 200 th rows, the area in which the (n + 1) th image frame is intra-coded is the pixel area in the 201 th to 400 th rows, and the area in which the (n + 2) th image frame is intra-coded is the pixel area in the 401 th to 800 th rows; the regions in which the error correction refresh frames are intra-coded may be equal or unequal.

In another possible embodiment, the regions of the error correction refresh frames in the same error correction sequence that are intra-coded may be overlapped by a portion; for example, the area in which the nth image frame is intra-coded is the pixel area in the 0 th to 300 th rows, the area in which the (n + 1) th image frame is intra-coded is the pixel area in the 201 th to 500 th rows, and the area in which the (n + 2) th image frame is intra-coded is the pixel area in the 401 th to 700 th rows; the regions in which the error correction refresh frames are intra-coded may be equal or unequal.

In this embodiment, the union of the regions in which all the error correction refresh frames of the preset number are intra-coded in the same error correction sequence is equal to the range of any error correction refresh frame. For example, all error correction refresh frames in the same error correction sequence require intra-coding of the respective pixel ranges of row 1 to row 1080. As shown in fig. 2, the union of the regions intra-coded for the nth image frame to the (n + 2) th image frame is equal to the range of one complete image frame. Therefore, the receiving end can completely refresh the reference data once by decoding the preset number of error correction refresh frames, so that the accuracy of decoding the subsequent image frames by the receiving end is improved, and the phenomenon of screen splash of the received video is prevented.

In some embodiments, as shown in fig. 11, the step S121 of determining a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame includes:

s1211, determining the same number of error correction refresh frames in the current image frame and/or the image frames after the current image frame according to the number of the strips in the current image frame.

A Slice (Slice) contains part or all of the data of a frame of a picture, in other words, a frame of a video picture can be encoded as one or several slices. A slice contains at least one Macroblock (Macroblock) and at most data of an entire frame of image. In different coding implementations, the number of slices constructed in the same frame of image is not necessarily the same.

In the present embodiment, each image frame includes at least two slices; the encoding and decoding between different strips in the same image frame are independent; the data referenced by the decoding process of a slice, such as predictive coding, cannot cross the boundary of the slice, preventing the spreading of errors.

Exemplarily, if the number of stripes in an image frame is 3, determining 3 error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame; as shown in fig. 2, the nth image frame to the (n + 2) th image frame are determined as the error correction refresh frame.

Illustratively, the step S1311 intra-coding the header area of the first error correction refresh frame includes:

s1311', intra-coding a first slice of the first error correction refresh frame.

Illustratively, the first slice of the first error correction refresh frame is intra-coded, and the other slices are inter-coded. Carrying out intra-frame coding on the strips to obtain I strips, namely I slices; and performing interframe coding on the strips to obtain P strips, namely Pslice.

Illustratively, as shown in FIG. 2, the first slice of the nth image frame is intra-coded.

Illustratively, only I-type macroblocks, i.e., intra-predicted macroblocks, are contained in I slices, and no P macroblocks, i.e., inter-predicted macroblocks, are contained; in a P slice, I macroblocks may be included in addition to corresponding P macroblocks.

Carrying out intra-frame coding or inter-frame coding on the strips to obtain I strips or P strips; the slice type information slice _ type included in the I slice is a code corresponding to the I slice, and the I slice further includes information of a position of a first macroblock included in the slice in the whole frame, and the like; the slice type information slice _ type included in a P slice is a code corresponding to the P slice, and the P slice further includes information of a position of a first macroblock included in the P slice in the entire frame, and the like.

After receiving the encoded stripe from the sending end, the receiving end judges whether to perform intra-frame decoding or inter-frame decoding according to the stripe type information slice _ type; if the slice type information slice _ type is a code corresponding to an I slice, performing intra-frame decoding, and if the slice type information slice _ type is a code corresponding to a P slice, performing inter-frame decoding; after decoding, the decoded data are spliced into the data of the image frame according to the position information of the first macro block in the whole frame contained in each strip.

For example, in step S1331, moving back an intra-coded region of a previous error correction refresh frame of the error correction refresh frame to obtain an intra-coded region of the error correction refresh frame, and intra-coding the intra-coded region of the error correction refresh frame, the method includes:

s1331', adding one to the strip sequence number of the previous error correction refresh frame of the error correction refresh frame for intra-frame coding to obtain the strip sequence number of the error correction refresh frame for intra-frame coding, and performing intra-frame coding on the strip corresponding to the strip sequence number in the error correction refresh frame.

Exemplarily, as shown in fig. 2, the serial number of the slice in which the nth image frame is intra-coded is increased by one, and the serial number of the slice in which the (n + 1) th image frame is intra-coded is obtained as two; then, carrying out intra-frame coding on a second strip of the (n + 1) th image frame, and carrying out inter-frame coding on a first strip and a third strip of the (n + 1) th image frame; then acquiring an n +2 th image frame, and adding one to the strip serial number of the n +1 th image frame for intra-frame coding to obtain a strip serial number of the n +2 th image frame for intra-frame coding, which is three; the third strip of the (n + 2) th image frame is then intra-coded, and the other two strips of the (n + 2) th image frame are inter-coded.

Each error correction refresh frame in the same error correction sequence carries out intra-frame coding on one strip, and the union of the intra-frame coding areas of all the error correction refresh frames with the preset number is equal to the range of any error correction refresh frame. After the receiving end decodes an error correction refresh frame, updating the reference information of a strip in the image frame; after all the error correction refresh frames with the preset number in the same error correction sequence are decoded, the reference information of each strip in the image frame is updated; the division of the area for intra-frame coding in the error correction refresh frame is combined into the coding unit, namely the strip, so that the calculation amount can be reduced, and the accuracy can be improved.

In other embodiments, the error feedback information received by the transmitting end from the receiving end includes area index information.

Illustratively, when an error occurs when receiving and decoding data of a certain region in a current image frame, a receiving end generates error feedback information according to the region index information of the region and sends the error feedback information to a sending end.

Illustratively, when the sending end sends the encoded current image frame to the receiving end, error feedback information sent by the receiving end is received, and the error feedback information includes area index information corresponding to pixel areas on lines 201 to 400 of the image frame.

As shown in fig. 12, the intra-coding the partial area of the first error correction refresh frame in step S131 includes:

s1312, determining the area of the first error correction refresh frame for intra-frame coding according to the area index information in the acquired error feedback information, and performing intra-frame coding on the area of the first error correction refresh frame for intra-frame coding.

Illustratively, the area in which the first error correction refresh frame is intra-coded is determined to be the pixel area from the 201 th line to the 400 th line of the first error correction refresh frame according to the area index information in the error feedback information.

Illustratively, the area in which the first error correction refresh frame is intra-coded, and the other areas are inter-coded. For example, the pixel regions of the 201 th line to the 400 th line of the nth image frame are intra-coded, and the other regions are inter-coded; and then transmitting the encoded nth image frame to a receiving end.

The receiving end decodes the nth image frame, especially after the area of the nth image frame which is intra-coded is intra-decoded, the reference data of the area can be updated; therefore, the reference data of the error area can be updated and received and decoded more quickly, the quick recovery of the error state is facilitated, and the probability of screen splash is reduced.

For example, as shown in fig. 13, the area in which the nth image frame is intra-coded is determined as the middle area according to the area index information in the error feedback information, then the middle area of the nth image frame is intra-coded, and the head area and the tail area of the nth image frame are inter-coded.

In this embodiment, the intra-coding the partial area of the error correction refresh frame in step S133 includes:

s1332, moving back the intra-frame coding region of the previous error correction refresh frame of the error correction refresh frame to obtain the intra-frame coding region of the error correction refresh frame, and performing intra-frame coding on the intra-frame coding region of the error correction refresh frame.

Exemplarily, the region of the nth image frame subjected to intra-frame coding is moved backward, and the region of the n +1 th image frame subjected to intra-frame coding is obtained as a tail; then, the tail region of the (n + 1) th image frame is intra-coded, and the head region and the middle region of the (n + 1) th image frame are inter-coded.

In some examples, the area intra-coded in the first error correction refresh frame is not at the head, so when the area intra-coded in a certain error correction refresh frame moves backward to the tail, the head area of the image frame may not be intra-coded yet, and the subsequent error correction refresh frame is required to intra-code the partial area.

For example, as shown in fig. 14, the step S1332 of moving back the intra-coded region of the error correction refresh frame before the error correction refresh frame to obtain the intra-coded region of the error correction refresh frame includes:

s320, judging whether the intra-frame coding area of the previous error correction refresh frame of the error correction refresh frame is at the tail of the previous error correction refresh frame.

If the intra-frame coding area of the previous error correction refresh frame is not at the tail of the previous error correction refresh frame, executing:

s321, moving the area of the previous error correction refresh frame subjected to intra-frame coding backward to obtain the area of the error correction refresh frame subjected to intra-frame coding.

For example, as shown in fig. 13, the region in which the nth image frame is intra-coded is a middle region, and if the region is not at the end of the nth image frame, the region in which the nth image frame is intra-coded is moved backward, and the region in which the (n + 1) th image frame is intra-coded is an end.

If the area of the previous error correction refresh frame for intra-frame coding is at the tail of the previous error correction refresh frame, executing:

s322, determining the head area of the error correction refresh frame as the area of the error correction refresh frame for intra-frame coding.

For example, as shown in fig. 13, if the region in which the (n + 1) th image frame is intra-coded is the tail, the region in which the (n + 2) th image frame is intra-coded is the head; then, the head region of the (n + 2) th image frame is intra-coded, and the middle region and the tail region of the (n + 2) th image frame are inter-coded.

Step S1332, moving back the intra-frame coding region of the previous error correction refresh frame of the error correction refresh frame to obtain the intra-frame coding region of the error correction refresh frame, and after partial intra-frame coding is performed on the error correction refresh frame, if the error correction refresh frame is not the last error correction refresh frame in the preset number corresponding to the error feedback information, returning to step S132 to obtain the next error correction refresh frame; if the error correction refresh frame is the last error correction refresh frame in the preset number corresponding to the error feedback information, the sending end may return to step S110 to obtain, for example, an image frame newly captured by the camera, that is, the current image frame, encode the current image frame, and send the encoded current image frame to the receiving end.

Carrying out backward shift of an intra-frame coding region through each error correction refresh frame in an error correction sequence corresponding to the error feedback information, and carrying out intra-frame coding by returning to the head after the intra-frame coding region reaches the tail; after partial areas of all error correction refresh frames in an error correction sequence are subjected to intra-frame coding, all areas in an image frame are subjected to intra-frame coding; after the image frame is sent to the receiving end, the receiving end can completely refresh the reference data corresponding to the whole image frame.

In some embodiments, as shown in fig. 15, the step S121 of determining a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame includes:

and S1212, determining the same number of error correction refresh frames in the current image frame and/or the image frame after the current image frame according to the number of the strips in the current image frame.

In the present embodiment, each image frame includes at least two slices; the encoding and decoding between different strips in the same image frame are independent; the data referenced by the decoding process of a slice, such as predictive coding, cannot cross the boundary of the slice, preventing the spreading of errors.

Exemplarily, if the number of stripes in an image frame is 3, determining 3 error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame; as shown in fig. 13, the nth image frame to the (n + 2) th image frame are determined as the error correction refresh frame.

For example, the determining the area of the first error correction refresh frame for intra-frame coding according to the area index information in the obtained error feedback information in step S1312 includes:

s1312', determining the strip corresponding to the strip number in the error feedback information in the first error correction refresh frame as the intra-coded region.

Illustratively, the sending end encodes each strip in the image frame and sends the encoded strip to the receiving end, and if the receiving end has errors during receiving and decoding a certain strip, error feedback information is generated according to the strip serial number of the strip and sent to the sending end.

For example, if the receiving end receives and decodes the second slice of the current image frame, the error feedback information sent to the sending end includes the slice number of the second slice.

And the sending end determines the area of the first error correction refresh frame for intra-frame coding as a second strip according to the strip sequence number in the error feedback information.

Illustratively, the second slice of the first error correction refresh frame is intra-coded, and the first slice and the third slice are inter-coded.

For example, the step S321 of moving back the intra-coded region of the previous error correction refresh frame to obtain the intra-coded region of the error correction refresh frame includes:

s321', adding one to the strip sequence number of the previous error correction refresh frame for intra-frame coding to obtain the strip sequence number of the error correction refresh frame for intra-frame coding.

For example, the serial number of the strip for intra-frame coding of the nth image frame is added by one to obtain the serial number of the strip for intra-frame coding of the (n + 1) th image frame, which is three; the third slice of the (n + 1) th image frame is then intra-coded, and the first and second slices of the (n + 1) th image frame are inter-coded.

For example, the determining the header area of the error correction refresh frame as the area in which the error correction refresh frame is intra-coded in step S322 includes:

s322', determining the first stripe of the error correction refresh frame as the stripe intra-coded with the error correction refresh frame.

For example, since the serial number of the strip for intra-coding the (n + 1) th image frame is the serial number of the strip corresponding to the tail of the image frame, that is, three, the (n + 2) th image frame is obtained, the serial number of the strip for intra-coding the (n + 1) th image frame is determined as one, then the first strip of the (n + 2) th image frame is intra-coded, and the other two strips of the (n + 2) th image frame are inter-coded.

Each error correction refresh frame in the same error correction sequence carries out intra-frame coding on one strip, and the union of the intra-frame coding areas of all the error correction refresh frames with the preset number is equal to the range of any error correction refresh frame. The first error correction refresh frame firstly carries out intra-frame coding on the area with errors in receiving and decoding, and the receiving end can firstly update the reference information with errors after decoding, so that the transmission errors can be recovered more quickly.

In some embodiments, as shown in fig. 16 and 17, the current image frame is encoded in step S110, including at least steps S111 to S112.

And S111, judging whether the current image frame is an interval refresh frame according to an interval refresh condition.

In this embodiment, when the current image frame is encoded in step S110, inter-frame encoding is performed on some image frames to obtain inter-frame encoded frames, i.e., P frames; some image frames may be intra-coded or partially intra-coded at intervals so that the receiving end updates the reference data in time.

Illustratively, when error feedback information sent by a receiving end is not received, the acquired image frames are encoded according to a refresh cycle.

Specifically, the refresh cycle includes a refresh range and a prediction range adjacent to each other.

Specifically, the image frames in the prediction range in the same refresh period are all located after all the image frames in the refresh range.

In a refreshing period, a sending end firstly carries out intra-frame coding on an image frame in a refreshing range, and a receiving end can update reference data of the image frame after decoding; then the sending end refers to the image frame in the refreshing range or the previous image frame in the predicting range in the refreshing period to carry out interframe coding on the newly acquired image frame; the receiving end can decode the inter-coded image according to the reference data.

For example, the step S111 determines whether the current image frame is an interval refresh frame according to an interval refresh condition, including: if the serial number of the current image frame is positioned in the refreshing range of any refreshing period, judging that the current image frame is an interval refreshing frame; and if the serial number of the current image frame is in the prediction range of any refresh period, judging that the current image frame is not an interval refresh frame.

For example, when error feedback information sent by the receiving end is not received, whether the current image frame is an interval refresh frame is determined according to a group of pictures (GOP) division rule.

A sequence of pictures, i.e. a group of pictures, is a set of consecutive pictures in a coded video stream. Each coded video stream consists of a sequence of consecutive pictures.

Illustratively, the data variation between image frames in the same image sequence is small; for example, the former of two adjacent image frames with larger data change may be used as the last frame in the previous image sequence, and the latter of two adjacent image frames with larger data change may be used as the first frame in the next image sequence.

Illustratively, the image sequence comprises a contiguous refresh sub-sequence and a prediction sub-sequence, the refresh sub-sequence comprising at least one image frame at the beginning of the image sequence.

In particular, the predictor sub-sequence in the same image sequence is located after the refresh sub-sequence.

When encoding the image frames in an image sequence, a sending end firstly performs intra-frame encoding on the image frames in a refreshing subsequence, and a receiving end can update the reference data of the image frames after decoding; then the sending end refers to the image frame in the refreshing subsequence or the previous image frame in the predicting subsequence to carry out interframe coding on the newly acquired image frame; the receiving end can decode the inter-coded image according to the reference data.

For example, the step S111 determines whether the current image frame is an interval refresh frame according to an interval refresh condition, including: if the serial number of the current image frame is positioned in a refreshing subsequence of any image sequence, judging that the current image frame is an interval refreshing frame; and if the serial number of the current image frame is positioned in a prediction subsequence of any image sequence, judging that the current image frame is not an interval refresh frame.

And S112, if the current image frame is not the interval refresh frame, performing interframe coding on the current image frame.

Illustratively, as shown in fig. 18, each 8 image frames is a refresh period, wherein the 1 st image frame in each refresh period is an interval refresh frame, and the 2 nd to 8 th image frames are not interval refresh frames; and performing interframe coding on the acquired 2 nd to 8 th image frames respectively to obtain corresponding interframe coding frames, namely P frames. For example, the 2 nd to 8 th image frames are inter-coded with reference to the 1 st image frame; or inter-coding the 8 th image frame with reference to any one of the 1 st to 7 th image frames.

Illustratively, as shown in fig. 19, each 8 image frames is a refresh period, wherein the 1 st to 3 rd image frames in each refresh period are interval refresh frames, and the 4 th to 8 th image frames are not interval refresh frames; and performing interframe coding on the obtained 4 th to 8 th image frames respectively to obtain corresponding interframe coding frames, namely P frames. For example, the 4 th to 8 th image frames are inter-coded with reference to the 1 st image frame; or inter-coding the 8 th image frame with reference to any one of the 1 st to 7 th image frames.

In some embodiments, as shown in fig. 16, the step S111, after determining whether the current image frame is an interval refresh frame according to an interval refresh condition, includes:

and S113, if the current image frame is an interval refresh frame, carrying out intra-frame coding on the current image frame.

Illustratively, as shown in fig. 18, the 1 st image frame in each refresh period is an interval refresh frame; then the whole frame of the image frame is intra-coded to obtain an intra-coded frame, i.e. an I-frame.

By carrying out intra-frame coding on the interval refresh frames in a refresh period or an image sequence, after receiving and decoding each interval refresh frame, a receiving end can completely update the reference data once, so as to improve the accuracy of decoding each non-interval refresh frame in the refresh period or the image sequence and prevent screen from being lost.

In some embodiments, as shown in fig. 17, the step S111, after determining whether the current image frame is an interval refresh frame according to an interval refresh condition, includes:

and S114, if the current image frame is an interval refresh frame, carrying out intra-frame coding on a partial area of the current image frame.

Illustratively, as shown in fig. 19, each 8 image frames is one refresh period, wherein the 1 st to 3 rd image frames in each refresh period are interval refresh frames; and intra-frame coding is carried out on the partial area of each interval refresh frame.

Illustratively, the regions intra-coded for every interval refresh frame in the same refresh cycle or image sequence are different, and the union of the regions intra-coded for all interval refresh frames in the same refresh cycle or image sequence is equal to the image range of every image frame.

Illustratively, the regions intra-coded in the same refresh cycle or in refresh frames at different intervals in the image sequence may partially coincide or may not coincide with each other.

By carrying out intra-frame coding of partial areas on each interval refresh frame in a refresh period or an image sequence, after a receiving end receives and decodes each interval refresh frame, the reference data can be completely updated once, so that the decoding accuracy of each non-interval refresh frame in the refresh period or the image sequence is improved, and screen blooming is prevented.

For example, if the current image frame is an interval refresh frame, the step S114 performs intra-frame coding on a partial area of the current image frame, including: if the current image frame is an interval refresh frame, intra-frame coding is carried out on partial area of the current image frame, and inter-frame coding is carried out on the rest area.

The intra-frame coding is carried out on the partial area of the interval refresh frame, so that the resource occupation of network resources can be reduced, the delay and jitter increase caused by code rate burst caused by the intra-frame coding of the complete image frame can be avoided, and the reference data required by decoding can be provided for a receiving end at intervals.

In some embodiments, as shown in fig. 20, before the step S110 acquires a current image frame, encodes the current image frame, and transmits the encoded current image frame to a receiving end, the method includes:

s101, carrying out intra-frame coding on an image frame, and sending the coded image frame to a receiving end.

Illustratively, as shown in fig. 21, an image frame a is a first image frame transmitted from a transmitting end to a receiving end at a certain time. The sending end carries out intra-frame coding on the image frame to obtain an I frame, and the coded image frame is sent to the receiving end; so that the receiving end can quickly obtain complete reference data after decoding.

For example, a first image frame acquired by a sending end through a camera after a certain startup is an initial image frame, and the sending end performs intra-frame coding on the image frame.

For example, after the step S101 intra-frame codes an image frame and transmits the coded image frame to the receiving end, the method further includes:

s102, inter-frame coding is carried out on a plurality of image frames behind the image frame according to the intra-frame coded image frame, and the coded image frame is sent to a receiving end.

After the image frame a is subjected to intra-frame coding and sent to a receiving end, the receiving end can acquire complete reference data; the sending end can refer to the image frame a to perform interframe coding on the obtained image frame b to obtain a P frame, and the P frame is sent to the receiving end, and the receiving end decodes the image frame b according to the reference data. The sending end can refer to the image frame a or the image frame b to perform interframe coding on the acquired image frame c to obtain a P frame, and the P frame is sent to the receiving end, and the receiving end decodes the image frame c according to the reference data.

Illustratively, the number of image frames inter-coded in step S102 is equal to the number of image frames in one refresh period minus 1. The sending end can conveniently plan the interval refresh frame and the non-interval refresh frame in the refresh period.

For example, when a partial region of the error correction refresh frame or the interval refresh frame is intra-coded and the rest regions are inter-coded, the region for inter-coding may refer to a corresponding region of the image frame before the error correction refresh frame or the interval refresh frame.

As shown in fig. 13, when a first slice of the n +2 th image frame is intra-coded and the other two slices of the n +2 th image frame are inter-coded, inter-coding may be performed with reference to a second slice and a third slice in the n +1 th image frame, and inter-coding may be performed with reference to a second slice and a third slice in the n th image frame.

Through the mode, the embodiment of the application carries out intra-frame coding on the partial area of the error correction refresh frame when the transmission of a certain image frame has errors, so that the resource occupation of network resources can be reduced, the increase of delay and jitter caused by code rate burst due to the fact that the complete image frame is subjected to intra-frame coding is avoided, and reference data required by decoding can be timely provided for a receiving end, so that the transmission errors can be quickly recovered.

Referring to fig. 22, fig. 22 is a schematic flowchart of a video receiving method for a receiving end according to an embodiment of the present application, where the video receiving method can be applied to a terminal device or a server, and is used for receiving a video from the receiving end; the terminal device may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device, a projector, a display, or the like, and the server may be an independent server or a server cluster, for example, a server providing a live video service. Further, the terminal device may be a virtual reality head-mounted display device, such as VR glasses, for acquiring video data from the drone, or a ground station or the like for acquiring video data from the drone.

As shown in fig. 22, the video transmission method of the present embodiment includes steps S210 to S230.

S210, receiving the coded current image frame from the sending end, and decoding the current image frame.

Illustratively, a sending end acquires a current image frame, encodes the current image frame, and sends the encoded current image frame to a receiving end. For example, the current image frame is an image frame obtained by a camera arranged on the sending end in real time, and represents an image frame obtained and to be sent by the sending end at the current moment.

Exemplarily, the receiving end receives each encoded image frame from the transmitting end in real time, and decodes the received image frame; the decoded video data is then stored and/or played.

S220, if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refreshing frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area.

In this embodiment, if the receiving end receives an error in a certain image frame or an error in a decoded image frame, it sends error feedback information to the sending end.

Exemplarily, the receiving end receives and decodes any image frame and then sends successful feedback information to the sending end, and decodes the image frame; the sending end acquires a new image frame after receiving the successful feedback information, encodes the new image frame and sends the encoded new image frame to the receiving end; and if the receiving end receives a certain image frame or a decoded image frame is in error, sending error feedback information to the sending end.

Illustratively, the receiving end sends the successful feedback information and the error feedback information to the sending end through an uplink channel between the receiving end and the sending end. For example, the uplink channel may also be used to transmit flight control instructions to a transmitting end, such as an unmanned aerial vehicle, and various control instructions, such as instructions for taking a picture, recording a video, returning a flight, and the like.

Illustratively, the error feedback information includes feedback information of the receiving end in a packet loss event, or includes feedback information of the receiving end in an integrity check error.

When a sending end sends a certain image frame to a receiving end, error feedback information sent by the receiving end is received, the sending or decoding of the image frame is failed, and an error correction refreshing frame is determined according to the image frame corresponding to the error feedback information.

Different from the prior art that intra-frame (Intraframe) coding is performed on the whole image frame, in this embodiment, the sending end performs intra-frame coding on a partial region of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end; and the receiving end receives an error correction refresh frame which is subjected to intra-frame coding by the transmitting end in response to the partial region sent by the error feedback information.

As shown in fig. 2, the transmitting end intra-codes partial areas of the nth to (n + 2) th image frames.

And S230, decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

When the receiving end decodes the area of the error correction refresh frame for intra-frame coding, the data can be decoded without referring to other image frames, and the decoded data can be used as reference data required by decoding other image frames, so that the updating of the reference data for decoding is realized.

In the video receiving method provided by this embodiment, when an error occurs in transmission of a certain image frame, a receiving and sending end performs an error correction refresh frame for intra-frame coding on a partial area; the method can reduce the resource occupation of network resources, avoid the delay and jitter increase caused by code rate burst when receiving the whole frame to carry out intra-frame coding, and the receiving end can update the reference data required by decoding in time, so that the transmission error can be recovered quickly.

Illustratively, as shown in fig. 23 and 24, the decoding of the current image frame in step S210 includes steps S2101 and S211.

S2101, analyzing the coding format corresponding to the current image frame.

Illustratively, when the sending end sends the encoded image frame to the receiving end, the sent data includes an encoding format corresponding to the image frame; the receiving end can obtain the coding format corresponding to the current image frame by analyzing the received data.

S211, if the coding format corresponding to the current image frame is interframe coding, decoding the current image frame according to the reference data for decoding.

In some embodiments, the sending end performs inter-frame (Interframe) coding on a current image frame to obtain a Predictive-coded image frame (P frame); the receiving end may decode the inter-coded image frame according to the reference data.

In some embodiments, as shown in fig. 23, decoding the current image frame in step S210 includes steps S2121 to S2122.

S2121, if the coding format corresponding to the current image frame is intra-frame coding, performing intra-frame decoding on the current image frame.

Illustratively, as shown in fig. 18, the 1 st image frame in each refresh period is an interval refresh frame; the sending end carries out intra-frame coding on the whole image frame to obtain an intra-frame coding frame, namely an I frame. And the receiving end receives the image frame and then performs intra-frame decoding on the image frame.

And S2122, refreshing the reference data according to the intra-frame decoding result.

For example, after receiving and decoding an interval refresh frame in an interval refresh period or an image sequence, such as an intra-frame encoded frame, the receiving end may completely update the reference data once, so as to improve the accuracy of decoding each non-interval refresh frame in the refresh period or the image sequence and prevent screen blooming.

Illustratively, as shown in fig. 23, after the step S2122 refreshes the reference data according to the result of intra-decoding, the method includes:

and S2123, sending correct feedback information to the sending end.

For example, as shown in fig. 18, after receiving and decoding the interval refresh frame in the refresh period, the receiving end correctly updates the reference data, and then sends correct feedback information to the transmitting end. And after receiving the correct feedback information, the sending end performs interframe coding on the 2 nd to 8 th image frames in the acquired refreshing period to obtain respective corresponding interframe coding frames, namely P frames, and then sends the P frames to the receiving end.

In other embodiments, as shown in fig. 24, the decoding of the current image frame in step S210 includes steps S2131 to S2132.

S2131, if the coding format corresponding to the current image frame is intra-frame coding of a partial region, intra-frame decoding is carried out on the intra-frame coded region of the current image frame, and refreshing segment data corresponding to the current image frame is obtained.

Illustratively, as shown in fig. 19, each 8 image frames is one refresh period, wherein the 1 st to 3 rd image frames in each refresh period are interval refresh frames; the sending end performs intra-frame coding on partial areas of each interval refresh frame and then sends the partial areas to the receiving end, and the receiving end performs intra-frame decoding on the intra-frame coded areas of the interval refresh frames after receiving the intra-frame coded interval refresh frames of the partial areas.

Illustratively, the regions intra-coded for every interval refresh frame in the same refresh cycle or image sequence are different, and the union of the regions intra-coded for all interval refresh frames in the same refresh cycle or image sequence is equal to the image range of every image frame.

Illustratively, the regions intra-coded in the same refresh cycle or in refresh frames at different intervals in the image sequence may partially coincide or may not coincide with each other.

S2132, updating a part of data in the reference data corresponding to the intra-coded region to the refresh segment data.

For example, when the receiving end decodes the region coded in the current image frame, the data can be decoded without referring to other image frames, and the decoded data can be used as reference data required by decoding other image frames, so as to realize periodic update of the reference data used for decoding.

By carrying out intra-frame coding of partial areas on each interval refresh frame in a refresh period or an image sequence, after a receiving end receives and decodes each interval refresh frame, the reference data can be completely updated once, so that the decoding accuracy of each non-interval refresh frame in the refresh period or the image sequence is improved, and screen blooming is prevented.

Illustratively, as shown in fig. 24, the decoding the current image frame in step S210 further includes:

and S2133, if the coding format corresponding to the current image frame is intra-frame coding of a partial area, inter-frame decoding is performed on the rest areas of the current image frame according to the reference data.

Exemplarily, the sending end performs intra-frame coding on a part of regions of the interval refresh frame, and performs inter-frame coding on the rest of regions; the method can reduce the resource occupation of network resources, avoid the increase of delay and jitter caused by code rate burst due to the intra-frame coding of the complete image frame, and provide reference data required by decoding for a receiving end at intervals.

Illustratively, as shown in fig. 24, after the step S2132 of updating the partial data corresponding to the intra-coded region in the reference data to the refresh segment data, the method includes:

and S2134, if all the data in the reference data are updated, sending correct feedback information to the sending end.

Illustratively, the union of the intra-coded regions of all the alternate refresh frames in the same refresh period or image sequence is equal to the image range of each image frame; as shown in fig. 19, after receiving and decoding three refresh frames at intervals in a refresh cycle, the receiving end may completely update the reference data once, and then send correct feedback information to the sending end.

After receiving the correct feedback information, the sending end can perform interframe coding on the 4 th to 8 th image frames in the acquired refresh period to obtain respective corresponding interframe coding frames, namely, the P frames, and then send the P frames to the receiving end.

Illustratively, when receiving and decoding an image frame encoded in an inter-frame encoding, intra-frame encoding, or intra-frame encoding of a partial region, if a reception error or a decoding error occurs, the receiving end sends error feedback information corresponding to the image frame with the error to the sending end. When the sending end receives the error feedback information corresponding to any image frame, the sending end determines an error correction refresh frame according to the image frame, carries out intra-frame coding on partial area of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end. Therefore, the sending end responds to the error feedback information of the receiving end, determines an error correction refresh frame corresponding to the error feedback information, performs intra-frame coding on a partial area of the error correction refresh frame, and sends the coded error correction refresh frame to the receiving end; and the receiving end receives and decodes the error correction refreshing frame, and refreshes the reference data for decoding according to the decoding result.

In some embodiments, the receiving end receives the encoded current image frame from the transmitting end, and decodes the current image frame, specifically: and the receiving end receives the image area data after encoding each area of the current image frame from the transmitting end and decodes each image area data of the current image frame.

Illustratively, when the sending end codes the image frame, the image frame is divided into a plurality of areas to be coded respectively to obtain image area data, and the image area data are sent to the receiving end respectively; the receiving end receives and decodes the image area data one by one.

For example, a transmitting end divides an image frame into a plurality of slices (slices), and encoding and decoding between different slices in the same image frame are independent. The sending end sends the image area data obtained by coding each strip to the receiving end; the receiving end receives and decodes the image area data one by one.

Illustratively, the image region data includes encoding format information, and the image region data can be decoded according to a decoding mode corresponding to the encoding format information.

For example, the sending end performs intra-frame coding and inter-frame coding on the slice to obtain image area data with coding format information of intra-frame coding and inter-frame coding, that is, i.e., I slice and P slice.

Coding format information included in the I slice, for example, slice type information slice _ type is a code corresponding to the I slice; coding format information included in the P slice, for example, slice type information slice _ type is a code corresponding to the P slice; illustratively, an I slice further includes information (first _ mb _ in _ slice) of a position of a first macroblock included in the slice in the entire frame, and the like, and a P slice further includes information (first _ mb _ in _ slice) of a position of a first macroblock included in the slice in the entire frame, and the like.

Illustratively, the encoding format is an intra-frame encoded current image frame, and the encoding format information of each image area data after encoding is intra-frame encoding; the encoding format is a current image frame of intra-encoding of a partial region, the encoding format information corresponding to image region data of an intra-encoding region is intra-encoding, and the encoding format information of image region data corresponding to other regions is inter-encoding.

For example, after receiving the image area data, the receiving end judges whether the image area data is an I-band or a P-band according to the coding format information in the image area data; if the image area data is the I strip, carrying out intra-frame decoding on the image area data; and if the image area data is the P stripe, performing interframe decoding on the image area data.

In some embodiments, if the current image frame is received with errors or decoded with errors in step S220, sending error feedback information to the sending end, where the sending end includes: and if a packet loss event occurs when the coded current image frame is received, sending error feedback information to the sending end.

Illustratively, the receiving end receives the encoded image frame from the transmitting end, performs protocol header analysis and fault-tolerant processing, and decodes the encoded image frame if the encoded image frame is determined to be correct; if the sequence number of the coding frame in the frame header protocol is discontinuous, the packet loss is indicated, namely, a packet loss event occurs.

For example, the sender has a global coded frame counter (frame _ idx) that records the frame number of a frame-by-frame code. The frame sequence number is sent along with the protocol head, if the frame numbers of the adjacent frames received by the receiving end are discontinuous, which indicates that there is a frame loss in the transmission process, error feedback information is sent to the sending end.

In some embodiments, if the encoded image area data is not correctly received from the sending end, error feedback information is generated according to the image area data, and the error feedback information is sent to the sending end.

Illustratively, the receiving end receives image area data after encoding of each area of the image frame from the sending end one by one, and if the area corresponding to the image area data received at a certain time is discontinuous with the area corresponding to the image area data received at the previous time, it indicates that the image area data in the middle is lost and not received, and then sends error feedback information to the sending end.

Exemplarily, the receiving end receives image area data after encoding of each strip of the image frame from the transmitting end one by one; a sending end strictly sends a data service packet according to image area data of a single strip, and a receiving end can correspondingly obtain a strip index of the image area data according to a service packet number index of a physical layer; if the strip index of the image area data received by the receiving end at a certain time is not continuous with the strip index of the image area data received at the previous time, the loss of the image area data corresponding to a strip is indicated and the image area data is not received, and error feedback information is sent to the sending end. For example, the transport protocol header may not include the slice index corresponding to the currently transmitted code stream.

Illustratively, the image area data received by the receiving end from the transmitting end after each area of the current image frame at the transmitting end is encoded includes a packet sequence number.

In some embodiments, if the packet sequence number of the image area data is not consecutive to the packet sequence number of the previous image area data, calculating area index information corresponding to the image area data according to the packet sequence number of the previous image area data and the packet sequence number of the first image area data corresponding to the current image frame; generating error feedback information including the region index information.

The region index information may indicate a region position in the image frame corresponding to the erroneous image region data. For example, if the packet number of the image area data received by the receiving end at a certain time is 17, but the packet number of the image area data received immediately before is 14, it is determined that the image area data with the packet numbers 15 and 16 are not received; if the data packet sequence number of the first image area data corresponding to the current image frame is 12, subtracting the data packet sequence number 12 of the first image area data corresponding to the current image frame from the packet sequence number 14 of the previous image area data, and then adding one to obtain area index information corresponding to the image area data as 3; or subtracting the data packet serial number 12 of the first image area data corresponding to the current image frame from the packet serial number 15 of the first image area data which is not received, so as to obtain that the area index information corresponding to the image area data is 3; it is indicated that the first image area data which is not received is the image area data corresponding to the band after the third band in the current image frame, that is, the image area data corresponding to the fourth band.

Illustratively, each image area data transmitted from the transmitting end to the receiving end includes a slice index corresponding to the encoded slice. For example, each transmission data packet sent from the sending end to the receiving end includes a slice indexed transmission protocol header information.

For example, if the slice index of the image area data is not consecutive to the slice index of the previous image area data, calculating the area index information corresponding to the image area data according to the slice index of the previous image area data and the slice index of the first image area data corresponding to the current image frame; generating error feedback information including the region index information.

For example, if the stripe index of the image area data received by the receiving end at a certain time is 6, but the stripe index of the image area data received before is 3, it indicates that the image area data with the stripe indexes 4 and 5 are not received; if the data strip index of the first image area data corresponding to the current image frame is 1, the data strip index 1 of the first image area data corresponding to the current image frame may be subtracted from the strip index 3 of the previous image area data, and then one is added to obtain area index information corresponding to the image area data as 3; or subtracting the data strip index 1 of the first image area data corresponding to the current image frame from the strip index 4 of the first image area data which is not received, so as to obtain area index information 3 corresponding to the image area data; it is indicated that the first image area data which is not received is the image area data corresponding to the band after the third band in the current image frame, that is, the image area data corresponding to the fourth band.

Illustratively, calculating area index information corresponding to the image area data, and generating error feedback information including the area index information; the error feedback information received by the transmitting end from the receiving end includes area index information. The sending end can determine the area of the first error correction refresh frame for intra-frame coding according to the area index information in the obtained error feedback information, and intra-frame coding is carried out on the area of the first error correction refresh frame for intra-frame coding; for example, the fourth slice in the first error correction refresh frame is intra-coded.

Illustratively, the receiving end decodes the first error correction refresh frame, and particularly after intra-frame decoding is performed on the area in which the first error correction refresh frame is intra-frame coded, the reference data of the area can be updated; therefore, the reference data of the error area can be updated and received and decoded more quickly, the quick recovery of the error state is facilitated, and the probability of screen splash is reduced.

For example, as shown in fig. 13, the transmitting end determines an area in which the nth image frame is intra-coded as a middle area according to the area index information in the error feedback information, then intra-codes the middle area of the nth image frame, and inter-codes a head area and a tail area of the nth image frame. And after receiving the image frame, the receiving end performs intra-frame decoding on the middle area of the nth image frame and updates data corresponding to the middle area in the reference data.

In other embodiments, the receiving end receives from the transmitting end, and the image area data obtained by encoding each area of the current image frame at the transmitting end includes the area index information.

Illustratively, certain image area data received by the receiving end from the transmitting end includes area index information, such as a slice index 3.

And if the coded image area data is not correctly received from the sending end, generating error feedback information comprising the area index information, and sending the error feedback information to the sending end.

For example, if a receiver receives an error in image area data corresponding to the area index information of the slice index 3, for example, the area index information is obtained through protocol header analysis, but the image data is incomplete or the error rate is too high, error feedback information including the area index information of the slice index 3 is generated, and the error feedback information is transmitted to the transmitter. The transmitting end may perform intra-frame coding on the third stripe in the first error correction refresh frame according to the region index information in the obtained error feedback information.

In some embodiments, if the current image frame is received with errors or decoded with errors in step S220, sending error feedback information to the sending end, where the sending end includes: and if the received current image frame does not pass the integrity check, sending error feedback information to the sending end, wherein the error feedback information comprises area index information corresponding to the head area of the current image frame.

Exemplarily, after the current image frame is received, if the length of the coding frame and the checksum (checksum) check in the parsed frame tail protocol fail to pass, it is determined that the integrity check fails; and sending error feedback information to the sending end.

For example, if the received current image frame does not pass the integrity check, which indicates that the current image frame is received with errors, the error feedback information that may be sent to the sending end includes area index information corresponding to the header area of the current image frame, such as a stripe index 1; and the sending end performs intra-frame coding on the first stripe in the first error correction refresh frame according to the error feedback information with the stripe index of 1.

Illustratively, the stripe index corresponding to the header area of the current image frame is 0, and the sending end performs intra-frame coding on the first stripe in the first error correction refresh frame according to the error feedback information with the stripe index of 0.

In some embodiments, if the current image frame is received with errors or decoded with errors in step S220, sending error feedback information to the sending end, where the sending end includes: and if the current image frame fails to be decoded, sending error feedback information to the sending end.

Illustratively, if a code word that does not exist in a code table is encountered when decoding the current image frame or image area data of the current image frame, or it is determined that a decoded value is not reasonable according to a value range of a parameter (parameter) specified by a syntax (syntax), it is determined that decoding the current image frame fails, and error feedback information is sent to the sending end.

For example, if decoding of certain image area data of the current image frame fails, error feedback information including the image area data area index information is sent to the sending end.

In some embodiments, the receiving of the error correction refresh frame intra-coded by the sender in response to the partial region sent by the error feedback information in step S220 includes: and receiving a preset number of encoded error correction refresh frames from the sending end, wherein the error correction refresh frames are obtained by carrying out intra-frame encoding on partial areas of the error correction refresh frames determined according to the error feedback information by the sending end.

In this embodiment, if the sending end receives the error feedback information sent by the receiving end, a preset number of error correction refresh frames are determined in an image frame corresponding to the error feedback information, that is, a current image frame and/or an image frame subsequent to the current image frame.

For example, after receiving error feedback information sent by a receiving end, a sending end takes a preset number of image frames starting with an image frame corresponding to the error feedback information as an error correction refresh frame.

For example, as shown in fig. 2, the transmitting end determines M image frames starting from the nth image frame, i.e., the nth to n + M-1 image frames, as M error correction refresh frames. And transmitting the error correction refresh frame to the receiving end by more rapidly responding to the error feedback information of the receiving end.

The step of the sending end performing intra-frame coding on the partial area of the error correction refresh frame determined according to the error feedback information has been described in the video sending method applied to the sending end, and is not described herein again.

For example, the area in which the receiving end intra-codes the first error correction refresh frame received from the transmitting end may be a header area, such as a first slice.

For example, the area in which the first error correction refresh frame received by the receiving end from the transmitting end is intra-coded may also be an area determined according to area index information in the error feedback information, such as a middle area or a fourth stripe.

In some embodiments, as shown in fig. 25, the receiving end decodes the error correction refresh frame in step S230, and refreshes the reference data for decoding according to the decoding result, including steps S231 to S232.

S231, carrying out intra-frame decoding on the intra-frame coded region of the error correction refresh frame to obtain refresh segment data corresponding to the error correction refresh frame.

And after receiving the error correction refresh frame from the sending end, the receiving end performs intra-frame decoding on the intra-coded region to obtain refresh segment data corresponding to the region.

Illustratively, when a receiving end receives an error correction refresh frame in which the transmitting end performs intra-frame coding on a partial region sent in response to the error feedback information, the receiving end receives coded image region data from the transmitting end, where the image region data includes data obtained by coding the partial region of the error correction refresh frame by the transmitting end and coding format information.

In the present embodiment, the coding format information of the image area data obtained by intra-coding a partial area in the error correction refresh frame is intra-coding, and the coding format information of the image area data obtained by inter-coding the remaining area is inter-coding.

Illustratively, if the encoding format information in the image area data received by the receiving end at a certain time is intra-frame encoding, intra-frame decoding is performed on the data after the partial area encoding in the image area data, so as to obtain refresh segment data corresponding to the image area data.

In some embodiments, the image area data further includes area index information of the partial area. Illustratively, the receiving end performs intra-frame decoding on the data after the partial region is encoded, so as to obtain refresh segment data corresponding to the region index information.

Illustratively, the sending end performs intra-frame coding on a third strip of a certain error correction refresh frame, performs inter-frame coding on the first strip and the second strip to obtain three image area data, wherein coding format information of the image area data corresponding to the third strip is intra-frame coding; and after receiving the image area data corresponding to the third stripe, the receiving end performs intra-frame decoding on the image area data to obtain refresh segment data corresponding to the third stripe.

Illustratively, the error correction refresh frame intra-coded region is intra-decoded, and the remaining regions are inter-decoded according to reference data for decoding. For example, after receiving the image region data corresponding to the first and second slices, the receiving end performs inter-frame decoding according to the data corresponding to the first and second slices in the reference data.

And S232, refreshing the reference data according to the refresh segment data corresponding to the error correction refresh frame.

Because only a partial region of each refresh frame is subjected to intra-frame coding, the receiving end decodes the intra-coded partial region to obtain partial reference data corresponding to the corresponding region, namely refresh segment data, and the partial data corresponding to the region in the reference data is updated according to the refresh segment data.

Illustratively, the partial data corresponding to the partial region in the reference data is updated as the refresh segment data.

In some embodiments, the image area data further includes area index information of the partial area. Illustratively, the receiving end updates a part of the data corresponding to the region index information in the reference data to the refresh segment data.

Illustratively, after receiving image area data corresponding to a third stripe, a receiving end performs intra-frame decoding on the image area data to obtain refresh segment data corresponding to the third stripe; updating a portion of the reference data corresponding to a third stripe to the refresh segment data.

In some embodiments, after the reference data is refreshed according to the refresh segment data corresponding to the preset number of error correction refresh frames, correct feedback information is sent to the sending end.

Illustratively, the union of the intra-frame coding regions of a preset number of error correction refresh frames corresponding to the same error feedback information is equal to the image range of each image frame; as shown in fig. 2, after receiving and decoding the three error correction refresh frames, the receiving end can completely update the reference data once, and then send correct feedback information to the sending end. And after the sending end receives the correct feedback information, the sending end can continue to encode the subsequent image frames.

Illustratively, the sending end determines an error correction sequence according to the error feedback information sent by the receiving end, where the error correction sequence includes a preset number of error correction refresh frames and a number of non-error correction refresh frames, such as prediction frames. After refreshing the reference data according to the refresh segment data corresponding to the error correction refresh frames with the preset number, the receiving end sends correct feedback information to the sending end; and after receiving the correct feedback information, the sending end performs interframe coding on the prediction frame in the error correction sequence and sends the coded prediction frame to the receiving end.

Illustratively, if an error occurs in an error correction refresh frame currently received and decoded by a receiving end, sending error feedback information to a sending end; and the sending end determines a preset number of error correction refresh frames according to the error correction refresh frames, and determines the error correction refresh frames and performs intra-frame coding of partial areas once again.

In some embodiments, after the receiving end refreshes the reference data according to the refresh segment data corresponding to the preset number of error correction refresh frames, returning to step S210 to continue execution; namely, the coded current image frame is continuously received from the sending end, and the step of decoding the current image frame is continuously executed until the current image frame is received or decoded wrongly.

Illustratively, after refreshing the reference data according to the refresh segment data corresponding to the preset number of error correction refresh frames, the receiving end sends correct feedback information to the sending end; after receiving the correct feedback information, the sending end acquires, for example, an image frame newly shot by a camera, that is, a current image frame, encodes the current image frame, and sends the encoded current image frame to the receiving end; and the receiving end receives the coded current image frame from the transmitting end and decodes the current image frame.

Referring to fig. 26, fig. 26 is a flowchart illustrating a video transmission method according to an embodiment of the present application, for transmitting a video from the transmitting end to the receiving end.

As shown in fig. 26, the video transmission method of the present embodiment includes steps S510 to S540.

And S510, the sending end encodes the acquired current image frame and sends the encoded current image frame to the receiving end.

In this embodiment, the transmitting end transmits image frames to the receiving end frame by frame, as shown in fig. 2; illustratively, a camera on a sending end acquires image frames in real time, encodes the acquired image frames frame by frame and sends the encoded image frames to a receiving end.

Illustratively, the sending end judges whether the current image frame is an interval refresh frame according to an interval refresh condition; and if the current image frame is not the interval refresh frame, the sending end carries out interframe coding on the current image frame.

For example, if the number of the current image frame is located in a refresh range of any refresh cycle, the sending end determines that the current image frame is an interval refresh frame, and the refresh cycle includes an adjacent refresh range and a prediction range; and if the serial number of the current image frame is in the prediction range of any refresh period, the sending end judges that the current image frame is not an interval refresh frame.

For example, if the number of the current image frame is located in a refresh subsequence of any image sequence, the sending end determines that the current image frame is an interval refresh frame, where the image sequence includes an adjacent refresh subsequence and a prediction subsequence, and the refresh subsequence includes at least one image frame at the beginning of the image sequence; and if the serial number of the current image frame is positioned in a prediction subsequence of any image sequence, the sending end judges that the current image frame is not an interval refresh frame.

In some embodiments, if the current image frame is an interval refresh frame, the sending end intra-codes the current image frame.

Illustratively, as shown in fig. 18, each 8 image frames is a refresh period, wherein the 1 st image frame in each refresh period is an interval refresh frame, and the 2 nd to 8 th image frames are not interval refresh frames; and performing interframe coding on the acquired 2 nd to 8 th image frames respectively to obtain corresponding interframe coding frames, namely P frames. By carrying out intra-frame coding on the interval refresh frames in a refresh period or an image sequence, after receiving and decoding each interval refresh frame, a receiving end can completely update the reference data once, so as to improve the accuracy of decoding each non-interval refresh frame in the refresh period or the image sequence and prevent screen from being lost.

In some other embodiments, if the current image frame is an interval refresh frame, the sending end intra-codes a partial region of the current image frame.

For example, if the current image frame is an interval refresh frame, the sending end performs intra-frame coding on a part of the area of the current image frame, and performs inter-frame coding on the rest of the area.

Illustratively, as shown in fig. 19, each 8 image frames is a refresh period, wherein the 1 st to 3 rd image frames in each refresh period are interval refresh frames, and the 4 th to 8 th image frames are not interval refresh frames; and performing interframe coding on the obtained 4 th to 8 th image frames respectively to obtain corresponding interframe coding frames, namely P frames.

Illustratively, the regions intra-coded for every interval refresh frame in the same refresh cycle or image sequence are different, and the union of the regions intra-coded for all interval refresh frames in the same refresh cycle or image sequence is equal to the image range of every image frame.

Illustratively, the regions intra-coded in the same refresh cycle or in refresh frames at different intervals in the image sequence may partially coincide or may not coincide with each other.

By carrying out intra-frame coding of partial areas on each interval refresh frame in a refresh period or an image sequence, after a receiving end receives and decodes each interval refresh frame, the reference data can be completely updated once, so that the decoding accuracy of each non-interval refresh frame in the refresh period or the image sequence is improved, and screen blooming is prevented.

In some embodiments, in step S510, a sending end encodes an acquired current image frame, and before sending the encoded current image frame to a receiving end, the sending end performs intra-frame encoding on an image frame and sends the encoded image frame to the receiving end; the sending end carries out interframe coding on a plurality of image frames behind the image frames according to the image frames coded in the frames and sends the coded image frames to the receiving end.

Illustratively, as shown in fig. 21, an image frame a is a first image frame transmitted from a transmitting end to a receiving end at a certain time. The sending end carries out intra-frame coding on the image frame to obtain an I frame, and the coded image frame is sent to the receiving end; so that the receiving end can quickly obtain complete reference data after decoding. After the image frame a is subjected to intra-frame coding and sent to a receiving end, the receiving end can acquire complete reference data; the sending end can refer to the image frame a to perform interframe coding on the obtained image frame b to obtain a P frame, and the P frame is sent to the receiving end, and the receiving end decodes the image frame b according to the reference data. The sending end can refer to the image frame a or the image frame b to perform interframe coding on the acquired image frame c to obtain a P frame, and the P frame is sent to the receiving end, and the receiving end decodes the image frame c according to the reference data.

S520, the receiving end receives and decodes the current image frame, and if the current image frame is received or decoded wrongly, error feedback information is sent to the sending end.

And the receiving end receives and decodes the current image frame sent by the sending end. For example, if the encoding format corresponding to the current image frame is inter-frame encoding, the receiving end decodes the current image frame according to the reference data for decoding.

In some embodiments, if the encoding format corresponding to the current image frame is intra-frame encoding, the receiving end performs intra-frame decoding on the current image frame; and the receiving end refreshes the reference data according to the result of the intra-frame decoding.

For example, after receiving and decoding the refresh period or the inter-refresh frame in the image sequence, such as an intra-coded frame, the receiving end may completely update the reference data once, so as to improve the accuracy of decoding each non-inter-refresh frame in the refresh period or the image sequence and prevent screen blooming.

Illustratively, after the receiving end refreshes the reference data according to the result of intra-frame decoding, the receiving end sends correct feedback information to the sending end.

In other embodiments, if the coding format corresponding to the current image frame is intra-frame coding of a partial region, the receiving end performs intra-frame decoding on the intra-frame coded region of the current image frame to obtain refresh segment data corresponding to the current image frame; and the receiving end updates partial data corresponding to the intra-coded region in the reference data into the refresh segment data.

For example, if the coding format corresponding to the current image frame is intra-frame coding of a partial region, the receiving end performs inter-frame decoding on the rest regions of the current image frame according to the reference data.

For example, when the receiving end decodes the region coded in the current image frame, the data can be decoded without referring to other image frames, and the decoded data can be used as reference data required by decoding other image frames, so as to realize periodic update of the reference data used for decoding.

Illustratively, if all the data in the reference data are updated, the receiving end sends correct feedback information to the sending end.

As shown in fig. 19, after receiving and decoding three refresh frames at intervals in a refresh cycle, the receiving end may completely update the reference data once, and then send correct feedback information to the sending end.

The sending end carries out intra-frame coding on partial areas of the interval refresh frame, and carries out inter-frame coding on the rest areas; the method can reduce the resource occupation of network resources, avoid the increase of delay and jitter caused by code rate burst due to the intra-frame coding of the complete image frame, and provide reference data required by decoding for a receiving end at intervals.

In some embodiments, if the receiving end determines that a packet loss event occurs or integrity check is wrong when receiving an image frame, sending error feedback information to the sending end; the error feedback information includes feedback information of the receiving end responding to a packet loss event, or includes feedback information of the receiving end responding to an integrity check error.

Illustratively, the receiving end sends the successful feedback information and the error feedback information to the sending end through an uplink channel between the receiving end and the sending end.

In some embodiments, if a packet loss event occurs when the receiving end receives the encoded current image frame, the receiving end sends error feedback information to the sending end.

Illustratively, the receiving end receives the encoded image frame from the transmitting end, performs protocol header analysis and fault-tolerant processing, and decodes the encoded image frame if the encoded image frame is determined to be correct; if the sequence number of the coding frame in the frame header protocol is discontinuous, the packet loss is indicated, namely, a packet loss event occurs.

For example, the receiving end receives image area data obtained by encoding each area of the current image frame from a transmitting end, and decodes each image area data of the current image frame.

Specifically, the image area data includes encoding format information. And the receiving end decodes the image area data according to the decoding mode corresponding to the coding format information.

Illustratively, if the receiving end does not correctly receive the encoded image area data from the transmitting end, error feedback information is generated according to the image area data, and the error feedback information is transmitted to the transmitting end.

Illustratively, the receiving end receives image area data after encoding of each area of the image frame from the sending end one by one, and if the area corresponding to the image area data received at a certain time is discontinuous with the area corresponding to the image area data received at the previous time, it indicates that the image area data in the middle is lost and not received, and then sends error feedback information to the sending end.

Illustratively, the image area data includes a packet sequence number.

Specifically, if the packet sequence number of the image area data is not consecutive to the packet sequence number of the previous image area data, the receiving end calculates area index information corresponding to the image area data according to the packet sequence number of the previous image area data and the packet sequence number of the first image area data corresponding to the current image frame; and the receiving end generates error feedback information comprising the area index information.

Illustratively, the image area data includes area index information.

Specifically, if the receiving end does not correctly receive the encoded image region data from the transmitting end, error feedback information including the region index information is generated.

In other embodiments, if the current image frame received by the receiving end fails to pass the integrity check, the receiving end sends error feedback information to the sending end, where the error feedback information includes area index information corresponding to a header area of the current image frame.

Illustratively, the stripe index corresponding to the header area of the current image frame is 0, and the sending end performs intra-frame coding on the first stripe in the first error correction refresh frame according to the error feedback information with the stripe index of 0.

In another embodiment, if the receiving end fails to decode the current image frame, the receiving end sends error feedback information to the sending end.

For example, if decoding of certain image area data of the current image frame fails, error feedback information including the image area data area index information is sent to the sending end.

S530, the sending end determines an error correction refresh frame according to the error feedback information, intra-frame coding is carried out on partial area of the error correction refresh frame, and the coded error correction refresh frame is sent to the receiving end.

For example, the sending end performs intra-frame coding on a partial region of the error correction refresh frame, and after sending the coded error correction refresh frame to the receiving end, returns to step S510 to code the obtained current image frame, and continues to perform the step of sending the coded current image frame to the receiving end until receiving the error feedback information sent by the receiving end.

In some embodiments, the sending end determines a preset number of error correction refresh frames in the current image frame and/or an image frame subsequent to the current image frame, and performs intra-frame coding on a partial area of each error correction refresh frame. For example, the sending end uses a preset number of image frames starting with the current image frame as error correction refresh frames.

As shown in fig. 2, M image frames starting from the nth image frame, i.e., the nth image frame through the (n + M-1) th image frame, are determined as M error correction refresh frames. And transmitting the error correction refresh frame to the receiving end by more rapidly responding to the error feedback information of the receiving end.

Illustratively, the sending end performs intra-frame coding on a partial region of the error correction refresh frame, and performs inter-frame coding on the rest regions.

Illustratively, the union of the regions intra-coded by the preset number of error correction refresh frames is equal to the range of any error correction refresh frame.

In some embodiments, the sending end uses the current image frame as a first error correction refresh frame, performs intra-frame coding on a partial area of the first error correction refresh frame, and sends the coded first error correction refresh frame to the receiving end; then the sending end acquires the next error correction refresh frame; then the sending end carries out intra-frame coding on partial area of the error correction refresh frame and sends the coded error correction refresh frame to the receiving end; and then returning to the sending end to obtain the next error correction refresh frame, carrying out intra-frame coding on a partial area of the error correction refresh frame by the sending end, and continuously executing the step of sending the coded error correction refresh frame to the receiving end until the last error correction refresh frame is coded and then sent to the receiving end.

In some embodiments, the sending end intra-codes a header region of the first error correction refresh frame; and the sending end moves the area of the previous error correction refresh frame of the error correction refresh frame subjected to intra-frame coding backward to obtain the area of the error correction refresh frame subjected to intra-frame coding, and performs intra-frame coding on the area of the error correction refresh frame subjected to intra-frame coding.

Illustratively, the sending end determines the same number of error correction refresh frames in the current image frame and/or image frames after the current image frame according to the number of stripes in the current image frame; the sending end performs intra-frame coding on a first stripe of the first error correction refresh frame; and the sending end adds one to the strip serial number of the intra-frame coding of the previous error correction refresh frame of the error correction refresh frame to obtain the strip serial number of the intra-frame coding of the error correction refresh frame, and performs the intra-frame coding on the strip corresponding to the strip serial number in the error correction refresh frame.

Performing intra-frame coding on partial areas of a preset number of error correction refresh frames, wherein the intra-frame coding is equivalently performed on all areas of a whole image frame; therefore, the receiving end can completely refresh the reference data once by decoding the preset number of error correction refresh frames, so that the accuracy of decoding the subsequent image frames by the receiving end is improved, and the phenomenon of screen splash of the received video is prevented.

In other embodiments, the error feedback information includes region index information.

Illustratively, when an error occurs when receiving and decoding data of a certain region in a current image frame, a receiving end generates error feedback information according to the region index information of the region and sends the error feedback information to a sending end.

The sending end determines the area of the first error correction refresh frame for intra-frame coding according to the area index information in the obtained error feedback information, and performs intra-frame coding on the area of the first error correction refresh frame for intra-frame coding; and the sending end moves the area of the previous error correction refresh frame of the error correction refresh frame subjected to intra-frame coding backward to obtain the area of the error correction refresh frame subjected to intra-frame coding, and performs intra-frame coding on the area of the error correction refresh frame subjected to intra-frame coding.

Specifically, if the intra-frame coding region of the previous error correction refresh frame is not at the tail of the previous error correction refresh frame, the sending end moves the intra-frame coding region of the previous error correction refresh frame backward to obtain the intra-frame coding region of the error correction refresh frame; if the area of the previous error correction refresh frame for intra-frame coding is at the tail of the previous error correction refresh frame, the sending end determines the head area of the error correction refresh frame as the area of the error correction refresh frame for intra-frame coding.

Illustratively, the sending end determines the same number of error correction refresh frames in the current image frame and/or image frames after the current image frame according to the number of stripes in the current image frame; the sending end determines a strip corresponding to a strip serial number in the error feedback information in the first error correction refresh frame as an intra-frame coding area; if the area of the previous error correction refresh frame for intra-frame coding is not at the tail of the previous error correction refresh frame, the sending end adds one to the strip serial number of the previous error correction refresh frame for intra-frame coding to obtain the strip serial number of the error correction refresh frame for intra-frame coding; and if the area of the previous error correction refresh frame for intra-frame coding is at the tail of the previous error correction refresh frame, the sending end determines the first stripe of the error correction refresh frame as the stripe of the error correction refresh frame for intra-frame coding.

Illustratively, the area in which the first error correction refresh frame is intra-coded is determined to be the pixel area from the 201 th line to the 400 th line of the first error correction refresh frame according to the area index information in the error feedback information.

For example, the pixel regions of the 201 th line to the 400 th line of the nth image frame are intra-coded, and the other regions are inter-coded; and then transmitting the encoded nth image frame to a receiving end.

The receiving end decodes the nth image frame, especially after the area of the nth image frame which is intra-coded is intra-decoded, the reference data of the area can be updated; therefore, the reference data of the error area can be updated and received and decoded more quickly, the quick recovery of the error state is facilitated, and the probability of screen splash is reduced.

And S540, the receiving end decodes the error correction refresh frame and refreshes the reference data for decoding according to the decoding result.

In some embodiments, the receiving end receives a preset number of encoded error correction refresh frames from the transmitting end.

Illustratively, the receiving end performs intra-frame decoding on the region intra-coded in the error correction refresh frame, and performs inter-frame decoding on the rest of the region according to the reference data for decoding.

Exemplarily, the receiving end performs intra-frame decoding on the intra-frame coded region of the error correction refresh frame to obtain refresh segment data corresponding to the error correction refresh frame; and the receiving end refreshes the reference data according to the refresh segment data corresponding to the error correction refresh frame.

Illustratively, when the receiving end decodes the area in which the error correction refresh frame is intra-coded, the data can be decoded without referring to other image frames, and the decoded data can be used as reference data required by decoding other image frames, so as to update the reference data used for decoding.

In some embodiments, when the receiving end receives a preset number of encoded error correction refresh frames from the transmitting end, the receiving end receives encoded image area data from the transmitting end, where the image area data includes data obtained by encoding a partial area of the error correction refresh frames by the transmitting end and encoding format information.

Illustratively, if the coding format information in the image region data is intra-frame coding, the receiving end performs intra-frame decoding on the data after the partial region coding to obtain refresh segment data corresponding to the image region data; and the receiving end updates partial data corresponding to the partial region in the reference data into the refresh segment data.

Illustratively, the image area data further includes area index information of the partial area.

Specifically, the receiving end performs intra-frame decoding on the data after the partial region coding to obtain refresh segment data corresponding to the region index information; and then the receiving end updates part of the data corresponding to the region index information in the reference data to the refresh segment data.

In some embodiments, after the reference data is refreshed by the receiving end according to the refresh segment data corresponding to the preset number of error correction refresh frames, correct feedback information is sent to the sending end.

Illustratively, the union of the intra-frame coding regions of a preset number of error correction refresh frames corresponding to the same error feedback information is equal to the image range of each image frame; as shown in fig. 2, after receiving and decoding the three error correction refresh frames, the receiving end can completely update the reference data once, and then send correct feedback information to the sending end. And after the sending end receives the correct feedback information, the sending end can continue to encode the subsequent image frames.

In some embodiments, if the sending end receives error feedback information sent by the receiving end, the sending end determines an error correction sequence according to the current image frame, where the error correction sequence includes a preset number of error correction refresh frames and prediction frames following the preset number of error correction refresh frames.

Illustratively, after the reference data is refreshed according to the refresh segment data corresponding to the preset number of error correction refresh frames, the receiving end sends correct feedback information to the sending end.

And the sending end performs intra-frame coding on a partial area of the error correction refresh frame, after the coded error correction refresh frame is sent to the receiving end, if correct feedback information sent by the receiving end when the error correction refresh frame is decoded is received, the inter-frame coding is performed on the prediction frame in the error correction sequence, and the coded prediction frame is sent to the receiving end.

Illustratively, if an error occurs in an error correction refresh frame currently received and decoded by a receiving end, sending error feedback information to a sending end; and the sending end determines a preset number of error correction refresh frames according to the error correction refresh frames, and determines the error correction refresh frames and performs intra-frame coding of partial areas once again.

In some embodiments, the sending end performs inter-frame coding on the prediction frame in the error correction sequence, after sending the coded prediction frame to the receiving end, returns to the sending end to code the obtained current image frame, and sends the coded current image frame to the receiving end, and the receiving end continues to perform the step of receiving and decoding the current image frame until the receiving end receives the current image frame or decodes the current image frame with errors, and sends error feedback information to the sending end.

Illustratively, each refresh period in the video transmission method includes N image frames, wherein the interval refresh frame includes M consecutive frames, and each image frame is divided into M slices for encoding, wherein N is greater than M. And the receiving end sends feedback information to the sending end through an uplink channel. For example, the feedback information includes an error status, where the error status is a first value, such as 1, indicating that the feedback information is an error feedback information, and the error status is a second value, such as 0, indicating that the feedback information is a correct feedback information; the error feedback information may further include a slice index slice _ idx for receiving and decoding an erroneous slice.

Fig. 27 is a schematic diagram of image frame transmission.

In this example, the processing logic on the transmit side is as follows:

1. carrying out intra-frame coding on the first frame, and sending the obtained I frame to a receiving end;

2. performing interframe coding on the second frame to the Nth frame, and sending the obtained P frame to a receiving end;

3. carrying out periodic refreshing, carrying out intra-frame coding on partial areas from the 1 st frame to the M th frame in a refreshing period, and sending the obtained interval refreshing frames to a receiving end, wherein the union set of the areas subjected to intra-frame coding of the M frames is equal to the range of one frame, namely the refreshing areas are superposed into a whole frame;

4. performing interframe coding on the (M + 1) th frame to the (N) th frame in the refreshing period, and sending the obtained P frame to a receiving end;

5. returning to 3 and 4 to continue execution;

6. if the error state in the feedback information received by the sending end at any time is a first numerical value, immediately determining M error correction refresh frames according to the frames corresponding to the feedback information, carrying out intra-frame coding on partial regions of each error correction refresh frame, and sending the obtained error correction refresh frames to the receiving end, wherein the union set of the regions of the M frames for intra-frame coding is equal to the range of one frame, namely the refresh regions are overlapped to form a whole frame; determining the area of the 1 st error correction refresh frame for intra-frame coding according to the slice index _ idx of the feedback information;

7. after M error correction refresh frames are coded and sent, the subsequent N-M frames are subjected to interframe coding, and the obtained P frame is sent to a receiving end;

8. returning to 3 and 4 to continue execution; if the error state in the feedback information sent by the receiving end is still the first numerical value, determining the area for carrying out intra-frame coding on the 1 st frame in the refresh period according to the strip index slice _ idx of the feedback information; if the error state in the feedback information sent by the receiving end is a second value, determining that the area of the 1 st frame in the refreshing period for intra-frame coding is a first strip;

in this example, the processing logic at the receiving end is as follows:

1. initializing the error state in the feedback information to be a second numerical value, wherein the slice index slice _ idx is 0;

2. receiving the encoded frame from the sending end, performing protocol header analysis and fault-tolerant processing, and then decoding;

3. when an I frame, a group of complete M interval refresh frames or a group of complete M error correction refresh frames is received and decoded correctly, setting the error state as a second numerical value and setting the strip index slice _ idx as 0;

4. returning to 2 and 3 to continue execution;

when the protocol header is analyzed, if the physical layer packet sequence number or the coding frame sequence number in the frame header protocol is discontinuous and indicates that packet loss exists, the difference between the first lost packet sequence number and the first packet sequence number of the frame is marked as slice _ idx, meanwhile, the error state is set as a first numerical value, and the feedback information is sent to the sending end. And if the length of the coding frame in the frame tail protocol and the checksum check are not passed, setting the error state as a first value, marking the slice _ idx as 0, and sending the feedback information to the sending end.

By the above manner, according to the video transmission method provided by the embodiment of the application, when an error occurs in transmission of a certain image frame, the sending end performs intra-frame coding on a partial area of an error correction refresh frame, and the receiving end can update reference data required by decoding in time, so that the transmission error is recovered quickly; and the resource occupation of network resources can be reduced, and the increase of delay and jitter caused by code rate burst caused by intra-frame coding of the complete image frame can be avoided.

Illustratively, the interval refresh frame and the error correction refresh frame are divided according to the strip division of the image frame, the number of each group of refresh frames is the same as the strip number of the image frame, and each strip corresponds to an intra-frame coding area of one refresh frame.

Illustratively, the receiving end sends feedback information to the sending end through an uplink channel, and the sending end can adjust a refresh period and a refresh mode according to the feedback information; if the original refresh period is broken after the error feedback information is received, M error correction refresh frames are coded in time so that the receiving end can update the reference data.

Illustratively, the transmission protocol ensures that the transmitted code stream data service contains the stripe index information, and the receiving end can perform correspondence between the stripe index information and the code stream data service, so that the stripe index of the error position can be judged. There can be, but are not limited to, the following two implementations: one is that each sending data packet adds a transmission protocol header information of a strip index, the other is that the transmission protocol header does not contain the strip index corresponding to the current sending code stream, the sending end strictly sends a data service packet according to a single strip, and the receiving end can correspondingly obtain the strip index according to the service packet number index of the physical layer.

For example, when the sending end receives the error feedback information, the refresh cycle and the refresh format are readjusted, which is not limited to the following implementation manners: if the error occurs, starting to perform intra-frame coding and sending by using a refresh area corresponding to the stripe index of the error feedback information, and finishing the area refresh of a whole frame after M frames; a new refresh period begins after a number of P frames have passed after the refresh is complete.

Illustratively, an I frame, or a group of M complete interval refresh frames, or a group of M complete error correction refresh frames, is received and correctly received and decoded as a basis for determining error recovery.

For example, the video sending method provided in the embodiment of the present application may still perform decoding display under the condition that the uplink packet error is relatively high or uplink feedback is not timely, and the sending end still maintains periodic refreshing; when an error occurs, the sending end breaks through the original refreshing cycle, an error correction refreshing frame is sent out at the first time, and the refreshing area is the area corresponding to the error strip index, so that the error recovery is carried out to the maximum extent; the method can give consideration to two most important graph transmission indexes of graph transmission delay jamming and error recovery speed. The video transmission method is particularly suitable for application scenes with extremely high requirements on image transmission delay jamming and error recovery speed, such as a traversing machine.

Referring to fig. 28, fig. 28 is a schematic block diagram of a video transmitting apparatus 600 according to an embodiment of the present application. The video transmitter 600 includes a processor 601 and a memory 602, and the processor 601 and the memory 602 are connected by a bus 603, such as an I2C (Inter-integrated Circuit) bus 603.

Specifically, the Processor 601 may be a Micro-controller Unit (MCU), a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or the like.

Specifically, the Memory 602 may be a Flash chip, a Read-Only Memory (ROM) magnetic disk, an optical disk, a usb disk, or a removable hard disk.

The processor 601 is configured to run a computer program stored in the memory 602, and when executing the computer program, implement the video transmission method.

Illustratively, the processor 601 is configured to run a computer program stored in the memory 602 and to implement the following steps when executing the computer program:

acquiring a current image frame, encoding the current image frame, and sending the encoded current image frame to a receiving end;

if error feedback information sent by the receiving end is received, determining an error correction refresh frame according to the current image frame;

and carrying out intra-frame coding on a partial area of the error correction refresh frame, and sending the coded error correction refresh frame to the receiving end.

Referring to fig. 29, fig. 29 is a schematic block diagram of an aircraft 10 according to an embodiment of the present application.

As shown in fig. 29, the aircraft 10 includes:

the image acquisition component 11 is used for acquiring images;

the video transmitting device 600 is configured to obtain an image acquired by the image acquisition component, encode the image and transmit the encoded image to a receiving end;

a flying assembly 12 for flying.

Referring to fig. 30, fig. 30 is a schematic block diagram of a video receiving apparatus 700 according to an embodiment of the present application. The video receiving apparatus 700 includes a processor 701 and a memory 702, and the processor 701 and the memory 502 are connected by a bus 703, such as an I2C (Inter-integrated Circuit) bus 703.

Specifically, the Processor 701 may be a Micro-controller Unit (MCU), a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or the like.

Specifically, the Memory 702 may be a Flash chip, a Read-Only Memory (ROM) magnetic disk, an optical disk, a usb disk, or a removable hard disk.

The processor 701 is configured to run a computer program stored in the memory 702, and when executing the computer program, implement the video receiving method described above.

Illustratively, the processor 701 is configured to run a computer program stored in the memory 702 and to implement the following steps when executing the computer program:

receiving the coded current image frame from a sending end, and decoding the current image frame;

if the current image frame is received to be in error or the current image frame is decoded to be in error, sending error feedback information to the sending end, and receiving an error correction refresh frame which is sent by the sending end in response to the error feedback information and carries out intra-frame coding in a partial area;

and decoding the error correction refresh frame, and refreshing the reference data for decoding according to the decoding result.

Referring to fig. 31, fig. 31 is a schematic block diagram of a video playback device 20 according to an embodiment of the present application.

As shown in fig. 31, the video playback device includes:

the video receiving apparatus 700 described above is configured to receive an image from a transmitting end and decode the image;

and a display component 21, configured to display the data decoded by the video receiving apparatus.

According to the video sending device, the aircraft, the video receiving device and the video playing device provided by the embodiment of the application, when an error occurs in transmission of a certain image frame, the sending end carries out intra-frame coding on a partial area of an error correction refreshing frame, and the receiving end can update reference data required by decoding in time, so that the transmission error can be recovered quickly; and the resource occupation of network resources can be reduced, and the increase of delay and jitter caused by code rate burst caused by intra-frame coding of the complete image frame can be avoided.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and the processor executes the program instructions to implement the steps of the video transmitting method provided in the foregoing embodiment.

The computer-readable storage medium may be an internal storage unit of the video transmission apparatus according to any of the foregoing embodiments, for example, a hard disk or a memory of the video transmission apparatus. The computer-readable storage medium may also be an external storage device of the video transmission apparatus, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like provided on the video transmission apparatus.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and the processor executes the program instructions to implement the steps of the video receiving method provided in the foregoing embodiment.

The computer-readable storage medium may be an internal storage unit of the video receiving apparatus according to any of the foregoing embodiments, for example, a hard disk or a memory of the video receiving apparatus. The computer readable storage medium may also be an external storage device of the video receiving apparatus, such as a plug-in hard disk provided on the video receiving apparatus, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like.

While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.