阅读说明：本技术 基于视频流的sei帧回放数据同步方法、系统、装置及介质 (SEI frame playback data synchronization method, system, device and medium based on video stream ) 是由 郑新越 白剑 黄海亮 梁瑛玮 张海林 鲁和平 李长杰 陈焕然 李乐 王浩 洪行健 于 2021-06-24 设计创作，主要内容包括：本发明提出了一种基于视频流的SEI帧回放数据同步方法、系统、装置及介质,所述方法包括：步骤1,获取录制好的视频文件并解析出视频文件中所有的SEI帧数据,确定每一SEI帧所对应的当时时间戳以及辅助文档的每一页在视频播放时间线上出现的相对时间；步骤2,分别计算两相邻SEI帧所对应当时时间戳的间隔时间△T,根据所述间隔时间△T确定两相邻SEI帧之间是否丢失视频帧数据；是则进入步骤3,否则不处理视频文件而让视频文件正常播放；步骤3,将辅助文档数据时间戳在这两相邻SEI帧时间戳内的辅助文档数据删除并将后面的辅助文档数据向前移动。本发明能够根据视频帧数据丢失准确地对辅助文档数据进行矫正,极大保证了回放中辅助文档和视频的同步性。(The invention provides a method, a system, a device and a medium for synchronizing SEI frame playback data based on video streams, wherein the method comprises the following steps: step 1, acquiring a recorded video file, analyzing all SEI frame data in the video file, and determining a current timestamp corresponding to each SEI frame and relative time of each page of an auxiliary document appearing on a video playing time line; step 2, respectively calculating interval time delta T of time stamps corresponding to two adjacent SEI frames, and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering step 3, otherwise, not processing the video file and enabling the video file to be played normally; and 3, deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forward. The method can accurately correct the auxiliary document data according to the loss of the video frame data, and greatly ensures the synchronism of the auxiliary document and the video in the playback.)

1. A method for synchronizing the playback data of SEI frames based on a video stream, comprising the steps of:

step 1, acquiring a recorded video file, analyzing all SEI frame data in the video file, and simultaneously determining a current timestamp corresponding to each SEI frame and the relative time of each page of an auxiliary document appearing on a video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

step 2, respectively calculating interval time delta T of time stamps corresponding to two adjacent SEI frames, and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and 3, deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forward.

2. The method for synchronizing SEI frame playback data of a video stream according to claim 1, wherein the step of determining whether video frame data is lost between two adjacent SEI frames according to the interval time Δ T in step 2 comprises:

and judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, and otherwise, not losing the video frame data between the two adjacent SEI frames.

3. The method for synchronizing the playback data of SEI frames based on video streams as claimed in claim 2, wherein the step of step 2 comprises:

step 201, determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1;

step 202, judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, otherwise, not losing video frame data between two adjacent SEI frames.

4. The video stream-based SEI frame playback data synchronization method according to claim 3, wherein the step of deleting the auxiliary document data having the auxiliary document data timestamp within the two adjacent SEI frame timestamps and moving the following auxiliary document data forward in step 3 comprises:

the auxiliary document data having the auxiliary document data time stamp between T1 and T2 is deleted, and the following auxiliary document data is moved forward (Δ T- Δ T).

5. A video stream-based SEI frame playback data synchronization system, comprising

The SEI frame analysis module is used for acquiring the recorded video file, analyzing all SEI frame data in the video file, and determining the current timestamp corresponding to each SEI frame and the relative time of each page of the auxiliary document appearing on the video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

the video frame loss determining module is used for respectively calculating the interval time delta T of the time stamps corresponding to the two adjacent SEI frames and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and the auxiliary document data proofreading module is used for deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forwards.

6. The video stream-based SEI frame playback data synchronization system according to claim 5, wherein the video frame loss determining module is specifically configured to determine whether the interval time Δ T is greater than a preset interval time Δ T, if so, video frame data is lost between two adjacent SEI frames, otherwise, no video frame data is lost between two adjacent SEI frames.

7. The video stream-based SEI frame playback data synchronization system as recited in claim 6, wherein the video frame loss determination module comprises

An interval time calculation unit for determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1;

and the video frame loss determining unit is used for judging whether the interval time delta T is larger than a preset interval time delta T, if so, video frame data are lost between two adjacent SEI frames, and otherwise, the video frame data are not lost between the two adjacent SEI frames.

8. The video stream-based SEI frame playback data synchronization system according to claim 7, wherein the auxiliary document data check module is specifically configured to delete auxiliary document data with an auxiliary document data timestamp between T1 and T2 and move the following auxiliary document data forward (Δ T- Δ T).

9. An SEI frame playback data synchronization apparatus based on video stream, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the SEI frame playback data synchronization method based on video stream according to any one of claims 1 to 4.

10. A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the video stream-based SEI frame playback data synchronization method according to any one of claims 1 to 4.

Technical Field

The invention relates to the technical field of video processing, in particular to a method, a system, a device and a medium for synchronizing SEI frame playback data based on video streams.

Background

At present, when live broadcast is carried out by pushing stream through live broadcast software, other auxiliary functions independent of the live broadcast stream, such as ppt data, are sometimes required to be added, and at the moment, when the whole live broadcast process is restored in live broadcast playback, video data and data of other auxiliary functions need to be played. And in the live broadcast process, due to unstable network link, the stream is easy to break, so that the stored partial video frame data can be lost, and at the moment, if the ppt data is normally stored, the problem that the ppt and the video are not synchronous can occur during playback.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a method, a system, a device and a storage medium for synchronizing SEI frame playback data based on video streams, which can accurately correct auxiliary document data according to video frame data loss, and greatly ensure the synchronism of the auxiliary document and the video in playback.

In order to achieve the above object, the present invention discloses a method for synchronizing SEI frame playback data based on a video stream, comprising the following steps:

step 1, acquiring a recorded video file, analyzing all SEI frame data in the video file, and simultaneously determining a current timestamp corresponding to each SEI frame and the relative time of each page of an auxiliary document appearing on a video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

step 2, respectively calculating interval time delta T of time stamps corresponding to two adjacent SEI frames, and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and 3, deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forward.

Further, in step 2, the step of determining whether video frame data is lost between two adjacent SEI frames according to the interval time Δ T includes:

and judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, and otherwise, not losing the video frame data between the two adjacent SEI frames.

Further, the step 2 comprises the following steps:

step 201, determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1;

step 202, judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, otherwise, not losing video frame data between two adjacent SEI frames.

Further, in step 3, the step of deleting the auxiliary document data with the auxiliary document data timestamp within the two adjacent SEI frame timestamps and moving the following auxiliary document data forward includes:

the auxiliary document data having the auxiliary document data time stamp between T1 and T2 is deleted, and the following auxiliary document data is moved forward (Δ T- Δ T).

On the other hand, the invention also discloses a SEI frame playback data synchronization system based on the video stream, which comprises

The SEI frame analysis module is used for acquiring the recorded video file, analyzing all SEI frame data in the video file, and determining the current timestamp corresponding to each SEI frame and the relative time of each page of the auxiliary document appearing on the video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

the video frame loss determining module is used for respectively calculating the interval time delta T of the time stamps corresponding to the two adjacent SEI frames and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and the auxiliary document data proofreading module is used for deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forwards.

Further, the video frame loss determining module is specifically configured to determine whether the interval time Δ T is greater than a preset interval time Δ T, if so, video frame data is lost between two adjacent SEI frames, otherwise, video frame data is not lost between two adjacent SEI frames.

Further, the video frame loss determining module includes:

an interval time calculation unit for determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1;

and the video frame loss determining unit is used for judging whether the interval time delta T is larger than a preset interval time delta T, if so, video frame data are lost between two adjacent SEI frames, and otherwise, the video frame data are not lost between the two adjacent SEI frames.

Further, the auxiliary document data proofreading module is specifically configured to delete the auxiliary document data with the auxiliary document data timestamp between T1 and T2, and move the following auxiliary document data forward (Δ T — Δ T).

In another aspect, the present invention further discloses a device for synchronizing video stream-based SEI frame playback data, including a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the method for synchronizing video stream-based SEI frame playback data is implemented.

In still another aspect, the present invention further discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for synchronizing the playback data of SEI frames based on video streams.

Compared with the prior art, the invention has the following advantages: the current timestamp is added into an SEI frame of the video in the live broadcast process, and meanwhile, the relative time of each page of the auxiliary document in the video playing time line can be recorded by the auxiliary document data (such as PPT data); when live broadcast playback is carried out, the current time stamp corresponding to each SEI frame and the relative time of each page of the auxiliary document on a video playing time line are determined by analyzing all SEI frame data in a video file, then whether the video frame data are lost or not is determined according to the interval time of the current time stamps corresponding to two adjacent SEI frames, if the video frame data are lost, the auxiliary document data of the auxiliary document data time stamps in the two adjacent SEI frame time stamps are deleted, the subsequent auxiliary document data are moved forward, therefore, the auxiliary document data can be accurately corrected, and the synchronism of the auxiliary document and the video in the playback process is greatly ensured.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of a video stream-based SEI frame playback data synchronization method according to the present invention;

FIG. 2 is a block diagram illustrating the structure of the SEI frame playback data synchronization method based on video stream according to the present invention;

FIG. 3 is a diagram illustrating a corresponding relationship between SEI frame data and PPT data of a video in a normal case;

fig. 4 is a time stamp mapping relationship diagram of SEI frame data and PPT data of a video in case of flow interruption;

fig. 5 is a time stamp corresponding relationship diagram of SEI frame data and PPT data of the adjusted video.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, an embodiment of the present invention discloses a method for synchronizing playback data of an SEI frame based on a video stream, including the following steps:

step 1, acquiring a recorded video file, analyzing all SEI frame data in the video file, and simultaneously determining a current timestamp corresponding to each SEI frame and the relative time of each page of an auxiliary document appearing on a video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

step 2, respectively calculating interval time delta T of time stamps corresponding to two adjacent SEI frames, and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and 3, deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forward.

Correspondingly, the invention discloses an SEI frame playback data synchronization system based on video stream, which comprises an SEI frame analysis module, a video frame loss determination module and an auxiliary document data correction module, wherein the SEI frame analysis module is used for analyzing the SEI frame, determining the loss of the video frame and correcting the auxiliary document data

The SEI frame analysis module is used for acquiring the recorded video file, analyzing all SEI frame data in the video file, and determining the current timestamp corresponding to each SEI frame and the relative time of each page of the auxiliary document appearing on the video playing time line; the current time stamp of the video file during live broadcasting and the relative time of each page of the auxiliary document appearing in the video playing time line are respectively recorded in an SEI frame of the video file;

the video frame loss determining module is used for respectively calculating the interval time delta T of the time stamps corresponding to the two adjacent SEI frames and determining whether video frame data are lost between the two adjacent SEI frames according to the interval time delta T; if yes, entering the step 3, otherwise, not processing the video file and enabling the video file to be played normally;

and the auxiliary document data proofreading module is used for deleting the auxiliary document data with the auxiliary document data time stamp in the two adjacent SEI frame time stamps and moving the subsequent auxiliary document data forwards.

In the present embodiment, the video stream-based SEI frame playback data synchronization method has the video stream-based SEI frame playback data synchronization system as an object of execution of the steps, or has components within the video stream-based SEI frame playback data synchronization system as an object of execution of the steps. Specifically, step 1 takes an SEI frame parsing module as an execution object of the step, step 2 takes a video frame loss determination module as an execution object of the step, and step 3 takes an auxiliary document data collation module as an execution object of the step.

The client adds a current timestamp into an SEI frame of a video in a live broadcast process, auxiliary document data, such as PPT data, also records the relative time of each page in PPT in a video playing time line, and a video file of mp4/m3u8 is generated after the live broadcast is finished; in the step 1, byte data of the file can be read online through the video file, all SEI frame data in the video file are analyzed, and therefore a timestamp corresponding to each SEI frame and the relative time of each page in the PPT appearing in a video playing time line are determined, and therefore the time of PPT can be corrected according to the read SEI when live broadcast playback is carried out.

In step 2, after the current time stamp corresponding to each SEI frame is calculated, whether video frame data is lost between two adjacent SEI frames is determined according to the interval time Δ T of the current time stamps corresponding to the two adjacent SEI frames, so as to determine that the auxiliary document data does not need to be processed; if the video file has a video frame loss situation, the auxiliary document data needs to be processed to ensure the synchronicity of the auxiliary document and the video during video playback.

Specifically, in step 2, the step of determining whether video frame data is lost between two adjacent SEI frames according to the interval time Δ T includes:

and judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, and otherwise, not losing the video frame data between the two adjacent SEI frames.

Correspondingly, in the SEI frame playback data synchronization system based on the video stream, the video frame loss determining module is further specifically configured to determine whether the interval time Δ T is greater than a preset interval time Δ T, if so, video frame data is lost between two adjacent SEI frames, otherwise, video frame data is not lost between two adjacent SEI frames.

Normally, the time difference between every two adjacent SEI frames is fixed, and is generally 2S, so if the time difference between the adjacent SEI frames exceeds 2S, it is considered that video frame data is lost in the middle, that is, a cut-out occurs. Therefore, in the present invention, the preset time interval is set as the time difference between two adjacent SEI frames when no current break occurs under normal conditions, and when the time difference between the adjacent SEI frames exceeds the preset time interval, it indicates that video frame data is lost between the two adjacent SEI frames, and the auxiliary document data needs to be corrected to ensure the synchronization between the auxiliary document and the video during video playback.

Specifically, in the SEI frame playback data synchronization system based on the video stream, step 2 includes the following sub-steps:

step 201, determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1; wherein n =0,1,2,3 … …;

step 202, judging whether the interval time delta T is larger than a preset interval time delta T, if so, losing video frame data between two adjacent SEI frames, otherwise, not losing video frame data between two adjacent SEI frames.

Correspondingly, the video frame loss determining module comprises

An interval time calculation unit for determining a timestamp T1 of an nth SEI frame and a timestamp T2 of an n +1 th SEI frame, and calculating an interval time Δ T between two adjacent SEI frames according to a formula Δ T = T2-T1;

and the video frame loss determining unit is used for judging whether the interval time delta T is larger than a preset interval time delta T, if so, video frame data are lost between two adjacent SEI frames, and otherwise, the video frame data are not lost between the two adjacent SEI frames.

Similarly, step 2 is to take the video frame loss determining module as an execution object of the step, or take the component in the video frame loss determining module as an execution object of the step; specifically, step 201 is an object of execution of the step of the inter-period calculation unit, and step 202 is an object of execution of the step of the video frame loss determination unit.

Step 201, because the client adds the current timestamp into the SEI frames of the video in the live broadcast process, the current timestamp corresponding to each SEI frame can be determined, and the interval time Δ T between two adjacent SEI frames can be calculated by using a formula Δ T = T2-T1 according to the timestamp T1 of the nth SEI frame and the timestamp T2 of the (n + 1) th SEI frame; step 202 determines whether video frame data is lost between every two adjacent SEI frames in the video file by determining whether the interval time Δ T is greater than a preset interval time Δ T.

In step 3, if video frame data are lost between two adjacent SEI frames in the video file, deleting the auxiliary document data with the auxiliary document data timestamp in the two adjacent SEI frame timestamps, and moving the following auxiliary document data forward.

Specifically, in step 3, the auxiliary document data having the auxiliary document data time stamp between T1 and T2 is deleted, and the following auxiliary document data is moved forward (Δ T- Δ T).

Correspondingly, in the video stream-based SEI frame playback data synchronization system, the auxiliary document data check module is specifically configured to delete the auxiliary document data with the auxiliary document data timestamp between T1 and T2, and move the following auxiliary document data forward (Δ T- Δ T).

The procedure for correcting PPT data is described as follows: under normal conditions, when there is no flow break in the stream pushing process, that is, there is no video frame loss in the video file, the corresponding relationship between the time stamps of the SEI frame data and the PPT data of the video is shown in fig. 3; when a stream is cut off in the stream pushing process, that is, when a video frame is lost in a video file, the corresponding relationship between the time stamps of the SEI frame data and the PPT data of the video is shown in fig. 4; as can be seen from fig. 3 and 4, the video segment in the 10 th and following video segments is moved forward for 4 seconds, the PPT starts from the third page, and the page appears at a time point that is not already at the correct position for video playing, that is, the third page of the PPT should appear at the SEI frame with the timestamp of 12s originally but now appears at the SEI frame with the time point of 16s, so that the data in the third and following PPT pages needs to be adjusted to move forward (Δ T); in the embodiment, Δ T =6s, and Δ T =2s, so that data on the third page and later of the PPT needs to be moved forward by 4 s; as shown in fig. 5, after the adjustment is completed, the adjusted PPT data is displayed at the original playing position, so that the synchronization between the PPT and the video during the video playback is ensured.

Another embodiment of the present invention further provides a device for synchronizing video stream-based SEI frame playback data, including a processor and a memory, where the memory stores a computer program, and the computer program is executed by the processor to implement the method for synchronizing video stream-based SEI frame playback data.

Another embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for synchronizing the playback data of SEI frames based on video streams.

In summary, the present invention adds the current timestamp into the SEI frame of the video during the live broadcast, and the auxiliary document data (such as PPT data) also records the relative time of each page of the auxiliary document appearing in the video playing timeline; when live broadcast playback is carried out, the current time stamp corresponding to each SEI frame and the relative time of each page of the auxiliary document on a video playing time line are determined by analyzing all SEI frame data in a video file, then whether the video frame data are lost or not is determined according to the interval time of the current time stamps corresponding to two adjacent SEI frames, if the video frame data are lost, the auxiliary document data of the auxiliary document data time stamps in the two adjacent SEI frame time stamps are deleted, the subsequent auxiliary document data are moved forward, therefore, the auxiliary document data can be accurately corrected, and the synchronism of the auxiliary document and the video in the playback process is greatly ensured.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.