阅读说明：本技术 一种在云导播台视频同步的方法及系统 (Method and system for synchronizing videos on cloud director ) 是由 唐杰 王喆 李庆瑜 戴立言 于 2020-12-31 设计创作，主要内容包括：本发明涉及云导播技术领域,提供了一种在云导播台视频同步的方法及系统,方法包括：在CDN云服务器上订阅音视频文件的输入流到本地服务器；在本地服务器上通过汇聚模块对音视频文件进行解码,并同时记录UTC时间戳,以作为后续时间同步统一基准；将解码后的音视频文件中的音视频数据通过数据分发器分发至预览、预监、主监任务；包括预览、预监、主监在内的任务,在对音视频数据重新编码时,将UTC时间戳写入音视频数据对应的信息中,将音视频数据发送到前端播放器；前端播放器,在播放时取出音视频数据中的UTC时间戳,实现包括预览、预监、主监在内的任务的视频同步,提升了用户操作和观看体验。(The invention relates to the technical field of cloud broadcasting guide, and provides a method and a system for video synchronization on a cloud broadcasting guide station, wherein the method comprises the following steps: subscribing an input stream of an audio and video file to a local server on a CDN cloud server; decoding the audio and video files on a local server through a convergence module, and simultaneously recording a UTC timestamp as a subsequent time synchronization unified reference; distributing the audio and video data in the decoded audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor; when the audio and video data are recoded, the UTC timestamp is written into the information corresponding to the audio and video data, and the audio and video data are sent to a front-end player; the front-end player takes out the UTC timestamp in the audio and video data during playing, achieves video synchronization of tasks including preview, pre-monitoring and main monitoring, and improves user operation and viewing experience.)

1. A method for video synchronization on a cloud director is characterized by comprising the following steps:

s1: establishing a local server for realizing the convergence-distribution of audio and video files, and subscribing the input stream of the audio and video files to the local server on a CDN cloud server through an RTMP protocol;

s2: establishing a convergence module on the local server, decoding the audio and video file through the convergence module, and simultaneously recording a UTC timestamp as a subsequent time synchronization unified reference;

s3: distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor;

s4: when the audio and video data are recoded, the UTC timestamp is written into the information corresponding to the audio and video data, and the audio and video data are sent to a front-end player;

s5: and the front-end player takes the UTC timestamp in the audio and video data during playing, and takes the UTC timestamp as a unified reference for time synchronization, so that video synchronization of tasks including preview, pre-monitoring and main monitoring is realized.

2. The method of video synchronization at a cloud director station as recited in claim 1, further comprising: in step S2, the method further includes: establishing a convergence-distribution model, which specifically comprises the following steps:

establishing a task management-switching module for realizing the switching function, closing and releasing the previous input source when the input source of the audio and video file needs to be switched, switching to a new input source, and simultaneously delivering the input data packet of the audio and video file to a subsequent module;

decapsulating the input stream of the audio/video file, and extracting an original compressed packet of the audio/video data;

decoding the audio data compression packet and the video data compression packet respectively to generate video data YUV and audio data PCM, and putting the video data YUV and the audio data PCM into the data distributor;

the data distributor adopts a producer consumer mode, data production comes from the video data YUV and the audio data PCM generated after decoding, and data consumers come from registration of preview, pre-monitoring and main monitoring tasks.

3. The method of video synchronization at a cloud director station as recited in claim 2, further comprising: the data distributor adopts a private TCP protocol as a data distribution protocol and keeps long connection with preview, pre-monitoring and main monitoring tasks.

4. The method for video synchronization on a cloud director platform according to claim 1, wherein in step S3, the method further comprises:

outputting two paths of data including low-resolution video data and high-resolution video data for the audio and video data distributed to the main monitoring task;

and the low-resolution video data is provided for a browser to play, and the high-resolution video data is provided for live broadcast for delayed use.

5. The method for video synchronization on a cloud director platform according to claim 2, wherein the step S4 specifically comprises:

scaling the resolution of the video data YUV for use by different front-end players;

resampling the audio data PCM, and setting a required audio output sampling rate and a required channel number;

encoding the video data YUV into video H.264, and PCM encoding the audio data into audio AAC;

writing the UTC timestamp into SEI information of the video H.264 coding;

and encapsulating the compressed packets of the video H.264 and the audio AAC, and sending the encapsulated packets to the front-end player.

6. The method for video synchronization on a cloud director platform according to claim 5, wherein the step S5 specifically comprises:

the front-end server subscribes output streams of tasks including preview, pre-monitoring and main monitoring;

decoding the output stream to obtain the video H.264 and the audio AAC;

fetching the UTC timestamp in the video H.264 coding SEI information;

and taking the UTC timestamp as a unified reference for time synchronization, and realizing video synchronization of tasks including preview, pre-monitoring and main monitoring.

7. The method of video synchronization at a cloud director station as recited in claim 1, further comprising:

tasks including preview, pre-monitoring and main monitoring are output to the front-end player in any one output mode including SRT protocol and RTMP protocol.

8. A system for performing the method of video synchronization at a cloud director station as claimed in any one of claims 1 to 7, comprising:

the server establishing module is used for establishing a local server for realizing convergence-distribution of the audio and video files, and subscribing the input stream of the audio and video files to the local server on the CDN cloud server through an RTMP protocol;

the aggregation module is used for establishing the aggregation module on the local server, decoding the audio and video files through the aggregation module, and simultaneously recording UTC time stamps to be used as a subsequent time synchronization unified reference;

the distribution module is used for distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor;

the task recoding module is used for providing tasks including preview, pre-monitoring and main monitoring, writing the UTC timestamp into information corresponding to the audio and video data when the audio and video data are recoded, and sending the audio and video data to a front-end player;

and the playing module is used for providing the video data to the front-end player, taking the UTC timestamp in the audio and video data during playing, and taking the UTC timestamp as a unified reference for time synchronization to realize video synchronization of tasks including preview, pre-monitoring and main monitoring.

9. An electronic device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by the processor to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, having stored thereon a computer program, characterized in that the computer program, when being executed by a processor, carries out the method according to any one of claims 1-7.

Technical Field

The invention relates to the technical field of cloud broadcasting guide, in particular to the technical field of video synchronization in the cloud broadcasting guide, and specifically relates to a method and a system for video synchronization in a cloud broadcasting guide. The transcoding engine is used as a core, and aims to synchronously play the same stream preview, pre-monitoring and main monitoring frequency pictures subscribed by the CDN cloud server, so that the operation and watching experience of a user are improved.

Background

The traditional cloud director broadcast station previews, pre-monitors and main monitor pictures respectively subscribe RTMP input sources from a CDN cloud server, are transcoded, packaged and output, and are released to the CDN cloud server by RTMP protocols respectively for users to watch. In this process, the subscription stream and the release stream may cause the asynchronism of the preview, pre-monitoring and main monitoring of the same stream. The operation and viewing experience of the user are seriously affected. The specific reasons are as follows:

(1) when the CDN cloud server cluster causes that preview, pre-monitoring and main monitoring subscribe the same stream, time difference is caused by different equipment access points, and then videos are asynchronous.

As is well known, CDN cloud servers are implemented using clusters. One implementation of cluster load balancing is through domain name. Namely, when a remote request data, it is decided which working host is used to provide service to the outside according to the idle state of the nodes in the cluster. Experiments prove that: the same server subscribes streams from the CDN cloud server for multiple times, and the obtained host machine IP addresses are different. The device access points are different, which in turn causes a time difference. I.e. the same stream is previewed, pre-monitored and main monitored, and the video is not synchronized at the moment of receiving.

(2) The CDN cloud server caches one gop video data, so that time difference exists when previewing, pre-monitoring and main monitoring subscribe the same stream, and further videos are not synchronous.

Some mainstream streaming media servers, like SRS, may cache video data of one gop inside, so that when a receiving end subscribes to a stream, a first frame of video sent is a key frame, and a picture can be shown in seconds. This also means that from the same server subscribing to the stream, the resulting video data is not currently up-to-date. The maximum is one gop time. Experiments prove that: previewing, pre-monitoring and main monitoring are carried out on the same flow, subscription requests are sent at the same time, and the time stamps of the received first frame of video sometimes have large difference.

Disclosure of Invention

In view of the above problems, an object of the present invention is to provide a method and a system for video synchronization in a cloud broadcast guiding station, which can overcome the defect that the preview, pre-monitoring, and main monitoring of the same channel of video subscribed by the cloud broadcast guiding station from a CDN cloud server are not synchronized, and implement the synchronous play of the preview, pre-monitoring, and main monitoring of the same channel of video subscribed by the cloud broadcast guiding station from the CDN cloud server, thereby improving the user operation and viewing experience.

The above object of the present invention is achieved by the following technical solutions:

a method for video synchronization at a cloud director station, comprising the steps of:

s1: establishing a local server for realizing the convergence-distribution of audio and video files, and subscribing the input stream of the audio and video files to the local server on a CDN cloud server through an RTMP protocol;

s2: establishing a convergence module on the local server, decoding the audio and video file through the convergence module, and simultaneously recording a UTC timestamp as a subsequent time synchronization unified reference;

s3: distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor;

s4: when the audio and video data are recoded, the UTC timestamp is written into the information corresponding to the audio and video data, and the audio and video data are sent to a front-end player;

s5: and the front-end player takes the UTC timestamp in the audio and video data during playing, and takes the UTC timestamp as a unified reference for time synchronization, so that video synchronization of tasks including preview, pre-monitoring and main monitoring is realized.

Further, in step S2, the method further includes: establishing a convergence-distribution model, which specifically comprises the following steps:

establishing a task management-switching module for realizing the switching function, closing and releasing the previous input source when the input source of the audio and video file needs to be switched, switching to a new input source, and simultaneously delivering the input data packet of the audio and video file to a subsequent module;

decapsulating the input stream of the audio/video file, and extracting an original compressed packet of the audio/video data;

decoding the audio data compression packet and the video data compression packet respectively to generate video data YUV and audio data PCM, and putting the video data YUV and the audio data PCM into the data distributor;

the data distributor adopts a producer consumer mode, data production comes from the video data YUV and the audio data PCM generated after decoding, and data consumers come from registration of preview, pre-monitoring and main monitoring tasks.

Further, the data distributor adopts a private TCP protocol as a data distribution protocol, and long connection with preview, pre-monitoring and main monitoring tasks is kept.

Further, in step S3, the method further includes:

outputting two paths of data including low-resolution video data and high-resolution video data for the audio and video data distributed to the main monitoring task;

and the low-resolution video data is provided for a browser to play, and the high-resolution video data is provided for live broadcast for delayed use.

Further, in step S4, specifically, the method includes:

scaling the resolution of the video data YUV for use by different front-end players;

resampling the audio data PCM, and setting a required audio output sampling rate and a required channel number;

encoding the video data YUV into video H.264, and PCM encoding the audio data into audio AAC;

writing the UTC timestamp into SEI information of the video H.264 coding;

and encapsulating the compressed packets of the video H.264 and the audio AAC, and sending the encapsulated packets to the front-end player.

Further, in step S5, specifically, the method includes:

the front-end server subscribes output streams of tasks including preview, pre-monitoring and main monitoring;

decoding the output stream to obtain the video H.264 and the audio AAC;

fetching the UTC timestamp in the video H.264 coding SEI information;

and taking the UTC timestamp as a unified reference for time synchronization, and realizing video synchronization of tasks including preview, pre-monitoring and main monitoring.

Further, the tasks including preview, pre-monitor and main monitor are output to the front-end player by any one of the output modes including SRT protocol and RTMP protocol.

A system for performing the above-described method of video synchronization at a cloud director, comprising:

the server establishing module is used for establishing a local server for realizing convergence-distribution of the audio and video files, and subscribing the input stream of the audio and video files to the local server on the CDN cloud server through an RTMP protocol;

the aggregation module is used for establishing the aggregation module on the local server, decoding the audio and video files through the aggregation module, and simultaneously recording UTC time stamps to be used as a subsequent time synchronization unified reference;

the distribution module is used for distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor;

the task recoding module is used for providing tasks including preview, pre-monitoring and main monitoring, writing the UTC timestamp into information corresponding to the audio and video data when the audio and video data are recoded, and sending the audio and video data to a front-end player;

and the playing module is used for providing the video data to the front-end player, taking the UTC timestamp in the audio and video data during playing, and taking the UTC timestamp as a unified reference for time synchronization to realize video synchronization of tasks including preview, pre-monitoring and main monitoring.

An electronic device comprising a processor and a memory, wherein at least one instruction, at least one program, a set of codes, or a set of instructions is stored in the memory, and wherein the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the method.

A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method as described above.

Compared with the prior art, the invention has the beneficial effects that:

(1) the method for synchronizing the videos on the cloud director station comprises the following steps: s1: establishing a local server for realizing the convergence-distribution of audio and video files, and subscribing the input stream of the audio and video files to the local server on a CDN cloud server through an RTMP protocol; s2: establishing a convergence module on the local server, decoding the audio and video file through the convergence module, and simultaneously recording a UTC timestamp as a subsequent time synchronization unified reference; s3: distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor; s4: when the audio and video data are recoded, the UTC timestamp is written into the information corresponding to the audio and video data, and the audio and video data are sent to a front-end player; s5: and the front-end player takes the UTC timestamp in the audio and video data during playing, and takes the UTC timestamp as a unified reference for time synchronization, so that video synchronization of tasks including preview, pre-monitoring and main monitoring is realized. According to the technical scheme, the defect that the same channel stream subscribed by the cloud broadcasting guide platform from the CDN cloud server previews, pre-monitors and main monitoring images are not played synchronously can be overcome, the same stream subscribed by the cloud broadcasting guide platform from the CDN cloud server previews, pre-monitors and main monitoring images are played synchronously, and user operation and watching experience are improved.

(2) A convergence-distribution model is employed. And changing the multi-time subscription and multi-time decoding of the same input stream into one-time subscription, one-time decoding and multi-time distribution.

Drawings

FIG. 1 is an overall flow diagram of a method of video synchronization at a cloud director station in accordance with the present invention;

fig. 2 is a schematic diagram of an overall architecture of a cloud director transcoding service according to the present invention;

FIG. 3 is a schematic diagram of a convergence-distribution model of the present invention;

FIG. 4 is a schematic diagram of a production consumer model in the data distributor of the present invention;

FIG. 5 is a flow chart of the preview function implementation of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The invention takes a transcoding engine as a core. Two video synchronization schemes of a server and a client are simultaneously supported, and the method comprises the following steps:

1. transcoding services video synchronization scheme.

The transcoding service employs a convergence-distribution model. I.e., subscribing to streams from CDN cloud servers to local, this process is called aggregation. The preview, pre-monitor and main monitor videos are read from the local convergence service, and are not directly read from the CDN cloud server. I.e. the data comes from the distribution of the converged service. And the multiple subscriptions and the multiple decodings of the input stream are converted into one subscription, one decoding and multiple distribution, so that the resource consumption of the system is reduced.

2. A client video synchronization scheme.

And the transcoding service subscribes streams from the CDN cloud server to the local, records the UTC time stamp of each received video packet, and writes the recorded UTC time stamp into H.264 coding SEI information during video coding. And the front-end player decodes the H.164, reads the UTC timestamp in the SEI information, and performs playing control by taking the UTC timestamp as a uniform time reference.

The following is illustrated by specific examples:

first embodiment

As shown in fig. 1 and fig. 2, the present embodiment provides a method for video synchronization on a cloud director, including the following steps:

s1: establishing a local server for realizing the convergence-distribution of audio and video files, and subscribing the input stream of the audio and video files to the local server on a CDN cloud server through an RTMP protocol.

Specifically, in order to implement aggregation of audio and video streams to facilitate subsequent uniform distribution, a local server needs to be established to store a video input from the CND cloud server.

S2: and establishing a convergence module on the local server, decoding the audio and video file through the convergence module, and simultaneously recording a UTC timestamp to serve as a subsequent time synchronization unified reference.

Specifically, in this step, it is necessary to establish a convergence-distribution model, as shown in fig. 3, the method specifically includes the following steps:

s21: establishing a task management-switching module for realizing the switching function, closing and releasing the previous input source when the input source of the audio and video file needs to be switched, switching to a new input source, and simultaneously delivering the input data packet of the audio and video file to a subsequent module;

s22: decapsulating the input stream of the audio/video file, and extracting an original compressed packet of the audio/video data;

s23: decoding the audio data compression packet and the video data compression packet respectively to generate video data YUV and audio data PCM, and putting the video data YUV and the audio data PCM into the data distributor;

s24: the data distributor adopts a producer-consumer mode (as shown in figure 4), data is produced from the video data YUV and the audio data PCM generated after decoding, and data consumers are from the registration of preview, pre-monitoring and main monitoring tasks.

Further, the data distributor adopts a private TCP protocol as a data distribution protocol, and long connection with preview, pre-monitoring and main monitoring tasks is kept, so that the communication efficiency is improved.

S3: and distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor.

For the audio and video data distributed to the main monitoring task, outputting two paths of data including low-resolution video data and high-resolution video data; and the low-resolution video data is provided for a browser to play, and the high-resolution video data is provided for live broadcast for delayed use.

S4: when the audio and video data is recoded, the UTC timestamp is written into information corresponding to the audio and video data, and the audio and video data is sent to a front-end player, as shown in fig. 5, specifically, the tasks include preview, pre-monitoring, and main monitoring:

s41: scaling the resolution of the video data YUV for use by different front-end players, such as in preview, scaling to a low resolution for use in web page playing;

s42: resampling the audio data PCM, and setting a required audio output sampling rate and a required channel number;

s43: encoding the video data YUV into video H.264, and PCM encoding the audio data into audio AAC;

s44: writing the UTC timestamp into SEI information of the video H.264 coding;

s45: and encapsulating the compressed packets of the video H.264 and the audio AAC, and sending the encapsulated packets to the front-end player.

S5: the front-end player takes out the UTC timestamp in the audio and video data during playing, and takes the UTC timestamp as a unified reference for time synchronization to realize video synchronization of tasks including preview, pre-monitoring and main monitoring, which specifically comprises the following steps:

the front-end server subscribes output streams of tasks including preview, pre-monitoring and main monitoring;

decoding the output stream to obtain the video H.264 and the audio AAC;

fetching the UTC timestamp in the video H.264 coding SEI information;

and taking the UTC timestamp as a unified reference for time synchronization, and realizing video synchronization of tasks including preview, pre-monitoring and main monitoring.

Further, the tasks including preview, pre-monitor and main monitor are output to the front-end player by any one of the output modes including SRT protocol and RTMP protocol.

Second embodiment

The present embodiment provides a system for performing the method for video synchronization on a cloud director in the first embodiment, including:

the server establishing module 1 is used for establishing a local server for realizing convergence-distribution of audio and video files, and subscribing the input stream of the audio and video files to the local server on a CDN cloud server through an RTMP protocol;

the aggregation module 2 is used for establishing an aggregation module on the local server, decoding the audio and video file through the aggregation module, and simultaneously recording a UTC timestamp to serve as a subsequent time synchronization unified reference;

the distribution module 3 is used for distributing the decoded audio and video data in the audio and video file to preview, pre-monitoring and main monitoring tasks through a data distributor;

the task recoding module 4 is used for providing tasks including preview, pre-monitoring and main monitoring, writing the UTC timestamp into information corresponding to the audio and video data when the audio and video data are recoded, and sending the audio and video data to a front-end player;

and the playing module 5 is used for providing the front-end player with the UTC timestamp, taking the UTC timestamp in the audio and video data during playing, and taking the UTC timestamp as a unified reference for time synchronization to realize video synchronization of tasks including preview, pre-monitoring and main monitoring.

It should be noted that other steps are the same as those in the first embodiment, and are not described again in this embodiment.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

It should be noted that the above embodiments can be freely combined as necessary. The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.