阅读说明：本技术 可视对讲系统的音视频同步方法、装置、设备及存储介质 (Audio and video synchronization method, device, equipment and storage medium of video intercom system ) 是由 曾义 杜其昌 吴艳茹 于 2019-10-30 设计创作，主要内容包括：本申请实施例公开了一种可视对讲系统的音视频同步方法、装置、电子设备及存储介质。所述方法包括：在音频帧和视频帧被创建时,根据预先设置的系统时间戳起点得到所述音频帧和所述视频帧的显示时间戳,所述显示时间戳用于指示所述音频帧和所述视频帧在解码后的显示顺序；在对所述音频帧和所述视频帧进行编码得到压缩数据包时,根据预先设置的系统时间戳起点得到所述音频帧和所述视频帧的解码时间戳,所述解码时间戳用于指示所述音频帧和所述视频帧的解码顺序；将所述压缩数据包、对应的显示时间戳及解码时间戳推流至服务器端。本申请实施例提供的技术方案,能够较好地确保可视对讲系统的音视频同步播放。(The embodiment of the application discloses an audio and video synchronization method and device of a visual intercom system, electronic equipment and a storage medium. The method comprises the following steps: when an audio frame and a video frame are created, obtaining display timestamps of the audio frame and the video frame according to a preset system timestamp starting point, wherein the display timestamps are used for indicating the display sequence of the audio frame and the video frame after decoding; when the audio frame and the video frame are coded to obtain a compressed data packet, obtaining decoding time stamps of the audio frame and the video frame according to a preset system time stamp starting point, wherein the decoding time stamps are used for indicating the decoding sequence of the audio frame and the video frame; and pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to a server side. According to the technical scheme provided by the embodiment of the application, the audio and video synchronous playing of the visual intercom system can be well ensured.)

1. An audio and video synchronization method of a visual intercom system is characterized by comprising the following steps:

when an audio frame and a video frame are created, obtaining display timestamps of the audio frame and the video frame according to a preset system timestamp starting point, wherein the display timestamps are used for indicating the display sequence of the audio frame and the video frame after decoding;

when the audio frame and the video frame are coded to obtain a compressed data packet, obtaining decoding time stamps of the audio frame and the video frame according to a preset system time stamp starting point, wherein the decoding time stamps are used for indicating the decoding sequence of the audio frame and the video frame;

and pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to a server side.

2. The audio-video synchronization method of a video intercom system according to claim 1, wherein before obtaining the display timestamps of the audio frame and the video frame according to a preset system timestamp starting point when the audio frame and the video frame are created, further comprising:

when the system software starts, the current time stamp is recorded and used as the starting point of the system time stamp.

3. The audio-video synchronization method of a video intercom system according to claim 2, wherein in obtaining the display timestamps of the audio frame and the video frame according to a preset system timestamp starting point, the display timestamp is obtained by differencing according to the corresponding timestamps when the audio frame and the video frame are created and the corresponding timestamp when system software is started;

correspondingly, in the decoding time stamps of the audio frame and the video frame obtained according to the preset system time stamp starting point, the decoding time stamps are obtained by performing difference according to the time stamps corresponding to the audio frame and the video frame when the audio frame and the video frame are coded and the time stamps corresponding to the audio frame and the video frame when the system software is started.

4. The audio-video synchronization method of a video intercom system according to claim 1, wherein in pushing the compressed data packet, the corresponding display timestamp and decoding timestamp to a server side, the duration information of the compressed data packet is also pushed to the server side.

5. The audio-video synchronization method of a video intercom system according to claim 4, wherein before pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to the server, further comprising:

converting the display timestamp, the decode timestamp, and the duration from an encoding time base to a system time base representation.

6. An audio and video synchronization device of a video intercom system, comprising:

the system comprises a creating module, a decoding module and a display module, wherein the creating module is used for obtaining display timestamps of an audio frame and a video frame according to a preset system timestamp starting point when the audio frame and the video frame are created, and the display timestamps are used for indicating the display sequence of the audio frame and the video frame after decoding;

the encoding module is used for obtaining decoding time stamps of the audio frames and the video frames according to a preset system time stamp starting point when the audio frames and the video frames are encoded to obtain compressed data packets, and the decoding time stamps are used for indicating the decoding sequence of the audio frames and the video frames;

and the stream pushing module is used for pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to a server side.

7. An audiovisual synchronization device of a visual intercom system as in claim 6, further comprising:

and the recording module is used for recording the current time stamp when the system software is started, and taking the current time stamp as the starting point of the system time stamp.

8. An audiovisual synchronization device of a visual intercom system as in claim 6, further comprising:

and the conversion module is used for converting the display time stamp, the decoding time stamp and the duration from an encoding time base to a system time base representation.

9. An electronic device, comprising:

a memory and one or more processors;

the memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the audiovisual synchronization method of the visual intercom system of any of claims 1-5.

10. A storage medium containing computer executable instructions for performing the audio video synchronization method of the visual intercom system as in any one of claims 1-5 when executed by a computer processor.

Technical Field

The embodiment of the application relates to the technical field of visual intercom systems, in particular to an audio and video synchronization method, device, equipment and storage medium of a visual intercom system.

Background

At present, along with the improvement of living standard of people, the awareness of prevention of personal and property safety is gradually enhanced. Generally, the building construction of a residential area has a unified security door, and when visitors visit, the visitors call residents to open the doors by pressing down doorbells of the relevant residents. In order to better confirm the visitor's identity, the building visual intercom system is then in transit. The video intercom system is used as a set of modern residential community service measures, two-way video communication between visitors and residents is provided, the visitors and owners can directly communicate through videos and open the anti-theft door locks for the visitors, and therefore double recognition of images and voice is achieved, and safety and reliability are improved.

However, in the existing video intercom system, when the stream is pulled from the server to play the audio and video of the client, the current audio and video display pictures are often out of synchronization, which affects the user experience.

Disclosure of Invention

The embodiment of the application provides an audio and video synchronization method and device of a visual intercom system, electronic equipment and a storage medium, and can well ensure the audio and video synchronization playing of the visual intercom system.

In a first aspect, an embodiment of the present application provides an audio and video synchronization method for a video intercom system, including:

when an audio frame and a video frame are created, obtaining display timestamps of the audio frame and the video frame according to a preset system timestamp starting point, wherein the display timestamps are used for indicating the display sequence of the audio frame and the video frame after decoding;

when the audio frame and the video frame are coded to obtain a compressed data packet, obtaining decoding time stamps of the audio frame and the video frame according to a preset system time stamp starting point, wherein the decoding time stamps are used for indicating the decoding sequence of the audio frame and the video frame;

and pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to a server side.

Further, before obtaining the display timestamps of the audio frame and the video frame according to a preset system timestamp starting point when the audio frame and the video frame are created, the method further includes:

when the system software starts, the current time stamp is recorded and used as the starting point of the system time stamp.

Further, in the obtaining of the display timestamps of the audio frames and the video frames according to the preset system timestamp starting point, performing subtraction according to the timestamps corresponding to the audio frames and the video frames when being created and the timestamps corresponding to the system software when being started to obtain the display timestamps;

correspondingly, in the decoding time stamps of the audio frame and the video frame obtained according to the preset system time stamp starting point, the decoding time stamps are obtained by performing difference according to the time stamps corresponding to the audio frame and the video frame when the audio frame and the video frame are coded and the time stamps corresponding to the audio frame and the video frame when the system software is started.

Furthermore, the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp are pushed to a server side, and the duration information of the compressed data packet is also pushed to the server side.

Further, before pushing the compressed data packet, the corresponding display timestamp, and the corresponding decoding timestamp to the server, the method further includes:

converting the display timestamp, the decode timestamp, and the duration from an encoding time base to a system time base representation.

In a second aspect, an embodiment of the present application provides an audio and video synchronization apparatus for a video intercom system, including:

the system comprises a creating module, a decoding module and a display module, wherein the creating module is used for obtaining display timestamps of an audio frame and a video frame according to a preset system timestamp starting point when the audio frame and the video frame are created, and the display timestamps are used for indicating the display sequence of the audio frame and the video frame after decoding;

the encoding module is used for obtaining decoding time stamps of the audio frames and the video frames according to a preset system time stamp starting point when the audio frames and the video frames are encoded to obtain compressed data packets, and the decoding time stamps are used for indicating the decoding sequence of the audio frames and the video frames;

and the stream pushing module is used for pushing the compressed data packet, the corresponding display timestamp and the corresponding decoding timestamp to a server side.

Specifically, still include:

and the recording module is used for recording the current time stamp when the system software is started, and taking the current time stamp as the starting point of the system time stamp.

Specifically, the creating module performs subtraction according to a timestamp corresponding to the audio frame and the video frame when the audio frame and the video frame are created and a timestamp corresponding to the system software when the system software is started in the display timestamps of the audio frame and the video frame obtained according to a preset system timestamp starting point to obtain the display timestamps;

and the coding module performs difference according to the time stamps corresponding to the audio frame and the video frame when coding is performed on the audio frame and the video frame and the time stamps corresponding to the system software when starting to obtain the decoding time stamps in the process of obtaining the decoding time stamps of the audio frame and the video frame according to the preset system time stamp starting point.

Specifically, the stream pushing module is further configured to push the duration information of the compressed data packet to a server side.

Specifically, still include:

and the conversion module is used for converting the display time stamp, the decoding time stamp and the duration from an encoding time base to a system time base representation.

In a third aspect, an embodiment of the present application provides an electronic device, including:

a memory and one or more processors;

the memory for storing one or more programs;

when the one or more programs are executed by the one or more processors, the one or more processors implement the audio-video synchronization method of the visual intercom system according to the first aspect.

In a fourth aspect, embodiments of the present application provide a storage medium containing computer-executable instructions for performing the audio-video synchronization method of the visual intercom system according to the first aspect when the computer-executable instructions are executed by a computer processor.

According to the method and the device, when the audio frame and the video frame are created, the display time stamp of the audio frame and the display time stamp of the video frame are obtained according to the starting point of the system time stamp, when the audio frame and the video frame are encoded to obtain the compressed data packet, the decoding time stamp of the audio frame and the decoding time stamp of the video frame are obtained according to the starting point of the system time stamp, and finally when the compressed data packet is pushed to flow, the display time stamp and the decoding time stamp information are sent to the server side together. By adopting the technical means, the audio and video compressed data packet can be decoded according to the decoding sequence indicated by the decoding time stamp when the pull stream decoding playing is carried out, and the audio and video data can be played according to the display sequence indicated by the display time stamp, so that the synchronous playing of audio and video pictures can be realized, and the use experience of a user on the visual intercom system can be further optimized.

Drawings

Fig. 1 is a flowchart of an audio and video synchronization method of a video intercom system according to an embodiment of the present application;

fig. 2 is a flowchart of an audio and video synchronization method of another video intercom system provided in the second embodiment of the present application;

fig. 3 is a flowchart of an audio and video synchronization method of another video intercom system provided in the third embodiment of the present application;

fig. 4 is a schematic structural diagram of an audio and video synchronization device of a video intercom system according to a fourth embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application will be described in detail with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

The audio and video synchronization method of the visual intercom system aims to achieve synchronous playing of audio and video when the audio and video data collected by the client side are played in a pull stream mode, and the situation that audio and video pictures are not synchronous is avoided, so that intercom experience of users is influenced. Compared with the existing video intercom system, when audio and video data are played in a pull-stream decoding mode, if the audio and video data are simply played according to the sampling rate of audio and the frame rate of video, audio and video synchronization is difficult to achieve due to the influence of various time difference factors such as the running speed of a machine and the decoding efficiency, and the time difference of the audio and video data can be linearly increased. Therefore, the audio and video playing is asynchronous, and the use experience of the user on the visual intercom system is influenced. Therefore, the audio and video synchronization method of the video intercom system provided by the embodiment of the application indicates the system to decode according to the corresponding decoding sequence and play audio and video according to the corresponding display sequence by recording the display time stamp and the decoding time stamp of each frame of audio frame and video frame, so as to realize the synchronous playing of the audio and video pictures. The problem that the audio and video playing of the existing visual intercom system is not synchronous is solved.