阅读说明：本技术 一种多屏网真终端的数据传输方法和装置 (Data transmission method and device for multi-screen telepresence terminal ) 是由 张斌超 于 2020-12-29 设计创作，主要内容包括：本发明公开了一种多屏网真终端的数据传输方法,所述多屏网真终端包括主屏幕设备和至少一个从屏幕设备,所述方法包括：所述主屏幕设备将来自所述从屏幕设备的第一数据加上预设的第一数据包头,获得第二数据；并向多点控制单元发送所述第二数据；其中,所述第二数据包括所述第一数据和预设的所述第一数据包头；所述主屏幕设备接收所述多点控制单元发送的第三数据,对所述第三数据预设的第二数据包头进行解析,得到第四数据；并将所述第四数据发送至所述从屏幕设备；其中,所述第三数据为其他终端发送至所述多点控制单元的数据。本发明能够解决现有技术中多屏网真终端在视频传输中,多个屏幕之间出现视频传输延时,不能同步的问题。(The invention discloses a data transmission method of a multi-screen telepresence terminal, wherein the multi-screen telepresence terminal comprises a main screen device and at least one slave screen device, and the method comprises the following steps: the main screen equipment adds a preset first data packet header to first data from the auxiliary screen equipment to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header; the main screen equipment receives third data sent by the multipoint control unit, and analyzes a second data packet header preset by the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals. The invention can solve the problem that in the prior art, video transmission delay occurs among a plurality of screens in video transmission of the multi-screen telepresence terminal, and synchronization cannot be realized.)

1. A data transmission method of a multi-screen telepresence terminal is characterized in that the multi-screen telepresence terminal comprises a main screen device and at least one slave screen device, and the method comprises the following steps:

the main screen equipment adds a preset first data packet header to first data from the auxiliary screen equipment to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

the main screen equipment receives third data sent by the multipoint control unit, and analyzes a second data packet header preset by the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

2. The data transmission method of claim 1, further comprising, after the primary screen device sending the second data to a multipoint control unit:

the multipoint control unit analyzes the first data packet header preset by the second data; and sending the analyzed data to other terminals.

3. The data transmission method according to claim 2, wherein the multipoint control unit comprises a management control layer and a media processing layer, the media processing layer comprises an apparent forwarding service and a media processing service; wherein the content of the first and second substances,

the visual forwarding service is used for analyzing the first data packet header preset by the second data; sending the analyzed data to other terminals;

the media processing service is used for coding the data sent by the single-screen terminal and sending the coded data to the visual forwarding service;

the visual forwarding service is also used for receiving the data from the media processing service to perform packet processing and sending the data after the packet processing to the main equipment.

4. The data transmission method according to claim 1 or 2, wherein the other terminal includes: multi-screen terminals and single-screen terminals.

5. The utility model provides a data transmission device of many screen real terminals, its characterized in that, many screen real terminals include main screen equipment and at least one from screen equipment, the device includes: a transmitting module and a receiving module; wherein the content of the first and second substances,

the sending module is used for the main screen device to add a preset first data packet header to the first data from the secondary screen device to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

the receiving module is configured to receive third data sent by the multipoint control unit, and analyze a second data packet header preset in the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

6. The data transmission apparatus according to claim 5, wherein after the sending module completes executing, the multipoint control unit parses the first data packet header preset by the second data; and sending the analyzed data to other terminals.

7. The data transmission apparatus according to claim 5, wherein the multipoint control unit comprises a management control layer and a media processing layer, the media processing layer comprising an apparent forwarding service and a media processing service; wherein the content of the first and second substances,

the visual forwarding service is used for analyzing the first data packet header preset by the second data; sending the analyzed data to other terminals;

the media processing service is used for coding the data sent by the single-screen terminal and sending the coded data to the visual forwarding service;

the visual forwarding service is also used for receiving the data from the media processing service to perform packet processing and sending the data after the packet processing to the main equipment.

8. The data transmission apparatus according to claim 5 or 6, wherein the other terminal comprises: multi-screen terminals and single-screen terminals.

9. A computer terminal device, comprising:

one or more processors;

a memory coupled to the processor for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement a data transmission method as claimed in any one of claims 1 to 4.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data transmission method according to any one of claims 1 to 4.

Technical Field

The invention relates to the technical field of telepresence, in particular to a method and a device for transmitting data of a multi-screen telepresence terminal, a computer terminal device and a computer readable storage medium.

Background

Telepresence (Telepresence) is a technology that creates an Immersive virtual meeting environment. Compared with conventional video conferencing systems, the Telepresence system has many advantages, including: the image of the size of a real person can be provided, the eye-mind communication effect is achieved, the movement is smoother, and the accurate limb behaviors of the remote participants are achieved; high-definition, studio-level video, lighting and audio effects; the unified meeting environment enables the participants to feel in the same meeting place, and the consistency of experience of different meeting places is ensured; and conference equipment such as a camera is hidden, so that the influence on a user is reduced, and the like.

At present, the telepresence technology is generally applied to a multi-screen telepresence terminal, and in the prior art, video transmission among a plurality of screens of the multi-screen telepresence terminal is delayed, so that the video transmission cannot be synchronized.

Disclosure of Invention

The purpose of the invention is: the data transmission method, the computer terminal device and the computer readable storage medium of the multi-screen telepresence terminal are provided, and the problem that video transmission delay among a plurality of screens cannot be synchronized in video transmission of the existing multi-screen telepresence terminal can be solved.

In order to achieve the above object, the present invention provides a data transmission method for a multi-screen telepresence terminal, where the multi-screen telepresence terminal includes a master screen device and at least one slave screen device, the method includes:

the main screen equipment adds a preset first data packet header to first data from the auxiliary screen equipment to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

the main screen equipment receives third data sent by the multipoint control unit, and analyzes a second data packet header preset by the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

Further, after the home screen device transmits the second data to the multipoint control unit, the method further includes: the multipoint control unit analyzes the first data packet header preset by the second data; and sending the analyzed data to other terminals.

Further, the multipoint control unit comprises a management control layer and a media processing layer, wherein the media processing layer comprises an apparent forwarding service and a media processing service; the visual forwarding service is used for analyzing the preset first data packet header of the second data; sending the analyzed data to other terminals; the media processing service is used for coding the data sent by the single-screen terminal and sending the coded data to the visual forwarding service; the visual forwarding service is also used for receiving the data from the media processing service to perform packet processing and sending the data after the packet processing to the main equipment.

Further, the other terminals include: multi-screen terminals and single-screen terminals.

An embodiment of the present invention further provides a data transmission device for a multi-screen telepresence terminal, where the multi-screen telepresence terminal includes a master screen device and at least one slave screen device, and the device includes: a transmitting module and a receiving module; wherein the content of the first and second substances,

the sending module is used for the main screen device to add a preset first data packet header to the first data from the secondary screen device to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

the receiving module is configured to receive third data sent by the multipoint control unit, and analyze a second data packet header preset in the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

Further, after the sending module completes execution, the multipoint control unit analyzes the first data packet header preset by the second data; and sending the analyzed data to other terminals.

Further, the multipoint control unit comprises a management control layer and a media processing layer, wherein the media processing layer comprises an apparent forwarding service and a media processing service; the visual forwarding service is used for analyzing the preset first data packet header of the second data; sending the analyzed data to other terminals; the media processing service is used for coding the data sent by the single-screen terminal and sending the coded data to the visual forwarding service; the visual forwarding service is also used for receiving the data from the media processing service to perform packet processing and sending the data after the packet processing to the main equipment.

Further, the other terminals include: multi-screen terminals and single-screen terminals.

An embodiment of the present invention further provides a computer terminal device, including: one or more processors; a memory coupled to the processor for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a data transmission method as in any one of the above.

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data transmission method according to any one of the above items.

Compared with the prior art, the data transmission method and the data transmission device for the multi-screen telepresence terminal have the advantages that:

1. the visual forwarding service realizes the forwarding between the terminal and the terminal, and reduces the time delay and the performance of the server.

2. The media processing service and the video forwarding service are integrated, so that the intelligent service and some conference control can not influence the time delay between the terminals and the performance of the server.

3. The multi-screen real terminal docking server only docks through main screen equipment, so data all send through main screen equipment, guaranteed the delay synchronization problem of many screens.

Drawings

Fig. 1 is a schematic flowchart of a data transmission method for a multi-screen telepresence terminal according to an embodiment of the present invention;

fig. 2 is a schematic diagram of a three-screen telepresence terminal architecture according to an embodiment of the present invention.

Fig. 3 is a schematic view of an MCU total architecture in a data transmission method for a multi-screen fax terminal according to an embodiment of the present invention;

fig. 4 is a schematic diagram of data transmission between a three-screen telepresence terminal and a telepresence forwarding service in a telepresence conference according to an embodiment of the present invention;

fig. 5 is a schematic diagram of data transmission when two three-screen telepresence terminals exist in a telepresence conference according to an embodiment of the present invention;

fig. 6 is a schematic diagram of data transmission when two three-screen telepresence terminals and an intelligent switch are present in a telepresence conference according to an embodiment of the present invention;

fig. 7 is a schematic diagram of data transmission when a three-screen telepresence terminal and a single-screen telepresence terminal exist in a telepresence conference according to an embodiment of the present invention;

fig. 8 is a schematic diagram of data transmission when a three-screen telepresence terminal enters a conference in a telepresence conference according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a data transmission device of a multi-screen fax terminal according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be understood that the step numbers used herein are for convenience of description only and are not intended as limitations on the order in which the steps are performed.

It is to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

The terms "comprises" and "comprising" indicate the presence of the described features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The term "and/or" refers to and includes any and all possible combinations of one or more of the associated listed items.

Interpretation of terms:

MCU: a multipoint control unit;

MPS: media processing services (MCU internal services);

TPS: a visual forwarding service (MCU internal service);

freeswitch: a soft-switching solution for telephony;

TELE: the full-spelling telepresence means a multi-screen telepresence terminal.

The first embodiment of the present invention:

referring to fig. 1 to 8, a data transmission method for a multi-screen telepresence terminal according to an embodiment of the present invention includes a master screen device and at least one slave screen device, where the method includes at least the following steps:

s101, adding a preset first data packet header to first data from the slave screen equipment by the master screen equipment to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

it should be noted that the multipoint control unit includes: an access Stratum (SGW), a management control layer (RMC) and a media processing layer (MPS, TPS); all resources of the media processing layer are controlled by the management control layer services. MPS is responsible for the access of the common terminal, and TPS is responsible for the access of the multi-screen real network terminal. MPS and TPS are deployed in a multi-master form, and resource management is unified by RMC.

It should be noted that, in the MCU cluster, the server includes:

sign GateWay (SGW): and the media interface gateway and the MCU are externally connected, adopt an RESTful Api form and are responsible for processing external service registration, authentication, callback and the like, convert an external interface protocol into an internal protocol, distribute the request to different services at the rear end of the MCU, and make stateless and transversely expand the capacity.

Reliable Media Controller (RMC): and the media conference control service is responsible for the functions of media resource allocation and scheduling, stacking relation establishment, conversation establishment between the TPS and the MPS and the like. The media processing service is deployed in a single-master mode (disaster recovery through a database), all conferences are established on the RMC, and when a call is incoming, which media processing service is specifically selected is determined according to the type of the incoming call.

Media Process Service (MPS): and the media processing service is responsible for the audio and video coding and decoding functions of the common terminal.

Telepresence Process Service (TPS): and the multi-screen real-network processing service is responsible for accessing the multi-screen real-network terminal and is in butt joint with the MPS when required.

It should be noted that, the selection of the main screen device and the slave screen device of the multi-screen telepresence terminal is selected by the user according to actual needs, for example: for a three-screen telepresence terminal, the screen in the middle is generally used as a main screen device, and the screens on both sides are used as slave screen devices.

It should be noted that, the preset first and second packet headers replace the original RTP header in the RTP format, and the original RTP packet header of the data is reassembled, so as to reassemble the key information and the custom information.

The key information includes SSRC, SeqNumber, time stamp, and other information required by the media. The customization information refers to some parameters that are needed for media processing, including what device is on its own, and the sequence number.

The transmitted data includes video and audio.

S102, the main screen equipment receives third data sent by the multipoint control unit, and analyzes a second data packet header preset by the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

It should be noted that, for the data from the multipoint control unit, the master screen device may choose to decode the data and then send the decoded data to the slave screen device; or the data is directly sent to the slave screen device, and the slave screen device carries out decoding processing by itself.

In one embodiment of the present invention, after the home screen device sends the second data to the multipoint control unit, the method further includes: the multipoint control unit analyzes the transmission protocol header of the second data; and sending the analyzed data to other terminals.

It should be noted that, since the main screen device transmits the second data to the multipoint control unit via the transmission protocol, the multipoint control unit needs to parse a header portion of the generated transmission protocol and then directly transmit the data to the other terminal.

In one embodiment of the present invention, the multipoint control unit includes a management control layer and a media processing layer, and the media processing layer includes a visual forwarding service and a media processing service; wherein the visual forwarding service parses a transport protocol header of the second data; sending the analyzed data to other terminals; and the media processing service encodes the data sent by the single-screen terminal and sends the encoded data to the visual transmitting service.

It should be noted that, in the data transmission process, data transmission of the multi-screen terminal is completed by the visual forwarding service, and data transmission of the single-screen terminal is completed by the media processing service; when the single-screen terminal transmits the data to the multi-screen terminal, the media processing service needs to process the data and then sends the processed data to the video and real forwarding service, and the video and real forwarding service sends the data to the multi-screen terminal.

In an embodiment of the present invention, the other terminal includes: multi-screen terminals and single-screen terminals.

It should be noted that the present invention is applicable to data transmission of a multi-screen telepresence terminal, and for better understanding of the present invention, the following specific example will be described by using a three-screen telepresence terminal in a conference, but the present invention is not limited to use in a three-screen telepresence terminal.

The three-screen telepresence terminal consists of 3 single-screen terminals, the three single-screen terminals are defined as left, middle and right, the MCU interacts with the main equipment when calling in the MCU, the RTP data left and right terminals are respectively sent to the main equipment, then the main equipment adds a preset first data packet header and sends the data to the MCU, the MCU sends the data to the terminal and only sends the data to the main equipment, then the main equipment analyzes a preset second data packet header, and then the data are forwarded to the left and right equipment or decoded by the main equipment, and the following description is performed on the possible conditions in the conference:

1. in the telepresence conference, under the condition that one telepresence conference has only 2 three-screen telepresence terminals, data of the three-screen telepresence terminals only pass through the TPS, after the TPS receives the data of the terminal 1, an RTP header is analyzed and then directly forwarded to the other three-screen telepresence terminal, and the middle part does not participate in encoding and decoding, so that the performance and the time delay are saved.

2. When there are only 2 three-screen telepresence terminals in a conference and some intelligent service switches (recording, voice recognition and intelligent sign-in), 3 channels are respectively opened for the terminal 1 on the TPS and the MPS on the original forwarding logic, 3 channels are opened for the terminal 2, sendOnly is opened on the TPS and recvOnly is opened on the MPS, when the TPS receives the data of the terminal, a preset first data packet header is firstly analyzed, then the data is forwarded to another terminal, and meanwhile, one piece of data is also sent to the MPS. All channels opened on the MPS side are considered as normal terminals and then the content is decoded and processed differently. Thus, the delay of 2 visual terminals is not increased, and unnecessary coding consumption is not increased.

3. When only one three-screen real network terminal accesses the meeting, data can be sent to the TPS from the terminal, then forwarded to the MPS by the TPS, sent to the TPS after being coded and decoded by the MPS, and then forwarded to the three-screen real network terminal.

4. When a three-screen real meeting and a single-screen terminal meet, the single-screen terminal is directly accessed to the MPS, and the three-screen real meeting terminal is accessed to the TPS. After receiving the data, the single-screen terminal sends the data to the TPS after MPS coding and decoding. Because the three-screen telepresence terminal has 3 paths, and the single-screen terminal has only 1 path, the data of the single-screen terminal is sent to the main channel, and the left channel and the right channel send static picture data; the picture seen by the terminal can be a picture formed by splicing three screens or a main picture, and the picture depends on the layout set by the user.

The process of the terminal entering the telepresence conference can be specifically summarized as follows: the method comprises the steps that a terminal access sends Sip Invite signaling to a Freeswitch (FS for short), the Freeswitch is converted into signaling application resources and sent to an SGW in an MCU (microprogrammed control Unit), the signaling application resources are sent to an RMC after the SGW is analyzed, the RMC applies for media resources to MPS and TPS respectively, the media resources are returned to the FS after the media resources are applied, the FS sends signaling for opening a channel to the SGW, the RMC forwards the signaling after the SGW is analyzed, and the RMC opens a specific channel according to the condition of the terminal in a current conference.

Compared with the prior art, the data transmission method of the multi-screen telepresence terminal has the beneficial effects that:

1. the visual forwarding service realizes the forwarding between the terminal and the terminal, and reduces the time delay and the performance of the server.

2. The media processing service and the video forwarding service are integrated, so that the intelligent service and some conference control can not influence the time delay between the terminals and the performance of the server.

3. The multi-screen real terminal docking server only docks through main screen equipment, so data all send through main screen equipment, guaranteed the delay synchronization problem of many screens.

Second embodiment of the invention:

referring to fig. 9, a data transmission apparatus 200 of a multi-screen telepresence terminal according to an embodiment of the present invention includes a master screen device and at least one slave screen device, and the apparatus includes: a sending module 201 and a receiving module 202; wherein the content of the first and second substances,

the sending module 201 is configured to add a preset first data packet header to the first data from the slave screen device by the master screen device to obtain second data; and sending the second data to a multipoint control unit; the second data comprises the first data and a preset first data packet header;

the receiving module 202 is configured to receive third data sent by the multipoint control unit, and analyze a second data packet header preset in the third data to obtain fourth data; and sending the fourth data to the slave screen device; and the third data is data sent to the multipoint control unit by other terminals.

In one embodiment of the present invention, after the sending module completes execution, the multipoint control unit parses the first data packet header preset in the second data; and sending the analyzed data to other terminals.

In one embodiment of the present invention, the multipoint control unit includes a management control layer and a media processing layer, and the media processing layer includes a visual forwarding service and a media processing service; the visual forwarding service is used for analyzing the preset first data packet header of the second data; sending the analyzed data to other terminals; the media processing service is used for coding the data sent by the single-screen terminal and sending the coded data to the visual forwarding service; the visual forwarding service is also used for receiving the data from the media processing service to perform packet processing and sending the data after the packet processing to the main equipment.

In an embodiment of the present invention, the other terminal includes: multi-screen terminals and single-screen terminals.

Compared with the prior art, the data transmission device of the multi-screen telepresence terminal has the advantages that:

1. the visual forwarding service realizes the forwarding between the terminal and the terminal, reduces the time delay and the performance of the server

2. The media processing service and the video forwarding service are integrated, so that the intelligent service and some conference control can not influence the time delay between the terminals and the performance of the server.

3. The multi-screen real terminal docking server only docks through main screen equipment, so data all send through main screen equipment, guaranteed the delay synchronization problem of many screens.

Third embodiment of the invention:

the embodiment of the present invention further provides a computer-readable storage medium including program instructions, where the program instructions, when executed by a processor, implement the steps of the data transmission method for a multi-screen real terminal according to any one of the above embodiments. For example, the computer readable storage medium may be the above memory including program instructions, and the program instructions may be executed by a processor of the computer terminal device to implement the data transmission method of the multi-screen telepresence terminal according to any one of the above embodiments, and achieve the technical effects consistent with the above method.

The fourth embodiment of the present invention:

the computer-readable storage medium provided by the embodiment of the invention stores a computer program thereon, and the computer program, when executed by a processor, implements the data transmission method of the multi-screen real terminal according to any one of the above items.

The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and substitutions can be made without departing from the technical principle of the present invention, and these modifications and substitutions should also be regarded as the protection scope of the present invention.