阅读说明：本技术 一种媒体优化方法、装置和存储介质 (Media optimization method, device and storage medium ) 是由 戚兴海 于 2020-06-28 设计创作，主要内容包括：本发明公开一种媒体优化方法、装置和存储介质,属于多流视频会议领域。该方法包括：本发明的技术方案,获取与多点控制单元MCU连接的每个客户端的网络信息；利用每个客户端的网络信息,确定客户端之间的链路信息；当与MCU连接的第一客户端与MCU之间的链路不通畅时,利用由每个客户端的网络信息和客户端之间的链路信息确定的能够传送媒体流的第二客户端,向第一客户端传送媒体流。本发明通过客户端的网络信息确定客户端之间的链路信息,在出现第一客户端与MCU之间的链路不通畅时,转由与第一客户端链路通畅的第二客户端向第一客户端转发MCU下发的媒体流,在弱网环境下,提升了客户端体验,分担了传统MCU压力,解决客户端及时获取媒体流的问题。(The invention discloses a media optimization method, a media optimization device and a storage medium, and belongs to the field of multi-stream video conferences. The method comprises the following steps: according to the technical scheme, network information of each client connected with a Multipoint Control Unit (MCU) is obtained; determining link information between the clients by using the network information of each client; and when the link between the first client connected with the MCU and the MCU is not smooth, transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream. According to the method and the device, the link information between the client sides is determined through the network information of the client sides, when the link between the first client side and the MCU is not smooth, the second client side which is smooth in link with the first client side forwards the media stream issued by the MCU to the first client side, the client side experience is improved under a weak network environment, the traditional MCU pressure is shared, and the problem that the client side acquires the media stream in time is solved.)

1. A method for media optimization, the method comprising:

acquiring network information of each client connected with a Multipoint Control Unit (MCU);

determining link information between the clients by using the network information of each client;

and when the link between the first client connected with the MCU and the MCU is not smooth, transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream.

2. The method of claim 1, wherein the obtaining network information of each client connected to the MCU comprises:

periodically acquiring network information sent by each client; or regularly acquiring the network information sent by each client; or acquiring the network information sent by each client in real time;

the network information sent by each client comprises the IP address of the client, the network segment where the client is located, and the bandwidth and the speed of a link between the client and the MCU.

3. The method of claim 2, wherein determining link information between clients using the network information of each client comprises:

sending a link test request carrying the client IP address of the link to be tested to all other clients connected with the MCU, so that all other clients respectively determine link information between the other clients and the client of the link to be tested according to the client IP address of the link to be tested in the link test request;

and receiving link information between the client side of the link to be tested and all other client sides returned by responding to the link test request.

4. The method of claim 2, wherein before the second client capable of transmitting the media stream determined by the network information of each client and the link information between the clients transmits the media stream to the first client when the link between the first client connected to the MCU and the MCU is not clear, the method further comprises:

and generating a routing link table containing second client information capable of transmitting the media stream for the first client according to the network information sent by each client and the link information between the clients determined by the network information of each client, which is periodically or regularly acquired.

5. The method of claim 4, wherein when a link between a first client connected to the MCU and the MCU is not clear, transmitting the media stream to the first client by using a second client capable of transmitting the media stream determined by the network information of each client and the link information between the clients comprises:

when a link between a first client and an MCU is not smooth, acquiring a routing link table of the first client;

and sending an instruction for transmitting the media stream for the first client to the second client indicated by the routing link table so as to transmit the media stream to the first client by the second client indicated by the routing link table.

6. The method of claim 2, wherein transmitting the media stream to the first client by using a second client capable of transmitting the media stream determined by the network information of each client and the link information between the clients when a link between the first client connected with the MCU and the MCU is not clear comprises:

when a link between a first client and an MCU is not smooth, generating a routing link table containing second client information capable of transmitting media streams for the first client according to network information sent by each client acquired in real time and the link information between the clients determined by the network information of each client acquired in real time;

and sending an instruction for transmitting the media stream for the first client to the second client indicated by the routing link table so as to transmit the media stream to the first client by the second client indicated by the routing link table.

7. The method according to claim 4 or 6, wherein the generating a routing link table for the first client containing second client information enabling the transmission of the media stream for the first client comprises:

determining the unobstructed degree information of the link between each client and the MCU according to the bandwidth and the rate of the link between each client and the MCU;

determining the clients in the same network segment according to the network segment where each client is located;

determining interconnected clients according to link information between the clients;

and generating a routing link table containing second client information capable of transmitting media streams for the first client according to the smoothness degree information of the link between each client and the MCU, the clients in the same network segment and the clients connected with each other.

8. The method of claim 7, wherein the generating a routing link table for a first client, which includes information of a second client capable of transmitting a media stream for the first client, according to information of a degree of smoothness of a link between each client and the MCU, the clients in the same network segment, and the clients connected to each other, comprises:

determining a first client end with an unsmooth link between each client end and the MCU according to the smoothness degree information of the link between each client end and the MCU;

in the clients which are in the same network segment with the first client and the clients which are mutually connected, the client which is not smoothly connected with the first client is searched for the first client to serve as a second client which can transmit media streams for the first client, and a routing link table containing the second client is generated for the first client.

9. An apparatus for media optimization, the apparatus comprising:

the acquisition module is used for acquiring the network information of each client connected with the multipoint control unit MCU;

the determining module is used for determining link information between the clients by utilizing the network information of each client;

and the optimization module is used for transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream when the link between the first client connected with the MCU and the MCU is not smooth.

10. A storage medium for computer readable storage, wherein the storage medium stores one or more programs which are executable by one or more processors to implement the steps of the media optimization method of any one of claims 1 to 8.

Technical Field

The present invention relates to the technical field of multi-stream video conferencing, and in particular, to a media optimization method, apparatus, and storage medium.

Background

In the current multi-stream video conference, the client respectively uploads own video data to the Multipoint Control Unit (MCU), the MCU is distributed to each client, the MCU is equivalent to a transfer station, and each client can only pull the video conference stream to be displayed by requesting the MCU, as shown in fig. 1.

Under the condition, when the network condition between a certain client and the MCU is poor, the speed and the quality of video pulling are greatly influenced, and the client can only wait for pulling the video conference stream.

Disclosure of Invention

The embodiments of the present invention mainly aim to provide a method, an apparatus, and a storage medium for media optimization, which aim to obtain a media stream when a network condition between a certain client and an MCU is poor.

In order to achieve the above object, an embodiment of the present invention provides a media optimization method, where the method includes:

acquiring network information of each client connected with a Multipoint Control Unit (MCU);

determining link information between the clients by using the network information of each client;

and when the link between the first client connected with the MCU and the MCU is not smooth, transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream.

Preferably, the acquiring network information of each client connected to the MCU includes:

periodically acquiring network information sent by each client; or regularly acquiring the network information sent by each client; or acquiring the network information sent by each client in real time;

the network information sent by each client comprises the IP address of the client, the network segment where the client is located, and the bandwidth and the speed of a link between the client and the MCU.

Preferably, the determining link information between the clients by using the network information of each client includes:

sending a link test request carrying the client IP address of the link to be tested to all other clients connected with the MCU, so that all other clients respectively determine link information between the other clients and the client of the link to be tested according to the client IP address of the link to be tested in the link test request;

and receiving link information between the client side of the link to be tested and all other client sides returned by responding to the link test request.

Preferably, before the transmitting the media stream to the first client by using the second client capable of transmitting the media stream determined by the network information of each client and the link information between the clients when the link between the first client connected with the MCU and the MCU is not clear, the method further includes:

and generating a routing link table containing second client information capable of transmitting the media stream for the first client according to the network information sent by each client and the link information between the clients determined by the network information of each client, which is periodically or regularly acquired.

Preferably, when a link between a first client connected to the MCU and the MCU is not clear, transmitting a media stream to the first client by using a second client capable of transmitting a media stream determined by network information of each client and link information between clients includes:

when a link between a first client and an MCU is not smooth, acquiring a routing link table of the first client;

and sending an instruction for transmitting the media stream for the first client to the second client indicated by the routing link table so as to transmit the media stream to the first client by the second client indicated by the routing link table.

Preferably, when a link between a first client connected to the MCU and the MCU is not clear, transmitting a media stream to the first client by using a second client capable of transmitting a media stream determined by network information of each client and link information between clients includes:

when a link between a first client and an MCU is not smooth, generating a routing link table containing second client information capable of transmitting media streams for the first client according to network information sent by each client acquired in real time and the link information between the clients determined by the network information of each client acquired in real time;

and sending an instruction for transmitting the media stream for the first client to the second client indicated by the routing link table so as to transmit the media stream to the first client by the second client indicated by the routing link table.

Preferably, the generating a routing link table containing second client information capable of transmitting the media stream for the first client includes:

determining the unobstructed degree information of the link between each client and the MCU according to the bandwidth and the rate of the link between each client and the MCU;

determining the clients in the same network segment according to the network segment where each client is located;

determining interconnected clients according to link information between the clients;

and generating a routing link table containing second client information capable of transmitting media streams for the first client according to the smoothness degree information of the link between each client and the MCU, the clients in the same network segment and the clients connected with each other.

Preferably, the generating, for the first client, a routing link table including second client information capable of transmitting a media stream for the first client according to the information on the degree of smoothness of a link between each client and the MCU, the clients in the same network segment, and the clients connected to each other includes:

determining a first client end with an unsmooth link between each client end and the MCU according to the smoothness degree information of the link between each client end and the MCU;

in the clients which are in the same network segment with the first client and the clients which are mutually connected, the client which is not smoothly connected with the first client is searched for the first client to serve as a second client which can transmit media streams for the first client, and a routing link table containing the second client is generated for the first client.

In order to achieve the above object, an embodiment of the present invention further provides a media optimization apparatus, where the apparatus includes:

the acquisition module is used for acquiring the network information of each client connected with the multipoint control unit MCU;

the determining module is used for determining link information between the clients by utilizing the network information of each client;

and the optimization module is used for transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream when the link between the first client connected with the MCU and the MCU is not smooth.

To achieve the above object, the present invention provides a storage medium for a computer-readable storage, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the steps of the aforementioned method.

According to the media optimization method, the device and the storage medium provided by the embodiment of the invention, the link information between the client sides is determined through the network information of the client sides, when the link between the first client side and the MCU is not smooth, the second client side which is smooth in link with the first client side forwards the media stream issued by the MCU to the first client side, the client side experience is improved under the weak network environment, the traditional MCU pressure is shared, and the problem that the client side acquires the media stream in time is solved.

Drawings

Fig. 1 is a schematic diagram of a prior art media streaming network.

Fig. 2 is a flowchart of a media optimization method according to an embodiment of the present invention.

Fig. 3 is a schematic structural diagram of a media optimization apparatus according to an embodiment of the present invention.

Fig. 4 is a flow chart of video stream optimization provided by the embodiment of the present invention.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

In the following description, suffixes such as "module", "part", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no peculiar meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.

Example one

As shown in fig. 2, the present embodiment provides a media optimization method, which may include:

step S100: and acquiring the network information of each client connected with the multipoint control unit MCU.

Specifically, the network information sent by each client may be periodically obtained; network information sent by each client can also be obtained at regular time; or the network information sent by each client can be acquired in real time.

The network information sent by each client comprises the IP address of the client, the network segment where the client is located, and the bandwidth and the speed of a link between the client and the MCU.

Step S120: link information between the clients is determined using the network information for each client.

Specifically, a link test request carrying the client IP address of the link to be tested is sent to all other clients connected to the MCU, so that all other clients respectively determine link information with the client of the link to be tested according to the client IP address of the link to be tested in the link test request, and then receive link information with the client of the link to be tested, which is returned by all other clients in response to the link test request.

Step S130: and when the link between the first client connected with the MCU and the MCU is not smooth, transmitting the media stream to the first client by using a second client which is determined by the network information of each client and the link information between the clients and can transmit the media stream.

Specifically, in one embodiment, a routing link table including second client information capable of transmitting a media stream for a first client may be generated in advance for the first client according to network information sent by each client, which is periodically or periodically acquired, and link information between the clients determined by using the network information of each client, which is periodically or periodically acquired. In this way, when the link between the first client and the MCU is not smooth, the pre-generated routing link table of the first client is obtained, and the instruction for transmitting the media stream for the first client is sent to the second client indicated by the routing link table, so that the second client indicated by the routing link table transmits the media stream to the first client. In another embodiment, when a link between a first client and an MCU is not clear, a routing link table containing second client information capable of transmitting a media stream for the first client is generated for the first client according to network information obtained in real time and link information between clients determined by using the network information obtained in real time for each client, and an instruction for transmitting the media stream for the first client is sent to the second client indicated by the routing link table, so that the second client indicated by the routing link table transmits the media stream to the first client.

The method comprises the steps of determining unobstructed degree information of a link between each client and an MCU (microprogrammed control unit) according to the bandwidth and the rate of the link between each client and the MCU, determining clients in the same network segment according to the network segment in which each client is located, determining interconnected clients according to the link information between the clients, determining the clients in the same network segment and the interconnected clients according to the unobstructed degree information of the link between each client and the MCU, generating a routing link table containing second client information capable of transmitting media streams for the first client, specifically, determining the first client with an unobstructed link between the first client and the MCU according to the unobstructed degree information of the link between each client and the MCU, and searching the client with the unobstructed link for the first client as the second client capable of transmitting media streams for the first client in the clients in the same network segment and the interconnected clients And generating a routing link table for the first client including the second client.

According to the media optimization method provided by the embodiment of the invention, the link information between the clients is determined through the network information of the clients, and when the link between the first client and the MCU is not smooth, the second client with the smooth link forwards the media stream issued by the MCU to the second client, so that the client experience is improved, the traditional MCU pressure is shared, and the problem that the client acquires the media stream in time is solved under the weak network environment.

As shown in fig. 3, the present embodiment provides a media optimization apparatus, which may include: an acquisition module 10, a determination module 20 and an optimization module 30. Wherein:

the acquiring module 10 is configured to acquire network information of each client connected to the multipoint control unit MCU, and specifically, the acquiring module 10 may periodically acquire the network information sent by each client; network information sent by each client can also be obtained at regular time; or the network information sent by each client can be obtained in real time, wherein the network information sent by each client comprises the IP address of the client, the network segment where the client is located, and the bandwidth and the speed of the link between the client and the MCU.

The determining module 20 is configured to determine link information between the clients by using network information of each client, and specifically, the determining module 20 sends a link test request carrying a client IP address of a link to be tested to all other clients connected to the MCU, so that all other clients respectively determine link information between the clients of the link to be tested according to the client IP addresses of the links to be tested in the link test request, and then receive link information between the clients of the link to be tested and returned by all other clients in response to the link test request.

And the optimization module 30 is configured to, when a link between a first client connected to the MCU and the MCU is not smooth, transmit the media stream to the first client by using a second client capable of transmitting the media stream, which is determined by the network information of each client and the link information between the clients. Specifically, in an embodiment, the optimization module 30 may generate a routing link table containing second client information capable of delivering a media stream for a first client for the first client in advance according to the network information sent by each client obtained periodically or at regular time and link information between the clients determined by using the network information of each client obtained periodically or at regular time. In this way, when the link between the first client and the MCU is not smooth, the pre-generated routing link table of the first client is obtained, and the instruction for transmitting the media stream for the first client is sent to the second client indicated by the routing link table, so that the second client indicated by the routing link table transmits the media stream to the first client. In another embodiment, when the link between the first client and the MCU is not smooth, the optimization module 30 generates a routing link table containing second client information capable of transmitting the media stream for the first client according to the network information obtained in real time and the link information between the clients determined by using the network information obtained in real time for each client, and sends an instruction for transmitting the media stream for the first client to the second client indicated by the routing link table, so that the second client indicated by the routing link table transmits the media stream to the first client. The method comprises the steps of determining unobstructed degree information of a link between each client and an MCU (microprogrammed control unit) according to the bandwidth and the rate of the link between each client and the MCU, determining clients in the same network segment according to the network segment in which each client is located, determining interconnected clients according to the link information between the clients, determining the clients in the same network segment and the interconnected clients according to the unobstructed degree information of the link between each client and the MCU, generating a routing link table containing second client information capable of transmitting media streams for the first client, specifically, determining the first client with an unobstructed link between the first client and the MCU according to the unobstructed degree information of the link between each client and the MCU, and searching the client with the unobstructed link for the first client as the second client capable of transmitting media streams for the first client in the clients in the same network segment and the interconnected clients And generating a routing link table for the first client including the second client.

According to the media optimization device provided by the embodiment of the invention, the link information between the clients is determined through the network information of the clients, and when the link between the first client and the MCU is not smooth, the second client with the smooth link forwards the media stream issued by the MCU to the second client, so that the client experience is improved, the traditional MCU pressure is shared, and the problem that the client acquires the media stream in time is solved under the weak network environment.

The following describes a common video conference scenario in detail.

As shown in fig. 4, the steps of implementing media optimization may include:

step 401: after the client enters the meeting, the network link condition (equivalent to the network information in the foregoing embodiment) of the client is uploaded to the service server at intervals, including the information of the ip, the network segment, the bandwidth, the rate, and the like of the client.

Step 402: and after collecting the relevant information of the client, the service server forms a data structure for operation and stores the relevant information of the client. Then, by issuing the ip address of the client needing to test the link to other clients, the link connection condition between the clients (equivalent to the link information between the clients in the foregoing embodiment) is retrieved.

Step 403: and establishing a routing link list of a plurality of clients according to the acquired connection condition of each client and by combining network segment and bandwidth information.

The route link table of the client can be generated according to the following principle of generating the route link table of the client: and step 1, classifying according to the connection state between the client and the MCU, and classifying into ABC three types according to the sequence of the smooth degree of a link from good to bad. And 2, classifying according to the connection state between the clients, classifying the same network segment into one class, classifying the rest of the network segments according to the smoothness degree of the links between the clients, and not classifying the network segments which are matched with each other. 3. According to the classification results of the two steps, searching the classification results generated in the step 2 for the C-type client (namely the client which is not smooth to the MCU link) in the step 1 to find the client which is smooth to be connected with the C-type client, and if the corresponding client is A-type in the step 1, confirming the routing relation between the client and the client; if the client is not in the class A, continuously searching the next class C client until the traversal is completed; if none, continue the lookup in class B until the routing relationship is confirmed.

Step 404: when detecting that the connection between the client a and the MCU is not smooth, the client a reports to the service server, and the service server distributes an instruction to the corresponding client according to the generated routing link table, where, for example, the link related to a is: and A- - -B- - -C, B acquires the video resource required by A from C, and A acquires from B.

Step 405: and the service server timely adjusts the routing link lists of the rest clients according to the link distribution and execution conditions.

The client of the embodiment can be a video conference client (soft terminal), and the method of the embodiment can improve the video experience of the client and share the pressure of the traditional MCU under the weak network environment.

For example: suppose that there are 6 clients participating in the session at the same time, which are respectively the codes a, b, c, d, e, f. The following treatment is carried out according to the principle:

1. the following three categories are classified according to the degree of patency with the MCU:

a is a and b;

b is c and d;

class C: e. f;

2. the connection states between the clients are classified into the following categories:

the method is divided into the following steps according to the same network segment: a. b, f and c, d, e;

the connection condition of each of them is divided into: a. d, f and b, c, e;

3. according to the link table generation rule, the generated link is the following condition:

the worst link between the current MCU and the MCU is e and f;

e and d are in the same network segment, the link conditions of d and a are good, a is the most unobstructed one connected with the MCU, and the routing table of e is e- - > d- - > a;

obtaining a routing table corresponding to the f as f- - > b in the same way;

thus, when e sends a request of link failure to the service server, the service server informs a to forward the video resource to d, d to forward the video resource to e, and simultaneously stops the MCU from transmitting the resource between d and e.

On the basis of the embodiment shown in fig. 4, the MCU may also notify the service server of its own load condition and the connection condition with the client through communication with the service server, so as to help the service server to perfect link information, so that the MCU shares a part of pressure with the client in this way when its own load is overloaded.

In the embodiment, the link condition between the clients is judged by uploading the network condition of the client in real time, and when the link between the client and the MCU is not smooth, the video stream issued by the MCU is forwarded to the client by other clients which have smooth networks with the client, so that the problem of video acquisition of the client is solved in time. The method is applied to the multi-stream conference, not only can the MCU share part of pressure to the client in the mode when the load of the MCU is overloaded, but also other clients with unobstructed network links can be searched to acquire videos in the mode when the client pulls the videos laboriously.

It should be noted that the bandwidth required by the current scheme is large, and in the 5G era, when the network speed and the bandwidth are no longer the limiting conditions, the service server in the scheme can theoretically replace the MCU.

Embodiments of the present invention also provide a storage medium, which stores one or more programs that can be executed by one or more processors to implement the steps of the aforementioned media optimization method.

One of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof.

In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.

The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and are not to be construed as limiting the scope of the invention. Any modifications, equivalents and improvements which may occur to those skilled in the art without departing from the scope and spirit of the present invention are intended to be within the scope of the claims.