阅读说明：本技术 一种视频信息传输方法、装置、电子设备及存储介质 (Video information transmission method and device, electronic equipment and storage medium ) 是由 肖建军 刘泓昊 韩瑞 张富才 于 2021-09-01 设计创作，主要内容包括：本发明提供了一种视频信息传输方法、装置、电子设备及存储介质,方法包括：对网络传输层中的不同视频信息传输通道进行监测,确定不同视频信息传输通道的状态；当网络传输层中的帧往返时延参数增加,并且超过帧往返时延参数阈值时,触发对视频播放终端的视频信息传输通道进行切换,将第一视频信息传输通道切换至目标视频信息传输通道；在不同视频信息传输通道中选择目标视频信息传输通道；通过目标视频信息传输通道,对视频信息进行传输,由此,选择最适合的目标视频信息传输通道对视频信息进行传输,以实现视频信息在网络传输层中的完整传输,可以减少由于网络环境的变化和终端解码能力的不足所造成的视频播放卡顿,提升用户的使用体验。(The invention provides a video information transmission method, a video information transmission device, electronic equipment and a storage medium, wherein the method comprises the following steps: monitoring different video information transmission channels in a network transmission layer, and determining the states of the different video information transmission channels; when the frame round-trip delay parameter in the network transmission layer is increased and exceeds the frame round-trip delay parameter threshold value, triggering to switch the video information transmission channel of the video playing terminal, and switching the first video information transmission channel to the target video information transmission channel; selecting a target video information transmission channel from different video information transmission channels; the video information is transmitted through the target video information transmission channel, and therefore the most suitable target video information transmission channel is selected to transmit the video information, so that the complete transmission of the video information in a network transmission layer is realized, video playing blockage caused by the change of a network environment and the insufficiency of terminal decoding capacity can be reduced, and the use experience of a user is improved.)

1. A method for transmitting video information, the method comprising:

monitoring the state of a network transmission layer of a video playing terminal to obtain a state change result of the network transmission layer;

monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determining the states of the different video information transmission channels;

when the frame round-trip delay parameter in the network transmission layer is increased and exceeds the frame round-trip delay parameter threshold value, triggering to switch the video information transmission channel of the video playing terminal, and switching the first video information transmission channel to a target video information transmission channel;

and transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer.

2. The method according to claim 1, wherein the monitoring different video information transmission channels in the network transport layer based on the change result of the state of the network transport layer to determine the state of the different video information transmission channels comprises:

when the video information transmission channel is a first video information transmission channel,

acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer through the video information transmitted by the first video information transmission channel;

and determining the state of a first video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter.

3. The method according to claim 1, wherein the monitoring different video information transmission channels in the network transport layer based on the change result of the state of the network transport layer to determine the state of the different video information transmission channels comprises:

when the video information transmission channel is a second video information transmission channel,

configuring a keep-alive message for the second video information transmission channel;

acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer in the second video information transmission channel through the keep-alive message;

and determining the state of a second video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter.

4. The method of claim 1, wherein said selecting a target video information transmission channel among said different video information transmission channels in response to a switching instruction of a video information transmission channel comprises:

when a first video information transmission channel of the different video information transmission channels is in an unavailable state,

determining the state of a second video information transmission channel according to a frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter of the second video information transmission channel;

and determining that the second video information transmission channel is a target video information transmission channel according to the state of the second video information transmission channel.

5. The method of claim 1, wherein said selecting a target video information transmission channel among said different video information transmission channels in response to a switching instruction of a video information transmission channel comprises:

when a first video information transmission channel of the different video information transmission channels is in an available state and the state of a second video information transmission channel in a low bit rate scene can be acquired,

configuring the same test video frame set for the first video information transmission channel and the second video information transmission channel;

respectively testing the first video information transmission channel and the second video information transmission channel through the test video frame set to obtain a test result of the video information transmission channels;

and determining a target video information transmission channel according to the test result of the video information transmission channel.

6. The method according to claim 1, wherein said transmitting video information through said target video information transmission channel to achieve complete transmission of said video information in a network transport layer comprises:

when it is determined that the second video information transmission channel is the target video information transmission channel,

retransmitting the buffered video frames in the second video information transmission channel;

receiving the retransmitted video frames in the video playing terminal, performing disorder recombination processing on the received video frames, and deleting repeated video frames;

video information is transmitted that has undergone deletion of duplicate video frames.

7. The method of claim 1, further comprising:

acquiring a video type identifier transmitted by a network transmission layer of the video playing terminal;

when the video type identification is used for representing a cloud game video, configuring a WIFI-2.4G type information transmission channel and a WIFI-5G type information transmission channel for a network transmission layer of the terminal;

when the video type identification is used for representing a live video, configuring a cellular communication type information transmission channel and a WIFI-5G type information transmission channel for a network transmission layer of the terminal;

and when the video type identifier is used for representing the short video, configuring a WIFI-2.4G type information transmission channel, a WIFI-5G type information transmission channel and a cellular communication type information transmission channel for a network transmission layer of the terminal.

8. The method of claim 1, further comprising:

when the target video is the cloud game video, determining identification information of a target user;

determining a data source cluster matched with the identification information based on a cloud server network;

according to the data source cluster, storing cloud game history information of a user matched with the target user into a cloud game server;

and selecting a target video information transmission channel from different video information transmission channels based on the cloud game historical information.

9. The method of claim 1, further comprising:

when a target video information transmission channel is selected from the different video information transmission channels, analyzing the video information to be transmitted to obtain a priority identifier of the video information to be transmitted;

sequencing the priority of the received video information to be transmitted according to the priority identification of the video information to be transmitted;

establishing a corresponding video information queue to be transmitted according to the priority of the video information to be transmitted;

and adjusting the network resource allocation of the target video information transmission channel according to the priority sequence of the video information to be transmitted.

10. The method according to claim 9, wherein the adjusting the network resource configuration of the target video information transmission channel according to the priority ranking of the video information to be transmitted comprises:

traversing a video information queue to be transmitted, and determining the video information to be transmitted with the highest priority;

determining link quality of each link in the network resources;

and configuring the link with the highest link quality in the network resources of the target video information transmission channel for the video information to be transmitted with the highest priority so as to process the video information to be transmitted in the video information queue to be transmitted through the configured link.

11. A video information transmission apparatus, characterized in that the apparatus comprises:

the information transmission module is used for monitoring the change of a network transmission layer of the video playing terminal;

the information processing module is used for monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer and determining the states of the different video information transmission channels;

the information processing module is used for triggering the switching of a video information transmission channel of the video playing terminal when the frame round-trip delay parameter in the network transmission layer is increased and exceeds a frame round-trip delay parameter threshold value, and switching the first video information transmission channel to a target video information transmission channel;

the information processing module is used for responding to a switching instruction of a video information transmission channel and selecting a target video information transmission channel from different video information transmission channels;

and the information processing module is used for transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer.

12. The apparatus of claim 11,

the information processing module is used for acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer through video information transmitted by the first video information transmission channel when the video information transmission channel is the first video information transmission channel;

and the information processing module is used for determining the state of a first video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter.

13. An electronic device, characterized in that the electronic device comprises:

a memory for storing executable instructions;

a processor for implementing the video information transmission method of any one of claims 1 to 10 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, implement the video information transmission method of any one of claims 1 to 10.

Technical Field

The present invention relates to video information transmission technologies, and in particular, to a video information transmission method and apparatus, an electronic device, and a storage medium.

Background

With the development of cloud games in related technologies, the number of cloud games deployed in a cloud server is increasing, but most of video forwarding servers of the cloud games process video information on the basis of a single channel. When the network shakes, the network blocking can be recovered only by discarding data and reducing code rate, so that some users with unstable network or limited decoding calculation power cannot timely adjust the transmission of video information according to the change of the network when running the cloud game, and the fluency of the use of the users is responded.

Disclosure of Invention

In view of this, embodiments of the present invention provide a video information transmission method, an apparatus, an electronic device, and a storage medium, which can monitor different video information transmission channels in a network transmission layer, determine states of the different video information transmission channels, and select a most suitable target video information transmission channel to transmit video information, so as to implement complete transmission of the video information in the network transmission layer, thereby reducing video playing jams caused by changes in a network environment and insufficient decoding capability of a terminal, and improving user experience.

The technical scheme of the embodiment of the invention is realized as follows:

the embodiment of the invention provides a video information transmission method, which comprises the following steps:

monitoring the change of a network transmission layer of the video playing terminal;

monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determining the states of the different video information transmission channels;

when the frame round-trip delay parameter in the network transmission layer is increased and exceeds a frame round-trip delay parameter threshold value, triggering to switch the video information transmission channel of the video playing terminal, and switching the first video information transmission channel to a target video information transmission channel;

responding to a switching instruction of a video information transmission channel, and selecting a target video information transmission channel from different video information transmission channels;

and transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer.

An embodiment of the present invention further provides a video information transmission apparatus, including:

the information transmission module is used for monitoring the change of a network transmission layer of the video playing terminal;

the information processing module is used for monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer and determining the states of the different video information transmission channels;

the information processing module is used for triggering the switching of a video information transmission channel of the video playing terminal when the frame round-trip delay parameter in the network transmission layer is increased and exceeds a frame round-trip delay parameter threshold value, and switching the first video information transmission channel to a target video information transmission channel;

the information processing module is used for responding to a switching instruction of a video information transmission channel and selecting a target video information transmission channel from different video information transmission channels;

and the information processing module is used for transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer through video information transmitted by the first video information transmission channel when the video information transmission channel is the first video information transmission channel;

and the information processing module is used for determining the state of a first video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for configuring a keep-alive message for the second video information transmission channel when the video information transmission channel is the second video information transmission channel;

the information processing module is used for acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer in the second video information transmission channel through the keep-alive message;

and the information processing module is used for determining the state of a second video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for determining the state of a second video information transmission channel according to a frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter of the second video information transmission channel when a first video information transmission channel in the different video information transmission channels is in an unavailable state;

and the information processing module is used for determining that the second video information transmission channel is a target video information transmission channel according to the state of the second video information transmission channel.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for acquiring the state of a second video information transmission channel in a low bit rate scene when a first video information transmission channel in the different video information transmission channels is in an available state,

configuring the same test video frame set for the first video information transmission channel and the second video information transmission channel;

the information processing module is used for respectively testing the first video information transmission channel and the second video information transmission channel through the test video frame set to obtain a test result of the video information transmission channel;

and the information processing module is used for determining a target video information transmission channel according to the test result of the video information transmission channel.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for determining that the second video information transmission channel is the target video information transmission channel,

retransmitting the buffered video frames in the second video information transmission channel;

the information processing module is used for receiving the retransmitted video frames in the video playing terminal and carrying out disorder recombination processing on the received video frames so as to delete repeated video frames in the retransmitted video frames.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for determining the identification information of the target user when the target video is the cloud game video;

the information processing module is used for determining a data source cluster matched with the identification information based on a cloud server network;

the information processing module is used for storing cloud game historical information of the user matched with the target user into a cloud game server according to the data source cluster;

and the information processing module is used for selecting a target video information transmission channel from different video information transmission channels based on the historical cloud game information.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for analyzing the video information to be transmitted and acquiring the priority identification of the video information to be transmitted when a target video information transmission channel is selected from the different video information transmission channels;

the information processing module is used for sequencing the priority of the received video information to be transmitted according to the priority identification of the video information to be transmitted;

the information processing module is used for creating a corresponding video information queue to be transmitted according to the priority of the video information to be transmitted;

and the information processing module is used for adjusting the network resource allocation of the target video information transmission channel according to the priority sequence of the video information to be transmitted.

In the above-mentioned scheme, the first step of the method,

the information processing module is used for traversing the video information queue to be transmitted and determining the video information to be transmitted with the highest priority;

the information processing module is used for determining the link quality of each link in the network resources;

the information processing module is configured to configure a link with the highest link quality in the network resources of the target video information transmission channel for the video information to be transmitted with the highest priority, so as to process the video information to be transmitted in the video information queue to be transmitted through the configured link.

An embodiment of the present invention further provides an electronic device, where the electronic device includes:

a memory for storing executable instructions;

and the processor is used for realizing the video information transmission method of the preamble when the executable instruction stored in the memory is operated.

The embodiment of the invention also provides a computer-readable storage medium, which stores executable instructions, and is characterized in that the executable instructions are executed by a processor to realize the video information transmission method of the preamble.

The embodiment of the invention has the following beneficial effects:

the embodiment of the invention monitors the change of the network transmission layer of the video playing terminal; monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determining the states of the different video information transmission channels; when the frame round-trip delay parameter in the network transmission layer is increased and exceeds a frame round-trip delay parameter threshold value, triggering to switch the video information transmission channel of the video playing terminal, and switching the first video information transmission channel to a target video information transmission channel; responding to a switching instruction of a video information transmission channel, and selecting a target video information transmission channel from different video information transmission channels; and transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer. Therefore, different video information transmission channels in the network transmission layer can be monitored, states of the different video information transmission channels are determined, the most suitable target video information transmission channel is selected to transmit the video information, and complete transmission of the video information in the network transmission layer is achieved.

Drawings

Fig. 1 is a schematic usage environment diagram of a video information transmission method according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a video information transmission apparatus according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a processing method for video information transmission display according to an embodiment of the present invention;

fig. 4 is a schematic flow chart of an alternative video information transmission method according to an embodiment of the present invention;

fig. 5 is a schematic usage environment diagram of a video information transmission method according to an embodiment of the present invention;

fig. 6 is a schematic usage environment diagram of a video information transmission method according to an embodiment of the present invention;

fig. 7 is a schematic flow chart of an alternative video information transmission method according to an embodiment of the present invention;

fig. 8 is a schematic diagram of an alternative state of a video information transmission method according to an embodiment of the present invention;

fig. 9 is a schematic flow chart illustrating an alternative process of a video information transmission method according to an embodiment of the present invention;

fig. 10 is a schematic flow chart illustrating an alternative process of a video information transmission method according to an embodiment of the present invention;

fig. 11 is a schematic flow chart illustrating an alternative process of a video information transmission method according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail with reference to the accompanying drawings, the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

Before further detailed description of the embodiments of the present invention, terms and expressions mentioned in the embodiments of the present invention are explained, and the terms and expressions mentioned in the embodiments of the present invention are applied to the following explanations.

1) Video transmission (Video Transcoding) refers to converting a Video stream that has been compression-coded into another Video stream to adapt to different network bandwidths, different terminal processing capabilities, and different user requirements.

2) The client, a carrier in the terminal for implementing a specific function, for example, a mobile client (APP) is a carrier of a specific function in the mobile terminal, for example, a function of performing live online (video push streaming) or a playing function of online video.

3) In response to: for indicating the condition or state on which the performed operation depends, when the condition or state on which the performed operation depends is satisfied, the performed operation or operations may be in real time or may have a set delay; there is no restriction on the order of execution of the operations performed unless otherwise specified.

4) Virtual environment: is a virtual environment that is displayed (or provided) when an application is run on the terminal. The virtual environment may be a simulation environment of a real world, a semi-simulation semi-fictional three-dimensional environment, or a pure fictional three-dimensional environment. The virtual environment may be any one of a two-dimensional virtual environment, a 2.5-dimensional virtual environment, and a three-dimensional virtual environment, and the following embodiments illustrate the virtual environment as a three-dimensional virtual environment, but are not limited thereto. Optionally, the virtual environment is also used for virtual environment engagement between at least two virtual objects. Optionally, the virtual environment is also used for a virtual firearm fight between at least two virtual objects. Optionally, the virtual environment is further configured to engage a virtual firearm between at least two virtual objects within a target area, the target area being smaller over time in the virtual environment.

5) Cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. Based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied in the cloud computing business model, a resource pool can be formed and used as required, and the cloud computing business model is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.

6) Cloud game: the game is characterized in that the game runs in a cloud server device, a game picture rendered by the cloud device is coded and then transmitted to a user terminal through a network, and a coded file is decoded by the user terminal and then rendered to a display screen for displaying, so that a user does not need to install the game locally, and the game interaction process can be completed only by establishing communication network connection with the cloud.

7) Frames per second FPS: (Frames Per Second), FPS is a definition in the field of images, and refers to the number of Frames transmitted Per Second of a picture, and refers to the number of pictures of animation or video in popular terms. The FPS measures the amount of information used to store and display the motion video. The greater the number of frames per second, the more fluid the displayed motion will be. Typically, a minimum of 30FPS is to avoid motion blindness.

The following description is made on a use environment of the video information transmission method provided by the present application, referring to fig. 1, fig. 1 is a schematic view of a use scene of the video information transmission method provided by an embodiment of the present invention, referring to fig. 1, a terminal (including a terminal 10-1 and a terminal 10-2) is provided with a client for running a cloud game, and a user can obtain game content stored in a cloud server through the set cloud game software client; the terminal is connected with the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the wide area network and the local area network, and data transmission is realized by using a wireless link, wherein the video information transmission method provided by the invention can serve clients (packaged in professional game terminals or different mobile electronic devices) with available types as a cloud service form, the video information transmission method is not particularly limited in the application, wherein the game runs in cloud server equipment, game pictures rendered by the cloud equipment are encoded and then transmitted to the user terminal through the network, and the encoded files are decoded by the user terminal and then rendered to a display screen for displaying the game, so that the user does not need to install the game locally, and can complete a game interaction process only by establishing communication network connection with the cloud. The organization structure of the cloud game comprises a terminal and a cloud server, wherein the terminal is used for receiving control operation of a user on the game process and sending a control instruction corresponding to the control operation to the cloud server, and the cloud server is used for controlling the game process and sending a video stream in the game process to the terminal for playing. That is, in the organization structure of the cloud game, the terminal is mainly responsible for receiving the game video and the game audio of the corresponding server in the game process to render and play, and sends the operation of the game user on the terminal side (including but not limited to the operation that the game user realizes through mouse, keyboard, operation handle, voice command, touch command and the like) to the server through the signaling channel connected with the server, and the server sends the operation information to the mouse driving process, sound card driving process and keyboard driving process matched with each other, thereby realizing the control of the cloud game, specifically, the processing content of the cloud server includes: 1) calculating operation, game rendering, picture acquisition coding and sound card sound acquisition coding in the running process of the game of the cloud game, and sending video information and audio information subjected to coding processing to a terminal in a streaming media mode; 2) and receiving an operation instruction of the terminal, and sending the operation instruction to the corresponding mouse driving process, the sound card driving process and the keyboard driving process, thereby realizing the control of the cloud game.

As will be described in detail below, the video information transmission apparatus according to the embodiment of the present invention may be implemented in various forms, such as a dedicated terminal with a processing function of the video information transmission apparatus, or an electronic device (a mobile phone or a tablet computer) with a processing function of the video information transmission apparatus, for example, the terminal 10-1 or the terminal 10-2 in the foregoing fig. 1. Fig. 2 is a schematic diagram of a composition structure of a video information transmission apparatus according to an embodiment of the present invention, and it can be understood that fig. 2 only shows an exemplary structure of the video information transmission apparatus, and not a whole structure, and a part of the structure or the whole structure shown in fig. 2 may be implemented as needed.

The video information transmission device provided by the embodiment of the invention comprises: at least one processor 201, memory 202, user interface 203, and at least one network interface 204. The various components of the video information transfer device are coupled together by a bus system 205. It will be appreciated that the bus system 205 is used to enable communications among the components. The bus system 205 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 205 in fig. 2.

The user interface 203 may include, among other things, a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touch pad, or a touch screen.

It will be appreciated that the memory 202 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The memory 202 in embodiments of the present invention is capable of storing data to support operation of the terminal (e.g., 10-1). Examples of such data include: any computer program, such as an operating system and application programs, for operating on a terminal (e.g., 10-1). The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application program may include various application programs.

In some embodiments, the video information transmission apparatus provided in the embodiments of the present invention may be implemented by a combination of hardware and software, and for example, the video information transmission apparatus provided in the embodiments of the present invention may be a processor in the form of a hardware decoding processor, which is programmed to execute the video information transmission method provided in the embodiments of the present invention. For example, a processor in the form of a hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Device (PLDs), Complex Programmable Logic Device (CPLD), Field-Programmable Gate arrays (FPGAs), or other electronic components.

As an example of the video information transmission apparatus provided by the embodiment of the present invention implemented by combining software and hardware, the video information transmission apparatus provided by the embodiment of the present invention may be directly embodied as a combination of software modules executed by the processor 201, where the software modules may be located in a storage medium, the storage medium is located in the memory 202, and the processor 201 reads executable instructions included in the software modules in the memory 202, and completes the video information transmission method provided by the embodiment of the present invention in combination with necessary hardware (for example, including the processor 201 and other components connected to the bus 205).

By way of example, the Processor 201 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic engine device, discrete gate or transistor logic engine device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor or the like.

As an example of the video information transmission apparatus provided by the embodiment of the present invention implemented by hardware, the apparatus provided by the embodiment of the present invention may be implemented by directly using the processor 201 in the form of a hardware decoding processor, for example, by being implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic engine devices (PLDs), Complex Programmable Logic engine devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components, to implement the video information transmission method provided by the embodiment of the present invention.

The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the video information transmission apparatus. Examples of such data include: any executable instructions for operating on the video information transmission apparatus, such as executable instructions, a program implementing the video information transmission method of an embodiment of the present invention may be included in the executable instructions.

In other embodiments, the video information transmission apparatus provided in the embodiments of the present invention may be implemented by software, and fig. 2 shows the video information transmission apparatus stored in the memory 202, which may be software in the form of programs, plug-ins, and the like, and includes a series of modules, and as an example of the programs stored in the memory 202, the video information transmission apparatus may include the following software modules, namely, an information processing module 2081 and an information processing module 2082. When the software modules in the video information transmission apparatus are read into the RAM by the processor 201 and executed, the video information transmission method provided by the embodiment of the present invention is implemented, where the functions of each software module in the video information transmission apparatus include:

the information transmission module 2081 is used for monitoring the change of a network transmission layer of the video playing terminal;

the information processing module 2082 is configured to monitor different video information transmission channels in the network transmission layer based on a change result of the state of the network transmission layer, and determine the states of the different video information transmission channels;

the information processing module 2082, configured to trigger switching of a video information transmission channel of the video playing terminal when a frame round-trip delay parameter in the network transmission layer increases and exceeds a frame round-trip delay parameter threshold, so as to switch a first video information transmission channel to a target video information transmission channel;

the information processing module 2082 is configured to respond to a switching instruction of a video information transmission channel, and select a target video information transmission channel from the different video information transmission channels;

the information processing module 2082 is configured to transmit video information through the target video information transmission channel, so as to implement complete transmission of the video information in a network transmission layer.

According to the electronic device shown in fig. 2, in one aspect of the present application, the present application also provides a computer program product or a computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes different embodiments and combinations of embodiments provided in various alternative implementations of the video information transmission method.

Referring to fig. 3, fig. 3 is a schematic diagram of an MPTCP transport layer architecture for video information transmission in an embodiment of the present invention, in a video information transmission process, a cloud game and a desktop capture coding scheme may capture a game picture and then encode and transmit the game picture to a user, and a desktop capture frame rate and a game frame rate are synchronized, in this process, MPTCP may be used, and an MPTCP technology allows data to be transmitted through a plurality of sub-streams during one connection period, so that an available bandwidth for data transmission is well increased. As shown in fig. 3, the connection channel of the traffic path for transmitting data may be a Subflow based on a multi-path transmission Control Protocol (MPTCP multi Transfer Control Protocol) or a multi-path Real-time Transport Protocol (MPRTP multi Real-time Transport Protocol), or may be a connection channel based on a common TCP, RTP, GRE, IPSec, or other forms. The UE and the service opposite terminal can define the multipath protocol. If the service opposite end does not support multipath, an MPTCP Proxy (Proxy) or an MPRTP Proxy, etc. may be set between the UE and the service opposite end to support multipath transmission, so as to avoid service interruption caused by changing an IP address by the UE. As an extension of TCP, the Multipath transport protocol standard (MPTCP), which is established by the internet engineering task force IETF, is a connection-oriented, reliable and ordered transport protocol, whose purpose is to allow TCP connections to use multiple data transmission paths to maximize channel resource usage. The transmission layer of the MPTCP is composed of two layers, namely an MPTCP connection layer and a TCP sub-stream layer. The MPTCP subflow exhibits the characteristics of TCP flow at the TCP subflow layer, and can be regarded as a single-path TCP flow for data packet transmission. The MPTCP connection layer is responsible for interacting with the application layer and implementing the management of multiple TCP subflows. The MPTCP data scheduling mainly completes the work of grouping data received from an application layer, and then distributing the data to different sub-streams for transmission. In use, the network architecture of TCP/IP includes, from top to bottom, an application layer, a transport layer, a network layer, and a link layer. When an upper layer application transmits data, the data is divided into a plurality of data blocks, which are called sub-data, and contain address information of a sender and a receiver. The sub-data is transmitted along different paths in one or more networks and recombined at the destination. For the transport layer, it may be determined from the application identification information in the packet name of the sub data from which upper layer application the data is coming.

As shown in fig. 3, in the transmission process of MPTCP, MPTCP includes an application layer and a transport layer. The transport layer is in turn divided into two sublayers: MPTCP and TCP layer. When a user uses MPTCP communication, N TCP connections can be opened simultaneously, where N is the number of paths between the sending end and the receiving end. The application layer sends the data to the MPTCP layer, which then breaks the data into segments and adds a control segment header to each segment. The control header contains a sequence number that is used to reassemble the data at the receiving end. The MPTCP layer then passes the data to the TCP layer. When receiving end receives data sent by opposite end of TCP layer, it reads data and rebuilds data according to control head, then transfers data to application layer

In some embodiments of the present invention, if an MPTCP-based connection is used between the UE and the service peer, the UE may send a TCP Synchronization (SYNC) message to the service peer, where the TCP Option (Option) of the newly-established TCP connection, such as MP _ JOIN (meaning adding Subflow), or ADD _ ADDR (meaning adding address), is carried in the TCP synchronization message, and the service peer may be instructed to establish a new Subflow based on the second IP address of the UE through the two TCP options.

Referring to fig. 4, fig. 4 is a schematic diagram illustrating transmission of an MPTCP subflow packet seq in the embodiment of the present invention, in an environment where the video information transmission method provided by the present invention is used, MPTCP is located above TCP in a protocol stack, and can perform data transmission by using the established TCP subflow. Among them, TCP is a connection-oriented 4-layer protocol that can provide reliable transport and possesses a congestion control mechanism. However, the protocol for single-path transmission of TCP cannot utilize a plurality of path resources even if these path resources exist on the network. Because TCP is implemented by quadruplets: { Source IP, destination IP, Source port, destination port } uniquely identifies a connection, which causes a connection to be broken when any one of the four-tuples changes. Because an application can only bind to a single quad using the TCP protocol, multiple IPs cannot be used simultaneously. This problem can be solved just by using MPTCP, as shown in fig. 4, an MPTCP session enables simultaneous transmission of data using multiple paths by establishing multiple subflows, which is similar to the process of TCP establishing a connection. These sub-flows are similar to normal TCP connections, establishing a connection through a three-way handshake and closing the connection with a four-way swipe. But these sub-streams are all bound to an existing MPTCP session. The data at the transmitting end can select any sub-stream for transmission.

In the processing engineering shown in fig. 4, taking the terminal as a mobile phone as an example, the two IPs can be used to establish an MPTCP session with the network server by using the WIFI and 5G of the mobile phone at the same time. However, if the network server has only 1 IP address, the network server YE1 may use different ports to establish connection with WIFI and 5G of the mobile phone, respectively, to form different video information transmission channels.

In an embodiment of the present application, since the usage scenarios of the video information are different, different video information transmission channels need to be established first, which specifically includes: acquiring a video type identifier transmitted by a network transmission layer of the video playing terminal; when the video type identification is used for representing a cloud game video, configuring a WIFI-2.4G type information transmission channel and a WIFI-5G type information transmission channel for a network transmission layer of the terminal; when the video type identification is used for representing a live video, configuring a cellular communication type information transmission channel and a WIFI-5G type information transmission channel for a network transmission layer of the terminal; and when the video type identifier is used for representing the short video, configuring a WIFI-2.4G type information transmission channel, a WIFI-5G type information transmission channel and a cellular communication type information transmission channel for a network transmission layer of the terminal.

In particular, for video streams such as cloud games, which are required in scenes with high code rate and ultra-low delay, although MPTCP is implemented in a transmission layer by using multiple subflows and using multiple paths, MPTCP in the transmission layer needs to comply with protocol specifications of the transmission layer. For example: when a plurality of subflows are used, the same seq space needs to be shared, but seq is sequentially increased, so when packet loss occurs in video information transmission, data packets which are vacant in the middle and discontinuous seq need to be retransmitted and recovered. Therefore, when one subflow sub-stream is lost, the degree of the total transmission time is still increased, which is not favorable for the user experience.

In order to overcome the above-mentioned drawbacks, a video information transmission method provided by the embodiment of the present invention is described with reference to the electronic device shown in fig. 2, referring to fig. 5, fig. 5 is an optional flowchart of the video information transmission method provided by the embodiment of the present invention, and it can be understood that the steps shown in fig. 5 may be executed by various servers operating the video information transmission apparatus, such as a dedicated terminal, a server or a server cluster with a video information transmission function. The following is a description of the steps shown in fig. 5.

Step 501: the video information transmission device monitors the state of a network transmission layer of the video playing terminal to obtain the state change result of the network transmission layer.

Step 502: and the video information transmission device monitors different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determines the states of the different video information transmission channels.

In some embodiments of the present invention, when the video information transmission channel is a first video information transmission channel, obtaining a frame round-trip delay parameter, a network delay parameter, and a packet loss rate parameter of a network transmission layer through video information transmitted by the first video information transmission channel; and determining the state of a first video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter. The first video information transmission channel can be a main channel for video information transmission, video information transmission in the cloud game process is kept, and channel quality of the video information transmission channel can be mainly expressed as indexes such as a frame round trip delay parameter (frame RTT), a network delay parameter, a frame number throughput capability parameter, a packet loss rate parameter, a stability parameter and the like.

In some embodiments of the present invention, when the video information transmission channel is a second video information transmission channel, a keep-alive packet may be configured for the second video information transmission channel; acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a network transmission layer in the second video information transmission channel through the keep-alive message; and determining the state of a second video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter. The second video information transmission channel can be a standby channel (the number of which is at least one) for video information transmission, and is used for ensuring the transmission of video information in the cloud game process when the video information is switched through the transmission channel, taking a TCP keep-alive message as an example, and utilizing the keep-alive message to detect the accident condition of the opposite end, so that when the accident happens, the half-open TCP connection can be released, the condition that the intermediate network is interrupted and the client and the server end are not aware of is avoided. Specifically, the TCP keep-alive message is an application layer data with 1 byte and a content of "00" obtained by subtracting 1 from the sequence number of the previous TCP message, and the TCP message before the keepalive probe message is sent, so that the states of different standby video information transmission channels are monitored.

Step 503: and when the frame round-trip delay parameter in the network transmission layer is increased and exceeds the frame round-trip delay parameter threshold value, the video information transmission device triggers the switching of the video information transmission channel of the video playing terminal and switches the first video information transmission channel to the target video information transmission channel.

It should be noted that, in different applications and different scenarios, when the video information transmission method provided in the present application is executed, the round trip delay parameter thresholds (RTT thresholds) corresponding to the switching frames are slightly different, which may be specifically expressed as: for applications with higher latency requirements, such as video transmission for cloud gaming, the lower the RTT threshold. Corresponding to application scenarios where latency requirements are not high, such as video conferencing, video telephony, for example, the RTT threshold may be set at the hundred millisecond (ms) level. In some embodiments of the present invention, a video type identifier transmitted by a network transport layer of the video playing terminal is obtained; when the video type identifier is used to represent a cloud game video, the frame round trip delay parameter threshold may be configured to be 20ms, when the video type identifier is used to represent a short video, the frame round trip delay parameter threshold may be configured to be 500ms, and for a video conference scene, the delay parameter threshold of the conference video may be configured to be 200ms, it should be noted that when the video information transmission method provided by the present application is used to transmit different types of video information, because different users have different usage requirements and usage feelings, the RTT threshold is not fixed, the RTT threshold may be automatically configured and adjusted according to different network states, and the RTT threshold may be dynamically adjusted according to instruction information of the user (for example, in a scene of a live broadcast application, the RTT threshold provided by the present application may be configured as a second-level threshold, for example, to be configured as 1s, the RTT threshold provided by the present application may be configured as 2s in the context of a live application).

Step 504: the video information transmission apparatus selects a target video information transmission channel among the different video information transmission channels in response to a switching instruction of the video information transmission channels.

Referring to fig. 6, fig. 6 is an alternative flowchart of a video information transmission method according to an embodiment of the present invention, and it can be understood that the steps shown in fig. 6 may be executed by various servers operating the video information transmission apparatus, such as a dedicated terminal, a server or a server cluster with a video information transmission function. The following is a description of the steps shown in fig. 6.

Step 601: and when a first video information transmission channel in the different video information transmission channels is in an unavailable state, acquiring a frame round-trip delay parameter, a network delay parameter and a packet loss rate parameter of a second video information transmission channel.

Step 602: and determining the state of a second video information transmission channel according to the frame round-trip delay parameter, the network delay parameter and the packet loss rate parameter of the second video information transmission channel.

Step 603: and determining that the second video information transmission channel is a target video information transmission channel according to the state of the second video information transmission channel.

Referring to fig. 7, fig. 7 is an alternative flowchart of a video information transmission method according to an embodiment of the present invention, and the following describes steps shown in fig. 7.

Step 701: determining that a first video information transmission channel in the different video information transmission channels is in an available state, and acquiring the state of a second video information transmission channel in a low-bit-rate scene;

step 702: configuring the same test video frame set for the first video information transmission channel and the second video information transmission channel;

step 703: respectively testing the first video information transmission channel and the second video information transmission channel through the test video frame set to obtain a test result of the video information transmission channels;

step 704: and determining a target video information transmission channel according to the test result of the video information transmission channel.

After the target video information transmission channel is determined through the steps shown in fig. 6 and 7, the process continues to step 505.

Step 505: the video information transmission device transmits the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer.

In some embodiments of the present invention, the video information is transmitted through the target video information transmission channel to implement complete transmission of the video information in the network transmission layer, which may be implemented by:

when a second video information transmission channel is determined to be a target video information transmission channel, retransmitting the cached video frames in the second video information transmission channel; and receiving the retransmitted video frames in the video playing terminal, and performing out-of-order recombination processing on the received video frames to delete repeated video frames in the retransmitted video frames.

In some embodiments of the present invention, when a target video information transmission channel is selected from the different video information transmission channels, the video information to be transmitted may be analyzed, and the priority identifier of the video information to be transmitted may be obtained; sequencing the priority of the received video information to be transmitted according to the priority identification of the video information to be transmitted; establishing a corresponding video information queue to be transmitted according to the priority of the video information to be transmitted; and adjusting the network resource allocation of the target video information transmission channel according to the priority sequence of the video information to be transmitted. Therefore, different customers can be treated differently. Simultaneously traversing a video information queue to be transmitted, and determining the video information to be transmitted with the highest priority; determining link quality of each link in the network resources; and configuring the link with the highest link quality in the network resources for the video information to be transmitted with the highest priority so as to transmit the video information to be transmitted in the video information queue to be transmitted through the configured link. Therefore, the utilization efficiency of resources is improved, timely processing of high-priority tasks can be guaranteed, the data processing speed of cloud server users is guaranteed, and the use experience of the users is improved.

Referring to fig. 8, fig. 8 is a schematic diagram of an optional state of a video information transmission method according to an embodiment of the present invention, and the following describes a processing procedure shown in fig. 8.

In some embodiments of the present invention, when the target video is a cloud game video, identification information of a target user may be determined; determining a data source cluster matched with the identification information based on a cloud server network; according to the data source cluster, storing cloud game history information of a user matched with the target user into a cloud game server; and selecting a target video information transmission channel from different video information transmission channels based on the cloud game historical information. As shown in fig. 8, when a video of a cloud game is played, a terminal receives a video frame encoded by a server in an operating environment of the cloud game and performs decoding playing, after the video playing terminal receives a video frame of the game video encoded by the server or a data frame carrying video information, the video frame of the game video or the data frame carrying video information needs to be stored in a play cache pool of the terminal, a decoding device of the terminal decodes the video frame of the game video or the data frame carrying video information in a real-time data frame queue of the play cache pool, and a game user obtains three complete video frames of the game video by decoding the video frames with the number 1, the number 2 and the number 3 in the game video to be displayed, wherein the better the transmission channel state of the target video information of the video playing terminal in the process, the smaller the number of frames in the real-time data frame queue of the play cache pool is, the less the number of frames in the real-time data frame queue of the play cache pool can be kept to be 0 in a cloud game use scene, so that a game user can be ensured to obtain a non-blocking game video in the running process of a cloud game.

In some embodiments of the present invention, the first resolution may be set to 720P for different target video information transmission channels, and when the resolution selected by the user in the cloud game process is greater than or equal to 720P, and when the network of the user terminal is switched from the broadband access cloud game server to the mobile network access cloud game server, fluctuation of the information transmission rate may occur, which may cause bandwidth instability.

Next, taking a virtual scene as an example, a description is continued on the video information transmission method provided in the embodiment of the present invention. Wherein, with the development of network games, in some games operated by mobile terminals, the requirements of game target users on game scenes are higher and higher, in the process of cloud games, because of different terminal types and different game types, the display range of mobile terminals is limited, and in the process of changing real-time scenes of game roles held by users, the users can obtain richer use feeling of cloud games through the game videos, because of the complexity of cloud game video transmission, when the traditional technology uses single transmission, dead retransmission may occur in a transmission layer, so that the transmission of the cloud game videos is slower, service problems are easily caused, and when the application layer is used for processing, the fluctuation of the network can be relieved only by reducing the video quality or sacrificing the fluency of the videos, and the use experience of the cloud games is seriously influenced, therefore, through the video information transmission method provided by the application, when the target game is a cloud game, identification information of a target user can be determined; determining a data source cluster matched with the identification information based on a cloud server network; and according to the data source cluster, storing the cloud game history information of the user matched with the target user into a cloud game server. The cloud game history information of the user can be called by other application programs (such as a game simulator or a motion sensing game device), and of course, a game data processing model matched with different types of games can also be migrated to an online fighting FPS game in an instant messaging process or an offline controlled FPS game and a cloud game.

With continuing reference to fig. 9, fig. 9 is an optional flowchart of a video information transmission method process according to an embodiment of the present invention, where the steps shown in fig. 9 may be executed by various game terminals or game accelerator processes in which the video information transmission apparatus is deployed, and specifically include the following steps:

step 901: and acquiring the video type identification transmitted by the network transmission layer of the video playing terminal.

Step 902: and when the video type identification is used for representing the cloud game video, configuring a WIFI-2.4G type information transmission channel, a WIFI-5G type information transmission channel and a cellular communication type information transmission channel for a network transmission layer of the terminal.

Step 903: and configuring a frame round-trip delay parameter threshold value of a network transmission layer for the cloud game video, wherein the frame round-trip delay parameter threshold value is 20 ms.

Where the RTT threshold may be set at the hundred millisecond (ms) level. When the video type identifier represents the cloud game video of the role playing class, the frame round trip delay parameter threshold value can be configured to be 20ms, when the video type identifier represents the strategy game video, the frame round trip delay parameter threshold value can be configured to be 200ms, and different game users have different use requirements and use feelings, so that the RTT threshold value can be automatically configured and adjusted according to different network states to adapt to the use requirements of the game, and the RTT threshold value can be dynamically adjusted according to instruction information of the users, so that the game users can obtain better use experience.

Step 904: monitoring the state of a network transmission layer of a video playing terminal to obtain a state change result of the network transmission layer;

step 905: monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determining the states of the different video information transmission channels;

step 906: and when the frame round-trip delay parameter in the network transmission layer is increased and exceeds the frame round-trip delay parameter threshold value by 20ms, triggering to switch the video information transmission channel of the video playing terminal, and switching the WIFI-2.4G type information transmission channel to the WIFI-5G type information transmission channel.

In some embodiments of the present invention, since the initial transmission channel of the cloud game video is the WIFI-2.4G type information transmission channel, and the WIFI-5G type information transmission channel and the cellular communication type information transmission channel are both in a connected state, in order to select a more suitable video information transmission channel, the same test video frame set may be configured; and testing the WIFI-5G type information transmission channel and the cellular communication type information transmission channel through the test video frame set, and selecting the video information transmission channel with the minimum frame round-trip delay parameter for switching.

When the switching of the video information transmission channel is performed, the video information transmission is lost or queued for a long time due to the jitter of the network channel, thereby causing the loss or serious delay of the video frame. After switching the video information transmission channel, in order to implement lossless fast switching, it is necessary to retransmit a video frame that has been sent but is not acknowledged by the client, with reference to fig. 10, fig. 10 is an optional flowchart of a process of a video information transmission method provided in an embodiment of the present invention, and specifically includes the following steps:

step 1001: transmitting cloud game video information in a network transmission layer of the video playing terminal;

step 1002: and judging whether the switching of the video information transmission channel is needed, if so, executing step 1003, otherwise, executing step 1005.

Step 1003: and retransmitting the cached cloud game video.

Step 1004: and continuously transmitting the cloud game video information.

Step 1005: and determining to insert the cloud game video cache queue.

Step 1006: and determining to delete the cloud game video cache queue.

Step 1007: and forming a cloud game video buffer queue.

Further, when the video information is transmitted through the target video information transmission channel to realize complete transmission of the video information in the network transmission layer, in order to realize the effect of lossless transmission, not only the cloud game video retransmission processing shown in fig. 10 needs to be performed on the server side, but also the video frames need to be reassembled out of order on the terminal side, and the repeated video frames are deleted. With continuing reference to fig. 11, fig. 11 is a schematic flow chart illustrating an optional process of a video information transmission method according to an embodiment of the present invention, which specifically includes the following steps:

step 1101: and acquiring the cloud game video and determining a corresponding return time delay parameter.

Step 1102: and judging whether the cloud game video is a continuous video frame, if so, executing step 1106, and otherwise, executing step 1103.

Step 1103: it is determined whether it is a repeated video frame and if so, step 1104 is performed, otherwise step 1105 is performed.

Step 1104: the repeated video frames are discarded.

For example, as shown in fig. 8, by decoding the video frames with the number 1, the number 2 and the number 3 in the game video, so that the game user obtains three complete video frames of the game video, if the video frame with the number 3 is received twice, the repeated video frame with the number 3 should be discarded.

Step 1105: and recombining the disordered video frames.

For example, as shown in fig. 8, by decoding the video frames with the number 1, the number 2, and the number 3 in the game video, when the game user obtains three video frames of the complete game video, if the number sequence number 2, the number 3, and the number 1 are received, the video frame numbers should be adjusted.

Step 1106: and transmitting the cloud game video through the target cloud game video transmission channel.

The beneficial technical effects are as follows:

the embodiment of the invention monitors the change of the network transmission layer of the video playing terminal; monitoring different video information transmission channels in the network transmission layer based on the change result of the state of the network transmission layer, and determining the states of the different video information transmission channels; when the frame round-trip delay parameter in the network transmission layer is increased and exceeds a frame round-trip delay parameter threshold value, triggering to switch the video information transmission channel of the video playing terminal, and switching the first video information transmission channel to a target video information transmission channel; responding to a switching instruction of a video information transmission channel, and selecting a target video information transmission channel from different video information transmission channels; and transmitting the video information through the target video information transmission channel so as to realize the complete transmission of the video information in a network transmission layer. Therefore, different video information transmission channels in the network transmission layer can be monitored, states of the different video information transmission channels are determined, the most suitable target video information transmission channel is selected to transmit the video information, and complete transmission of the video information in the network transmission layer is achieved.

The above description is only exemplary of the present invention and should not be taken as limiting the scope of the present invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.