阅读说明：本技术 一种自由视点视频播放控制方法及设备 (Free viewpoint video playing control method and device ) 是由 翟世平 高雪松 陈维强 于 2020-06-23 设计创作，主要内容包括：本申请公开了一种自由视点视频播放控制方法及设备。本申请中,获取目标节目中用户指定的目标对象,在目标节目对应的单视点视频序列中,对目标对象进行视觉目标跟踪；当根据视觉目标跟踪判定目标对象的大小和/或位置满足虚拟视点视频切换条件时,根据目标对象的大小和/或位置,从目标节目对应的至少两个虚拟视点视频序列中选择目标虚拟视点视频序列；将目标虚拟视点视频序列发送给播放设备,使得播放设备播放所述目标虚拟视点视频序列。(The application discloses a free viewpoint video playing control method and equipment. In the method, a target object specified by a user in a target program is obtained, and visual target tracking is carried out on the target object in a single-view video sequence corresponding to the target program; when the size and/or the position of the target object meet/meets the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to a target program according to the size and/or the position of the target object; and sending the target virtual viewpoint video sequence to the playing equipment, so that the playing equipment plays the target virtual viewpoint video sequence.)

1. A free viewpoint video playing control method is characterized by comprising the following steps:

acquiring a target object designated by a user in a target program;

performing visual target tracking on the target object in a single-view video sequence corresponding to the target program;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

2. The method of claim 1, wherein the method further comprises:

acquiring the free viewpoint splicing video sequence;

decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence;

and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

3. The method of claim 1, wherein selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or position of the target object comprises:

determining an optimal viewing angle according to the size and/or the position of the target object;

and selecting one virtual viewpoint video sequence matched with the optimal view angle from the at least two virtual viewpoint video sequences corresponding to the target program as a target virtual viewpoint video sequence according to the respective corresponding view angles of the at least two virtual viewpoint video sequences corresponding to the target program.

4. A streaming media server, comprising: a memory and a processor coupled with the memory, the processor configured to:

acquiring a target object designated by a user in a target program;

performing visual target tracking on the target object in a single-view video sequence corresponding to the target program;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

5. The streaming media server of claim 4, wherein the processor is further configured to:

acquiring the free viewpoint splicing video sequence;

decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence;

and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

6. A video processing terminal, comprising: a memory and a processor coupled with the memory, the processor configured to:

acquiring a target object designated by a user in a target program;

receiving a single-viewpoint video sequence and a free-viewpoint spliced video sequence corresponding to the target program, wherein the free-viewpoint spliced video sequence is formed by splicing single-viewpoint video sequences respectively acquired by at least two video acquisition devices for the same scene;

performing visual target tracking on the target object in the single-viewpoint video sequence;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; wherein the at least two virtual view video sequences are generated from the free view stitched video sequence;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

7. The video processing terminal of claim 6, wherein the processor is further configured to:

acquiring the free viewpoint splicing video sequence;

decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence;

and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

8. A streaming media server, comprising:

the target acquisition module is used for acquiring a target object specified by a user in a target program;

the target tracking module is used for carrying out visual target tracking on the target object in a single-view video sequence corresponding to the target program;

the video switching module is used for selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and the sending module is used for sending the target virtual viewpoint video sequence to the playing equipment so that the playing equipment plays the target virtual viewpoint video sequence.

9. A video processing terminal, comprising:

the video receiving module is used for receiving a single-viewpoint video sequence and a free-viewpoint spliced video sequence corresponding to a target program from a streaming media server, wherein the free-viewpoint spliced video sequence is formed by splicing single-viewpoint video sequences respectively acquired by at least two video acquisition devices aiming at the same scene;

the virtual viewpoint video sequence generating module is used for generating at least two virtual viewpoint video sequences according to the free viewpoint splicing video sequence;

the target acquisition module is used for acquiring a target object specified by a user in the target program;

the target tracking module is used for carrying out visual target tracking on the target object in the single-view video sequence;

the video switching module is used for selecting a target virtual viewpoint video sequence from the at least two virtual viewpoint video sequences according to the size and/or the position of the target object when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking;

and the sending module is used for sending the target virtual viewpoint video sequence to the playing equipment so that the playing equipment plays the target virtual viewpoint video sequence.

Technical Field

The present application relates to the field of free viewpoint video technologies, and in particular, to a free viewpoint video playing control method and device.

Background

The free viewpoint video is obtained by shooting videos aiming at the same scene by a plurality of single viewpoint cameras and synthesizing the videos shot at different angles. The free viewpoint-based video playing service may enable a user to watch videos at arbitrary angles. The free viewpoint video playing supports video live broadcast and video recording and playing.

When the free viewpoint video playing service is used, a user can select videos with corresponding visual angles according to interested visual angles for watching. When the user needs to switch to other visual angles for watching, the video switching of different visual angles is carried out in a manual mode, and the user experience is poor.

Disclosure of Invention

An exemplary embodiment of the present application provides a free viewpoint video playing control method and device, which are used for automatically switching a free viewpoint view angle based on visual tracking and positioning to improve user experience.

According to an aspect of the exemplary embodiments, there is provided a free viewpoint video play control method including:

acquiring a target object designated by a user in a target program;

performing visual target tracking on the target object in a single-view video sequence corresponding to the target program;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

In some embodiments, the method further comprises: acquiring the free viewpoint splicing video sequence; decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence; and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

In some embodiments, the selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object includes: determining an optimal viewing angle according to the size and/or the position of the target object; and selecting one virtual viewpoint video sequence matched with the optimal view angle from the at least two virtual viewpoint video sequences corresponding to the target program as a target virtual viewpoint video sequence according to the respective corresponding view angles of the at least two virtual viewpoint video sequences corresponding to the target program.

According to an aspect of an exemplary embodiment, there is provided a streaming server including: a memory and a processor coupled with the memory, the processor configured to:

acquiring a target object designated by a user in a target program;

performing visual target tracking on the target object in a single-view video sequence corresponding to the target program;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

In some embodiments, the processor is further configured to: acquiring the free viewpoint splicing video sequence; decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence; and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

According to an aspect of the exemplary embodiments, there is provided a video processing terminal including: a memory and a processor coupled with the memory, the processor configured to:

acquiring a target object designated by a user in a target program;

receiving a single-viewpoint video sequence and a free-viewpoint spliced video sequence corresponding to the target program, wherein the free-viewpoint spliced video sequence is formed by splicing single-viewpoint video sequences respectively acquired by at least two video acquisition devices for the same scene;

performing visual target tracking on the target object in the single-viewpoint video sequence;

when the size and/or the position of the target object are judged to meet the virtual viewpoint video switching condition according to the visual target tracking, selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object; wherein the at least two virtual view video sequences are generated from the free view stitched video sequence;

and sending the target virtual viewpoint video sequence to a playing device, so that the playing device plays the target virtual viewpoint video sequence.

In some embodiments, the processor is further configured to: acquiring the free viewpoint splicing video sequence; decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence; and generating at least two virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different viewpoints of the same scene.

According to an aspect of an exemplary embodiment, there is provided a streaming server including:

the target acquisition module is used for acquiring a target object specified by a user in a target program;

the target tracking module is used for carrying out visual target tracking on the target object in a single-view video sequence corresponding to the target program;

the video switching module is used for selecting a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking; the at least two virtual viewpoint video sequences are generated according to a free viewpoint splicing video sequence corresponding to the target program, and the free viewpoint splicing video sequence is formed by splicing single viewpoint video sequences respectively collected by at least two video collecting devices aiming at the same scene;

and the sending module is used for sending the target virtual viewpoint video sequence to the playing equipment so that the playing equipment plays the target virtual viewpoint video sequence.

According to an aspect of the exemplary embodiments, there is provided a video processing terminal including:

the video receiving module is used for receiving a single-viewpoint video sequence and a free-viewpoint spliced video sequence corresponding to a target program from a streaming media server, wherein the free-viewpoint spliced video sequence is formed by splicing single-viewpoint video sequences respectively acquired by at least two video acquisition devices aiming at the same scene;

the virtual viewpoint video sequence generating module is used for generating at least two virtual viewpoint video sequences according to the free viewpoint splicing video sequence;

the target acquisition module is used for acquiring a target object specified by a user in the target program;

the target tracking module is used for carrying out visual target tracking on the target object in the single-view video sequence;

the video switching module is used for selecting a target virtual viewpoint video sequence from the at least two virtual viewpoint video sequences according to the size and/or the position of the target object when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking;

and the sending module is used for sending the target virtual viewpoint video sequence to the playing equipment so that the playing equipment plays the target virtual viewpoint video sequence.

In the embodiment of the application, the target object is subjected to visual target tracking in the single-viewpoint video sequence corresponding to the target program, and when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking, the target virtual viewpoint video sequence is selected from the multiple virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object for playing, so that automatic switching of free viewpoint and visual angle based on visual tracking and positioning is realized, and the user experience can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.

Fig. 1 is a diagram schematically illustrating generation of a free viewpoint video sequence in the present embodiment;

fig. 2 is a schematic diagram illustrating a video playing system in an embodiment of the present application;

fig. 3 is a schematic structural diagram illustrating a streaming media server in an embodiment of the present application;

fig. 4 is a schematic diagram illustrating a video playback control flow executed on the streaming media server side in the embodiment of the present application:

fig. 5 is a schematic diagram illustrating a video playback system in a further embodiment of the present application;

fig. 6 is a schematic structural diagram schematically illustrating a video processing terminal in an embodiment of the present application;

fig. 7 is a schematic flowchart illustrating a video playback control method implemented on the video processing terminal side in the embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below with reference to the drawings in the exemplary embodiments of the present application, and it is obvious that the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments.

All other embodiments, which can be derived by a person skilled in the art from the exemplary embodiments shown in the present application without inventive effort, shall fall within the scope of protection of the present application. Moreover, while the disclosure herein has been presented in terms of exemplary one or more examples, it is to be understood that each aspect of the disclosure can be utilized independently and separately from other aspects of the disclosure to provide a complete disclosure.

It should be understood that the terms "first," "second," "third," and the like in the description and in the claims of the present application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used are interchangeable under appropriate circumstances and can be implemented in sequences other than those illustrated or otherwise described herein with respect to the embodiments of the application, for example.

Furthermore, the terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to those elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.

The term "module" as used herein refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.

The embodiment of the application provides a free viewpoint video playing control method and device, and videos with corresponding visual angles are automatically selected for a user by performing visual tracking and positioning on video contents which are interested by the user. The embodiment of the application is suitable for live video scenes and also suitable for video on demand scenes.

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Fig. 1 schematically illustrates a schematic diagram of generating a free viewpoint video sequence in an embodiment of the present application.

As shown in fig. 1, a plurality of cameras may be positioned around a target scene, the plurality of cameras forming an array of cameras, the different cameras filming from different perspectives, so as to film the scene from a multi-dimensional spatial perspective, forming a multi-way single-view video sequence. The multi-path single-viewpoint video sequence is transmitted to a video synthesis server, and the video synthesis server performs splicing processing to obtain a free viewpoint splicing video sequence.

In the embodiment of the application, a multi-channel video real-time panoramic stitching algorithm can be adopted to stitch a multi-channel single-viewpoint video sequence into a free-viewpoint stitched video sequence. It should be noted that, in the embodiment of the present application, a generation method of a free viewpoint splicing video sequence is not limited.

Fig. 2 schematically shows an architecture diagram of a video playing system in an embodiment of the present application.

As shown in the figure, the multiple single-view video sequences are transmitted to the video composition server 201, and after the video composition server 201 performs the splicing process to form the free-view spliced video sequence, the free-view spliced video sequence is transmitted to the streaming media server 202 for storage.

The streaming media server 202 processes the free viewpoint spliced video sequence to generate a plurality of virtual viewpoint video sequences. Different virtual viewpoint video sequences correspond to different visual angles of the same scene, so that the video watching requirement of a user on any visual angle is met. When a streaming media server provides media playing service for a user, visual tracking and positioning are carried out on a target object in a single-viewpoint video sequence shot by a camera according to the target object appointed by the user, when the size and/or the position of the target object are judged to meet a virtual viewpoint video switching condition according to a visual target tracking result, a target virtual viewpoint video sequence is selected from a plurality of virtual viewpoint video sequences according to the size and/or the position of the target object, and the target virtual viewpoint video sequence is sent to playing equipment through a network 203, so that the playing equipment plays the target virtual viewpoint video sequence.

Wherein, the playback device refers to an electronic device with a video playback function, as shown in the figure, the playback device includes: smart phone 204, tablet 205, laptop 206, smart television 207, etc.

Based on the functions of the streaming server described above, fig. 3 exemplarily shows a structure of the streaming server.

As shown, the streaming server may include: the system comprises a target acquisition module 301, a target tracking module 302, a video switching module 303 and a sending module 304. Wherein:

a target obtaining module 301, configured to obtain a target object specified by a user in a target program, where the target object is an object in a video that the user is interested in;

a target tracking module 302, configured to perform visual target tracking on the target object in a single-view video sequence corresponding to the target program;

the video switching module 303 is configured to, when it is determined that the size and/or the position of the target object meets the virtual viewpoint video switching condition according to the visual target tracking, select a target virtual viewpoint video sequence from at least two virtual viewpoint video sequences corresponding to a target program according to the size and/or the position of the target object;

the sending module 304 is configured to send the target virtual viewpoint video sequence to the playing device, so that the playing device plays the target virtual viewpoint video sequence.

In some embodiments, the target tracking module 302 may determine an optimal viewing angle according to the size and/or the position of the target object, and select one virtual viewpoint video sequence matching the optimal viewing angle from the multiple virtual viewpoint video sequences as the target virtual viewpoint video sequence according to the respective corresponding viewing angles.

In some embodiments, the streaming media server may further include a virtual viewpoint video sequence generating module 305, configured to: the method comprises the steps of obtaining a free viewpoint splicing video sequence, decoding the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence, and generating a plurality of virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding.

Fig. 4 schematically shows a flow chart of a free viewpoint video playing control method implemented by a streaming media server in the embodiment of the present application.

In the embodiment of the application, the streaming media server may decode the free viewpoint spliced video sequence after obtaining the free viewpoint spliced video sequence to obtain a single viewpoint video sequence contained in the free viewpoint spliced video sequence, and generate a plurality of virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, where different virtual viewpoint video sequences correspond to different viewpoints of the same scene. The plurality of virtual viewpoint video sequences can be stored in a video list form, so that the virtual viewpoint video sequences are selected according to the situation and are sent to the playing equipment at the user side for video playing.

A virtual viewpoint video sequence may be generated based on the depth information and using a convolutional neural network. It should be noted that, in the embodiment of the present application, a virtual viewpoint video sequence generation algorithm is not limited.

Referring to fig. 4, the flow of the streaming media server performing video playing control may include the following steps:

s401: and the streaming media server acquires a target object specified by the user in the target program.

In the embodiment of the application, a user can send a video playing request through a terminal or a playing device, for example, in a video-on-demand scene, the user selects a certain target program from a video resource list through the terminal or the playing device and requests to play the program; for another example, in a live video scene, a user selects a live channel through a terminal or a playing device to request to watch a live target program. After a user initiates a video playing request, a target object which the user is interested in can be selected based on a user interface provided by a system, and information of the target object is sent to a server. For example, for a live program of a basketball game requested to be watched, the user may select a basketball player participating in the game as the target object.

S402: and the streaming media server performs visual target tracking on the target object in a single-view video sequence corresponding to the target program.

In the embodiment of the application, a target program may correspond to a free viewpoint splicing video sequence and a single viewpoint video sequence, and the single viewpoint video sequence may be a single-path video sequence included in the free viewpoint splicing video sequence or may not be in multiple-path video sequences included in the free viewpoint splicing video sequence.

In this step, the streaming media server may obtain a single-view video sequence corresponding to a target program from its video resource library according to the target program requested to be viewed by a user. A single-viewpoint video sequence acquired by a video acquisition device with a large shooting range can be appointed in advance to be used for tracking a target object visual target.

Visual target tracking is to predict the size and position of a target in an initial frame of a video sequence given the size and position of the target in a subsequent frame. In the embodiment of the application, the following method can be adopted to track the target object:

firstly, inputting an initialization target frame, wherein the target frame comprises a target object; then, a plurality of candidate frames are generated in the next frame, the features (Feature Extractor) of the candidate frames are extracted, the candidate frames are scored (update Model), and finally, one candidate frame with the highest score is found in the scores to be used as a prediction target, or a plurality of predicted values are fused (Ensemble) to obtain a better prediction target.

In order to realize more accurate target tracking, the embodiment of the application performs target tracking by a tracking method based on Correlation Filter (Correlation Filter) and Convolutional Neural Network (CNN). It should be noted that, the visual target tracking method used in the embodiments of the present application is not limited.

S403: and if the size and/or the position of the target object meet/meets the virtual viewpoint video switching condition according to the visual target tracking, turning to S404, otherwise, returning to S402 to continue the target tracking.

In this step, virtual viewpoint video switching conditions may be set in advance, for example, the conditions may be set as: when the position of the target object is moved to the edge area of the video frame (the edge area of the video frame can be defined in advance), the virtual viewpoint video switching condition is met; the condition may also be set as: when the size of the target object is smaller than the sizes of other objects or is smaller than a set size (generally, the size of the target object is related to the distance between camera lenses, and the farther the distance, the smaller the target object), the virtual viewpoint video switching condition is satisfied. The condition may also be set in connection with the size and position of the target object.

And after judging that the virtual viewpoint video switching condition is met based on the visual target tracking of the target object, switching to the subsequent steps to select the virtual viewpoint video with the optimal view angle, otherwise, keeping the virtual viewpoint video with the current view angle unchanged, and continuously carrying out the visual target tracking on the target object.

S404: and the streaming media server selects a target virtual viewpoint video sequence from a plurality of virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object.

In this step, the streaming media server may first calculate an optimal viewing angle, for example, a viewing angle closest to the target object and having the character in the center of the video frame, according to the size and/or position of the target object; and selecting a virtual viewpoint video sequence matched with the optimal viewpoint from the virtual viewpoint video sequences as a target virtual viewpoint video sequence according to the respective corresponding viewpoints of the virtual viewpoint video sequences.

S405: and the streaming media server sends the target virtual viewpoint video sequence to the playing equipment, so that the playing equipment plays the target virtual viewpoint video sequence.

And when the video sequence requested to be watched by the user is played completely or the video processing terminal receives an end instruction of the user, ending the process.

Based on the above flow, taking the video requested to be watched by the user as a basketball game program as an example, the streaming media server may generate video streams of N (N is an integer greater than 1) virtual viewpoints in advance according to the free viewpoint splicing video streams (i.e., video sequences) of the program. In the process of providing a video server for a user, performing visual target tracking on a basketball player appointed by the user and interested by the user according to a single-view video stream of the program, determining an optimal viewing angle (such as the distance is the nearest and a person is in the center of a video picture) according to a target tracking result, selecting a matched virtual view video stream according to the optimal viewing angle and sending the matched virtual view video stream to a playing device at the user side for playing.

By taking a playing device for playing a basketball game in a live broadcast manner as an example, in practical application, a single-viewpoint video sequence shot by a camera capable of shooting a global picture of a game scene is taken as the single-viewpoint video sequence for tracking a visual target, so that all players in a game field can be tracked.

Before the basketball game is played, the user may select a basketball player to be tracked (e.g., basketball player a) from a list of participating players in the user interface. The name or the identification of the basketball player selected by the user, the identification of the user and other information are sent to a streaming media server, and characteristic information such as the face of each player in the game is stored on the streaming media server for visual target tracking.

After the live broadcast starts, the streaming media service tracks the video target of the athlete concerned by the user based on the single viewpoint video sequence, and in the process, judges whether the virtual viewpoint video switching condition is met or not according to the position, the size and the like of the tracked athlete A, if the virtual viewpoint video switching condition is met, a target virtual viewpoint video sequence is selected from the virtual viewpoint video sequences according to the motion direction, the position and the like of a target object, and the selected target virtual viewpoint video sequence can enable the athlete A to be located in a picture center area or enable the athlete A to occupy more pictures compared with other athletes. And the streaming media server switches the virtual viewpoint video sequence corresponding to the user to the target virtual viewpoint video sequence, so that the user can watch the competition process of the athlete A concerned by the user.

In some embodiments of the present application, the user may also be allowed to select or re-select a basketball player to be tracked after the live broadcast begins. For example, during the live game viewing process using the smart phone, the user may trigger the live pause function option through a screen touch operation, and perform a frame selection on the currently still image for the athlete needing attention (e.g., circle the outline of the target athlete in the still image on the touch screen, and trigger the "track" function option to select the tracked target). The playing device may transmit the still image and the information about the area selected by the user to the streaming server. After the streaming media server obtains the still image, the streaming media server can identify the object in the area to obtain the target to be tracked by the user. And the subsequent streaming media server can switch the virtual viewpoint video sequence according to the visual tracking of the target.

As can be seen from the above description, in the embodiments of the present application, the target object is subjected to the visual target tracking in the single-viewpoint video sequence corresponding to the target program, and when it is determined that the size and/or the position of the target object satisfy the virtual viewpoint video switching condition according to the visual target tracking, the target virtual viewpoint video sequence is selected from the multiple virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object for playing, so that the free viewpoint view angle automatic switching based on the visual tracking positioning is realized, and the user experience can be improved.

Fig. 5 is a schematic diagram illustrating an architecture of a video playing system in another embodiment of the present application.

As shown in the figure, after a plurality of single-viewpoint video sequences are transmitted to the video composition server and are subjected to splicing processing by the video composition server to form a free-viewpoint spliced video sequence, the free-viewpoint spliced video sequence is transmitted to the streaming media server 501 for storage. In the process that the streaming media server provides the video playing service for the user, the streaming media server 501 sends the free viewpoint splicing video sequence corresponding to the program requested to be played by the user to the video processing terminal 503 on the user side through the network 502. The streaming media server 501 may also send a single-view video sequence corresponding to the program to the video processing terminal 503 for performing visual target tracking.

The video processing terminal 503 processes the free viewpoint spliced video sequence to generate a plurality of virtual viewpoint video sequences. Different virtual viewpoint video sequences correspond to different visual angles of the same scene, so that the video watching requirement of a user on any visual angle is met. When the streaming media server provides media playing service for a user, visual tracking and positioning are carried out on a target object in a single viewpoint video sequence according to the target object appointed by the user, when the size and/or the position of the target object are judged to meet a virtual viewpoint video switching condition according to a visual target tracking result, a target virtual viewpoint video sequence is selected from a plurality of virtual viewpoint video sequences according to the size and/or the position of the target object, and the target virtual viewpoint video sequence is sent to playing equipment, so that the playing equipment plays the target virtual viewpoint video sequence.

Wherein, the playback device refers to an electronic device with a video playback function, as shown in the figure, the playback device includes: smart tv 504, smart phone 505, PC 506, etc.

The video processing terminal 503 and various playing devices in the above-mentioned architecture may be in the same local area network, and the video processing terminal 503 may be a device used for performing video processing in an intelligent home scene, for example, may specifically be an intelligent set-top box.

Based on the functions of the video processing terminal described above, fig. 6 exemplarily shows a structure of a video processing terminal.

As shown, the video processing terminal may include: the system comprises a video receiving module 601, a virtual viewpoint video sequence generating module 602, a target obtaining module 603, a target tracking module 604, a video switching module 605 and a sending module 606. Wherein:

the video receiving module 601 is configured to receive a single-viewpoint video sequence corresponding to a target program and a free-viewpoint spliced video sequence from a streaming media server;

a virtual viewpoint video sequence generating module 602, configured to generate a plurality of virtual viewpoint video sequences according to the free viewpoint splicing video sequence;

a target obtaining module 603, configured to obtain a target object specified by a user in a target program;

a target tracking module 604, configured to perform visual target tracking on a target object in the single-viewpoint video sequence;

a video switching module 605, configured to select a target virtual viewpoint video sequence from the multiple virtual viewpoint video sequences according to the size and/or the position of the target object when it is determined that the size and/or the position of the target object meets the virtual viewpoint video switching condition according to the visual target tracking;

a sending module 606, configured to send the target virtual viewpoint video sequence to the playing device, so that the playing device plays the target virtual viewpoint video sequence.

In some embodiments, the target tracking module 604 may determine an optimal viewing angle according to the size and/or the position of the target object, and select one virtual viewpoint video sequence matching the optimal viewing angle from the multiple virtual viewpoint video sequences as the target virtual viewpoint video sequence according to the respective corresponding viewing angles.

Fig. 7 schematically shows a flowchart of a free viewpoint video playing control method implemented by a video processing terminal in the embodiment of the present application. As shown in the figure, the process of controlling video playing by the video processing terminal may include the following steps:

s701: and the video processing terminal receives a single-viewpoint video sequence and a free viewpoint splicing video sequence corresponding to a target program requested by a user from the streaming media server.

In the embodiment of the application, a target program may correspond to a free viewpoint splicing video sequence and a single viewpoint video sequence, and the single viewpoint video sequence may be a single-path video sequence included in the free viewpoint splicing video sequence or may not be in multiple-path video sequences included in the free viewpoint splicing video sequence.

In this step, based on the target program requested to be watched by the user, the video processing terminal may send a video acquisition request to the streaming media server, and the streaming media server sends the single-viewpoint video sequence and the free-viewpoint stitching video sequence corresponding to the target program requested to be watched by the user to the video processing terminal.

S702: and the video processing terminal generates a plurality of virtual viewpoint video sequences according to the free viewpoint splicing video sequence.

In the step, after receiving a free viewpoint splicing video sequence sent by a streaming media server, a video processing terminal decodes the free viewpoint splicing video sequence to obtain a single viewpoint video sequence contained in the free viewpoint splicing video sequence, and generates a plurality of virtual viewpoint video sequences according to all the single viewpoint video sequences obtained after decoding, wherein different virtual viewpoint video sequences correspond to different view angles of the same scene. The plurality of virtual viewpoint video sequences can be stored in a video list form, so that the virtual viewpoint video sequences are selected according to the situation and are sent to the playing equipment at the user side for video playing. The generation algorithm of the virtual viewpoint video sequence can be referred to the description of the foregoing embodiment. It should be noted that, in the embodiment of the present application, a virtual viewpoint video sequence generation algorithm is not limited.

S703: and the video processing terminal acquires a target object specified by the user in the target program.

S704: and the video processing terminal carries out visual target tracking on the target object in the single-view video sequence corresponding to the target program.

S705: and if the size and/or the position of the target object meet/meets the virtual viewpoint video switching condition according to the visual target tracking, turning to S706, otherwise, returning to S704 to continue the target tracking.

S706: and the video processing terminal selects a target virtual viewpoint video sequence from a plurality of virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object.

S707: and the video processing terminal sends the target virtual viewpoint video sequence to the playing equipment, so that the playing equipment plays the target virtual viewpoint video sequence.

The specific implementation manner of some steps in the above flow (such as the above S703 to S707) is basically the same as the specific implementation manner of the relevant steps in the flow shown in fig. 4, and is not repeated here.

And when the video sequence of the target program requested to be watched by the user is played completely or the video processing terminal receives an end instruction of the user, ending the process.

In the embodiment of the application, the target object is subjected to visual target tracking in the single-viewpoint video sequence corresponding to the target program, and when the size and/or the position of the target object is judged to meet the virtual viewpoint video switching condition according to the visual target tracking, the target virtual viewpoint video sequence is selected from the multiple virtual viewpoint video sequences corresponding to the target program according to the size and/or the position of the target object for playing, so that automatic switching of free viewpoint and visual angle based on visual tracking and positioning is realized, and the user experience can be improved.

According to an aspect of the exemplary embodiments, an embodiment of the present application further provides a streaming media server, including: a memory and a processor coupled to the memory, the processor configured to perform the method performed by the server described in the embodiments of the present application.

According to an aspect of the exemplary embodiments, an embodiment of the present application further provides a video processing terminal, including: a memory and a processor coupled to the memory, the processor configured to perform the method performed by the video processing terminal described in the above embodiments of the present application.

According to an aspect of the exemplary embodiments, the present application also provides a computer storage medium, in which computer program instructions are stored, and when the instructions are run on a computer, the computer is caused to execute the above method.

On the basis of the common knowledge in the field, the above preferred conditions can be combined randomly to obtain the preferred embodiments of the application.

Since the communication terminal and the computer storage medium in the embodiment of the present application may be applied to the processing method, reference may also be made to the above method embodiment for obtaining technical effects, and details of the embodiment of the present application are not described herein again.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

While specific embodiments of the present application have been described above, it will be appreciated by those skilled in the art that these are by way of example only, and that the scope of the present application is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and principles of this application, and these changes and modifications are intended to be included within the scope of this application.