阅读说明：本技术 在360度视频中搜寻视频失真 (Searching for video distortion in360degree video ) 是由 塞思·拉裴尔 于 2018-12-11 设计创作，主要内容包括：一种方法包括：从显示流送360度视频的第一帧的一部分的设备,接收触发搜寻操作的指示,所述第一帧的该部分基于所述设备的用户的视点；基于所述搜寻操作来选择所述流送360度视频的第二帧；编码所述第二帧的第一部分和所述第二帧的第二部分,所述第二帧的第一部分对应于所述第一帧的一部分；以及将所述第二帧的经编码的部分通信到所述设备。(One method comprises the following steps: receiving, from a device displaying a portion of a first frame of a streaming 360degree video, an indication that triggers a seek operation, the portion of the first frame based on a viewpoint of a user of the device; selecting a second frame of the streaming 360degree video based on the seek operation; encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to a portion of the first frame; and communicating the encoded portion of the second frame to the device.)

1. A method, comprising:

displaying a portion of a first frame of a streaming 360degree video, the portion of the first frame being based on a viewpoint of a user of a device;

receiving an indication for triggering a search operation;

selecting a second frame of the streaming 360degree video based on the seek operation;

encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to the portion of the first frame; and

communicating the encoded portion of the second frame to the apparatus.

2. The method of claim 1, wherein the second portion of the second frame extends outward from a center of the first portion of the second frame and begins at a boundary of the first portion of the second frame.

3. The method of claim 1 or 2, wherein the seek operation is one of a fast forward operation or a look back operation.

4. The method of any of claims 1-3, wherein selecting the second frame based on the seek operation comprises:

determining whether the seek operation is a fast forward operation, an

In response to determining that the seek operation is a fast forward operation,

selecting a subsequent frame of the 360degree video that advances in sequence as the second frame.

5. The method of any of claims 1-4, wherein selecting the second frame based on the seek operation comprises:

determining whether the search operation is a look-back operation, an

In response to determining that the seek operation is a look-back operation,

selecting a previous frame of the 360degree video that advances in sequence as the second frame.

6. The method of any of claims 1-5, wherein encoding the first portion of the second frame and the second portion of the second frame comprises encoding the second frame at a frame rate based on a seek speed of the seek operation.

7. The method of any of claims 1 to 6, further comprising:

encoding the portion of the first frame of the streaming 360degree video by using a projection technique; and

communicating the encoded first frame to the device, wherein encoding the second frame comprises:

centering the first portion of the second frame based on the viewpoint,

projecting the second frame by using a projection technique, an

The second portion is added around the centered first portion.

8. The method of any of claims 1 to 7, further comprising:

encoding the portion of the first frame of the streaming 360degree video by using a first projection technique; and

communicating the encoded first frame to the device, wherein encoding the second frame comprises encoding the second frame by using a second projection technique.

9. The method of any of claims 1 to 8, further comprising:

receiving, from the device, an indication that the viewpoint has changed during the seek operation; and

encoding the second frame of the streaming 360-degree video based on the changed viewpoint.

10. The method of any of claims 1 to 9, further comprising:

receiving, from the device, an indication that the viewpoint has changed during the seek operation;

encoding the second frame of the streaming 360-degree video based on the changed viewpoint;

receiving, from the device, an indication to trigger a playback operation during the seek operation;

encoding a portion of a third frame of the streaming 360degree video based on the changed viewpoint; and

communicating the encoded third frame to the apparatus.

11. A content server, comprising:

a controller configured to:

receiving, from a device rendering a first portion of a first frame of a streaming 360degree video, an indication to trigger a seek operation, the portion of the frame being based on a viewpoint of a user of the device;

selecting a second frame of the streaming 360degree video based on the seek operation; and

communicating the encoded second frame to the device; and

an encoder configured to encode a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to the portion of the first frame.

12. The content server of claim 11, wherein the second portion of the second frame extends outward from a center of the first portion of the second frame and begins at a boundary of the first portion of the second frame.

13. The content server of claim 11 or 12, wherein selecting the second frame based on the seek operation comprises:

determining whether the seek operation is a fast forward operation, an

In response to determining that the seek operation is a fast forward operation,

selecting a subsequent frame of the 360degree video that advances in sequence as the second frame.

14. The content server of any of claims 11 to 13, wherein selecting the second frame based on the seek operation comprises:

determining whether the search operation is a look-back operation, an

In response to determining that the seek operation is a look-back operation,

selecting a previous frame of the 360degree video that advances in sequence as the second frame.

15. The content server of any of claims 11 to 14, wherein the encoder is further configured to:

centering the first portion of the second frame based on the viewpoint,

projecting the second frame by using a projection technique, an

The second portion is added around the centered first portion.

16. The content server of any of claims 11 to 15, wherein the encoder is further configured to:

encoding the first frame of the streaming 360degree video by using a first projection technique; and

encoding the second frame of the streaming 360degree video by using a second projection technique.

17. The content server of any of claims 11 to 16, wherein the controller is further configured to:

receiving, from the device, an indication that the viewpoint has changed during the seek operation; and

causing the encoder to encode the second frame of the streaming 360 degrees video based on the changed viewpoint.

18. The content server of any of claims 11 to 17, wherein the controller is further configured to:

receiving, from the device, an indication that the viewpoint has changed during the seek operation;

causing the encoder to encode the second frame of the streaming 360-degree video based on the changed viewpoint;

receiving, from the device, an indication to trigger a playback operation during the seek operation;

causing the encoder to encode a portion of a third frame of the streaming 360 degrees video based on the changed viewpoint; and

communicating the encoded third frame to the apparatus.

19. A non-transitory computer readable storage medium having stored thereon computer executable program code which when executed on a computer system causes the computer system to perform steps comprising:

receiving, from a device displaying a first portion of a first frame of a streaming 360degree video, an indication that triggers a seek operation, the portion of the frame being based on a viewpoint of a user of the device;

selecting a second frame of the streaming 360degree video based on the seek operation;

encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to the portion of the first frame; and

communicating the encoded second frame to the apparatus.

20. The non-transitory computer-readable storage medium of claim 19, wherein the second portion of the second frame extends outward from a center of the first portion of the second frame and begins at a boundary of the first portion of the second frame.

Technical Field

Background

Typically, a streaming server encodes a two-dimensional (2D) representation of 360 degrees of video and communicates a portion of the encoded 2D representation to a device capable of rendering 360 degrees of video. Alternatively (or additionally), the streaming server encodes a three-dimensional (3D) video representation of the 360degree video. In one case, two separate 2D views (one for each eye) are included or volumetric video (volumetric video) is encoded and communicated, or in another case, 2D or stereoscopic video including depth information is encoded and communicated. The device then decodes the 2D (or 3D) representation, converts the decoded 2D representation into 360degree video, and renders a portion of the 360degree video.

Disclosure of Invention

Example embodiments describe techniques for encoding, decoding, and streaming video during seek operations. In a general aspect, a method includes: displaying a portion of a first frame of a 360degree video, the portion of the first frame based on a viewpoint of a user of a device; receiving an indication for triggering a search operation; selecting a second frame of the 360degree video based on the seek operation; encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to a portion of the first frame; and communicating the encoded portion of the second frame to the device. In a preferred embodiment, 360degree video is streamed, such as from a server or second device that has higher processing power than the device performing the display. The first portion of the second frame corresponding to a portion of the first frame may be the portion of the second frame based on a viewpoint of the user for the first frame. In an embodiment, the device performing the display may be a different device or the same device as the device receiving the indication of the seek operation. For example, the head mounted device may perform the display, and the mobile device or the control device may receive an indication that triggers the seek operation. In other embodiments, when the video is displayed on another device, such as a head-mounted device, the computing device may stream or otherwise provide 360 the video and receive an indication that a seek operation is triggered. There is further provided a method comprising: receiving, from a device displaying a portion of a first frame of a streaming 360degree video, an indication that triggers a seek operation, the portion of the first frame based on a viewpoint of a user of the device; selecting a second frame of the streaming 360degree video based on the seek operation; encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to the portion of the first frame; and communicating the encoded portion of the second frame to the device.

In another general aspect, a server includes a controller and an encoder. The controller is configured to: controlling rendering of a first portion of a first frame of a 360degree video, the portion of the frame based on a viewpoint of a user of a device; receiving an indication for triggering a search operation; selecting a second frame of the 360degree video based on the seek operation; and communicating the encoded second frame to the device. The encoder is configured to: encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to the portion of the first frame. In a preferred embodiment, 360degree video is streamed and the server is a content server.

In yet another general aspect, a non-transitory computer-readable storage medium having computer-executable program code stored thereon, the code when executed on a computer system causing the computer system to perform steps comprising: displaying a portion of a first frame of a 360degree video, the portion of the first frame based on a viewpoint of a user of a device; receiving an indication for triggering a search operation; selecting a second frame of the 360degree video based on the seek operation; encoding a first portion of the second frame and a second portion of the second frame, the first portion of the second frame corresponding to a portion of the first frame; and communicating the encoded portion of the second frame to the device. In a preferred embodiment, 360degree video is streamed, such as from a server or second device that has higher processing power than the device performing the display.

Drawings

The exemplary embodiments will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only, and thus do not limit the exemplary embodiments, and wherein:

fig. 1 is a flow diagram of a method for displaying a streaming 360degree video frame according to at least one example embodiment.

Fig. 2A illustrates a frame of 360degree video in accordance with at least one example embodiment.

Fig. 2B illustrates a block diagram of a 2D cube representation of a frame of 360degree video, according to at least one example embodiment.

FIG. 2C illustrates a portion of a frame within a block diagram of a 2D cube representation.

Fig. 2D illustrates this portion of the frame within the rearranged block diagram of the 2D cube representation.

Fig. 2E illustrates a block diagram of a 2D representation of a frame of 360degree video, according to at least one example embodiment.

Fig. 3 illustrates a user interacting with a control device.

Fig. 4,5, 6 and 7 are flowcharts of a method for encoding/decoding a video frame according to at least one example embodiment.

FIG. 8 is an exemplary system diagram illustrating interaction with a computer-generated Virtual Reality (VR) environment.

Fig. 9A illustrates a video encoder system according to at least one example embodiment.

Fig. 9B illustrates a video decoder system according to at least one example embodiment.

FIG. 10 illustrates a system in accordance with at least one example embodiment.

FIG. 11 illustrates an example of a computer device and a mobile computer device according to at least one example embodiment.

It should be noted that these drawings are intended to illustrate the general features of methods, structures and/or materials utilized in certain exemplary embodiments, and to supplement the written description provided below. The drawings are not necessarily to scale, and may not precisely reflect the precise structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the value ranges or characteristics encompassed by example embodiments. For example, the relative thicknesses and positions of structural elements may be reduced or exaggerated for clarity. The use of like or identical reference numbers in the various figures is intended to indicate the presence of like or identical elements or features.

Embodiments relate to rendering 360degree streaming video during seek operations.