Reference picture derivation and motion compensation for 360-degree video coding

文档序号：1440083 发布日期：2020-02-14 浏览：22次中文

阅读说明：本技术 用于360度视频译码的参考图片推导及运动补偿 (Reference picture derivation and motion compensation for 360-degree video coding ) 是由 M·Z·科班 G·范德奥维拉 M·卡切维奇于 2018-07-03 设计创作，主要内容包括：本发明描述用于从360度视频数据的立方体贴图投影或经调整立方体贴图投影产生用延伸面封装的参考帧的技术。用所述延伸面封装的所述参考帧可用于360度视频数据的后续帧的帧间预测。(This disclosure describes techniques for generating extended surface encapsulated reference frames from a cube map projection or adjusted cube map projection of 360 degree video data. The reference frame encapsulated with the extended surface may be used for inter-prediction of subsequent frames of 360 degree video data.)

1. A method of decoding 360 degree video data, the method comprising:

receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data;

decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged in the packing plane;

deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data;

sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent encoded frames of the 360 degree video data are decoded using an inter-prediction process and the derived extended reference frame.

2. The method of claim 1, wherein the projection is a cube map projection or an adjusted cube map projection (ACP).

3. The method of claim 1, wherein sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface comprises sampling the decoded sphere of 360 degrees video data using the projection according to a number of extended pixels to generate an extended surface.

4. The method of claim 3, further comprising:

receiving an indication of the number of extended pixels at a picture level or a sequence level.

5. The method of claim 3, wherein the number of extended pixels is equal to a maximum prediction unit size in a Coding Tree Unit (CTU) of the encoded frame of 360 degree video data.

6. The method of claim 1, wherein decoding the subsequent encoded frame of 360 degree video data using the inter-prediction process and the derived extended reference frame comprises:

rotating a current prediction unit in a current cube face of the encoded frame of 360 degrees video data based on a rotation angle of the reference cube face, the reference cube face containing a reference block for the current prediction unit.

7. An apparatus configured to decode 360 degree video data, the apparatus comprising:

a memory configured to store encoded frames of 360 degree video data; and

one or more processors in communication with the memory, the one or more processors configured to:

receiving the encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data;

decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged in the packing plane;

deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent encoded frames of the 360 degree video data are decoded using an inter-prediction process and the derived extended reference frame.

8. The apparatus of claim 7, wherein the projection is a cube map projection or an adjusted cube map projection (ACP).

9. The apparatus of claim 7, wherein to sample the decoded sphere of 360 degrees video data using the projection to generate an extended surface, the one or more processors are further configured to sample the decoded sphere of 360 degrees video data using the projection according to a number of extended pixels to generate an extended surface.

10. The apparatus of claim 9, wherein the one or more processors are further configured to:

receiving an indication of the number of extended pixels at a picture level or a sequence level.

11. The apparatus of claim 9, wherein the number of extended pixels is equal to a maximum prediction unit size in a Coding Tree Unit (CTU) of the encoded frame of 360 degree video data.

12. The apparatus of claim 7, wherein to decode the subsequent encoded frame of 360 degrees video data using the inter-prediction process and the derived extended reference frame, the one or more processors are further configured to:

13. The apparatus of claim 7, further comprising:

a display configured to display at least a portion of the decoded sphere of 360 degrees video data.

14. An apparatus configured to decode 360 degree video data, the apparatus comprising:

means for receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data;

means for decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged with the packing plane;

means for deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data;

means for sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data;

means for deriving an extended reference frame from the extended surface; and

means for decoding subsequent encoded frames of 360 degree video data using an inter-prediction process and the derived extended reference frame.

15. A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to decode video data to:

receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data;

decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged in the packing plane;

deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent encoded frames of the 360 degree video data are decoded using an inter-prediction process and the derived extended reference frame.

16. A method of encoding 360 degree video data, the method comprising:

receiving a sphere of 360 degree video data;

arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data;

encoding the frames of encapsulation face to form frames of encoded 360-degree video data;

reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data arranged in the encapsulation face;

deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data;

sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent frames of the 360 degree video data are encoded using an inter-prediction process and the derived extended reference frame.

17. The method of claim 16, wherein the projection is a cube map projection or an adjusted cube map projection (ACP).

18. The method of claim 16, wherein sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface comprises sampling the reconstructed sphere of 360 degrees video data using the projections according to a number of extended pixels to generate an extended surface.

19. The method of claim 18, further comprising:

generating an indication of the number of extended pixels at a picture level or a sequence level.

20. The method of claim 18, wherein the number of extension pixels is equal to a maximum prediction unit size in a Coding Tree Unit (CTU) of the frame of a packing plane.

21. The method of claim 16, wherein encoding the subsequent frame of 360 degree video data using the inter-prediction process and the derived extended reference frame comprises:

rotating a current prediction unit in a current cube face of the frame of a packing face based on a rotation angle of the reference cube face, the reference cube face containing a reference block for the current prediction unit.

22. An apparatus configured to encode 360 degree video data, the apparatus comprising:

a memory configured to store a sphere of 360 degrees video data; and

one or more processors in communication with the memory, the one or more processors configured to:

receiving the sphere of 360 degrees video data;

arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data;

encoding the frames of encapsulation face to form frames of encoded 360-degree video data;

deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent frames of the 360 degree video data are encoded using an inter-prediction process and the derived extended reference frame.

23. The apparatus of claim 22, wherein the projection is a cube map projection or an adjusted cube map projection (ACP).

24. The apparatus of claim 22, wherein to sample the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, the one or more processors are further configured to sample the reconstructed sphere of 360 degrees video data using the projections according to a number of extended pixels to generate an extended surface.

25. The apparatus of claim 24, wherein the one or more processors are further configured to:

generating an indication of the number of extended pixels at a picture level or a sequence level.

26. The apparatus of claim 24, wherein the number of extension pixels is equal to a maximum prediction unit size in a Coding Tree Unit (CTU) of the encoded frame of an encapsulation plane.

27. The apparatus of claim 22, wherein to encode the subsequent frame of 360 degree video data using the inter-prediction process and the derived extended reference frame, the one or more processors are further configured to:

rotating a current prediction unit in a current cube face of the encoded frame of a packing face based on a rotation angle of the reference cube face, the reference cube face containing a reference block for the current prediction unit.

28. The apparatus of claim 22, further comprising:

a camera configured to capture the sphere of 360 degrees video data.

29. An apparatus configured to encode 360 degree video data, the apparatus comprising:

means for receiving a sphere of 360 degrees video data;

means for arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data;

means for encoding the frames of encapsulation face to form frames of encoded 360 degree video data;

means for reconstructing the frame of encoded 360 degree video data to obtain a reconstructed frame of 360 degree video data, the reconstructed frame of 360 degree video data arranged with the encapsulation face;

means for deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data;

means for sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data;

means for deriving an extended reference frame from the extended surface; and

means for encoding a subsequent frame of 360 degree video data using an inter-prediction process and the derived extended reference frame.

30. A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to encode video data to:

receiving a sphere of 360 degree video data;

arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data;

encoding the frames of encapsulation face to form frames of encoded 360-degree video data;

deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data;

deriving an extended reference frame from the extended surface; and

subsequent frames of the 360 degree video data are encoded using an inter-prediction process and the derived extended reference frame.

Technical Field

The present invention relates to encoding and decoding video data.

Background

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, Personal Digital Assistants (PDAs), laptop or desktop computers, tablet computers, electronic book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smart phones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265, also known as High Efficiency Video Coding (HEVC), and extensions of these standards. Video devices may more efficiently transmit, receive, encode, decode, and/or store digital video information by implementing these video coding techniques.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as Coding Tree Units (CTUs), Coding Units (CUs) and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in inter-coded (P or B) slices of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame and a reference picture may be referred to as a reference frame.

Recently, techniques have been developed for coding and transmitting 360-degree video, such as for Virtual Reality (VR) applications. Due to recent developments in VR video technology, the video environment experienced by the user has become as important as the subject matter of the video itself. This VR video technology may use 360 degree video technology, which involves real-time streaming of 360 degree video graphics and/or real-time streaming of 360 degree video from a 360 degree video camera or website to a real-time video display, such as a VR Head Mounted Display (HMD). VR HMDs allow a user to experience all the actions that occur around him by changing the viewing angle as the head is turned. To create a 360 degree video, a special set of cameras may be used to record all 360 degrees of the scene simultaneously, or multiple views (e.g., video and/or computer generated images) may be stitched together to form an image.

After the video data has been encoded, the video data may be packetized for transmission or storage. The video data may be compiled into a video file that conforms to any of a variety of standards, such as the international organization for standardization (ISO) base media file format and extensions thereof, such as the AVC file format.

Disclosure of Invention

In general, this disclosure relates to techniques for encoding and decoding video data. In some examples, this disclosure describes reference picture derivation and motion compensation techniques for 360-degree video coding. In some examples, this disclosure describes techniques for generating extended surface encapsulated reference frames from cube map projections or adjusted cube map projections of 360 degree video data. The reference frame encapsulated with the extended surface may be used for inter prediction of subsequent frames of 360 degree video data. By generating reference frames with extended faces, distortion and coding efficiency issues caused by distortion and discontinuities at boundaries between packing faces can be mitigated.

In one example, this disclosure describes a method of decoding 360-degree video data, the method comprising: receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data; decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged with the encapsulation face; deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data; sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and decoding subsequent encoded frames of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes an apparatus configured to decode 360 degree video data, the apparatus comprising: a memory configured to store encoded frames of 360 degree video data; and one or more processors in communication with the memory, the one or more processors configured to: receiving the encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data; decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged with the encapsulation face; deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data; sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and decoding subsequent encoded frames of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes an apparatus configured to decode 360 degree video data, the apparatus comprising: means for receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data; means for decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged with the encapsulation face; means for deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data; means for sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data; means for deriving an extended reference frame from the extended surface; and means for decoding subsequent encoded frames of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to decode video data to: receiving an encoded frame of 360 degree video data, the encoded frame of 360 degree video data arranged with a packing surface obtained from a projection of a sphere of the 360 degree video data; decoding the frame of encoded 360-degree video data to obtain a decoded frame of 360-degree video data, the decoded frame of 360-degree video data arranged with the encapsulation face; deriving a decoded sphere of 360 degrees video data from the decoded frame of 360 degrees video data; sampling the decoded sphere of 360 degrees video data using the projection to generate an extended surface, wherein the extended surface is greater than the packing surface of the decoded frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and decoding subsequent encoded frames of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes a method of encoding 360 degree video data, the method comprising: receiving a sphere of 360 degree video data; arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data; encoding the frames of encapsulation face to form frames of encoded 360-degree video data; reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data arranged with the encapsulation face; deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data; sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and encoding a subsequent frame of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes an apparatus configured to encode 360-degree video data, the apparatus comprising: a memory configured to store a sphere of 360 degrees video data; and one or more processors in communication with the memory, the one or more processors configured to: receiving the sphere of 360 degrees video data; arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data; encoding the frames of encapsulation face to form frames of encoded 360-degree video data; reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data arranged in the encapsulation face; deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data; sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and encoding a subsequent frame of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes an apparatus configured to encode 360-degree video data, the apparatus comprising: means for receiving a sphere of 360 degrees video data; means for arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data; means for encoding the frames of encapsulation face to form frames of encoded 360 degree video data; means for reconstructing the frame of encoded 360 degree video data to obtain a reconstructed frame of 360 degree video data, the reconstructed frame of 360 degree video data arranged with the encapsulation face; means for deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data; means for sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data; means for deriving an extended reference frame from the extended surface; and means for encoding a subsequent frame of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

In another example, this disclosure describes a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device configured to encode video data to: receiving a sphere of 360 degree video data; arranging the sphere of 360 degrees video data into a frame of an envelope surface obtained from a projection of the sphere of 360 degrees video data; encoding the frames of encapsulation face to form frames of encoded 360-degree video data; reconstructing the frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data, the reconstructed frame of 360-degree video data arranged in the encapsulation face; deriving a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data; sampling the reconstructed sphere of 360 degrees video data using the projections to generate an extended surface, wherein the extended surface is greater than the packing surface of the reconstructed frame of 360 degrees video data; deriving an extended reference frame from the extended surface; and encoding a subsequent frame of the 360 degree video data using an inter-prediction process and the derived extended reference frame.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

Drawings

FIG. 1 is a block diagram depicting an example video encoding and decoding system configured to perform the techniques of this disclosure.

Fig. 2A and 2B are conceptual diagrams showing representations of models of displays used to present 360 degree and/or panoramic video data.

Fig. 3 is a conceptual diagram showing an example 3 x 2 package structure for 360 degree video.

FIG. 4 is a conceptual diagram showing deformation at a cube face boundary.

FIG. 5 is a conceptual diagram depicting discontinuities at cube face boundaries.

Fig. 6 is a conceptual diagram showing a process for cube face extension.

Fig. 7 is a conceptual diagram showing a reconstructed adjusted cube map projection (ACP) frame and an extended reference frame.

FIG. 8 is a conceptual diagram showing a derived extended reference frame that may improve inter-prediction at deformed cube face boundaries.

FIG. 9 is a conceptual diagram showing a derived extended reference frame that may improve inter-prediction at discontinuity cube face boundaries.

FIG. 10 is a conceptual diagram showing an example prediction unit rotation, according to one example of this disclosure.

FIG. 11 is a block diagram showing an example video encoder configured to perform the techniques of this disclosure.

FIG. 12 is a block diagram showing an example video decoder configured to perform the techniques of this disclosure.

FIG. 13 is a flow chart showing an example encoding method of the present invention.

FIG. 14 is a flow chart showing an example decoding method of the present invention.

Detailed Description

In general, this disclosure relates to techniques for encoding and decoding video data. In some examples, this disclosure describes reference picture derivation and motion compensation techniques for 360-degree video coding. In some examples, this disclosure describes techniques to generate a reference frame encapsulated with extended faces from a cube map projection or adjusted cube map projection of 360-degree video data. The reference frame encapsulated with the extended surface may be used for inter prediction of subsequent frames of 360 degree video data. By generating reference frames with extended faces, distortion and coding efficiency issues caused by distortion and discontinuities at boundaries between packing faces can be mitigated.

Fig. 1 is a block diagram showing an example video encoding and decoding system 10 that may utilize techniques for reference picture derivation and motion compensation of 360 degree video data. As shown in fig. 1, system 10 includes a source device 12, source device 12 providing encoded video data to be decoded by a destination device 14 at a later time. In particular, source device 12 provides video data to destination device 14 via computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Destination device 14 may receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may comprise any type of medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may comprise a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the internet. The communication medium may include routers, switches, base stations, or any other equipment that may be used to facilitate communication from source device 12 to destination device 14.

In some examples, the encoded data may be output from output interface 22 to a storage device. Similarly, encoded data may be accessed from the storage device by the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by source device 12. Destination device 14 may access stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to destination device 14. Example file servers include web servers (e.g., for a website), FTP servers, Network Attached Storage (NAS) devices, or local hard drives. Destination device 14 may access the encoded video data via any standard data connection, including an internet connection. Such a connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, internet video streaming, such as dynamic adaptive streaming over HTTP (DASH), encoding of digital video onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

In the example of fig. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes input interface 28, video decoder 30, display device 32, and region determination unit 34. In other examples, the source device and the destination device may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18, such as an external camera. Likewise, destination device 14 may interface with an external display device, rather than including an integrated display device.

The depicted system 10 of FIG. 1 is merely one example. The techniques for reference picture derivation and motion compensation of 360-degree video data may be performed by any digital video encoding and/or decoding device. Although the techniques of this disclosure are generally performed by a video encoding device, the techniques may also be performed by a video encoder/decoder (commonly referred to as a "CODEC"). Source device 12 and destination device 14 are merely examples of such coding devices that source device 12 generates coded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetric manner such that each of devices 12, 14 includes video encoding and decoding components. Thus, system 10 may support one-way or two-way video transmission between video devices 12, 14, e.g., for video streaming, video playback, video broadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device such as a video camera, a library of video files containing previously captured video, and/or a video feed interface to receive video from a video content provider. As a further alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding in general, and may be applicable to wireless and/or wired applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output by output interface 22 onto computer-readable medium 16.

In an example of the present disclosure, video source 18 may be configured to capture 360 degrees of video data. For example, video source 18 may be a set of cameras, typically consisting of a plurality of individual cameras pointing in different directions and ideally collectively encompassing all viewpoints around the set of cameras. Video source 18 may be further configured to perform image stitching, where video pictures taken by multiple individual cameras are synchronized in the time domain and stitched into spherical video in the spatial domain, but mapped to a rectangular format, such as an equal-quantity rectangular map (similar to a world map) or a cube map.

In one example, video encoder 20 may encode data for a full 360 degree panorama at multiple resolutions (e.g., 6k, 4k, HD (1080p), and 720 p). That is, video encoder 20 may encode video data for each region (or "tile") at each of these multiple resolutions. In this way, the die granularity may be the same for each resolution. Video encoder 20 may avoid inter-layer dependencies when encoding various resolutions. Thus, video decoder 30 may decode video data for a slice at different resolutions in a selective manner (e.g., as selected by region determination unit 34). For example, region determination unit 34 may select the highest available resolution for the region at the center of the user's current viewpoint. In the case of moving away from the center of the current view, the decoded resolution may gradually decrease. That is, the region determining unit 34 may select a resolution that becomes proportionally lower for regions (dies) farther from the center of the current viewpoint. Thus, video decoder 30 may decode video data at the lowest available resolution for a die behind the user's current view.

Computer-readable medium 16 may include: transitory media such as wireless broadcast or wired network transmission; or a storage medium (i.e., a non-transitory storage medium) such as a hard disk, a flash drive, a compact disk, a digital video disk, a blu-ray disk, or other computer-readable medium. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, e.g., transmit over a network. Similarly, a computing device of a media production facility, such as an optical disc stamping facility, may receive encoded video data from source device 12 and produce an optical disc containing the encoded video data. Thus, in various examples, computer-readable medium 16 may be understood to include one or more computer-readable media in various forms.

Input interface 28 of destination device 14 receives information from computer-readable medium 16. The information of computer-readable medium 16 may include syntax information defined by video encoder 20, which is also used by video decoder 30, including syntax elements that describe characteristics and/or processing of blocks and other coded units. Display device 32 displays the decoded video data to a user, and may comprise any of a variety of display devices, such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.

In accordance with example techniques of this disclosure, output interface 22 and input interface 28 may correspond to network interfaces, such as a Network Interface Card (NIC) implementing one or more network protocols, such as ethernet. Computer-readable media 16 may correspond to a network connection, which may traverse a private or public network, such as the Internet.

The display device 32 may correspond to a panoramic display. For example, display device 32 may correspond to a Head Mounted Display (HMD), or substantially or completely enclose one or more screens of a user. Region determination unit 34 may be configured to determine a plurality of regions of display device 32. For example, display device 32 may include a plurality of dies, e.g., corresponding to one or more portions of a cube face of a spherical display (or a display that can simulate a spherical display, such as an HMD).

As discussed herein, zone determination unit 34 may determine one or more of the zones to which the visual focus of the user (not shown in fig. 1) is directed. Region determination unit 34 may cause input interface 28 to retrieve video data for a first subset of regions of display device 32 to which the visual focus of the user is directed.

Destination device 14 may include a memory, such as a hard disk and/or a buffer, configured to store the retrieved video data. Such memory may be included within video decoder 30, within region determination unit 34, or elsewhere within destination device 14.

Video encoder 20 and video decoder 30 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard, also known as ITU-T h.265, or a new h.266 standard being studied by the joint video experts group (jfet). Alternatively, video encoder 20 and video decoder 30 may operate according to other proprietary or industry standards, such as the ITU-t h.264 standard, alternatively referred to as MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions of these standards. However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in fig. 1, in some aspects, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units, or other hardware and software, to handle encoding of both audio and video in a common data stream or separate data streams. Where applicable, the multiplexer-demultiplexer unit may conform to the ITU h.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP). In general, video decoder 30 performs a substantially similar, but reciprocal, process to that performed by video encoder 20 to decode the encoded data.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder or decoder circuitry, e.g., including one or more processors, such as one or more microprocessors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), discrete logic, software, hardware, firmware, or any combinations thereof. When the techniques are implemented in part in software, a device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Thus, an encoder or decoder may be formed from any of a variety of integrated processing circuitry, including one or more processors implemented as fixed hardware processing circuitry, programmable processing circuitry, and/or a combination of both fixed and programmable processing circuitry. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (codec) in the respective device. Devices including video encoder 20, video decoder 30, and/or region determination unit 34 may include integrated circuits, microprocessors, and/or wireless communication devices, such as cellular telephones.

Various video coding techniques are described below with reference to the HEVC standard. However, the techniques of this disclosure may be used with any video coding technique used with 360 degree video, including future video coding standards such as h.266.

In HEVC and other video coding specifications, a video sequence typically includes a series of pictures. Pictures may also be referred to as "frames". A picture may contain three arrays of samples, denoted as S_L、S_CbAnd S_Cr。S_LIs a two-dimensional array (i.e., block) of luma samples. S_CbIs a two-dimensional array of Cb chroma samples. S_CrIs a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to herein as "chroma" samples. In other casesA picture may be monochrome, and may include only an array of luma samples.

To generate an encoded representation of a picture, video encoder 20 may generate a set of Coding Tree Units (CTUs). Each of the CTUs may comprise a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples, and syntax structures used to code samples of the coding tree blocks. In a monochrome picture or a picture having three separate color planes, a CTU may comprise a single coding tree block and syntax structures used to code the samples of the coding tree block. A coded block may be an nxn block of samples. A CTU may also be referred to as a "treeblock" or a "largest coding unit" (LCU). The CTUs of HEVC or other standards may be roughly similar to the macroblocks of other standards such as h.264/AVC. However, the CTUs are not necessarily limited to a particular size and may include one or more Coding Units (CUs). A slice may include an integer number of CTUs ordered consecutively in raster scan order.

The CTB contains a quad-tree whose nodes are coding units. The size of the CTB may range from 16 × 16 to 64 × 64 in the HEVC main specification (although 8 × 8CTB sizes may be supported technically). But the Coding Unit (CU) may be the same size as the CTB and as small as 8 x 8. Each decoding unit is decoded using one mode. When a CU is inter coded, the CU may be further partitioned into 2 or 4 Prediction Units (PUs), or into only one PU when further partitioning is not applicable. When two PUs are present in one CU, they may be half-sized rectangles or two rectangles of size 1/4 or 3/4 of the CU.

To generate a coded CTU, video encoder 20 may recursively perform quadtree partitioning on a coding tree block of the CTU to divide the coding tree block into coding blocks, hence the name "coding tree unit. The coding block may be an nxn block of samples. A CU may include a coding block of luma samples and two corresponding coding blocks of chroma samples of a picture having a luma sample array, a Cb sample array, and a Cr sample array, as well as syntax structures used to code the samples of the coding blocks. In a monochrome picture or a picture having three separate color planes, a CU may comprise a single coding block and syntax structures used to code the samples of the coding block.

Video encoder 20 may partition the coding block of the CU into one or more prediction blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples to which the same prediction is applied. A Prediction Unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and a syntax structure used to predict the prediction blocks. In a monochrome picture or a picture with separate color planes, a PU may include a single prediction block and syntax structures used to predict the prediction block. Video encoder 20 may generate, for a luma prediction block, a Cb prediction block, and a Cr prediction block for each PU of the CU, a predictive luma block, a predictive Cb block, and a predictive Cr block.

Video encoder 20 may use intra prediction or inter prediction to generate the predictive blocks for the PU. If video encoder 20 uses intra prediction to generate the predictive blocks for the PU, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of the picture associated with the PU. If video encoder 20 generates the predictive blocks of the PU using inter prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more pictures other than the picture associated with the PU. When a CU is inter coded, there may be a set of motion information for each PU. In addition, each PU may be coded with a unique inter prediction mode to derive the set of motion information.

After video encoder 20 generates the predictive luma block, the predictive Cb block, and the predictive Cr block for one or more PUs of the CU, video encoder 20 may generate the luma residual block for the CU. Each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predictive luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. In addition, video encoder 20 may generate a Cb residual block for the CU. Each sample in the Cb residual block of the CU may indicate a difference between the Cb sample in one of the predictive Cb blocks of the CU and the corresponding sample in the original Cb coding block of the CU. Video encoder 20 may also generate a Cr residual block for the CU. Each sample in the Cr residual block of the CU may indicate a difference between a Cr sample in one of the predictive Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

Moreover, video encoder 20 may use quad-tree partitioning to decompose the luma, Cb, and Cr residual blocks of the CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (e.g., square or non-square) block of samples to which the same transform is applied. A Transform Unit (TU) of a CU may comprise a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. A luma transform block associated with a TU may be a sub-block of a luma residual block of a CU. The Cb transform block may be a sub-block of a Cb residual block of the CU. The Cr transform block may be a sub-block of a Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may comprise a single transform block and syntax structures used to transform the samples of the transform block.

Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar. Video encoder 20 may apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of a TU to generate a Cr coefficient block for the TU.

After generating the coefficient blocks (e.g., luma coefficient blocks, Cb coefficient blocks, or Cr coefficient blocks), video encoder 20 may quantize the coefficient blocks. Quantization generally refers to the process of quantizing transform coefficients to possibly reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform Context Adaptive Binary Arithmetic Coding (CABAC) on syntax elements that indicate quantized transform coefficients.

Video encoder 20 may output a bitstream that includes a sequence of bits that forms a representation of coded pictures and associated data. The bitstream may comprise a sequence of NAL units. A NAL unit is a syntax structure that contains an indication of the type of data in the NAL unit and bytes containing the data, in the form of RBSPs interspersed with emulation prevention bits if necessary. Each of the NAL units includes a NAL unit header and encapsulates a RBSP. The NAL unit header may include a syntax element indicating a NAL unit type code. The NAL unit type code specified by the NAL unit header of the NAL unit indicates the type of the NAL unit. An RBSP may be a syntax structure containing an integer number of bytes encapsulated within a NAL unit. In some cases, the RBSP includes zero bits.

Different types of NAL units may encapsulate different types of RBSPs. For example, a first type of NAL unit may encapsulate an RBSP for a PPS, a second type of NAL unit may encapsulate an RBSP for a coded slice, a third type of NAL unit may encapsulate an RBSP for an SEI message, and so on. NAL units that encapsulate RBSPs for video coding data (as opposed to RBSPs for parameter sets and SEI messages) may be referred to as VCL NAL units.

Video decoder 30 may receive a bitstream generated by video encoder 20. In addition, video decoder 30 may parse the bitstream to obtain the syntax elements from the bitstream. Video decoder 30 may reconstruct pictures of the video data based at least in part on syntax elements obtained from the bitstream. The process to reconstruct the video data may be substantially reciprocal to the process performed by video encoder 20. In addition, video decoder 30 may inverse quantize coefficient blocks associated with TUs of the current CU. Video decoder 30 may perform an inverse transform on the coefficient blocks to reconstruct transform blocks associated with TUs of the current CU. Video decoder 30 may reconstruct the coding blocks of the current CU by adding samples for predictive blocks of PUs of the current CU to corresponding samples of transform blocks of TUs of the current CU. By reconstructing the coding block for each CU of a picture, video decoder 30 may reconstruct the picture.

Fig. 2A and 2B are conceptual diagrams showing representations of models of displays used to present panoramic and/or 360 degree video data. FIG. 2A shows an example of a cube map projection 100, while FIG. 2B shows an equivalent rectangular projection 110.

In FIG. 2A, each of the 6 faces 102A-102F of cube 104 (face 102) is divided into four dies (24 dies total). However, in some examples, each of the faces may include only one die. The die of the visible faces (i.e., faces 102A, 102B, and 102C) are labeled as die 106A-106L. Specifically, facet 102C is divided into dice 106A-106D, facet 102B is divided into dice 106E-106H, and facet 102A is divided into dice 106I-106L. The die of the hidden faces (i.e., faces 102D, 102E, and 102F) are not labeled in fig. 2A for readability, but it should be understood that faces 102D-102F are also divided into dies. A "die" may also be referred to as a region. Each of the cube faces 102 in fig. 2A corresponds to a 90 degree by 90 degree field of view (FoV). Any arbitrary 90 x 90 degree tile of a sphere may need to decode 1/3 of a panorama at high resolution. FoV rarely spans more than eight dice. Thus, the span of high resolution decoding can be limited to eight tiles or fewer without loss of visual quality.

FIG. 2B depicts the canvas 118 divided into eight tiles 116A-116H. In this example, when the user is observing the "pole" of the sphere (e.g., north pole 112, with the user's field of view when observing north pole 112 being represented by region 114), the entire upper half of the canvas 118 (i.e., the dies 116A-116D) will need to be decoded at high resolution. Splitting cell slices 116A-116D into more vertical cell slices would not help solve the problem. Thus, in this example, half of the panorama would need to be decoded at high resolution.

As discussed above, projection and mapping may be used to represent a 3D surface on a 2D map. In 360 degree video applications, projection is used to map 360 degree video data represented on a sphere onto a two dimensional video frame. Example projections include cube map projections and adjusted cube map projections (ACPs). In general, video encoder 20 may use cube map projection and/or ACP to map points on the surface of a sphere of 360 degree video to points on a plane tangent to the sphere surface (e.g., a face of a cube), e.g., as shown in fig. 2A. The resulting cube may be mapped to a 2D frame by encapsulating the video data using various encapsulation schemes. Although this disclosure discusses the projection of a spherical 360 degree video onto six faces of a cube, it should be understood that the techniques of this disclosure may be used with other types of projections, including other cube-based projections as well as projections to other shapes.

In some examples of this disclosure, video encoder 20 may be configured to generate and signal, in an encoded video bitstream, one or more syntax elements indicating the type of projection. Video decoder 30 may be configured to receive and parse one or more syntax elements to determine projections. In other examples, the projection to be used may be predetermined and stored at both video encoder 20 and video decoder 30. As will be explained in more detail below, video decoder 30 may be configured to use the same projections as those used by video encoder 20 to generate reference frames having extended faces (e.g., faces larger than the faces generated by video encoder 20). The extended face may have more samples than the initial sampling face.

Fig. 3 shows an example of a 3 x 2 frame encapsulation that may be used by video encoder 20 and video decoder 30 to encapsulate cube map projections and/or ACPs. As shown in fig. 3, six faces of a cube map projection or ACP (or other projection type) for a frame of 360 degree video data may be packaged into a frame 200. Frame 200 is a data structure and may be considered similar to a frame or picture of 360 degree video data and may be processed similarly to a frame or picture of 2D video data (e.g., an HEVC picture). However, frame 200 contains video data from each of the six faces of a projection of 360 degrees of video data onto the cube (e.g., cube map projection or ACP).

As shown in fig. 3, video data for left cube face 202 is packed in the upper left corner of frame 200, video data for front cube face 204 is packed in the middle-upper right corner of frame 200, video data for right cube face 206 is packed in the upper right corner of frame 200, video data for bottom cube face 208 is packed in the lower left corner of frame 200, video data for back cube face 210 is packed in the lower middle corner of frame 200, and top cube face 212 is packed in the lower right corner of frame 200. As shown in fig. 3, the arrangement of text for the left cube face, front cube face, right cube face, bottom cube face, back cube face, and top cube face also indicates the orientation of the video data within frame 200. It should be understood that the 3 x 2 packing arrangement of frames 200 is merely an example, and other orders and orientations of cube faces may be used.

In some examples of this disclosure, video encoder 20 may be configured to generate and signal, in an encoded video bitstream, one or more syntax elements indicating a packaging scheme used to package cube map projections and/or ACP projections into frames 200. Video decoder 30 may be configured to receive and parse one or more syntax elements to determine a packaging scheme. In other examples, the packaging scheme to be used may be predetermined and stored at both video encoder 20 and video decoder 30. As will be explained in more detail below, video decoder 30 may be configured to use the same packing scheme as that used by video encoder 20 to generate reference frames having extended faces (e.g., faces larger than the faces generated by video encoder 20). The extended face may have more samples than the initial sampling face.

When generating and packaging cube map projections and/or ACPs, there may be discontinuities or distortions along the edges of each cube face. The distortion arises from the projection technique used, while the discontinuity may be a result of the packaging scheme. Due to the projection technique used, deformations can often occur at certain cube face boundaries (e.g., deformations between left and front cube faces or between front and right cube faces).

The top columns of the three cube faces are continuous, representing the left side, front face, and right side. Similarly, the bottom column of three cube faces represents the top cube face, the back cube face, and the bottom cube face. However, the bottom column is rotated 90 degrees. Thus, there may be discontinuities along the boundaries of the cube faces. For example, video data at the bottom of the front cube face 204 may not flow directly to the top edge of the back cube face 210 even if it is adjacent. This is because, with a 90 degree clockwise rotation of the back cube face 210 in the packaging scheme shown in fig. 3, the top edge of the back cube face 210 is actually the left edge of the video data of the back cube face 210.

These kinds of deformations and discontinuities along the cube faces may reduce coding efficiency and/or increase distortion when performing inter-prediction during video coding. Objects that intersect a cube face over a duration of time may not be efficiently predicted due to discontinuities and deformations.

Fig. 4 shows an example of possible inter-prediction inefficiencies that may be due to deformation at cube face boundaries. As shown in fig. 4, the frame 200 includes a left side 202, a front side 204, a right side 206, a bottom side 208, a back side 210, and a top side 212. The frame 200 is encapsulated in the same manner as described above with reference to picture 3. Video encoder 20 may generate the faces of frame 200 using any projection technique, including cube map projection or ACP. In the example of fig. 4, video encoder 20 and/or video decoder 30 may be configured to perform inter-prediction for block 214. For illustrative purposes, FIG. 4 shows block 214 as including an elliptical object. This elliptical object is meant to represent any portion of the image represented in the 360 degree video data.

Reference frame 300 is a frame of encapsulated 360 degree video data that has been previously reconstructed and/or decoded by video encoder 20 or video decoder 30. The reference frame 300 may be stored in a decoded picture buffer. The reference frame 300 includes a left side 302, a front side 304, a right side 306, a bottom side 308, a back side 310, and a top side 312. The reference frame 300 may be packaged in the same manner as the frame 200, and the faces of the reference frame 300 may be generated using the same projection techniques as the frame 200.

As shown in fig. 4, the reference frame 300 may include a reference block 314 corresponding to the block 214 in the frame 200. Video encoder 20 may locate reference block 314 using a motion estimation process. Video encoder 20 may indicate the location of reference block 314 to video decoder 30. As shown in fig. 4, the reference block 314 intersects the boundaries of the front face 304 and the right side face 306. Due to the deformation caused by the projection process, the portion of the elliptical object in the reference block 314 along the edges of the front face 304 and the right side face 306 is deformed relative to the elliptical object in the block 214 of the frame 200. As such, predicting the block 214 from the reference block 314 may result in distortion and/or coding efficiency losses.

Fig. 5 shows an example of possible inter-prediction inefficiencies that may be due to discontinuities at cube face boundaries. As shown in fig. 5, the frame 200 includes a left side 202, a front side 204, a right side 206, a bottom side 208, a back side 210, and a top side 212. The frame 200 is encapsulated in the same manner as described above with respect to fig. 3. Video encoder 20 may generate the faces of frame 200 using any projection technique, including cube map projection or ACP. In the example of fig. 5, video encoder 20 and/or video decoder 30 may be configured to perform inter-prediction for block 216. For illustrative purposes, FIG. 5 shows block 216 as including an elliptical object. This elliptical object is meant to represent any portion of the image represented in the 360 degree video data.

As shown in fig. 5, the reference frame 300 may include a reference block 316 corresponding to the block 216 in the frame 200. Video encoder 20 may locate reference block 316 using a motion estimation process. Video encoder 20 may indicate the location of reference block 316 to video decoder 30. As shown in fig. 5, the reference block 316 intersects the boundaries of the right side face 306 and the top face 306. Due to the discontinuity between the faces caused by the packaging scheme, the portion of the oval-shaped article in the reference block 314 above the edge of the top face 212 is actually located in the bottom face 308. As such, predicting the block 216 from the reference block 316 may result in distortion and/or coding efficiency losses because all of the elliptical objects are not located in the reference block 316.

One technique to address the preservation of neighboring information (i.e., the portions of the image that are near or cross face boundaries) and the reduction of potential distortion involves projecting objects from neighboring cube face planes to a plane that resides in the current block to be coded, as an extension of the face plane of the currently coded block. Examples of these techniques are described in each of the following: sauer, m.wien, "geometric correction for motion compensation of planar projected 360VR video" (jviet-D0067, 2016; and x.ma, h.yang, z.zhao, l.li, h.li, "Co-projection screen based motion compensated prediction for cubic format VR content" (Co-projection-based motion compensated VR content), jfet-D0061, 2016. Examples of these techniques are also depicted in FIG. 6.

As shown in fig. 6, video encoder 20 and/or video decoder 30 may employ wider projections (shown as extension 400 and extension 402) to extend the video data represented in face 1. Extension 400 may include portions of video data that may be in facet 3, while extension 402 may include portions of video data that may be in facet 2. Likewise, video encoder 20 and/or video decoder 30 may employ wider projections (shown as extensions 404 and 406) to extend the video data represented in plane 2. Extension 404 may include portions of video data that may be in facet 1, while extension 406 may include portions of video data that may be in facet 4. Thus, the resulting extension of the boundary of a cube face includes pixels from the cube face adjacent to the particular cube face. Video encoder 20 and/or video decoder 30 may use ACP to derive extended samples for video data initially projected using ACP, and may use cube map projection to derive extended samples for video data initially projected using regular cube map projection.

To address the shortcomings of using inter-prediction to code cube maps and ACP projections, this disclosure describes techniques that include extending the faces of a decoded encapsulated cube map frame or ACP frame and forming a reference frame according to the encapsulated extended faces. In this way, objects in the video data that are near the boundary of a face will be more likely to be within the same face of a reference frame having an extended face. Thus, the problems described above caused by deformation and discontinuities along the face boundaries may be reduced.

The techniques of this disclosure may be performed by both video encoder 20 and video decoder 30. For example, after encoding a frame of video data, video encoder 20 may be configured to reconstruct (e.g., decode) the encoded frame of video data and store the encoded frame of video data as a reference frame in a decoded picture buffer. Using the techniques of this disclosure, video encoder 20 may be configured to process reconstructed frames of video data to create extended planes and package and store the extended planes as extended plane reference frames in a decoded picture buffer. Likewise, using the techniques of this disclosure, video decoder 30 may be configured to process decoded frames of video data to establish extended planes and package and store the extended planes as extended plane reference frames in a decoded picture buffer. Video decoder 30 may use the same process as that used by video encoder 20 to establish the extended plane reference frame. Video encoder 20 and video decoder 30 may then use the extended plane reference frame as a reference for inter-prediction. The following techniques will be described with reference to video decoder 30. It should be understood, however, that the same techniques may be performed by video encoder 20 in the reconstruction loop when forming the reference pictures.

In one example of this disclosure, video decoder 30 may be configured to receive encoded frames of 360-degree video data in an encoded video bitstream. The encoded frames of 360-degree video data may be arranged in a packing plane obtained from a projection of a sphere of 360-degree video data (e.g., cube map projection or ACP). For example, encoded frames of 360 degree video data may be packaged as shown in fig. 3. Of course, other packaging arrangements may be used. Video decoder 30 may be further configured to decode frames of encoded 360-degree video data to obtain decoded frames of 360-degree video data, the decoded frames of 360-degree video data being arranged in the same arrangement of packing planes.

In accordance with the techniques of this disclosure, video decoder 30 may be configured to process decoded frames of 360-degree video data to generate reference frames of 360-degree video data having extended cube faces. Video decoder 30 may be configured to extend the cube face by sampling a sphere of 360-degree video data derived from a decoded frame of 360-degree video data, e.g., as shown in fig. 5. That is, depending on the projection used, video decoder 30 may derive a decoded sphere of 360 degrees video data from a decoded frame of 360 degrees video data. As discussed above, the projections used may be predetermined and stored at both video encoder 20 and video decoder 30, and/or video encoder 20 may signal the projections used to video decoder 30.

Video decoder 30 may then sample the decoded sphere of 360 degrees video data back into the extended surface using the projection used by video encoder 20 (e.g., cube map projection, ACP, or other projections as described with respect to fig. 2A and 2B). However, unlike the projection used to create the encoded frame of 360 degree video data, video decoder 30 may sample the decoded sphere of 360 degree video to generate an extended surface of the packing surface that is greater than the decoded frame of 360 degree video data. That is, the extended face includes the boundaries of the extended pixels surrounding the original received and decoded cube face. Video decoder 30 may then derive the extended reference frame from the extended surface by encapsulating the extended surface into a reference frame using an encapsulation scheme. The encapsulation scheme may be the same encapsulation scheme used for encoded frames of 360 degree video data. Likewise, the encapsulation scheme used may be predetermined and stored at both video encoder 20 and video decoder 30, and/or video encoder 20 may signal the encapsulation scheme used to video decoder 30.

The derivation of the extended pixels may be regular because the extended pixels will be used for prediction of subsequent frames. In some examples, an approximation of a floating point projection of extended pixels may be specified. In some examples, the amount of extension (e.g., in terms of the number of pixels) may be configurable. For example, video encoder 20 may generate and signal syntax at the picture and/or sequence level in the parameter set that indicates how many pixels are to make the extended cube face larger relative to the original cube face. In other examples, the number of extended pixels may be predetermined and stored at both video encoder 20 and video decoder 30. In one example, the number of extended pixels may be the maximum prediction unit size for a given CTU size.

Fig. 7 shows a decoded/reconstructed encapsulated frame 500 and an extended reference frame 502 derived from the reconstructed encapsulated frame 500. As can be seen in fig. 7, each of the cube faces of the extended reference frame 502 is larger than the cube face of the decoded/reconstructed encapsulated frame 500. The cube face of extended reference frame 502 includes all of the video data of the cube face of decoded/reconstructed encapsulated frame 500 (e.g., white portion 504 of the left cube face), and a number of extended pixels surrounding the cube face (e.g., gray portion 506 of the left cube face). Thus, each of the cube faces in the extended reference frame 502 includes more neighboring pixels relative to the original boundaries between the cube faces of the decoded/reconstructed encapsulated frame 500. Thus, when using the reconstructed encapsulated frame 500 as a reference for inter-prediction, fewer pixels will be subject to the distortion and discontinuities caused by the projection process.

Video decoder 30 may then use extended reference frame 502 to decode a subsequently received encoded frame of 360 degrees video data using an inter-prediction process. FIG. 8 is a conceptual diagram showing a derived extended reference frame that may improve inter-prediction at deformed cube face boundaries. Compare the deformation shown in fig. 8 with fig. 4. Frame 200 of fig. 8 is the same as frame 200 in fig. 4. Instead of generating the reference frame 300 as shown in fig. 4, video encoder 20 and video decoder 30 may be configured to generate the extended reference frame 600 shown in fig. 8. Video encoder 20 and video decoder 30 may generate extended reference frame 600 using the techniques described above.

The extended reference frame 600 includes an extended left side 602, an extended front 604, an extended right side 606, an extended bottom 608, an extended back 610, and an extended top 612. As shown in fig. 8, the extended reference frame 308 may include reference blocks 614 that correspond to the blocks 214 in the frame 200. Video encoder 20 may locate reference block 614 using a motion estimation process. Video encoder 20 may indicate the location of reference block 614 to video decoder 30. As shown in fig. 4, the reference block 314 intersects the boundaries of the front face 304 and the right side face 306, causing inter-prediction errors due to distortion. However, as shown in fig. 8, the reference block 614 is entirely within the extended front face 604. Thus, any distortion that may have been present in the reference frame 300 of fig. 4 is mitigated due to the extended reference plane generated when the extended reference frame 600 was generated.

FIG. 9 is a conceptual diagram showing a derived extended reference frame that may improve inter-prediction at discontinuity cube face boundaries. The discontinuities shown in fig. 9 and 5 are compared. Frame 200 of fig. 9 is the same as frame 200 in fig. 5. Instead of generating reference frame 300 as shown in fig. 5, video encoder 20 and video decoder 30 may be configured to generate extended reference frame 600 shown in fig. 9. Video encoder 20 and video decoder 30 may generate extended reference frame 600 using the techniques described above.

The extended reference frame 600 includes an extended left side 602, an extended front 604, an extended right side 606, an extended bottom 608, an extended back 610, and an extended top 612. As shown in fig. 9, the extended reference frame 600 may include a reference block 616 corresponding to the block 216 in the frame 200. Video encoder 20 may locate reference block 616 using a motion estimation process. Video encoder 20 may indicate the location of reference block 616 to video decoder 30. As shown in fig. 5, the reference block 316 intersects the boundaries of the right side 306 and top 308 faces, causing inter-prediction error due to discontinuities. However, as shown in fig. 9, the reference block 616 is completely within the extended right side 606. Thus, any discontinuity that may have been present in the reference frame 300 of fig. 5 is mitigated due to the extended reference plane generated when the extended reference frame 600 was generated.

As discussed above, the extended reference frame generated using the techniques of this disclosure is larger than the decoded frame (e.g., includes more pixels). The collocated pixels of the currently decoded frame on the reference frame correspond to the locations of the pixels in the non-extended region (face) in the extended frame (e.g., white region 504 of fig. 7), i.e., all zero motion vector predictions will be pre-extended for the reference frame, i.e., before extension is applied. Extending the deformations and discontinuities at the disposal edge region. However, in some packaging schemes, the cube faces on the top and bottom columns are rotated 90 degrees. This may prevent efficient prediction from bottom to top, or vice versa, with large motions that move the object sufficiently across the cube face boundaries.

To handle motion compensation from a rotational plane, video encoder 20 and video decoder 30 may be configured to rotate a block (e.g., a prediction unit) of a currently coded block to align the orientation of the current block and its predicted block (e.g., a reference block or predicted block in a rotated cube plane of a reference frame). Video encoder 20 and video decoder 30 may be configured to determine whether to rotate the current block (or rotate the reference block) by determining a face orientation at a location of an upper left corner of the prediction block pointed to by the motion vector relative to a face orientation of the coded block.

FIG. 10 shows an example of current block 702 having PU1 and PU2 located in the back face 700 of current frame 650, current block 702 being predicted from PUs from the right side and top face. As shown in fig. 10, Motion Vector (MV)656 for the upper PU (PU1) of block 702 points to PU 1662 in the right side of reference frame 660, and MV658 for the lower PU (PU2) of block 702 points to PU 2664 in the top side of reference frame 660. In some examples, the faces of reference frame 660 may be extended using the techniques described above, such as shown in fig. 10. However, in other examples, the reference frame need not have an extended surface.

As shown in fig. 10, with the packing scheme used, the right side of both current frame 650 and reference frame 660 are rotated 90 degrees counterclockwise relative to the back side of current frame 650 and reference frame 660. Thus, in accordance with the techniques of this disclosure, video encoder 20 and video decoder 30 may be configured to rotate a PU in a current frame relative to an orientation of a reference block in a reference frame in order to match the orientation of the current block with the reference block. In the example of fig. 10, video encoder 20 and video decoder 30 may be configured to rotate PU1 of block 702 counterclockwise 90 degrees to match reference block 662. Video encoder 20 and video decoder 30 may be configured to rotate PU2 of block 702 clockwise by 90 degrees to match reference block 664.

Moreover, after prediction, video encoder 20 and video decoder 30 may rotate the residual samples back to the orientation of the back face of current frame 650 to form a final prediction block. For coded blocks located in the bottom and top faces predicted from the left, front, or right side, the PU will be rotated 90 degrees clockwise to align with the object in the face. A similar approach may be applied to the case where the blocks in the left, front and right sides are predicted from the blocks from the bottom, back and top faces to align face orientation. No rotation may be applied to predictions between faces that do not rotate relative to each other. As one example, no rotation may be applied for prediction between the left, front, and right flanks. As another example, no rotation may be applied for prediction between the bottom, back and top sides. Table 1 below provides an overview of example rotations of prediction blocks applied to all faces.

For packaging schemes using different cube-face orientations, a similar scheme of aligning the orientation of the cube-faces is applied. The encapsulation scheme of the encapsulation frame may be signaled in the parameter set. Block prediction rotation may also be applied to non-cube based projections, such as Rotating Spherical Projection (RSP), where some regions are rotated relative to another region in the envelope. In table 1, CW means clockwise rotation and CCW means counterclockwise rotation.

Table 1: a predicted block rotation across the face.

Fig. 11 is a block diagram showing an example of a video encoder 20 that may implement the techniques of this disclosure. Video encoder 20 may perform intra-coding and inter-coding of video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy of video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of a number of spatial-based coding modes. An inter mode, such as uni-directional prediction (P-mode) or bi-directional prediction (B-mode), may refer to any of a number of temporally based coding modes.

As shown in fig. 11, video encoder 20 receives a current frame of video data to be encoded. In an example of this disclosure, a video frame may be a frame of 360 degrees video data. A frame of 360 degrees video data may be an encapsulated cube face formed by a cube map projection of a sphere of 360 degrees video data or ACP.

In the example of fig. 11, video encoder 20 includes mode select unit 40, reference picture memory 64, which may also be referred to as a Decoded Picture Buffer (DPB), summer 50, transform processing unit 52, quantization unit 54, extension plane generation unit 63, and entropy encoding unit 56. Mode select unit 40, in turn, includes motion compensation unit 44, motion estimation unit 42, intra-prediction unit 46, and partition unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. A deblocking filter (not shown in fig. 11) may also be included to filter block boundaries to remove blockiness artifacts from the reconstructed video. The deblocking filter will typically filter the output of summer 62, if desired. In addition to deblocking filters, additional filters (in-loop or post-loop) may also be used. These filters are not shown for simplicity, but may filter the output of summer 62 (as an in-loop filter) if desired.

During the encoding process, video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into a plurality of video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive encoding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-prediction unit 46 may alternatively intra-predict the received video block using pixels of one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Video encoder 20 may perform multiple coding passes, e.g., to select an appropriate coding mode for each block of video data.

Furthermore, partition unit 48 may partition a block of video data into sub-blocks based on an evaluation of previous partition schemes in previous coding passes. For example, partition unit 48 may initially partition a frame or slice into a plurality of LCUs, and partition each of the LCUs into sub-CUs based on a rate-distortion analysis (e.g., rate-distortion optimization). Mode select unit 40 may further generate a quadtree data structure indicating the partitioning of the LCU into sub-CUs. Leaf-node CUs of a quad-tree may include one or more PUs and one or more TUs.

Mode select unit 40 may select one of the prediction modes (intra or inter), e.g., based on the error results, and provide the resulting prediction block to summer 50 to generate residual data and to summer 62 to reconstruct the encoded block for use as a reference frame. Mode select unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit 56.

Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of the video block. For example, the motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a predictive block within a reference frame (or other coded unit) relative to a current block being coded within the current frame (or other coded unit). A predictive block is a block that is found to closely match the block to be coded in terms of pixel differences, which may be determined by Sum of Absolute Differences (SAD), Sum of Squared Differences (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for sub-integer pixel positions of reference pictures stored in reference picture memory 64. For example, video encoder 20 may interpolate values for a quarter-pixel position, an eighth-pixel position, or other fractional-pixel positions of a reference picture. Thus, motion estimation unit 42 may perform a motion search with respect to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

Motion estimation unit 42 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the locations of the PUs to the locations of predictive blocks of the reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each of which identifies one or more reference pictures stored in reference picture memory 64. Motion estimation unit 42 sends the calculated motion vectors to entropy encoding unit 56 and motion compensation unit 44.

The motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on the motion vector determined by motion estimation unit 42. Again, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Summer 50 forms a residual video block by subtracting the pixel values of the predictive block from the pixel values of the current video block being coded, forming pixel difference values, as discussed below. In general, motion estimation unit 42 performs motion estimation with respect to luma components, and motion compensation unit 44 uses motion vectors calculated based on luma components for both chroma and luma components. Mode select unit 40 may also generate syntax elements associated with the video blocks and the video slice for use by video decoder 30 in decoding the video blocks of the video slice.

Further, motion compensation unit 44 may be configured to perform any or all of the techniques of this disclosure (either alone or in any combination). Although discussed with respect to motion compensation unit 44, it should be understood that mode selection unit 40, motion estimation unit 42, partition unit 48, and/or entropy encoding unit 56 may also be configured to perform certain techniques of this disclosure, either alone or in combination with motion compensation unit 44.

As an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44 as described above, intra-prediction unit 46 may intra-predict the current block. In particular, intra-prediction unit 46 may determine the intra-prediction mode to be used to encode the current block. In some examples, intra-prediction unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction unit 46 (or mode selection unit 40 in some examples) may select an appropriate intra-prediction mode from the tested modes for use.

For example, intra-prediction unit 46 may calculate rate-distortion values using rate-distortion analysis for various tested intra-prediction modes, and may select the intra-prediction mode having the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between an encoded block and an original, unencoded block, which is encoded to produce an encoded block, and the bit rate (i.e., number of bits) used to produce the encoded block. Intra-prediction unit 46 may calculate ratios from the distortions and rates of various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

After selecting the intra-prediction mode for the block, intra-prediction unit 46 may provide information to entropy encoding unit 56 indicating the selected intra-prediction for the block. Entropy encoding unit 56 may encode information indicating the selected intra-prediction mode. Video encoder 20 may include the following in the transmitted bitstream: configuration data, which may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables); definition of coding context for various blocks; and an indication of a most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table to be used for each of the contexts.

Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents one or more components that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform, to the residual block, producing a video block that includes transform coefficient values. Wavelet transforms, integer transforms, subband transforms, Discrete Sine Transforms (DST), or other types of transforms may be used instead of DCT. In any case, transform processing unit 52 applies the transform to the residual block, producing a block of transform coefficients. The transform may convert the residual information from the pixel domain to a transform domain, such as the frequency domain. Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The quantization level may be modified by adjusting a quantization parameter.

After quantization, entropy encoding unit 56 entropy codes the quantized transform coefficients. For example, entropy encoding unit 56 may perform Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), Probability Interval Partition Entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the contexts may be based on neighboring blocks. After entropy coding by entropy encoding unit 56, the encoded bitstream may be transmitted to another device (e.g., video decoder 30), or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain. In particular, summer 62 adds the reconstructed residual block to the motion compensated prediction block generated earlier by motion compensation unit 44 or intra-prediction unit 46 to generate a reconstructed video block for storage in reference picture memory 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-code a block in a subsequent video frame.

In accordance with the techniques of this disclosure, extended surface generation unit 63 may form extended reference frames from the reconstructed video blocks using the techniques described above. For example, using the techniques described above, video encoder 20 may receive and encode frames of 360 degree video data. The encoded frames of 360 degree video may be arranged in a packing plane obtained from a projection of a sphere of 360 degree video data. Video encoder 20 may reconstruct a frame of encoded 360-degree video data to obtain a reconstructed frame of 360-degree video data. Reconstructed frames of 360-degree video data are also arranged in a packing plane. Extended surface generation unit 63 may be configured to derive a decoded sphere of 360 degrees video data from a reconstructed frame of 360 degrees video data. Extended face generation unit 63 may be further configured to sample a decoded sphere of 360 degrees video data using projection (e.g., cube map projection or ACP) to generate an extended face. The extended surface is greater than the packing surface of a reconstructed frame of 360 degrees video data. Extended surface generation unit 63 may be configured to derive an extended reference frame from the extended surface and store the extended reference frame in reference picture memory 64. Video encoder 20 may then encode subsequent frames of the 360 degree video data using the inter-prediction process and the derived extended reference frame.

Fig. 12 is a block diagram showing an example of a video decoder 30 that may implement the techniques of this disclosure. In the example of fig. 12, video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference picture memory 82, an extended plane generation unit 81, and a summer 80. In some examples, video decoder 30 may perform a decoding pass that is substantially reciprocal to the encoding pass described with respect to video encoder 20 (fig. 11). Motion compensation unit 72 may generate prediction data based on the motion vectors received from entropy decoding unit 70, while intra-prediction unit 74 may generate prediction data based on the intra-prediction mode indicator received from entropy decoding unit 70.

As shown in fig. 12, a video decoder receives an encoded video bitstream that includes a current encoded frame of video data to be decoded. In an example of this disclosure, an encoded video frame may be an encoded frame of 360 degree video data. An encoded frame of 360-degree video data may be an encapsulated cube face formed by a cube map projection of a sphere of 360-degree video data or ACP.

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from video encoder 20. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, or intra-prediction mode indicators, among other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

When a video slice is coded as an intra-coded (I) slice, intra-prediction unit 74 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When a video frame is coded as an inter-coded (e.g., B or P) slice, motion compensation unit 72 generates predictive blocks for the video blocks of the current video slice based on the motion vectors and other syntax elements received from entropy decoding unit 70. The predictive block may be generated from one of the reference pictures within one of the reference picture lists. Video decoder 30 may use a default construction technique to construct the reference frame list based on the reference pictures stored in reference picture memory 82: list 0 and list 1.

Motion compensation unit 72 determines prediction information for the video blocks of the current video slice by parsing the motion vectors and other syntax elements and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) to code video blocks of the video slice, an inter-prediction slice type (e.g., B-slice or P-slice), construction information for one or more of the reference picture lists of the slice, an action vector for each inter-coded video block of the slice, an inter-prediction state for each inter-coded video block of the slice, and other information used to decode video blocks in the current video slice.

Motion compensation unit 72 may also perform interpolation based on interpolation filters for sub-pixel accuracy. Motion compensation unit 72 may use interpolation filters as used by video encoder 20 during encoding of video blocks to calculate interpolated values for sub-integer pixels of a reference block. In this case, motion compensation unit 72 may determine the interpolation filter used by video encoder 20 from the received syntax element and use the interpolation filter to generate the predictive block. Further, motion compensation unit 72 may be configured to perform any or all of the techniques of this disclosure (either alone or in any combination).

Inverse quantization unit 76 inverse quantizes (i.e., de-quantizes) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 70. The inverse quantization process may include using a quantization parameter QP calculated by video decoder 30 for each video block in a video slice_YTo determine the degree of quantization that should be appliedAnd likewise the degree of inverse quantization that should be applied.

The inverse transform unit 78 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to generate a residual block in the pixel domain.

After motion compensation unit 72 generates the predictive block for the current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform unit 78 with the corresponding predictive block generated by motion compensation unit 72. Summer 80 represents one or more components that perform this summation operation. Optionally, a deblocking filter may also be applied to filter the decoded blocks in order to remove blockiness artifacts. Other in-loop filters may also be used (within or after the coding loop) to smooth pixel transitions or otherwise improve video quality. The decoded video blocks in a given frame or picture are then stored in reference picture memory 82, reference picture memory 82 storing reference pictures for subsequent motion compensation. Reference picture memory 82 also stores decoded video for later presentation on a display device, such as display device 32 of fig. 1. For example, reference picture memory 82 may store decoded pictures.

In accordance with the techniques of this disclosure, extended surface generation unit 81 may use the techniques described above to form extended reference frames from decoded video blocks of a video frame. For example, using the techniques described above, video decoder 30 may receive encoded frames of 360 degrees video data. The encoded frames of 360 degree video may be arranged in a packing plane obtained from a projection of a sphere of 360 degree video data. Video decoder 30 may decode frames of encoded 360-degree video data to obtain decoded frames of 360-degree video data. The decoded frames of 360-degree video data are also arranged in a packing plane. Extended surface generation unit 81 may be configured to derive a decoded sphere of 360 degrees video data from a decoded frame of 360 degrees video data. The extended face generation unit 81 may be further configured to sample a decoded sphere of 360 degrees video data using projection (e.g., cube map projection or ACP) to generate an extended face. The extended surface is greater than an encapsulation surface of a decoded frame of 360 degrees video data. Extended surface generation unit 81 may be configured to derive an extended reference frame from the extended surface and store the extended reference frame in reference picture memory 82. Video decoder 30 may then decode subsequent encoded frames of the 360 degree video data using the inter-prediction process and the derived extended reference frame.

FIG. 13 is a flow chart showing an example encoding method of the present invention. Video encoder 20, including extended facet generation unit 63, may be configured to perform the techniques of fig. 13.

In one example of this disclosure, video encoder 20 may be configured to receive a sphere of 360 degrees video data (1300), and arrange the sphere of 360 degrees video data into a frame of an encapsulation surface obtained from a projection of the sphere of 360 degrees video data (1302). Video encoder 20 may be further configured to encode the frames of the packing plane to form frames of encoded 360-degree video data (1304), and then reconstruct the frames of encoded 360-degree video data to obtain reconstructed frames of 360-degree video data, the reconstructed frames of 360-degree video data being arranged in the packing plane (1306). Video encoder 20 may be further configured to derive a reconstructed sphere of 360 degrees video data from the reconstructed frame of 360 degrees video data (1308), and sample the reconstructed sphere of 360 degrees video data using projections to generate an extended plane, wherein the extended plane is greater than an encapsulation plane of the reconstructed frame of 360 degrees video data (1310). Video encoder 20 may be further configured to derive an extended reference frame from the extended surface (1312), and encode a subsequent frame of 360-degree video data using the inter-prediction process and the derived extended reference frame (1314).

In one example of the invention, the projection is a cube map projection or an adjusted cube map projection (ACP).

In another example of this disclosure, to sample a reconstructed sphere of 360 degrees video data using projections to generate an extended surface, video encoder 20 may be further configured to sample the reconstructed sphere of 360 degrees video data using projections according to a number of extended pixels to generate the extended surface.

In another example of this disclosure, video encoder 20 is further configured to generate an indication of the number of extended pixels at the picture level or the sequence level. In one example, the number of extension pixels is equal to the maximum prediction unit size in a Coding Tree Unit (CTU) of the encoded frame of the packing plane.

In another example of this disclosure, video encoder 20 is further configured to rotate a current prediction unit in a current cube face of an encoded frame of the encapsulation face based on a rotation angle of the reference cube face, the reference cube face containing a reference block for the current prediction unit.

FIG. 14 is a flow chart showing an example decoding method of the present invention. Video decoder 30, including extended facet generation unit 81, may be configured to perform the techniques of fig. 14.

In one example of this disclosure, video decoder 30 may be configured to receive an encoded frame of 360 degree video data, the encoded frame of 360 degree video data being arranged in a packing plane obtained from a projection of a sphere of 360 degree video data (1400), and decode the frame of encoded 360 degree video data to obtain a decoded frame of 360 degree video data, the decoded frame of 360 degree video data being arranged in a packing plane (1402). Video decoder 30 may be configured to derive a decoded sphere of 360 degrees video data from a decoded frame of 360 degrees video data (1404), and sample the decoded sphere of 360 degrees video data using projection to generate an extended surface, wherein the extended surface is greater than an encapsulation surface of the decoded frame of 360 degrees video data (1406). Video decoder 30 may be further configured to derive an extended reference frame from the extended surface (1408), and decode a subsequent encoded frame of 360 degrees video data using the inter-prediction process and the derived extended reference frame (1410).

In one example, the projection is a cube map projection or an adjusted cube map projection (ACP).

In another example of this disclosure, to sample a decoded sphere of 360 degrees video data using projections to generate an extended surface, video decoder 30 may be further configured to sample a decoded sphere of 360 degrees video data using projections according to a number of extended pixels to generate an extended surface. In another example of this disclosure, video decoder 30 may be further configured to receive an indication of the number of extended pixels at a picture level or a sequence level. In one example, the number of extended pixels is equal to a maximum prediction unit size in a Coding Tree Unit (CTU) of an encoded frame of 360 degrees video data. In another example of this disclosure, video decoder 30 may be further configured to rotate a current prediction unit in a current cube face of an encoded frame of 360 degrees video data based on a rotation angle of the reference cube face, the reference cube face containing a reference block for the current prediction unit.

It should be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, added, combined, or left out entirely (e.g., not all described acts or events are necessary to practice the techniques). Further, in some examples, acts or events may be performed concurrently, e.g., via multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include: a computer-readable storage medium corresponding to a tangible medium such as a data storage medium; or communication media including any medium that facilitates transfer of a computer program from one place to another, such as according to a communication protocol. In this manner, the computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but rather refer to non-transitory tangible storage media. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more Digital Signal Processors (DSPs), general purpose microprocessors, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor," as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functions described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding or incorporated in a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses including a wireless handset, an Integrated Circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, the various units may be combined in a codec hardware unit, as described above, or provided in conjunction with suitable software and/or firmware through a set of interoperability hardware units (including one or more processors as described above).

Various examples have been described. These and other examples are within the scope of the following claims.

41页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：用于在视频译码中构造候选列表的基于运动的优先级

Reference picture derivation and motion compensation for 360-degree video coding

相关技术

网友询问留言