Efficient sub-picture extraction
阅读说明：本技术 有效子图片提取 (Efficient sub-picture extraction ) 是由 罗伯特·斯库皮 科尼利厄斯·海勒格 瓦莱丽·乔治 托马斯·斯基尔勒 亚格·桑切斯·德·拉·富 于 2019-01-21 设计创作，主要内容包括：通过为每个切片提供开始位置信息,子图片提取变得不太复杂,开始位置信息指示相对于起始位置的开始位置,从开始位置向前,图片沿着编码路径被编码到相应切片中,编码路径在起始位置处开始遍历开始位置位于的段。可替换地或额外地,子图片提取过程被处理为类似于从多层数据流中选择一个层。(Sub-picture extraction becomes less complex by providing each slice with start position information indicating a start position relative to a start position from which onwards the picture is encoded into the respective slice along an encoding path starting at the start position through the segment at which the start position is located. Alternatively or additionally, the sub-picture extraction process is processed similar to selecting a layer from the multi-layer data stream.)
1. A data stream having pictures (18) encoded therein,
wherein the picture (18) is encoded into the data stream along an encoding path (22) in units of slices (26) into which the picture is divided,
wherein the picture (18) is subdivided into segments (30) that are sequentially traversed segment by the encoding path (22), wherein each slice (26) has a portion of a segment (30) or one or more segments (30) in their entirety encoded therein, wherein the picture (18) has segments encoded therein that do not have encoding interdependencies,
wherein each slice (26) comprises:
start position information (42) indicating a start position (40), the picture being encoded into the respective slice (26) along the encoding path (22) from the start position (40) onwards, wherein the start position information (42) indicates a start position (40) relative to a start position (44) at which the encoding path (22) starts traversing the segment (30) in which the start position (40) is located, and
segment information (48) indicating a segment (30) at which the start position (40) is located.
2. The data stream of claim 1, wherein the start position information (42) comprises:
a first syntax element (122) having a start address encoded therein, the start address addressing the start position (40) along the encoding path (22), the pictures being encoded into respective slices (26) proceeding from the start position (40), wherein the start address addresses the start position relative to the start position (44), the encoding path (22) beginning at the start position (44) traversing a segment (30) in which the start position (40) is located.
3. The data stream as set forth in claim 1,
wherein the start position information (42) comprises:
a marker (120) indicating whether the respective slice (26) is a first slice along a segment (30) of the encoding path.
4. The data stream as set forth in claim 3,
wherein the start position information (42) comprises, if the marker (120) indicates that the respective slice (26) is not the first slice (26) of the segment (30) along the encoding path (22):
a first syntax element (122) having a start address encoded therein, the start address addressing the start position (40) along the encoding path (22), the picture being encoded into a respective slice from the start position (40) onwards, wherein the start address addresses the start position (40) with respect to a start position (44), the encoding path (22) beginning at the start position (44) traversing a segment in which the start position is located.
5. The data stream according to claim 2 or 4, wherein the first syntax element (122) has the start address encoded therein using a variable length code.
6. The data stream of any one of claims 2, 4 and 5,
wherein the data stream comprises a start position coding mode flag (60; 112), the start position coding mode flag (60; 112)
Switchable between a first state and a second state, the first state indicating the start address encoded into the first syntax element (122) addressing the start position (40) relative to the start position (44), the encoding path beginning at the start position (44) traversing the segment at which the start position is located, the second state indicating the start address encoded into the first syntax element (122) addressing the start position (40) relative to a picture start position (24), the encoding path beginning at the picture start position (24) traversing the picture, and
is set to the first state.
7. The data stream as set forth in claim 6,
wherein the start position information (42) comprises:
a flag (120), said flag (120) indicating whether said respective slice (26) is a first slice of a segment (30) along said encoding path (22) if said start position coding mode flag (60; 112) assumes said first state, and said flag (120) indicating whether said respective slice is a first slice of said picture (18) along said encoding path if said start position coding mode flag (60; 112) assumes said second state.
8. The data stream of any one of claims 1 to 7,
wherein the data stream comprises a start position coding mode flag (60; 112), the start position coding mode flag (60; 112)
Switchable between a first state indicating that the start position information (42) indicates the start position (40) relative to the start position (44) at which the encoding path starts traversing the segment in which the start position is located, and a second state indicating that the start position information indicates the start position (40) relative to a picture start position (24) at which the encoding path (22) starts traversing the picture (18),
is set to the first state.
9. The data stream of any one of claims 1 to 8,
wherein the data stream has encoded therein video (20) consisting of a sequence of pictures including the picture (18),
wherein the sequence of pictures is spatially subdivided in a manner consistent with the subdivision of the pictures (18) into the segments (30), wherein the sequence of pictures is encoded into the data stream (10) picture by picture and in a manner such that each segment is encoded independently of portions of the sequence of pictures that are spatially outside the segment.
10. The data stream of any one of claims 1 to 9, the segment information (48) comprising:
a second syntax element (124) having a segment index encoded therein, the segment index indexing a segment (30) at which the start position (40) is located.
11. The data stream of claim 10, the second syntax element (124) having the segment index encoded therein using fixed length coding.
12. The data stream of claim 10 or 11, further comprising:
an index data field defining an association of a segment index value (114) with each segment (30), wherein the segment index encoded into the second syntax element uses the association.
13. The data stream of any one of claims 1 to 12, further comprising:
a base address data field (130), the base address data field (130) defining, for each segment (30), a base address of the starting position (44) of the respective segment, the base address addressing the starting position (44) relative to a picture starting position (24).
14. The data stream of any of claims 1 to 13, wherein the data stream comprises parameter sets (36) signaling a subdivision of the picture (18) into the segments (30), the signaled subdivision
In the range of the picture,
within a picture sequence, or
Within a picture range and a picture sequence range, wherein a subdivision within the picture range overrules a subdivision within the picture sequence range.
15. The data stream of any one of claims 1 to 14,
wherein the data stream comprises parameter sets (36), the parameter sets (36) indicating
The size of the picture;
an indication of a first decoder capability level required to decode the picture from the data stream;
for at least one sub-picture decoding option (148),
a further picture size corresponding to a segment cluster consisting of a subset of said segments, an
A second level of decoder capability required to decode the segment cluster from an extracted version of the data stream generated by stripping from the data stream slices having segments encoded therein that are spatially offset from the segment cluster.
16. The data stream as set forth in claim 15,
wherein the parameter set (36) comprises
For each of the at least one sub-picture decoding options,
at least one segment index set (156) indexing one or more segments that together form a version of a segment cluster of reduced picture size.
17. The data stream of claim 16, wherein the parameter set (36) comprises
For each of the at least one sub-picture decoding options,
for each of the at least one set of segment indices,
a priority indication indicating a priority at which the one or more segments indexed by the respective segment index set for the respective sub-picture decoding option are to be selected for forming the segment cluster.
18. The data stream of claim 17, wherein the priority indication
Separately within each of the at least one sub-picture decoding options,
or
Globally within all of the at least one sub-picture decoding options
The priority is indicated.
19. The data stream of any one of claims 16 to 18,
wherein the data stream is an extracted data stream (12), pictures having encoded therein the one or more segments restricted to a predetermined one of the at least one segment index set for a predetermined one of the at least one sub-picture decoding options, wherein slices having segments encoded therein that are spatially offset from the segment clusters are stripped from the data stream to form the extracted version of the data stream,
wherein the extracted data stream comprises extraction information indicating the predetermined one of the at least one segment index set for the predetermined one of the at least one sub-picture decoding option.
20. The data stream of any one of claims 16 to 19,
wherein the parameter set (36) comprises
For each of the at least one sub-picture decoding options,
for each of the at least one set of segment indices,
a decoding timestamp and/or an encoded picture buffer retrieval time and/or a buffer size (158) for decoding a version of the data stream, the version having pictures encoded therein restricted to the one or more segments indexed by the respective segment index set for the respective sub-picture decoding option, and the version being generated from the data stream by stripping slices having segments encoded therein that are spatially offset from the respective segment index set.
21. A decoder for decoding pictures (18) from a data stream (10),
wherein the picture (18) is encoded into the data stream along an encoding path (22) in units of slices (26) into which the picture is divided,
wherein the picture (18) is subdivided into segments (30) that are sequentially traversed segment by the encoding path (22), wherein each slice (26) has a portion of a segment (30) or one or more segments (30) in their entirety encoded therein, wherein the picture (18) has segments (30) encoded therein that do not have encoding interdependencies,
wherein the decoder is configured to, for each slice,
decoding start position information from the data stream,
-locating the start position (40) using the start position information (42) by using the start position information (42) for locating the start position (40) relative to a start position (44), the picture being encoded into a respective slice (26) from the start position (40) onwards, the encoding path starting at the start position along the encoding path traversing a segment (30) in which the start position is located, and
segment information (48) indicating a segment (30) in which the start position is located is decoded from the data stream.
22. Decoder according to claim 21, wherein the start position information (42) comprises
A first syntax element having a start address encoded therein, the start address addressing the start position along the encoding path, the picture being encoded into a respective slice from the start position onwards, wherein the decoder is configured to locate the start position by applying the start address relative to the start position at which the encoding path starts traversing the segment in which the start position is located.
23. The decoder of claim 21 or 22, wherein the decoder is configured to decode from the data stream
A marker (120) comprised by the start position information and indicating whether the respective slice is a first slice of a segment along the encoding path.
24. The decoder of claim 23, wherein the decoder is configured to
If the flag (120) indicates that the corresponding slice is not the first slice of a segment along the encoding path,
decoding from the data stream a first syntax element (120) comprised by the start position information and having encoded therein a start address addressing a start position along the encoding path from which the picture was encoded onwards into the respective slice, and
locating the start position by applying the start address as an offset relative to the start position at which the encoding path begins traversing the segment at which the start position is located.
25. The decoder of claim 22 or 24, configured to decode the first syntax element from the data stream using a variable length code.
26. The decoder of any of claims 22, 24 and 25, configured to
A mode flag (60; 112) is encoded from the data stream decoding start position,
if the start position coding mode flag is in the first state,
locating the start position by applying the start address relative to the start position at which the encoding path starts traversing the segment in which the start position is located, an
If the start position coding mode flag is in the second state,
the encoding path begins traversing the picture at a picture start position by applying the start address relative to the picture start position to locate the start position.
27. The decoder of claim 26, wherein the decoder is further configured,
wherein the decoder is configured to decode the marker included by the start position information from the data stream,
if the start position coding mode flag assumes the first state,
if the flag is set, determining that the corresponding slice is a first slice of a segment along the encoding path, an
If the start position coding mode flag assumes the second state,
determining that the respective slice is a first slice of the picture along the encoding path if the flag is set.
28. The decoder of any of claims 21 to 27, configured to
The mode flag is encoded from the data stream decoding start position,
if the start position coding mode flag has a first state,
using the start position information for positioning the start position relative to the start position, the encoding path starts traversing the segment at which the start position is located at the start position, an
If the start position coding mode flag has the second state,
using the start position information for positioning the start position relative to a picture start position, the encoding path starts traversing the picture at the picture start position.
29. Decoder according to any of the claims 21 to 28,
wherein the data stream has encoded therein video consisting of a sequence of pictures including the picture,
wherein the sequence of pictures is spatially subdivided in a manner consistent with the subdivision of the pictures into the segments, wherein the sequence of pictures is encoded into the data stream picture by picture and in a manner such that each segment is encoded independently of portions of the sequence of pictures that are spatially outside the segment.
30. The decoder of any of claims 21 to 29, further configured to
Decoding, from the data stream, a second syntax element included by the segment information, the second syntax element having a segment index encoded therein, the segment index indexing a segment at which the start position is located.
31. The decoder of claim 30, configured to decode the second syntax element using a fixed length code.
32. The decoder of claim 30 or 31, configured to
Decoding from the data stream an index data field defining an association of a segment index value with each segment,
using the association for identifying the segments indexed by the segment index.
33. The decoder of any of claims 21 to 31,
configured to decode from the data stream a base address data field defining for each segment a base address of the starting position of the respective segment, the base address addressing the starting position along the encoding path relative to the picture starting position, or configured to calculate for each segment the base address of the starting position of the respective segment, the base address addressing the starting position along the encoding path relative to the picture starting position, and
is configured to calculate, for each slice, a slice address for the respective slice using the base address at which the start position of the respective slice is located and the start position information decoded from the data stream for the respective slice.
34. Decoder according to any of the claims 21 to 33, wherein the parameter set (36) signals a subdivision of the picture (18) into the segments (30), the signaled subdivision
In the range of the picture,
within a picture sequence, or
Within a picture range and a picture sequence range, wherein a subdivision within the picture range overrules a subdivision within the picture sequence range.
35. Decoder according to any of the claims 21 to 34,
is configured to decode a parameter set from the data stream, the parameter set indicating
The size of the picture;
an indication of a first decoder capability level required to decode the picture from the data stream;
for at least one of the sub-picture decoding options,
a reduced picture size corresponding to a cluster of segments consisting of an appropriate subset of said segments, an
A second level of decoder capability required to decode the segment clusters from an extracted version of the data stream generated from the data stream by stripping from the data stream slices having segments encoded therein that are spatially offset from the segment clusters,
checking compliance of the first decoder capability level when decoding the picture from the data stream, an
Checking compliance of the second decoder capability level when decoding the segment cluster from the extracted version of the data stream.
36. The decoder of claim 35, configured as
Decoding at least one segment index set from each of the data stream or at least one sub-picture decoding option,
the at least one segment index set is comprised by the parameter set and indexes one or more segments that together form a version of the segment cluster of the reduced picture size.
37. The decoder of claim 36, configured to
Decoding from the data stream
For each of the subset of the set of at least one segment index set of all of the at least one sub-picture decoding options,
a priority indication comprised by the set of parameters,
decoding the one or more segments indexed by the set of segment indices from the data stream, the set of segment indices being prioritized by the priority indication in the subset of at least one set of segment indices.
38. The decoder of claim 37, configured to
By passing
Analyzing the segments of the data stream covered by the slices, or
External signalling, or
The side-information in the data stream is,
determining a subset of a set of the at least one segment index sets for all of the at least one sub-picture decoding options, the subset defining one or more segment index sets encoded into the data stream.
39. A decoder as claimed in claim 37 or 38, wherein the priority indication is
Within each of the at least one sub-picture decoding option, or
Globally within all of the at least one sub-picture decoding options,
prioritizing the at least one set of segment indices for each of the at least one sub-picture decoding option.
40. The decoder of any of claims 36 to 39,
wherein the parameter set comprises
For each of the at least one sub-picture decoding options,
for each of the at least one set of segment indices,
decoding timestamps and/or encoded picture buffer retrieval times for decoding a version of the data stream, the version having pictures encoded therein restricted to the one or more segments indexed by the respective segment index set for the respective sub-picture decoding option, and the version being generated from the data stream by stripping slices having segments encoded therein that are spatially offset from the respective segment index set.
41. Decoder according to any of the claims 21 to 40, wherein the decoder is configured to decode from the data stream the composition of sub-pictures (54) of the subset of segments of the picture (18), and to perform the decoding, locating and decoding subject to a slice whose segment index indicates any of the subset of segments and with respect to a subdivision of the sub-pictures into the subset of segments.
42. A data stream having pictures encoded therein,
wherein the pictures (18) are encoded into the data stream (10) along an encoding path (22) in units of slices (26) into which the pictures are divided,
wherein the picture is subdivided into segments (30) traversed sequentially, segment by segment, by the encoding path, wherein each slice has a portion of a segment or one or more segments in its entirety encoded therein, wherein the picture has segments encoded therein that do not have encoding interdependencies,
wherein each slice comprises
Start position information (42) indicating a start position (40) from which forward the picture is encoded into the respective slice,
wherein the data stream comprises a start position coding mode flag (60) switchable between a first state and a second state, wherein
If in the first state of the network it is,
the start position information indicates the start position relative to a start position (44) at which the encoding path starts traversing a segment in which the start position is located, an
Each slice includes segment information (48) indicating a segment at which the start position is located, an
If in the second state of the network it is,
the start position information indicates the start position relative to a picture start position (24) at which the encoding path starts traversing the picture, wherein the slice is free of the segment information (48).
43. The data stream of claim 42, wherein the start position information comprises:
a first syntax element having a start address encoded therein, the start address addressing the start position along the encoding path, the picture being encoded into the respective slice from the start position onwards, wherein
If the start position encoding mode flag is in the first state, the start address addresses the start position relative to the start position (44) at which the encoding path begins traversing the segment in which the start position is located, and
if in the second state, the start address addresses the start position relative to the picture start position (24).
44. The data stream according to claim 42 or 43, wherein said first syntax element has said start address encoded therein using a variable length code.
45. The data stream of any one of claims 42 to 44,
wherein the start position information includes
A flag indicating whether the corresponding slice is a first slice of a segment along the encoding path if the start position encoding mode flag assumes the first state, and whether the corresponding slice is a first slice of the picture along the encoding path if the start position encoding mode flag assumes the second state.
46. The data stream of any one of claims 42 to 45,
wherein the data stream has encoded therein video consisting of a sequence of pictures including the picture,
wherein the sequence of pictures is spatially subdivided in a manner consistent with the subdivision of the pictures into the segments, wherein the sequence of pictures is encoded into the data stream picture by picture and in a manner such that each segment is encoded independently of portions of the sequence of pictures that are spatially outside the segment.
47. The data stream of any one of claims 42 to 46, the segment information comprising:
a second syntax element having a segment index encoded therein, the segment index indexing the segment at which the starting position is located.
48. The data stream of claim 47, the second syntax element having the segment index encoded therein using a fixed length code.
49. The data stream of claim 47 or 48, further comprising:
an index data field defining an association of a segment index value to each segment, wherein the segment index encoded into the second syntax element uses the association.
50. The data stream of any one of claims 42 to 49,
if the start position coding mode flag is in the first state, further comprising;
a base address data field defining, for each segment, a base address of the starting position of the respective segment, the base address addressing the starting position relative to the picture starting position along the encoding path.
51. A decoder for decoding pictures from a data stream,
wherein a picture is encoded into the data stream along an encoding path in units of slices into which the picture is divided,
wherein the picture is subdivided into segments traversed sequentially, segment by segment, by the encoding path, wherein each slice has a portion of a segment or one or more segments in its entirety encoded therein, wherein the picture is encoded into the segments without encoding interdependencies,
wherein the decoder is configured to, for a predetermined slice,
decoding start position information indicating a start position from which the picture is encoded into the predetermined slice,
decoding a start position encoding mode flag switchable between a first state and a second state from the data stream,
if it is in the first state, then,
using the start position information to locate the start position relative to a start position at which the encoding path starts traversing the segment in which the start position is located, and decoding segment information from the data stream indicating the segment in which the start position is located, an
If it is in the second state, the first state,
using the start position information to locate the start position relative to a picture start position at which the encoding path starts traversing the picture, and determining the segment at which the start position is located based on the start position.
52. A data stream having pictures (18) encoded therein,
wherein the pictures (18) are encoded into the data stream (10) along an encoding path (22) in units of slices (26) into which the pictures are divided,
wherein the picture (18) is subdivided into segments (30) traversed sequentially, segment by segment, by the encoding path (22), wherein each slice has a portion of a segment or one or more segments in its entirety encoded therein, wherein the picture has segments (30) encoded therein that do not have encoding interdependencies,
wherein each slice comprises segment information (48) indicating the segments covered by the respective slice,
wherein the data stream comprises parameter sets (36), the parameter sets (36) indicating
The size of the picture;
an indication of a first decoder capability level required to decode the picture from the data stream;
for at least one of the sub-picture decoding options,
a reduced picture size corresponding to a cluster of segments consisting of an appropriate subset of said segments, an
A second level of decoder capability required to decode the segment cluster from an extracted version of the data stream generated from the data stream by stripping from the data stream slices having segments encoded therein that are spatially offset from the segment cluster.
53. The data stream of claim 52 wherein the data stream is selected from the group consisting of,
wherein the parameter set (36) comprises:
for each of the at least one sub-picture decoding options,
indexing at least one segment index set of one or more segments that together form a version of the reduced picture size segment cluster.
54. The data stream of claim 53, wherein the parameter sets comprise:
for each of the at least one sub-picture decoding options,
for each of the at least one set of segment indices,
a priority indication indicating a priority at which the one or more segments indexed by the respective segment index set for a respective sub-picture decoding option are to be selected for forming the segment cluster.
55. The data stream of claim 54, wherein the priority indication
Within each of the at least one sub-picture decoding option, or
Globally within all of the at least one sub-picture decoding options
Prioritizing the at least one set of segment indices for each of the at least one sub-picture decoding option.
56. The data stream of any one of claims 53 to 55,
wherein the data stream has encoded therein pictures restricted to the one or more segments indexed by a predetermined one of the at least one segment index set for a predetermined one of the at least one sub-picture decoding option, wherein slices having segments encoded therein that are spatially offset from the segment clusters are stripped from the data stream to form the extracted version of the data stream,
wherein the data stream includes extraction information indicating the predetermined one of the at least one segment index set for the predetermined one of the at least one sub-picture decoding option.
57. The data stream of any one of claims 53 to 56, wherein the set of parameters comprises:
for each of the at least one sub-picture decoding options,
for each of the at least one set of segment indices,
decoding timestamps and/or encoded picture buffer retrieval times and/or buffer sizes for decoding a version of the data stream, the version having the pictures encoded therein restricted to the one or more segments indexed by the respective segment index set for the respective sub-picture decoding option, and the version being generated from a data stream by stripping slices having segments encoded therein that are spatially offset from the respective segment index set.
58. A decoder configured to decode a parameter set (36) from a data stream (10),
deriving a decoding option indication (58) whether picture decoding or sub-picture decoding is to be performed on the data stream,
if the picture decoding is to be performed on the data stream,
deriving from the parameter set an indication of a size of a picture and a first decoder capability level required to decode the picture from the data stream, an
Deriving, from the parameter set, first information on subdividing the picture (18) into a first set of segments, the segments of the first set being encoded into the data stream without encoding interdependencies,
decoding the picture from the data stream along a first encoding path (22) in units of the slice (26) into which the picture is divided, wherein the first encoding path traverses the picture sequentially, segment by segment, and each slice has a portion of a segment or one or more segments in its entirety in a first set of the segments encoded therein,
if the sub-picture decoding is to be performed on the data stream,
deriving from the parameter set an indication of a further picture size and a second decoder capability level required for decoding sub-pictures of the further picture size from the data stream, an
Deriving, from the set of parameters, second information on subdividing the sub-picture into a second set of segments, the segments in the second set being encoded into the data stream without encoding interdependencies, and the second set being a subset of the first set of segments,
decoding the sub-picture from the data stream along a second encoding path in units of the slice into which the sub-picture is divided, wherein the second encoding path traverses the sub-picture sequentially, segment by segment, and each slice has a portion of a segment or a complete one or more segments of the second set of segments encoded therein,
wherein the decoder is configured to, for each slice,
segment information indicating a segment in which a start position in the first set of segments is located is decoded from the data stream, and the corresponding slice is encoded from the start position onward.
59. The decoder of claim 58, configured to, for each slice,
decoding start position information from the data stream, an
If the picture decoding is to be performed on the data stream,
by using the start position information relative to a first start position to locate the start position of a respective slice in the picture, the first encoding path starting at the first start position through a segment at which the start position of the respective slice is located,
if the sub-picture decoding is to be performed on the data stream,
the second encoding path starts traversing a segment at a second start position of the respective slice by locating the start position of the respective slice in the sub-picture using the start position information relative to the second start position.
60. The decoder of claim 59, wherein the start position information comprises:
a first syntax element having a start address encoded therein that addresses the start position within the segment in which the start position is located along a portion of the first and second encoding paths.
61. The decoder of claim 59 in which the decoder,
wherein the decoder is configured to decode from the data stream
A marker included by the start position information and indicating whether the respective slice is a first slice of a segment along the portion of the first and second encoding paths.
62. The decoder of claim 61 wherein the first and second decoders are,
wherein the decoder is configured to:
if the flag indicates that the respective slice is not the first slice of a segment along the portion of the first and second encoding paths,
decoding, from the data stream, a first syntax element having a start address encoded therein that addresses the start position within the segment in which the start position is located along a portion of the first and second encoding paths.
63. The decoder of claim 60 or 62, configured to decode the first syntax element from the data stream using a variable length code.
64. The decoder of any of claims 58 through 63, further configured to decode, from the data stream, a second syntax element included by the segment information, the second syntax element having a segment index encoded into a segment in which the segment at which the start position is located is indexed.
65. The decoder of claim 64 configured to decode the second syntax element using a fixed length code.
66. The decoder of any of claims 58 to 65, configured to:
by passing
Deriving from the first information a first index data field defining an association of a segment index value with each segment in the first set of segments,
deriving from the second information a second index data field that uses the association to define how the sub-picture is composed from the second set of segments,
to derive a subdivision of said sub-picture into said second set of segments,
wherein the decoder is configured to use the association for each slice to identify the segment at which the start position of the respective slice is located based on the segment information of the respective slice.
67. The decoder of any of claims 58 to 66,
is configured to derive from the set of parameters,
for each of the at least one sub-picture decoding options,
an instance of the reduced picture size and an indication of the second decoder capability level.
68. The decoder of claim 67 wherein the first and second decoders,
is configured to derive from the parameter set
For each of the at least one sub-picture decoding options,
a set of at least one instance of the second information.
69. The decoder of claim 68, configured as
Deriving from said parameter set
For each of the at least one sub-picture decoding options,
for each of the set of at least one instance,
an indication of the priority level is provided,
using the priority indication to select one of the sub-picture decoding options indicated by the decoding option indication as a basis for the sub-picture decoding to be performed on the data stream.
70. The decoder of claim 68 or 69, configured to
Deriving from said parameter set
For each of the at least one sub-picture decoding options,
for each of the set of at least one instance,
the decoding timestamp and/or the encoded picture buffer retrieval time and/or the buffer size.
71. The decoder of any of claims 58 to 70, configured to
By passing
Analyzing the segments covered by the slices present in the data stream, or
External signalling, or
The side-information in the data stream is,
receiving the decoding option indication.
72. A method for decoding a picture (18) from a data stream (10),
wherein the pictures are encoded into the data stream along an encoding path (22) in units of slices (30) into which the pictures are divided,
wherein the picture (18) is subdivided into segments (30) traversed sequentially segment by the encoding path, wherein each slice has a portion of a segment (30) or one or more segments (30) in their entirety encoded therein, wherein the picture has segments (30) encoded therein that do not have encoding interdependencies,
wherein for each slice, the method comprises:
decoding start position information from the data stream,
-using start position information (42) for locating a start position with respect to a start position (44) at which the coding path starts traversing a segment (30) in which the start position is located along the coding path, -using start position information (42) for locating a start position from which the picture is coded into the respective slice (26), and-locating a start position (40) using start position information (42), and
segment information (48) indicating a segment (30) in which the start position is located is decoded from the data stream.
73. A method for decoding pictures from a data stream,
wherein the picture is encoded into the data stream along an encoding path in units of slices into which the picture is divided,
wherein the picture is subdivided into segments traversed sequentially, segment by segment, by the encoding path, wherein each slice has a portion of a segment or one or more segments in its entirety encoded therein, wherein the picture is encoded into the segments without encoding interdependencies,
wherein the method comprises, for a predetermined slice,
decoding start position information indicating a start position from which the picture is encoded into the predetermined slice,
decoding a start position encoding mode flag switchable between a first state and a second state from the data stream,
if it is in the first state, then,
using the start position information to locate the start position relative to a start position at which the encoding path begins traversing a segment in which the start position is located; and said decoding of segment information from the data stream indicating said segment at which said start position is located, an
If it is in the second state, the first state,
using the start position information to locate the start position relative to a picture start position at which the encoding path starts traversing the picture; and determining the segment at which the start position is located based on the start position.
74. A decoding method, comprising:
decoding a parameter set (36) from a data stream (10),
deriving a decoding option indication (58) whether picture decoding or sub-picture decoding is to be performed on the data stream,
if the picture decoding is to be performed on the data stream,
deriving from the parameter set an indication of a size of a picture and a first decoder capability level required to decode the picture from the data stream, an
Deriving, from the parameter set, first information on subdividing the picture (18) into a first set of segments, wherein segments of the first set are encoded into the data stream without encoding interdependencies,
decoding the picture from the data stream along a first encoding path (22) in units of slices (26) into which the picture is divided, wherein the first encoding path traverses the picture sequentially, segment by segment, and each slice has a portion of a segment or a complete segment or segments of the first set of segments encoded therein,
if sub-picture decoding is to be performed on the data stream,
deriving from the parameter set an indication of a further picture size and a second decoder capability level required for decoding sub-pictures of the further picture size from the data stream, an
Deriving, from the set of parameters, second information on subdividing the sub-picture into a second set of segments, the segments of the second set being encoded into the data stream without encoding interdependencies, and the second set being a subset of the first set of segments,
decoding the sub-picture from the data stream along a second encoding path in units of slices into which the sub-picture is divided, wherein the second encoding path traverses the sub-picture sequentially, segment by segment, and each slice has a portion of a segment or a complete one or more segments of the second set of segments encoded therein,
for each of the slices, the slice is,
decoding segment information from the data stream indicating a segment in which a start position in the first set of segments is located, the corresponding slice being encoded from the start position onwards.
75. An encoder for encoding pictures (18) into a data stream (10),
wherein the encoder is configured to encode the picture into the data stream in a manner such that the picture is encoded along an encoding path (22) in units of slices (30) into which the picture is divided,
wherein the encoder is configured to encode the picture into the data stream in such a way that the picture (18) is subdivided into segments (30) which are traversed sequentially segment by the encoding path, wherein each slice has a part of the segments (30) encoded therein or one or more segments (30) in their entirety,
wherein the encoder is configured to encode the segments (30) without encoding interdependencies,
wherein the encoder is configured to, for each slice,
encoding start position information into the data stream such that using the start position information (42), a start position (40) from which the picture is encoded into the respective slice (26) is locatable by using the start position information (42) for locating the start position relative to a start position (44), the encoding path starting at the start position (44) along the encoding path traversing a segment (30) at which the start position is located, and
encoding into the data stream segment information (48) indicating the segment (30) at which the start position is located.
76. An encoder for encoding pictures into a data stream,
wherein the encoder is configured to encode the picture into the data stream in a manner such that the picture is encoded along an encoding path (22) in units of slices (30) into which the picture is divided,
wherein the encoder is configured to encode the picture into the data stream in such a way that the picture (18) is subdivided into segments (30) which are traversed sequentially segment by the encoding path, wherein each slice has a part of the segments (30) encoded therein or one or more segments (30) in their entirety,
wherein the encoder is configured to encode the segments (30) without encoding interdependencies,
wherein the encoder is configured to, for a predetermined slice,
encoding start position information indicating a start position from which the picture is encoded into a predetermined slice,
encoding a start position encoding mode flag switchable between a first state and a second state into the data stream such that
If it is in the first state, then,
the start position information locating the start position relative to a start position at which the encoding path starts traversing the segment in which the start position is located, and the encoder being configured to encode segment information into the data stream indicating the segment in which the start position is located, and
if it is in the second state, the first state,
the start position information positions the start position relative to a picture start position at which the encoding path starts traversing the picture such that a segment in which the start position is located is determinable based on the start position.
77. An encoder configured to
Encoding parameter sets (36) into a data stream (10), wherein
The parameter set indicates that the parameter set is,
for picture decoding to be performed on the data stream, an indication of a size of a picture and a first level of decoder capability required for decoding the picture from the data stream, and first information on subdividing the picture (18) into a first set of segments, wherein the segments of the first set are encoded into the data stream without encoding interdependencies, wherein the picture is decodable from the data stream along a first encoding path (22) in units of slices (26) into which the picture is divided in accordance with the picture decoding, wherein the first encoding path traverses the picture sequentially, segment by segment, and each slice has a portion of or one or more segments in full of the segments in the first set of segments encoded therein,
for sub-picture decoding to be performed on the data stream, a further picture size and an indication of a second decoder capability level required for decoding a sub-picture having the further picture size from the data stream, and second information on subdividing the sub-picture into a second set of segments, the segments of the second set being encoded into the data stream without encoding interdependencies, and the second set being a subset of a first set of the segments, wherein the sub-picture is decodable from the data stream along a second encoding path in units of the slice into which the sub-picture is divided in accordance with the sub-picture decoding, wherein the second encoding path sequentially traverses the sub-picture segment by segment, and each slice has a part of or one or more segments in their entirety encoded into the second set of segments,
for each of the slices, the slice is,
encoding segment information into the data stream, the segment information indicating a segment in which a start position in the first set of segments is located, from which forward a respective slice is encoded.
78. A method for encoding pictures (18) into a data stream (10),
wherein the method comprises encoding the picture into the data stream in such a way that the picture is encoded along an encoding path (22) in units of slices (30) into which the picture is divided,
wherein the pictures are encoded into the data stream in such a way that the pictures (18) are subdivided into segments (30), the segments (30) being traversed sequentially segment by the encoding path, wherein each slice has a portion of a segment (30) encoded therein or one or more segments (30) in their entirety,
wherein the segments (30) are encoded without encoding interdependencies,
wherein for each slice, the method comprises:
encoding start position information into the data stream such that using the start position information (42), a start position (40) is locatable by using the start position information (42) for locating the start position relative to a start position (44), the picture being encoded into a respective slice (26) proceeding from the start position (40), the encoding path starting at the start position (44) along the encoding path traversing a segment (30) in which the start position is located, and
segment information (48) indicating the segment (30) in which the start position is located is encoded into the data stream.
79. A method for encoding pictures into a data stream,
wherein the method comprises encoding the picture into the data stream in such a way that the picture is encoded along an encoding path (22) in units of slices (30) into which the picture is divided,
wherein the picture is encoded into the data stream such that the picture (18) is subdivided into segments (30) that are sequentially traversed segment by the encoding path, wherein each slice has a portion of a segment (30) encoded therein or one or more segments (30) in their entirety,
wherein the segments (30) are encoded without encoding interdependencies,
wherein for a predetermined slice, the method comprises:
encoding start position information indicating a start position from which the picture is encoded into the predetermined slice,
encoding a start position coding mode flag switchable between a first state and a second state into the data stream such that
If it is in the first state, then,
the start position information positions the start position relative to a start position at which the encoding path starts traversing the segment in which the start position is located, and the method comprises encoding into the data stream segment information indicating the segment in which the start position is located, and
if it is in the second state, the first state,
the start position information positions the start position relative to a picture start position at which the encoding path starts traversing the picture such that the segment in which the start position is located is determinable based on the start position.
80. An encoding method, comprising:
encoding parameter sets (36) into a data stream (10), wherein
The parameter set indicates that the parameter set is,
for picture decoding to be performed on the data stream, an indication of a size of a picture and a first level of decoder capability required for decoding the picture from the data stream, and first information on subdividing the picture (18) into a first set of segments, wherein the segments of the first set are encoded into the data stream without encoding interdependencies, wherein the picture is decodable from the data stream along a first encoding path (22) in units of slices (26) into which the picture is divided in accordance with the picture decoding, wherein the first encoding path traverses the picture sequentially, segment by segment, and each slice has a portion of or one or more segments in full of the segments in the first set of segments encoded therein,
for sub-picture decoding to be performed on the data stream, a further picture size and an indication of a second decoder capability level required for decoding a sub-picture having the further picture size from the data stream, and second information on subdividing the sub-picture into a second set of segments, the segments of the second set being encoded into the data stream without encoding interdependencies, and the second set being a subset of a first set of the segments, wherein the sub-picture is decodable from the data stream along a second encoding path in units of slices into which the sub-picture is divided, in accordance with the sub-picture decoding, wherein the second encoding path sequentially traverses the sub-picture segment by segment, and each slice has a part of a segment or one or more segments in their entirety encoded into the second set of segments,
for each of the slices, the slice is,
encoding segment information into the data stream, the segment information indicating a segment in which a start position in the first set of segments is located, from which forward a respective slice is encoded.
81. A computer program having a program code for performing the method of any one of claims 72 to 74 and 78 to 80 when run on a computer.
Technical Field
The present application relates to concepts for efficient sub-picture extraction.
Background
Sub-picture extraction is a process in which a data stream having video pictures already encoded therein is adapted to fit sub-picture regions without requiring re-encoding. For example, HEVC DAM3 MCTS extraction allows extracting sub-picture specific data streams from the original full-picture data stream without any re-encoding, e.g. in the case of motion compensated prediction and residual coding, by constantly subdividing the picture into tiles coded independently of each other, and grouping the tiles into tile sets, even motion compensated prediction is restricted to not cross tile set boundaries with respect to the tile sets. However, this MCTS extraction requires adjusting each NAL unit carrying slice data to adjust the slice segment address of the slice header carried.
Therefore, a sub-picture extraction process is known, but it would be advantageous to have an upcoming concept that reduces the complexity of the tasks involved.
Disclosure of Invention
It is an object of the invention to provide a data stream, a decoder and/or an encoder according to the concept of making sub-picture extraction less complex.
This object is achieved by the subject matter of the independent claims of the present application.
According to a first aspect of the application, sub-picture extraction is made less complex by providing start position information to each slice, the start position information indicating a start position relative to a start position from which the picture is encoded forward into the respective slice along an encoding path starting at the start position through the segment at which the start position is located. A picture is subdivided into segments that are traversed sequentially, segment by segment, by an encoding path, where each slice has a portion of the segments or one or more segments in its entirety encoded therein, where the picture is encoded into the segments without encoding interdependencies. Pictures are encoded into a data stream along an encoding path in units of slices into which the pictures are divided. In addition, each slice includes segment information indicating a segment at which the start position is located. Thus, the start position information and the segment information together enable a determination to be made within which segment the start position of the respective slice is located, and where in this segment. Since segments are encoded without coding interdependency, it is possible to remove one or more slices related to one segment without affecting the decodability of another segment. Furthermore, even if a new sub-picture is compiled using segments of a picture by shuffling or rearranging the segments of the picture and/or discarding some of the segments of the picture, it is possible to form a respective sub-picture specific data stream based on the original data stream by discarding the slices forming the segments of the sub-picture that are not encoded therein and employing not-yet discarded slices that still refer to the correct segments and that due to the relative position indication indicate the correct position of the start position of the respective slice within the respective segment. Thus, according to the first aspect of the present application, although the segments of the picture are rearranged and/or some segments are omitted in the sub-picture data stream, the data stream enables easy extraction of the sub-picture without modifying the start position information and the segment information. In other words, the sub-picture extraction process is made easier by having the opportunity to only omit or discard slices that do not belong to any segment included in the sub-picture to be extracted, and to adopt the remaining slices without having to modify the start position information.
According to an embodiment, the start position information comprises a variable length coded start address syntax element. Variable length coding can be used without any loss during sub-picture extraction, since the inversion start position information is not needed anyway during sub-picture extraction.
According to a further embodiment, the data stream is switchable between start position information by the start position coding mode flag, the start position information indicating a start position of the respective slice relative to a start position of the segment, or absolutely relative to a start position of the respective slice of the picture start position at which the coding path starts traversing the picture. According to this option, existing codecs that so far use absolute start position indication can be extended to take advantage of the relative position indication discussed herein. The decoder may be able to understand both types of start position indications provided by the start position information, or only one of them.
According to a further embodiment, the data stream further comprises a base address data field defining, for each segment, a base address of the start position of the respective segment, the base address addressing the start position along the encoding path relative to the picture start position. The transfer of these base addresses reuses the computational overhead for computing the base addresses based on subdividing the picture itself into segments.
According to an embodiment, the segment information includes a segment syntax element having a segment index encoded therein, the segment index indexing the segment at which the start position is located. The syntax element may be coded using fixed length coding. The data stream may include an index data field defining an association of a segment index value with each segment. In other words, the index data field may explicitly associate a tag, i.e., a segment index value, with a segment and set segment information of a slice to the tag or segment index value associated with the segment at which the start position of the corresponding slice is located. This flag can be easily modified during sub-picture extraction.
Another aspect of the application that may be combined with the first aspect is to make the sub-picture extraction process easier by handling a sub-picture extraction process similar to selecting a layer from a multi-layer data stream. That is, according to the second aspect, the extractable data stream comprises parameter sets indicating not only the size of the pictures of the data stream and an indication of a first decoder capability level required to decode the pictures from the data stream, but also at least one sub-picture decoding option and a reduced picture size for the at least one sub-picture decoding option and a second decoder capability level required to decode the sub-pictures from the extracted version of the data stream. This sub-picture decoding option is treated as a sub-layer with respect to the higher layer corresponding to the full picture: an extracted version of the data stream is derived from the data stream by stripping or discarding from the data stream slices having segments encoded therein that are spatially offset from the cluster of segments that make up the sub-picture. The slice may or may not include the start position information and the segment information as discussed above with respect to the first aspect. However, it may be sufficient if each slice comprises segment information indicating the segments covered by the respective slice. Thus, according to the second aspect, the process of sub-picture extraction is similar to the conversion between several layers of a multi-layer data stream and indeed only slices not belonging to the intended layer need to be omitted, i.e. here slices not covering any segment included in the sub-picture to be extracted. There is no need to "convert" or "modify" the slice of the data stream to be taken into the extracted data stream.
Advantageous aspects are the subject matter of the dependent claims.
Drawings
Preferred embodiments of the present application are described below with reference to the accompanying drawings, in which:
fig. 1 shows a schematic diagram illustrating the concept of using relative start position indications for slices for mitigating an extractable data stream of a sub-picture extraction process, wherein fig. 1 shows a data stream not yet extracted and the extraction process and the extracted data stream as well as participating entities including encoders, decoders and optional network devices;
fig. 2 shows a schematic diagram illustrating a portion of a picture and a data stream to illustrate an embodiment in which the data stream is switchable between a relative slice start position indication and an absolute slice start position indication;
fig. 3 shows a schematic diagram illustrating an embodiment of a data stream as presented in relation to fig. 1 as a variant of an HEVC data stream;
fig. 4 shows pseudo code for illustrating content conveyed within a parameter set, here illustratively a picture parameter set, as it may be used in the embodiment of fig. 3;
fig. 5 shows a schematic diagram of a picture and its segmentation into segments, and parameters used in pseudo code presented in relation to the embodiment example provided in fig. 3;
fig. 6 shows pseudo code indicating possible content transmitted in slice headers according to the embodiment example of fig. 3;
FIG. 7 shows pseudo code indicating a block syntax (tiling syntax) that may be used to implement the codec example of FIG. 3;
fig. 8 shows pseudo code illustrating possible contents of parameter sets, here illustratively sequence parameter sets using the block syntax of fig. 7;
fig. 9 shows pseudo code for illustrating possible contents of parameter sets, here illustratively picture parameter sets using the block syntax of fig. 7;
FIG. 10 shows pseudo code illustrating possible content for implementing the content of the slice header of the example of FIG. 3, differing from FIG. 6 in that the segments may be components of one or more tiles;
fig. 11 shows pseudo code illustrating possible contents of a picture parameter set, here exemplarily a sequence parameter set, illustrating the possibility of the parameter set to convey information for more than a full picture decoding option, but additionally at least one sub-picture decoding option.
Detailed Description
A first embodiment of the present application is described with reference to fig. 1. Fig. 1 shows a data stream 10 and a sub-picture specific data stream 12 derived from the data stream 10 by sub-picture extraction 14. The data stream 10 may have been generated by an encoder 16. In particular, encoder 16 has encoded picture 18 into data stream 10. Fig. 1 shows picture 18 as one picture of video 20, but it should be noted that embodiments of the present application are not limited to video data streams. Rather, the embodiments described herein can be easily transferred to a picture codec. However, the following description illustrates embodiments related to video encoding.
In encoding picture 18 into data stream 10, encoder 16 follows or uses an encoding path 22, encoding path 22 traversing picture 18 from a picture start position 24, which may be located at the upper left corner of picture 18 as shown in fig. 1, to an opposite picture corner, such as the lower right corner, for example, to continue traversing another picture, such as video 20. For example, compliance or use of encoding path 22 determines availability of spatially adjacent portions of picture 18, such as samples or parameters derived therefrom by encoder 18 for encoding a current portion of picture 18, such as syntax elements. For example, encoder 16 may use predictive coding. To this end, encoder 16 may predict the current portion, such as its sample content or syntax elements describing it, based on spatially neighboring portions of picture 18, provided these neighboring portions precede the current portion along encoding path 22. Additionally or alternatively, encoder 16 may use other spatial coding dependencies for the current portion, such as deriving entropy contexts for encoding syntax elements describing the current portion of picture 18 based on neighboring portions of picture 18. In addition, encoder 16 uses an encoding path to divide or encapsulate the data into which picture 18 is encoded. To this end, encoder 16 subdivides the encoded data along encoding path 22 into slices. Thus, each slice 26 of the data comprising a picture 18 in the data stream 10 covers the corresponding portion or is encoded into a slice 28 in which the corresponding portion, i.e. the picture 18, has been already, wherein the slices 28 of the picture 18 are sequentially traversed by the encoding path 22 without interleaving the slices 28. In other words, the encoding path 22 traverses each of the slices 28 into which the picture 18 is subdivided only once.
As also shown in fig. 1, in encoding picture 18 into data stream 10, encoder 16 also follows the subdivision of picture 18 into segments 30. Fig. 1 exemplarily shows that the picture 18 is subdivided into four segments 30, exemplarily arranged in a 2 × 2 array. However, the number of segments is not critical and may vary. However, subdividing the picture 18 into segments 30 may be such that the segments 30 are arranged in rows and columns. Subdividing the picture 18 into segments 30 may be such that the segments 30 completely seamlessly cover the picture 18, with segments within a segment row having the same height and segments within a segment column having the same width. However, the size of such rectangular segments 30 may vary, as the segment columns and/or segment rows may differ in width and height, respectively. The compliance of subdividing the picture 18 into segments 30 may be related to coding interdependencies: encoder 16 encodes pictures 18 into data stream 10 in such a way that the encoding interdependencies do not cross segment boundaries. For this reason, no part within a segment 30 of a picture 18 is encoded depending on the part of the picture 18 outside this segment. In other words, the encoder 16 encodes the segments 30 independently of each other. As regards the relationship between the slice 28 on the one hand and the segment 30 on the other hand, the slice 28 either completely covers one or more segments 30 or is only within one segment 30. For illustrative purposes only, fig. 1 shows that each segment 30 is composed of two slices 28. This in turn means that the encoding path 22 traverses the segments 30 sequentially, i.e., each segment 30 is completely traversed by the encoding path 22 before the encoding path 30 traverses the next segment 30 in encoding order.
Thus, encoder 16 encodes pictures 18 into data stream 10 in units of slices 28. Thus, each slice 26 in the data stream 10 has a corresponding picture slice 28 encoded therein. Each slice 26 comprises payload data 32, the payload data 32 encoding the content of the corresponding slice portion 28 of the picture 18 in the form of, for example, one or more prediction parameters such as a prediction mode distinguishing, for example, intra or inter coding modes, motion parameters for intra prediction blocks, intra prediction sub-modes for intra prediction blocks, subdivision information for subdividing the picture 18 into blocks, and residual data such as transform coefficients representing prediction residuals.
Before proceeding to describe slice header 34 and its components or syntax elements, respectively, of slice 26, a brief continuation will be described of how encoder 16 encodes picture 18 into data stream 10. As previously mentioned, the segments 30 are segments that are encoded independently of each other. However, to date, the description has focused only on the encoding of picture 18. However, where the picture 18 is a picture of the video 20, the encoder 16 may employ subdividing the picture into segments 30 of the picture sequence for the video 20, i.e., subdividing the pictures of the picture sequence in the same manner, i.e., such that other pictures are segmented into the same number of segments 30 and same size segments 30, with segment boundaries spatially coinciding between the pictures. This is the case depicted in fig. 1, for example, where preceding and succeeding pictures relative to picture 18 are shown to be subdivided in the same manner as picture 18, i.e. into four segments 30. In addition to the mutual independence of the encoded segments 30 within one picture, encoder 16 may encode each segment of picture 18 in a manner such that the encoding of such a respective segment 30 of picture 18 does not depend on portions of another picture, referred to as a reference picture, that are external to or spatially offset from the segment collocated with the respective segment of picture 18. In other words, the collocated segments 30 of pictures within the sequence of pictures of the video 20 may form a spatio-temporal region within the video 20 within which the encoder 16 performs encoding independent of the spatio-temporal region formed by another segment of these pictures. Therefore, the aforementioned inter prediction, i.e., motion compensated prediction, will be restricted in the following way: such that a block within one of the segments 30 of the picture 18 encoded using inter prediction will not reference, by way of motion vectors, portions of a reference picture that are located outside of the segment of the reference picture that is collocated with the segment in which the current block is located. Instead, encoder 16 will instead, for example, select intra prediction for the block.
It should even be noted that the data stream 10 comprises further information comprising higher level coding parameters, hereinafter referred to as parameter sets, in addition to the slice header 34 of the slice 26, which is described in more detail below. This set of
In addition to the details described so far, it should be noted that the encoder 16 may form the data stream 10 in such a way that slices 26 included in the data stream 10 relating to one picture, such as the picture 18, are not interleaved with other slices 26 in the data stream 10 relating to another picture of the video 20. Instead they form a continuous part of the data stream 10, the so-called
Having described the general framework for encoding of the data stream 10, the description continues with a description of the slice header 34 and its contents. In particular, the encoder 16 provides start position information for each slice header 34, the start position information indicating a
Advantageously, however, the start position information 42 indicates the
To compensate for the relative indication of the
A decoder 38 receiving the data stream 10 is able to decode the start position information 42 of the corresponding slice 26 from the data stream 10 and is able to use the start position information 42 to locate the
The above has outlined the case where the relative indication of the
The extraction process 14 is intended to form a data stream 12 with pictures 54 encoded therein, the pictures 54 being composed of only a suitable subset of the segments 30 of the pictures 18 and/or the mutual positioning of the segments 30 within the picture areas of the pictures 18 and 54, respectively, being different in comparison to the pictures 18. Fig. 1 shows the case where a picture 54 of the data stream 12 consists of only one of the four segments 30 of the picture 18 of the original data stream 10, segment B, but this is only an example and the picture 54 may alternatively consist of more than one segment 30 of the picture 18 as long as a rectangular picture area of the picture 54 is generated.
Notably, with respect to the slices 26 included in the data stream 10, the extraction process 14 involves merely discarding or omitting slices 26 relating to segments 30 of the picture 18 that are not included in the picture 54, while slices 26 relating to any segments 30 contributing to the composition of the picture 44 are taken or retained in the extracted data stream 12 without any modification, in particular without any modification of the start position information 42 and the segment information 48. In particular, since the start position information 42 indicates the
The only information present in the extracted data stream 12 that has not been discussed in relation to the original data stream 10 are those encoding parameters associated with the modified composition of the pictures 54 of the extracted data stream 12 based on the selected set of segments 30. There are several possibilities regarding this subject. According to a first alternative, the extraction process 14 may involve modification of the aforementioned parameter sets 36 such that corresponding parameter sets 36' in the extracted data stream 12 are modified to reflect a modified picture size of pictures 54 of the extracted video 56, subdivision of the pictures 54 into segments, a level of decoder capability required for decoding the extracted data stream 12 that may be lower than a level of decoder capability required for decoding the complete video 20 or pictures 18 from the data stream 10, and/or a modified decoding timestamp, encoded picture buffer retrieval time, and/or buffer size for decoding the extracted data stream 12. However, according to an alternative embodiment, the aspect of using the relative start position indication within the start position information 42 may be combined with the aspect of the present application that even the parameter sets 36' of the extracted data stream 12 may remain unchanged, i.e. by providing two information items to the parameter sets 36 of the original data stream 10: according to this alternative, parameter set 36 would indicate all encoding parameters for decoding the entire picture 18 from data stream 10, and at the same time all encoding parameters for decoding picture 54 from data stream 10 or 12. The encoding parameters indicated by parameter set 36 for the extracted data stream 12 and its pictures 54 may be indicated at least in part in parameter set 36 in a manner that is relative to or different from the encoding parameters indicated by parameter set 36 for decoding pictures 18 from data stream 10. For example, as outlined in more detail below, parameter set 36 may indicate a subdivision of picture 18 into segments 30 such that the size of segments 30 is clear from this portion of parameter set 36. The encoding parameters of the parameter set 36 relating to the picture 54 may rely on this knowledge and may simply indicate which segments contribute to the picture 54 and which mutual arrangement of these contributing segments applies within the picture 54. The aforementioned base address indication even for the start position 44 within the picture 18 may be repeated in the
More details on how decoder 38/52 operates to decode an inbound data stream (which may be data stream 10 or data stream 12) are set forth at the end of the description with respect to fig. 11.
In the following description, specific examples for implementing the above-described embodiments are described. In doing so, assuming that the data stream includes a start position
Further details that will become clear from the embodiments explained later are the fact that the slice position indication 42 can actually be signaled by a combination of a flag and a conditionally signaled slice address: in the case of the
And as a further note before the following more detailed explanation of the starting embodiment, it is noted that there are different possibilities for indicating the starting
Another possibility, which will become clear from the following description, is as follows: so far, the segments 30 have been described as possible spatial cross-sections of spatio-temporal regions, where the coding interdependencies do not cross the boundaries of these spatio-temporal regions. For example, the segments 30 may be defined as a set of one or more tiles, the picture 18 may be subdivided into tiles, and the tiles are encoded independently of each other with respect to the encoding of one picture 18. Thus, in the case where one segment 30 is composed of one block, the segment 30 may be a block, and in the variant a explained later, this is the case, while the variant B assumes that the segment 30 may be composed of one or more blocks. In addition, according to the explanations that follow, the aforementioned coding blocks 64 are CTUs (coding tree units), which means that these blocks 64 are tree root blocks that are further hierarchically subdivided by multi-tree subdivision into coding blocks, prediction blocks and/or transform blocks, the encoder 16 selects the coding mode to be used in units of coding blocks, prediction blocks and/or transform blocks, i.e. inter-or intra-coding of the corresponding coding block, sets the prediction parameters of the corresponding selected coding mode in units of coding blocks, prediction blocks and/or transform blocks, i.e. for each prediction block that may be a departure node of a coding block, and the transformation of the prediction residual takes place in units of coding blocks, prediction blocks and/or transform blocks, i.e. in units of transform blocks that may also be a departure block of a coding block. The subdivision of the CTU 64 into coding units, prediction blocks, and transform blocks may be signaled as part of the corresponding slice payload 32.
Hence, in the following, the implementation of the embodiment described with respect to fig. 1 is illustrated as a possible modification of the HEVC framework. As described in the introductory part of the present specification, in HEVC DAM3 MCTS extraction, an adjustment of the slice segment address for each slice header is necessary, which in turn may affect even the byte alignment of each slice header due to the variable length coding of the slice address. In particular, if HEVC DAM3 MCTS is used, the slice addresses of the slices employed in the MCTS extracted data stream have to be modified due to their absolute indication option, i.e. they have to be modified to refer to the new picture start position of the picture 54, and due to the variable length coding this may result in different lengths of the slice addresses and thus different byte alignments. A modification of the HEVC framework explained later will overcome this. In particular, by signaling portions of slice addresses to imply or express association of slices with tiles/segments, relatively simple MCTS sub-stream extraction results.
In the following, a first variant, variant a, of a modification of the HEVC framework is described. As outlined above with respect to fig. 1, according to this variant, the first CTU with respect to the respective tile or combination of tiles (i.e., the current segment 30) signals the slice address, as opposed to the first CTU with respect to the picture. "first" means the leftmost and topmost CTUs when HEVC coding order is used.
The difference between the two references (i.e., the first CTU of the current segment 30 or the first CTU of a picture) may be derived by the decoder, i.e., decoder 38 or 52, by combining the picture and tile instruction information from parameter sets 36 and 36', respectively, the tile instruction information being the association of the slice 28/26 with this segment 30 (which may be a tile or a set of tiles), where the association may be explicitly transmitted in the data stream as outlined below.
An array having information on the segment size in the CTB and the slice segment address offset of each segment 30 may be determined at the decoder side. In the case of the
Fig. 3 provides a top view of an embodiment derived using HEVC naming. As indicated, parameter sets 36 are distributed over different ranges of picture parameter sets 70, sequence parameter sets 72, and video parameter sets 74. Fig. 3 shows only a portion of the data stream 10, namely two VCL NAL units 76 and 78, each of which includes a NALU header 80 indicating that it includes segment information, followed by a corresponding slice 26 consisting of the slice header 34 and the slice payload 32. The slice header 34 references the corresponding picture parameter set 70 indicated by arrow 80, and the picture parameter set 70 in turn points to the corresponding sequence parameter set indicated by 82, and the sequence parameter set 72 in turn points to the corresponding active video parameter set indicated by 84. For example, parameter set 36 suggests that either the picture parameter set 70 or the sequence parameter set 72 include a slice syntax 86, i.e., a syntax that defines the subdivision of pictures 14 into slices 88, which, according to one alternative, form the segment 30 discussed above in fig. 1. Thus, based on the block syntax 86, it is possible for the decoder to calculate the base slice segment address indicated by arrow 90, for example. The sum 92 of the offset slice segment address 96 and the base slice segment address 94, which are communicated as part of the start position information 42 and the slice header 34, yields a corresponding slice segment address 98 for the corresponding slice 26, i.e., the slice segment address as measured absolutely relative to the picture start position 24. Fig. 3 shows the two slices 26 depicted in fig. 3 as belonging to one tile of tile 88, tile number 2, which is in fact derivable for the decoder based on the segment information 48 also comprised by the slice header 34, as will become clear from the following description.
In particular, according to the embodiment illustrated in fig. 4, a block structure or block syntax 86 may be included in the picture parameter set 70. Note that alternative embodiments are possible where, for example, the partition syntax 86 is present in the sequence parameter set 72 instead of the picture parameter set.
In particular, as depicted in fig. 4, portion 100 of parameter set 70 indicates a subdivision of picture 18 into segments 30 (here, tiles). The number of segment columns is indicated at 102, the number of segment rows is indicated at 104, and the marker 106 optionally provides an opportunity to signal in the data stream that the width of the segment columns and the height of the segment rows are set evenly. If not, or if the flag 106 is not set, then the width of the segment column is indicated individually at 108 and the height of the segment row is indicated individually at 110.
In addition, an explicit flag of a segment is provided if the flag slice _ segment _ base _ addr _ per _ segment _ enable _ flag 112 is set, i.e., by explicitly signaling the segment index value tile _ id _ in _ pps [ i ] of each segment i 30 of the picture 18 at 114. The segment index is transmitted in the data stream, i.e. in the parameter set 70, by using a default order in the regular array of segments 30, such as a row-by-row fashion as indicated by the dashed arrow 116 in fig. 1.
The semantics of the syntax elements in fig. 4 are summarized in detail below:
slice _ segment _ base _ addr _ per _ tile _ enable _ flag equal to 0 specifies: the sliced variable CtbAddrInRs is uniquely derived from slice _ segment _ address. slice _ segment _ base _ addr _ per _ tile _ enable _ flag equal to 1 specifies: the derivation of CtbAddrInRs is based on slice _ segment _ address and tile dependent base address. When slice _ segment _ base _ addr _ per _ tile _ enable _ flag is not present, it is inferred to be equal to 0.
tile _ id _ in _ pps [ i ] specifies the id of the tile in the bitstream order. the value of tile _ id _ in _ pps [ i ] should be in the range of 0 to 255, and for i not equal to j, tile _ id _ in _ pps [ i ] should not have the same value as tile _ id _ in _ pps [ j ]. When tile _ id _ in _ pps is not present, it is inferred to be equal to 0.
This may be a constraint on bitstream conformance: when slice _ segment _ base _ addr _ per _ tile _ enable _ flag is equal to 1, the access unit delimiter NAL unit is present in the bitstream and tiles _ fixed _ structure _ flag is equal to 1.
The pseudo code below shows how a decoder, such as decoder 38 or 52, calculates a particular size of segment 30 and a base address of segment 30, for example, based on information available in the data stream.
The encoding tree block grid and tile scan conversion process is described below.
The following (pseudocode 1) yields a list colWidth [ i ] for i, which ranges from 0 to num _ tile _ columns _ minus1 (inclusive), specifying the width of the ith tile column in units of Code Tree Blocks (CTBs):
colWidth [ i ] is the width of the ith segment column in
num _ tile _ columns _ minus1 is the number of segments minus 1.
The following (pseudocode 2) yields a list rowHeight [ j ] for j, ranging from 0 to num _ tile _ rows _ minus1 (inclusive), specifying the height of the jth tile row in CTB:
rowHeight [ i ] is the height of the ith segment row in
num _ tile _ rows _ minus1 is the number of segment rows minus 1.
The following (pseudocode 3) yields a list TileId [ ctbsadrts ] for ctbsadrts, which ranges from 0 to PicSizeInCtbsY-1 (inclusive), the TileId [ ctbsadrts ] specifying the translation from CTB address in a tile scan to tile ID:
tileidxtileldmap [ ] is interpreted as tileIdx that maps the N segment index values included in the vector tile _ id _ in _ pps [ ] to the segment.
tile _ id _ in _ pps [ ] is a vector that includes
tileIdx indexes segment 30 in raster scan segment order 116.
TileId [ i ] is interpreted as mapping the CTB address i, i.e. the address of the block 64, measured in coding order or along the coding path 22 to the tileIdx of the segment 39 in which this CTB is located.
Ctbsaddr rstots [ i ] is a function that maps the rank i of a block 64 (which is the i-th block 64 of picture 18 in picture raster scan order leading through blocks of picture 18 line by line) to its address measured in coding order 22.
colBd [ i ] is a
Note that all parameters used so far in the pseudo code, although they have been described as being used to determine a base address in the case of full picture decoding (i.e. with respect to picture 18), may also be used to determine a base address of the start position of a segment in sub-picture 54. Herein, the parameters N, colWidth [ i ], num _ tile _ columns _ minus1, rowHeight [ i ], num _ tile _ rows _ minus1, tileidxtileldmap [ ], tile _ id _ in _ pps [ ], ctbddrrstats [ i ], colBd [ i ] are respectively specific to this sub-picture 54 and its composition from the segment 30 and its subdivision into corresponding blocks 64. Similarly, all references to the encoding path 22 and raster segment order 116 are applied to the corresponding sub-picture. the tilledxtiledmap [ ] ensures that the association between the segment index value tile _ id _ in _ sh, on the one hand, assuming any of tile _ id _ in _ pps [ ] and the segments of the sub-picture 54 and their indices in the raster scan segment order 116, i.e., the tileIdx, on the other hand, is maintained. Incidentally, it should be noted that the syntax element suffix "_ PPS" has been chosen here to reflect the example of carrying the segment index value to segment association in a PPS, but as already outlined above, this is merely an example and the signaling may also be implemented elsewhere, such as in SPS or VPS. Consistent with the latter statement, the suffix "_ ps" is alternatively used, and the corresponding substitution naturally also applies to the present embodiment. Similarly, the following example will reveal that 1:1 correspondence between tiles and segments can be abandoned. This has also been outlined with reference to fig. 1. Accordingly, the corresponding subsequent embodiments are replaced with the name component "area" or "area _" or "_ area", which will become clear from the following description. In addition, as also mentioned in the description of fig. 1, in an alternative embodiment, an explicit assignment of labels or segment index values to segments may be preserved, wherein a mapping between segment index values in segment information 48 and segments in sub-picture 54 may be implicitly derived, such as by alternatively using, for example, an index field tileIdx, in which case transmission of such an association in parameter set 36 of picture 18 may be stopped. However, the explicit assignment of segment index values to segments 30 of pictures 18 mitigates the mixing of tiles of two separate data streams.
The following (pseudocode 4) yields a list tilesizelnctbsy [ k ], for k ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) — inclusive), the tilesizelnctbsy [ k ] specifying the size of the kth tile within a tile in CTB units:
for(j＝0,k＝0；j<＝num_tile_rows_minus1；j++)
for(i＝0；i<＝num_tile_columns_minus1；i++,k++)
TileSizeInCtbsY[k]＝colWidth[i]*rowHeight[j]
TilesizeinCtbsY [ i ] is a vector that includes
The following (pseudo code 5) yields a list tileselgbaseaddr [ k ] for k ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) — inclusive, the tileselgbaseaddr [ k ] specifying, in CTB units, the slice address offset in the tile scan of the first slice of the kth tile within the picture in bitstream order:
tileslicicesegbaseaddr [ i ] is a
Herein, "tile scan" means a scan 22 as depicted in fig. 1. Tileslicicesegbaseaddr represents the aforementioned base address of the start location 44 of the segment, which, according to this example, is equal to the tile. The calculations indicated above may be used by the decoders 38 and 52, respectively, to calculate the base address. If sub-picture decoding is applied, the number of segments and their arrangement is modified.
Fig. 6 shows how the slice segment head, i.e. the slice head 34 described above, looks. In particular, herein, the slice header 34 includes a flag 120, i.e., a first _ slice _ segment _ in _ area _ flag, which forms the start position information 42 together with a syntax element slice _ segment _ address 122. Only if the flag 120 is not set, slice _ segment _ address exists and indicates an offset of the
In particular, the semantics may be as follows:
first _ slice _ segment _ in _ area _ flag equal to 1 specifies that a slice segment is the first slice of a picture in decoding order when slice _ segment _ base _ addr _ per _ tile _ enable _ flag is equal to 0. Otherwise, when slice _ segment _ base _ addr _ per _ tile _ enable _ flag is equal to 1, first _ slice _ segment _ in _ area _ flag equal to 1 specifies that the slice is the first slice of the tile of the picture in decoding order. first _ slice _ segment _ in _ pic _ flag equal to 0 specifies that the slice is not the first slice of a picture in decoding order or the first slice of a tile in decoding order, respectively. Thus, slice _ segment _ base _ addr _ per _ tile _ enable _ flag serves as an example of the
Note 1-this syntax element may be used to detect boundaries between coded pictures that are consecutive in decoding order. However, when IDR pictures are consecutive in decoding order and have the same NAL unit type, the loss of the first slice of the IDR picture can lead to problems of detection of boundaries between coded pictures. This may occur, for example, in the transmission of full intra-coded video in error-prone environments. This problem can be mitigated by alternately using two different IDR NAL unit types (IDR _ W _ RADL and IDR _ N _ LP) for any two consecutive IDR pictures. The use of a temporal sub-layer zero index SEI message may also be helpful, since the SEI message comprises the syntax element IRAP _ pic _ id, whose value is different for consecutive IRAP pictures in decoding order. Some system environments have other provisions that may also facilitate picture boundary detection, such as using presentation time stamps in Rec.ITU-T H.222.0| ISO/IEC13818-1 systems, access units framed by ISO/IEC14496-12 ISO base media file formats, or flag bits in IETF RFC3550 real-time transport protocol headers.
dependency _ slice _ segment _ flag equal to 1 specifies that the value of each slice segment header syntax element that is not present is inferred to be equal to the value of the corresponding slice segment header syntax element in the slice header. When not present, the value of dependent _ slice _ segment _ flag is inferred to be equal to 0.
The variable sliceaaddrrs is given as follows:
-setting sliceaddr equal to ctbsaddr if dependent slice segment flag is equal to 0.
-otherwise, the sliceaddrs is set equal to sliceadadrrs of a previous slice of a coding tree block comprising a coding tree block address ctbsaddtstors [ ctbsadrrstots [ ctbsaddlnrs ] -1], wherein the variable ctbsaddlnrs is specified in the semantics of slice _ segment _ address.
tile _ id _ in _ sh specifies the index of the tile to which the slice belongs. The value of area _ id _ in _ sh should be in the range of 0 to 255. When there is no area _ id _ in _ sh, it is inferred to be equal to 0. No more than one tile within a tile will have the same value of tile _ id _ in _ sh.
slice _ segment _ address specifies the coding tree block raster scan of the picture (when slice _ segment _ base _ addr _ per _ tile _ enable _ flag is equal to 1) and the address of the first coding tree block in a slice in the slice scan of the picture, otherwise:
the variable maxNumCtbY is defined as (slice _ segment _ base _ addr _ per _ tile _ enable _ flag; herein, tileidxtiledmap [ tile _ id _ in _ sh ] maps tile _ id _ in _ sh of the currently decoded/encoded slice included by its segment information 48 onto the correct tile idx, i.e., the correct segment in the reference picture 18 or 54, and tilesizelnctbsy generates the maximum value of the relative slice address for this segment, i.e., maxNumCtbsY-1, to be represented/representable by the slice _ segment _ address of its start position information 42. The decoder and encoder may use the pseudo-codes 1-4 to calculate this information.
The length of the slice _ segment _ address syntax element is ceil (Log2(maxNumCtbsY)) bits. The value of slice _ segment _ address should be in the range of 0 to maxNumCtbsY-1, inclusive. When slice _ segment _ base _ address _ per _ tile _ enable _ flag is equal to 0, the value of slice _ segment _ address should not be equal to the value of slice _ segment _ address of any other coded slice NAL unit of the same coded picture. When slice _ segment _ base _ address _ per _ tile _ enable _ flag is equal to 1, the value of slice _ segment _ address should not be equal to the value of slice _ segment _ address of any other coded slice NAL unit belonging to the same tile within the same coded picture. When slice _ segment _ address does not exist, it is inferred to be equal to 0.
A variable ctbddrnrs specifying the coding tree block address in the coding tree block raster scan of the picture is set equal to ctbddrtstors [ slice _ segment _ address + (slice _ segment _ base _ address _ per _ tile _ enable _ flag. A variable CtbAddrInTs specifying the code tree block address in a tile scan is set equal to CtbAddrRsToTs [ CtbAddrInRs ]. A variable CuQpDeltaVal specifying the difference between the luma quantization parameter of the coding unit including cu _ qp _ delta _ abs and its prediction is set equal to 0. Variables CuQpOffsetCb and CuQpOffsetCr specifying values to be used when determining the respective values of the Qp 'Cb and Qp' Cr quantization parameters of the coding unit including cu _ chroma _ Qp _ offset _ flag are both set equal to 0. Thus, in the case of the
Regarding the order of VCL NAL units and their association with coded pictures, the following can be said.
In particular, the order of VCL NAL units and their association with coded pictures is specified below.
Each VCL NAL unit is part of a coded picture.
The order of VCL NAL units in a coded picture is constrained as follows:
first slice segment in pic flag of the first VCL NAL unit of the coded picture should be equal to 1.
Let sliceselgaddra and slicesegadaddrb be the values of ctbsaddtstors [ slice _ segment _ base _ address _ per _ tile _ enable _ flag ] tileselgbaseaddr [ tiledidxtend map [ tile _ id _ in _ sh ] ] of any two coded slice NAL units a and B within the same coded picture. The coded slice NAL unit a should precede the coded slice NAL unit B when any of the following conditions is true:
-TileId [ CtbAddrRsToTs [ SliceSegAddrA ] ] is less than TileId [ CtbAddrRsToTs [ SliceSegAddrB ] ].
-TileId [ CtbAddrRsToTs [ SliceSegAddrA ] ] is equal to TileId [ CtbAddrRsToTs [ SliceSegAddrB ] ] and CtbAddrRsToTs [ SliceSegAddrA ] is less than CtbAddrRsToTs [ SliceSegAddrB ].
Decoders 38 and 52 may monitor these conditions to determine when a
For example, the variant a described so far may be modified in different respects to allow easier implementation and processing.
For example, in variant a, the segment 30 need not be exactly a tile, as already described above with respect to fig. 1. To this end, the slice segment base address calculation need not occur exclusively at tile granularity, but rather multiple tiles may be joined into regions, then form a segment 30 according to the description set forth above, and then for a segment 30, the slice segment base address may be defined collectively, i.e., the slice address in the slice header given relative to the first CTU in the bitstream order of a given region belonging to multiple tiles forming the segment 30.
Additionally, additionally or alternatively, according to variant B described below, it becomes clear that instead of burdening the client side or decoder with the calculation of the slice segment base address for each segment 30, explicit signaling of the slice segment base address may be provided in the data stream.
Even additionally or alternatively, by allowing the partition syntax to be carried in the SPS or PPS, the burden of repeating tile structure signaling within each PPS of the video bitstream while ensuring, for example, that tile fixed structure equals 1 when slice segment base address per tile enable flag equals 1, where a PPS instance of the partition syntax can override the partition syntax settings carried within the SPS.
The following figures are shown by highlighting the variations relative to the variant a described above. For example, fig. 7 shows a portion of the block syntax that may be used in a PPS or SPS, where when the block syntax of fig. 7 is transmitted in the PPS, it overrules the tile syntax provided by the corresponding SPS, i.e., it has a higher precedent. Fig. 7 shows the possibility of the parameter set 36 of fig. 1 carrying the base address of the segment start location 44 via the base address data field 130.
The semantics may be as follows.
slice _ segment _ base _ address _ per _ area _ enable _ flag equal to 0 specifies the variable ctbddrinrs from which the slice is uniquely derived. slice _ segment _ base _ addr _ per _ area _ enable _ flag equal to 1 specifies that derivation of ctcoadprinrs is based on slice _ segment _ address and tile dependent offset. When slice _ segment _ base _ addr _ per _ area _ enable _ flag is not present, it is inferred to be equal to 0.
area _ ID _ in _ ps [ i ] specifies the ID of the tile in the bitstream order. The value of area _ id _ in _ ps [ i ] should be in the range of 0 to 255, and for i not equal to j, area _ id _ in _ ps [ i ] should not have the same value as area _ id _ in _ ps [ j ]. When area _ id _ in _ sh does not exist, it is inferred to be equal to 0.
slice _ segment _ base _ address [ i ] specifies the substrate segment addresses in the tile scan order for all segments belonging to a region having the same value of area _ id _ in _ ps [ i ]. When not present, the value of slice _ segment _ base _ address [ i ] is inferred to be equal to 0.
A requirement for bitstream conformance is that when area _ id _ enable _ flag is equal to 1, an access unit delimiter NAL unit is present in the bitstream.
The requirement for bitstream conformance is that the value of slice segment base address i-1 is not equal for any two values of i, where area _ id _ in _ ps [ i ] has equal value.
Fig. 8 and 9 provide examples of SPS and PPS syntax, with separate enable flags for block syntax signaling. Herein, the following semantics are used.
tiles _ enabled _ in _ SPS _ flag equal to 1 specifies the default partition syntax that SPS carries the default structure describing the CVS.
tiles enabled in PPS flag equal to 1 specifies that the PPS carries the partition syntax describing the actual partition structure of the picture of the reference PPS of the CVS and overwrites the partition syntax within the referenced SPS.
The coding tree block trellis and block scan conversion process will be as follows.
The following (pseudo code 1') yields a list colWidth [ i ] for i, which ranges from 0 to num _ tile _ columns _ minus1 (inclusive), specifying the width of the ith tile column in Code Tree Blocks (CTB):
the following (pseudo-code 2') results in a list rowHeight [ j ] for j, ranging from 0 to num _ tile _ rows _ minus1 (inclusive), the rowHeight [ j ] specifying the height of the jth tile row in CTB units:
the following (pseudocode 3') yields a list TileId [ ctbsadrts ] for ctbsadrts, which ranges from 0 to PicSizeInCtbsY-1 (inclusive), the TileId [ ctbsadrts ] specifying the translation from CTB address in a tile scan to tile ID:
the variable retains its meaning as described above, however, N represents the number of tiles that may be larger than the number M of segments 30 due to the collection of the tiles into the segments 30.
The following (pseudocode 4') yields a list tilesizelnctbsy [ k ] for k, ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) -1) (inclusive), tilesizelnctbsy [ k ] specifying the size of the kth tile within a tile in CTB:
area _ id _ in _ ps [ ] is a
AreaSizeInCtbsY [ ] is interpreted as a vector of entries that become each element of the field that includes a possible value for a segment index in segment information 48, where the value indicates, for each segment index that occurs in picture 18 or sub-picture 54, the number of blocks 64 covered thereby, regardless of the current decoding objective/option.
The mapping from AreaId to AreaIdx is derived as follows:
NumTilesInArea [ ] is interpreted as a vector of entries that become each element of the field that includes a possible value for a segment index in segment information 48, where for each segment index value occurring in a picture 18 or sub-picture 54 indicates the number of tiles covered by the corresponding segment with that segment index value, regardless of the current decoding goal/option.
Areaidtoracidx [ ] is interpreted as a vector of entries that become each element of a field that includes possible values for the segment index in segment information 48, where for each segment index value that appears in picture 18 or sub-picture 54 indicates its rank or index when sequentially assigned or measured along a raster scan segment that is directed segment-by-segment row from top left to bottom right, regardless of the current decoding goal/option.
The list, areaslicesesgbaseaddr [ k ], for k, ranging from 0 to (NumAreas-1), inclusive, is derived as follows (pseudo code 5'), the slice address offset of the first slice of the kth region within the picture in bitstream order being specified in CTB by areaslicesesgbaseaddr [ k ], where:
the areasliceseggbaseaddr [ i ] is interpreted to become a
Thus, it is noted that the base address data field 130 defined by the syntax element slice _ segment _ base _ address defines for each segment 30 the base address of the start position 44 of the respective segment addressing the start position 44 relative to the picture start position 24, although in case an explicit transmission is applied to the sub-picture decoding option(s) an explicit transmission of the base address of the first segment 30 of a picture 18 or sub-picture 54 in raster scan segment order is also omitted, since the base address is also zero anyway. By way of the base address data field, this definition of a base address of zero is a conventional result of explicit base address transfers starting from the second segment in raster scan segment order. Of course, alternative embodiments may be evaluated where instead an explicit base address transfer is used for all segments.
In case no explicit base address signaling is used but an implicit derivation of the base address is used, the decoder or encoder may perform the derivation of the base address as follows. The following calculations are used by the encoder to calculate the base address even in the case of explicit base address signaling.
The following (pseudo code 5 ") yields a list tileselgbaseaddr [ k ] for k ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) -1), inclusive, tileselgbaseaddr [ k ] specifying the slice segment address offset of the first slice segment of the kth tile within the bitstream order picture in CTB units, and a list artarslicesegbaseaddr [ k ] for k ranging from 0 to (numares-1), inclusive, specifying the slice segment address offset of the first slice segment of the kth region within the bitstream order picture in CTB units:
in addition, fig. 10 shows how slice headers look according to variant B, where possible semantics are shown below.
first _ slice _ segment _ in _ area _ flag equal to 1 specifies that a slice segment is the first slice segment of a picture in decoding order when tiles _ enabled _ flag is equal to 0. Otherwise, when tiles _ enabled _ flag is equal to 1, first _ slice _ segment _ in _ pic _ flag is equal to 1 specifying the slice as the first slice of the picture in decoding order. first _ slice _ segment _ in _ pic _ flag equal to 0 specifies that the slice is not the first slice of a picture in decoding order.
Note 1-this syntax element may be used to detect boundaries between consecutive coded pictures in decoding order. However, when IDR pictures are consecutive in decoding order and have the same NAL unit type, the loss of the first slice of the IDR picture may cause problems with the detection of boundaries between coded pictures. This may occur, for example, in the transmission of full intra-coded video in error-prone environments. This problem can be alleviated by alternately using two different IDR NAL unit types (IDR _ W _ RADL and IDR _ N _ LP) for any two consecutive IDR pictures. The use of a temporal sub-layer zero index SEI message may also be helpful, since the SEI message comprises the syntax element IRAP _ pic _ id, whose value is different for consecutive IRAP pictures in decoding order. Some system environments have other provisions that may also facilitate picture boundary detection, such as using presentation time stamps in Rec.ITU-TH.222.0| ISO/IEC13818-1 systems, access units framed by ISO/IEC14496-12 ISO base media file formats, or flag bits in IETF RFC3550 real-time transport protocol headers.
dependency _ slice _ segment _ flag equal to 1 specifies that the value of each slice segment header syntax element that is not present is inferred to be equal to the value of the corresponding slice segment header syntax element in the slice header. When not present, the value of dependent _ slice _ segment _ flag is inferred to be equal to 0.
The variable sliceaaddrrs is given as follows:
-setting sliceaddr equal to ctbsaddr if dependent slice segment flag is equal to 0.
-otherwise, the sliceaddrs is set equal to sliceadadrrs of a previous slice of a coding tree block comprising a coding tree block address ctbsaddtstors [ ctbsadrrstots [ ctbsaddlnrs ] -1], wherein the variable ctbsaddlnrs is specified in the semantics of slice _ segment _ address.
area _ id _ in _ sh specifies the index of the tile to which the slice belongs. The value of area _ id _ in _ sh should be in the range of 0 to 255. When there is no area _ id _ in _ sh, it is inferred to be equal to 0.
slice _ segment _ address specifies the address of the first coding tree block in a slice in the coding tree block raster scan of the picture (when slice _ segment _ base _ address _ per _ tile _ enable _ flag is equal to 1) and in the slice scan of the picture, otherwise:
the variable maxNumCtbY is defined as (slice _ segment _ addr _ offset _ per _ tile _ enable _ flag.
The length of the slice _ segment _ address syntax element is ceil (Log2(maxNumCtbsY)) bits. The value of slice _ segment _ address should be in the range of 0 to maxNumCtbsY-1, inclusive. When slice _ segment _ address _ offset _ per _ tile _ enable _ flag is equal to 0, the value of slice _ segment _ address should not be equal to the value of slice _ segment _ address of any other coded slice NAL unit of the same coded picture. When slice _ segment _ address _ offset _ per _ tile _ enable _ flag is equal to 1, the value of slice _ segment _ address should not be equal to the value of slice _ segment _ address of any other coded slice NAL unit of the same region within the same coded picture. When slice _ segment _ address does not exist, it is inferred to be equal to 0.
A variable ctbdradinrs specifying the coding tree block address in the coding tree block raster scan of the picture is set equal to ctbddrtstors [ slice _ segment _ address + (slice _ segment _ address _ offset _ per _ tile _ enable _ flag. The variable CtbAddrInTs specifying the coding tree block address in a tile scan is set equal to CtbAddrRsToTs [ CtbAddrInRs ]. A variable CuQpDeltaVal specifying the difference between the luma quantization parameter of the coding unit including cu _ qp _ delta _ abs and its prediction is set equal to 0. Variables CuQpOffsetCb and CuQpOffsetCr, which specify values to be used when determining the respective values of the Qp 'Cb and Qp' Cr quantization parameters of the coding unit including cu _ chroma _ Qp _ offset _ flag, are both set equal to 0.
As regards the order of VCL NAL units and their association with coded pictures and
each VCL NAL unit is part of a coded picture.
The order of VCL NAL units in a coded picture is constrained as follows:
first slice segment in pic flag of the first VCL NAL unit of the coded picture should be equal to 1.
Let slicesegadadadra and slicesegadaddrb be the values of ctbsadrtstors [ slice _ segment _ address + ctbsadrsttots [ (slice _ segment _ base _ address _ per _ tile _ enable _ flag. The coded slice NAL unit a should precede the coded slice NAL unit B when either of the following conditions is true:
-TileId [ CtbAddrRsToTs [ SliceSegAddrA ] ] is less than TileId [ CtbAddrRsToTs [ SliceSegAddrB ] ].
-TileId [ CtbAddrRsToTs [ SliceSegAddrA ] ] is equal to TileId [ CtbAddrRsToTs [ SliceSegAddrB ] ] and CtbAddrRsToTs [ SliceSegAddrA ] is less than CtbAddrRsToTs [ SliceSegAddrB ].
The description of the possible HEVC codec extensions for efficient sub-picture extraction presented above is preliminarily extended with respect to the aspect already announced with respect to fig. 1, according to which the sub-picture extraction process is not only mitigated due to the advantageous relative indication of the start position of the slice with respect to the start position of the current segment, but additionally by providing parameter sets in the data stream that provide not only information about the decoding of the entire picture/video, but also information about the decoding of sub-picture decoding options, or alternatively information about several sub-picture decoding options, such that the extraction process may actually be limited only to actions regarding the omission or discarding of slices that are not needed according to the sub-picture decoding option of interest. Even in other words, according to such an extension of the HEVC modification described above, the interest in decoding only sub-pictures, i.e. the selection of sub-picture decoding options, is handled like omitting layers of a multi-layer data stream, while selecting all corresponding layers corresponding to the decoding of a full picture region of the original data stream. Later, it will be noted that the latter aspect may also be used independently of the relative slice starting point indication discussed above.
Thus, the extension now described allows for an easier extraction process without parameter set rewriting. As in the extraction process of layered bitstreams, removing NAL units carrying slice data and parameter sets is sufficient to create a conforming bitstream, and a decoder receiving the bitstream can easily determine the actual operation points represented in the extracted bitstream. In case tiles are used as segments, the operation points in this context exist in the space utilized by at least the segment grid or a subset of the tile grid, and optionally in other scalability dimensions such as spatial resolution, temporal resolution, SNR scalability, etc.
According to the current embodiment, a parameter set such as VPS or SPS defines several operating points in a tiling environment, hereinafter referred to as an Output Tile Set (OTS), for which several characteristics are described. A specific embodiment is depicted in fig. 11, which herein exemplarily assumes that parameters related to OTS are included in the SPS.
As shown in fig. 11, the parameter set (here exemplarily SPS) indicates how to decode the complete picture 18 (compared to fig. 1), the
When using the embodiment of fig. 11, the decoding process occurring in the decoder 38 or 52 indicates which of the qualified OTS is selected for decoding using an additional input parameter, which may be referred to as a target output tile set, where this input parameter represents the signal 58 discussed above with respect to fig. 1.
The parameter, TargetOutputTileSet, may be determined in various ways:
an indication of external means, such as signaling through the system layer in file format, transport stream, SDP, or others.
The presence of SEI messages in the bitstream indicates which tiles or OTS are present in the bitstream, i.e. a description of the previous extraction process performed on the bitstream. Thus, the extraction information may be included in a data stream indicating one or more decodable OTS.
Parse the value of the syntax element tile _ id _ sh in the slice header of the NAL unit within the access unit to determine which of the defined OTS can be decoded using the content of the bitstream.
Priorities may be assigned to the defined OTS to help determine the OTS to decode, for example by:
explicitly signaling the assignment of priority to OTS in the num _ output _ tile _ set loop or SEI message.
Each order in the num _ output _ tile _ set loop.
SEI messages of present/removed tiles in the bitstream, with indication of all included OTS.
Thus, for each decoding option signaled by 148, and for each segment set defined by decoding
The tileselcesegsegbaseaddr derivation (highlighted in cyan) needs to be adjusted relative to variant a as follows:
the list tileId [ ctbAddrTs ] for ctbAddrTs, which ranges from 0 to PicSizeInCtbsY-1, inclusive, specifies the translation from CTB address to tile ID in a tile scan is derived as follows:
the list tilesezelnctbsy [ k ], k ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) -1), inclusive, for k, specifying the size of the kth tile within a tile in CTB, is given by:
for(j＝0,k＝0；j<＝num_tile_rows_minus1；j++)
for(i＝0；i<＝num_tile_columns_minus1；i++,k++)
TileSizeInCtbsY[k]＝colWidth[i]*rowHeight[j]
for the ith target OTS, a list of tileslicitsegbaseaddr [ k ], ranging from 0 to ((num _ tile _ rows _ minus1+1) (num _ tile _ columns _ minus1+1) -1), inclusive, for k, specifying the slice address offset in a tile scan of the first slice of the kth tile within the sequential bitstream tile in CTB, is derived as follows:
wherein TileIdxInTargetOTS is true for all blocks belonging to TargetOutputTiLESet, otherwise it is false. Note that with the above, the tile arrangement within the decoded picture of TargetOutputTileSet remains similar to the tile arrangement in the original bitstream. Only tiles within the TargetOutputTileSet are considered for grid-to-tile scan conversion, and vice versa.
As mentioned above, for example, the extensions just described with respect to fig. 11 may be used in a framework where the aforementioned start position information 42 of the slice is not present, the extensions involving the parameter set 36 and its signaling of picture size and corresponding decoder capability level not only for the original picture 18 but also for the sub-picture 54. For example, conventionally, a slice may be restricted to not start within a segment 30, but only at the segment start position. Then, for example, the segment information 48 may be sufficient. Other examples are also possible. In either case, the decoder 38 or 52 receiving the stream 10 or 12 (with or without the start position information 42) uses the information in the parameter set 36 equal to 36' as follows. In particular, it decodes the parameter set 36 from the inbound data stream, which may be 10 or 12, and derives a decoding option indication 58, i.e. whether picture decoding or sub-picture decoding is to be performed on the inbound data stream. Picture decoding represents the target of decoding a picture 18 from the data stream 10, i.e. synthesizing the picture 10 from all segments 30. Therefore, all slices 26 are required in the data stream 10 to achieve this. The sub-picture decoding purpose is the object of decoding a picture 54 from the data stream 12. "picture 54" indicates that more than one sub-picture decoding option may be signaled in parameter set 36, each sub-picture decoding option differing in picture size and shape and in decoder capability level. For example, in the example of fig. 1, different sub-picture decoding options may result from forming a sub-picture 54 from two segments 30 side-by-side, a sub-picture 54 from two segments 30 one above the other, or a sub-picture 54 from only one segment 30, where these examples may be extended by forming a sub-picture from three segments 30, for example.
If picture decoding is to be performed on the data stream, the decoder derives from the parameter set 36 the size of the picture 18, such as via 142 and 144 for
However, in case the decoding option indication 58 suggests that sub-picture decoding is to be performed on the inbound data stream, the decoder derives from the parameter set 36 a further picture size of the sub-picture 54 and an indication of the decoder capability level required to decode the sub-picture 54 from the data stream, such as for i ≠ 0, via 142, 144 and 146, derives from the parameter set 36 information about subdividing the sub-picture 54 into a second set of segments, which are encoded into the data stream without encoding interdependencies and which are a subset of the set of segments of the picture 18, and decodes the sub-picture 54 from the data stream in units of those slices 26 to which the sub-picture 54 is encoded. Even here, the decoder uses or follows the encoding path 22'. This time it traverses the other target purpose, namely sub-picture 54. As encoding path 22 traverses picture 18, encoding path 22' traverses sub-picture 54 sequentially, segment by segment, and each of those slices 26 that belong to any portion of sub-picture 54 has encoded therein a portion of a segment, or a complete one or more segments, of the second set of segments that form the composition of sub-picture 54. Note that the encoding path or order 22' may traverse the subset of segments 30 participating in the constituent sub-pictures 54 in an order different from the order in which the encoding path 22 traversed within the original picture 18. However, this does not affect the decoding result, since the segment coding independence prohibits the coding interdependencies across segment boundaries anyway. Within each segment 30, however, the paths 22 and 22' coincide, and this is important to keep decoding synchronicity and placement of the content of the slice unaffected even when the start position information 42 is used. Likewise, the decoder may check whether decoder 38/52 satisfies the decoder capability level for the sub-picture decoding option.
As described above, the decoder may derive the decoding option indication 58 in different ways. For example, the decoder 52 may analyze the segments covered by the slices present in the inbound data stream (i.e., 10 or 12) to determine whether the slices and corresponding segments have been removed, e.g., by the omission of the network device 50, and if so, which slices and corresponding segments have been removed. Additionally, or alternatively, an external signaling, shown as 58 in fig. 1, may be used by the decoder 38 or 52 to determine the decoding options or objectives to be used. The external signaling may have uniquely identified the encoding option to be used, or may exclude only some encoding options that are not used or are not available due to, for example, the intermediate extraction process 14, in which case the external signaling may originate from the network device 50, or may positively identify a set of encoding options from which the decoder will select one based on further information. Additionally, or alternatively, side information in the inbound data stream, such as explicit information about still decodable options, may be used by the decoder 38 or 52 to determine the decoding options or targets to use. As with the external information, the side information may already be comprised by the extraction entity, i.e. the device 50. In addition, the decoder may check the decoder capability level associated with the decoding options present in the parameter set 54 to exclude some of the list of possible decoding option candidates when they conflict with the capabilities of the decoder. Combinations of some or all of these cues may also form decoding option indications. In case of any remaining ambiguities, the decoder may use the above priorities to determine the one of the remaining possible decoding option candidates with the highest priority.
Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices, such as microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.
The data stream of the present invention may be stored on a digital storage medium or may be transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the internet.
Embodiments of the present invention may be implemented in hardware or software, depending on the particular implementation requirements. The implementation can be performed using a digital storage medium, e.g. a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a flash memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Accordingly, the digital storage medium may be computer-readable.
Some embodiments according to the invention comprise a data carrier with electronically readable control signals capable of cooperating with a programmable computer system so as to perform one of the methods described herein.
In general, embodiments of the invention may be implemented as a computer program product with a program code operable to perform one of the methods when the computer program product runs on a computer. The program code may be stored on a machine-readable carrier, for example.
Other embodiments include a computer program stored on a machine-readable carrier or non-transitory storage medium for performing one of the methods described herein.
Thus, in other words, an embodiment of the inventive method is thus a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
Thus, a further embodiment of the inventive method is a data carrier (or digital storage medium, or computer readable medium) comprising a computer program recorded thereon for performing one of the methods described herein. Data carriers, digital storage media or recording media are usually tangible and/or non-transitory.
Thus, a further embodiment of the inventive method is a data stream or a signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may for example be arranged to be transmitted via a data communication connection, for example via the internet.
Further embodiments include a processing apparatus, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.
Further embodiments include a computer having installed thereon a computer program for performing one of the methods described herein.
Further embodiments according to the present invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, mobile device, memory device, or the like. For example, the apparatus or system may comprise a file server for transmitting the computer program to the receiver.
In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware means.
The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.
The apparatus described herein or any component of the apparatus described herein may be implemented at least in part in hardware and/or software.
The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.
Any components of the methods described herein or the apparatuses described herein may be performed, at least in part, by hardware and/or software.
The above-described embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is the intention, therefore, to be limited only by the scope of the claims appended hereto, and not by the specific details presented by way of description and explanation of the embodiments herein.
- 上一篇：一种医用注射器针头装配设备
- 下一篇：有损图像压缩中的混合噪声和细致纹理合成