Partition-based intra coding concept for achieving high efficiency

文档序号：1895380 发布日期：2021-11-26 浏览：10次中文

阅读说明：本技术 实现高效率的基于分区的帧内编码构思 (Partition-based intra coding concept for achieving high efficiency ) 是由圣地亚哥·德鲁西安埃尔南德斯本杰明·布鲁斯阮藩黄松瓦勒日·乔治海科·施瓦茨德特勒于 2020-03-11 设计创作，主要内容包括：一种用于从数据流中基于块地解码图片的解码器,被配置为针对图片的预定块,解码帧内编码模式和划分维度标志,并且根据划分维度标志将划分维度设置为水平或垂直。解码器被配置为沿着预定维度将预定块划分为变换分区,变换分区在垂直于预定维度的方向上与预定块一样宽。此外,解码器被配置为针对每个变换分区,解码预测残差的变换；以取决于帧内编码模式的方式,根据与预定块相邻的一个或多个已重构样本,对预定块进行帧内预测,以获得预定块的预测子；并且通过使用针对相应变换分区解码的预测残差的变换校正每个变换分区内的预测子,来重构预定块。(A decoder for block-based decoding of a picture from a data stream is configured to decode an intra coding mode and a division dimension flag for a predetermined block of the picture, and set a division dimension to be horizontal or vertical according to the division dimension flag. The decoder is configured to divide the predetermined block into transform partitions along the predetermined dimension, the transform partitions being as wide as the predetermined block in a direction perpendicular to the predetermined dimension. Furthermore, the decoder is configured to decode, for each transform partition, a transform of the prediction residual; intra-predicting the predetermined block based on one or more reconstructed samples adjacent to the predetermined block in a manner dependent on an intra-coding mode to obtain a predictor of the predetermined block; and reconstructing the predetermined block by correcting the predictor within each transform partition using a transform of the prediction residual decoded for the corresponding transform partition.)

1. A decoder for block-based decoding of a picture (12) from a data stream (14), configured to:

-decoding an intra coding mode (116) of a predetermined block (80) of the picture from the data stream (14);

decoding a partition dimension flag (114) of a predetermined block of the picture from the data stream and setting a partition dimension (104) to horizontal or vertical according to the partition dimension flag,

dividing the predetermined block (80) along a predetermined dimension (104) into transform partitions (300), the transform partitions (300) being as wide as the predetermined block in a direction perpendicular to the predetermined dimension;

for each transform partition, decoding a transform of a prediction residual from the data stream (182);

intra-predicting the predetermined block from one or more reconstructed samples adjacent to the predetermined block in a manner dependent on the intra-coding mode to obtain a predictor of the predetermined block; and

-correcting (122) the predictor within each transform partition by using a transform of the prediction residual decoded for the respective transform partition to reconstruct the predetermined block.

2. The decoder according to claim 1, wherein the decoder is configured to,

wherein the number of transform partitions is greater than 2 and/or the transform partitions are one sample wide along the predetermined dimension.

3. The decoder according to claim 1 or 2,

wherein the decoder is configured to, for each transform partition, retransform the transform into the spatial domain for correcting the predictor within the respective transform partition.

4. Decoder according to any of claims 1 to 3,

wherein the decoder is configured to decode the transform from the data stream for each partition by:

decoding a coded transform partition flag (188) from the data stream;

setting a prediction residual of the corresponding transform partition to zero if the coded transform partition flag is not set, an

Decoding transformed transform coefficients of prediction residuals of respective transform partitions from the data stream if the coded transform partition flag is set.

5. The decoder of claim 4, configured to:

the transcoding partition flags of the transform partitions are decoded sequentially from the data stream, and if all previous transcoding partition flags are not set, it is inferred that the transcoding partition flag of the last transform partition in transform partition order is set.

6. The decoder of claim 5, configured to:

decoding the coded transform partition flags of the respective transform partition from the data stream by using context dependent entropy decoding, the context dependent entropy decoding using a context dependent on the coded transform partition flags decoded for a previous transform partition preceding the respective transform partition in a predetermined transform partition order.

7. Decoder according to any of claims 1 to 6,

wherein the decoder is configured to decode, for a predetermined transform partition, a transform of a prediction residual of the predetermined partition from the data stream by:

decoding a last position indication (190) from the data stream, the last position indication indicating a last transform coefficient position of the transform along a predetermined scan order, the predetermined scan order being for scanning transform coefficients of a one-dimensional transform; and

transform coefficients in the transform up to a last transform coefficient position along the predetermined scan order are decoded from the data stream (198), and transform coefficients in the transform beyond the last transform coefficient position along the predetermined scan order are inferred to be zero.

8. The decoder of claim 7, wherein the transform partition is one sample wide along the predetermined dimension, and the transform is a one-dimensional transform.

9. The decoder according to any of claims 1 to 8,

the transformation is a DCT transformation in case the intra prediction mode is not a planar mode, the transformation is a DST transformation in case the intra prediction mode is a planar mode, or

The transformation is a linear transformation, the type of which is selected based on an intra prediction mode, a block size of a predetermined block, and/or a dedicated syntax element.

10. The decoder according to any of claims 1 to 9, configured to:

-decoding a partitioning mode flag (160) of a predetermined block of said picture from said data stream;

performing decoding of the partition dimension flag, the partitioning, and decoding of the transform of each transform partition if the partition mode flag indicates a first partition mode;

decoding a transform of the prediction residual within the predetermined block if the partition mode flag indicates a second partition mode without performing the following: decoding the partition dimension indicator, the partitioning, and decoding the transform for each transform partition.

11. The decoder of any preceding claim, configured to:

decoding the partition dimension flag (114) by using context dependent entropy decoding using a context that depends on the intra coding mode.

12. The decoder of any preceding claim, configured to:

decoding the partition dimension flag (114) by using context dependent entropy decoding using one of three contexts including:

signaling intra coding modes other than angle mode,

signaling the intra coding mode of the horizontal mode,

the intra coding mode of the vertical mode is signaled.

13. The decoder according to any of claims 1 to 12, configured to:

setting a width of the transform partition measured along a predetermined dimension (104) in dependence on:

a size of the predetermined block (80) along the predetermined dimension (104), and/or

Intra coding mode of said predetermined block, and/or

Whether an intra coding mode of the predetermined block is an angle mode.

14. Decoder according to any of claims 1 to 13,

wherein the decoder is configured to perform intra-prediction of the predetermined block from one or more reconstructed samples adjacent to the predetermined block to obtain a predictor of the predetermined block in a manner dependent on the intra-coding mode, and to reconstruct the predetermined block by correcting (122) the predictor within each transform partition using a transform of the prediction residual decoded for the respective transform partition:

sequentially reconstructing transform partition groups according to a predetermined sub-partition order (126) into which transform partitions of the predetermined block are grouped to form sub-partitions (102; 112) for each transform partition group by, for a current sub-partition and before continuing processing subsequent sub-partitions, sequentially traversing the sub-partitions along a predetermined dimension:

deriving (122) a predictor for a current sub-partition by padding the current sub-partition according to one or more reconstructed samples neighboring the current sub-partition in a manner dependent on the intra coding mode;

the predictor within each transform partition comprised by the set of transform partitions forming the current sub-partition is corrected (122) by using the transform of the respective transform partition to reconstruct the current sub-partition.

15. The decoder of claim 14, configured such that:

the number of transform partitions (300) per sub-partition depends on the size of the predetermined block.

16. The decoder of claim 15, configured such that:

the number of transform partitions (300) of each sub-partition is 1 in case the size of the predetermined block exceeds a predetermined threshold, and the number of transform partitions (300) of each sub-partition is greater than 1 in case the size of the predetermined block does not exceed the predetermined threshold.

17. The decoder of any of claims 14 to 16, configured such that:

the number of sub-partitions in the predetermined block depends on the size of the predetermined block and/or the predetermined dimension (104).

18. The decoder of claim 17, configured such that:

the number of sub-partitions (300) in the predetermined block is 1 in case the size of the predetermined block is below a further predetermined threshold, and the number of sub-partitions (300) in the predetermined block is greater than 1 in case the size of the predetermined block is not below the further predetermined threshold.

19. The decoder of any of claims 17 to 18, configured such that:

the number of sub-partitions in the predetermined block depends on the size of the predetermined block in the following respects: the number of sub-partitions in the predetermined block is equal to a first number in a case where the size of the predetermined block employs a first width and a first height, and is equal to a second number different from the first number in a case where the size of the predetermined block employs a second width equal to the first height and a second height equal to the first width.

20. An encoder for block-based encoding of a picture (12) into a data stream (14), configured to:

encoding an intra coding mode (116) of a predetermined block (80) of the picture into the data stream (14);

encoding a partition dimension flag (114) for a predetermined block of the picture into the data stream, the partition dimension flag signaling that a partition dimension (104) is to be set to horizontal or vertical,

for each transform partition, a transform (182) of the prediction residual is encoded into the data stream, such that a predictor within each transform partition can be corrected (122) by using the transform of the prediction residual encoded for the respective transform partition to reconstruct the predetermined block.

21. The encoder according to claim 1, wherein the encoder is a digital encoder,

wherein the number of transform partitions is greater than 2 and/or the transform partitions are one sample wide along the predetermined dimension.

22. The encoder of claim 21 or 22, configured to:

for each transform partition, the prediction residuals in the respective partition are transformed into the spectral domain for correcting the predictors within the respective transform partition.

23. Encoder according to any of the claims 21 to 23,

wherein the encoder is configured to encode the transform into a data stream for each partition by:

encoding a transcoding partition flag (188) into the data stream;

wherein if the coded transform partition flag is not set, the coded transform partition flag (188) signals that the prediction residual of the corresponding transform partition is zero, and

the encoder is configured to encode transformed transform coefficients of a transform of a prediction residual of a transform partition into the data stream, if the coded transform partition flag is set.

24. The encoder of claim 24, configured to:

encoding the transcoding partition flags of the transform partitions into the data stream sequentially except for the transcoding partition flag of the last transform partition in transform partition order, wherein the transcoding partition flag of the last transform partition in transform partition order is inferred to be set if all previous transcoding partition flags are not set.

25. The encoder of claim 25, configured to:

encoding the coded transform partition flags of the respective transform partitions into the data stream by using context-dependent entropy encoding that uses contexts that depend on the coded transform partition flags encoded for previous transform partitions preceding the respective transform partitions in a predetermined transform partition order.

26. Encoder according to any of claims 21 to 26, configured to encode, for a predetermined transform partition, a transform of a prediction residual of the predetermined partition into the data stream by:

encoding a last position indication (190) into the data stream, the last position indication indicating a last transform coefficient position of the transform along a predetermined scan order, the predetermined scan order being for scanning transform coefficients of a one-dimensional transform; and

encoding into the data stream transform coefficients (198) in the transform up to a last transform coefficient position along the predetermined scan order, wherein transform coefficients in the transform exceeding the last transform coefficient position along the predetermined scan order are inferred to be zero.

27. The encoder of claim 27, wherein the transform partition is one sample wide along a predetermined dimension, and the transform is a one-dimensional transform.

28. The encoder according to any one of claims 21 to 28,

the transformation is a DCT transformation in case the intra prediction mode is not a planar mode, the transformation is a DST transformation in case the intra prediction mode is a planar mode, or

The transformation is a linear transformation, the type of which is selected based on an intra prediction mode, a block size of a predetermined block, and/or a dedicated syntax element.

29. The encoder according to any of claims 21 to 29, configured to:

encoding a partition mode flag (160) for a predetermined block of the picture into the data stream;

performing encoding of the partition dimension flag, the partitioning, and encoding of the transform for each transform partition if the partition mode flag indicates a first partition mode;

encoding a transform of a prediction residual within the predetermined block if the partition mode flag indicates a second partition mode without performing the following: encoding the partition dimension indicator, the partition, and encoding a transform for each transform partition.

30. The encoder according to any of the preceding claims 21 to 30, configured to:

encoding the partition dimension flag (114) by using context-dependent entropy encoding that uses a context that depends on the intra-coding mode.

31. The encoder according to any of the preceding claims 21 to 31, configured to:

encoding the partition dimension flag (114) by using context-dependent entropy encoding, the context-dependent entropy encoding using one of three contexts including:

signaling intra coding modes other than angle mode,

signaling the intra coding mode of the horizontal mode,

the intra coding mode of the vertical mode is signaled.

32. The encoder according to any of claims 21 to 32, configured to:

setting a width of the transform partition measured along a predetermined dimension (104) in dependence on:

a size of the predetermined block (80) along a predetermined dimension (104), and/or

Intra coding mode of said predetermined block, and/or

Whether an intra coding mode of the predetermined block is an angle mode.

33. The encoder according to any of claims 21 to 33, configured to perform intra-prediction of the predetermined block from one or more reconstructed samples neighboring the predetermined block to obtain a predictor of the predetermined block in a manner dependent on the intra-coding mode, and to determine a prediction residual of the predetermined block for correcting (122) the predictor within each transform partition using a transform of the prediction residual to be encoded for the respective transform partition by:

deriving predictors for a current sub-partition by populating the current sub-partition with one or more reconstructed samples adjacent to the current sub-partition in a manner dependent on the intra coding mode, sequentially predicting groups of transform partitions in a predetermined sub-partition order (126) into which the transform partitions of the predetermined blocks are grouped to form sub-partitions (102; 112) for each group of transform partitions, the predetermined sub-partition order sequentially traversing the sub-partitions along a predetermined dimension, determining a transform of prediction residuals within each transform partition comprised by the group of transform partitions forming the current sub-partition for correcting (122) the predictor within each transform partition comprised by the group of transform partitions forming the current sub-partition by using the transform of the respective transform partition, before proceeding to processing subsequent sub-partitions, to reconstruct the current sub-partition.

34. The encoder of claim 34, configured such that:

the number of transform partitions (300) per sub-partition depends on the size of the predetermined block.

35. The encoder of claim 35, configured such that:

36. The encoder of any of claims 34 to 36, configured such that:

the number of sub-partitions in the predetermined block depends on the size of the predetermined block and/or the predetermined dimension (104).

37. The encoder of claim 37, configured such that:

38. The encoder according to any of claims 37 to 38, configured such that:

39. A method for block-based decoding of a picture (12) from a data stream (14), comprising:

-decoding an intra coding mode (116) of a predetermined block (80) of the picture from the data stream (14);

for each transform partition, decoding a transform of a prediction residual from the data stream (182);

-correcting (122) the predictor within each transform partition by using a transform of the prediction residual decoded for the respective transform partition to reconstruct the predetermined block.

40. A method for block-based encoding of a picture (12) into a data stream (14), comprising:

encoding an intra coding mode (116) of a predetermined block (80) of the picture into the data stream (14);

41. A data stream generated by the method of claim 41.

42. A computer program having a program code for performing the method according to claim 40 and/or 41 when running on a computer.

Technical Field

The present application relates to intra coding concepts for use in block-based codecs, such as hybrid video codecs.

Background

Given a certain block, intra prediction is performed in HEVC by extrapolating decoded boundary samples of neighboring blocks, following certain modes (i.e., 33 angular modes and DC and planar mode [1 ]). One intra prediction mode that minimizes the rate-distortion cost is then signaled to the decoder. Although known codecs support many Intra Prediction Modes (IPMs), intra prediction thus achieved is still the subject of development to find better intra predictors leading to higher coding efficiency. This applies not only to HEVC but also to other block-based codecs that use intra prediction. Finding a set of intra-prediction modes suitable for efficient coding of the interior of a block requires consideration of signaling overhead of the intra-prediction modes and the resulting quality of predictors obtained by these intra-prediction modes due to the fact that more accurate predictors reduce the prediction residual, thereby reducing the signaling overhead associated with coding the prediction residual. In order to keep the signaling overhead associated with the intra-prediction mode low, the intra-prediction block should be large, i.e. signaling that the granularity of the intra-prediction mode should remain coarse, but on the other hand, the spatial prediction of a large block tends to be less accurate due to the higher average sample distance of the samples inside the intra-prediction block (i.e. the samples to be predicted) from the decoded/encoded samples (i.e. the reference samples) neighboring the block. HEVC somewhat alleviates this dilemma (catch-22) by allowing a transform residual block, which forms leaf blocks with respect to the corresponding coding units, to inherit the intra prediction mode of its corresponding coding unit, which is subdivided into these leaf blocks by multi-tree subdivision. However, this still requires signaling overhead for signaling from the encoder to the decoder the subdivision of the respective intra-coded coding unit into transform blocks.

In the newly developed multifunctional video coding (VVC) standard, an intra sub-partition (ISP) coding mode proposes a newly developed intra coding concept, but there is a need to improve implementation efficiency.

Therefore, the presently advantageous idea is to further improve the implementation efficiency of intra coding with comparable coding efficiency.

Disclosure of Invention

It is therefore an object of the present invention to provide a more efficient concept for intra coding.

This object is achieved by the subject matter of the independent claims of the present application.

Advantageous aspects of the invention are the subject matter of the dependent claims.

According to a first aspect of the present invention, the inventors of the present application have realized that one problem encountered when using sub-partitioning in connection with intra-coding is that the number of sub-partitions per block for which prediction is individually made should be limited, considering the resulting sub-partition size, the required minimum throughput (e.g. 16 samples per cycle) and/or the minimum coding width advance per coding cycle (e.g. 4 sample width advances per prediction). These ideas lead to the idea of building flag-controlled intra prediction modes/decisions for intra coded (predetermined) blocks, which leads to a partitioning of the predetermined block in terms of prediction residual transformation, while a fine partitioning in terms of intra prediction, i.e. whether the predetermined block is intra predicted in an overall manner (all at once), or whether transform partitions are used for intra prediction intermediate in order and partition manner using the prediction residual, and the correction of the just intra predicted sub-partition is also used for intra prediction on the next sub-partition, or whether a group of transform partitions of such predicted sub-partitions can be freely implemented as needed, e.g. to render the latter choice in terms of target block size, e.g. to avoid that intra prediction leads to too few samples per performed intra prediction or too few advanced intra prediction widths. Note that the encoding and decoding of the transform partitions can be done independently between transform partitions, i.e. the transform partitions can be encoded/decoded in parallel, thus not causing any minimum samples per loop or width-advance issues per loop. This enables the intra prediction block to be divided into partitions with e.g. less than 16 samples, since more than one partition can be intra predicted and reconstructed in the same cycle. It is advantageous if all sub-partitions encoded or decoded in the same cycle together comprise at least 16 samples. Also, according to the variant described herein, the coding of the coding supports many block sizes and, depending on the size of the predetermined intra-predicted block and/or depending on its width and/or height, the decoder and the encoder set the partitioning in terms of prediction to yield one of the following options:

1) global intra-prediction block, i.e. all (at once) or as a whole (in other words, for example, predicting the whole predetermined block at once, or even in other words, predicting all samples within the predetermined block based on neighboring samples located exclusively outside the predetermined block and processing the transform partitions of the predetermined block independently (i.e. performing the transform regionally within each transform partition)), and/or

2) Sequential intra prediction in units of transform partitions, which then also serve as predictor partitions (in other words, for example, to predict a transform partition, to encode/decode the prediction residual of the transform partition while obtaining reconstructed samples within the transform partition, and then to predict the next transform partition in a predetermined block using the reconstructed samples obtained for the previous transform partition, to encode/decode the prediction residual of the next transform partition, and so on), and/or

3) Sequential intra prediction in units of transform partition groups (each transform partition belongs to exactly one partition group) (in other words, for example, a transform partition group (i.e., a predictor partition) is predicted based on neighboring samples located exclusively outside the predictor partition, prediction residuals of the predictor partition are encoded/decoded, while reconstructed samples within the predictor partition are obtained in units of the transform partition within the predictor partition (i.e., transform is performed by region within each transform partition), and then a next transform partition group (i.e., a next predictor partition) in a predetermined block is predicted using the reconstructed samples, the reconstructed samples including samples obtained for a previous predictor partition but not including samples located inside the next predictor partition, prediction residuals of the next predictor partition are encoded/decoded in units of the transform partitions, and so on).

Thus, according to a first aspect of the present application, a decoder for block-based decoding of a picture from a data stream is configured to decode an intra coding mode of a predetermined block of the picture from the data stream. The decoder is configured to decode a division dimension flag of a predetermined block of a picture from the data stream, and set a division dimension to horizontal or vertical according to the division dimension flag. In other words, the division dimension flag indicates whether the division dimension is horizontal or vertical. The decoder is configured to divide the predetermined block into transform partitions along a predetermined dimension (i.e. along a dividing dimension), the transform partitions being as wide as the predetermined block in a direction perpendicular to the predetermined dimension. Transform partitions may be associated with vertically stacked horizontal blocks if the partition dimension is vertical, and with vertical blocks arranged horizontally side-by-side if the partition dimension is horizontal. For each transform partition, the decoder is configured to decode a transform of the prediction residual from the data stream. Furthermore, the decoder is configured to intra-predict the predetermined block from one or more reconstructed samples adjacent to the predetermined block to obtain a predictor of the predetermined block in a manner dependent on an intra-coding mode, and correct the predictor within each transform partition by using a transform of the prediction residual decoded for the corresponding transform partition to reconstruct the predetermined block.

According to a first option, the decoder is configured to sequentially perform intra prediction of predictors one by one for the transform partitions, and to correct the predictors within each transform partition by using a transform of the prediction residual decoded for the respective transform partition, to reconstruct the transform partition.

According to a second alternative option, the decoder is configured to perform intra prediction of the predictor for each transform partition, and to decode a transform of the prediction residual from the data stream for each transform partition. Then, the decoder is configured to correct the predictor by using a transform of the prediction residual decoded for the corresponding transform partition to reconstruct the predetermined block. Therefore, all predictors are first intra predicted and all transforms of the prediction residual are decoded, and then the predictors within each transform partition are corrected by using the transforms of the prediction residual decoded for the respective transform partition to reconstruct all transform partitions. Thus, for example, in one step, all predictors are corrected.

Instead, according to the first option, the transform partitions are reconstructed one by one. In other words, according to the first option, the predictor is intra predicted and corrected for the current transform partition, and then the new predictor is intra predicted and corrected for the subsequent transform partition.

According to a third alternative, the decoder is configured to intra-predict the whole predetermined block in one step to obtain a prediction signal (i.e. predictor) and to divide this prediction signal into predictors for, for example, transform partitions of the predetermined block. According to an embodiment, each predictor is associated with a different transform partition. For example, transform partitions are processed independently by the decoder. Thus, for example, the decoder is configured to decode, for each transform partition, a transform of the prediction residual from the data stream, and correct the predictor by using the transform of the prediction residual decoded for the respective transform partition, to reconstruct the predetermined block. Alternatively, this is not performed once for the entire block, but on sub-partitions of the predetermined block that may be further divided into transform partitions. In this case, for example, the decoder is configured to intra-predict the sub-partitions of the predetermined block in one step to obtain a prediction signal (i.e., predictor), and divide the prediction signal into predictors for transform partitions, e.g., the sub-partitions of the predetermined block.

According to an embodiment, the decoder is configured to divide the predetermined block into sub-partitions according to the block size, wherein a minimum prediction width 4 is established to reduce hardware implementation complexity. The invention is not limited by the following examples of different partitions performed by the decoder. It is clear that also other sub-partitions and/or transform partitions can be implemented by the decoder.

4X 4 Block (example 1)

O horizontal segmentation: one 4 × 4 PU (prediction unit) and four independent 4 × 1 TUs (transform units).

O vertical segmentation: one 4 × 4 PU and four independent 1 × 4 TUs.

In other words, the entire 4 × 4 block is predicted at once and then divided into four transform partitions for independent processing.

8 × 4 Block (example 2)

O horizontal segmentation: two 8 × 2 PUs and four 8 × 1 TUs. The second PU is predicted using reconstructed samples of the second TU.

O vertical segmentation: two 4 × 4 PUs and four 2 × 4 TUs. The second PU is predicted using reconstructed samples of the second TU.

In other words, an 8 × 4 block is divided into two sub-partitions (i.e., PUs), and each sub-partition is divided into two transform partitions for independent processing.

4X 8 Block (example 3)

O horizontal segmentation: two 4 × 4 PUs and four 4 × 2 TUs. The second PU is predicted using reconstructed samples of the second TU.

O vertical segmentation: one 4 × 8 PU and four independent 1 × 8 TUs.

In other words, in the horizontal division, the 4 × 8 block is divided into two sub-partitions (i.e., PUs), each of which is divided into two transform partitions, to be independently processed, and in the vertical division, the entire 4 × 8 block is predicted at a time and then divided into four transform partitions to be independently processed.

4X 8 Block (example 3'; alternative to example 3)

Horizontal partitioning (no modification compared to the processing of sub-partitions in terms of prediction and transform residual encoding/decoding): two 4 × 4 PUs are used, which simultaneously form two 4 × 4 TUs. The second PU is predicted using the reconstructed samples of the first PU.

Vertical segmentation (modification): one 4 × 8 PU and two separate 2 × 8 TUs.

In other words, in the horizontal division, a 4 × 8 block is divided into two sub-partitions (i.e., PUs), each of which is eventually one transform partition, and in the vertical division, the entire 4 × 8 block is predicted once and then divided into two transform partitions to be processed independently.

4 XM blocks (example 4)

The entire 4 xm block is predicted at once and then divided into four 1 xm transform partitions for independent processing.

4 XM blocks (example 4'; where M >8)

Horizontal partitioning (no modification compared to the processing of sub-partitions in terms of prediction and transform residual encoding/decoding): predicting a 4 × M block with four 4 × (M/4) PUs, each PU being simultaneously one of four transform partitions

O vertical segmentation: the entire 4 xm block is predicted at once and then divided into four 1 xm transform partitions for independent processing.

8 XN Block (example 5)

The 8 xn block is divided into two 4 xn sub-partitions, which may be further divided into four 1 xn transform partitions.

8 XN blocks (example 5'; where N >4)

Horizontal partitioning (no modification compared to the processing of sub-partitions in terms of prediction and transform residual encoding/decoding): an 8 × N block is divided into four 8 × N (N/4) sub-partitions (for prediction and transform residual coding/decoding).

O vertical segmentation: the 8 xn block is divided into two 4 xn partitions (for prediction purposes), which can be further divided into two 2 xn transform partitions.

The examples outlined above relate to different block sizes and may be applied to a codec according to the corresponding embodiment (i.e. to a decoder and an encoder, respectively) individually, completely or in combination of two or more of them. It can be seen that, according to an embodiment, the decision how to select among the above options 1 to 3 (two of 1 to 3) may be different for at least one predetermined block size (e.g., comparative examples 3 to 5) depending on the division direction: while one option is selected for horizontal partitioning, e.g. option 2, where each TU is also a PU and the number of PUs and the number of TUs are therefore the same, a different option may be selected for vertical partitioning, e.g. option 1, where the entire block acts as a PU but is divided into TUs, the number of PUs and the number of TUs being different, or option 3, where a predetermined block is divided into PUs, each PU in turn being divided into TUs, the number of PUs and the number of TUs being different. Additionally or alternatively, for another block size (comparative example 2), the decision may eventually be the same option regardless of the segmentation direction. Thus, selecting the correlation to the segmentation direction among the just mentioned options may thus avoid the already mentioned block size direction, but naturally it may be applicable without the latter.

An embodiment according to the invention relates to an encoder for block-based encoding of a picture into a data stream, configured to encode an intra coding mode of a predetermined block of the picture into the data stream. The encoder is configured to encode a division dimension flag of a predetermined block of a picture into a data stream, the division dimension flag signaling that a division dimension is to be set to horizontal or vertical. In other words, the division dimension flag indicates whether the division dimension is horizontal or vertical. The encoder is configured to divide the predetermined block into transform partitions along a predetermined dimension (i.e. along a dividing dimension), the transform partitions being as wide as the predetermined block in a direction perpendicular to the predetermined dimension. Transform partitions may be associated with vertically stacked horizontal blocks if the partition dimension is vertical, and with vertical blocks arranged horizontally side-by-side if the partition dimension is horizontal. Furthermore, the encoder is configured to intra-predict the predetermined block from one or more reconstructed samples neighboring the predetermined block to obtain a predictor of the predetermined block in a manner dependent on the intra-coding mode. For each transform partition, the encoder is configured to encode a transform of the prediction residue into the data stream, such that the predictor within each transform partition can be corrected by using the transform of the prediction residue encoded for the respective transform partition to reconstruct the predetermined block.

The encoder as described above and the methods performed by any encoder and decoder as described herein and the data stream generated by the methods performed by any encoder as described herein are based on the same considerations as the decoder as described above. Incidentally, these methods can be carried out with all the features and functions which are also described with respect to the decoder and/or the encoder.

Drawings

Preferred embodiments of the present application are described below with reference to the accompanying drawings, in which:

fig. 1 shows a block diagram of an apparatus for predictive coding a picture as an example of an encoder capable of implementing the ISP concept;

fig. 2 shows a block diagram of an apparatus for predictive decoding of pictures, suitable for the apparatus of fig. 1, as an example of a decoder capable of implementing the ISP concept;

fig. 3 shows a schematic diagram illustrating an example of the relationship between a prediction residual signal, a prediction signal and a reconstructed signal in order to illustrate the possibility of setting a subdivision for coding mode selection, transform selection and transform performance, respectively;

FIG. 4 shows a schematic diagram illustrating the division of an intra-coded fast forward line according to the ISP variant which allows selection between different division dimensions (i.e. horizontal and vertical);

FIG. 5 shows a schematic diagram illustrating the sequential processing of partitions of ISP coding blocks;

FIG. 6 shows a schematic diagram illustrating the prediction derivation of the filling process of a partition;

fig. 7 illustrates an example of ISP blocks partitioned according to a horizontal partition mode and a vertical partition mode, respectively, and having two different intra prediction modes associated therewith, respectively, in order to illustrate the likelihood of a relevant determination of partition order being present on the intra prediction mode associated with an intra prediction block;

FIG. 8 shows a schematic diagram illustrating the possible signalization spent on intra-predicted blocks 80 processed using the partitioning option;

fig. 9 shows a schematic diagram illustrating a possible way of sending the prediction residuals of a partition;

FIG. 10 shows a schematic diagram illustrating partial sum determination of coding costs associated with partitions of intra-prediction modes in order to be able to abort a test when it is clear that it does not become better than any normal intra-prediction mode; and

FIG. 11 shows a flow diagram of the mode or operation of an encoder to perform a split mode test; and

FIG. 12 shows a schematic diagram of a decoder for block-based decoding of pictures that implements the ISP concept of the present invention;

FIG. 13 shows a diagram illustrating the use of a last position syntax element;

FIG. 14 shows a schematic diagram of an encoder for block-based decoding of pictures illustrating the ISP concept of the present invention;

fig. 15a to 15d show schematic diagrams illustrating intra prediction of respective transform partitions of a vertically divided 4 × 4 block;

fig. 16a to 16d show schematic diagrams illustrating intra prediction of respective transform partitions of a horizontally partitioned 4 × 4 block;

FIG. 17 shows a schematic diagram illustrating intra prediction of respective transform partitions of a vertically partitioned 4 × 8 block;

FIG. 18 shows a schematic diagram illustrating intra prediction of respective transform partitions of a horizontally partitioned 8 × 4 block;

FIG. 19a shows the vertical partitioning of 4M blocks (M >8) in ISP designs in VVC draft 5 (left) and proposal version (right); and

FIG. 19b shows the vertical partitioning of 8N blocks (N >4) in ISP designs in VVC draft 5 (left) and proposed version (right);

the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

Detailed Description

The following description of the figures starts with the presentation of a description of an encoder and decoder of a block-based prediction codec for encoding pictures of video in order to form an example of an encoding framework for embodiments in which an intra-prediction codec may be built in. The aforementioned encoder and decoder are described with reference to fig. 1 to 3. In the following, a description of variants of the ISP concept is presented together with a description on how these concepts are built into the encoder of fig. 1 and the decoder of fig. 2, respectively, although the concepts described with the figures following fig. 4 and 4 may also be used to form encoders and decoders that do not operate according to the encoding framework under the encoder of fig. 1 and the decoder of fig. 2. Later, embodiments utilizing ISPs, but with improvements in implementation efficiency, will be described. Furthermore, embodiments are described that utilize variations of partition-based intra coding.

Fig. 1 shows an apparatus for predictive coding a picture 12 into a data stream 14, illustratively using transform-based residual coding. The apparatus or encoder is indicated with reference numeral 10. Fig. 2 shows a corresponding decoder 20, namely apparatus 20 configured to also predictively decode a picture 12 'from the data stream 14 using transform-based residual decoding, wherein an apostrophe has been used to indicate that the picture 12' reconstructed by the decoder 20 deviates from the picture 12 originally encoded by the apparatus 10 in terms of coding loss introduced by quantization of the prediction residual signal. Fig. 1 and 2 exemplarily use transform-based prediction residual coding, although embodiments of the present application are not limited to such prediction residual coding. The same is true for the other details described with reference to fig. 1 and 2, which will be outlined below.

The encoder 10 is configured to perform a spatial to spectral transformation on the prediction residual signal and to encode the prediction residual signal thus obtained into a data stream 14. Also, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to perform a spectral-to-spatial transformation on the prediction residual signal thus obtained.

Internally, the encoder 10 may include a prediction residual signal former 22, the prediction residual signal former 22 generating a prediction residual 24 in order to measure the deviation of the prediction signal 26 from the original signal (i.e. picture 12). The prediction residual signal former 22 may be, for example, a subtractor that subtracts the prediction signal from the original signal (i.e., picture 12). The encoder 10 then further comprises a transformer 28, which transformer 28 performs a spatial-to-spectral transformation of the prediction residual signal 24 to obtain a spectral domain prediction residual signal 24 ', which spectral domain prediction residual signal 24' is then quantized by a quantizer 32 also comprised in the encoder 10. The prediction residual signal 24 "thus quantized is encoded into the bitstream 14. To this end, the encoder 10 may optionally comprise an entropy encoder 34, the entropy encoder 34 entropy encoding the transformed and quantized prediction residual signal into the data stream 14. The prediction residual 24 is generated by a prediction stage 36 of the encoder 10 based on a prediction residual signal 24 "encoded into the data stream 14 and decodable from the data stream 14. To this end, as shown in fig. 1, the prediction stage 36 may internally comprise an inverse quantizer 38, the inverse quantizer 38 dequantizing the prediction residual signal 24 "so as to obtain a spectral domain prediction residual signal 24" ' corresponding to the signal 24 ' except for quantization losses, the prediction stage 36 then comprising an inverse transformer 40, the inverse transformer 40 inverse transforming, i.e. spectral-to-spatial transforming, the latter prediction residual signal 24 "' so as to obtain the prediction residual signal 24" "corresponding to the original prediction residual signal 24 except for quantization losses. The combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24 "", for example by addition, in order to obtain a reconstructed signal 46, i.e. a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12'. Prediction module 44 of prediction stage 36 then generates prediction signal 26 based on signal 46 using, for example, spatial prediction (i.e., intra-prediction) and/or temporal prediction (i.e., inter-prediction).

Likewise, decoder 20 may be internally composed of components corresponding to prediction stage 36 and interconnected in a manner corresponding to prediction stage 36. In particular, the entropy decoder 50 of the decoder 20 may entropy decode the quantized spectral domain prediction residual signal 24 ″ from the data stream, whereupon the inverse quantizer 52, the inverse transformer 54, the combiner 56 and the prediction module 58, interconnected and cooperating in the manner described above with reference to the modules of the prediction stage 36, restore the reconstructed signal based on the prediction residual signal 24 ″ so that, as shown in fig. 2, the output of the combiner 56 produces the reconstructed signal, i.e., the picture 12'.

Although not specifically described above, it is readily apparent that the encoder 10 may set some encoding parameters, including, for example, prediction modes, motion parameters, etc., according to some optimization scheme, such as in a manner that optimizes some rate-distortion criteria. For example, encoder 10 and decoder 20 and corresponding modules 44, 58, respectively, may support different prediction modes, such as intra-coding modes and inter-coding modes. The granularity at which the encoder and decoder switch between these prediction mode types may correspond to the subdivision of the pictures 12 and 12' into coded slices or blocks, respectively. For example, a picture may be subdivided into intra-coded blocks and inter-coded blocks in units of these coded slices. As outlined in more detail below, intra-coded blocks are predicted based on the spatially already encoded/decoded neighborhood of each block. There may be several intra-coding modes including directional or angular intra-coding modes, and an intra-coding mode is selected for each intra-coded segment, according to which each segment is filled by extrapolating the sample values of the neighborhood into the corresponding intra-coded segment along a certain direction specific to each directional intra-coding mode. The intra-coding modes may also include, for example, one or more other modes such as a DC coding mode according to which the prediction for each intra-coded block assigns a DC value to all samples within each intra-coded segment and/or a planar intra-coding mode according to which the prediction for each block is approximated or determined as a spatial distribution of sample values described by a two-dimensional linear function over sample positions of each intra-coded block, wherein slopes and offsets of planes defined by the two-dimensional linear function are derived based on neighboring samples. In contrast to this, inter-coded blocks may be predicted, for example, temporally. For inter-coded blocks, a motion vector may be signaled in the data stream, indicating the spatial offset of the portion in the previously coded picture of the video to which picture 12 belongs, at which position the previously coded/decoded picture is sampled in order to obtain a prediction signal for the respective inter-coded block. This means that, in addition to the residual signal coding comprised by the data stream 14, such as the entropy coded transform coefficient levels representing the quantized spectral domain prediction residual signal 24 ″, the data stream 14 may have encoded therein coding mode parameters for assigning a coding mode to the respective block, prediction parameters for some blocks (e.g. motion parameters for inter-coded slices), and optionally other parameters (e.g. parameters for controlling and signaling the subdivision of the pictures 12 and 12' into slices, respectively). The decoder 20 uses these parameters to subdivide the picture in the same way as the encoder, assigns the same prediction mode to the slices, and performs the same prediction to produce the same prediction signal.

Fig. 3 illustrates the relation between the reconstructed signal, i.e. the reconstructed picture 12', on the one hand, and the combination of the prediction residual signal 24 "" signaled in the data stream and the prediction signal 26, on the other hand. As described above, the combination may be an addition. The prediction signal 26 is shown in fig. 3 as subdividing the picture area into intra-coded blocks, schematically indicated with hatching, and inter-coded blocks, not schematically indicated with hatching. The subdivision may be any subdivision, e.g. regularly subdividing the picture area into rows and columns of blocks or blocks, or multi-tree subdividing the picture 12 into different sized leaf blocks, e.g. quad-tree subdividing into blocks, etc., a mix of which is shown in fig. 3, where the picture area is first subdivided into rows and columns of tree root blocks, which are then further subdivided according to a recursive multi-tree subdivision. Likewise, intra-coded modes for intra-coded blocks 80 may be encoded in data stream 14, with one of several supported intra-coded modes being assigned to a respective intra-coded block 80. Further details are described below. For an inter-coded block 82, one or more motion parameters may be encoded in the data stream 14. In general, inter-coded blocks 82 are not limited to being temporally coded. Alternatively, the inter-coded block 82 may be any block predicted from a previously coded portion other than the current picture 12 itself (e.g., a previously coded picture of the video to which the picture 12 belongs, or a picture of another view or a layer lower in the hierarchy in the case where the encoder and decoder are scalable encoders and decoders, respectively). The prediction residual signal 24 "" in fig. 3 is also shown as subdividing the picture area into blocks 84. These blocks may be referred to as transform blocks in order to distinguish them from the encoded blocks 80 and 82. In practice, fig. 3 shows that encoder 10 and decoder 20 may subdivide pictures 12 and 12' into blocks using two different subdivisions, respectively, namely, one into encoded blocks 80 and 82, respectively, and the other into block 84. Both subdivisions may be the same, i.e. each encoded block 80 and 82 may form a transform block 84 at the same time, but fig. 3 shows a situation where, for example, the subdivision into transform blocks 84 forms an extension to the subdivision of encoded blocks 80, 82, such that any border between two of blocks 80 and 82 covers the border between two blocks 84, or in other words each block 80/82 coincides with either one of transform blocks 84, or with a cluster of transform blocks 84. However, the subdivisions may also be determined or selected independently of one another such that transform block 84 may alternatively span block borders between blocks 80/82. A similar statement is thus true with respect to the subdivision into transform blocks 84 as proposed with respect to the subdivision into blocks 80, 82, i.e. the blocks 84 may be the result of a regular subdivision of the picture area into blocks/blocks (arranged in rows and columns), the result of a recursive multi-tree subdivision of the picture area, or a combination thereof or any other type of way of partitioning (blocking). Incidentally, it is to be noted that the blocks 80, 82, and 84 are not limited to square, rectangular, or any other shape.

Fig. 3 illustrates that the combination of the prediction signal 26 and the prediction residual signal 24 "" directly results in the reconstructed signal 12'. It should be noted, however, that more than one prediction signal 26 may be combined with the prediction residual signal 24 "" to produce picture 12' according to alternative embodiments.

In fig. 3, the transform segment 84 should have the following meaning. Transformer 28 and inverse transformer 54 perform their transforms in units of these transform segments 84. For example, many codecs use some DST or DCT on all transform blocks 84. Some codecs allow the transform to be skipped so that for some transform slices 84 the prediction residual signal is coded directly in the spatial domain. However, according to the embodiments described below, the encoder 10 and the decoder 20 are configured in such a way that they support several transforms. For example, the transforms supported by the encoder 10 and decoder 20 may include:

o DCT-II (or DCT-III), where DCT stands for discrete cosine transform

Omicron DST-IV, where DST stands for discrete sine transform

οDCT-IV

οDST-VII

Omicron equal transformation (IT)

Naturally, transformer 28 should support all forward transformed versions of these transforms, while decoder 20 or inverse transformer 54 should support its corresponding backward or inverse version:

omicron inverse DCT-II (or inverse DCT-III)

Omicron inverse DST-IV

Omicron inverse DCT-IV

Omicron inverse DST-VII

Omicron equal transformation (IT)

The subsequent description provides more details about which transforms the encoder 10 and decoder 20 can support. In any case, it should be noted that the set of supported transforms may include only transforms such as one spectral-to-spatial transform or spatial-to-spectral transform.

As mentioned above, fig. 1 to 3 have been presented as examples, wherein the intra prediction concept described further below may be implemented. In this regard, the encoder of fig. 1 and the decoder of fig. 2 represent possible implementations of the encoder and decoder, respectively, described below. As will be outlined in more detail below, when embodiments for intra prediction according to the present application, which will be explained later, are built into the encoder of fig. 1 and the decoder of fig. 2, the encoder of fig. 1 and the decoder of fig. 2 support at least one option to process the intra prediction block 80 in a manner outlined in more detail below. Thus, the embodiments described below refer to an encoder equivalent to encoder 10 of fig. 1, encoder 10 of fig. 1 processing intra-coded blocks 80 in the manner outlined in more detail below, and is equally applicable to the decoder of fig. 2, which thus represents an example of a decoder according to an embodiment in which intra-coded blocks are processed in the manner outlined in more detail below. However, fig. 1 and 2 are only specific examples. However, an encoder according to embodiments of the present application may perform block-based encoding of picture 12 using concepts outlined in more detail below, and differs from the encoder of fig. 1 in, for example: not a video encoder, does not support inter prediction, or performs the subdivision into blocks 80 in a different way than illustrated in fig. 3, or even if, depending on the embodiment, the encoder does not use transform prediction residual coding when encoding the prediction residual, e.g. instead directly in the spatial domain. Likewise, a decoder according to embodiments of the present application may perform block-based decoding of pictures 12' from a data stream 14 using intra-prediction coding concepts outlined further below, but may differ from the decoder 20 of fig. 2, for example, in the following respects: not a video decoder but a still picture decoder, does not support intra prediction, or subdivides the picture 12' into blocks in a different manner than described with reference to fig. 3, and/or derives prediction residuals from the data stream 14 not in the transform domain but in the spatial domain.

It has been shown that the following description focuses first on the description of ISP-based intra prediction. According to ISP intra-prediction, intra-prediction blocks (e.g., block 80 in fig. 4) are allowed to be partitioned into one-dimensional horizontal partitions or one-dimensional vertical partitions. The availability to process blocks in this manner may be provided for intra-predicted blocks 80 of any size, or limited to blocks 80 of a block size within a predefined range, e.g., blocks larger than a certain size. "one-dimensional" refers to the fact that: when relating to partitions as a result of partitioningThe partition is only one sample width along the partition dimension. However, the one-dimensional nature of the partitioning scheme discussed herein refers to the fact that: the division occurs along a certain dimension, the resulting division being similar to a stripe extending completely in the block along and transverse to the direction of division. For example, referring to fig. 4, fig. 4 shows an intra-predicted block 80, i.e., a block to be decoded or a block to be encoded, on the left side. Its size is W × H. That is, it is a block of size W × H, where H and W are the height and width, respectively, of block 80 measured in samples. According to fig. 4, there are two available segmentation or partitioning options, namely horizontal segmentation 100, according to which a block 80 is segmented or divided along a vertical axis (i.e. a partitioning dimension 104) into a plurality of partitions 102₁、102₂、102₃And 102₄. According to the example of FIG. 4, which is an example applied in the following description, each partition 102 is shown with a double-headed arrow 106₁To 102₄Is one sample wide, such that the resulting partitions 102 from the block 80₁To 102₄Is equal to H, i.e. the height of the block 80 in units of samples 108 of the block 80, it should be clear that the encoder and decoder may also perform the partitioning according to different ways agreed between the encoder and decoder, e.g. may result in a predefined number of partitions 102_iThe division of the blocks 80 along the dimension 104 is done in a predefined number, e.g., greater than two, or a mixture thereof, the size of the blocks 80 being evenly distributed over a predefined number of partitions along the division dimension.

Another encoding option depicted in FIG. 4 and indicated by reference numeral 110 corresponds to partitioning the block 80 into vertical partitions 112₁、112₂、……112₈. That is, according to option 110, block 80 is divided into partitions 112 along a horizontal axis (i.e., horizontal division dimension 104)_i. In the case of option 100, each partition 102_iAs wide as the block 80, i.e. having the width W of the block, and the partition 112_iThe height H of the block 80 is taken, i.e. has a height H. To summarize, in a manner similar to the description of option 100, vertical partition 110 may partition block 80 into a number W of partitions 112_iW represents the water measured by the block 80 in units of samples 108Flat width such that each section 112_iOne sample width in the horizontal direction, however, the partitioning according to option 110 therein may also be performed in another way agreed between the encoder and the decoder.

Thus, according to fig. 4, the encoder is free to divide the block 80 into H W × 1 partitions 102 according to the horizontal partitioning option 100_iOr, the block 80 is divided into W H1 partitions 112 according to the vertical partition option 110_iAnd the partitioning option selected by the encoder for the block 80 may be signaled in the data stream 14 for the block 80, e.g., by a corresponding partition dimension flag 114 in the data stream 14. However, it should be clear that embodiments of the present application also cover the following encoders and decoders: by default only one of the options 100 and 110 is used without the flag 114 in the data stream. Still further, in other examples, the flag 114 may be transmitted in the data stream 80 depending on the intra-coding mode 116 signaled from the encoder to the decoder in the data stream 14 for the block 80. As outlined above, the intra-coding mode may indicate one of a set of available/supported intra-coding modes, the set comprising, for example, an angular mode and optionally one or more non-angular modes, such as a DC mode or a planar mode. That is, according to alternative embodiments not discussed further below, the flag 114 may be conditionally conveyed in the data stream 14 depending on the intra-coding mode 116. According to the embodiments described below, for block 80, flag 114 is present in data stream 14 independent of intra-coding mode 116 signaled for block 80 in data stream 14. However, there may be a correlation with respect to flag switching between the division process of intra-coded block 80 just discussed and the different ways of intra-coding of block 80 that will be outlined below.

Each of the partitions 102/112 is predicted, transformed, quantized, and encoded individually according to the ISP, in such a way that the partitions are processed sequentially. Thus, reconstructed samples of a certain partition will be able to be used for any subsequent partition 102/112 in partition order in the partition into which the prediction block 80 has been divided, and in this way the process of intra-prediction is on the partition 102/112 into which the block 80 has been dividedAnd (6) circulating. Fig. 5 exemplarily shows an intra-predicted block 80 partitioned according to option 100. For each partition 102 of block 80₁To 102₄Making predictions, i.e. deriving the corresponding partitions 102_iAnd a prediction residual correlation task, i.e., correcting the predictor using the prediction residual. The latter task may be performed by combining the prediction residual and the predictor. This is done in the decoder for reconstruction. The encoder performs the determination of the prediction residual as a prediction residual related task, including for example transformation and quantization, and uses the prediction residual to correct the predictor, i.e. to keep the prediction loop synchronized with the decoder by filling the decoded picture buffer in the encoder with the reconstruction of the picture. For partition 102₁To 102₄Separate and within partition 102₁To 102₄The just mentioned tasks, i.e. prediction and residual processing, are performed sequentially in between. After these two steps for the currently processed partition, the next partition 102 according to the partition order is processed in the same manner_i. The partitioning order is exemplarily illustrated in fig. 5 using three arrows 126.

FIG. 5 illustrates the immediately adjacent partition 102 as it continues processing₂Before etc., the partition comprising the top-left pixels of the block 80 should be processed first, corresponding to the partition 102 in fig. 5₁To 102₄But this order is merely an example and, as will become clear from the description below, the partition order may be selected in a manner dependent on other settings, such as intra coding mode and/or size of the block 80, the former dependency being discussed below.

In examples discussed further below, the partition order 126 is such that immediately following partitions are immediately adjacent to each other in a manner that only varies between those orders of traversing the partitions 102/112, such that in the case of partition type 100, the partition order is top-to-bottom or bottom-to-top, respectively, and in the case of partition type 110, the partition order is left-to-right or right-to-left, respectively. However, it should be mentioned that other examples are also conceivable. For example, the partition order may be selected in such a way that in two scans the partitions are scanned in the adjacent order just outlined, in the first scan from top to bottom, bottom to top, left to right or right to left (as applicable), every other partition is processed once, and then the remaining partitions between them are processed in the same sequential direction or in the opposite sequential direction.

In any event, FIG. 5 shows the first partition 102₁To be processed first and is the currently processed partition. For the first partition, here illustrated as 102₁Selection for forming partition 102 may be based only on samples located outside the bounding box of block 80₁Of the predictor 118₁Since the samples of block 80 have not been processed, i.e., reconstructed or encoded, while the first partition of block 80 was processed. That is, the set 118 has been reconstructed in the encoder using any prediction and correction of the respective predictor using the prediction residual sent in the data stream₁The sample of (1). They belong to previously encoded/decoded picture blocks and may be inter-coded or intra-coded or any other encoded block. With respect to forming the first partition 102₁Of the predictor 118₁Also depends on the intra coding mode assigned to the block 80. This intra-coding mode is used in common, or equally, for each partition of the processing block 80, as will be discussed below. To complete the first partition 102₁By processing according to set 118₁Fills the partition 102 with one or more reconstructed/encoded samples₁The partitions 102 are derived in a decoder and encoder₁Determines its prediction residual, i.e. by transformation and quantization as described above, in the case of an encoder, which prediction residual (in the version sent in the data stream, i.e. including quantization losses) is then used to correct the predictor by using the prediction residual in the data stream 14 to reconstruct the partition 102₁. For example, FIG. 5 shows partition 102₁Is predicted, an example is at 120₁To (3). That is, 120₁Includes and partition 102₁Of the prediction residual, wherein the data 120 is the transform coefficients corresponding to the transform of the prediction residual₁The description of which will be discussed in more detail below.

Now it isGo to the next partition in partition order, partition 102 in the example of FIG. 5₂. The situation has changed because it was used to export the partition 102₂The set of neighbouring reconstructed/coded samples of the predictor of (2) may now consist of samples located outside the block 80 and/or samples within the block 80, i.e. those located in any already processed partition, here the current partition 102 in the example of fig. 5₁Since for these samples the prediction residual is already determined and already available in the data stream 14. That is, the encoder and decoder derive the partition 102 separately₂Then the prediction residual determination in the encoder is made and used in the encoder and decoder to correct the predictor. The process then continues with the next partition, one by one, in partition order, whereby all partitions of block 80 are processed in order.

As already mentioned above, the partition order 126 may be selected in another way than traversing partitions, such that directly consecutive partitions are immediately adjacent partition neighborhoods. That is, the partition order may skip from one partition to the next. This means for filling the corresponding partition 102_iTo derive a set of neighboring samples 118 for the respective predictor_iNot limited to the immediate sample neighborhood of the respective partition as shown in fig. 5. This also relates to the selection of the start of the partition order 126. For example, assume partition 102₄Is the first partition in partition order. The method may then proceed by basing the set 118 of neighboring samples₄(not shown in FIG. 5) populating partition 102₄To derive its predictor, the set 118₄Samples located along the periphery of the block 80, on the left and top of the block 80, are collected. Set 118₄Will not be immediately adjacent to partition 102₄. Incidentally, this will correspond to the case where the last sample line is filled in the normal intra prediction padding of the block 80 as a whole. The just mentioned possibility is also true for any subsequently processed partition, i.e. the second partition and the further partitions in partition order. That is, their neighboring sample sets 118_iOr may include partitions 102 that are not associated with the corresponding partition_iA certain in the immediate vicinityThese samples were used. And still further, any second or subsequently processed partition 102 without limiting the partition order to traverse partitions in such a way that consecutive partitions are immediately adjacent to each other_i118 of the reference sample_iMay not only collect data located in the corresponding partition 102_iThe left and top samples of (1), and the samples located in the corresponding partitions 102 may also be collected₁Lower sample, depending on whether partition 102 is compared to partition order₁Any partitions of block 80 are processed earlier. That is, set 180_iMay include being located in partition 102_iMore than samples on both sides.

Briefly, FIG. 5 herein illustrates the sequential processing of partitions 102/112 of a block 80 with respect to horizontal partitions by way of example, but the same description applies to vertical partitions 112_iThe vertical mode 110. For each partition 102_iThe corresponding prediction residual 102_iContained in the data stream 14. Data 120₁To 120₄Together forming the prediction residual, i.e. 120, of block 80. It should be noted that according to an alternative embodiment of the present application, the prediction residual 120 of the block 80 may not be used, i.e. may be signaled directly in the data stream 14 (e.g. in the spatial domain). In this case, each partition 102₁To 102₄Data 120 of₁To 120₄The partition individual fields in the data stream 14 shown in FIG. 5 may not be included, with each data portion 120_iThe representation is for the corresponding partition 102_iSignaling of a certain transition of (2). Rather, in that case, the prediction residual 120 of block 80 may form one field of data 14. In this alternative embodiment, the decoder will be processing a certain partition 102_iThe time from field 120 is collected about the partition 102_iInformation of the prediction residual of (1). This procedure can also be applied when a fully reversible version of the transform is used, so that the quantization can be done in the spatial domain.

Thus, fig. 5 shows for each partition 102 in an encoder and a decoder_iTwo tasks are performed, namely: 1) prediction derivation task 122, producing corresponding partition 102₁Is predicted or predicted, i.e.For the corresponding partition 102_iAnd 2) the prediction residual correlation task performed thereafter, i.e. the prediction residual derivation at the encoder, including quantizing the prediction residual for encoding it into the data stream 14, and the corresponding partitioning 102 by combining or correcting the prediction residual and the predictor_iTo obtain the partition 102_iThe reconstructed sample of (1). To predict the derived tasks, the later reconstructed samples may be used as subsequent processing partitions 102 following the partition order 126_jOf the neighboring sample set 118_jTo reserve (c).

Before continuing with further details of the ISP conception possibilities, FIG. 6 illustrates the partitioning 102 by population of the current process_iTo perform the process of prediction derivation 122, where it is noted that the description of the horizontal partition 102 is chosen only illustratively and the same description also refers to the vertical partition 112. FIG. 6 illustrates a currently processed partition 102_iAnd its corresponding set 118 of reconstructed/encoded neighboring samples_i. As already indicated above with respect to FIG. 5, set 118_iMay not be limited to the partition 102_iDirectly adjacent or neighboring samples 128. However, due to partitioning, partition 102 when averaging all samples of block 80_iSample and set 118 of_iIs lower than performing intra prediction of block 80, e.g. as known from h.264 or HEVC. As described with respect to FIG. 5, the intra-prediction mode associated with the block 80 is used, for each partition 102_iPredictor derivation or padding 122 is performed, the mode indicating one of a set of available intra prediction modes. The set may include different angle or directivity patterns from each other in the angle or direction 132, along which angle or direction 132 the adjacent sample sets 118_iTo the partition 102_iIn the sample 134. To perform the copy, it may be based on the collection 118_iThe number of adjacent samples 134 positioned in a direction opposite to direction 132 relative to samples 134 in deriving partition 102_iOf each sample 134. This number is used, for example, to derive the sample set 118_iIs defined by the kernel of the interpolation filter at the inter-pixel location between samples 128. For example, FIG. 6 shows usage collection 118_iThree samples 128 to calculate the currently processed partition 102_iOf one sample 134. Due to the relatively small average distance 130, the partition 102_iThe number of reference samples 134 per sample 134 may be kept low. More details will be presented below. However, for completeness, it should be noted that the set of available intra-prediction modes may also include a DC mode according to which a DC value is assigned to partition 102_iBy performing a set 118 of neighboring samples on all samples 134_iAveraged to derive the DC value. Furthermore, there may be a planar mode according to which the partition 102 passes_iA linear function at the sample position within to define a prediction value for the sample 134, based on the neighboring samples 118_iTo derive the slope and offset of the linear function. Further, note that neighbor set 118_iMay differ depending on the intra-prediction mode selected for block 80, and may particularly differ between angular and non-angular modes (DC/plane), for example.

For example, in the most advanced JEM decoder, there are 67 intra prediction modes available: of these 65 are angular modes and two are model non-directional textures, DC and planar. A 1D partitioning (referred to as a 1D partitioning mode) mode (i.e., predictor derivation 122 performed for partition 102/112 outlined above and below) according to which block 80 is partitioned/partitioned into partitions along dimension 104, the resulting partitions extending across the full width of block dimension and dimension 104, and being one sample wide or more than one sample wide along direction 104, may be combined with, or in other words, may be implemented using, any of them. As already described in relation to fig. 5, all partitions 102/112 (e.g. coding units CU) of one block 80 use the same associated intra prediction mode of the block 80, thereby avoiding excessive overhead of signalling, since the intra prediction mode 116 only needs to be sent once in the data stream 14 for the block 80.

That is, the prediction 122 can be performed in the same manner as the two-dimensional case outlined in the JEM decoder. However, only one line (whether horizontal or vertical) is calculated for the currently processed partition 102/112, so that the prediction process 122 would adjust accordingly as compared to JEM. In the case of selecting a partition order for traversing partitions in such a way that consecutive partitions are next to each other, the prediction process 122 may correspond to the two-dimensional case of JEM, but only for the first row, i.e. the row closest to the reconstructed/encoded neighborhood. In some cases, both HEVC and JEM allow certain filters to be applied to reference samples 128 or to the resulting predictors. This is useful in the two-dimensional case for better predicting samples within the prediction block 80 that are far away from the reference sample 128 to reduce boundary discontinuities. However, by using a partition to partition 102/112, it is possible and should be a goal to take advantage of the high correlation between nearby pixels.

That is, a reduced average distance 130 should be utilized. Excessive smoothing can degrade the quality. Thus, provided that the encoder or decoder is capable of performing two types of intra prediction, i.e. using partitioned intra prediction as outlined in relation to fig. 4 to 6 and below, the intra filter (i.e. the filter involved in predictor derivation 122) is either disabled or at least the number of contributing samples 134 per partitioned sample 134 is reduced relative to the number of samples contributing to one sample in the two-dimensional case where the intra prediction of the block 80 is performed on or according to HEVC, i.e. decomposed into leaf blocks of a hierarchical quad-tree, subdividing the block 80 into rectangular blocks.

As is clear from the discussion set out above, in order to perform the prediction residual correlation task 124, the decoder decodes, for example, a transform of the corresponding prediction residual of the currently processed partition from the data stream 14 and performs an inverse transform (e.g., a spectral to spatial transform) on the transform to produce a prediction residual, which is then used to correct the predictor obtained at 122 by combining/adding. The same is true in the encoder in order to maintain a prediction loop that is synchronized with the decoder. In addition, for the partition currently being processed, the encoder performs a transform on the prediction error of the predictor determined using 122, transforming the prediction error (e.g., spatial to spectral)Transform), followed by quantization of the transform coefficients, and then encoding of the transform into the data stream 14 to obtain the partition 102 currently being processed_iCorresponding data 120 of_i. With respect to the transformation, all partitions 102/112 within a block 80 may be processed using the same transformation. The transform may be DCT-II except in the case of planar mode, in which DST-VII may be used, for example. For this reason, if the block 80 is coded using intra prediction mode in a partitioning manner outlined so far with respect to fig. 4 to 7 and further below, all tools relating to transform and inverse transform that the encoder and decoder can use for other blocks, such as transform skip (i.e. coding in the spatial domain), EMT (EMT ═ explicit multi-core transform), NSST (NSST ═ mode dependent inseparable secondary transform), or others, may be disabled to avoid unnecessary overhead bits. Even alternatively, the transform may be a linear transform, the type of linear transform being selected based on one or more of an intra prediction mode, a dedicated syntax element, and a predetermined partition order.

Some of the ink has been spent on the partition order 126 that is used to sequentially process the partitions 102/112 of the current processing block 80. It should be emphasized that this embodiment is merely an example, and that according to alternative embodiments, the partitioning order may be static, or may vary in different ways according to the examples set forth below. Fig. 7 indicates a possible partitioning/processing order by using inscribed numbers as indicated by arrow 126 in fig. 5. Here, the order follows the inscribed number in ascending order. Fig. 5 shows an example of the sequence 126 starting with the partition containing the top left pixel/sample 140 of the block 80 down to the lowest partition. Similarly, if the partition type is vertical, the processing order will start again to the right starting from the leftmost partition containing the top left pixel/sample. However, this is not the best case for all existing intra prediction modes. This is illustrated in fig. 7, which also shows that for the diagonal mode, fig. 7 shows the vertical and horizontal division of the block 80, i.e. the copy angle/direction 132 is directed from the lower left side to the upper right side at 45 °, and 34, i.e. the copy angle/direction 132 is directed from the upper left side to the lower right side at-45 °. In the former case, if the partition is horizontal, starting from the upper left corner of block 80, a partition whose reconstructed samples do not have any effect on the prediction of the subsequent partitions will be generated. Therefore, it is more reasonable to start from the lower left corner of the block, so that the reconstructed samples of each partition can be used to predict the next partition in partition order. However, as can be observed in the above figures, in vertical segmentation this is not necessary. On the other hand, mode 34 does not have any of these problems given that the sample is divided horizontally and vertically from both sides. Therefore, a normal processing order can be adopted in both divisions.

Table 1 shows a complete list of processing orders according to intra prediction modes and partition types.

Table 1: according to the processing order of the intra mode and the partition type. HOR _ DIR and VER _ DIR are horizontal and vertical modes, respectively, and VDIA _ DIR is vertical diagonal mode

With reference to fig. 8, the ISP concept described so far with respect to signalling overhead is summarized. Fig. 8 shows the content transmitted for block 80 according to an embodiment of the application. In particular, there is intra prediction mode signaling 116 to signal which intra prediction mode is to be applied to the block 80. Thus, the signalization 116 indicates, for example, one of the angular modes, or one of the available modes including angular mode and non-angular modes (e.g., DC and planar). In addition to this signalization 116, there is a partitioning flag 160, which is encoded by the encoder into the data stream 14 and decoded by the decoder from the data stream 14 for the block 80, indicating whether the partitioning process according to fig. 4 to 7 is applied to the block 80, or whether the block 80 is processed "normally", e.g. in whole blocks or in one or two dimensions, i.e. using only samples outside the block 80 to form the reference sample reserve 118, to predict each sample within the block 80. Alternatively, the flag 160 may switch between the partitioning process discussed with respect to fig. 4-7 on the one hand and the decomposition of the block 80 into transform blocks using quadtree subdivision, and then process these transform blocks sequentially, however the disadvantage is that the decomposition must be signaled in the data stream 14. If the split flag 160 indicatesIllustrating partitioning according to fig. 4, the data stream 14 contains a partition dimension flag 114 for the block 80 that toggles between partition types 100 and 110 discussed with respect to fig. 4. Then, for each partition of block 80 (the partition subdivided/divided by block 80), if the division flag 160 indicates the division option, the data stream 14 includes signaling/data 120₁Signaling/data 120₁With the prediction residuals of the respective partitions coded, for example, into the transform domain as indicated above.

With respect to fig. 8, it should be noted that the prediction residual data 120₁、120₂… … may be encoded into the data stream 14 in an order corresponding to the partitioning/encoding order 126. As described above, the latter can be uniquely determined by the intra prediction mode indicated by the signalization 116. However, an alternative is that the partition order 126 is determined based at least in part on optional additional signalization in the data stream 14.

Another alternative to the description presented herein is that the signalization 116 may alternatively be used in order to indicate the fact whether the partitioning option is used. In other words, one syntax element may share responsibility for the signaling of 116 and 160. Such syntax elements would take one value of a range of values, each value corresponding to a combination of intra prediction mode and an indication of whether block partitioning is used. In this case, the inventors can also provide partitioning options for only a subset of intra prediction modes. Finally, it should be noted that the partition flag 160 may also be conditionally transmitted in the data stream 14 only if the intra-prediction mode indicated by the signalization 116 assumes a certain subset of the available intra-prediction modes.

FIG. 9 exemplarily shows a partition 102/112 having a certain partition_iData 120 of prediction residual_iWhat should look like. According to fig. 9, the prediction residual is coded into the data stream 14 in the transform domain. That is, the encoder generates a transform 182 of the prediction residual by transform 180, and the decoder derives the prediction residual in the spatial domain by inverse transform 184. Fig. 9 shows, for example, transform coefficients 186 of the transform 182 corresponding to different spectral frequencies f. Data 120_iMay include a coded block flag CBF, data 120_iMay include coded block marksCBF 188, CBF 188 indicates whether transform 182 includes any significant transform coefficients 186, i.e., whether transform 182 is completely zero. If CBF 188 is set, transform 182 is not zero and data 120 is_iA Last Position (LP) syntax element 190 may be included that indicates the last position 192 of the significant transform coefficient (i.e., non-zero transform coefficient 186) along the increasing spectral frequencies (see axis 194) starting from the lowest or DC coefficient 196 at 192. Then, the data 120_iSignaling 198 is included to signal the transform coefficients from 196 to 192.

That is, FIG. 9 shows each partition 102_i/112_iIts prediction residual may be encoded into the data stream 14 by CBF 188, LP 190 and transform coefficient data 198. That is, for a block 80 having n partitions 102/112, there are n CBFs 188, one LP 190 for each partition having a non-zero CBF 188, and transform coefficient data 198 only for those partitions having a non-zero CBF 188 associated therewith. Coefficient data 198 may be encoded in the same manner as is done for a normally processed intra-predicted block (i.e., block 80 with partition flag 160 indicating a non-partition option), except that: if the partition is one sample wide, only one coordinate is needed per LP 190 (otherwise 2 coordinates are needed as usual), i.e., x for the horizontal partition 100 and y for the vertical partition 110. However, in the case of two-dimensional partitioning, LP 190 indicates the last position along the scan direction or path, using sequential indications, or using x and y coordinates. The context of each CBF 188 may be selected as the value of the previously encoded CBF, i.e., the CBF of the previous partition in partition order 126. Further, due to the partitioning, the transform coefficient data 198 is associated with different shapes. That is, the transforms 182 also have different shapes. In the case where the partitions are one-dimensional partitions, as discussed with respect to FIG. 4, the transform 182 is a one-dimensional transform. That is, transform 182 may be a W/H long vector of transform coefficients 186, depending on the partition type 100 or 110.

Regarding the flags 160 and 114 of FIG. 8 and their encoding, the following is noted. Flag 160, which indicates whether block 80 is divided into partitions 102/112, defines the condition under which it is checked for block 80 whether flag 114 is transmitted in data stream 14. Specifically, if flag 160 indicates a partition to partition 102/112, flag 114 is present in data stream 14 and sent to the decoder to signal which type of partitioning 100/110, i.e., horizontal or vertical, is to be performed. Just as flag CBF, flag 114 may also be encoded using context-dependent entropy encoding/decoding. Depending on the intra prediction mode of block 80, the context of flag 114 may have three possibilities: 0 for non-angular mode, 1 for horizontal mode, and 2 for vertical mode.

Although fig. 9 shows that the CBF 188 may occur once in each partition i of the current block 80, additionally or alternatively, the transforms 182 of the partition 120i of the current block may each be divided into one or more sub-blocks, signaling for each sub-block within the data 120i an encoded sub-block flag indicating whether the transform coefficients 186 within that sub-block are all zero or at least one coefficient thereof is non-zero. Thus, only the encode sub-block flag signals that coefficients 186 within the sub-block that have non-zero coefficients will be encoded. The coded sub-block flag signals that other coefficients within the sub-block where no non-zero coefficients exist will be inferred to be zero at the decoder side. Note that since each partition 120i is transformed separately, the sub-blocks belonging to a partition differ in the spectral components of the transform 182 of that partition, and in the transform coefficients 186 in the transform that they comprise. For example, the sub-blocks may be set such that only the corresponding partition 102 is needed_i/112_iAre equal to or greater than four samples 140, they are 4 x 4 coefficient blocks, so as long as the corresponding partition 102 is a block of coefficients_i/112_iIs equal to or greater than 4 coefficients 186 in both the x and y dimensions. In the case of 4 × N partitions, the sub-blocks form columns of m 4 × 4 sub-blocks, where m × 4 ═ N and m is an integer. In the case of an N × 4 partition, the sub-blocks form rows of m 4 × 4 sub-blocks, where m × 4 ═ N and m is an integer. For a broader partition, a 4 × 4 array of sub-blocks arranged in rows and columns may be produced. However, such partitions wider than 4 samples and/or as wide as 4 samples may not occur, depending on the embodiment. Whether present or not, for narrower partitions, i.e. with one of their sizes smaller than 4 samplesI.e. less than 4 samples wide in at least one dimension x or y, sub-block partitioning of its transform 180 into sub-blocks may be done, each sub-block collecting a different set of coefficients of the transform 180, such that a sub-block has a minimum number M of coefficients in all possible cases for the size of the current block. That is, partitions may be set as large as the block width N along one dimension, while partitioning is done along the other dimension 104. Thus, the size of the transform 180 for each partition may be 1 × N, 2 × N, N × 01, or N × 12. In fact, the transform 180 for a certain partition may have a number of coefficients equal to the number of samples in that partition. In the case of 1 × N partitioning/transform, the sub-blocks may then form columns of M1 × M sub-blocks, where M × M ═ N and M is an integer. In the case of an N × 1 partition, the sub-blocks form rows of M × 1 sub-blocks, where M × M ═ N and M is an integer. In the case of a 2 xn partition/transform, the sub-blocks may then form a column of M2 × (M/2) sub-blocks, where M × (M/2) ═ N and M is an integer. In the case of an N × 2 partition, the sub-blocks may form rows of M (M/2) × 2 sub-blocks, where M × N (M/2) and M is an integer. For the exemplary case where the minimum number of coefficients M ═ 16, this is exemplarily shown in table 2.

Table 2: entropy coding coefficient group size

Although fig. 9 shows that the CBF 188 may exist once for each partition i of the current block 80, it may be agreed between the decoder and the encoder that at least one of the n partitions among the partitions of the current block 80 has a non-zero CBF 188. Thus, if n is the number of sub-partitions and the first n-1 sub-partitions in coding order have produced zero CBF, the CBF of the nth partition will be inferred to be 1. Therefore, it is not necessary to decode it nor encode it. Thus, the CBF of data 120n will be lost, data 120₁To 120_n-1The CBF in (1) signals non-zero and the decoder may infer that the CBF signals the presence of at least one non-zero coefficient in the transform of the partition.

The following may hold for intra coding mode signaling 116. The coding mode signalization 116 can be sent as a pointer or index to one of the Most Probable Mode (MPM) lists. The latter MPM list can then be determined in the same way by the encoder and decoder based on the intra-prediction mode (e.g., spatially and/or temporally neighboring intra-prediction modes) used for the previously encoded/decoded intra-prediction block. Thus, the MPM list may represent a suitable subset of the available/supported intra prediction modes, i.e. one or more of the aforementioned angular modes and/or DC and planar modes. As described above, in addition to the blocks of the classical intra prediction, there may be intra prediction blocks using the LIP or ISP scheme, such as the block 80 (i.e., the entire block) in the drawing or blocks in units of transform blocks using recursive quadtree partitioning. Both types of intra-prediction blocks may support the same set of available/supported intra-prediction modes. Whereas for the subsequent normal/classical intra-predicted block, an MPM flag (which is decoded by the decoder and encoded by the encoder) may be signaled in the data stream indicating whether the mode of the block is selected from the MPM list, in which case a pointer/index into the MPM list is sent (which is decoded by the decoder and encoded by the encoder), and in case an intra-predicted block using LIP or ISP schemes (e.g. block 80) should be inferred to signal MPM list restriction. If for a certain normal/classical intra-predicted block the MPM flag signals that no MPM mode is used, there is no index/pointer in the data stream for this block, instead a replacement pointer/index pointing to the remaining list of intra-predicted modes is sent in the data stream for this block. The remaining list may also be a suitable subset of the set of available/supported intra prediction modes, and in particular may be a complementary set of MPM lists compared to the set of available/supported intra prediction modes, i.e. each member of the set of available/supported intra prediction modes is either a member of the MPM list or a member of the remaining set. The pointers/indices pointing to the MPM list may be VLC coded, while the pointers/indices pointing to the remaining sets may be coded using fixed length codes. Naturally, even for intra-predicted blocks of the LIP or ISP schemes, the MPM flag is sent and the encoder will freely select any mode from the set of available/supported intra-prediction modes, wherein the MPM flag is set depending on whether the selected mode is in the MPM list or in the remaining set.

The MPM list may be the same for normal/classical intra-predicted blocks and for ISP/LIP intra-predicted blocks, i.e. may be determined by the encoder and the encoder in the same way. However, whether or not applying the restrict to MPM list and inferring MPM flag to signal MPM list usage of ISP/LIP intra-predicted blocks, alternatively, MPM lists may be determined differently for ISP/LIP intra-predicted blocks in order to accommodate statistics of ISP/LIP modes. For example, changes may be made to exclude DC intra-modes from the MPM list and prioritize horizontal intra-modes for ISP horizontal partitioning (i.e., horizontal direction 104) and vertical intra-modes for vertical partitioning (i.e., vertical direction 104). That is, for normal/classical intra-predicted blocks, the MPM list may form a suitable subset of the set of available/supported intra-prediction modes, the modes being selected and ordered according to some concept. For the ISP/LIP intra-predicted block 80, the MPM index may point to an MPM list depending on the partitioning direction 104 signaled by the flag 114 and/or the appropriate subset of no DC mode or no DC and planar modes forming the set of available/supported intra-prediction modes, i.e. the appropriate subset of angular modes in the set of available/supported intra-prediction modes. The MPM list construction based on previously used intra prediction modes of previous encoding/decoding may prefer an angular mode closer to the horizontal dimension in the angular intra prediction directions if the flag 114 indicates that the partition direction 104 is horizontal, and prefer an angular mode closer to the vertical dimension in the angular intra prediction directions if the flag 114 indicates that the partition direction 104 is vertical.

With respect to the description just presented, it is again noted that the juxtaposition between the intra-prediction mode of the normal processing and the intra-prediction mode using the partitioning processing outlined herein need not be so. That is, the encoder and decoder may inevitably process the intra-predicted block 80 using the partitioning proposed herein, and then, accordingly, the partition flag 160 becomes obsolete, for example. However, if the partitioning option signaled by flag 160 is available as a decision for the encoder, the following description reveals how the encoder performs the decision or finds out whether the partitioning mode applies to a certain block 80, and the possibility of which partitioning type (i.e. horizontal or vertical) is the best. To perform this operation, the encoder should test two options for different intra prediction modes for each block. The encoder will therefore be slower because more options have to be tested than if the encoder had only one option (e.g. the normal option). To reduce this effect, the partitioning pattern signaled by the flag 160 may be tested by encoding according to the following strategy, wherein reference is made to fig. 10 and 11.

1) The 1D partition mode is the last intra mode to be tested.

2) When the 1D partition mode is to be tested, set C_minAt the least cost so far.

3) A combination of intra mode and partition type to be tested is selected.

4) A block is partitioned into N1D partitions, and let i denote the index of each of these partitions, where i ═ 1, N.

5) After each partition is encoded, its sub-cost J is calculated_i. Thus, the inventors can know the sum of all the sub-costs available after partition i has been encoded, i.e. the sum of all the sub-costsThis process is described in FIG. 10, so FIG. 10 shows the accumulation of 1D partition sub-costs to obtain the final cost for the entire block.

6) After processing each partition, the expression S is evaluated_i<C_min. If true, the partition continues to be encoded until the end. Otherwise, it can be ensured that the test pattern does not produce the ratio C_minLower RD cost and therefore the process is interrupted, we will proceed with the next combination of intra mode and partition type.

7) If all 1D partitions are all coveredEncoding, then test mode is the best mode and C is updated accordingly_min。

The advantage of this procedure is that processing unnecessary 1D partitions is avoided, since it is well known that the 1D partition mode does not generate a better cost than the already existing minimum cost. Furthermore, it does not have any disadvantages in terms of RD loss. The whole process is shown in the flow chart of fig. 11.

Note again that all of the above ISP examples illustrate the partitioning as being done in one sample wide stripe transverse to direction 104, alternatively the partitioning may be done in a manner that results in a wider partition, resulting in a two-dimensional partition. Further alternatives regarding partitioning are listed below.

In the particular ISP example, the width along the splitting direction 104 is defined based on: 1) whether the intra prediction mode is angular or non-angular, and 2) the width of the intra prediction block along the direction 104.

1) A block 80 of W × H (where W and H are assumed to be powers of 2) may be divided horizontally or vertically (e.g., as indicated by the syntax element 114 sent to the decoder) into K equal partitions 102/112 of size W × H, where the values of W and H are described in table 3. According to table 3, a block with W16 and H8, which is predicted using a non-angular intra mode and divided vertically (i.e., the direction 104 is vertical), is divided into, for example, 4 partitions 102, all of which have a size W16 and H2. If the same block 80 is predicted using angular intra mode, the block 80 is partitioned into 8 partitions 102, each having a size w-16 and h-1.

Table 3: values of w, h and K for additional layout example 1

2) Alternatively, a block 80 of wxh (where W and H are assumed to be powers of 2) may be divided horizontally or vertically (e.g., indicated by a syntax element 114 sent to the decoder) into K equal partitions of size W × H, where the value of K is not fixed (and thus sent to the decoder using the syntax element), which may range to any power of 2 between 2 and S, where S is the value of the divided dimension (width for vertical division, height for horizontal division). The values of w and h were obtained as described in table 4.

Variables of	Vertical segmentation	Horizontal segmentation
			w	W/K	W
h	H	H/K

Table 4: values of w and h for additional layout example 2

Alternatively, the width of a partition along dimension 104 can be signaled directly for block 80.

3) Alternatively, a block 80 of wxh (where W and H are assumed to be powers of 2) may be split horizontally or vertically (e.g., as indicated by syntax element 114 sent to the decoder) into sizes W_i×h_iK partitions (where K depends on W and H), where i ═ 1,2, …, K. If the division is horizontal, let S be H, S_i＝h_iIf vertical, let S equal W, S_i＝w_i. For S, which measures the width of block 80 along dimension 104, and S, which measures the width of partition i along dimension 104_iDifferent values of (a), s is described in Table 5_iVarious options for the value of (c).

Table 5: s for different partition layouts_iValue of (A)

The options used by the decoder are fixed and can also be implicitly determined from the values of the existing parameters at the decoder side.

4) Alternatively, a block 80 of wxh (where W and H are assumed to be powers of 2) may be split horizontally or vertically (e.g., as indicated by syntax element 114 sent to the decoder) into sizes W_i×h_iK partitions (where K depends on W and H), where i ═ 1,2, …, K. If the division is horizontal, let S be H, S_i＝h_iIf vertical, let S equal W, S_i＝w_i。s_iWill be determined by a syntax element indicating which of the three options presented in example 3) is to be used for dividing the block into sub-partitions.

Thus, as illustrated in examples 1 to 4 above, the partitioning may be done along one dimension 104 such that the partitions are as wide as the predetermined block in a direction perpendicular to the predetermined dimension, while the width of the partitions measured along the predetermined dimension 104 is selected from at least two different width settings or options. Explicit or implicit signaling concepts may be used to maintain selective synchronization between the encoder and decoder. Thus, the choice is such that although the partitioning may vary between blocks of the same size and shape, the overhead associated with such variation is kept at a fairly low level. For example, the selection may be done according to the intra coding mode of the predetermined block, e.g. depending on whether the intra coding mode of the predetermined block is an angle mode or not. The selection may also be made according to an index in the data stream for the predetermined block, the index being used to index one of the at least two different width settings shown in example 4. The partition may be one or more samples wide along the partition dimension. Within a block, the partition width along the dividing/predetermined direction may vary. One may be one sample wide, i.e. a one-dimensional stripe, and the other more than one sample wide, a two-dimensional sample field.

In the case of residual coding, as described above, the residual coding may be done using transform coding. Each sub-partition 102/112 may have its own Coded Block Flag (CBF)188, Last Position (LP) syntax element 190, and transform coefficients 198 in the data stream that are to be sent to the decoder. Thus, for a block 80 (e.g., a CU) having K sub-partitions 102/112, there will be K CBFs 188 and one LP 190 for each partition 102/112 having a non-zero CBF. The context used to encode each CBF 188 may depend on the values of the CBFs of previously encoded sub-partitions within the same block (e.g., along the order 126). In addition, another syntax element not yet mentioned may also be sent to the decoder in the data stream to indicate whether to use the ISP concept for each block, or to indicate whether to use the partitioned intra prediction concepts described herein at a range corresponding to the entire data stream or a certain picture or for a slice of a certain picture, e.g., whether to use the partitioned intra prediction concepts described herein on each intra prediction block 80 within the range, or whether to signal processing of some intra prediction blocks as a slice, i.e., as if only partitioned into one partition.

Also, it has been described above that each sub-partition may be transformed separately using one transform, thereby obtaining one transform for each partition 102/112 that is not quantized to all zeros. As a transformation for a certain partition 102/112, a 2-D transformation may be used, except in the case where one of the dimensions of the partition 102/112 is 1, in which case a 1-D transformation would be applied. The transform kernel may be DCT-II or any other transform determined by existing parameters at the decoder side when the sub-partition is to be decoded. For example, the transform may be selected according to a combination of intra mode, sub-partition index, and sub-partition dimension or some subset of the latter parameters. The transform may also be signaled directly to the decoder, or in other words, in the form of an additional syntax element sent, for example, for all partitions within a block 80 or separately for each partition 102/112 of a block 80.

One aspect that has also been discussed above is the fact that: the residual of partition 102/112 of block 80 may be further transformed, lossless or (although) reversible, by performing a partition-specific transform on the prediction residual of each partition, and then quantizing the transform coefficients to achieve quantization in the spatial or some intermediate transform domain. Thus, the decoder will be able to obtain transformed transform coefficient levels for the entire block 80, inverse lossless transforming the transform coefficient levels to obtain, for each partition 102/112, a prediction residual in the spatial domain or in the intermediate transform domain, obtained by re-transforming each partition 102/112.

Furthermore, it has been discussed above that the prediction residuals of the respective partitions of the block 80 are quantized and encoded into the data stream sequentially, partition by partition, in an alternative way with respect to the respective intra prediction of these partitions. This has been described with respect to fig. 5 and with respect to fig. 10 and 11. However, the decoder need not perform decoding by alternating between: the residual decoding on the one hand and the reconstruction of the individual partitions by combining the intra prediction on the one hand and the prediction residual on the other hand. That is, when decoding a certain block 80, the decoder may separate the decoding of the prediction residuals 120 of the respective partitions from the actual reconstruction process of the respective intra prediction including the respective partitions. Referring to fig. 5: the decoder may decode the prediction residuals 120 of all partitions, i.e. the prediction residuals 120 of the block 80, from the data stream 14 according to one processing task₁To 120₄And the decoder may use the partition 102 according to another task_iIs predicted to be the residual 120_iTo reconstruct the interior of the block 80 partition by partition according to the partition order 126. To this end, the decoder uses the intra prediction mode of block 80 for each partition 120 in the second task_iPerforms intra prediction and then adds the prediction residual 120 obtained from the first task_iAnd then proceeds to the next partition 120 by following partition order 126_i+1The reconstruction of block 80 is continued to perform spatial prediction and then reconstructed using the prediction residual of the partition to correct the intra prediction result. The decoder may perform the first task of deriving the prediction residual 120 from the data stream 14 completely before starting the second task of prediction and prediction correction using the prediction residual, or the decoder may perform both tasks in parallel while providing measures to ensure thatA certain partition 102_iIs predicted to be the residual 120_iReady when needed, i.e. for that partition 102_iWhen the prediction result of (2) has been obtained using the intra prediction mode of the block and correction is required. In particular, during a first task or phase, the decoder may partition all non-zeros in parallel (i.e., signal the prediction residual 120)_iAll partitions 102 that are non-zero) perform all inverse transforms.

In one aspect, note that when partitioning i, the residual 120 is_iWhen quantized in the transform domain, it may happen that the reconstructed samples of these partitions may deviate (i.e. exceed or inherit) from some allowed range of sample values. As described above, they may be used as members of the reference sample 118j for subsequent partitions j in the order 126. According to the ISP variant, these samples are left as they are in order to predict the subsequent partition j in the order 126, while clipping of these samples of the block 80 is performed as a final clipping step of the entire block 80, thereby increasing the implementation friendliness, for example, at the decoder side. Thus, in the export partition 102_iThe predictor according to the partition order 126 can be used in the partition 102_iOne or more reconstructed samples 118 of a previous partition used as a reference for a current partition_iWherein after performing the sequential reconstruction, a state is finally performed in which the reconstructed samples are clipped from the not-clipped state to an allowable sample value range to finally reconstruct the predetermined block. At the encoder side, clipping is performed only to obtain a reconstructed version of such samples to be used as a prediction reference for a subsequent encoded block, in order to maintain reference synchronization with the decoder. However, this final clean-up type of clipping is merely an example, and alternatively clipping may be performed immediately, i.e., before reconstructed samples of partition i are used as reference samples 118j of partition j for subsequent processing.

An ISP example is illustrated in detail below. In particular, according to this example, for intra-coded block 80, data stream 14 is signaled by partition mode flag 160 whether it is coded using the ISP scheme. The corresponding syntax element in the data stream 14 may be named intra _ sub _ modes _ flag. For example, if the flag is 1, then the intra-coded block 80 may be encoded using the LIP or ISP scheme, otherwise the block 80 is encoded using normal intra prediction. For example, an LIP or ISP scheme may be used for the current intra-coded block 80 only if a certain condition or conditions are met. For example, the one or more conditions may include: intra-coded block 80 may need to be larger than some minimum size, e.g., in terms of the number of samples of block 80, and/or intra-coded block 80 may not be allowed to exceed a certain size (at least not allowed in both the horizontal and vertical directions), e.g., in order not to cause too large a transform size. More specifically, the LSP or ISP mode may only be available if block 80 is less than or equal to the just-mentioned maximum transform correlation size in at least one direction (i.e., horizontal or vertical). Thus, intra _ sub _ modes _ flag may only be present in the data stream if block 80 satisfies the conditions just mentioned. Otherwise, the decoder may conclude that intra-coded block 80 is normally intra-coded. In the case where the partition mode flag intra _ sub _ partitions _ mode _ flag indicates that the intra-coded block 80 is an LSP or ISP-coded block, a partition dimension flag 114 may be further signaled for the intra-coded block 80. However, it may not be inevitable to signal the intra _ sub _ modes _ flag explicitly, but in some cases is inferred to indicate a certain partition dimension 104. For example, in the case where the width of intra-coded block 80 exceeds the maximum transform size described above (but the height does not exceed), then partition dimension 104 may have to be horizontal, and in the case where the height of block 80 exceeds the just-mentioned maximum transform size (but the width does not exceed), then dimension 104 may have to be vertical. In both cases, the intra _ sub _ partitions _ split _ flag is not explicitly signaled in the data stream, but is inferred accordingly by the decoder. The intra-coding mode 116 may be signaled in the data stream as described above, i.e. by using the most probable intra-prediction mode list constructed at the encoder and decoder side. For LIP or ISP intra-coded blocks 80, however, data stream 14 may signal the intra-coding mode via an MPM list pointer, e.g., referred to as intra _ luma _ MPM _ IDX, which inevitably points to a list of possible intra-prediction modes, which may be preceded in data stream 14 by an MPM flag in the case of intra-coded blocks not coded with the ISP scheme. If the MPM flag (e.g., referred to as intra _ luma _ MPM _ flag) has a flag state other than the pointer to the most probable intra prediction mode list, pointers to the remaining list of intra prediction modes are signaled in the data stream. However, as mentioned above, this is only an example, the set of intra prediction modes that may be signaled may be the same for both the normally encoded intra prediction block and the ISP intra prediction block, i.e. covering all supported intra prediction modes. For example, intra _ luma _ mpm _ flag may be sent for two types of intra-coded blocks. Alternatively, the pointers sent for both types of intra-predicted blocks can point directly to the complete list of supported intra-prediction modes without any MPM flags for both types of intra-coded blocks. If intra-coded block 80 is encoded using the ISP scheme, the number of partitions 102/112 may be defined as follows. Specifically, the encoder and decoder may determine the number of partitions according to the size of the block 80. No signal is consumed in the data stream. For smaller block sizes, the number may be 2, otherwise the number of partitions 102/112 is 4. The partition order in which the intra prediction of partitions and the encoding of prediction residuals in the data stream are performed may point sequentially from the leftmost partition to the farthest partition along the partition direction 104 in the case of the horizontal direction n14 and from the uppermost partition to the farthest partition in the case of the vertical partition direction. Nor does it consume signaling for this purpose. As described above, the residual transform may be done for each partition 102/112. That is, each partition may be transformed separately. In contrast, in the case of normal intra-coded blocks 80, the number of transforms may depend on the size of intra-coded block 80 as follows: if the intra-coded block is smaller than the above-mentioned maximum transform size in the horizontal and vertical directions, the residual of intra-coded block 80 is coded using one transform, i.e. the residual of block 80 is completely transformed by one transform. In the case where the maximum transform size is exceeded in the horizontal direction, intra-coded block 80 is divided into two halves or a corresponding number of transform blocks in the horizontal direction such that the two halves or transform blocks satisfy the maximum transform size, and one transform is performed per half/transform block for the residual of block 80. The same applies if the block 80 exceeds the maximum transform size in the vertical direction. If the maximum transform size is exceeded in both the vertical and horizontal directions, then the residual of block 80 is transformed into a corresponding number of transformed blocks, using four or a corresponding number of transforms, in four quadrants of the block 80 or a regular two-dimensional subdivision of the block 80. Further, the processing of normal intra-coded blocks 80 may deviate from LIP or ISP-coded intra-coded blocks 80 in the following respects: normal intra coded blocks are intra predicted in whole blocks. That is, the normal intra-coded blocks are not subdivided. A further difference may relate to the coding used in the transform to encode the prediction residual of block 80. For each transform, a coded block flag 188 may be sent, e.g., tu _ CBF _ luma, but for a normal intra-coded block 80, this flag may be coded inevitably for each transform within block 80, which may be inferred to be 1 for the last transform of block 80 in the case of ISB coding of block 80 and all previous CBFs of previous transforms being zero. Furthermore, the choice of the size of the sub-blocks within each transform may differ between normal intra-coded blocks 80 on the one hand and ISP-coded blocks 80 on the other hand. The details are as above. However, the subdivision of the transform 182 into sub-blocks may alternatively be done identically for normal intra-coded blocks and ISP-coded blocks. For example, let log2SbW and log2SbH be the log-pair of the subblock width and height, respectively, and log2TbWidth and log2TbHeight be the width and height of the transform, respectively. The sub-block size may then be determined as follows:

the above pseudo-code obtains sub-blocks of sizes listed in table 2. Due to the inherent minimum size of intra-coded block 80 and the non-subdivision of normal intra-coded blocks, only sub-blocks of 4 x 4 coefficients are generated for normal intra-coded block 80. Finally, it should be noted that the example just outlined may result in ISP intra-predicted blocks of different sizes, including blocks that are divided into only two partitions 102/112, but regardless of whether such ISP intra-predicted blocks are present, there are ISP intra-predicted blocks that are divided into more than two partitions.

This is the starting point for the description of the embodiment efficiency improvement embodiments presented below. Note that its description is proposed as an alternative to handling ISP blocks of certain sizes (e.g. 4 x 4, 8 x 4, 4 x 8 blocks) while handling larger ISP blocks as described above, but even further alternative embodiments may be realized by transferring the following partition-based intra coding concept to a block-based codec without the ISP processing of blocks outlined so far.

Fig. 12 shows a decoder 20 for block-based decoding of pictures from the data stream 14 according to an embodiment. Accordingly, the predetermined block 80 may be decoded, and thus the predetermined block may be divided into sub-blocks, which may be understood as the transform partition 300. The data stream 14 may include an encoded intra coding mode 116, an encoded partition dimension flag 114, and several encoded transforms 120 of the prediction residual. For the predetermined block 80, an intra coding mode 116 and a partition dimension flag 114 are signaled, for example. Instead, the transform 120 of the prediction residual is signaled, for example, for each transform partition of the predetermined block 80.

The decoder 20 is configured to decode a partition dimension flag 114 of a predetermined block 80 of pictures from the data stream 14 and set the partition dimension 104 to the level 104 according to the partition dimension flag 114₁Or vertical 104₂. For example, the partition dimension 104 indicates a direction along which the decoder 20 is to partition 105 the predetermined block 80.

According to an embodiment, the decoder 20 is configured to decode the partition dimension flag 114 by using context-dependent entropy decoding using a context that depends on the intra coding mode 116. According to an embodiment, the decoder 20 is configured to decode the partition dimension flag 114 by using context dependent entropy decoding using one of three contexts including: an intra-coding mode signaling a non-angle mode, an intra-coding mode signaling a horizontal mode, and an intra-coding mode signaling a vertical mode.

If dimension 104 is divided₁Horizontal, the predetermined block 80 is divided or partitioned into vertical transform partitions 300, for example_1bTo 300_6bAcross a predetermined block 80₁The entire vertical height of (a). Alternatively, if the partition dimension 1042 is vertical, the predetermined block is divided or partitioned into horizontal transform partitions 300, for example_1aTo 300_4aAcross a predetermined block 80₂The entire horizontal width of (a). In other words, the decoder is configured to be along the predetermined dimension 104₁Or 104₂Dividing the predetermined block 80 into transform partitions 300, the transform partitions being aligned perpendicular to the predetermined dimension 104₁Or 104₂Direction 103 of₁Or 103₂As wide as the predetermined block 80. The predetermined dimension is, for example, a partition dimension 104.

The number of transform partitions 300 is, for example, greater than 2 and/or the transform partitions 300 are along the predetermined dimension 104₁Or 104₂Is one sample wide 101₁Or 101₂. Alternatively, transform partition 300 may be along predetermined dimension 104₁Or 104₂Two or more samples wide.

According to an embodiment, the decoder 20 is configured to set the width 101 of the transform partition 300 measured along the predetermined dimension 104 depending on: a size 101a of the predetermined block 80 along the predetermined dimension 104, and/or an intra-coding mode 116 of the predetermined block 80, and/or whether the intra-coding mode 116 of the predetermined block 80 is an angular mode.

According to an embodiment, the predetermined block 80 of width W and height H, i.e. W × H size, is for example divided horizontally or vertically into K transform partitions of the same size, each transform partition having a size W × H/K for horizontal division (along the vertical division dimension) and a size W/K × H for vertical division (along the horizontal division dimension).

The decoder 20 is configured to reconstruct 123 the predetermined block 80 based on: reconstructed samples 118 adjacent to the predetermined block 80, intra coding mode 116, and transform 120 of the prediction residual.

Thus, the decoder is configured to decode a transform 120 of the prediction residual from the data stream for each transform partition. According to an embodiment, the decoder 20 is configured to decode the transform 120 from the data stream by decoding the coded transform partition flag 188 from the data stream for each partition. If the coded transform partition flag 188 is not set, the decoder 20 is configured to set the prediction residuals of the corresponding transform partition 300 to zero, and if the coded transform partition flag is set, the decoder 20 is configured to decode the transform coefficients of the transform 120 of the prediction residuals of the corresponding transform partition from the data stream. For example, if the transcoding partition flag 188 is zero, the transcoding partition flag 188 is not set.

According to the embodiment shown in fig. 12, for each transform partition 300, a coded transform partition flag 188 is optionally coded in the data stream 14, based on the coded transform partition flag 188, the decoder 20 is configured to either decode the transform 120 of the prediction residual, or infer the prediction residual to be zero and the decoder 20 does not have to decode the transform 120 for that transform partition. Alternatively, the decoder does not decode the coded transform partition flag 188, but rather, the decoder is configured to decode the transform 120 of the prediction residual directly or one transform of the entire predetermined block 80 for each transform partition 300.

According to an embodiment, the decoder 20 is configured to decode the transcoding partition flags 188 of the transform partition 300 sequentially from the data stream 14 and to infer the last transform partition (e.g., transform partition 300) in the partition order 210 if all previous transcoding partition flags 188 are not set_4aOr transform partition 300_6b) The transcoding partition flag 188 is set.

According to an embodiment, the decoder 20 is configured to decode a respective transform partition (e.g. the second transform partition 300) from the data stream 14 by using context dependent entropy decoding_2aOr 300_2b) Coded transform partition flag 188₂Context-dependent entropy decoding uses contexts that depend on the respective transform partition in accordance with a predetermined transform partition order 210A previous transform partition before a zone (e.g., first transform partition 300)_1aOr 300_1b) Decoded transcoding partition flag 188₁。

Furthermore, the decoder 20 is configured to decode the intra coding mode 116 of the predetermined block 80 of pictures from the data stream 14.

The decoder 20 is configured to intra-predict 122 the predetermined block 80 from one or more reconstructed samples 118 adjacent to the predetermined block 80 in a manner dependent on the intra-coding mode 116 to obtain a predictor of the predetermined block 80. According to an embodiment, the decoder 20 is configured to intra-predict 122 the predictor for each transform partition 300.

According to a first option, the transform partitions 300 are sequentially reconstructed by the decoder 20. Thus, the decoder is configured to partition the first transform 300_1aOr 300_1bIs intra predicted 122₁And use partition 300 for the first transform_1aOr 300_1bTransform 120 of decoded prediction residual₁To correct the predictor and then partition 300 for the second transform_2aOr 3002_bIs intra predicted 122₂And use partition 300 for the second transform_2aOr 300_2bTransform 120 of decoded prediction residual₂To correct the predictor. According to the divided predetermined blocks 80 shown in fig. 12₁And 80₂Block 80₂Subsequent transformation partition 300_3aAnd 300_4aAnd a block 80₁Subsequent transformation partition 300_3b、300_4b、300_5bAnd 300_6bAre intra predicted and corrected accordingly.

According to a second option, the transform partition 300 is reconstructed in one step. Thus, the decoder 20 is configured to decode a transform 120 of the prediction residual from the data stream 14 for each transform partition 300 and intra-predict 122 the predictor for each transform partition 300. For example, the decoding of the transform 120 is performed independently of intra prediction. According to an embodiment, at the reconstruction 123 of the predetermined block 80, all transforms 120 of the prediction residual are decoded by the decoder 20 and all predictors are intra predicted by the decoder 20. In other words, at the reconstruction 123 of the predetermined block 80, for each transform partition 300, the transform 120 of the prediction residual decoded for the respective transform partition 300 and the predictor associated with the respective transform partition 300 are available.

According to an embodiment, the decoder 20 is configured to, for each transform partition 300, retransform the transform 120 into the spatial domain for correcting the predictors within the respective transform partition 300. In other words, the inverse transform may be applied to the transform coefficients of the transform 120 of the prediction residual to obtain the prediction residual in the spatial domain.

According to an embodiment, in case the intra prediction mode (i.e. intra coding mode 116) is not planar mode, the transform 120 is a DCT transform, and in case the intra prediction mode is planar mode, the transform 120 is a DST transform, or is a linear transform, the type of linear transform being selected based on the intra prediction mode, the block size of the predetermined block 80 and/or the dedicated syntax elements. Thus, the decoder is for example configured to determine a transform type of the transform 120 of the prediction residual based on the intra coding mode. For the above re-transformation, the decoder uses, for example, an inverse transform type.

According to an embodiment, the decoder 20 is configured to decode the partition mode flag 160 of a predetermined block 80 of a picture from the data stream 14. If partition mode flag 160 indicates the first partition mode, decoder 20 is configured to perform, for each transform partition, decoding of partition dimension flag 114, decoding of partition 105, and decoding of transform 120. If the partition mode flag 160 indicates the second partition mode, the decoder 20 is configured to decode one transform 120 of the prediction residual within the predetermined block 80, instead of performing the decoding of the partition dimension flag 114, the decoding of the partition 105, and the transform 120 for each transform partition 300. In other words, the first partition mode indicates that the predetermined block 80 is decoded based on the transform partition, and the second partition mode indicates that the entire predetermined block 80 is decoded without the partition 105. Thus, in the second partition mode, decoder 20 does not use any transform partitions and, for example, does not encode partition dimension flags 114, coded transform partition flags 188, and individual transforms 120 of the prediction residual associated with the transform partitions in data stream 14 for predetermined blocks 80.

Fig. 13 shows an embodiment of a transform 120 capable of decoding the prediction residual of a transform partition 300 from a data stream 14, performed by the decoder shown in fig. 12. According to an embodiment, the decoder 20 is configured to target a predetermined transform partition 300_3aOr 300_3b(this embodiment is shown for predetermined block 80₂Along the vertical division dimension 104₂Divided, for predetermined blocks 80₁Along the horizontal division dimension 104₁Partitioned) by decoding the last position indication 190 from the data stream 14₃To decode a predetermined transform partition 300 from a data stream 14_3aOr 300_3b120 of the prediction residual₃Last position indication 190₃Indicating a transformation 120₃The predetermined scan order 193 scans the one-dimensional transform 120 along the last transform coefficient position 191 of the predetermined scan order 193₃Of the transform coefficients 120₃₁To 120₃₆(and transform partition 300_3aAssociated) or 120₃₁To 120₃₄(and transform partition 300_3bAssociated). In addition, the decoder 20 is configured to partition 300 for predetermined transforms_3aOr 300_3bDecoding a predetermined transform partition 300 from a data stream 14 by_3aOr 300_3b120 of the prediction residual₃: decoding transforms 120 from data stream 14₃Up to the last transform coefficient position 191 along a predetermined scan order 193 of the transform coefficients 120₃₁To 120₃₃(and transform partition 300_3bAssociated) or 120₃₁To 120₃₃(and transform partition 300_3bAssociated) and infer the transform coefficient 120 of the transform 120 that exceeds the last transform coefficient position 191 along the predetermined scan order 193₃₄To 120₃₆(and transform partition 300_3aAssociated) or 120₃₄(and transform partition 300_3bAssociated) is zero.

According to an embodiment, the last position indication 190 may be encoded in the data stream 14 in addition to the transcoding partition flag 188, such as described with respect to fig. 9. Alternatively, only the last position indication 190 is encoded in the data stream 14, and the transcoding partition flag 188 is not encoded.

According to an embodiment, transform partition 300 is one sample wide along predetermined dimension 104, and transform 120 is a one-dimensional transform.

Fig. 14 shows an embodiment of the encoder 10 for block-based encoding of a picture into a data stream 14, the encoder 10 being configured to encode a partition dimension flag 114 of a predetermined block 80 of the picture into the data stream 14, the partition dimension flag 114 signaling that the partition dimension 104 is to be set to the level 104₁Or vertical 104₂. The encoder 10 determines 200 whether the predetermined block 80 should be divided, and if the predetermined block 80 is to be divided, decides which division dimension 104 should be selected, for example, based on the block size of the predetermined block 80 and/or one or more division decisions 200 of the encoder on one or more previously encoded blocks of the picture.

The encoder 10 is configured to divide 105 the predetermined block 80 along a predetermined dimension 104 into transform partitions 300, the transform partitions 300 being as wide 103 as the predetermined block in a direction perpendicular to the predetermined dimension. The predetermined dimension is, for example, a partition dimension 104. Thus, the encoder 10 is configured if the dimensions 104 are divided₁Is horizontal, then vertical partitioning is performed, and if the dimensions 104 are divided₂If vertical, horizontal segmentation is performed. In the context of the decoder, optionally, the partitioning is performed as described in fig. 12 and 13 and/or as described with respect to fig. 15 to 18 in the context of the decoder.

The encoder 10 is configured to intra-predict the predetermined block 80 from one or more reconstructed samples 118 adjacent to the predetermined block 80 in a manner dependent on the intra-coding mode 116 to obtain a predictor of the predetermined block 80. This is performed, for example, by prediction module 44 of prediction stage 36, with prediction stage 36 generating prediction signal 26 using intra-prediction mode 116. According to an embodiment, the encoder 10 is configured to perform intra prediction 300 for each transform partition of the predetermined block separately. Transform partition 300 is, for example, sequentially intra-predicted, transformed 28, quantized 32, and encoded 34 into data stream 14.

According to an embodiment, the encoder 10 is configured to encode the intra coding mode 116 of the predetermined block 80 of the picture into the data stream 14. The encoder 10 is configured to use, for example, the same intra coding mode 116 for the complete predetermined block 80 to obtain a prediction residual 24 for each transform partition of the predetermined block 80, which prediction residual 24 may be transformed 28, quantized 32 and encoded 34 into the data stream 14.

The encoder 10 is configured to encode, for each transform partition 300, a transform 120 of the prediction residues into the data stream 14, such that the predictor within each transform partition 300 is corrected by using the transform 120 of the prediction residues encoded for the respective transform partition 300 to reconstruct the predetermined block 80. In other words, the encoder 10 is for example configured to transform 28, for each transform partition 300, the prediction residuals 24 in the respective transform partition 300 into the spectral domain for correcting the predictors within the respective transform partition 300.

According to an embodiment, the number of transform partitions is larger than 2 and/or the transform partitions are one sample wide along the predetermined dimension.

According to an embodiment, the encoder 10 is configured to encode the transform 120 into the data stream 14 for each partition 300 by encoding the coded transform partition flag 188 into the data stream 14. If the coded transform partition flag 188 is not set, the coded transform partition flag 188 is selected to signal that the prediction residual 24 of the corresponding transform partition 300 is zero, and if the coded transform partition flag 188 is set, the encoder is configured to encode the transform coefficients of the transform 120 of the prediction residual 24 of the corresponding transform partition 300 into the data stream 14. For example, if the transcoding partition flag 188 is zero, the transcoding partition flag 188 is not set.

According to an embodiment, the encoder 10 is configured to transform partitions 300 except the last transform partition in the transform partition order 210 (e.g., for vertical partitioning_8bOr for horizontal partitioning, transform partition 300_4b) Coded transform partition flag of (e.g., for vertical partitioning, coded transform partition flag 188)₈Or for horizontal segmentation, the transcoding partition flag 188₄) In addition, the transcoding partition flags 188 of the transform partition 300 are sequentially encoded into the data stream 14 if all previous transcoding partition flags (e.g., for vertical partitioning, all previous transcoding partition flags) were presentCode transform partition flag 188₁To 188₇Or for horizontal partitioning, all previous transcoding partition flags 188₁To 188₃) None is set, it is inferred that the transcoding partition flag for the last transform partition is set.

According to an embodiment, the encoder 10 is configured to encode the coded transform partition flags 188 of the respective transform partition 300 into the data stream 14 by using context-dependent entropy coding, which uses a context that depends on the coded transform partition flag 188 encoded for a previous transform partition 300 preceding the respective transform partition 300 in the predetermined transform partition order 210.

According to an embodiment, the encoder 10 is configured to encode the transform 120 of the prediction residual of the predetermined partition 300 into the data stream 14 by encoding a last position indication 190 into the data stream 14 for the predetermined transform partition 300, the last position indication 190 indicating a last transform coefficient position of the transform along a predetermined scan order, the predetermined scan order scanning the transform coefficients of the one-dimensional transform. Furthermore, the encoder is configured to encode, for a predetermined transform partition 300, a transform 120 of the prediction residual of the predetermined partition 300 into the data stream 14 by encoding into the data stream, for the transform partition 300, transform coefficients along a predetermined scan order up to a last transform coefficient position in the transform, wherein the transform coefficients exceeding the last transform coefficient position along the predetermined scan order in the transform are inferred to be zero. This may be performed similarly to the decoder, as described in fig. 13.

For example, the transform partition is one sample wide along the predetermined dimension 104, and the transform is a one-dimensional transform.

For example, in the case where the intra prediction mode 116 is not the planar mode, the transform is a DCT transform, and in the case where the intra prediction mode 116 is the planar mode, the transform is a DST transform. Alternatively, the transform is a linear transform, the type of linear transform being selected based on the intra prediction mode 116, the block size of the predetermined block 80, and/or the dedicated syntax element.

According to an embodiment, the encoder 10 is configured to encode the partition mode flag 160 of the predetermined block 80 of the picture into the data stream 14. If the partition mode flag indicates the first partition mode, the encoder is configured to perform the encoding of the partition dimension flag 114, the partitioning, and the encoding of the transforms 120 for each transform partition 300. If the partition mode flag indicates the second partition mode, the encoder is configured to encode one transform 120 of the prediction residual within the predetermined block without performing the encoding of the partition dimension flag 114, the partitioning, and the encoding of the transform 120 of each transform partition 300.

According to an embodiment, the encoder is configured to encode the partition dimension flag 114 by using context dependent entropy coding, the context depending on the intra coding mode.

According to an embodiment, the encoder is configured to encode the partition dimension flag 114 by using context-dependent entropy coding using a context comprising one of the following three contexts: the non-angular mode intra-coding mode 116 is signaled, the horizontal mode intra-coding mode 116 is signaled, and the vertical mode intra-coding mode 116 is signaled.

According to an embodiment, the encoder is configured to set the width of the transform block 300 measured along the predetermined dimension 104 depending on: a size of the predetermined block 80 along the predetermined dimension 104, and/or an intra-coding mode 116 of the predetermined block 80, and/or whether the intra-coding mode 116 of the predetermined block 80 is an angular mode.

The modification of the extension of intra sub-partition (ISP) coding modes to the number of sub-partitions of 4 x 4 and 4 x 8 and 8 x 4 blocks is described below and is motivated. To maintain a worst case throughput of 16 samples/cycle, there are sub-partitions that are independent of each other. The experimental results show that for the AI configuration, the gains for CTC and F classes are 0.1% and 0.47%, respectively, whereas in the RA case, the gains for CTC and F classes are 0.01% and 0.25%, respectively. The impact on the coding run time is 102% in the case of AI and 100% in the case of RA.

Additional test results are provided regarding the impact of extending the ISP concept to all ISP blocks that produce sub-partitions less than 4 wide. The experimental results show that the gains for CTCs and class F configured for AI are 0.05% and 0.44%, respectively, and in the case of RA, the gain for class F is 0.24% (CTCs not changed).

On the other hand, it has been tested to completely remove the effect of sub-partitions with widths less than 4 for current ISP designs. This additional information shows that for the AI configuration, the losses for CTC and F classes are 0.14% and 0.31%, respectively, whereas in the RA case, the losses for CTC and F classes are 0.04% and 0.23%, respectively.

Brief introduction to 1

As proposed in [1], the intra sub-partition (ISP) coding mode divides the luma intra prediction block vertically or horizontally into 2 or 4 equal-sized sub-partitions according to the CU size. Table 6 shows different possibilities.

Table 6: number of sub-partitions created by ISP according to CU size in current VVC draft

Each sub-partition is predicted, transformed, quantized, and the entropy encoded coefficients are sent to a decoder. The reconstructed samples of the sub-partition are then used to generate a prediction for the next sub-partition, but the intra mode used is shared among all sub-partitions.

Since this process is located in the intra-prediction critical path (generally, sub-partitions cannot be decoded before obtaining the previous sub-partition reconstructed samples), it is necessary to force a minimum of 16 samples in each sub-partition, which guarantees a worst-case throughput of 16 samples/cycle. This is why, in essence, 4 x 8 and 8 x 4 blocks have two sub-partitions (instead of four), whereas 4 x 4 blocks cannot be subdivided.

The following description proposes an extension of the ISP allowing use on 4 x 4 blocks and modifying the number of sub-partitions in the case of 4 x 8 and 8 x 4 blocks while maintaining a throughput limit of 16 samples/cycle. This object can be achieved by generating sub-partitions independent of each other, i.e. not using reconstructed samples of the sub-partitions to predict the next sub-partition.

The division and reconstruction of the predetermined block 80 as described below, for example, with respect to fig. 15 to 18, for 4 × 8 and 8 × 4 blocks, may be performed by a decoder as described with respect to fig. 12 and 13 or by an encoder as described with respect to fig. 14.

Note that the subsequent presentation of specific modifications to the ISP represents only a proposition of possible embodiments, and variants thereof are therefore readily available. For example, less complex or more straightforward variations are listed in section 6 below.

2 ISP expansion to New Block sizes

ISP modifications introduced in this contribution affect 4 x 4, 8 x 4 and 4 x 8 blocks.

2.14 × 4 block

According to the embodiment shown in fig. 15 and 16, the predetermined block 80 is divided into four 4 × 1 (horizontal division) or 1 × 4 (vertical division) partitions. However, the prediction signal 122 for each of them is generated using the neighboring samples 118 of the CU boundary as reference samples. Thus, each sub-partition (i.e., transform partition 300) is independent of the other, and all sub-partitions can be processed in parallel in a single step. In other words, and using the terminology applied above repeatedly, the block 80 is not divided or reserved as one partition 112, and the reference samples 118 used for intra prediction or prediction derivation are all located outside the block 80. Instead, the partitioning is applied to transform-based residual coding as described below. Here, one partition 112 (i.e., block) is divided into transform partitions 300, each of which is transformed separately, resulting in a corresponding transform 182 encoded in the manner described above with respect to data structure 120.

Fig. 15a to 15d show an embodiment of vertical partitioning of a 4 x 4 block, i.e. partitioning along the horizontal partitioning dimension. The reference samples 118 used to generate each 1 x 4 prediction are only CU boundary samples. Note that the prediction is the same as the non-ISP case.

Note that the prediction samples for all transform partitions 300 are the same as those generated for blocks that do not use an ISP (not including the reference samples and PDPC filtering operations disabled for all ISP blocks). The difference between 4 x 4 blocks using ISP and 4 x 4 blocks without ISP is the transformation (there are four 1-D transformations in the ISP case and a single 4 x 4 transformation in the non-ISP case) and entropy coding of the coefficients.

2.28 × 4 and 4 × 8 blocks

According to the embodiment shown in FIG. 17 or FIG. 18, the ISP design is changed so that the number of transform partitions 300 is 4, where the sub-partition 112₁And 112₂The number of (2). Each sub-partition 112₁And 112₂Subdivided into two transform partitions 300. The prediction signals 122 for transform partitions 2 and 4 cannot be generated using reconstructed samples for transform partitions 1 and 3, respectively, as shown in the embodiments of fig. 17 or 18. Thus, transform partition 2 is independent of transform partition 1, and similarly, transform partition 4 is independent of transform partition 3. Thus, a 4 × 8 or 8 × 4 block can be processed in 2 cycles, which corresponds to a throughput of 16 samples/cycle. The figure shows the situation for this horizontal and vertical division.

Fig. 17 shows an embodiment of vertical division of the predetermined block 80. The predetermined block 80 is, for example, a 4 xM block divided into four 1 xM transform partitions 300, where M ≧ 8, an 8 xN block divided into four 2 xN transform partitions 300, where N ≧ 4, or a 16 xO block divided into four 4 xO transform partitions 300, where O ≧ 1. Reconstructed samples from transform partition 2 may be used to predict transform partition 3, but reconstructed samples from transform partitions 1 and 3 may not be used to predict transform partitions 2 and 4, respectively. Vertical partitioning corresponds to partitioning along a horizontal partitioning dimension.

Fig. 18 shows an embodiment of horizontal division of the predetermined block 80. The predetermined block 80 is, for example, an Mx 4 block divided into four Mx 1 transform blocks, where M ≧ 8, an Nx 8 block divided into four Nx 2 transform partitions 300, where N ≧ 4, or an Ox 16 block divided into four Ox 4 transform partitions 300, where O ≧ 4. Reconstructed samples from transform partition 2 may be used to predict transform partition 3, but reconstructed samples from transform partitions 1 and 3 may not be used to predict transform partitions 2 and 4, respectively. Horizontal partitioning corresponds to partitioning along a vertical partitioning dimension.

As shown in fig. 17 and 18, the decoder described herein may be configured to perform in a manner dependent on the intra-coding mode, from one or more reconstructed samples 118 adjacent to the predetermined block 80₁Performing intra-prediction 122 of the predetermined block 80 to obtain a predictor of the predetermined block, and by using the predictor for the corresponding blockTransformation of the prediction residuals decoded by the transform partition 300 to correct the predictor within each transform partition 300 to reconstruct the predetermined block 80: by targeting the current child partition (e.g., child partition 112 shown in FIG. 17)₁Or sub-partition 102 as shown in FIG. 18₁) And continue processing subsequent child partitions (e.g., child partition 112 shown in fig. 17)₂Or sub-partition 102 as shown in FIG. 18₂) The following operations were previously performed to sequentially reconstruct the transform partition 300 groups according to the predetermined sub-partition order 126, with the transform partitions 300 of the predetermined blocks 80 grouped into transform partition groups to form sub-partitions (e.g., 112 shown in fig. 17) for each transform partition 300 group₁And 112₂And 102 as shown in FIG. 18₁And 102₂) The predetermined sub-partition order 126 sequentially traverses the sub-partitions 102/112 along the predetermined dimension 104. The decoder is configured to decode the current sub-partition by deriving from one or more reconstructed samples 118 adjacent to the current sub-partition in a manner dependent on the intra-coding mode₁The current sub-partition is populated to derive 122 a predictor for the current sub-partition. Furthermore, the decoder is configured to reconstruct the current sub-partition by correcting the predictors within each transform partition 300 comprised by the set of transform partitions forming the current sub-partition using the transform of the respective transform partition 300.

As shown in fig. 17 and 18, the encoder described herein is similar to the decoder, e.g., configured to perform in a manner dependent on the intra-coding mode, from one or more reconstructed samples 118 adjacent to the predetermined block 80₁Intra-prediction 122 is performed on the predetermined block 80 to obtain a predictor of the predetermined block 80, and a prediction residual of the predetermined block 80 is determined for correcting the predictor within each transform partition 300 using a transform of the prediction residual to be encoded for the respective transform partition 300: by means of one or more reconstructed samples 118 adjacent to the current sub-partition in a manner dependent on the intra coding mode₁Filling the current sub-partition, sequentially predicting the set of transform partitions 300 according to a predetermined sub-partition order 126 to derive the current sub-partition (e.g., 112 shown in FIG. 17)₁Or 102 shown in FIG. 18₁) Is predicted byThe transform partitions 300 of the predetermined block 80 are grouped into transform partition 300 groups to form sub-partitions (e.g., 112 shown in FIG. 17) for each transform partition group₁And 112₂Or 102 shown in FIG. 18₁And 102₂) The predetermined sub-partition order 126 sequentially traverses the sub-partition 102/112 along the predetermined dimension 104, continuing with subsequent sub-partitions (e.g., 112 shown in FIG. 17)₂Or 102 shown in FIG. 18₂) Previously, the transformation of the prediction residuals within each transform partition 300 comprised by the set of transform partitions forming the current sub-partition 102/112 is determined for use in correcting the predictors within each transform partition 300 comprised by the set of transform partitions forming the current sub-partition 102/112 by using the transformation of the respective transform partition to reconstruct the current sub-partition.

According to fig. 17 and 18, the decoder and/or encoder is for example configured to use reconstructed samples 118 adjacent to the predetermined block 80₁To predict 122 grouping together to form a first sub-partition (e.g., sub-partition 112 shown in fig. 17)₁Or the sub-partition 102 shown in FIG. 18₁) The predictors of the first and second transform partitions 300. Grouped together to form a second sub-partition (e.g., sub-partition 112 shown in FIG. 17) for prediction 122₂Or the sub-partition 102 shown in FIG. 18₂) The decoder uses, for example, reconstructed samples 118 adjacent to the predetermined block 80₂And/or reconstructed samples 118 of a second transform partition 300 adjacent to a third transform partition 300₂. In other words, to predict 122 the second sub-partition 112₂/102₂For example, the decoder is configured to use the predictor of the second sub-partition 112₂/102₂Neighboring reconstructed samples 118₂Wherein at least some of the reconstructed samples may be associated with reconstructed samples of a previous sub-partition, such as sub-partition 112 shown in FIG. 17₁Or the sub-partition 102 shown in FIG. 18₁。

According to an embodiment, the sub-partitions of the predetermined block 80 are sequentially reconstructed or encoded. In other words, the first sub-partition (e.g., sub-partition 112 according to FIG. 17)₁Or sub-partition 102 according to FIG. 18₁) Is reconstructed or encoded in a first loop and then a second sub-partition (e.g., sub-partition 112 according to fig. 17)₂Or sub-partition 102 according to FIG. 18₂) Reconstructed or encoded in a second loop. Thus, all transform partitions of the same sub-partition are reconstructed or encoded in the same loop. In other words, transform partitions associated with the same sub-partition may be reconstructed or encoded in parallel. Thus, according to an embodiment, the complete predetermined block 80 in fig. 15 or fig. 16 may be understood as one partition (i.e., sub-partition) because all transform partitions 300 are reconstructed in parallel.

According to an embodiment, the decoder and/or encoder is configured such that the number of transform partitions 300 of each sub-partition 102/112 depends on the size of the predetermined block 80.

According to an embodiment, the decoder and/or encoder is configured such that the number of transform partitions 300 of each sub-partition 102/112 is 1 if the size of the predetermined block 80 exceeds a predetermined threshold, and is greater than 1 if the size of the predetermined block 80 does not exceed the predetermined threshold. If the predetermined block exceeds a certain size, it is advantageous to avoid further partitioning of the sub-partitions to increase the decoding or encoding run time. The predetermined threshold is, for example, a block size of 64 × 64 samples. In other words, the use of the ISP is limited to the case where the size of the predetermined block 80 does not exceed the predetermined threshold.

According to an embodiment, the decoder and/or encoder is configured such that the number of sub-partitions 102/112 in the predetermined block depends on the size and/or predetermined dimensions of the predetermined block.

According to an embodiment, the decoder and/or encoder is configured such that the number of sub-partitions 102/112 in the predetermined block 80 is 1 if the size of the predetermined block 80 is below a further predetermined threshold and is larger than 1 if the size of the predetermined block 80 is not below the further predetermined threshold. The further predetermined threshold is for example determined such that the predetermined block 80 comprises at least 16 samples. Another predetermined threshold is, for example, the case where the size of the predetermined block 80 is larger than 4 × 4 samples, 8 × 2 samples, 2 × 8 samples, 1 × 16 samples, or 16 × 1 samples. In the case of 4 × 4 samples, as shown in fig. 15 or 16, the predetermined block 80 includes one sub-partition having, for example, four 1 × 4 transform partitions or four 4 × 1 transform partitions. In the case of 2 × 8 samples, the predetermined block 80 includes one sub-partition having, for example, two 1 × 8 transform partitions, and in the case of 8 × 2 samples, the predetermined block 80 includes one sub-partition having, for example, two 8 × 1 transform partitions. In case that the predetermined block has a size of 1 × 16 or 16 × 1 samples, the entire block is both a sub-partition and a transform partition. If the predetermined block 80 comprises only one sub-partition, the sub-partition is for example equal to the entire predetermined block 80. Alternatively, as shown in fig. 17 and 18, the larger size of the predetermined block 80 may result in 2 or more sub-partitions, which include 2 or more transform partitions.

According to an embodiment, the decoder and/or encoder is configured such that the number of sub-partitions in the predetermined block depends on the size of the predetermined block in the following respects: the number is equal to the first number in the case where the dimension of the predetermined block adopts a first width and a first height, and is equal to a second number different from the first number in the case where the dimension of the predetermined block adopts a second width equal to the first height and a second height equal to the first width. As shown in fig. 15 to 18, the number of sub-partitions is different for blocks having the same width and height than for blocks having the same width but different heights. For example, the predetermined block 80 of 4 × 4 size is not divided, i.e., the entire block represents one partition (i.e., sub-partition) having four transform partitions. In contrast, a predetermined block having a size of 4 × 8 or a size of 8 × 4 is divided into two sub-partitions. This also applies to larger blocks. For example, the predetermined block 80 having a size of 128 × 128 is not divided, but the predetermined block 80 having a size of 128 × 64 may be vertically (along the horizontal division dimension) divided into four sub-partitions.

3 results of the experiment

The proposed method is evaluated with VTM-4.0.1 software for intra-frame only (AI), Random Access (RA) and low latency b (ldb) configuration according to the common test condition [2 ]. The corresponding simulation was performed on an Intel Xeon cluster (E5-2697A v4, AVX2 on, turbo boost off) with Linux operating system and GCC 7.2.1 compiler.

Table 7: result of AI configuration

Table 8: results of RA configuration

Table 9: results of LDB configuration

4 additional information

4.1 use of sub-partitions with widths less than 4

Considering that samples are typically distributed in a raster scan fashion, and that some implementations write the output of reconstructed samples into 4 x 1 groups, the use of sub-partitions with widths less than 4 has been mentioned on the jfet reflection as a potential hardware problem. Thus, for example, a 1 × N or 2 × N sub-partition may cause problems. Prediction from 1 xn sub-partitions is not a big problem, writing 1 xn sub-partition data becomes a problem because data typically writes 4 or 8 samples horizontally per cycle. In order for it to work, it is necessary to reserve a 4 × N intermediate buffer (register) to store these data and then write 4 samples at a time (e.g. for deblocking) to the memory, for example. The same problem arises with reading the 1 xn inverse transform output for reconstruction. This increases latency and may also require double buffering. To this end, additional information is provided below to evaluate the impact of using these sub-partitions.

4.1.1 removing sub-partitions with widths less than 4

This modification affects the vertical partitioning of 4 × N and 8 × N blocks as follows:

4 × N: there is no longer a vertical split, i.e. along the horizontal division dimension. Thus, whenever an ISP is used on one of these blocks, the decoder or encoder concludes that horizontal partitioning is used, and therefore does not have to parse the partition flag syntax element. Thus, for example, 4 × 1 transform partitions are used instead of 1 × N transform partitions.

8 × N: the vertical partitioning results in two sub-partitions instead of four. The predetermined block 80 is thus divided into, for example, 4 xn sub-partitions instead of 1 xn or 2 xn sub-partitions.

Note that this modified ISP method is proposed in [1] as test 1.1.2 in CE3 of the mahalanobis meeting in month 1 2019. The results of this method above VTM-4.0.1 are presented in Table 10, Table 11 and Table 12.

Table 10: configuring results for AI with no child partitions of width less than 4

Table 11: configuring results for RA with no sub-partitions with widths less than 4

Table 12: configuring results for LDBs with no child partitions with widths less than 4

4.1.2 extending ISPs of independent subdivisions to subdivisions less than 4 wide

In this case, sub-partitions with widths less than 4 would not be removed from the ISP fabric. Instead, the same designs as 4 × 4, 8 × 4, and 4 × 8 described in section 2 are applied to them. Two different cases are distinguished:

the vertical division of the 4 × N block is handled in the same manner as the vertical division of 4 × 4 explained in section 2.1. For example, a 4 x 32 block may be vertically divided into four separate 1 x 32 transform partitions 300, which form one partition 112.

The vertical division of the 8 × N block is handled in the same manner as the vertical division of the 8 × 4 block explained in section 2.2. For example, an 8 × 16 block may be partitioned into four 2 × 16 transform partitions 300, where transform partitions 1 and 2 form partition 112₁The transform partitions 3 and 4 form partition 112₂Transform partitions 2 and 4 cannot use the reconstructed samples of transform partitions 1 and 3, respectively, to generate their corresponding prediction signals.

This method allows writing reconstructed samples in a set of at least 4 x 4 samples in all vertical splitting cases (the minimum value is still 16 x 1 in the case of horizontal splitting). The results of this method (using independent transform partitions on 4 × 4, 4 × 8, 8 × 4 and vertical partitioning of 4 × N and 8 × N blocks) above VTM-4.0.1 are presented in table 13, table 14 and table 15.

Thus, according to an embodiment, the decoder is configured to divide the predetermined block into 16M sub-blocks, where M ≧ 1, or alternatively into 4N sub-blocks or N × 4 sub-blocks, where N ≧ 4. As described above, the sub-block is optionally divided into two or more smaller transform partitions.

Table 13: results of vertical partitioning using independent sub-partitions on 4 × 4, 4 × 8, 8 × 4 and 4 × N and 8 × N blocks for AI configuration

Table 14: configuring results of vertical partitioning using independent sub-partitions and 4 xN and 8 xN blocks on 4 x 4, 4 x 8, 8 x 4 for RA

Table 15: results of using independent sub-partitions on 4 × 4, 4 × 8, 8 × 4 and vertical partitioning of 4 × N and 8 × N blocks for LDB configuration

5 conclusion

The results reported show gains for CTCs and classes F with little (AI) or no impact on the coding run-time (RA and LDB).

With respect to the additional information presented, experimental results show that removing sub-partitions with widths less than 4 produces significant penalties, especially in the case of classes C and E in AI and class F in all configurations. Furthermore, this loss is negligible for the reduction of the coding run time. On the other hand, the new ISP design is extended to produce blocks with sub-partitions less than 4 wide with better performance in BD-Rate gain than completely removing these sub-partitions, since it does not produce any penalty, it produces significant gain in class F, and the impact on the coding run time is very small.

6 ISP partitioning Structure complexity reduction aspect

As mentioned above, there are variations of the concepts outlined above, one variation being presented here. As mentioned above, in a typical decoder hardware implementation, one of the most critical aspects affecting the top-level system pipeline is processing dependencies. In the ISP case, the minimum 16 sample constraint ensures that the loop correlation present in intra prediction is not an issue in terms of throughput, since the smallest luma block in VVC has a size of 4 × 4, i.e., 16 samples.

However, ISPs have introduced new very narrow shapes in VVC design, namely 1 × N, 2 × N, M × 2 and M × 1. In the case of 1 × N and 2 × N sub-partitions, processing dependencies can become an issue because pixels are typically allocated in a raster scan fashion in an internal line buffer memory, with 4 × 1 as a set of access samples. For this reason, filling the buffer is inefficient for all sub-partitions with widths less than 4. The impact of this problem can be reduced by different hardware implementations (e.g. modifying the allocation of pixels or using transpose memory), but in any case it still implies an increase in hardware implementation complexity.

Fig. 19a and 19b show examples of different sub-partitions 112/P and transform partitions 300/T for different block sizes of the predetermined block 80, wherein the left example shows a conventional partition and the right example shows a proposed partition according to the invention described herein. The decoding and/or encoding of the predetermined block 80 on the right side of fig. 19a may be performed similar to or as described in fig. 15a to 15d, and the decoding and/or encoding of the predetermined block 80 on the right side of fig. 19a may be performed similar to or as described in fig. 17.

To reduce the complexity of the hardware implementation, a minimum prediction width of 4 is established (the transform size remains unchanged). This affects the vertical partitioning of 4 × M (M >4) and 8 × N (N >4) blocks by:

4 × M: as shown in fig. 1a, the entire block is predicted at once like a non-ISP block and a 4 × M residual signal is calculated. The residual is then split into four independently processed 1 × M (or two 2 × 8, for M ═ 8) transform sub-partitions.

8 × N: as shown in fig. 1b, the block is divided into two 4 xn predictor partitions, P1 and P2. After the prediction for P1 is computed, a 4 × N residual signal is generated and divided into two 2 × N transform sub-partitions, T1 and T2, which are processed separately. The same procedure was then similarly repeated for P2, T3, and T4. However, in this case, the prediction signal may use reconstructed samples of the P1 region.

Reference 7 to the accompanying drawings

[1] S.De-Lux n-Hern ndz, V.George, J.Ma, T.Nguyen, H.Schwarz, D.Marpe and T.Wiegand, "CE3: Intra Sub-Partitions Coding model (Tests 1.1.1 and 1.1.2)," document JFET-M0102, Marrakech, MA, 2019.

[2] Bossen, J.Boyce, X.Li, V.Seregin and K.S Huhring, "JVT common test conditions and software reference configurations for SDR video," document JVT-M1010, Marrakech, MA, 2019.

Although some aspects have been described in the context of an apparatus, it will be clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of a respective block or item or a feature of a respective apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

The data stream of the present invention may be stored on a digital storage medium or may be transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium (e.g., the internet).

Embodiments of the invention may be implemented in hardware or in software, depending on certain implementation requirements. Implementation may be performed using a digital storage medium (e.g. a floppy disk, a DVD, a blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a flash memory) having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Accordingly, the digital storage medium may be computer-readable.

Some embodiments according to the invention comprise a data carrier with electronically readable control signals capable of cooperating with a programmable computer system so as to perform one of the methods described herein.

Generally, embodiments of the invention can be implemented as a computer program product having a program code operable to perform one of the methods when the computer program product runs on a computer. The program code may be stored, for example, on a machine-readable carrier.

Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

In other words, an embodiment of the inventive method is thus a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

Thus, another embodiment of the inventive method is a data carrier (or digital storage medium or computer readable medium) having a computer program recorded thereon for performing one of the methods described herein. The data carrier, the digital storage medium or the recording medium is typically tangible and/or non-transitory.

Thus, another embodiment of the inventive method is a data stream or a signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may for example be arranged to be transmitted via a data communication connection (e.g. via the internet).

Another embodiment comprises a processing device, e.g., a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

Another embodiment comprises a computer having a computer program installed thereon for performing one of the methods described herein.

Another embodiment according to the present invention comprises an apparatus or system configured to transmit a computer program (e.g., electronically or optically) to a receiver, the computer program being for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may for example comprise a file server for transmitting the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the method is preferably performed by any hardware device.

The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The apparatus described herein or any component of the apparatus described herein may be implemented at least in part in hardware and/or software.

The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

Any components of the methods described herein or the apparatus described herein may be performed at least in part by hardware and/or by software.

The above-described embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended that the scope of the appended patent claims be limited only by the details of the description and the explanation of the embodiments herein, and not by the details of the description and the explanation.

68页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：用于视频编码和解码的装置、方法和计算机程序

Partition-based intra coding concept for achieving high efficiency

相关技术

网友询问留言