阅读说明：本技术 用于视频编解码的去方块滤波器 (Deblocking filter for video coding and decoding ) 是由 张莉 朱维佳 张凯 刘鸿彬 许继征 于 2020-05-11 设计创作，主要内容包括：在示例性方面,用于视觉媒体处理的方法包括：识别两个视频块的垂直边缘和/或水平边缘处的边界；基于使用组合的帧内帧间预测(CIIP)模式编码跨越垂直边缘或水平边缘的两个视频块中的至少一个,计算滤波器的边界强度；决定打开还是关闭所述滤波器；在打开所述滤波器的情况下选择所述滤波器的强度；以及基于所述选择,对所述视频块执行去方块滤波(DB)处理。(In an exemplary aspect, a method for visual media processing includes: identifying boundaries at vertical edges and/or horizontal edges of two video blocks; calculating a boundary strength of a filter based on encoding at least one of two video blocks spanning a vertical edge or a horizontal edge using a Combined Intra Inter Prediction (CIIP) mode; deciding whether to turn on or off the filter; selecting a strength of the filter if the filter is turned on; and based on the selection, performing a deblocking filter (DB) process on the video block.)

1. A method for visual media processing, comprising:

identifying boundaries at vertical edges and/or horizontal edges of two video blocks;

calculating a boundary strength of a filter based on at least one of the two video blocks crossing a vertical edge or a horizontal edge coded using a Combined Intra Inter Prediction (CIIP) mode;

deciding whether to turn on or off the filter;

selecting a strength of the filter if the filter is turned on; and

based on the selection, a deblocking filter (DB) process is performed on the video block.

2. The method of claim 1, wherein CIIP codec blocks are considered intra codec blocks when calculating the boundary strength.

3. The method of claim 1, wherein the boundary strength is selected to be 2 if at least one of two video blocks that span a vertical or horizontal edge are CIIP coded.

4. The method of claim 1, wherein the boundary strength is selected to be 1 in the case where only one of two blocks across a vertical or horizontal edge is CIIP coded and the other video block is inter coded.

5. The method of claim 1, wherein the boundary strength is selected to be 1 in the case where only one of two blocks across an edge is coded with CIIP and the other video block is coded with inter or Intra Block Copy (IBC).

6. A method for visual media processing, comprising:

determining a codec scheme applicable to transitions between video blocks of visual media data and bitstream representations of the video blocks; and

selectively enabling or disabling application of an in-loop filter to the video block during a transition of the video block based on the determination of the coding scheme.

7. The method of claim 6, wherein the in-loop filter comprises one of: deblocking filters (DB), Sample Adaptive Offset (SAO) filters, and/or Adaptive Loop Filters (ALF).

8. The method of any one or more of claims 6 to 7, wherein the coding scheme comprises refining predicted samples and/or motion information and/or reconstructed samples using optical flow.

9. The method of claim 8, wherein the codec that refines the predicted samples using the optical flow comprises a bi-directional optical flow (BDOF).

10. The method of claim 9, wherein the determination of the coding scheme comprises: identifying whether the predicted samples are located within the bidirectional optical flow (BDOF); and in response to determining that the predicted samples are within the bi-directional optical flow (BDOF), disabling the application of the in-loop filter.

11. The method of claim 10, wherein at least one of a boundary strength and/or a filter strength of the in-loop filter is determined based on use of the bi-directional optical flow (BDOF).

12. The method of claim 9, wherein the determination of the coding scheme comprises: identifying whether the predicted samples are located within the bi-directional optical flow (BDOF), and wherein the applying of the in-loop filter is enabled in response to determining that the predicted samples are located within the bi-directional optical flow (BDOF), and further wherein the in-loop filter is weaker in strength than another in-loop filter associated with predicted samples that are not located within the BDOF.

13. A method for visual media processing, comprising:

determining a type of transform matrix applicable to a transition between a video block of visual media data and a bitstream representation of the video block; and

selectively enabling or disabling application of a Deblocking (DB) filtering process to the video block during conversion of the video block based on the determination of the type of the transform matrix.

14. The method of claim 13, wherein the type of transform matrix comprises DCT-II, DST-VII, DCT-VIII, and/or transform skip mode.

15. The method of any one or more of claims 13 to 14, wherein the applying of the deblocking filtering process further comprises a clipping operation.

16. The method of claim 15, wherein the clipping operation is location dependent based on a location of samples in the video block relative to the DB filter.

17. The method of claim 16, wherein the clipping operation is disabled for samples located in transform skip coded video blocks.

18. The method of claim 17, wherein the deblocking filtering (DB) processing is disabled on blocks of the transform-skip codec or when it is determined that the video block codec is exempt from transform-based codec techniques.

19. The method of claim 18, wherein the transform-based codec technique comprises Residual Differential Pulse Codec Modulation (RDPCM) or quantized residual differential pulse codec modulation (QR-BDPCM).

20. The method of claim 15, wherein if the DB filtering process is enabled for a group of samples, all samples included in the group will be subjected to the same parameters of the clipping operation.

21. The method of claim 16, wherein parameters of the position-dependent clipping operation are related to basis functions of a transformation matrix applied to the transformation.

22. A method for visual media processing, comprising:

determining to apply an in-loop filtering step to a video block of visual media data during a transition between the video block and a bitstream representation of the video block, wherein the in-loop filtering step comprises a clipping operation applied to samples in the video block to be filtered; and

selectively enable or disable the cropping operation in response to determining one or more conditions associated with the video block.

23. The method of claim 22, wherein the one or more conditions associated with the video block comprise:

(a) content of the video block, or

(b) Sequence Parameter Set (SPS)/Video Parameter Set (VPS)/Picture Parameter Set (PPS)/picture header/slice group header/Largest Coding Unit (LCU)/Coding Unit (CU)/LCU row/LCU group.

24. The method of any one or more of claims 1 to 23, wherein the converting comprises generating the bitstream representation from the video block.

25. The method of any one or more of claims 1 to 23, wherein the converting comprises generating pixel values of the video block from the bitstream representation.

26. A video encoder device, comprising: a processor configured to implement the method of any one or more of claims 1 to 23.

27. A video decoder device, comprising: a processor configured to implement the method of any one or more of claims 1 to 23.

28. A computer readable medium having code stored thereon, the code comprising processor-executable instructions for implementing the method of any one or more of claims 1 to 23 by a processor.

Technical Field

This patent document relates to video encoding/decoding techniques, devices, and systems.

Background

Despite advances in video compression, digital video still accounts for the greatest bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth required to pre-count the usage of digital video will continue to grow.

Disclosure of Invention

Devices, systems, and methods related to video encoding/decoding techniques, and in particular to deblocking filters in image/video codecs, are described. The described method can be applied to standards under development (e.g. multifunctional video codecs) and future video codec standards or video codecs.

In one aspect, the disclosed technology may be used to provide a method for visual media processing. The method comprises the following steps: identifying boundaries at vertical edges and/or horizontal edges of two video blocks; calculating a boundary strength of a filter based on at least one of two video blocks crossing a vertical edge or a horizontal edge encoded using a Combined Intra Inter Prediction (CIIP) mode; deciding whether to turn on or off the filter; selecting a strength of the filter if the filter is turned on; and based on the selection, performing a deblocking filter (DB) process on the video block.

In another aspect, the disclosed technology may be used to provide a method for visual media processing. The method comprises the following steps: determining a coding scheme applicable to transitions between video blocks of visual media data and bit stream representations of the video blocks; and selectively enabling or disabling application of an in-loop filter to the video block during the transition of the video block based on the determination of the encoding scheme.

In yet another aspect, the disclosed technology may be used to provide a method for visual media processing. The method comprises the following steps: determining a type of transform matrix applicable to a transition between a video block of visual media data and a bitstream representation of the video block; and selectively enabling or disabling application of a Deblocking (DB) filtering process to the video block during the conversion of the video block based on the determination of the type of transform matrix.

In yet another aspect, the disclosed technology may be used to provide a method for visual media processing. The method comprises the following steps: determining to apply an in-loop filtering step to a video block during a transition between the video block of visual media data and a bitstream representation of the video block, wherein the in-loop filtering step comprises a clipping operation applied to samples in the video block to be filtered; and selectively enable or disable the cropping operation in response to determining one or more conditions associated with the video block.

In yet another aspect, the above-described method is embodied in the form of processor executable code and stored in a computer readable program medium.

In yet another aspect, an apparatus configured or operable to perform the above method is disclosed. The apparatus may include a processor programmed to implement the method.

In yet another aspect, a video decoder device may implement a method as described herein.

In yet another aspect, a video encoder device may implement a method as described herein.

The above and other aspects and features of the disclosed technology are described in more detail in the accompanying drawings, the description and the claims.

Drawings

Fig. 1 shows an example of an encoder block diagram for a multifunction video codec (VVC).

Fig. 2 shows an example process flow of the deblocking filtering process.

Fig. 3 illustrates how the Bs value is calculated based on an intra coding mode, the presence of non-zero transform coefficients and motion information, a reference picture, the number of motion vectors, and a motion vector difference.

Fig. 4 shows an example of reference information for calculating Bs at a CTU boundary.

Fig. 5 shows an example of a pixel related to filter on/off decision.

Fig. 6A shows an example of division of 4 × 8 and 8 × 4 blocks.

Fig. 6B shows an example of division of all blocks except for 4 × 8, 8 × 4, and 4 × 4.

Fig. 7A shows an example of Affine Linear Weighted Intra Prediction (ALWIP) for a4 × 4 block.

Fig. 7B shows an example of ALWIP for an 8 × 8 block.

Fig. 7C shows an example of ALWIP for an 8 × 4 block.

Fig. 7D shows an example of ALWIP for a 16 × 16 block.

Fig. 8 shows an example of quadratic transformation in JEM.

Fig. 9 illustrates an example of a simplified quadratic transformation (RST) based on an embodiment of the disclosed technology.

FIG. 10 shows examples of sub-block transform modes SBT-V and SBT-H.

Fig. 11 shows an example of the subblock MV VSB and the pixel Δ v (i, j).

Fig. 12 illustrates a flow diagram of an example method for video processing, in accordance with some embodiments of the disclosed technology.

Fig. 13 illustrates a flow diagram of an example method for video processing in accordance with some embodiments of the disclosed technology.

Fig. 14 illustrates a flow diagram of an example method for video processing, in accordance with some embodiments of the disclosed technology.

Fig. 15 illustrates a flow diagram of an example method for video processing in accordance with some embodiments of the disclosed technology.

Fig. 16 illustrates a flow diagram of an example method for video processing, in accordance with some embodiments of the disclosed technology.

Fig. 17 illustrates a flow diagram of an example method for video processing in accordance with some embodiments of the disclosed technology.

Fig. 18 illustrates a flow diagram of an example method for video processing, in accordance with some embodiments of the disclosed technology.

Fig. 19 illustrates a flow diagram of an example method for video processing, in accordance with some embodiments of the disclosed technology.

Fig. 20 is a block diagram of an example of a hardware platform for implementing the visual media decoding or visual media encoding techniques described in this document.

FIG. 21 is a block diagram of an example video processing system in which the disclosed techniques may be implemented.

FIG. 22 shows a flow diagram of an example method for visual media processing.

FIG. 23 shows a flow diagram of an example method for visual media processing.

FIG. 24 shows a flow diagram of an example method for visual media processing.

FIG. 25 shows a flow diagram of an example method for visual media processing.

Detailed Description

Due to the increasing demand for higher resolution video, video coding and decoding methods and techniques are ubiquitous in modern technology. Video codecs typically include electronic circuits or software that compress or decompress digital video, and are continually being improved to provide higher coding efficiency. The video codec converts uncompressed video into a compressed format and vice versa. There is a complex relationship between video quality, the amount of data used to represent the video (determined by the bit rate), the complexity of the encoding and decoding algorithms, susceptibility to data loss and errors, ease of editing, random access, and end-to-end delay (lag). The compression format typically conforms to a standard Video compression specification, such as the High Efficiency Video Codec (HEVC) standard (also known as h.265 or MPEG-H part 2), the Versatile Video Coding to be done (VVC) standard, or other current and/or future Video codec standards.

In some embodiments, reference software called Joint Exploration Model (JEM) is used to explore future video codec techniques. In JEM, sub-block based prediction is employed in several coding tools, such as affine prediction, optional temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), bi-directional optical flow (BIO), frame rate up-conversion (FRUC), Locally Adaptive Motion Vector Resolution (LAMVR), Overlapped Block Motion Compensation (OBMC), Local Illumination Compensation (LIC), and decoder-side motion vector refinement (DMVR).

Embodiments of the disclosed techniques may be applied to existing video codec standards (e.g., HEVC, h.265) and future standards to improve runtime performance. This document provides various techniques that may be used by a decoder of an image or video bitstream to improve the quality of decompressed or decoded digital video or images. For the sake of brevity, the term "video" is used herein to include a sequence of pictures (conventionally referred to as video) and a single image. In addition, the video encoder may also implement these techniques during the encoding process in order to reconstruct the decoded frames for further encoding.

Section headings are used in this document to improve the readability of the description, and do not in any way limit the discussion or embodiments (and/or implementations) to individual sections.

1. Example of an encoding flow for a typical video codec

Fig. 1 shows an example of an encoder block diagram for a VVC, which contains three in-loop filter blocks: deblocking Filter (DF), Sample Adaptive Offset (SAO), and ALF. Unlike DF using predefined filters, SAO and ALF utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by increasing the offset and applying a Finite Impulse Response (FIR) filter, respectively, where the encoded side information signals the offset and filter coefficients. ALF is located in the last processing stage of each picture and can be viewed as a tool that attempts to capture and fix artifacts created by previous stages.

Deblocking scheme example in HEVC

The deblocking filter process is performed on each CU in the same order as the decoding process. First, vertical edges are filtered (horizontal filtering), and then horizontal edges are filtered (vertical filtering). For both luma and chroma components, filtering is applied to determine the 8x8 block boundaries to filter. To reduce complexity, the 4x4 block boundaries are not processed.

Fig. 2 shows the overall process flow of the deblocking filtering process. The boundary may have three filter states: no filtering, weak filtering, and strong filtering. Each filtering decision is based on the boundary strength Bs and the thresholds β and t_C。

2.1 boundary decision

Three boundaries may be involved in the filtering process: CU boundaries, TU boundaries, and PU boundaries. Since CU boundaries are always also TU boundaries or PU boundaries, filtering always involves CU boundaries (i.e., the outer edges of the CU). When the PU shape is 2NxN (N >4) and the RQT depth is equal to 1, filtering involves TU boundaries at an 8x8 block grid and PU boundaries between each PU inside the CU. One exception is that when the PU boundary is within a TU, the boundary is not filtered.

2.2 boundary Strength calculation

Fig. 3 shows a flow chart for Bs calculation. In general, the boundary strength (Bs) reflects how strong filtering is required for the boundary. If Bs is large, then use of strong filtering should be considered. Let P and Q be defined as the blocks involved in the filtering, where P denotes the blocks located to the left of the boundary (vertical edge case) or above (horizontal edge case) and Q denotes the blocks located to the right of the boundary (vertical edge case) or above the boundary (horizontal edge case). Fig. 3 illustrates how the Bs value is calculated based on an intra coding mode, the presence of non-zero transform coefficients and motion information, a reference picture, the number of motion vectors, and a motion vector difference.

Bs is calculated based on 4x4 blocks, but remaps it to an 8x8 grid. The maximum value of two values of Bs corresponding to 8 pixels composed of lines in a4 × 4 grid is selected as Bs for the boundary in the 8 × 8 grid.

Fig. 4 shows an example of reference information for calculating Bs at a CTU boundary. To reduce the line buffer memory requirements, the information in each second block (4 x4 grid) on the left or above is reused only for CTU boundaries, as shown in fig. 4.

2.3 β and t_CDecision making

Deriving thresholds beta and t relating to filter on/off decision, strong and weak filter selection and weak filtering processing based on luminance quantization parameters QPP and QPQ of P and Q blocks, respectively_C. For deriving beta and t_CQ of (A) is calculated as follows.

Q＝((QP_P+QP_Q+1)>>1)

As shown in table 1, the variable β is derived based on Q. If Bs is greater than 1, then with Clip3(0,55, Q +2) as an input, a variable t is specified as shown in Table 1_C. Otherwise (BS is equal to or less than 1), Q is taken as input, and the variable t is specified as shown in Table 1_C。

Table 1: deriving threshold variables β and t from input Q_C

Q	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
																				β	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	6	7	8
tC	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
																				Q	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37
β	9	10	11	12	13	14	15	16	17	18	20	22	24	26	28	30	32	34	36
																				tC	1	1	1	1	1	1	1	1	2	2	2	2	3	3	3	3	4	4	4
Q	38	39	40	41	42	43	44	45	46	47	48	49	50	51	52	53	54	55
																				β	38	40	42	44	46	48	50	52	54	56	58	60	62	64	64	64	64	64
tC	5	5	6	6	7	8	9	9	10	10	11	11	12	12	13	13	14	14

2.4 Filter on/off decision every 4 lines

Fig. 5 shows an example of a pixel related to filter on/off decision. Note that in the following description, pNM denotes the left nth sample in the mth row with respect to the vertical edge or the upper nth sample in the mth column with respect to the horizontal edge, and qNM denotes the right nth sample in the mth row with respect to the vertical edge or the lower nth sample in the mth column with respect to the horizontal edge. Examples of pNM and qNM are shown in fig. 5.

Note that in the following description, pN denotes the left nth sample in a row with respect to a vertical edge or the upper nth sample in a column with respect to a horizontal edge, and qN denotes the right nth sample in a row with respect to a vertical edge or the lower nth sample in a column with respect to a horizontal edge.

The on/off decision of the filter is made in units of four rows. Fig. 5 shows a pixel related to filter on/off decision. The 6 pixels in the two red boxes of the first four rows are used to determine the filter on/off of the 4 rows. The 6 pixels in the two red boxes of the next four rows are used to determine the filter on/off of the next four rows.

If dp0+ dq0+ dp3+ dq3< β, then the filtering of the first four rows is turned on and the strong/weak filter selection process is applied. Each variable is derived as follows.

dp0＝|p2₀–2*p1₀+p0₀|,dp3＝|p2₃–2*p1₃+p0₃|,dp4＝|p2₄–2*p1₄+p0₄|,dp7＝|p2₇–2*p1₇+p0₇|

dq0＝|q2₀–2*q1₀+q0₀|,dq3＝|q2₃–2*q1₃+q0₃|,dq4＝|q2₄–2*q1₄+q0₄|,dq7＝|q2₇–2*q1₇+q0₇|

If the condition is not met, the first 4 rows are not filtered. Further, if the condition is satisfied, dE, depp 1, and dEp2 may be derived for weak filtering processing. The variable dE is set equal to 1. If dp0+ dp3< (β + (β > >1)) > >3, then variable dEp1 is set equal to 1. If dq0+ dq3< (β + (β > >1)) > >3, then variable dEq1 is set equal to 1. For the next four rows, the decision is made in the same manner as described above.

2.5 Strong/Weak Filter selection for 4 rows

After the first four rows are examined to determine that the filters are enabled, and if the following two conditions are met, then the first four rows are filtered using a strong filter. Otherwise, a weak filter will be used for filtering. The pixels involved are the same as those used for filter on/off decision, as shown in fig. 5.

—2*(dp0+dq0)<(β>>2),|p3₀–p0₀|+|q0₀–q3₀|<(β>>3) And | p0₀–q0₀|<(5*t_C+1)>>1

—2*(dp3+dq3)<(β>>2),|p3₃–p0₃|+|q0₃–q3₃|<(β>>3) And | p0₃–q0₃|<(5*t_C+1)>>1

Similarly, the next four rows are filtered using a strong filter if the following two conditions are met. Otherwise, a weak filter will be used for filtering.

—2*(dp4+dq4)<(β>>2),|p3₄–p0₄|+|q0₄–q3₄|<(β>>3) And | p0₄–q0₄|<(5*t_C+1)>>1

—2*(dp7+dq7)<(β>>2),|p3₇–p0₇|+|q0₇–q3₇|<(β>>3) And | p0₇–q0₇|<(5*t_C+1)>>1

2.5.1 Strong Filtering

For strong filtering, the filtered pixel value is obtained by the following equation. Note that three pixels are modified using four pixels as inputs to each P and Q block, respectively.

p₀′＝(p₂+2*p₁+2*p₀+2*q₀+q₁+4)>>3

q₀′＝(p₁+2*p₀+2*q₀+2*q₁+q₂+4)>>3

p₁′＝(p₂+p₁+p₀+q₀+2)>>2

q₁′＝(p₀+q₀+q₁+q₂+2)>>2

p₂′＝(2*p₃+3*p₂+p₁+p₀+q₀+4)>>3

q₂′＝(p₀+q₀+q₁+3*q₂+2*q₃+4)>>3

2.5.2 Weak Filtering

Δ is defined as follows:

–Δ＝(9*(q₀–p₀)–3*(q₁–p₁)+8)>>4

-when abs (Δ) is less than t_CAt the time of 10, the number of the grooves,

Δ＝Clip3(-t_C,t_C,Δ)

p₀′＝Clip1_Y(p₀+Δ)

q₀′＝Clip1_Y(q₀-Δ)

-if dEp1 equals 1, then

Δp＝Clip3(-(t_C>>1),t_C>>1,(((p₂+p₀+1)>>1)–p₁+Δ)>>1)

p₁′＝Clip1_Y(p₁+Δp)

-if dEq1 equals 1, then

Δq＝Clip3(-(t_C>>1),t_C>>1,(((q₂+q₀+1)>>1)–q₁–Δ)>>1)

q₁′＝Clip1_Y(q₁+Δq)

Notably, up to two pixels are modified using three pixels as inputs to each P and Q block, respectively.

2.5.3 chroma filtering

The chroma filtered Bs is inherited from the luma. Chroma filtering is performed if Bs >1 or if there is a case of coded chroma coefficients. There are no other filtering decisions. Only one filter is applied to the chrominance. The filter selection process for chrominance is not used. The filtered sample values p0 'and q0' are derived as follows:

Δ＝Clip3(-t_C,t_C,((((q₀–p₀)<<2)+p₁–q₁+4)>>3))

p₀′＝Clip1_C(p₀+Δ)

q₀′＝Clip1_C(q₀-Δ)

example of deblocking scheme in VTM-4.0

In the current VTM, VTM-4.0, the deblocking scheme described in JFET-M0471 is used. Vertical edges in the picture are filtered first. The horizontal edges in the picture are then filtered using the samples modified by the vertical edge filtering process as input. The vertical and horizontal edges in the CTB of each CTU are processed on a coding unit basis. Vertical edges of coding blocks in a coding unit are filtered starting from the edge on the left side of the coding block, passing through the edges in the geometric order of the edges and heading to the right side of the coding block. Horizontal edges of coding blocks in a coding unit are filtered starting from an edge at the top of the coding blocks, crossing the edge in its geometric order, towards the bottom of the coding blocks.

3.1 boundary decision

Filtering is applied to the 8x8 block boundaries. In addition, it must be a transform block boundary or a coded sub-block boundary (e.g., due to the use of affine motion prediction, ATMVP). For those without such a boundary, the filter will be disabled.

3.2 boundary Strength calculation

For transform block boundaries/coded sub-block boundaries, if they are located in an 8x8 grid, they may be filtered and the bS [ xD ] of the edge is filtered_i][yD_j]Setting (wherein [ xD)_i][yD_j]Representing coordinates) are defined as follows:

if samples p₀Or q₀In an encoding block of a coding unit encoded in intra prediction mode, then bS [ xD [ ]_i][yD_j]Set equal to 2.

Else, if the block edge is also a transform block edge and the sample point p₀Or q₀In a transform block containing one or more non-zero transform coefficient levels, then bS [ xD [ ] is set_i][yD_j]Is set equal to 1.

Else, if containing a sample point p₀The prediction mode and the included sampling point q of the coding sub-block₀If the prediction mode of the coding sub-block is different, then bS [ xD ] is set_i][yD_j]Is set equal to 1.

-otherwise, if one or more of the following conditions are met, then bS [ xD [ ]_i][yD_j]Set equal to 1:

containing spots p₀And containing samples q₀Each encoded sub-block of (a) is encoded in IBC prediction mode and the absolute difference between the horizontal or vertical components of the motion vectors used in the prediction of the two encoded sub-blocks is greater than or equal to 4 (in units of quarter-luminance samples).

For inclusion of a sample point p₀And the prediction of the coded sub-blocks, and the inclusion of samples q₀Using different references than the prediction of the coded sub-blocksPictures or different numbers of motion vectors.

Note 1 the determination of whether the reference pictures for the two encoded sub-blocks are the same or different is based only on which pictures are referenced, regardless of whether the prediction is formed by using the index to reference picture list 0 or the index to reference picture list 1, and regardless of whether the index positions within the reference picture lists are different.

Note that 2 the number of motion vectors used to predict the encoded sub-block with the top left corner sample coverage (xSb, ySb) is equal to PredFlagL0[ xSb ] [ ySb ] + PredFlagL1[ xSb ] [ ySb ].

Using a motion vector to predict the contained sample point p₀And one motion vector is used to predict the encoded subblock containing samples q0, and the absolute difference between the horizontal or vertical components of the used motion vector is greater than or equal to 4 (in units of quarter luminance samples).

-predicting a contained sample point p using two motion vectors and two different reference pictures₀Using two motion vectors of two identical reference pictures to predict a coded sub-block containing a sample point q₀And the absolute difference between the horizontal or vertical components of two motion vectors used in prediction for two encoded sub-blocks of the same reference picture is greater than or equal to 4 (in units of quarter-luma samples).

Two motion vectors for the same reference picture are used to predict the inclusion of a sample point p₀For the same reference picture, two motion vectors are used to predict the contained sample point q₀And both of the following conditions are true:

-the absolute difference between the horizontal or vertical components of the list 0 motion vectors used to predict the two coded sub-blocks is greater than or equal to 4 (in quarter luma samples), or the absolute difference between the horizontal or vertical components of the list 1 motion vectors used to predict the two coded sub-blocks is greater than or equal to 4 (in quarter luma samples).

- -in the containing spot p₀The water of list 0 motion vector used in prediction of the encoded subblock of (1)The sum of the horizontal or vertical component and the mean square of the contained sample point q₀The absolute difference between the list 1 motion vectors used in the prediction of the encoded subblocks of (a) is greater than or equal to 4 (in units of quarter-luminance samples), or in the inclusion of a sample p₀The horizontal or vertical component of the list 1 motion vector used in the prediction of the coded sub-block and the included sample point q₀The absolute difference between the list 0 motion vectors used in the prediction of the encoded subblock of (1) is greater than or equal to 4 (in units of quarter-luma samples).

Else, the variable bS [ xD ]_i][yD_j]Is set equal to 0.

Table 2 and table 3 summarize the BS calculation rules.

TABLE 2 Border Strength (SPS IBC disabled)

TABLE 3 Border Strength (SPS IBC disabled)

3.3 deblocking decisions for luminance Components

In this section, the deblocking decision process is described. A wider and stronger luminance filter is a filter that is used only when all of Condition 1(Condition1), Condition 2(Condition2), and Condition 3(Condition 3) are true.

Condition1 is a "large block" condition. This condition checks whether the samples on the P and Q sides belong to a large block, which is represented by the variables bsidepislargblk and bsideqisllargblk, respectively. bSidePisLargeBlk and bsideqislrgeblk are defined as follows.

bsidepis largeblk ((edge type is vertical, and p)₀Is of width>CU) | | (edge type is horizontal,and p is₀Is of height>CU of 32))? True: false

bSideQisLargeBlk ((edge type is vertical, and q is vertical, q is q)₀Is of width>CU) | | (edge type is horizontal, and q | |, 32 ═ c | | (edge type is horizontal, and q |)₀Is of height>CU of 32))? True: false

Based on bSidePisLargeBlk and bsideqislrgeblk, condition1 is defined as follows.

Condition1 ═ is (bsidepis largeblk | | | bsidepis largeblk)? TRUE FALSE next, if Condition 1(Condition1) is TRUE, then Condition 2(Condition2) will be further checked. First, the following variables are derived:

first derive dp0, dp3, dq0, dq3 as in HEVC;

if (p-side is greater than or equal to 32), then

dp0＝(dp0+Abs(p5₀-2*p4₀+p3₀)+1)>>1

dp3＝(dp3+Abs(p5₃-2*p4₃+p3₃)+1)>>1

-if (q-side is greater than or equal to 32), then

dq0＝(dq0+Abs(q5₀-2*q4₀+q3₀)+1)>>1

dq3＝(dq3+Abs(q5₃-2*q4₃+q3₃)+1)>>1

Condition 2? TRUE, where d ═ dp0+ dq0+ dp3+ dq3 is as shown in section 2.4.

If conditions 1 and 2 are valid, it is further checked whether any block uses a sub-block:

finally, if both condition1 and condition2 are valid, the proposed deblocking method will check condition 3 (large block strong filter condition), which is defined as follows.

In condition 3(strong filtercondition), the following variables are derived:

dpq is derived as in HEVC.

Deriving sp as in HEVC₃＝Abs(p₃-p₀)

If (p side is greater than or equal to 32)

If (Sp ═ 5)

sp₃＝(sp₃+Abs(p₅-p₃)+1)＞＞1

Otherwise

sp₃＝(sp₃+Abs(p₇-p₃)+1)＞＞1

Deriving sq as in HEVC₃＝Abs(q₀-q₃)

If (q side is greater than or equal to 32)

If (Sq ═ 5)

sq₃＝(sq₃+Abs(q₅-q₃)+1)＞＞1

Otherwise

sq₃＝(sq₃+Abs(q₇-q₃)+1)＞＞1

As in HEVC, StrongFilterCondition (dpq is less than (β >2), sp3+ sq3 is less than (3 β > 5), and Abs (p0-q0) is less than (5 tC +1) > 1)? TRUE: FALSE3.4 for the stronger deblocking filter of the luminance (designed for larger blocks)

When samples on either side of the boundary belong to a large block, a bilinear filter is used. Samples belonging to the large chunk are defined as: for a vertical edge, when width is 32 and for a horizontal edge, height is 32. The bilinear filter is listed below.

Then, block boundary samples pi for i 0 to Sp-1 and block boundary samples qi for j 0 to Sq-1 (pi and qi are the ith sample in the row for filtering the vertical edge or the ith sample in the column for filtering the horizontal edge described in the HEVC deblocking above) are replaced by linear interpolation, as follows:

p_i′＝(f_i*Middle_s，t+(64-f_i)*P_s+32) > 6) cut to p_i±tcPD_i

q_j′＝(g_j*Middle_s，t+(64-g_j)*Q_s+32) > 6) cut to q_j±tcPD_j

Wherein tcPD_iAnd tcPD_jIs the position-dependent cropping described in section 3.6, and g_j、f_i、Middle_s，t、P_sAnd Q_sAs follows:

TABLE 4

Deblocking control of 3.5 chroma

Chroma strong filters are used on both sides of the block boundary. Here, the chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (chroma position) and the following three conditions are satisfied: the first condition is used to decide the boundary strength and the large block. The proposed filter may be applied when the block width or height orthogonal to the block edges is equal to or greater than 8 in the chroma sampling domain. The second and third are essentially the same as HEVC luma deblocking decisions, on/off decisions and strong filter decisions, respectively.

In a first decision, the boundary strength (bS) of the chroma filtering is modified as shown in table 2. The conditions in table 2 were checked in turn. If the condition is satisfied, the remaining conditions of lower priority are skipped.

When bS is equal to 2, or bS is equal to 1 when a large block boundary is detected, chroma deblocking is performed.

The second and third conditions are substantially the same as the HEVC luma strong filter decisions as follows. Under a second condition:

d is then derived as in HEVC luma deblocking. When d is less than β, the second condition is true.

Under a third condition, StrongFilterCondition is derived as follows:

as in HEVC, dpq is derived.

As in HEVC, deriving sp₃＝Abs(p₃-p₀)

As in HEVC, sq is derived₃＝Abs(q₀-q₃)

As in HEVC design, strong filtercondition ═ (dpq is less than (β > >2), sp3+ sq3 is less than (β > >3), and Abs (p0-q0) is less than (5 × tC +1) > > 1.

3.6 Strong deblocking Filter for chroma

The following strong deblocking filters for chrominance are defined:

p₂′＝(3*p₃+2*p₂+p₁+p₀+q₀+4)>>3

p₁′＝(2*p₃+p₂+2*p₁+p₀+q₀+q₁+4)>>3

p₀′＝(p₃+p₂+p₁+2*p₀+q₀+q₁+q₂+4)>>3

the proposed chroma filter performs deblocking on a 4x4 grid of chroma samples.

3.7 position-dependent clipping

Position dependent clipping tcPD is applied to the output samples of the luminance filtering process involving the strong filter and the long filter, which modify 7, 5 and 3 samples at the boundaries. Assuming a quantization error distribution, it is proposed to increase the clipping values of the samples expected to have higher quantization noise, so that the deviation of the reconstructed sample values from the true sample values is expected to be larger.

For each P or Q boundary filtered with an asymmetric filter, depending on the outcome of the decision process in section 3.2, the location-dependent threshold table is selected as side information from the two tables provided to the decoder (i.e., Tc7 and Tc3 listed in the table below):

Tc7＝{6,5,4,3,2,1,1}；

Tc3＝{6,4,2}；

tcPD＝(Sp＝＝3)？Tc3:Tc7；

tcQD＝(Sq＝＝3)？Tc3:Tc7；

for P or Q boundaries filtered with a short symmetric filter, a lower magnitude position-dependent threshold is applied:

Tc3＝{3,2,1}；

after defining the thresholds, the filtered p 'i and q' i sample values are clipped according to tcP and tcQ clipping values:

p″_i＝Clip3(p′_i+tcP_i,p′_i–tcP_i,p′_i)；

q″_j＝Clip3(q′_j+tcQ_j,q′_j–tcQ_j,q′_j)；

where p 'i and q' i are filtered sample values, p "i and q" j are clipped output sample values, and tcPi, tcQi are clipping thresholds derived from the VVC tc parameter and tcPD and tcQD. The function Clip3 is a clipping function specified in VVC.

3.8 subblock deblocking adjustment

In order to implement parallel friendly deblocking using both long filters and sub-block deblocking, the long filters are limited to modify a maximum of 5 samples on the side where sub-block deblocking (AFFINE or ATMVP or DMVR) is used, as shown in luminance control of long filters. In addition, sub-block deblocking is adjusted such that sub-block boundaries close to CU or implicit TU boundaries on an 8x8 grid are constrained to modify a maximum of two samples on each side.

The following applies to sub-block boundaries that are not aligned with CU boundaries.

Where an edge (edge) equal to 0 corresponds to a CU boundary and an edge equal to 2 or equal to orthogonalLength-2 corresponds to 8 samples from a sub-block boundary of a CU boundary or the like. If the implicit nature of TU is used: (implimit), then the implicit TU is true.

3.9 luma/chroma restricted to 4CTU/2CTU line buffers

When the horizontal edge is aligned with the CTU boundary, the filtering of the horizontal edge limits Sp-3 for luminance, Sp-1 and Sq-1 for chrominance.

4. Examples of Intra sub-Block partitioning (ISP)

Fig. 6A shows an example of division of 4 × 8 and 8 × 4 blocks. Fig. 6B shows an example of division of all blocks except for 4 × 8, 8 × 4, and 4 × 4.

In jfet-M0102, an ISP is proposed, as shown in table 3, which divides a luminance intra-predicted block vertically or horizontally into 2 or 4 sub-partitions depending on the size dimension of the block. Fig. 6A and 6B show examples of two possibilities. All sub-partitions satisfy the condition of having at least 16 samples. For block sizes 4 × N or N × 4(N >8), there may be 1 × N or N × 1 sub-partitions, if allowed.

TABLE 5

For each of these sub-partitions, a residual signal is generated by entropy-decoding the coefficients transmitted by the encoder, and then inverse-quantizing and inverse-transforming them. Then, intra prediction is performed on the sub-partitions, and finally, corresponding reconstructed samples are obtained by adding the residual signal to the prediction signal. Thus, the reconstructed value of each sub-partition will be available to generate the next prediction, the process is repeated, and so on. All sub-partitions share the same intra mode.

TABLE 6

5. Example of affine linearly weighted Intra prediction (ALWIP, also known as matrix-based Intra prediction)

Affine linear weighted intra prediction (ALWIP, also known as matrix-based intra prediction (MIP)) is proposed in jfet-N0217.

5.1 generating a simplified prediction Signal by matrix vector multiplication

Neighboring reference samples are first downsampled via averaging (averaging) to produce a reduced reference signal bdry_red. The simplified prediction signal pred is then calculated by calculating the matrix vector product and adding the offset_red。

pred_red＝A·bdry_red+b

Where A is a group having W_red·H_redA matrix of columns, and if W ═ H ═ 4, a has 4 columns, and in all other cases a has 8 columns. b is the dimension W_red·H_redThe vector of (2).

5.2 Explanation of the entire ALWIP Process

The overall process of averaging, matrix vector multiplication, and linear interpolation is shown in fig. 7A-7D for different shapes. Note that the remaining shapes are processed as in one of the cases shown.

Fig. 7A shows an example of Affine Linear Weighted Intra Prediction (ALWIP) for a4 × 4 block. Given a 4x4 block, the ALWIP takes two averages along each axis of the boundary. The resulting four input samples enter a matrix vector multiplication. The matrix being taken from the set S₀. After adding the offset, 16 final predicted samples will be obtained. Linear interpolation is not necessary for generating the prediction signal. Therefore, a total of (4 · 16)/(4 · 4) ═ 4 multiplications are performed per sample.

Fig. 7B shows an example of ALWIP for an 8 × 8 block. Given an 8x8 block, the ALWIP takes four averages along each axis of the boundary. The resulting eight input samples enter matrix vector multiplication. The matrix being taken from the set S₁. This produces 16 samples at odd positions of the prediction block. Therefore, a total of (8 · 16)/(8 · 8) ═ 2 multiplications are performed per sample. After adding the offset, these samples are vertically interpolated by using the narrowed top boundary. By using the originalThe left border of the start is interpolated horizontally.

Fig. 7C shows an example of ALWIP for an 8 × 4 block. Given an 8x4 block, the ALWIP takes four averages along the horizontal axis of the boundary and four original boundary values at the left boundary. The resulting eight input samples enter matrix vector multiplication. The matrix being taken from the set S₁. This produces 16 samples at odd horizontal and each vertical position of the prediction block. Therefore, a total of (8 · 16)/(8 · 4) — 4 multiplications are performed per sample. After adding the offset, the samples are interpolated horizontally by using the original left boundary. The transposed (transposed) case is handled accordingly.

Fig. 7D shows an example of ALWIP for a 16 × 16 block. Given a 16x16 block, the ALWIP takes four averages along each axis of the boundary. The resulting eight input samples enter matrix vector multiplication. The matrix being taken from the set S₂. This results in 64 samples at odd positions of the prediction block. Therefore, a total of (8 · 64)/(16 · 16) ═ 2 multiplications are performed per sample. After adding the offset, these samples are interpolated vertically by using the eight averages of the top boundary. Horizontal interpolation is performed by using the original left border. In this case, the interpolation process does not add any multiplication. Thus, in general, two multiplications are required per sample to compute the ALWIP prediction.

For larger shapes, the process is essentially the same and it is easy to check whether the number of multiplications per sample is less than four.

For a W × 8 block with W >8, only horizontal interpolation is needed, since samples are given at both the odd horizontal and each vertical position.

Finally, for W>W × 4 block of 8, let A_kIs a matrix that is generated by discarding each row corresponding to an odd number along the horizontal axis of the downsampling block. Thus, the output size is 32, again, only horizontal interpolation has to be performed. The transposed (transposed) case is handled accordingly.

Examples of Multiple Transformation Sets (MTS) in VVC

6.1 explicit Multiple Transform Sets (MTS)

In VTM4, a maximum of 64 × 64 large block size transform is enabled, which is mainly used for higher resolution video, such as 1080p and 4K sequences. For a transform block with size (width or height, or both) equal to 64, the high frequency transform coefficients are zeroed out so that only the low frequency coefficients remain. For example, for an M × N transform block, where M is the block width and N is the block height, when M equals 64, only the left 32 columns of transform coefficients are retained. Similarly, when N equals 64, only the transform coefficients of the first 32 rows are retained. When the transform skip mode is used for large blocks, the entire block will be used without zeroing out any values.

In addition to DCT-II, which has been adopted in HEVC, inter and intra coded blocks are residual encoded using a Multiple Transform Selection (MTS) scheme. It uses a number of selected transforms of DCT8/DST 7. The newly introduced transformation matrices are DST-VII and DCT-VIII. Table 7 shows the basic functions of the selected DST/DCT.

To preserve the orthogonality of the transform matrices, the transform matrices are quantized more accurately than the transform matrices in HEVC. In order to keep the median of the transformed coefficients in the 16-bit range, all coefficients should have 10 bits after the horizontal and vertical transforms.

To control the MTS scheme, separate enable flags are specified at the SPS level for intra and inter frames, respectively. When MTS is enabled on SPS, CU level flag is signaled to indicate whether MTS is applied. Here, MTS is only applicable to luminance. The MTS CU level flag will be signaled when the following conditions are met.

-width and height both less than or equal to 32

-CBF flag equal to 1

TABLE 7

If the MTS CU flag is equal to zero, then the DCT2 applies in both directions. However, if the MTS CU flag is equal to 1, two more flags are additionally signaled to indicate the transform type in the horizontal and vertical directions, respectively. The transformation and signaling mapping table is shown in table 8. When transform matrix precision is involved, an 8-bit primary transform kernel is used. Thus, all transform kernels used in HEVC remain the same, including 4-point DCT-2 and DST-7, 8-point, 16-point, and 32-point DCT-2. In addition, other transform kernels (including 64-point DCT-2, 4-point DCT-8, 8-point, 16-point, 32-point DST-7, and DCT-8) all use an 8-bit primary transform kernel.

Table 8: mapping of decoded values of tu _ mts _ idx and corresponding transformation matrices for horizontal and vertical directions

To reduce the complexity of large sizes of DST-7 and DCT-8, the high frequency transform coefficients of DST-7 and DCT-8 with a size (width or height, or both width and height) equal to 32 are zeroed out. Only the coefficients in the 16x16 low frequency region are retained. In addition to applying different transforms, VVC also supports a mode called Transform Skip (TS), which is similar to the TS concept in HEVC. TS is considered a special case of MTS.

6.2 implicit Multiple Transform Set (MTS)

Note that ISP, SBT and MTS are enabled, but all with implicit signaling are considered implicit MTS.

In this patent document, implicitMtsEnabled is used to define whether implicit MTS is enabled.

8.7.4 transform processing for scaling transform coefficients

8.7.4.1 overview

The variable implicitimtsenenabled is derived as follows:

-if sps _ mts _ enabled _ flag is equal to 1, and one of the following conditions is met, then implitmenabled is set equal to 1:

-intrasubportionssplittype is not equal to ISP _ NO _ SPLIT;

-cu sbt flag is equal to 1 and Max (nTbW, nTbH) is less than or equal to 32;

-sps _ explicit _ mts _ INTRA _ enabled _ flag and sps _ explicit _ mts _ inter _ enabled _ flag are both equal to 0, and CuPredMode [ xTbY ] [ yTbY ] is equal to MODE _ INTRA;

-otherwise, implitmtsenabled is set equal to 0.

The variable trTypeHor specifying the horizontal transform kernel and the variable trTypeVer specifying the vertical transform kernel are derived as follows:

-if cIdx is greater than 0, setting trTypeHor and trTypeVer equal to 0.

Otherwise, if implicitMtsEnabled is equal to 1, then the following applies:

-if IntraSubPartitionsSplitType is not equal to ISP _ NO _ SPLIT, then trTypeHor and trTypeVer are specified in tables 8-15 depending on intraPredMode.

Otherwise, if cu _ sbt _ flag is equal to 1, the trTypeHor and trTypeVer are specified in tables 8-14 depending on cu _ sbt _ horizontal _ flag and cu _ sbt _ pos _ flag.

Otherwise (sps _ explicit _ mts _ intra _ enabled _ flag and sps _ explicit _ mts _ inter _ enabled _ flag equal to 0), the derivation of trTypeHor and trTypeVer is as follows:

trTypeHor＝(nTbW>＝4&&nTbW<＝16&&nTbW<＝nTbH)？1:0 (8-1030)

trTypeVer＝(nTbH>＝4&&nTbH<＝16&&nTbH<＝nTbW)？1:0 (8-1031)

otherwise, trTypeHor and trTypeVer are specified in tables 8-13 depending on tu _ mts _ idx [ xTbY ] [ yTbY ].

Tables 8-13-TrTypeHor and TrTypeVer Specifications for TrTypeHor and TrTypeVer depending on tu _ mts _ idx [ x ] [ y ]

tu_mts_idx[x0][y0]	0	1	2	3	4
						trTypeHor	0	1	2	1	2
trTypeVer	0	1	1	2	2

Tables 8-14-specification of trTypeHor and trTypeVer depending on cu _ sbt _ horizontal _ flag and cu _ sbt _ pos _ flag

cu_sbt_horizontal_flag	cu_sbt_pos_flag	trTypeHor	trTypeVer
				0	0	2	1
0	1	1	1
				1	0	1	2
1	1	1	1

Example of simplified quadratic transformation (RST) proposed in JFET-N0193

7.1 indistinguishable quadratic transformation (NSST) in JEM

Fig. 8 shows an example of quadratic transformation in JEM. In JEM, a quadratic transform is applied between the forward main transform and the quantization (at the encoder), and between the inverse quantization and the inverse main transform (at the decoder side). As shown in fig. 8, a 4x4 (or 8x8) quadratic transform is performed according to the block size. For example, for every 8x8 block, a 4x4 quadratic transform is applied to the smaller block (i.e., min (width, height) <8), and an 8x8 quadratic transform is applied to the larger block (i.e., min (width, height) > 4).

The application of the indivisible transformation is described below taking the input as an example. To apply the indivisible transform, the following 4X4 input block X is first represented as a vector

The indivisible transformation is calculated asWhereinIndicating a transform coefficient vector, T is a 16x16 transform matrix. The 16x1 coefficient vector is then scanned using the scan order (horizontal, vertical, or diagonal) of the blockReorganized into 4x4 chunks. The coefficients with smaller indices will be placed in the 4x4 coefficient block with smaller scan indices. There are a total of 35 transform sets, each using 3 indivisible transform matrices (kernels). The mapping from the intra prediction mode to the transform set is predefined. For each transform set, the selected undivided quadratic transform (NSST) candidate is further specified by an explicitly signaled quadratic transform index. After transforming the coefficients, each intra CU is signaled once an index in the bitstream.

7.2 simplified quadratic transformation (RST) in JFET-N0193

Fig. 9 illustrates an example of a simplified quadratic transformation (RST) based on an embodiment of the disclosed technology. RST (low frequency inseparable transform (LFNST)) was introduced in JFET-K0099, and 4 transform set (instead of 35 transform set) mappings were introduced in JFET-L0133. In this jfet-N0193, 16x48 and 16x16 matrices are used for 8x8 and 4x4 blocks, respectively. For convenience, the 16x48 transform is denoted as RST8x8, and the 16x16 transform is denoted as RST4x 4.

7.3 application of RST

The choice of RST is defined as follows:

for 4 × N or N × 4 blocks (N < ═ 8), RST4x4 is applied to the top left 4 × 4CG (e.g. 4x4, 4x8, 8x4)

Applying RST4x4 twice to two adjacent top left 4x4 CGs (e.g. 4x16, 16x4, etc.) for 4x N or N x4 blocks (N >8)

For all other cases, a quadratic transform is applied to the top left three 4x4 CGs to generate the top left 4x4CG and the other two CGs with coefficients of zero.

8. Examples of sub-block transformations

For inter-predicted CUs with CU cbf equal to 1, CU sbt flag may be signaled to indicate whether the entire residual block or a sub-portion of the residual block was decoded. In the former case, the inter-frame MTS information is further parsed to determine the transform type of the CU. In the latter case, a part of the residual block is encoded by the inferred adaptive transform, while another part of the residual block is zeroed out. SBT is not applicable to combined inter-frame intra mode.

FIG. 10 shows examples of sub-block transform modes SBT-V and SBT-H. In the sub-block transform, a position-dependent transform is applied to a luminance transform block in SBT-V and SBT-H (always using the chroma TB of DCT-2). The two positions of SBT-H and SBT-V are associated with different core transformations. More specifically, the horizontal and vertical transforms of each SBT location are specified in fig. 10. For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is greater than 32, the corresponding transform is set to DCT-2. Thus, the sub-block transforms collectively specify TU slices, cbf, and horizontal and vertical transforms of the residual block, which may be considered a syntax shortcut for the case where the main residual of the block is on one side of the block.

Example of Joint chroma residual coding in JVT-N0054

Since the Cb and Cr residuals seem to be inversely proportional to each other, it is proposed to exploit this phenomenon and introduce a mode for jointly encoding the chroma residuals. In this mode, a single residual is indicated for both chroma blocks of the transform unit. The indicated residual is added to the prediction block in the first channel (usually denoted Cb) and subtracted from the prediction block in the second channel (usually denoted Cr).

If the coded block flag (cbf) for both Cb and Cr is true, the joint residual mode is indicated with a flag in the bitstream. If this mode is activated, a single residual block will be decoded. The bitstream syntax and decoding process of the joint residual block follows the Cb residual in VTM-4. The residual of the Cr block is generated by negating the decoded residual. Since a single residual is used to represent the residuals of two blocks, it may be generally desirable that the QP used for this mode is lower than the QP used for separate encoding of the chroma residual. In CE testing, chroma QP offset-1 is used for joint mode, while +1 is used for separate chroma coding (as specified in the jfet general test conditions).

Example of JFET-N0236

This contribution proposes a method to refine sub-block based affine motion compensated prediction with optical flow. After performing sub-block based affine motion compensation, the prediction samples are refined by adding the differences derived from the optical flow equations, which is called prediction refinement with optical flow (PROF). The proposed method can achieve inter prediction at pixel level granularity without increasing memory access bandwidth.

To achieve finer granularity of motion compensation, this contribution proposes a method to refine sub-block based affine motion compensation prediction with optical flow. After performing sub-block based affine motion compensation, the luma prediction samples are refined by adding the differences derived by the optical flow equations. The proposed PROF (predictive refinement with optical flow) is described as the following four steps.

Step 1) performs sub-block based affine motion compensation to generate sub-block prediction I (I, j).

Step 2) use 3 tap filter [ -1, 0, 1[ -1]Calculating the spatial gradient g of the subblock prediction at each sample point position_x(i, j) and g_y(i，j)。

g_x(i，j)＝I(i+1，j)-I(i-1，j)

g_y(i，j)＝I(i，j+1)-I(i，j-1)

Fig. 11 shows an example of the subblock MV VSB and the pixel Δ v (i, j). The sub-block prediction expands one pixel on each side for gradient calculation. To reduce memory bandwidth and complexity, pixels on the extended bounding box are copied from the nearest integer pixel position in the reference picture. Thus, additional interpolation of the fill area is avoided.

Step 3) the luminance prediction refinement (expressed as Δ I) is calculated by the optical flow equation.

ΔI(i，j)＝g_x(i，j)*Δv_x(i，j)+g_y(i,j)*Δv_y(i,j)

Where the increment MV (denoted as Δ v (i, j)) is the difference between the pixel MV (denoted by v (i, j)) calculated for the sampling position (i, j) and the subblock MV of the subblock to which the pixel (i, j) belongs, as shown in fig. 11.

Since the affine model parameters and pixel position relative to the center of the sub-blocks do not vary between sub-blocks, Δ v (i, j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let x and y be the horizontal and vertical offsets from the pixel location to the center of the sub-block, then Δ v (x, y) can be derived by the following equation:

for a 4-parameter affine model,

for a6 parameter affine model

Wherein (v)_0x，v_0y)、(v_1x，v_1y)、(v_2x，v_2y) Are the top left, top right and bottom left control point motion vectors, and w and h are the width and height of the CU.

Step 4) finally, add the luma prediction refinement to the sub-block prediction I (I, j). The final prediction I' is generated according to the following formula:

I′(i，j)＝I(i，j)+ΔI(i，j)

11. disadvantages of the existing embodiments

In some prior implementations (e.g., current VVC/VTM deblocking designs), the following problems are encountered:

(1) BDOF refines each predicted sample. Therefore, they may not need to be filtered again in the deblocking process.

(2) In the deblocking filtering process, the CIIP coded block is regarded as an inter prediction mode, and the intra prediction process still needs to be performed.

12. Exemplary method of improving the Square Block Filtering (DB)

Embodiments of the presently disclosed technology overcome the disadvantages of the prior implementations, thereby providing video codecs with higher coding efficiency. The improvement of deblocking filtering (DB) that can enhance existing and future video codec standards based on the disclosed techniques is set forth in the following examples described for various embodiments. The examples of the disclosed technology provided below illustrate the general concepts and are not meant to be construed as limiting. In examples, various features described in these examples may be combined unless explicitly indicated to the contrary.

In some embodiments of the disclosed technology, it is proposed to modify the deblocking filtering (DB) process in several ways. However, it should be noted that the methods described in this document may also be applicable to other types of in-loop filtering methods (e.g., SAO, ALF) and post-reconstruction filters (e.g., Hadamard transform domain filters, bilateral filters, diffusion filters applied to the predicted signal or reconstructed signal before deblocking filters).

Note that the chroma components may represent Cb/Cr color components or B/R color components for RGB format. In the following description, we take "Cb/Cr" as an example.

Example 1.Whether DB is enabled and/or how filtering is applied in DB processing may depend on the use of BDOF and/or other coding tools that use optical flow to refine motion/prediction samples/reconstruction samples.

(a) In one example, for samples located within a BDOF coded block, the DB need not be applied.

(b) Alternatively, the boundary strength/filter strength may depend on the use of BDOF.

(i) Alternatively, for samples located within a BDOF coded block, a weaker filter may be applied.

Example 2.Whether DB is enabled and/or how filtering is applied in DB processing may depend on the transform matrix type (e.g., DCT-II, DST-VII, DCT-VIII, transform skip mode (with identification matrix)) and the encoding method (such as RDPCM/QR-BDPCM mode) that does not apply the transform.

(a) In one example, position-dependent clipping may be disabled for samples to be filtered that are located at blocks coded with a particular transform type (e.g., transform skip coded blocks).

(i) Alternatively, and in addition, the same clipping parameters are used for all samples to be filtered.

(b) In one example, the position-dependent clipping parameter may depend on the basis of a transformation matrix applied to the block.

(c) In one example, DB may not be performed for blocks that transform skip mode or do not apply the coding method of the transform (such as RDPCM mode).

Example 3.Clipping operations and/or location-dependent clipping may be disabled based on:

(a) video content (e.g., screen content or natural content)

(b) A message signaled in a SPS/VPS/PPS/picture header/slice group header/Largest Coding Unit (LCU)/Coding Unit (CU)/LCU row/LCU group.

Example 4.Whether DB is enabled and/or how filtering is applied in DB processing may depend on the use of Combined Intra Inter Prediction (CIIP).

(a) In one example, CIIP coded blocks may be treated similarly to intra-coded blocks.

(b) In one example, the boundary strength is set to 2 when at least one of the two blocks crossing the edge is coded in CIIP mode.

(c) Alternatively, when only one of two blocks crossing an edge is encoded in the CIIP mode and the other is inter-encoded, the boundary strength is set to 1.

(d) Alternatively, when only one of two blocks crossing an edge is encoded in the CIIP mode and the other is inter-frame or IBC encoded, the boundary strength is set to 1.

Example 5.The Boundary Strength (BS) of an IBC encoded block may be independent of the motion vector difference.

(a) In one example, when neighboring blocks are all IBC encoded, the boundary strength is set to M (e.g., M-0/1/2) regardless of MV differences and regardless of whether there are non-zero coefficients.

(b) Alternatively, the BS depends on whether the horizontal and/or vertical MV difference is greater than M0/M1 pixels (M > 1).

(i) In one example, both M0 and M1 are set to 4.

(ii) In one example, M0 is set to the minimum CU width for the MV difference of the horizontal component, and M1 is set to the minimum CU height for the MV difference of the horizontal component.

(iii) In one example, M0 is set to the minimum allowed IBC coding CU width for MV differences of horizontal components, and M1 is set to the minimum allowed IBC coding CU height for MV differences of horizontal components.

(c) Alternatively, the DB may not be applied to the boundary between two IBC coding blocks.

(d) Alternatively, the DB may not be applied to the samples of the IBC coded blocks.

Example 6.The Boundary Strength (BS) values of the P-side or Q-side may be different.

(a) In one example, a block on the P-side or Q-side may have its own boundary strength. Deblocking performed on the P-side and Q-side may be based on BS values on the P-side and Q-side, respectively.

(i) In one example, if the block on the P-side is an intra block and the block on the Q-side is an RDPCM encoded block, the deblocking condition may be checked on the P-side and deblocking may be performed only on the P-side.

(ii) In one example, the BSs on the P and Q sides may depend on the QPs on both sides.

(iii) In one example, the BSs of the P-side and the Q-side may depend on the LMCS coefficients used in both sides.

Example 7.In deblocking, an RDPCM encoded block may be considered an intra block in deblocking.

(a) In one example, if the RDPCM encoded block is located on the P-side and/or the Q-side, the Boundary Strength (BS) value may be set to 1 or 2.

Example 8.The diagonal edges in a delta-mode coded block may be deblocked.

(a) In one example, the deblocking filter and/or deblocking control logic in the current VVC may also be applied to diagonal edges in the delta-mode coded block.

Example 9.

(a) In one example, if the palette coding block is on the Q side and the blocks not coded in palette mode are on the P side, deblocking may be applied on the P side.

(b) In one example, if the palette coding block is on the P side and the block that is not coded in palette mode is on the Q side, deblocking may be applied on the Q side.

(c) In one example, if both the P-side and Q-side are palette coded blocks, deblocking may not be applied on both sides.

Example 10.Sub-regions within a block that are encoded with DMVR or other decoder-side motion refinement coding tools may be considered sub-block boundaries and may be filtered.

(a) In one example, the sub-region is defined as a sub-unit to which motion vector refinement is applied.

(i) In one example, a sub-region is defined as a MxN sub-block within a DMVR coded block, where M and N may be set to 64 or 16.

Example 11.Whether DB is enabled and/or how filtering is applied in DB processing may depend on whether a quadratic transform (also known as NSST or RST or LFNST) is applied.

(a) In one example, the boundary of the sub-region where RST is applied may be considered a transform block boundary or a coded sub-block boundary.

(i) Alternatively, the samples located in the sub-area to which the RST is applied may be filtered.

(ii) Alternatively, position-dependent clipping may be disabled for sub-regions to which RST is applied.

Example 12.Whether deblocking is performed on sub-block boundaries (e.g., intra sub-block boundaries) within an affine mode block may depend on whether interleaved prediction or/and PROF or/and phase-change (phase-variant) affine sub-block motion compensation is applied to the block.

(a) In one example, deblocking may be disabled when interleaved prediction or/and PROF or/and phase change affine sub-block motion compensation is applied to a block.

(b) In one example, deblocking may be enabled when interleaved prediction or/and PROF or/and phase change affine sub-block motion compensation is not applied to the block.

Fig. 12 illustrates a flow diagram of an example method 1200 for video processing in accordance with some embodiments of the disclosed technology. The method 1200 for video processing includes, at step 1210, performing determining an encoding scheme suitable for conversion between a video block and a bitstream representation of the video block, at step 1220, determining a mode of in-loop filtering processing including at least one of deblocking filtering (DB) processing, Sample Adaptive Offset (SAO), and Adaptive Loop Filtering (ALF) based on the determination of the encoding scheme, and, at step 1230, performing conversion using the mode of in-loop filtering. Other features related to the method 1200 for video processing may be found in example 1, example 2, example 3, example 11 discussed above.

Fig. 13 illustrates a flow diagram of another example method 1300 for video processing, in accordance with some embodiments of the disclosed technology. The method 1300 for video processing includes, at step 1310, determining that Combined Intra Inter Prediction (CIIP) encoding is applicable to a current block, and, at step 1320, performing a conversion between the current block and a bitstream representation of the current block, and, based on the use of CIIP, performing a deblocking filtering (DB) process by: identifying boundaries between blocks and vertical and horizontal edges; calculating the boundary strength; deciding whether to turn the filter on or off; and selecting the strength of the filter. Other features related to the method 1300 for video processing may be found in example 4 discussed above.

Fig. 14 illustrates a flow diagram of another example method 1400 for video processing, in accordance with some embodiments of the disclosed technology. The method 1400 for video processing includes, at step 1410, determining whether neighboring blocks are all encoded in Intra Block Copy (IBC), and, at step 1420, performing a deblocking filtering (DB) process by identifying boundaries between blocks and vertical and horizontal edges, calculating boundary strengths, deciding whether to turn on or off filters, and selecting a strength of a filter based on the determination. The boundary strength is independent of the motion vector difference. Other features related to the method 1400 for video processing may be found in example 5 discussed above.

Fig. 15 illustrates a flow diagram of another example method 1500 for video processing in accordance with some embodiments of the disclosed technology. The method 1500 for video processing includes, at step 1510, determining boundary strengths on the P-side and the Q-side, and, at step 1520, performing deblocking filtering (DB) processing on the P-side and the Q-side by identifying boundaries between blocks and vertical and horizontal edges, calculating the boundary strengths, determining whether to turn on or off a filter, and selecting the strength of the filter, based on the determination. Other features related to the method 1500 for video processing may be found in example 6 discussed above.

Fig. 16 illustrates a flow diagram of another example method 1600 for video processing in accordance with some embodiments of the disclosed technology. The method 1600 for video processing includes, at step 1610, determining a Residual Differential Pulse Code Modulation (RDPCM) coded block, and, at step 1620, performing deblocking filtering (DB) processing on the P-side and Q-side by identifying boundaries between the block and vertical and horizontal edges, calculating boundary strengths, determining whether to turn on or off a filter, and selecting a strength of the filter, based on the determination. RDPCM encoded blocks are treated as intra blocks in a deblocking filtering (DB) process. Other features related to the method 1600 for video processing may be found in example 7 discussed above.

Fig. 17 illustrates a flow diagram of another example method 1700 for video processing in accordance with some embodiments of the disclosed technology. The method 1700 for video processing includes, at step 1710, determining a diagonal edge in a delta-mode encoded block, and, at step 1720, performing a deblocking filtering (DB) process by identifying a boundary between the block and vertical and horizontal edges, calculating a boundary strength, determining whether to turn on or off a filter, and selecting a strength of the filter based on the determination. Other features related to the method 1700 for video processing may be found in example 8 discussed above.

Fig. 18 illustrates a flow diagram of another example method 1800 for video processing in accordance with some embodiments of the disclosed technology. The method 1800 for video processing includes, at step 1810, determining whether a palette coding block is on the P-side or the Q-side, and, at step 1820, performing deblocking filtering (DB) processing on diagonal boundaries by identifying boundaries between blocks and vertical and horizontal edges, calculating boundary strengths, determining whether to turn on or off filters, and selecting a strength of a filter, based on the determination. In case the palette coding block is located on the P-side or the Q-side, a deblocking filtering (DB) process is applied on one side. Other features related to the method 1800 for video processing may be found in example 9 discussed above.

Fig. 19 shows a flowchart of another example method 1900 for video processing, in accordance with some embodiments of the disclosed technology. Method 1900 for video processing includes, at step 1910, determining a subregion within a block that is encoded with decoder-side motion vector refinement (DMVR) or other decoder-side motion refinement encoding tools, and, at step 1920, performing a deblocking filtering (DB) process on the subregion by identifying boundaries between the block and vertical and horizontal edges, calculating boundary strength, determining whether to turn on or off a filter, and selecting a strength of the filter, based on the determination. Other features related to the method 1900 for video processing may be found in example 10 discussed above.

Fig. 20 is a block diagram of the video processing apparatus 2000. Apparatus 2000 may be used to implement one or more of the methods described herein. The apparatus 2000 may be embodied in a smartphone, tablet, computer, internet of things (IoT) receiver, or the like. The apparatus 2000 may include one or more processors 2002, one or more memories 2004, and video processing circuitry 2006. The processor 2002 may be configured to implement one or more of the methods described herein (including, but not limited to, methods 1200 and 1900). The memory(s) 2004 may be used for storing data and code for implementing the methods and techniques described herein. The video processing circuit 2006 may be used to implement some of the techniques described in this document in hardware circuits.

In some embodiments, the video codec method may be implemented using an apparatus implemented on a hardware platform as described with respect to fig. 20.

Fig. 21 is a block diagram illustrating an example video processing system 2100 in which various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 2100. The system 2100 may include an input 2102 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10 bit multi-component pixel values), or may be received in a compressed or encoded format. Input 2102 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as ethernet, Passive Optical Network (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

The system 2100 may include an encoding component 2104 that may implement various encoding or encoding methods described herein. The encoding component 2104 can reduce an average bit rate of video from an input 2102 to an output of the encoding component 2104 to produce an encoded representation of the video. Thus, the encoding techniques are sometimes referred to as video compression or video transcoding techniques. The output of the encoding component 2104 can be stored or transmitted via a connected communication, as represented by component 2106. A stored or transmitted bitstream (or encoded) representation of the video received at input 2102 can be used by component 2108 to generate pixel values or displayable video that is sent to display interface 2110. The process of generating user-visible video from a bitstream representation is sometimes referred to as video decompression. Furthermore, although certain video processing operations are referred to as "encoding" operations or tools, it should be understood that encoding tools or operations are used at the encoder and corresponding decoding tools or operations that reverse the results of the encoding will be performed by the encoder.

Examples of a peripheral bus interface or display interface may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, among others. Examples of storage interfaces include SATA (serial advanced technology attachment), PCI, IDE interfaces, and the like. The techniques described herein may be embodied in various electronic devices such as mobile phones, laptops, smart phones, or other devices capable of performing digital data processing and/or video display.

FIG. 22 shows a flow diagram of an example method for visual media processing. In section 12 of this document, the steps of this flowchart are discussed in connection with example 4. At step 2202, the process identifies boundaries of vertical edges and/or horizontal edges of two video blocks. At step 2204, the process calculates a boundary strength of the filter based on at least one of the two video blocks that span either a vertical edge or a horizontal edge encoded using a Combined Intra Inter Prediction (CIIP) mode. At step 2206, the process decides whether to turn the filter on or off. At step 2208, with the filter turned on, the process selects the strength of the filter. In step 2210, the process performs a deblocking filter (DB) process on the video block based on the selection.

FIG. 23 shows a flow diagram of an example method for visual media processing. In section 12 of this document, the steps of this flowchart are discussed in connection with example 1. In step 2302, the process determines an encoding scheme applicable to conversion between video blocks of visual media data and a bitstream representation of the video blocks. Based on the determination of the encoding scheme, the process selectively enables or disables the in-loop filter to be applied to the video block during the transition of the video block at step 2304.

FIG. 24 shows a flow diagram of an example method for visual media processing. In section 12 of this document, the steps of this flowchart are discussed in connection with example 2. In step 2402, the process determines the type of transform matrix applicable to the conversion between the video block of the visual media data and the bitstream representation of the video block. Based on the determination of the type of transform matrix, the process selectively enables or disables the application of a Deblocking (DB) filtering process to the video block during the conversion of the video block at step 2404.

FIG. 25 shows a flow diagram of an example method for visual media processing. In section 12 of this document, the steps of the flowchart are discussed in connection with example 3. At step 2502, during a transition between a video block of visual media data and a bitstream representation of the video block, the process determines to apply an in-loop filtering step to the video block, wherein the in-loop filtering step includes a clipping operation applied to samples in the video block to be filtered. At step 2504, the process selectively enables or disables the cropping operation in response to determining one or more conditions associated with the video block.

Some embodiments disclosed herein are now presented in a clause-based format.

X1. a method for visual media processing, comprising:

identifying boundaries at vertical edges and/or horizontal edges of two video blocks;

calculating a boundary strength of a filter based on encoding at least one of two video blocks spanning a vertical edge or a horizontal edge using a Combined Intra Inter Prediction (CIIP) mode;

deciding whether to turn the filter on or off;

selecting a strength of the filter in case the filter is turned on; and

based on the selection, a deblocking filter (DB) process is performed on the video block.

X2. the method according to clause X1, wherein CIIP encoded blocks are considered intra-encoded blocks when calculating boundary strength.

X3. the method of clause X1, wherein, if at least one of the two video blocks that span a vertical or horizontal edge is coded with CIIP, then the boundary strength is selected to be 2.

X4. the method according to clause X1, wherein the boundary strength is selected to be 1 in case only one of two blocks across a vertical or horizontal edge is CIIP encoded and the other video block is inter-coded.

X5. the method according to clause X1, wherein the boundary strength is selected to be 1 in case only one of the two blocks crossing the edge is CIIP encoded and the other video block is inter or Intra Block Copy (IBC) encoded.

A1. A method for visual media processing, comprising:

determining a coding scheme applicable to transitions between video blocks of visual media data and bit stream representations of the video blocks; and

the application of the in-loop filter to the video block is selectively enabled or disabled during the transition of the video block based on the determination of the encoding scheme.

A2. The method of clause a1, wherein the in-loop filter comprises one of: deblocking filters (DB), Sample Adaptive Offset (SAO) filters, and/or adaptive in-loop filters (ALF).

A3. The method according to any one or more of clauses a1-a2, wherein the coding scheme comprises refining predicted samples and/or motion information and/or reconstructed samples using optical flow.

A4. The method of clause a3, wherein the coding scheme that uses optical flow to refine the predicted samples comprises bi-directional optical flow (BDOF).

A5. The method according to clause a4, wherein the determining of the coding scheme comprises: identifying whether a predicted sample point is located within a bidirectional optical flow (BDOF); and disabling application of an in-loop filter in response to determining that the predicted samples are within a bidirectional optical flow (BDOF).

A6. The method of clause a5, wherein at least one of the boundary strength and/or the filter strength of the in-loop filter is determined based on the use of bi-directional optical flow (BDOF).

A7. The method according to clause a4, wherein the determining of the coding scheme comprises: identifying whether the predicted samples are located within a bi-directional optical flow (BDOF), and wherein, in response to determining that the predicted samples are located within the bi-directional optical flow (BDOF), enabling application of an in-loop filter, and further wherein the strength of the in-loop filter is weaker than the strength of another in-loop filter associated with predicted samples that are not located within the BDOF.

B1. A method for visual media processing, comprising:

determining a type of transform matrix applicable to a transition between a video block of visual media data and a bitstream representation of the video block; and

based on the determination of the type of transform matrix, applying a Deblocking (DB) filtering process to the video block is selectively enabled or disabled during conversion of the video block.

B2. The method according to clause B1, wherein the type of transform matrix comprises DCT-II, DST-VII, DCT-VIII and/or transform skip mode.

B3. The method of any one or more of clauses B1-B2, wherein the applying of the deblocking filter process further comprises a clipping operation.

B4. The method of clause B3, wherein the clipping operation is location dependent based on the location of samples in the video block relative to the DB filter.

B5. The method of clause B4, wherein the clipping operation is disabled for samples located in transform skip coded video blocks.

B6. The method of clause B5, wherein the deblocking filtering (DB) process is disabled on blocks that are transform skip coded or when it is determined that video block coding is exempt from transform based coding techniques.

B7. The method of clause B6, wherein the transform-based coding technique comprises Residual Differential Pulse Code Modulation (RDPCM) or quantized residual differential pulse code modulation (QR-BDPCM).

B8. The method according to clause B3, wherein, if the DB filtering process is enabled for a group of samples, all samples included in the group will be subjected to the same parameters of the clipping operation.

B9. The method according to clause B4, wherein the parameters of the position-dependent clipping operation are related to basis functions of a transformation matrix applicable to the transformation.

C1. A method for visual media processing, comprising:

determining to apply an in-loop filtering step to a video block during a transition between the video block of visual media data and a bitstream representation of the video block, wherein the in-loop filtering step comprises a clipping operation applied to samples in the video block to be filtered; and

the cropping operation is selectively enabled or disabled in response to determining one or more conditions associated with the video block.

C2. The method of clause C1, wherein the one or more conditions associated with the video block include:

(a) content of video blocks, or

(b) Sequence Parameter Set (SPS)/Video Parameter Set (VPS)/Picture Parameter Set (PPS)/picture header/slice group header/Largest Coding Unit (LCU)/Coding Unit (CU)/LCU line/LCU group.

C3. The method of any one or more of clauses X1-C2, wherein the converting comprises generating the bitstream representation from the video block.

C4. The method of any one or more of clauses X1-C2, wherein converting comprises generating pixel values of the video block from the bitstream representation.

C5. A video encoder device, comprising: a processor configured to implement the method of any one or more of clauses X1-C2.

C6. A video decoder device, comprising: a processor configured to implement the method of any one or more of clauses X1-C2.

C7. A computer readable medium having code stored thereon, the code embodying processor executable instructions for performing the method of any one or more of clauses X1-C2.

In this document, the term "video processing" or "visual media processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of the video to a corresponding bitstream representation, and vice versa. As defined by the syntax, the bitstream representation of the current video block may, for example, correspond to bits collocated or interspersed in different locations within the bitstream. For example, a block may be encoded according to the transform and encoded error residual values and also using bits in the header and other fields in the bitstream. Here, the block may be a pixel grouping according to the performed operation, such as a coding unit, a prediction unit, or a transform unit. Furthermore, as described in the above solutions, during the transition, the decoder may parse the bitstream based on the determination with known presence or absence of certain fields. Similarly, the encoder may determine whether certain syntax fields are included and generate the encoded representation accordingly by including or excluding syntax fields from the encoded representation.

From the foregoing it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the invention. Accordingly, the presently disclosed technology is not limited, except as by the appended claims.

Embodiments of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of transitory and non-transitory computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The terms "data processing unit" and "data processing apparatus" include all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, be considered exemplary only, with the examples being meant as examples. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, the use of "or" is intended to include "and/or" unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Also, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few embodiments and examples have been described and other embodiments, enhancements and variations can be made based on what is described and illustrated in this patent document.