Method and apparatus for intra-inter prediction for video encoding and decoding
阅读说明：本技术 用于视频编解码的帧内-帧间预测的方法及装置 (Method and apparatus for intra-inter prediction for video encoding and decoding ) 是由 赖贞延 陈庆晔 徐志玮 黄毓文 于 2018-06-07 设计创作，主要内容包括：本申请公开一种视频编解码的方法及装置。依据一种方法,当前块中的边界区域被确定。对于边界区域中的每个像素,帧内预测子和帧间预测子可以被推导出。透过混合帧内预测子和帧间预测子,帧内-帧间预测子被生成。使用帧内-帧间预测子,当前块的边界区域中的像素被编码或解码。对于当前块中不属于边界区域的剩余像素,使用帧间预测子或帧内预测子,这些像素被编码或解码。在另一方法中,透过将帧内加权因子与帧间加权因子之和缩放成2的幂,帧内-帧间预测流程被实现,以避免除法操作的需要。在又一方法中,帧内-帧间预测基于当前块尺寸被应用。(The application discloses a video coding and decoding method and device. According to one method, a boundary region in a current block is determined. For each pixel in the boundary region, an intra predictor and an inter predictor may be derived. By mixing the intra predictor and the inter predictor, an intra-inter predictor is generated. Pixels in the boundary area of the current block are encoded or decoded using the intra-inter predictor. For the remaining pixels of the current block, which do not belong to the boundary region, the pixels are encoded or decoded using the inter predictor or the intra predictor. In another approach, an intra-inter prediction process is implemented by scaling the sum of the intra weighting factor and the inter weighting factor to a power of 2 to avoid the need for a division operation. In yet another approach, intra-inter prediction is applied based on the current block size.)
1. A method of video coding performed by a video encoder or a video decoder, the method comprising:
receiving, at an encoder side, data related to a current block in a current image, or, at a video decoder side, a video bitstream comprising compressed data of the current block;
for each boundary region pixel in the boundary region in the current block:
deriving an intra predictor from intra reference pixels located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture, the intra reference pixels being coded prior to the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating an intra-inter predictor by mixing the intra predictor and the inter predictor; and
encoding or decoding each boundary region pixel using the intra-inter predictor; and
for a plurality of remaining pixels of the current block that do not belong to the boundary region:
the plurality of remaining pixels are encoded or decoded using an inter predictor or an intra predictor.
2. The method of claim 1, wherein the boundary region corresponds to at least one of a first N rows of the current block near the intra-reference pixels and a first N columns of the current block near the intra-reference pixels, wherein N is an integer greater than 0.
3. The method of claim 2, wherein a position associated with a maximum gradient in the first column or the first row of the current block is derived based on the intra-predictor or inter-predictor or a difference between the intra-predictor and the inter-predictor, wherein the position is used to determine the value of N.
4. The method of video coding according to claim 2,
a position associated with a maximum gradient located at a first column or a first row of the current block is derived based on the intra predictor or the inter predictor, or a difference value between the intra predictor and the inter predictor; and
if the maximum gradient is substantially greater than an average gradient of a plurality of vertical gradients and a plurality of horizontal gradients calculated for the first column or the first row of the current block, the position is used to determine a value of N, otherwise N is set to a default value.
5. The method of claim 2, wherein a position associated with a maximum difference between the intra predictor and a plurality of neighboring reconstructed pixels, between the inter predictor and the plurality of neighboring reconstructed pixels, or between the inter predictor and the intra predictor is derived and used to determine the value of N.
6. The method of claim 2, wherein N is predefined or signaled based on the size of the current block.
7. The method of claim 1, further comprising:
detecting model edges in the current block based on the inter predictor, the intra predictor, or a difference between the intra predictor and the inter predictor; and
the bounding region is determined based on the plurality of model edges.
8. The method of claim 7, wherein detecting the plurality of model edges comprises:
the maximum gradient is calculated based on the inter predictor, the intra predictor, or a difference between the intra predictor and the inter predictor for each row or each column depending on whether an intra prediction mode associated with the intra predictor is close to a vertical direction or a horizontal direction.
9. The method of claim 8, wherein the boundary region comprises an edge pixel in each row or column.
10. The method of claim 9, wherein the border region further comprises any neighboring pixels adjacent to an edge pixel in a same row or a same column of the current block.
11. The method of claim 10, wherein the boundary region further comprises any additional pixels in a current row or column between the current edge pixel in the current row or column and a previous edge pixel in a previous row or column.
12. The method of claim 9, wherein for a target pixel in the current block that does not belong to the boundary region, if the target pixel is located above or to the left of an edge pixel in the same row or column as the target pixel, the target pixel is encoded or decoded using the intra predictor.
13. The method of claim 9, wherein for a target pixel in the current block that does not belong to the boundary region, if the target pixel is located below or to the right of an edge pixel in the same row or column as the target pixel, the target pixel is encoded or decoded using the inter predictor.
14. An apparatus for video coding, in a video coding system, the apparatus comprising one or more electronic devices or processors configured to:
receiving, at an encoder side, data related to a current block in a current image, or, at a video decoder side, a video bitstream comprising compressed data of the current block;
for each boundary region pixel in the boundary region:
deriving an intra predictor from intra reference pixels located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture, the intra reference pixels being coded prior to the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating an intra-inter predictor by mixing the intra predictor and the inter predictor; and
encoding or decoding each boundary region pixel using the intra-inter predictor; and
for a plurality of remaining pixels of the current block that do not belong to the boundary region:
the plurality of remaining pixels are encoded or decoded using an inter predictor or an intra predictor.
15. A method of video coding performed by a video encoder or a video decoder, the method comprising:
receiving, at an encoder side, data related to a current block in a current image, or, at a video decoder side, a video bitstream comprising compressed data of the current block;
deriving an intra predictor from intra reference pixels located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture, the intra reference pixels being coded prior to the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating a weighted sum of the intra predictor and the inter predictor, wherein generating the weighted sum of the intra predictor and the inter predictor comprises:
adding a weighted intra predictor derived by multiplying the intra predictor by an intra weighting factor and a weighted inter predictor derived by multiplying the inter predictor by an inter weighting factor, the sum of the intra weighting factor and the inter weighting factor scaled to a power of 2;
generating a normalized intra-inter predictor by applying a right shift operation to the weighted sum of the intra predictor and the inter predictor; and
encoding or decoding the current block using the normalized intra-inter predictor.
16. The method of claim 15, wherein generating the weighted sum of the intra predictor and the inter predictor comprises:
adding an offset value to the result of the addition of the weighted intra predictor and the weighted inter predictor.
17. An apparatus for video coding, in a video coding system, the apparatus comprising one or more electronic devices or processors configured to:
receiving, at an encoder side, input data related to a current block in a current image, or, at a video decoder side, a video bitstream including compressed data of the current block;
deriving an intra predictor from intra reference pixels located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture, the intra reference pixels being coded prior to the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating a weighted sum of the intra predictor and the inter predictor by adding a weighted intra predictor derived by multiplying the intra predictor by an intra weighting factor and a weighted inter predictor derived by multiplying the inter predictor by an inter weighting factor, the sum of the intra weighting factor and the inter weighting factor being scaled to a power of 2;
generating a normalized intra-inter predictor by applying a right shift operation to the weighted sum of the intra predictor and the inter predictor; and
encoding or decoding the current block using the normalized intra-inter predictor.
18. A method of video coding performed by a video encoder or a video decoder, the method comprising:
receiving, at an encoder side, input data related to a current block in a current image, or, at a video decoder side, a video bitstream including compressed data of the current block;
disabling an intra-inter prediction mode for the current block if the size of the current block is greater than a maximum block size or less than a minimum block size;
when the intra-inter prediction mode is selected for the current block, then:
deriving an intra predictor from a plurality of reference pixels, wherein the plurality of intra reference pixels are located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture and the plurality of intra reference pixels are coded before the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating an intra-inter predictor by mixing the intra predictor and the inter predictor; and
encoding or decoding the current block using the intra-inter predictor; and
when the intra-inter prediction mode is disabled for the current block, then:
deriving the intra predictor or deriving the inter predictor; and
encoding or decoding the current block using the inter predictor or the intra predictor.
19. An apparatus for video coding, in a video coding system, the apparatus comprising one or more electronic devices or processors configured to:
receiving, at an encoder side, input data related to a current block in a current image, or, at a video decoder side, a video bitstream including compressed data of the current block;
disabling an intra-inter prediction mode for a current block if the size of the current block is greater than a maximum block size or less than a minimum block size;
when the intra-inter prediction mode is selected for the current block, then:
deriving an intra predictor from a plurality of reference pixels, wherein the plurality of intra reference pixels are located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture and the plurality of intra reference pixels are coded before the current block;
deriving an inter predictor comprising a plurality of inter reference pixels located in a reference block from a reference picture, wherein the reference block is coded before the current block;
generating an intra-inter predictor by mixing the intra predictor and the inter predictor; and
encoding or decoding the current block using the intra-inter predictor; and
when the intra-inter prediction mode is disabled for the current block, then:
deriving the intra predictor or deriving the inter predictor; and
encoding or decoding the current block using the inter predictor or the intra predictor.
Technical Field
The present application relates to video codec inter-frame prediction. More particularly, the present application relates to a new inter prediction through hybrid inter and intra prediction.
Background
Video data requires a large storage space to store or a wide bandwidth to transmit. As higher resolutions and higher frame rates increase, storage or transmission bandwidth requirements will be enormous if video data is stored or transmitted in uncompressed form. Thus, using video codec techniques, video data is typically stored or transmitted in a compressed format. With newer video compression formats, such as the h.264/AVC standard and the High Efficiency Video Coding (HEVC) standard, coding Efficiency has actually been improved. To maintain a controllable complexity, a picture is typically partitioned into blocks, e.g., Macroblocks (MBs) or Largest Coding Units (LCUs)/Coding Units (CUs), for video codec. Video coding standards typically employ inter/intra prediction based on block adaptation.
Fig. 1 illustrates an exemplary adaptive inter/intra video codec system that integrates loop processing. For inter prediction, Motion Estimation (ME)/Motion Compensation (MC) 112 is used to provide prediction data based on video data from one or more other pictures. The
As shown in fig. 1, in the encoding system, input video data undergoes a series of processes. The reconstructed video data from
In intra prediction, a current block is predicted from previously decoded boundary samples from spatially neighboring blocks. For HEVC, there are 33 different angular, DC, and planar modes for block sizes from 4x4 to 32x 32. Next generation video codecs are being developed to further improve codec efficiency, which discloses improved intra prediction. For example, in JEM-6(Chen et al, "Algorithm Description of Joint amplification Test Model 6(JEM 6)", Joint Video amplification Team (JVT) of ITU-T SG16 WP 3and ISO/IEC JTC1/SC 29/WG 11,6th Meeting: Hobart, AU,31 March-7 April 2017, Document: JVT-F1001), finer-grained arbitrary direction intra prediction is disclosed, in which 67 intra prediction modes are used, including a planar mode, a DC mode, and 65 angular modes. The coding units supported by the next generation video codec are expanded to 265x 256.
In inter prediction, each block will attempt to find a similar block in one or more previously encoded frames (called reference frames), and motion information will be signaled or derived. In HEVC, the motion information typically includes a horizontal motion vector translation value and a vertical motion vector translation value, one or two reference picture indices if a prediction region in a B slice, and an identification of which reference picture list is associated with each index.
In HEVC, inter prediction also includes a merge mode to inherit motion information from spatially or temporally neighboring blocks. The merge candidate lists are maintained at both the encoder side and the decoder side. The candidates selected from the list may be represented by the merge index without signaling motion information. It is called a merge mode because it forms a merge area to share all motion information. The merge mode is similar to the direct mode and skip mode in AVC.
In the next generation of video coding, different inter prediction techniques have been disclosed to improve the performance of coding efficiency. For example, in VCEG-AZ07(Chen et al, "flame improvements to HMKTA-1.0", Video Coding Extensions Group (VCEG) of ITU-T SG 16Q, 2015), a model-based MV derivation (PMVD) method is disclosed. The PMVD method is also referred to as Frame Rate Up Conversion (FRUC) in VCEG-AZ 07. FRUC includes bilateral matching for bi-directional prediction blocks and template matching for uni-directional prediction blocks. Predictors derived from PMVD (also known as FRUC) can be inserted into the merge list as another candidate.
Fig. 2 illustrates an example of a Frame Rate Up Conversion (FRUC) bi-directional matching mode, in which motion information of the
Fig. 3 illustrates an example of template matching FRUC pattern. The neighboring areas (i.e., 320a and 320b) of the
In HEVC, only a translational motion model is applied to Motion Compensation Prediction (MCP). Although in the real world there are many kinds of movements, such as zoom in/out, rotation, perspective movement and other irregular movements. Affine models can describe two-dimensional block selection and two-dimensional deformation to transform a square (or rectangle) into a parallelogram. This model can be described as follows:
x′＝ax+by+θ，
y′＝cx+dy+f. (1)
in this model, a total of 6 parameters (i.e., a, b, c, d, e, and f) were used. For each pixel a (x, y) in the region of interest, a motion vector between this pixel and its corresponding reference pixel a ' (x ', y ') can be derived using the Motion Vector Field (MVF) of the block.
In next generation video coding, simplified affine transform motion compensated prediction is applied in order to improve coding efficiency. The affine motion field of the block is described by motion vectors associated to two control points, the affine model using only 4 parameters instead of 6.
In addition to PMVD and affine motion models, other improved inter prediction techniques, such as spatial-temporal motion vector prediction (STMVP) and different sub-prediction unit-based motion vector prediction, are also proposed to next-generation video editing codes.
In the present application, a new inter prediction mode is disclosed, which combines an intra prediction mode with an inter prediction mode.
Disclosure of Invention
The application discloses a method and a device for video coding and decoding, which are executed by a video encoder or a video decoder. According to one method, a boundary region in a current block is determined. For each pixel in the boundary region, deriving an intra predictor from a plurality of intra reference pixels; deriving an inter predictor also in the reference picture comprising a plurality of inter reference pixels located in the reference block; generating an intra-inter predictor by mixing the intra predictor and the inter predictor; and encoding or decoding the boundary region pixels using the intra-inter predictor. The reference block is coded before the current block. The plurality of intra reference pixels are located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture. The intra reference pixels are coded prior to the current block. For a plurality of remaining pixels in the current block, which do not belong to the boundary region, the plurality of remaining pixels are encoded or decoded using the inter predictor or the intra predictor.
In one embodiment, the boundary region corresponds to at least one of a first N columns of the current block near the plurality of intra reference pixels and a first N rows of the current block near the plurality of intra reference pixels, where N is an integer greater than 0. N may be predefined or signaled based on the size of the current block. The location associated with the maximum gradient located at the first column or first row of the current block may be derived based on an intra predictor or an inter predictor, or a difference between the intra predictor and the inter predictor, the location being used to determine the value of N. Alternatively, the location associated with the maximum gradient located at the first column or first row of the current block may be derived based on an intra-predictor or an inter-predictor, or a difference between the intra-predictor and the inter-predictor, and the location is used to determine the value of N if the maximum gradient is substantially (substandally) greater than an average gradient of a plurality of vertical gradients and a plurality of horizontal gradients calculated for the first column or first row of the current block, and otherwise the default value is used for N. When the maximum gradient is greater than a certain amount or percentage, it may be considered "substantial". For example, when the maximum gradient 50% is greater than the average gradient, it is considered "substantial". In another case, a position related to a maximum difference between the intra predictor and the plurality of neighboring reconstructed pixels, between the inter predictor and the plurality of neighboring reconstructed pixels, or between the inter predictor and the intra predictor may be derived, or between the inter predictor and the intra predictor, and the position is used to determine the value of N. The remaining pixels in the current block, which do not belong to the boundary region, are encoded or decoded using the inter predictor.
In one embodiment, determining the boundary region in the current block includes: a model edge is detected in the current block based on the inter predictor, the intra predictor, or a difference between the intra prediction and the inter predictor, and the boundary region is determined according to the model edge. Detecting the plurality of model edges may include: the maximum gradient is calculated based on the inter predictor, the intra predictor, or the difference between the intra predictor and the inter predictor for each row or each column depending on whether the intra prediction mode related to the intra predictor is close to the vertical direction or the horizontal direction. The border area may include one edge pixel in each row or column. The boundary region may further include any neighboring pixels that are adjacent to one edge pixel in the same row or the same column of the current block. The boundary region may also include any additional pixels in the current row or column between the current edge pixel in the current row or column and the previous edge pixel in the previous row or column. For a target pixel in the current block, which does not belong to the boundary region, if the target pixel is located above or to the left of one edge pixel in the same row or column as the target pixel, the target pixel may be encoded or decoded using the intra predictor. For a target pixel in the current block, which does not belong to the boundary region, if the target pixel is located below or to the right of one edge pixel in the same row or column as the target pixel, the target pixel may be encoded or decoded using the inter predictor.
The application also discloses another video coding and decoding method and device, which are executed by a video encoder or a video decoder. After the intra predictor and the inter predictor are derived, a weighted sum of the intra predictor and the inter predictor is generated, wherein generating the weighted sum of the intra predictor and the inter predictor comprises: the weighted intra predictor is added to the weighted inter predictor, wherein the weighted intra predictor is derived by multiplying the intra predictor by an intra weighting factor, and the weighted inter predictor is derived by multiplying the inter predictor by an inter weighting factor, the sum of the intra weighting factor and the inter weighting factor being scaled to a power of 2. A normalized intra-inter predictor is generated by applying a right shift operation to the weighted sum of the intra predictor and the inter predictor. Then, the current block is encoded or decoded using the normalized intra-inter predictor. An offset may be added to the sum of the weighted intra predictor and the weighted inter predictor.
The application also discloses a video coding and decoding method and a video coding and decoding device, which are executed by a video coder or a video decoder. According to this method, if the size of the current block is greater than the maximum block size or less than the minimum block size, the intra-inter prediction mode for the current block is disabled.
A method and apparatus for video encoding or decoding performed by a video encoder or decoder that integrates a coding tool for intra prediction with multiple directional intra prediction modes. According to the present invention, when the intra prediction mode is the directional mode, the one or more coding tool settings are determined based on the intra prediction mode, so that the one or more coding tool settings determined based on the intra prediction mode do not need to be explicitly signaled. At the encoder side, intra prediction and encoding and decoding tools are applied to a current block to generate an intra prediction residue or a transformed intra prediction residue of the current block. At the decoder side, intra prediction and coding tools are applied to the intra prediction residue or the transformed intra prediction residue to generate a reconstructed current block. Intra-prediction is applied according to a current intra-prediction mode, and coding tools are applied according to one or more target settings.
Drawings
Fig. 1 illustrates an exemplary adaptive inter/intra video codec system that integrates loop processing.
Fig. 2 illustrates an example of FRUC bilateral matching mode, where motion information for a current block is derived based on two reference pictures.
Fig. 3 illustrates an example of a template matching FRUC mode, in which a neighboring area of a current block in a current image is used as a template to match a corresponding template in a reference image.
Fig. 4 is a block diagram illustrating an exemplary encoder that selects among an intra mode, an inter mode, and an intra-inter mode to encode input video, according to an embodiment of the present invention.
Fig. 5 illustrates an example of generating an intra-inter predictor by combining an intra predictor having a weighting coefficient and an inter predictor.
Fig. 6 illustrates an example of weighting factor selection by angle mode, where all angle modes are divided into two groups, each group using a respective weighting block having four coefficient bands.
Fig. 7 illustrates another example of weighting factor selection by angle mode, where intra modes are grouped by their prediction direction into five groups, each group using its own weighting block.
Fig. 8 illustrates an example of adjusting the weighting coefficients according to the distance between the current sample and the reference sample (810 and 812) when the intra mode is the diagonal mode.
Fig. 9 illustrates an example of deriving a boundary region for intra-inter prediction according to one embodiment.
Fig. 10 illustrates an exemplary flowchart of a video coding and decoding system using intra-inter prediction, in which the intra-inter prediction is applied only to a boundary region, according to an embodiment of the present invention.
Fig. 11 illustrates another exemplary flow diagram of a video codec system using intra-inter prediction according to an embodiment of the present invention, wherein the sum of the intra weighting factor and the inter weighting factor is scaled to a power of 2 to avoid the need for division operations.
Fig. 12 illustrates still another exemplary flowchart of a video coding and decoding system using intra-inter prediction according to an embodiment of the present invention, in which intra-inter prediction is applied based on a current block size.
Detailed Description
The following description is of the preferred embodiments of the invention. The following examples are provided only for illustrating the technical features of the present invention and are not intended to limit the present invention. The protection scope of the present invention should be determined by the claims.
In conventional coding systems, blocks are coded in an intra prediction mode or in an inter prediction mode. In this application, a combined intra prediction and inter prediction mode is disclosed for inter pictures in video compression. In this application, signaling of intra-inter modes is also disclosed. Intra-inter modes may be applied to sub-block partitions (e.g., Prediction Units (PUs) and Transform Units (TUs)). Sub-block (i.e., coding unit, prediction unit, and transform unit) layer signaling related to the proposed intra-inter prediction modes is also disclosed. In addition, various aspects related to the new intra-inter prediction mode are also disclosed, such as adjusting the number of intra modes used for intra-inter, modifying weighting coefficients used for intra-predictor and inter-predictor, and controlling parameters or turning on/off of the intra-inter modes in high level syntax.
Intra-inter mode
Intra-inter mode is a new mode that combines intra mode and inter mode to form a new inter prediction. In an exemplary encoder configuration as shown in fig. 4, the
In another embodiment, the intra-inter mode may be treated as a specific inter mode under an inter prediction branch. When the current block is coded in inter prediction mode, a flag will be signaled on the encoder side or derived on the decoder side to indicate whether the current prediction block is normal inter mode or intra-inter mode. When the current block uses intra-inter mode, intra coding information (e.g., intra mode) and inter coding information (reference list index, motion vector, bi-prediction, or uni-prediction) will be signaled at the encoder side or derived at the decoder.
In another embodiment, intra-inter modes may be treated as specific intra modes under intra prediction branches. When the current block is coded in intra prediction mode, a flag will be signaled on the encoder side or derived on the decoder side to indicate whether the current prediction block is normal intra mode or intra-inter mode. When the current block uses intra-inter mode, intra coding information (e.g., intra mode) and inter coding information (reference list index, motion vector, bi-prediction, or uni-prediction) will be signaled at the encoder side or derived at the decoder.
Intra-inter predictor generation
In the proposed method, for each block, there will be an intra predictor and an inter predictor. As shown in fig. 5, the intra predictor will be combined with the inter predictor with weighting coefficients to generate a new predictor. In fig. 5, the
In the present application, intra prediction means a predictor derived from the adjacent boundary of a current block in the same picture, and there may be many variables regarding how adjacent pixels are used to predict the current block. In an embodiment, the current picture may use conventional HEVC intra mode. In another embodiment, the current picture may use more advanced intra prediction as disclosed in JEM disclosed in JFET-F1001. In another embodiment, the current block may use fewer directions in angular intra mode. In another embodiment, the current block may derive the intra predictor based on filtered or unfiltered samples of the neighboring block, and the intra predictor may also be refined by filtering samples of the current block.
In this application, inter prediction means that the predictor is from a previous frame, which may be a conventional inter mode predictor, a merge mode predictor, a FRUC mode predictor, an affine mode predictor, or other predictor from a previous frame.
Signaling of intra-inter modes
In this section, signaling of intra-inter modes is disclosed.
In one example, intra-inter modes are treated as inter prediction modes. In this case, when the current block is coded in inter prediction, the flag is used to indicate whether it is coded in intra-inter prediction. If intra-inter mode, intra information will be signaled; and inter-frame information (i.e., motion information) is also signaled at the encoder side or derived at the decoder side. Otherwise, only inter-frame information is signaled or derived. An exemplary syntax design based on the HEVC syntax is provided in table 1.
Table 1 syntax example of intra-inter mode at prediction unit layer
In table 1, for the skip mode, a flag (i.e., intra _ inter _ mode _ flag) is used (as shown by the notation (1-1 a)) to indicate whether the intra-inter mode is used for the current prediction unit. The value of intra _ inter _ mode _ flag is detected to determine whether this value is set to 1 (i.e., "if (intra _ inter _ mode _ flag)" as shown by the comment (1-2 a)). If this value is set to 1, the syntax element intra _ pred _ mode as shown in the notation (1-3a) is used to represent intra prediction information. Similar explanations are included in the syntax tables for the merge mode (as shown by comments (1-1c), comments (1-2c), and comments (1-3 c)) and the unidirectional prediction mode and the bidirectional prediction mode (as shown by comments (1-1b), and comments (1-2 b)). In the above example, the intra _ inter mode _ flag may be context-coded or bypass-coded. It may use one context, two contexts, or three contexts.
In case of encoding and decoding the intra-inter flag using two contexts, if any one of the upper block and the left block uses the intra-inter mode, context 0 is selected; otherwise, context 1 is selected. In case of using three context codecs intra-inter flag, if neither the upper block nor the left block uses intra-inter mode, context 0 is selected; if one of the upper block and the left block uses the intra-inter mode, context 1 is selected; and if the upper block and the left block both use the intra-inter mode, context 2 is selected.
Weighting coefficients for intra-to-inter modes
When combining intra mode and inter mode, different weights may be applied to the intra predictor and the inter predictor. Let w _ Intra and w _ Inter denote weighting coefficients for Intra and Inter modes. The intra-inter predictor can be derived as follows:
Pred＝w_Intra*Pred_Intra+w_Inter*Pred_Inter (2)
in the above equation, Pred is the combined predictor, Pred _ Intra is the Intra predictor, Pred _ Inter is the Inter predictor, and w _ Intra + w _ Inter is 1.
In one embodiment, the weighting coefficients are related to the intra mode direction. For example, w _ Intra may decrease along the Intra mode direction. On the other hand, w _ Inter may increase along the intra mode direction.
In an example, if the sample is mostly predicted by the above reference pixels (i.e., the Intra prediction direction is close to vertical mode), the weighting of the Intra predictor (w _ Intra) decreases column by column in the row by row positive downward direction. If the samples are mostly predicted by the left reference pixel (i.e. the Intra prediction direction is close to horizontal mode), the weighting of the Intra predictor (w _ Intra) is decreasing column by column to the right. If the Intra mode is DC mode or planar mode, w _ Intra is set to 0.5 and w _ Inter is also set to 0.5 for the entire block. In another example, w _ Intra and w _ Inter may be other values. For example, the sum of w _ Intra and w _ Inter may be a value other than 1. However, if the Intra mode is the DC mode or the planar mode, w _ Intra and w _ Inter are still equal to 1. Fig. 6 illustrates an example of weighting factor selection according to the angular mode.
In fig. 6, all the angular modes 610 are divided into two groups, in which all the angular intra modes having prediction directions closer to the vertical mode are classified into a vertical group 612 and all the angular intra modes having prediction directions closer to the horizontal mode are classified into a horizontal group 613. If the block is Intra coded by an Intra mode belonging to a vertical group, an Intra weight set 620 is used, progressive with Intra weights column by column divided into four bands with weights decreasing from w _ Intra1 to w _ Intra4 (i.e., w _ Intra1 to w _ Intra 4). If the block is Intra-coded by an Intra-mode belonging to a horizontal group, the Intra weight set 630 is used in which Intra weights are divided column by column into four bands (i.e., w _ Intra1 to w _ Intra4) with weights decreasing from w _ Intra1 to w _ Intra 4.
In the above example, for four Intra-predictor weighting coefficients (w _ Intra1, w _ Intra2, w _ Intra _3, and w _ Intra4), the Inter-predictor weighting coefficients (i.e., w _ Inter1, w _ Inter2, w _ Inter _3, and w _ Inter4) can be derived as follows: w _ Inter1 ═ 1-w _ Intra1, w _ Inter2 ═ 1-w _ Intra2, w _ Inter3 ═ 1-w _ Intra3, and w _ Inter4 ═ 1-w _ Intra 4. In this example, for 4 sample bands weighted within a frame, there are 4 weighting coefficients that decrease from a boundary close to the reference pixel to a boundary far from the reference pixel. In another embodiment, for each block, there may be N weighting coefficients, where N is an integer greater than 1. In another embodiment, for larger block sizes, a larger number of weighting coefficients may be used (i.e., more weighting coefficients may be selected for deriving intra-inter predictors), and for smaller block sizes, a smaller number of weighting coefficients may be used. In another embodiment, the regions having the same coefficients may be non-linear for each group, meaning that the width of each region need not be the same.
In another example, the intra modes are grouped by their prediction direction. For example, the intra modes may be divided into 5 groups, which correspond to the vertical group (labeled as group a), the horizontal group (labeled as group B), the diagonal group (labeled as group C), the vertical diagonal group (labeled as group D), and the mode 2 group (labeled as group E) in fig. 7. As is known for intra prediction in HEVC or next generation video coding, mode 2 corresponds to the upper right diagonal prediction direction. If the intra direction is close to the vertical mode 712 (i.e., group a), then the
In fig. 7, w _ Intra1, w _ Intra2, w _ Intra _3, and w _ Intra4 are Intra predictor weighting coefficients for a sample band. The Inter-prediction sub-weighting coefficients (i.e., w _ Inter1, w _ Inter2, w _ Inter _3, and w _ Inter4) can be derived as follows: w _ Inter1 ═ 1-w _ Intra1, w _ Inter2 ═ 1-w _ Intra2, w _ Inter3 ═ 1-w _ Intra3, and w _ Inter4 ═ 1-w _ Intra 4. In the present example, there are 4 weighting coefficients that decrease from a boundary close to the reference pixel to a boundary far from the reference pixel. In another embodiment, for each block, there may be N weighting coefficients, where N is an integer greater than 1. In another embodiment, for larger block sizes, a larger number of weighting coefficients may be used (i.e., more weighting coefficients may be selected for deriving intra-inter predictors), and for smaller block sizes, a smaller number of weighting coefficients may be used. In another embodiment, the regions having the same coefficients may be non-linear for each group, meaning that the width of each region need not be the same.
In a further embodiment, the weighting factor is related to the distance between the current pixel and the reference pixel. When the intra mode is the angle mode, if the current pixel is close to the reference pixel, the intra predictor has a greater weight than the inter predictor. When the intra mode is the angular mode, if the current pixel is far from the reference pixel, the inter predictor has a greater weight than the intra predictor. When the Intra mode is the DC mode or the planar mode, w _ Intra and w _ Inter are both set to 0.5 for the entire block. In yet other embodiments, w _ Intra and w _ Inter may be other values. For example, if the Intra mode is DC mode or planar mode, w _ Intra and w _ Inter may be values other than 1, while the sum of w _ Intra and w _ Inter is still equal to 1.
Fig. 8 illustrates an example of adjusting the weighting coefficients according to the distance between the current sample (represented by area 820) and the reference samples (810 and 812) when the intra mode is the diagonal mode. In fig. 8, S _ r corresponds to a reference sample, and S _ p corresponds to a predicted sample. The distance between S _ p and S _ r is denoted by D. The Intra prediction weights w _ Intra may be expressed as a function of D. For example, it can be calculated as follows:
w_intra(D)＝a*D2+b (3)
in the above equation, a and b are positive integers. The values of a and b may be determined through training and may be signaled at the Sequence reference Set (SPS), Picture Parameter Set (PPS), slice header or hard-coded at both the encoder and decoder sides. Likewise, when D is smaller, w _ intra should be larger. The closer the distance between the reference sample and the current sample, the larger the w _ intra will be.
In another embodiment, the intra prediction weights w _ intra may be represented by other functions.
In an embodiment, a weighting parameter may be used to adjust the w _ intra of each frame. This parameter may be signaled at the sequence reference set, picture parameter set, or slice header. The new predictor will be calculated from the following equation:
Pred＝w_Intra*Adjustment_Intra*Pred_Intra+(1-w_Intra*Adjustment_Intra)*Pred_Inter (4)
the Adjustment Intra parameter may be derived from a previous frame. Different ways of deriving the intra-prediction weighting coefficients have been described above, which can be applied to the above equations.
Edge detection for intra-inter mode
According to the method, if an approximate model edge can be detected, weighting coefficients of an intra predictor and an inter predictor can be adaptively determined. In an embodiment, an inter predictor may be used to detect model edges. First, the vertical gradient and the horizontal gradient of the inter predictor are calculated (e.g., the difference between the inter predictors in the vertical direction can be used to derive the vertical gradient and the difference between the inter predictors in the horizontal direction can be used to derive the horizontal gradient), and then the two gradients are averaged for the next flow. If the intra prediction mode is close to the vertical mode, the sample having the maximum value of each row is selected. For simplicity, in the present invention, the pixel having the maximum value of each row is referred to as an edge pixel. For other samples in this row that are above this sample with the maximum value (i.e., the edge pixel), only the intra predictor is used, without a blending process; and for those samples in the same row that are below the sample with the largest value (i.e., the edge pixel), only the inter predictor is used for inter-intra prediction. Next, in order to reduce discontinuity between samples processed using the intra predictor or using the inter predictor, a blending procedure is applied to these pixels around the boundary sample (i.e., edge pixel) having the maximum value. For simplicity, the edge pixels and the pixels surrounding the edge pixels are referred to as boundary region pixels. The region including the boundary region pixels is referred to as a boundary region. If the intra prediction mode is close to the horizontal mode, the sample having the maximum value of each column is selected. For other samples in this column that are to the left of the sample with the maximum value (i.e., the edge pixel), only the intra predictor is used, without a mixing flow; and for those samples in the same column that are to the right of the sample having the largest value, only inter predictor is used for inter-intra prediction, wherein the other samples in the column to the left or right of the sample having the largest value (i.e., edge pixel) may be regarded as target pixels in the current block, which do not belong to the boundary region. Next, in order to reduce discontinuity between samples processed using the intra predictor or using the inter predictor, a blending procedure is applied to these pixels around the boundary sample having the maximum value. Also, for simplicity, the edge pixels and the pixels surrounding the edge pixels are referred to as boundary region pixels. The region including the boundary region pixels is referred to as a boundary region. In another embodiment, intra predictors may be used to detect model edges and process the same rules as in the previous embodiment to complete the segmentation. In another embodiment, the difference between the inter and intra predictors can be used to detect model edges, also applying the same rules as in the previous embodiment to accomplish intra-inter prediction.
Fig. 9 illustrates an example of deriving a boundary region for intra-inter prediction according to the above-described embodiment, in which a
Operation of weighted normalization in intra-inter mode
As previously described, the weighting coefficients w _ Intra and w _ Inter may have values selected for each sample band of the block. A mixed intra-inter predictor is derived by combining the intra predictor and the inter predictor using the weighted sum. For computing mixed intra-inter frames, a division operation may be avoided because it would require more processing power to implement the division operation than an addition operation and a multiplication operation. During the calculation of the intra-inter predictor, a division operation is used for weight normalization. To avoid division operations, in an embodiment of the present application, the w _ Intra and w _ Inter sums are scaled to powers of 2, so that the shift operation can be used for weighted normalization:
Pred＝(w_Intra*Pred_Intra+w_Inter*Pred_Inter)>>(Log2(w_Intra+w_Inter))(5)
in the above equation, the symbol "> >" represents a right shift operation, and Log2 represents a base-2 logarithmic operation. In another embodiment, a bias (bias) may be included to reduce rounding errors:
Pred＝((w_Intra*Pred_Intra+w_Inter*Pred_Inter)+bias)>>(Log2(w_Intra+w_Inter))(6)
smoothing filter for intra prediction of intra-inter modes
Intra prediction in intra-inter mode is used to compensate for some information loss due to inter prediction. In intra prediction, some smoothing and boundary filters are conditionally employed to further improve the intra predictor. For intra-inter modes, the generation of intra predictors may be the same as for conventional intra prediction. Alternatively, these smoothing or boundary filters may always be enabled or disabled in intra prediction generation for intra-inter modes.
Improvement of first row/column of intra-inter mode
Information of neighboring coding units may be referred to increase the accuracy of the current prediction. In one embodiment, samples belonging to the boundary of the current coding unit are generated by selecting the best one among the intra predictor, the inter predictor, and a mixture of the inter predictor and the intra predictor. The best one may be defined as a predictor having a minimum difference compared to neighboring samples in order to generate a minimum gradient between samples in the current coding unit of a smooth boundary or a minimum gradient between a sample in the current coding unit and a sample in a neighboring coding unit.
Reference previous coding unit parameter setting for intra-inter mode
To reduce the computational complexity of intra-inter modes at the encoder, some information of the intra-inter modes in a first Rate-Distortion Optimization (RDO) flow may be saved and reused in subsequent RDO flows in different coding unit partitions. In one embodiment, the on/off decision of intra-inter mode is reused. After this, the predictor may follow the first decision to process or skip this mode. However, the prediction modes of the neighboring coding units may be different, so that the performance of intra prediction and inter prediction in the intra-inter mode may be changed. Thus, in another embodiment, disabled intra-inter modes will always be tested, although the first decision is enabled. In another embodiment, the best intra prediction mode of the intra-inter mode may also be reused to reduce complexity.
Blending with boundaries of intra-inter modes
In intra-inter mode, intra prediction is used to compensate for the drawbacks of inter prediction. However, for larger size coding units, the predicted samples that are far from the reference sample are generally inaccurate in intra prediction. To overcome this problem, boundary constraints for intra-inter mixing are proposed. In an embodiment, only the first few columns or rows near the reference sample will undergo an intra-inter mixing flow; and other samples are derived directly from inter prediction. If the intra-prediction mode is close to the vertical mode, the intra-prediction sub-weights are reduced column by column within the first N rows close to the reference sample. If the intra-prediction mode is close to horizontal mode, the intra-predictor weights are reduced row by row in the first N columns close to the reference sample (i.e., the intra-predictor is reduced from column 1 to N, and the intra-predictor weights of the remaining columns (if any) are all set to 0). For DC mode or planar Intra mode, w _ Intra and w _ Inter equal to 0.5 are only applied in L-type regions (the first N columns and N rows near the reference sample). In another embodiment, the gradients of the samples in the first column and the first row are calculated first. If the maximum gradient in the first column/row is much larger than the average gradient calculated for this column/row, then the location of the maximum gradient is the boundary in the blending flow; otherwise, a default scope boundary is used. With this blending boundary, intra prediction in intra-inter can be simplified by generating only the previous N columns or N rows for further blending. In some embodiments, N is predefined or signaled based on the size of the current block. In another embodiment, the difference between the inter/intra predictor in the L-type of the top coding unit and the left coding unit and the neighboring reconstructed pixel is calculated, and the position with the largest difference is detected. The hybrid boundary may be set using any of the boundary derivation methods disclosed above, or a default boundary may be used for this constraint.
Coding unit size constraint for intra-inter modes
For smaller coding units, each syntax element requires more side information overhead than for larger coding units. Thus, removing one flag for a smaller coding unit may achieve a better break point in terms of codec efficiency. In an embodiment, intra-inter modes are forced to be off for smaller coding units. In another embodiment, the intra-inter mode for a larger coding unit is forced to be off due to a lower probability that the intra-inter mode is selected in the larger coding unit.
Deriving intra prediction modes by analyzing intra-to-inter correlation in intra-to-inter modes
With intra-predictors and inter-predictors used in intra-inter modes, some useful information can be generated by comparing their correlation. In an embodiment, rather than signaling the intra-prediction mode used in the intra-inter mode, the intra-inter mix result is compared at the encoder to the current coding unit, and the comparison between the intra-predictor and the inter-predictor at the decoder may be used to select the intra-prediction mode used in the intra-inter mode. In this case, the intra prediction mode used in signaling intra-inter modes may be eliminated.
Fig. 10 illustrates an exemplary flowchart of a video coding and decoding system using intra-inter prediction according to an embodiment of the present invention. The steps shown in this flowchart may be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder side. The steps shown in this flowchart may also be implemented on a hardware basis, such as one or more electronic devices or processors for executing the steps in the flowchart. According to the method, in step 1010, at the encoder side, data relating to a current block in a current picture are received, or at the video decoder side, a video bitstream comprising compressed data of the current block is received. In step 1020, a boundary region is determined in the current block. In step 1030, it is detected whether the current pixel is a boundary pixel. When the current pixel is a boundary pixel (i.e., "yes" path from step 1030), steps 1040 through 1070 are performed. Otherwise (i.e., the "no" path from step 1030), step 1080 is performed. In step 1040, an intra predictor is derived from intra reference pixels, wherein the intra reference pixels are located above a top boundary of the current block or to the left of a left boundary of the current block in the current picture, and the intra reference pixels are coded prior to the current block. In step 1050, an inter predictor including inter reference pixels located in a reference block that was coded prior to the current block is derived in the reference picture. In step 1060, an intra-inter predictor is generated by mixing the intra predictor and the inter predictor. In step 1070, each boundary region pixel is encoded at the encoder side or decoded at the decoder side using the intra-inter predictor. In step 1080, the remaining pixels are encoded at the encoder side or decoded at the decoder side using inter-or intra-predictors.
Fig. 11 illustrates another exemplary flowchart of a video coding and decoding system using intra-inter prediction according to an embodiment of the present invention. According to the method, in
Fig. 12 illustrates an exemplary flowchart of a video coding and decoding system using intra-inter prediction according to an embodiment of the present invention. According to the method, in
Any of the methods proposed above may be implemented in an encoder and/or decoder. As is known in the art of video coding and decoding, intra predictors and inter predictors can be derived at the encoder side as well as at the decoder side. By mixing the intra predictor and the inter predictor using weighting coefficients, an intra-inter predictor can be generated. The weighting coefficients may be derived or signaled. Thus, both the encoder and decoder can perform the same hybrid flow.
The flow chart shown is used to illustrate an example of video codec according to the present invention. It is possible for those skilled in the art to modify each step, recombine the steps, divide a step, or combine the steps to implement the invention without departing from the spirit of the invention. In the present invention, specific syntax and semantics have been used to illustrate examples of implementing embodiments of the present invention. By replacing the syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention, one of ordinary skill in the art can practice the present invention.
The foregoing description will enable one skilled in the art to practice the invention in the context of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. Thus, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In the foregoing detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. Nonetheless, it will be appreciated by those skilled in the art that the invention can be practiced.
The embodiments of the invention described above may be implemented in various hardware, software code, or combinations of both. For example, an embodiment of the invention may be circuitry integrated within a video compression chip or program code integrated into video compression software to perform the processing described herein. An embodiment of the invention may also be program code executing on a Digital Signal Processor (DSP) to perform the processing described herein. The invention may also include functions performed by a computer processor, digital signal processor, microprocessor, or Field Programmable Gate Array (FPGA). According to the present invention, the processors may be configured to perform certain tasks by executing machine readable software code or firmware code that defines certain methods implemented by the invention. Software code or firmware code may be developed in different programming languages and in different formats or styles. The software code may also be compiled into different target platforms. However, different code formats, styles and languages of software code, and other forms of configuration code to perform the tasks of the invention, do not depart from the spirit and scope of the invention.
The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
- 上一篇：一种医用注射器针头装配设备
- 下一篇：用于360度视频译码的参考图片推导及运动补偿