阅读说明：本技术 图像处理装置、方法和程序 (Image processing apparatus, method and program ) 是由 近藤健治 于 2019-09-18 设计创作，主要内容包括：本公开内容涉及使得可以抑制帧内预测中可能发生的延迟的图像处理装置、方法和程序。在要分别确定亮度的CU划分结构和色度的CU划分结构时,编码装置：针对亮度的CU划分结构,去除具有最小竖直和/或水平尺寸的CU划分结构以确定色度的CU划分结构；利用亮度的CU划分结构和色度的CU划分结构对图像进行编码；以及生成编码流。本公开内容可以例如应用于图像处理装置。(The present disclosure relates to an image processing apparatus, method, and program that make it possible to suppress a delay that may occur in intra prediction. When a luminance CU partition structure and a chrominance CU partition structure are to be determined, respectively, an encoding device: removing a CU partition structure having a smallest vertical and/or horizontal size for the CU partition structure of luminance to determine a CU partition structure of chrominance; encoding an image by using a luminance CU division structure and a chrominance CU division structure; and generating the encoded stream. The present disclosure can be applied to, for example, an image processing apparatus.)

1. An image processing apparatus comprising:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for the chrominance, a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of the chrominance; and

an encoding section configured to encode an image with the CU partition structure of the luminance and the CU partition structure of the chrominance to generate an encoded stream.

2. The image processing apparatus according to claim 1,

the size of the CU partition structure, which is the smallest size in at least one of the vertical direction and the horizontal direction, is 2 × 2, 4 × 2, or 2 × 4.

3. The image processing apparatus according to claim 1, further comprising:

a cost calculation unit configured to calculate a cost of the CU partition structure for the luminance and a cost of the CU partition structure for the chrominance, respectively,

for the luminance, the control portion determines a CU division structure of the luminance based on a cost of the CU division structure for the luminance, and

for the chroma, the control portion suppresses calculation of a CU partition structure cost for the chroma for a minimum CU partition structure, and determines the CU partition structure for the chroma based on the cost of the CU partition structure for the chroma.

4. The image processing apparatus according to claim 1, further comprising:

a setting section configured to set a syntax describing the minimum CU partition structure, wherein,

the control portion determines, for the chroma, a CU partition structure of the chroma by excluding the minimum CU partition structure based on the syntax.

5. The image processing apparatus according to claim 4, further comprising:

a transmitting section configured to transmit the syntax and the coded stream.

6. The image processing apparatus according to claim 4,

the syntax includes SPS.

7. The image processing apparatus according to claim 1,

the minimum CU partition structure is defined by a standard.

8. An image processing method performed by an image processing apparatus, comprising:

excluding, for the chroma, a CU partition structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine a CU partition structure for the chroma, when the CU partition structure for the luminance and the CU partition structure for the chroma are to be determined, respectively; and

and encoding an image by using the luminance CU partition structure and the chrominance CU partition structure to generate an encoding stream.

9. A program that causes a computer to function as:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for the chrominance, a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of the chrominance; and

an encoding section configured to encode an image with the CU partition structure of the luminance and the CU partition structure of the chrominance to generate an encoded stream.

10. An image processing apparatus comprising:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the CU division structure of chrominance, and determine the CU division structure of luminance and the CU division structure of chrominance based on an encoding stream generated by encoding an image using the CU division structure of luminance and the CU division structure of chrominance; and

a decoding section configured to decode the encoded stream based on the CU partition structure of the luminance and the CU partition structure of the chrominance to generate the image.

11. An image processing method performed by an image processing apparatus, comprising:

when a luminance CU division structure and a chrominance CU division structure are to be determined, respectively, excluding a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the chrominance CU division structure, and determining the luminance CU division structure and the chrominance CU division structure based on an encoding stream generated by encoding an image using the luminance CU division structure and the chrominance CU division structure; and

decoding the encoded stream based on the luma CU partition structure and the chroma CU partition structure to generate the image.

12. A program that causes a computer to function as:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the CU division structure of chrominance, and determine the CU division structure of luminance and the CU division structure of chrominance based on an encoding stream generated by encoding an image using the CU division structure of luminance and the CU division structure of chrominance; and

a decoding section configured to decode the encoded stream based on the CU partition structure of the luminance and the CU partition structure of the chrominance to generate the image.

Technical Field

The present disclosure relates to an image processing apparatus, method, and program, and particularly relates to an image processing apparatus, method, and program that can reduce a delay that may occur in intra prediction.

Background

In HEVC (high efficiency video coding), a CU (coding unit) of chroma (Cb/Cr) is divided into 4 × 4 (pixels). In VVC (general video coding), it is allowed to divide a CU into a size of 4 × 4 for luminance (Y) and a size of 2 × 2 for Cb/Cr (non-patent document 1).

[ list of references ]

[ non-patent document ]

[ non-patent document 1]

Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 2)", JVT-K1001-v 5, ITU-T SG16WP3 and ISO/IEC JTC1/SC29/WG11, conference number eleventh: lubuerga, SI, 7 months, 10 days to 18 days 2018.

Disclosure of Invention

[ problem ] to

Since it becomes possible to divide the CU of Cb/Cr into 2 × 2 sizes, the throughput of the CU of which the size of Cb/Cr is 2 × 2 in intra prediction becomes a bottleneck.

The present disclosure has been made in consideration of the situation as described above, and makes it possible to reduce a delay that may occur in intra prediction.

[ solution of problem ]

An image processing apparatus according to an aspect of the present disclosure includes: a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for chrominance, the CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of chrominance; and an encoding section configured to encode the image with a CU partition structure of luminance and a CU partition structure of chrominance to generate an encoded stream.

In one aspect of the present disclosure, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, the CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction is excluded for chrominance, and the CU division structure of chrominance is determined. Then, the image is encoded with a CU partition structure of luminance and a CU partition structure of chrominance, and an encoded stream is generated.

An image processing apparatus of another aspect of the present disclosure includes: a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for chrominance, a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of chrominance, and determine the CU division structure of luminance and the CU division structure of chrominance based on an encoding stream generated by encoding an image with the CU division structure of luminance and the CU division structure of chrominance; and a decoding section configured to decode the encoded stream based on the CU partition structure of luminance and the CU partition structure of chrominance to generate an image.

In another aspect of the present disclosure, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, the CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction is excluded for chrominance, and the CU division structure of chrominance is determined. Further, the CU partition structure of luminance and the CU partition structure of chrominance are determined based on an encoded stream generated by encoding an image with the CU partition structure of luminance and the CU partition structure of chrominance. Then, the encoded stream is decoded based on the luminance-based CU partition structure and the chrominance-based CU partition structure, and an image is generated.

Drawings

[ FIG. 1]

Fig. 1 is a diagram showing an example of division of a CU as a coding unit.

[ FIG. 2]

Fig. 2 depicts a diagram depicting a partitioning method of a CU.

[ FIG. 3]

Fig. 3 is a diagram depicting the order of intra prediction in the case where a CU of Y has a size of 4 × 4 and a CU of Cb/Cr has a size of 2 × 2.

[ FIG. 4]

Fig. 4 is a diagram depicting the order of intra prediction in the case where luma chroma prediction is performed for Y to Cb/Cr.

[ FIG. 5]

Fig. 5 is a block diagram depicting an example of the configuration of an embodiment of an encoding apparatus to which the present disclosure is applied.

[ FIG. 6]

Fig. 6 is a diagram depicting an example of a tree for determining a partitioning structure of a CU.

[ FIG. 7]

Fig. 7 is a block diagram depicting an example of a configuration of the encoding section of fig. 5.

[ FIG. 8]

Fig. 8 is a flowchart illustrating a stream generation process of the encoding apparatus of fig. 5.

[ FIG. 9]

Fig. 9 is a flowchart showing details of the encoding process in step S12 of fig. 8.

[ FIG. 10]

Fig. 10 is a flowchart showing details of the encoding process in step S12 of fig. 8, continuing from fig. 9.

[ FIG. 11]

Fig. 11 is a flowchart showing details of the CU division structure determination process in step S33 of fig. 9.

[ FIG. 12]

Fig. 12 is a flowchart showing details of the RD cost calculation process of BT in step S125 of fig. 11.

[ FIG. 13]

Fig. 13 is a flowchart showing details of the RD cost calculation process of TT in step S126 of fig. 11.

[ FIG. 14]

Fig. 14 is a flowchart illustrating another example of the CU division structure determination process in step S33 of fig. 9.

[ FIG. 15]

Fig. 15 is a flowchart showing details of the RD cost calculation process of BT in step S184 of fig. 14.

[ FIG. 16]

Fig. 16 is a flowchart showing details of the RD cost calculation process of TT in step S185 of fig. 14.

[ FIG. 17]

Fig. 17 is a flowchart illustrating details of the prediction mode determination process for the CU in step S186 in fig. 14.

[ FIG. 18]

Fig. 18 is a block diagram depicting an example of the configuration of an embodiment of a decoding apparatus to which the present disclosure is applied.

[ FIG. 19]

Fig. 19 is a block diagram depicting an example of the configuration of the decoding section of fig. 18.

[ FIG. 20]

Fig. 20 is a flowchart illustrating an image generation process of the decoding apparatus of fig. 18.

[ FIG. 21]

Fig. 21 is a flowchart showing details of the decoding process in step S303 of fig. 20.

[ FIG. 22]

Fig. 22 is a diagram depicting an example of an individual tree.

[ FIG. 23]

Fig. 23 is a flowchart illustrating another example of the CU division structure determination process in step S33 of fig. 9.

[ FIG. 24]

Fig. 24 is a flowchart showing a different example of the CU division structure determination process in step S33 of fig. 9, continuing from fig. 23.

[ FIG. 25]

Fig. 25 is a flowchart showing details of the RD cost calculation process of BT in step S411 of fig. 24.

[ FIG. 26]

Fig. 26 is a flowchart showing details of the RD cost calculation process of TT in step S412 of fig. 24.

[ FIG. 27]

Fig. 27 is a diagram depicting an example of semantics with respect to an individual tree.

[ FIG. 28]

Fig. 28 is a diagram depicting an example of a partitioning structure of a CU that can be controlled using information regarding fig. 27.

[ FIG. 29]

Fig. 29 is a diagram depicting an example in which only a Cb/Cr CU using the information on fig. 27 has a minimum size of 4 × 4.

[ FIG. 30]

Fig. 30 is a diagram depicting an example in which the minimum size of the QT partition of Cb/Cr is 0.

[ FIG. 31]

Fig. 31 is a diagram depicting an example of the partition structure of a CU provided with Cb/Cr of new information.

[ FIG. 32]

Fig. 32 is a block diagram depicting an example of the configuration of an embodiment of an encoding apparatus to which the present disclosure is applied.

[ FIG. 33]

Fig. 33 is a flowchart showing a stream generation process of the encoding apparatus of fig. 32.

[ FIG. 34]

Fig. 34 is a flowchart illustrating another example of the CU division structure determination process in step S33 of fig. 9.

[ FIG. 35]

Fig. 35 is a flowchart showing another example of the CU division structure determination process in step S33 of fig. 9 described above, continuing from fig. 34.

[ FIG. 36]

Fig. 36 is a flowchart showing another example of the CU division structure determination process in step S33 of fig. 9 described above, continuing from fig. 35.

[ FIG. 37]

Fig. 37 is a flowchart showing another example of the CU division structure determination process in step S33 of fig. 9.

[ FIG. 38]

Fig. 38 is a flowchart showing another example of the CU division structure determination process in step S33 of fig. 9, continuing from fig. 37.

[ FIG. 39]

Fig. 39 is a flowchart showing another example of the CU division structure determination process in step S33 of fig. 9, continuing from fig. 38.

[ FIG. 40]

Fig. 40 is a block diagram depicting an example of the configuration of an embodiment of a decoding apparatus to which the present disclosure is applied.

[ FIG. 41]

Fig. 41 is a flowchart illustrating an image generation process of the decoding apparatus of fig. 40.

[ FIG. 42]

Fig. 42 is a flowchart illustrating an example of the CU division structure determination process in step S333 of fig. 21.

[ FIG. 43]

Fig. 43 is a diagram depicting an example of a tree to be used for CU partition structure determination processing.

[ FIG. 44]

Fig. 44 is a block diagram showing an example of a configuration of hardware of a computer.

Detailed Description

Hereinafter, a mode for carrying out the present disclosure (hereinafter referred to as "embodiment") is described. Note that the description is given in the following order.

0. Overview

1. First embodiment (restriction of 2X 2 of CU of Cb/Cr)

2. Second embodiment (Single tree)

3. Third embodiment (adding minimum CU to grammar)

4. Fourth embodiment (determination of minimum CU by criterion)

5. Fifth embodiment

<0. overview >

(encoding method)

Hereinafter, the present technology is described taking as an example a case where the present technology is applied to image encoding and decoding of a VVC (general video coding) method.

(description of coding Unit)

Fig. 1 is a diagram depicting an example of division of a CU (coding unit) as a unit for coding.

The picture image of the color image is constructed from pixels according to resolution. The pixel has information on chrominance (Cb/Cr) and luminance (Y) indicating luminance.

In the format called 4:2:0 chroma format, the resolution of Cb/Cr is half relative to luminance in both the vertical and horizontal directions, as shown in fig. 2. When the resolution of Y is 1920 × 1080, the resolution of Cb/Cr is 960 × 540.

In the picture image of fig. 1, CTUs (coding tree units) are arranged in raster order. The CTU is a predetermined unit and is divided into coding units called CUs. Note that the CTUs may not be divided, but may be directly used as CUs. For Y, CTUs of size 128 × 128 are typically used.

In fig. 1, a thick line indicates a boundary between CTUs, and a solid line, a dotted line, and a dotted line indicate a boundary of a CU.

Fig. 2 is a diagram depicting a partitioning method of a CU.

In a of fig. 2, a CU of 8 × 8 size of Y and a CU of 4 × 4 size of Cb/Cr are depicted. In B of fig. 2, a CU obtained by being divided into two in the horizontal direction is depicted. CU with size of 8 × 4 for Y and CU with size of 4 × 2 for Cb/Cr are depicted.

In C of fig. 2, a CU obtained by being divided into two in the vertical direction is depicted. A CU with a size of 4 × 8 for Y and a CU with a size of 2 × 4 for Cb/Cr are depicted. In fig. 2D, an example is depicted in which the CU is divided into four.

In HEVC (high efficiency video coding), a CU is configured to have a size of 8 × 8 for Y and a size of 4 × 4 for Cb/Cr.

In VVC, a CU may be divided into a size of 4 × 4 for Y, and a size of 2 × 2 for Cb/Cr.

Here, if an image is input to the device on the encoding side, difference determination with a prediction signal, orthogonal transformation, quantization, dequantization, inverse orthogonal transformation, reconstruction by adding a prediction signal, and intra prediction in CU units by using the reconstructed signal are performed. In intra prediction, surrounding pixel values may be used in some cases. Therefore, if the loop from differencing to reconstruction is not complete, subsequent intra prediction for the CU cannot be performed. Therefore, as the size of the CU of the chroma decreases, the delay increases, and the processing time increases.

Fig. 3 is a diagram depicting the order of intra prediction in the case where the size of CU of Y is 4 × 4 and the size of CU of Cb/Cr is 2 × 2.

In the example of fig. 3, the arrow mark indicates the order of intra prediction of the CU of Cb/Cr.

As indicated by the arrow marks, intra prediction of a CU is performed in the order of top left, top right, bottom left, and bottom right. For the upper right CU, if the processing for the upper left CU is not completed, the processing cannot be started. For the lower left CU, if the processing for the upper right CU is not completed, the processing cannot be started. For the lower right CU, if the processing for the lower left CU is not completed, the processing cannot be started.

In the case where all CUs have a size of 4 × 4 of Y and a size of 2 × 2 of Cb/Cr obtained by dividing into minimum sizes in such a manner as described above, the delay is maximized.

Meanwhile, in VVC, a technique of predicting Cb/Cr from Y is intended to be employed, which is called luminance-chrominance prediction.

Fig. 4 is a diagram depicting the order of intra prediction in the case where luma chroma prediction is performed for Y to Cb/Cr.

In the example of fig. 4, the arrow marks indicate the order of intra prediction for the CU of Y and the CU of Cr/Cr.

Since the pixel values of the upper left CU of Y are used for intra prediction of the upper left CU of Cb/Cr, the above-described loop of differencing the prediction signal of Y to reconstruction needs to be completed already.

In the case of a moving image, an image according to the frame rate is input to the apparatus on the encoding side for one second. Therefore, in the real-time processing, the encoding processing must be completed in a certain unit such as one second or one frame; however, there is a risk that processing cannot be performed in time due to some delay.

Similar to the encoding apparatus, since the apparatus on the decoding side also adds the signal after the inverse orthogonal transform and the signal for intra prediction to reconstruct the signal, and performs intra prediction by using the signal after reconstruction, the apparatus on the decoding side is also affected by the delay.

Therefore, in the first embodiment, the device on the decoding side of a CU whose Cb/Cr size is 2 × 2 is excluded from the determination of the partition structure.

<1 > first embodiment (restriction of Cb/Cr 2 × 2 >

(example of configuration of the first embodiment of the encoding apparatus)

Fig. 5 is a block diagram depicting an example of the configuration of an embodiment of an encoding apparatus to which the present disclosure is applied.

The encoding device 10 of fig. 5 includes a control section 11, a setting section 12, an encoding section 13, and a transmission section 14, and encodes an image by a method conforming to the VVC method.

Specifically, the control section 11 of the encoding device 10 includes, for example, a CPU (central processing unit), a microprocessor, and the like. The control section 11 executes a predetermined program by the CPU to execute various types of calculation processing, operation control of associated blocks, and the like, and operates as a central processing device in the encoding device 10.

For example, the control section 11 controls the encoding section 13 to calculate the RD cost, and performs CU partition structure determination processing for determining the partition structure of the CU based on the RD cost supplied from the encoding section 13. At this time, the control section 11 determines the division structure of the CU except for the case where the size of each CU of Y after the division is equal to or smaller than 4 × 4. In particular, except for the case where the size of each CU of Cr/Cb is 2 × 2.

Further, the control section 11 determines the prediction mode based on the RD cost. The control section 11 controls the setting section 12 to set a parameter set based on the determined CU partition structure and prediction mode, and controls the encoding section 13 to perform encoding processing.

The setting section 12 sets SPS (sequence parameter set), PPS (picture parameter set), VUI (video usability information), SEI (supplemental enhancement information), and the like according to an instruction of the control section 11. The setting section 12 outputs a setting parameter set such as SPS, PPS, VUI, and SEI to the encoding section 13.

The image in units of frames is input to the encoding unit 13. The encoding unit 13 encodes the input image in accordance with the instruction of the control unit 11. The encoding section 13 generates an encoded stream from the parameter set and encoded data obtained as a result of the encoding, and outputs the encoded stream to the transmitting section 14.

The transmitting section 14 transmits the encoded stream supplied from the encoding section 13 to a decoding apparatus 110 described later.

Fig. 6 is a diagram depicting an example of a tree for determining a partitioning structure of a CU.

In the example of fig. 6, an example is depicted in which Y, Cb and Cr have the same CU partition structure. The tree has branches. The 0 and 1 indicated on the left and right sides of each branch indicate the value of the flag. After the partition structure is determined, the value of the flag is placed as a partition flag into the bitstream and transmitted together with the bitstream, which is decoded at the decoding side. Accordingly, the decoding side can find the CU partition structure determined by the apparatus of the decoding side.

At branch b1, the CTUs are divided by QT split (1) or by a split other than QT split (0). The partitions other than the QT partition include BT (binary tree) partition, TT (ternary tree) partition, or No partition (No split).

In case QT split (1) is chosen, the CTUs are split into QTs. In the case of dividing the CTU into QTs, the processing is returned as indicated by the thick line arrow, and each CT is recursively divided.

In the case where a partition (0) other than the QT partition is selected, the process proceeds to branch b 2. At branch b2, BT/TT partition (1) or not (0) is selected.

In the case where the non-division (0) is selected, the non-division is determined, and the process ends.

In the case where the BT/TT division (1) is selected, the process proceeds to branch b 3. At branch b3, horizontal (1) or vertical (0) is selected.

In the case of selection level (1), the process proceeds to branch b 4. At branch b4, either BT (1) or TT (0) is selected.

In case BT (1) is selected, CTUs are divided into BT. In the case where TT (1) is selected, CTUs are divided into TTs. Not only in the case of dividing the CTU into the BT but also in the case of dividing the CTU into the TT, the processing is returned as indicated by the dotted arrow, and the division is recursively performed.

In the case where vertical (0) is selected, processing proceeds to branch b 5. At branch b5, either BT (1) or TT (0) is selected.

In case BT (1) is selected, CTUs are divided into BT. In the case where TT (1) is selected, CTUs are divided into TTs. Not only in the case of dividing the CTU into the BT but also in the case of dividing the CTU into the TT, the process returns as indicated by the arrow, and the CTU is recursively divided.

The selection here is performed based on a technique called RD optimization (rate distortion optimization). In the RD optimization, the RD cost is calculated, and a decision is made to make the value of the RD cost small, so that a decision of high coding efficiency can be performed. The RD cost J is represented by the following expression (1).

[ mathematical formula 1]

J＝D+λR...(1)

D is the difference (distortion) between the original image and the decoded image, R is the generated code amount including up to the orthogonal transform coefficient, and λ is a lagrangian (Lagrange) undetermined multiplier given as a function of the quantization parameter QP.

Specifically, the control section 11 controls the encoding section 13 to calculate QT, BT, TT and RD cost of non-partitioning at each branch, and determines the CU partitioning structure based on the RD cost.

However, in the case where the size of each CU of Y after the QT division is 8 × 8, the control section 11 returns the processing and selects not to divide (0) without calculating the RD cost. In the case where the size of each CU of Y after BT division is 8 × 8, the control section 11 returns to the process and selects not to divide (0) without calculating the RD cost. In the case where the size of each CU of Y after TT division is 8 × 8, the control section 11 returns to the processing and selects not to divide (0) without calculating RD cost.

By determining the CU division structure in the manner as described above, the size of each CU of Qr/Qb does not become 2 × 2. Therefore, a large delay in the encoding process and the decoding process of the intra prediction can be avoided.

(example of configuration of encoding section)

Fig. 7 is a block diagram depicting an example of a configuration of the encoding section of fig. 5.

The encoding unit 13 of fig. 7 includes an a/D conversion unit 31, a screen image sorting buffer 32, a calculation unit 33, an orthogonal transform unit 34, a quantization unit 35, a lossless encoding unit 36, an accumulation buffer 37, a dequantization unit 38, an inverse orthogonal transform unit 39, and an addition unit 40. The encoding section 13 further includes a deblocking filter 41, an adaptive offset filter 42, an adaptive loop filter 43, a frame memory 44, a switch 45, an intra prediction section 46, a motion prediction and compensation section 47, a prediction image selection section 48, and a rate control section 49.

The coding unit 13 uses the CU having the partition structure determined by the control unit 11. Further, the division flag determined by the CU division structure determination process is output to the lossless encoding section 36.

The a/D conversion section 31 of the encoding section 13 a/D-converts an image in units of frames of an encoding target. The a/D conversion section 31 outputs the converted image in the form of a digital signal to the screen image sorting buffer 32 to be stored.

The picture image sorting buffer 32 sorts the images in units of frames in the display order into images in the encoding order according to the GOP structure. The screen image sorting buffer 32 outputs the sorted images to the calculation unit 33, the intra prediction unit 46, and the motion prediction and compensation unit 47.

The calculation section 33 performs encoding by subtracting the predicted image supplied from the predicted image selection section 48 from the image supplied from the screen image sorting buffer 32. The calculation unit 33 outputs the subtracted image to the orthogonal transformation unit 34 as residual information (difference). Note that, in a case where the prediction image is not supplied from the prediction image selecting section 48, the calculating section 33 outputs the image read out from the screen image sorting buffer 32 as it is to the orthogonal transforming section 34 as a residual.

The orthogonal transformation unit 34 performs orthogonal transformation processing on the residual information from the calculation unit 33 on a CU-by-CU basis. The orthogonal transformation unit 34 outputs the image after the orthogonal transformation to the quantization unit 35.

The quantization unit 35 quantizes the image after the orthogonal transform process supplied from the orthogonal transform unit 34. The quantization unit 35 outputs the quantized value to the lossless encoding unit 36.

The lossless encoding unit 36 acquires intra-prediction mode information, which is information indicating the optimal intra-prediction mode, from the intra-prediction unit 46. Further, the lossless encoding unit 36 acquires inter prediction mode information, a motion vector, information specifying a reference image, and the like from the motion prediction and compensation unit 47 as information indicating the optimal inter prediction mode.

Further, the lossless encoding section 36 acquires offset filter information on the offset filter from the adaptive offset filter 42, and acquires filter coefficients from the adaptive loop filter 43.

The lossless encoding section 36 performs lossless encoding such as variable length encoding (e.g., CAVLC (context adaptive variable length coding)), arithmetic encoding (e.g., CABAC (context adaptive binary arithmetic coding)), or the like on the quantized value supplied from the quantization section 35.

Further, the lossless encoding unit 36 performs lossless encoding on the intra prediction mode information or the inter prediction mode information, the motion vector, information specifying the reference picture, the offset filter information, and the filter coefficient as encoding information relating to encoding. The division flag is losslessly encoded as encoding information of each CTU. The lossless encoding unit 36 outputs the encoded information subjected to lossless encoding and the quantized value to the accumulation buffer 37 as encoded data to accumulate.

The accumulation buffer 37 temporarily stores the encoded data supplied from the lossless encoding section 36. Further, the accumulation buffer 37 outputs the encoded data stored therein to the transmission section 14 as an encoded bit stream together with the parameter set supplied from the setting section 12 of fig. 5.

The quantized value output from the quantization unit 35 is also input to the dequantization unit 38. The dequantization unit 38 dequantizes the quantized value. The dequantization unit 38 outputs the dequantized orthogonal transform processing result to the inverse orthogonal transform unit 39.

The inverse orthogonal transform unit 39 performs inverse orthogonal transform processing on the orthogonal transform processing result supplied from the dequantization unit 38 on a CU-by-CU basis. As a method for inverse orthogonal transform, for example, IDCT (inverse discrete cosine transform) and IDST (inverse discrete sine transform) can be used. The inverse orthogonal transform unit 39 outputs residual information obtained as a result of the inverse orthogonal transform process to the addition unit 40.

The adder 40 adds the residual information supplied from the inverse orthogonal transform unit 39 and the predicted image supplied from the predicted image selector 48 to perform decoding. The adder 40 outputs the decoded image to the deblocking filter 41 and the frame memory 44.

The deblocking filter 41 performs a deblocking filtering process for removing block distortion of the decoded image supplied from the addition section 40. The deblocking filter 41 outputs the image after the deblocking filtering process to the adaptive offset filter 42.

The adaptive offset filter 42 performs an adaptive offset filter (SAO (sample adaptive offset)) process to mainly eliminate ringing of an image after the deblocking filtering process by the deblocking filter 41.

The adaptive offset filter 42 outputs the image after the adaptive offset filtering process to the adaptive loop filter 43. Further, the adaptive offset filter 42 outputs information indicating the type of the adaptive offset filtering process and the offset to the lossless coding section 36 as offset filter information.

The adaptive loop filter 43 includes, for example, a two-dimensional Wiener (Wiener) filter. The adaptive loop filter 43 performs an Adaptive Loop Filter (ALF) process on the image after the adaptive offset filtering process.

The adaptive loop filter 43 outputs the image after the adaptive loop filter processing to the frame memory 44. The adaptive loop filter 43 outputs the filter coefficient used in the adaptive loop filter process to the lossless encoding unit 36.

The frame memory 44 accumulates the image supplied from the adaptive loop filter 43 and the image supplied from the addition section 40. The image adjacent to the CU among the images accumulated in the frame memory 44 and not subjected to any filtering process is output as a surrounding image to the intra prediction section 46 through the switch 45. On the other hand, the image accumulated in the frame memory 44 and subjected to the filter processing is output as a reference image to the motion prediction and compensation section 47 through the switch 45.

The intra prediction unit 46 performs intra prediction processing of all intra prediction modes considered as candidates using the surrounding image read out from the frame memory 44 through the switch 45 on a CU-by-CU basis.

Further, the intra prediction unit 46 calculates RD costs for all intra prediction modes considered as candidates based on the image read out from the picture image sorting buffer 32 and the predicted image predicted by the intra prediction process. The calculated RD cost is output to the control unit 11. The control section 11 determines the intra prediction mode in which the lowest RD cost becomes the smallest as the optimal intra prediction mode.

The intra prediction unit 46 outputs the prediction image generated in the optimal intra prediction mode to the prediction image selection unit 48 in accordance with the instruction from the control unit 11. When notifying the intra prediction unit 46 of the selection of the prediction image generated in the optimal intra prediction mode from the control unit 11, the intra prediction unit 46 outputs the intra prediction mode information to the lossless encoding unit 36. Note that the intra prediction mode is a mode indicating the size of the CU, the prediction direction, and the like.

The motion prediction and compensation section 47 performs motion prediction and compensation processing in units of CUs in all the inter prediction modes considered as candidates. Specifically, the motion prediction and compensation section 47 detects motion vectors in units of CUs in all the inter prediction modes considered as candidates based on the image supplied from the picture image sorting buffer 32 and the reference image supplied from the picture image sorting buffer 32. Then, the motion prediction and compensation section 47 performs compensation processing on the reference image in CU units based on the motion vector to generate a prediction image.

At this time, the motion prediction and compensation unit 47 calculates the RD costs of all the inter prediction modes considered as candidates based on the image supplied from the picture image sorting buffer 32 and the prediction image. The calculated RD cost is output to the control unit 11. The control section 11 determines the inter prediction mode in which the RD cost becomes the lowest as the optimal inter prediction mode.

The motion prediction and compensation unit 47 outputs the prediction image in the optimal inter prediction mode to the prediction image selection unit 48 in accordance with an instruction from the control unit 11. When the motion prediction and compensation unit 47 is notified of the selection of the optimal inter prediction mode prediction image from the control unit 11, the motion prediction and compensation unit 47 outputs the inter prediction mode information, the corresponding motion vector, the information specifying the reference image, and the like to the lossless encoding unit 36. Note that the inter prediction mode is a mode indicating the size of the CU and the like.

The control unit 11 determines one of the optimal intra prediction mode and the optimal inter prediction mode, which has a low RD cost, as the optimal prediction mode. Then, the predicted image selecting unit 48 outputs the predicted image in the optimal prediction mode to the calculating unit 33 and the adding unit 40 in accordance with the instruction of the control unit 11.

The rate control section 49 controls the rate of the quantization operation of the quantization section 35 based on the encoded data accumulated in the accumulation buffer 37 so that overflow or underflow does not occur.

(operation of the encoding apparatus)

Fig. 8 is a flowchart illustrating a stream generation process of the encoding apparatus of fig. 5.

In step S11 of fig. 8, the setting unit 12 of the encoding device 10 sets a parameter set. The setting section 12 outputs the set parameter set to the encoding section 13.

In step S12, the encoding unit 13 performs encoding processing for encoding an image in units of frames. Details of this encoding process are described with reference to fig. 9 and 10 described later. The encoded data is generated by an encoding process. The encoded data is accumulated in the accumulation buffer 37.

Note that the partition structure, prediction mode, and the like of the CU are determined based on the RD cost calculated by the encoding process in step S12. The partition structure and prediction mode of the CU based on the RD cost are used in the encoding process of step S12.

In step S13, a coded stream is generated from the parameter sets supplied from the setting section 12 and the accumulated coded data, and the coded stream is output to the transmission section 14.

In step S14, the transmission section 14 transmits the encoded stream supplied from the encoding section 13 to the decoding apparatus 110 described later, and the process ends.

Fig. 9 and 10 are flowcharts showing details of the encoding process in step S12 of fig. 8.

In step S31 of fig. 9, the a/D conversion section 31 of the encoding section 13 converts an image in units of frames input as an encoding target. The a/D conversion section 31 outputs the converted image in the form of a digital signal to the screen image sorting buffer 32 to be stored.

In step S32, the picture image sorting buffer 32 sorts the images in units of frames in the display order into images in the encoding order according to the GOP structure. The screen image sorting buffer 32 outputs the sorted images in units of frames to the calculation unit 33, the intra prediction unit 46, and the motion prediction and compensation unit 47.

In step S33, the control section 11 executes CU partition structure determination processing. The details of the CU division structure determination process are described with reference to fig. 11 described later. The CU of the partition structure determined by the CU partition structure determination process is used in the subsequent encoding process. The partition structure information on the CU is output to the associated block. Further, the division flag determined by the CU division structure determination process is output to the lossless encoding section 36.

In step S34, the intra prediction unit 46 performs intra prediction processing on all the intra prediction modes that become candidates on a CU-by-CU basis. Further, the intra prediction unit 46 calculates RD costs for all intra prediction modes considered as candidates based on the image read out from the picture image sorting buffer 32 and the predicted image predicted by the intra prediction process. The control section 11 determines the intra prediction mode in which the RD cost becomes the lowest as the optimal intra prediction mode. The intra prediction unit 46 outputs the prediction image generated in the optimal intra prediction mode to the prediction image selection unit 48.

Further, the motion prediction and compensation section 47 performs motion prediction and compensation processing on all the inter prediction modes considered as candidates on a CU basis. Further, the motion prediction and compensation section 47 calculates RD costs for all the inter prediction modes considered as candidates based on the image supplied from the picture image sorting buffer 32 and the prediction image. The control section 11 determines the inter prediction mode in which the RD cost becomes the lowest as the optimal inter prediction mode. The motion prediction and compensation unit 47 outputs the prediction image in the optimal inter prediction mode to the prediction image selection unit 48.

In step S35, the control section 11 determines one of the modes in which the RD cost becomes the lowest as the optimal prediction mode based on the RD costs of the optimal intra prediction mode and the optimal inter prediction mode. Then, the predicted image selector 48 outputs the predicted image in the optimal prediction mode to the calculator 33 and the adder 40.

In step S36, the control unit 11 determines whether or not the optimal prediction mode is the optimal inter prediction mode. When determining in step S36 that the optimal prediction mode is the optimal inter prediction mode, the control unit 11 notifies the motion prediction compensation unit 47 of the selection of the prediction image generated in the optimal inter prediction mode.

Then, in step S37, the motion prediction and compensation section 47 outputs the inter prediction mode information, the motion vector, and the information specifying the reference picture to the lossless encoding section 36. Thereafter, the process proceeds to step S39.

In contrast, in the case where the optimal prediction mode is the optimal intra prediction mode in step S36, the control unit 11 notifies the intra prediction unit 46 of selection of the prediction image generated by the optimal intra prediction mode. Then, in step S38, the intra prediction unit 46 outputs the intra prediction mode information to the lossless encoding unit 36. Thereafter, the process proceeds to step S39.

In step S39, the calculation section 33 performs encoding by subtracting the predicted image supplied from the predicted image selection section 48 from the image supplied from the screen image sorting buffer 32. The calculation unit 33 outputs the subtracted image to the orthogonal transformation unit 34 as residual information.

In step S40, the orthogonal transform unit 34 performs orthogonal transform processing on the residual information on a CU-by-CU basis. The orthogonal transform unit 34 outputs the orthogonal transform processing result after the orthogonal transform processing to the quantization unit 35.

In step S41, the quantization unit 35 quantizes the orthogonal transform processing result supplied from the orthogonal transform unit 34. The quantization unit 35 outputs the quantized value to the lossless encoding unit 36 and the dequantization unit 38.

In step S42 of fig. 10, the dequantization unit 38 dequantizes the quantized value from the quantization unit 35. The dequantization unit 38 outputs the dequantized orthogonal transform processing result to the inverse orthogonal transform unit 39.

In step S43, the inverse orthogonal transform unit 39 performs inverse orthogonal transform processing on the orthogonal transform processing result in CU units. The inverse orthogonal transform unit 39 outputs the residual information after the inverse orthogonal transform process to the addition unit 40.

In step S44, the adder 40 adds the residual information supplied from the inverse orthogonal transform unit 39 and the predicted image supplied from the predicted image selector 48 to perform decoding. The adder 40 outputs the decoded image to the deblocking filter 41 and the frame memory 44.

In step S45, the deblocking filter 41 performs a deblocking filtering process on the image supplied from the addition section 40. The deblocking filter 41 outputs the image after the deblocking filtering process to the adaptive offset filter 42.

In step S46, the adaptive offset filter 42 performs the adaptive offset filtering process on the image after the deblocking filtering process. The adaptive offset filter 42 outputs the image after the adaptive offset filtering process to the adaptive loop filter 43. The adaptive offset filter 42 outputs the offset filter information to the lossless encoding unit 36.

In step S47, the adaptive loop filter 43 performs the adaptive loop filter process on the image after the adaptive offset filter process. The adaptive loop filter 43 outputs the image after the adaptive loop filter processing to the frame memory 44. The adaptive loop filter 43 outputs the filter coefficient used in the adaptive loop filter process to the lossless encoding unit 36.

In step S48, the frame memory 44 accumulates the image supplied from the adaptive loop filter 43 and the image supplied from the addition section 40. The image adjacent to the CU among the images accumulated in the frame memory 44 and not subjected to the filtering process is output as a surrounding image to the intra prediction section 46 through the switch 45. In contrast, the image accumulated in the frame memory 44 and subjected to the filter processing is output as a reference image to the motion prediction and compensation section 47 through the switch 45.

In step S49, the lossless encoding unit 36 performs lossless encoding on the intra prediction mode information or the inter prediction mode information, the motion vector, the information specifying the reference picture, the offset filter information, and the filter coefficient as encoding information. The division flag is losslessly encoded as encoding information of each CTU.

In step S50, the lossless encoding section 36 performs lossless encoding on the quantized value supplied from the quantization section 35. Then, the lossless encoding section 36 generates encoded data from the encoded information that has been lossless encoded by the processing in step S49 and the quantization value that has been lossless encoded, and outputs the encoded data to the accumulation buffer 37.

In step S51, the accumulation buffer 37 temporarily accumulates the encoded data supplied from the lossless encoding section 36.

In step S52, the rate control section 49 controls the rate of the quantization operation of the quantization section 35 based on the encoded data accumulated in the accumulation buffer 37 so that overflow or underflow does not occur. Then, the process returns to step S12 of fig. 8, and then proceeds to step S13.

It is to be noted that although, in the encoding processes of fig. 9 and 10, the intra prediction process and the motion prediction and compensation process are generally performed for the sake of simplifying the description, in actual use, in some cases, depending on the picture type or the like, only any one of the above-described processes may be performed.

Fig. 11 is a flowchart showing details of the CU division structure determination process in step S33 of fig. 9.

In step S121 of fig. 11, the control unit 11 determines whether or not the CTUs are divided by QT.

In the case where it is determined in step S121 that the CTUs are divided by QT, the process proceeds to step S122.

In step S122, the control portion 11 determines whether the size of each CU of Y after the QT division is not 8 × 8.

In a case where it is determined in step S122 that the size of each CU of Y after the QT division is not 8 × 8, the process proceeds to step S123.

In step S123, the encoding unit 13 calculates the RD cost of the CU divided by QT.

In contrast, in the case where it is determined in step S121 that the CTU is not divided by QT, or in the case where it is determined in step S122 that the size of each CU of Y after QT division is 8 × 8, the process proceeds to step S124.

In step S124, the encoding unit 13 performs the RD cost calculation process for the CU that is not divided.

In step S125, the encoding unit 13 performs the RD cost calculation process for the CU divided by BT. Details of the calculation process of the RD cost of a CU divided by BT are described with reference to fig. 12 described later. By executing step S125, the RD cost of the CU divided by BT is calculated.

In step S126, the encoding unit 13 performs the RD cost calculation process for the CU divided by TT. Details of the calculation process of the RD cost of the CU divided by TT are described with reference to fig. 13 described later. By executing step S126, the RD cost of the CU divided by TT is calculated.

In step S127, the control unit 11 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the RD costs calculated as described above.

Fig. 12 is a flowchart showing details of the RD cost calculation process divided by BT in step S125 of fig. 11.

In step S141 of fig. 12, the control portion 11 determines whether the size of each CU of Y after the horizontal BT division is not 8 × 8.

In the case where it is determined in step S141 that the size of each CU of Y after the horizontal BT division is not 8 × 8, the process proceeds to step S142.

In step S142, the encoding section 13 performs horizontal division to calculate the RD cost.

In the case where it is determined in step S141 that the size of each CU of Y after the horizontal BT division is 8 × 8, the process skips step S142 and proceeds to step S143.

In step S143, the control portion 11 determines whether the size of each CU of Y after the vertical BT division is not 8 × 8.

In a case where it is determined in step S143 that the size of each CU of Y after the vertical BT division is not 8 × 8, the process proceeds to step S144.

In step S144, the encoding section 13 divides vertically and calculates RD cost.

In the case where it is determined in step S143 that the size of each CU of Y after the vertical BT division is 8 × 8, the process skips step S144 and proceeds to step S145.

In step S145, the control unit 11 adopts the division structure with the lowest RD cost among the calculated RD costs, and uses the adopted division structure for the RD costs of the CUs divided by BT.

Fig. 13 is a flowchart showing details of the RD cost calculation processing divided by TT in step S126 of fig. 11.

In step S161 of fig. 13, the control section 11 determines whether the size of each CU of Y after horizontal TT division is 8 × 8.

In a case where it is determined in step S161 that the size of each CU of Y after horizontal TT division is not 8 × 8, the process proceeds to step S162.

In step S162, the encoding section 13 is divided horizontally to calculate the RD cost.

In the case where it is determined in step S161 that the size of each CU of Y after the horizontal TT division is 8 × 8, the process skips step S162 and proceeds to step S163.

In step S163, the control section 11 determines whether the size of each CU of Y after the vertical TT division is not 8 × 8.

In a case where it is determined in step S163 that the size of each CU of Y after the vertical TT division is not 8 × 8, the process proceeds to step S164.

In step S164, the encoding section 13 divides vertically and calculates RD cost.

In the case where it is determined in step S163 that the size of each CU of Y after the vertical TT division is not 8 × 8, the process skips step S164 and proceeds to step S165.

In step S165, the control section 11 adopts the division structure with the lowest RD cost among the calculated RD costs, and uses the adopted division structure for the RD costs of the CUs divided by TT.

In the partition structure determination processing of CUs, a case where the size of each CU of Y becomes smaller than 8 × 8 is excluded in the manner described above. In particular, the partition structure of the CU in the case where the size of each CU of Cb/Cr becomes 2 × 2 may be excluded from the selection for determining the CU partition structure. Accordingly, the delay of intra prediction can be reduced.

It is to be noted that the description given above is directed to the case where the size of each CU of Cb/Cr becomes 2 × 2 is excluded from the selection for determining the CU division structure. As a different method, a method may be used which does not use a case where the size of each CU of Cb/Cr becomes 2 × 2 when intra prediction is performed.

Fig. 14 is a flowchart illustrating another example of the CU division structure determination process in step S33 of fig. 9.

Note that the processing in fig. 14 is the same as that of the normal CU division structure determination processing, except for the determination processing of the prediction mode of the CU in step S186.

In step S181 of fig. 14, the control unit 11 determines whether or not the CTUs are divided by QT.

In the case where it is determined in step S181 that the CTUs are divided by QT, the process proceeds to step S182.

In step S182, the encoding unit 13 calculates the RD cost for each CU divided by QT.

In contrast, in the case where it is determined in step S181 that the CTU is not divided by QT, the process proceeds to step S183.

In step S183, the encoding unit 13 performs the RD cost calculation process for each CU that is not divided.

In step S184, the encoding unit 13 executes the calculation process of the RD cost per CU divided by BT. Details of the calculation process of the RD cost of each CU divided by BT will be described with reference to fig. 15 described later. In step S184, the RD cost of the CU divided by BT is calculated.

In step S185, the encoding unit 13 performs the calculation process of the RD cost per CU divided by TT. Details of the calculation process of the RD cost of each CU divided by TT will be described with reference to fig. 16 described later. In step S185, the RD cost of the CU divided by TT is calculated.

In step S186, the control unit 11 executes the prediction mode determination process for the CU. Details of the prediction mode determination processing of the CU are described with reference to fig. 17 described later. By step S186, the RD cost of the CU of Cr/Cb other than the intra prediction mode of the CU of size 2 × 2 is obtained.

In step S187, the control unit 11 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

Fig. 15 is a flowchart showing details of the calculation process of the RD cost divided by BT in step S184 of fig. 14.

In step S201 of fig. 15, the encoding section 13 divides horizontally and calculates the RD cost.

In step S202, the encoding section 13 divides vertically and calculates RD cost.

In step S203, the control portion 11 adopts the division structure in which the RD cost is the lowest among the calculated RD costs, and uses the adopted division structure for the RD costs of CUs divided by BT.

Fig. 16 is a flowchart showing details of the RD cost calculation processing divided by TT in step S185 of fig. 14.

In step S221 of fig. 16, the encoding section 13 divides horizontally and calculates the RD cost.

In step S322, the encoding section 13 divides vertically and calculates RD costs.

In step S223, the control section 11 adopts the division structure in which the RD cost is the lowest among the calculated RD costs, and uses the adopted division structure for the RD costs of CUs divided by TT.

Fig. 17 is a flowchart showing details of the CU prediction mode determination processing in step S186 of fig. 14.

In step S241 of fig. 17, the control section 11 determines whether the size of each CU of Cb/Cr is 4 × 4 or more.

In the case where it is determined in step S241 that the size of each CU of Cb/Cr is 4 × 4 or more, the processing proceeds to step S242.

In step S242, the encoding unit 13 calculates the RD cost using intra prediction. Thereafter, the process proceeds to step S243.

In contrast, in the case where it is determined in step S241 that the size of each CU of Cb/Cr is not 4 × 4 or more, the process skips step S242 and proceeds to step S243.

In step S243, the encoding unit 13 calculates the RD cost using inter prediction.

In step S244, the control unit 11 adopts the division structure with the lowest RD cost among the calculated RD costs, and uses the adopted division structure for the RD cost of the CU.

In the partition structure determination processing of the CU, the intra prediction calculates the RD cost only in the case where the size of each CU of Cb/Cr is 4 × 4 or more. In other words, also in the case of this processing, the partition structure of the CU in the case where the size of each CU of Cb/Cr is 2 × 2 can be excluded from the selection for determining the CU partition structure. This can reduce the delay of intra prediction.

(example of configuration of embodiment of decoding apparatus)

Fig. 18 is a block diagram showing an example of the configuration of an embodiment of a decoding apparatus that decodes a coded stream transmitted from the encoding apparatus of fig. 5 and to which the present disclosure is applied.

The decoding device 110 of fig. 18 includes a control unit 111, a receiving unit 112, an extracting unit 113, and a decoding unit 114.

The control unit 111 of the decoding device 110 includes, for example, a CPU, a microprocessor, and the like. The control section 111 executes a predetermined program by the CPU to perform various types of calculation processing or operation control of associated components, and operates as a central processing device in the decoding device 110.

The control section 111 extracts information included in the parameter set. The control section 111 controls the decoding section 114 based on the information included in the parameter set. Specifically, the control section 111 outputs information included in a parameter set (described below with reference to fig. 27) to the decoding section 114, and causes the decoding section 114 to execute the CU partition structure determination process.

The receiving unit 112 receives the coded stream transmitted from the encoding device 10 of fig. 8, and outputs the coded stream to the extracting unit 113.

The extraction section 113 extracts a parameter set and encoded data from the encoded stream supplied from the reception section 112. The extraction unit 113 outputs the parameter set to the control unit 111. The extraction unit 113 outputs the encoded data to the decoding unit 114.

The decoding unit 114 decodes the encoded data supplied from the extracting unit 113 by a method conforming to the VVC method in accordance with an instruction from the control unit 111. At this time, the decoding unit 114 performs the partition structure determination process on the CU according to the instruction of the control unit 111, and decodes the encoded data using the CU of the determined partition structure. The decoding unit 114 outputs the decoded image.

(example of configuration of decoding section)

Fig. 19 is a block diagram describing an example of the configuration of the decoding section of fig. 18.

The decoding unit 114 in fig. 19 includes an accumulation buffer 131, a lossless decoding unit 132, a dequantization unit 133, an inverse orthogonal transform unit 134, an addition unit 135, a deblocking filter 136, an adaptive offset filter 137, an adaptive loop filter 138, and a picture image sorting buffer 139. The decoding section 114 further includes a D/a conversion section 140, a frame memory 141, a switch 142, an intra prediction section 143, a motion compensation section 144, and a switch 145.

The accumulation buffer 131 of the decoding section 114 receives and accumulates the encoded data from the extraction section 113 of fig. 18. The accumulation buffer 131 outputs the encoded data accumulated therein to the lossless decoding section 132.

The lossless decoding section 132 performs lossless decoding such as variable length decoding or arithmetic decoding on the encoded data from the accumulation buffer 131 to obtain a quantized value and encoded information. The lossless decoding unit 132 outputs the quantized value to the dequantization unit 133. The encoding information is configured by a division flag, intra prediction mode information, inter prediction mode information, a motion vector, information for specifying a reference picture, offset filter information, a filter coefficient, and the like.

The lossless decoding unit 132 executes CU partition structure determination processing using the partition flag in accordance with an instruction from the control unit 111. The CU partition structure can be found by reading out flags corresponding to branches of the tree from the partition flags.

In the decoding section 114, the CU of the partition structure determined by the encoding device 10 and found by the lossless decoding section 132 is used.

The lossless decoding unit 132 also outputs the intra prediction mode information and the like to the intra prediction unit 143. The lossless decoding unit 132 outputs the motion vector, the inter prediction mode information, the information for specifying the reference picture, and the like to the motion compensation unit 144.

The lossless decoding unit 132 outputs the intra prediction mode information or the inter prediction mode information to the switch 145. The lossless decoding unit 132 outputs the offset filter information to the adaptive offset filter 137. The lossless decoding unit 132 outputs the filter coefficient to the adaptive loop filter 138.

The dequantization unit 133, the inverse orthogonal transform unit 134, the addition unit 135, the deblocking filter 136, the adaptive offset filter 137, the adaptive loop filter 138, the frame memory 141, the switch 142, the intra prediction unit 143, and the motion compensation unit 144 perform processes similar to those of the dequantization unit 38, the inverse orthogonal transform unit 39, the addition unit 40, the deblocking filter 41, the adaptive offset filter 42, the adaptive loop filter 43, the frame memory 44, the switch 45, the intra prediction unit 46, and the motion prediction and compensation unit 47 of fig. 7, respectively. Thus, the picture is decoded.

In particular, the dequantizing section 133 is configured similarly to the dequantizing section 38 of fig. 7. The dequantization unit 133 dequantizes the quantized value from the lossless decoding unit 132 on a CU-by-CU basis. The dequantization unit 133 outputs the dequantized orthogonal transform processing result to the inverse orthogonal transform unit 134.

The inverse orthogonal transform unit 134 is configured similarly to the inverse orthogonal transform unit 39 of fig. 7. The inverse orthogonal transform unit 134 performs an inverse orthogonal transform process on the orthogonal transform process result supplied from the dequantization unit 133. The inverse orthogonal transform unit 134 outputs the residual information after the inverse orthogonal transform process to the addition unit 135.

The adder 135 adds the residual information supplied from the inverse orthogonal transform unit 134 and the prediction image supplied from the switch 145 to perform decoding. The adder 135 outputs the decoded image to the deblocking filter 136 and the frame memory 141.

The deblocking filter 136 performs a deblocking filtering process on the image supplied from the addition section 135, and outputs the image after the deblocking filtering process to the adaptive offset filter 137.

The adaptive offset filter 137 performs the type of adaptive offset filtering process indicated by the offset filter information on the image after the deblocking filtering process using the offset indicated by the offset filter information from the lossless decoding unit 132. The adaptive offset filter 137 outputs the image after the adaptive offset filtering process to the adaptive loop filter 138.

The adaptive loop filter 138 performs an adaptive loop filter process on the image supplied from the adaptive offset filter 137 using the filter coefficient supplied from the lossless decoding section 132. The adaptive loop filter 138 outputs the image after the adaptive loop filter process to the frame memory 141 and the picture image sorting buffer 139.

The picture image sorting buffer 139 stores the image after the adaptive loop filter processing in units of frames. The screen image sorting buffer 139 sorts images in frame units in the encoding order into images in the original display order, and outputs the sorted images to the D/a conversion section 140.

The D/a conversion section 140 performs D/a conversion on the image in units of frames supplied from the screen image sorting buffer 139, and outputs the resultant image.

The frame memory 141 accumulates the image after the adaptive loop filter processing and the image supplied from the addition section 135. Among the images stored in the frame memory 141 and not subjected to any filtering process, the image adjacent to the CU is supplied as a surrounding image to the intra prediction section 143 through the switch 142. Meanwhile, the image stored in the intra memory 141 and subjected to the filter processing is output as a reference image to the motion compensation section 144 through the switch 142.

The intra prediction unit 143 performs intra prediction processing of the optimal intra prediction mode indicated by the intra prediction mode information supplied from the lossless decoding unit 132, using the surrounding image read out from the frame memory 141 through the switch 142. The intra prediction unit 143 outputs the prediction image generated as a result of the intra prediction process to the switch 145.

The motion compensation section 144 reads out the reference picture specified by the information supplied from the lossless decoding section 132 and specifying the reference picture from the frame memory 141 through the switch 142. The motion compensation unit 144 executes motion compensation processing of the optimal inter prediction mode indicated by the inter prediction mode information supplied from the lossless decoding unit 132, using the reference image and the motion vector supplied from the lossless decoding unit 132. The motion compensation unit 144 outputs the prediction image generated as a result of the motion compensation process to the switch 145.

When the intra prediction mode information is supplied from the lossless decoding unit 132, the switch 145 outputs the prediction image supplied from the intra prediction unit 143 to the adder 135. In contrast, in the case where the inter prediction mode information is supplied from the lossless decoding unit 132, the switch 145 outputs the prediction image supplied from the motion compensation unit 144 to the addition unit 135.

(description of processing of decoding apparatus)

Fig. 20 is a flowchart illustrating an image generation process of the decoding apparatus of fig. 18.

In step S301 of fig. 20, the receiving unit 112 of the decoding device 110 receives the coded stream transmitted from the encoding device 10 of fig. 8, and outputs the coded stream to the extracting unit 113.

In step S302, the extraction section 113 extracts encoded data and parameter sets from the encoded stream supplied from the reception section 112. The extraction unit 113 outputs the encoded data to the decoding unit 114. The extraction unit 113 outputs the parameter set to the control unit 111.

The control section 111 controls the decoding section 114 based on information included in the parameter set.

In step S303, the decoding unit 114 decodes the encoded data supplied from the extracting unit 113 in accordance with the instruction of the control unit 111. Details of the decoding process will be described with reference to fig. 21 described later. After that, the image generation processing ends.

Fig. 21 is a flowchart showing details of the decoding process in step S303 of fig. 20.

In step S331 of fig. 20, the accumulation buffer 131 of the decoding section 114 receives and accumulates encoded data in units of frames from the extraction section 113. The accumulation buffer 131 outputs the encoded data accumulated therein to the lossless decoding unit 132.

In step S332, the lossless decoding section 132 performs lossless decoding on the encoded data from the accumulation buffer 131 to obtain a quantized value and encoded information. The lossless decoding unit 132 outputs the quantized value to the dequantization unit 133.

The lossless decoding unit 132 outputs the intra prediction mode information and the like to the intra prediction unit 143. The lossless decoding unit 132 outputs the motion vector, the inter prediction mode information, the information for specifying the reference picture, and the like to the motion compensation unit 144.

The lossless decoding unit 132 also outputs the intra-prediction mode information or the inter-prediction mode information to the switch 145. The lossless decoding section 132 supplies the offset filter information to the adaptive offset filter 137, and outputs the filter coefficient to the adaptive loop filter 138.

In step S333, the lossless decoding unit 132 executes the partition structure determination process of the CU based on the information included in the parameter set and the partition flag according to the instruction of the control unit 111. Through step S333, the CU partition structure is determined. The components of the decoding section 114 perform respective processes using the CU of the partition structure determined by the lossless decoding section 132.

In step S334, the dequantization section 133 dequantizes the quantized value supplied from the lossless decoding section 132. The dequantization unit 133 outputs the dequantized orthogonal transform processing result to the inverse orthogonal transform unit 134.

In step S335, the inverse orthogonal transform unit 134 performs orthogonal transform processing on the orthogonal transform processing result supplied from the dequantization unit 133.

In step S336, the motion compensation section 144 determines whether or not the inter prediction mode information has been supplied from the lossless decoding section 132. In the case where it is determined in step S335 that the inter prediction mode information has been provided, the process proceeds to step S336.

In step S337, the motion compensation section 144 reads out a reference image based on the reference image specification information supplied from the lossless decoding section 132, and performs motion compensation processing of the optimal inter prediction mode indicated by the inter prediction mode information using the motion vector and the reference image. The motion compensation unit 144 outputs the prediction image after the motion compensation to the adder 135 via the switch 145. Thereafter, the process proceeds to step S339.

In contrast, in the case where it is determined in step S336 that the inter prediction mode information is not supplied, that is, in the case where the intra prediction mode information has been supplied to the intra prediction section 143, the process proceeds to step S338.

In step S338, the intra prediction unit 143 performs intra prediction processing of the intra prediction mode indicated by the intra prediction mode information using the surrounding image read out from the frame memory 141 by the switch 142. The intra prediction unit 143 outputs the prediction image generated as a result of the intra prediction process to the adder 135 via the switch 145. Thereafter, the process proceeds to step S339.

In step S339, the adder 135 adds the residual information supplied from the inverse orthogonal transform unit 134 and the prediction image supplied from the switch 145 to perform decoding. The adder 135 outputs the decoded image to the deblocking filter 136 and the frame memory 141.

In step S340, the deblocking filter 136 performs a deblocking filtering process on the image supplied from the addition section 135 to remove block distortion. The deblocking filter 136 outputs the image after the deblocking filtering process to the adaptive offset filter 137.

In step S341, the adaptive offset filter 137 performs the adaptive offset filtering process on the image after the deblocking filtering process based on the offset filter information supplied from the lossless decoding unit 132. The adaptive offset filter 137 outputs the image after the adaptive offset filtering process to the adaptive loop filter 138.

In step S342, the adaptive loop filter 138 performs an adaptive loop filter process on the image supplied from the adaptive offset filter 137 for each LCU using the filter coefficient supplied from the lossless decoding section 132. The adaptive loop filter 138 supplies the image after the adaptive loop filter process to the frame memory 141 and the picture image sorting buffer 139.

In step S343, the frame memory 141 accumulates the image supplied from the addition section 135 and the image supplied from the adaptive loop filter 138. Among the images accumulated in the frame memory 141 and not subjected to the filter processing, the image adjacent to the CU is supplied as a surrounding image to the intra prediction section 143 through the switch 142. Meanwhile, the image stored in the frame memory 141 and subjected to the filtering process is supplied as a reference image to the motion compensation section 144 through the switch 142.

In step S344, the picture image sorting buffer 139 stores the image supplied from the adaptive loop filter 138 in units of frames, sorts the images in units of frames in the encoding order into images in the original display order, and outputs the resultant images to the D/a conversion section 140.

In step S345, the D/a conversion section 140 performs D/a conversion on the image after the adaptive loop filter processing, and outputs the resultant image. Then, the process returns to step S303 of fig. 20 and ends.

In the CU division structure determination process, the encoding side excludes the case where the size of each CU of Y in the above manner is smaller than 8 × 8. In other words, the partition structure of the CU in the case where the size of each CU of Cb/Cr is 2 × 2 may be excluded from the selection for determining the CU partition structure. Accordingly, delay due to intra prediction can be reduced.

It should be noted that the first embodiment described above is directed to the case where the CU of Y and the CU of Cb/Cr have the same division structure.

According to the technique of the first embodiment, the division of CUs is not reduced, and the processing is ended to avoid a large delay. However, since the CU of Cb/Cr is not reduced, the CU of Y cannot be reduced either.

In fact, if a 4 × 4 size cannot be used for a CU of Y, there is a fear of a decrease in coding efficiency.

Therefore, in the second embodiment described below, a separate tree is applied.

<2. second embodiment (Individual Tree) >

(description of the Individual Tree)

Fig. 22 is a diagram depicting an example of an individual tree.

In fig. 22, a tree of the partition structure of the CU for determining Y and a tree of the partition structure of the CU for determining Cb/Cr are depicted.

As shown in fig. 22, a technique for making the CU of Y and the CU of Cb/Cr have different partition structures from each other is available, and this technique is called a separate tree. Specifically, with respect to the individual tree, CU partition structure determination processing of Y and partition structure determination processing of Cb/Cr are performed.

In view of this, in the second embodiment, the application of the separate tree causes the CU of Y to have a size of 4 × 4, thereby reducing the reduction in coding efficiency.

It should be noted that the example of the first embodiment and the example of the second embodiment are different from each other only in that a separate tree is used and in that CU partition structure determination processing. Therefore, since the configuration and other processes of the encoding device 10 and other processes of the decoding device 110 are substantially the same as each other, the CU partition structure determination process will be described below using only the encoding device 10.

(operation of the encoding apparatus)

Fig. 23 and 24 are flowcharts showing another example of the CU division structure determination process in step S33 of fig. 9.

In step S401 of fig. 23, the control portion 11 determines whether the CTUs of Y have been divided by QT.

In the case where it is determined in step S401 that the CTU of Y has been divided by QT, the process proceeds to step S402.

In step S402, the encoding unit 13 calculates the RD cost of the CU divided by QT.

In contrast, in the case where it is determined in step S401 that the CTU of Y has not been divided by QT, the process proceeds to step S403.

In step S403, the encoding section 13 performs the RD cost calculation process for each CU that is not divided.

In step S404, the encoding section 13 performs the calculation process of the RD cost per CU divided by BT. The details of the RD cost calculation process for each CU divided by BT are substantially the same as the RD cost calculation process divided by BT of fig. 15 described above. By executing step S404, the calculation of the RD cost of the CU divided by BT is not subject to any limitation by the CU.

In step S405, the encoding unit 13 performs the calculation process of the RD cost for each CU divided by TT. The details of the calculation process of the RD cost per CU divided by TT are substantially the same as the above-described calculation process of the RD cost divided by TT of fig. 16. By executing step S405, the RD cost of the CU divided by TT is calculated without any restriction by the CU.

In step S406, the control unit 11 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

In step S407 of fig. 24, the control section 11 determines whether the CTUs of Cb/Cr have been divided by QT.

In the case where it is determined in step S407 that the CTUs of Cb/Cr have been divided by QT, the processing proceeds to step S408.

In step S408, the control portion 11 determines whether the size of each CU of Cb/Cr after QT division is not 2 × 2.

In the case where it is determined in step S408 that the size of each CU of Cb/Cr after QT division is not 2 × 2, the process proceeds to step S409.

In step S409, the encoding unit 13 calculates the RD cost for each CU divided by QT.

In contrast, in the case where it is determined in step S407 that the CTU of Cb/Cr has not been divided by QT, or in the case where it is determined in step S408 that the size of each CU of Cb/Cr after QT division is 2 × 2, the processing proceeds to step S410.

In step S410, the encoding unit 13 performs the RD cost calculation process for each CU that is not divided.

In step S411, the encoding section 13 performs the calculation process of the RD cost per CU divided by BT. Details of the calculation process of the RD cost of each CU divided by BT are described with reference to fig. 28 described later. By executing step S411, the RD cost of the CU divided by BT is calculated.

In step S412, the encoding unit 13 performs the calculation process of the RD cost per CU divided by TT. Details of the calculation process of the RD cost of each CU divided by TT are described with reference to fig. 26 described later. By executing step S412, the RD cost of the CU divided by TT is calculated.

In step S413, the control unit 11 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

Note that the above-described steps S401 to S406 correspond to the CU partition structure determination processing of Y. Steps S407 to S413 correspond to the CU partition structure determination processing of Cr/Cb.

Fig. 25 is a flowchart showing details of the calculation process of the RD cost divided by BT in step S411 of fig. 24.

In step S431 of fig. 25, the control portion 11 determines whether the size of each CU of Cb/Cr after the horizontal BT division is not 2 × 2.

In the case where it is determined in S431 that the size of each CU of Cb/Cr after horizontal BT division is not 2 × 2, the process proceeds to step S432.

In step S432, the encoding section 13 divides horizontally and calculates the RD cost.

In the case where it is determined in step S431 that the size of each CU of Cb/Cr after horizontal BT division is 2 × 2, the processing skips step S432 and proceeds to step S433.

In step S433, the control portion 11 determines whether the size of each CU of Cb/Cr after the vertical BT division is not 2 × 2.

If it is determined in step S433 that the size of each CU of Cb/Cr after the vertical BT division is not 2 × 2, the process proceeds to step S434.

In step S434, the encoding section 13 divides vertically and calculates the RD cost.

In the case where it is determined in step S433 that the size of each CU of Cb/Cr after the vertical BT division is 2 × 2, the process skips step S434 and proceeds to step S435.

In step S435, the control unit 11 adopts the division structure with the lowest RD cost among the calculated RD costs, and uses the division structure for the RD costs of CUs divided by BT.

Fig. 26 is a flowchart showing details of the calculation processing of the RD cost divided by TT in step S412 of fig. 24.

In step S451 of fig. 26, the control section 11 determines whether the size of each CU of Cb/Cr after horizontal TT division is not 2 × 2.

In a case where it is determined in step S451 that the size of each CU of Cb/Cr after horizontal TT division is not 2 × 2, the process proceeds to step S452.

In step S452, the encoding section 13 divides horizontally and calculates the RD cost.

In the case where it is determined in step S451 that the size of each CU of Cb/Cr after horizontal TT division is 2 × 2, the process skips S452 and proceeds to step S453.

In step S453, the control section 11 determines whether the size of each CU of Cb/Cr after the vertical TT division is not 2 × 2.

In a case where it is determined in step S453 that the size of each CU of Cb/Cr after vertical TT division is not 2 × 2, the process proceeds to step S454.

In step S454, the encoding section 13 divides in the vertical direction and calculates the RD cost.

In the case where it is determined in step S453 that the size of each CU of Cb/Cr after vertical TT division is 2 × 2, the process skips step S454 and proceeds to step S455.

In step S455, the control unit 11 adopts the division structure with the lowest RD cost among the calculated RD costs, and uses the division structure for the RD costs of the CUs divided by TT.

The case where the size of each CU of Cb/Cr is smaller than 2 × 2 as described above is excluded in the CU division structure determination processing. Therefore, the partition structure of the CU in the case where the size of each CU of Cb/Cr is 2 × 2 can be excluded from the selection for determining the partition structure of the CU. Accordingly, the delay of intra prediction can be reduced.

Further, since a separate tree is used, 4 × 4 can be used as the size of each CU of Y. Therefore, the reduction of the coding efficiency can be reduced.

It is to be noted that, although the above description is directed to an example of a case in which the encoding side avoids the size of each CU of Cb/Cr becoming 2 × 2, in the embodiment described below, the partition structure of the CU is limited by a high-level syntax.

<3. third embodiment (adding minimum CU to syntax) >

(description of syntax)

Fig. 27 is a diagram depicting an example of semantics with respect to an individual tree.

In fig. 27, semantics defining information required for decoding an SPS related to an individual tree are extracted and indicated.

Semantics define information about resolution, 4:2:0 chroma format, etc. In particular, since SPS or the like is used by the overall encoding of a moving image, it is referred to as a high level syntax.

Note that in fig. 27, numerals are added on the left side for convenience of description.

In the first to three rows, "qtbtt _ dual _ tree _ intra _ flag equal to 1 is indicated to specify that each CTU is divided into coding units with 64 × 64 luma samples using implicit quadtree partitioning for I slices, and that these coding units are the roots of two separate coding _ quadtree syntax structures for luma and chroma. The description dual _ tree is an alias of the individual tree. The individual tree is also referred to as a dual _ tree. The first to three lines describe information specifying whether or not the individual tree is to be used.

In the fourth row, "log 2_ CTU _ size _ minus2plus2 specifies the luminance code tree block size of each CTU" is indicated. In the fourth row, information specifying the size of a coding tree block of luminance in each CTU is indicated.

In the sixth to eighth rows, "log 2_ min _ qt _ size _ intra _ slices _ minus2plus2 is indicated to specify the minimum luminance of a leaf block resulting from quad-tree division of CTUs in a slice whose slice _ type is equal to 2 (I). The value of log2_ min _ qt _ size _ intra _ slices _ minus2 should be in the range of 0 to Ctbog 2SizeY-2, inclusive ".

In the ninth row, "MinQtLog 2 sizeintrany ═ log2_ min _ qt _ size _ intra _ slices _ minus2+2 (7-22)" is indicated.

In the tenth row, "[ Ed. (BB): the leaves of the quadtree may be the coding units or roots of the nested polytype trees are indicated. ]".

In the eleventh to thirteenth rows, "log 2_ min _ qt _ size _ inter _ slices _ minus2plus2 is indicated to specify the minimum luminance size of a leaf block resulting from quad-tree division of CTUs in a slice whose slice _ type is equal to 0(B) or 1 (P). The value of log2_ min _ qt _ size _ inter _ slices _ minus2 should range between 0 and CtbLog2SizeY-2 (inclusive).

In the fourteenth row, "MinQtLog 2SizeInter ═ log _ min _ qt _ size _ inter _ scales _ minus2+2 (7-23)" is indicated.

In the fifteenth to seventeenth rows, "max _ mtt _ hierarchy _ depth _ inter _ slices are indicated to specify the maximum hierarchical depth of a coding unit resulting from multi-type tree division of quadtree leaves in a slice having a slice _ type equal to 0(B) or 1 (P). max _ mtt _ hierarchy _ depth _ inter _ scales should range from 0 to CtbLog2SizeY-MinTbLog2SizeY, inclusive. Description mtt is an alias of BT/TT. BT/TT is also referred to as mtt ".

In the eighteenth to twentieth rows, "max _ mtt _ hierarchy _ depth _ intra _ slices are indicated to specify the maximum hierarchical depth of a coding unit resulting from multi-type tree division of quadtree leaves in a slice having a slice _ type equal to 2 (I). The values of max _ mtt _ hierarchy _ depth _ intra _ scales range from 0 to CtbLog2SizeY-MinTbLog2SizeY, inclusive.

Fig. 28 is a diagram depicting a partitioning structure of a CU that can be controlled using the information described in fig. 27.

Referring to FIG. 28, the shaded circles represent the QT partition. The black circles indicate the BT/TT split.

The description "log 2_ min _ QT _ size _ intra _ slices _ minus 2" (line 11 of fig. 27) is information specifying the minimum size of the QT partition in the case of an I slice.

In fig. 28, an example of a QT partition with a minimum size of 1 is indicated as "log 2_ min _ QT _ size _ intra _ slices _ minus 2": 1. Expression "log 2_ min _ qt _ size _ intra _ scales _ minus 2": 1 represents the cube of 2 and is interpreted as 8. In other words, 8 × 8 is the minimum value of the QT partition.

If the CTUs are partitioned by QT from the top until 8 × 8 is reached, no further QT partitioning can be performed by "log 2_ min _ QT _ size _ intra _ slices _ minus 2": 1. In the case of FIG. 28, only BT/TT partitioning can be performed after the second top QT partitioning up to 8 × 8.

Further, "max _ mtt _ hierarchy _ depth _ intra _ slices" (the eighteenth row of fig. 27) is information specifying the depth to which BT/TT partitioning is possible after QT partitioning in the case of I slices.

In FIG. 28, an example of a depth of 3 that the BT/TT split may reach is indicated as "max _ mtt _ hierarchy _ depth _ intra _ slices": 3. In the case of fig. 28, after reaching the second top QT partition of 8 × 8, the BT/TT partition may be performed up to the third stage.

In actual use, since the division into 4 × 4 is allowed, if the BT division is performed vertically and the BT division is performed horizontally, 4 × 4 is reached, and therefore the division cannot be performed any more.

Fig. 29 is a diagram depicting an example in which a separate tree is assumed and the control of fig. 28 is made executable for Y and Cb/Cr, respectively.

In fig. 29, "log 2_ min _ qt _ size _ intra _ slices _ luma _ minus 2" and "log 2_ min _ qt _ size _ intra _ slices _ chroma _ minus 2" are indicated. The description "log 2_ min _ QT _ size _ intra _ slices _ luma _ minus 2" is information specifying the minimum size of the QT partition of Y. The description "log 2_ min _ QT _ size _ intra _ slices _ chroma _ minus 2" is information specifying the minimum size of the QT partition of Cb/Cr.

Further, in fig. 29, "max _ mtt _ hierarchy _ depth _ intra _ luma _ slices" and "max _ mtt _ hierarchy _ depth _ intra _ chroma _ slices" are indicated. Description "max _ mtt _ hierarchy _ depth _ intra _ luma _ slices" is information specifying the depth at which BT/TT splitting of Y can be performed. Description "max _ mtt _ hierarchy _ depth _ intra _ chroma _ slices" is information specifying the depth at which BT/TT division of Cb/Cr can be performed.

It should be noted that, although the partition structure of the CU depicted in fig. 29 is similar to that of fig. 28, in fig. 29, only one example of Y or Cb/Cr is depicted. Furthermore, although not described semantically, in VVC software, Y and Cb/Cr may be specified separately only for I slices.

Fig. 30 is a diagram depicting an example in which the information described in fig. 27 is used so that the minimum size of each CU of Cb/Cr alone is 4 × 4.

In fig. 30, an example of depth where the minimum size of the QT partition of Y is 0 is indicated as "log 2_ min _ QT _ size _ intra _ slices _ luma _ minus 2": 0. The description "log 2_ min _ qt _ size _ intra _ slices _ luma _ minus 2" represents the square of 2 and is interpreted as 4. In other words, 4 × 4 is the minimum value of the QT partition of Y.

Further, FIG. 30 depicts an example of depth 3, where BT/TT partitioning may be done, indicated as "max _ mtt _ hierarchy _ depth _ intra _ luma _ slices": 3.

Note that, in fig. 30, an example of the partition structure of the CU of Y is not shown, and only one example of the partition structure of the CU of Cb/Cr is depicted.

In fig. 30, an example in which the minimum size of the QT partition of Cb/Cr is 0 is indicated as "log 2_ min _ QT _ size _ intra _ slices _ chroma _ minus 2": 0. The expression "log 2_ min _ qt _ size _ intra _ slices _ chroma _ minus 2": 0 represents the square of 2 and is interpreted as 4. In other words, 4 × 4 is the minimum of the QT partition of Cb/Cr.

Further, in fig. 30, an example in which the depth at which BT/TT division can be made is 0 is described as "max _ mtt _ hierarchy _ depth _ intra _ chroma _ slices": 0. In the case of fig. 30, BT/TT partitioning cannot be performed any more after reaching the second top QT partitioning of 4 × 4.

By determining the CU division structure in the above manner, the minimum size of each CU of Cb/Cr can be made 4 × 4.

However, since the method described in fig. 30 prohibits the use of BT/TT division, the coding efficiency is lowered. Therefore, as depicted in fig. 31, new information is set.

Fig. 31 is a diagram depicting an example of the partitioning structure of a CU of Cb/Cr set by new information.

In fig. 31, "log 2_ min _ qt _ size _ intra _ slices _ luma _ minus 2" and "log 2_ min _ qt _ size _ intra _ slices _ chroma _ minus 2" described above with reference to fig. 29 are indicated. "max _ mtt _ hierarchy _ depth _ intra _ luma _ slices" and "max _ mtt _ hierarchy _ depth _ intra _ chroma _ slices" are indicated.

Further, in fig. 31, "min _ cu _ size _ idc _ intra _ slice _ luma", "min _ cu _ size _ idc _ inter _ slice _ luma", and "min _ cu _ size _ idc _ intra _ slice _ luma" are newly indicated.

The description "min _ CU _ size _ idc _ intra _ slice _ luma" is information specifying the minimum size of a CU of Y in an I slice (intra prediction).

The description "min _ CU _ size _ idc _ intra _ slice _ chroma" is information specifying the minimum size of a CU of Cb/Cr in an I slice.

Description "min _ CU _ size _ idc _ inter _ slice _ luma" is information specifying the minimum size of a CU of Y in a P/B slice (inter prediction).

Description "min _ CU _ size _ idc _ inter _ slice _ choma" is information specifying the minimum size of a CU of Cb/Cr in a P/B slice.

Further, for CU of Cb/C, 0 to 2 are defined as follows. In particular, in the case where the information specifying the minimum size of the CU of Cb/Cr is 0, a size as small as 2 × 2 may be used. In the case where the information specifying the minimum size of the CU of Cb/Cr is 1, sizes as small as 2 × 4 and 4 × 2 may be used. In the case where the information specifying the minimum size of the CU of Cb/Cr is 2, a size as small as 4 × 4 may be used.

For CU of Y, 0 to 2 are defined as follows. In particular, in the case where the information specifying the minimum size of the CU of Y is 0, a size as small as 4 × 4 may be used. In the case where the information specifying the CU minimum size of Y is 1, sizes as small as 4 × 8 and 8 × 4 may be used. In the case where the information specifying the minimum size of the CU with Y is 2, a size as small as 8 × 8 may be used.

The above new information is transmitted together with the SPS in such a manner that the encoded side is included in the SPS.

(example of configuration of embodiment of encoding apparatus)

Fig. 32 is a block diagram depicting an example of the configuration of an embodiment of an encoding apparatus to which the present disclosure is applied.

The encoding device 210 of fig. 32 includes a control section 211, a setting section 212, an encoding section 13, and a transmission section 14, and encodes an image by a method conforming to the VVC method.

Of the components shown in fig. 32, the same components as those described above with reference to fig. 5 are denoted by the same reference numerals. Duplicate descriptions are appropriately omitted.

Specifically, the control section 211 of the encoding device 210 is configured similarly to the control section 11 of fig. 5, and operates as a central processing device in the encoding device 210.

For example, the control unit 211 controls the setting unit 212 to set "min _ cu _ size _ idc _ intra _ slice _ chroma" and "min _ cu _ size _ idc _ inter _ slice _ chroma" of the SPS.

Further, the control section 211 controls the encoding section 13 to calculate the RD cost, and determines the partition structure of the CU based on the RD cost supplied from the encoding section 13.

At this time, in the case where "min _ CU _ size _ idc _ intra _ slice _ luma" (or min _ CU _ size _ idc _ inter _ slice _ luma) "is 2, the control unit 211 controls the encoding unit 13 to calculate the RD cost except when the size of each CU of Cr/Cb after division is 2 × 2, 2 × 4, or 4 × 2. In this case, when the size of each CU of Cr/Cb is 2 × 2, 2 × 4, or 4 × 2, RD cost is not calculated, and when the size of each CU of Cr/Cb is 2 × 2, 2 × 4, or 4 × 2, a flag is not transmitted.

Meanwhile, in the case where "min _ CU _ size _ idc _ intra _ slice _ luma" (or min _ CU _ size _ idc _ inter _ slice _ luma) "is 1, the control section 211 controls the encoding section 13 to calculate the RD cost except when the size of each CU of Cr/Cb is 2 × 2. In this case, the RD cost when the size of each CU of Cr/Cb is 2 × 2 is not calculated, and the flag when the size of each CU of Cr/Cb is 2 × 2 is not transmitted.

Further, similar to the control portion 11, the control portion 211 determines the prediction mode based on the RD cost. The control section 211 controls the setting section 212 to set a parameter set based on the determined partition structure and prediction mode of the CU, and controls the encoding section 13 to perform encoding processing.

The setting section 212 sets SPS, PPS, VUI, SEI, and the like according to an instruction of the control section 211. In this case, the setting unit 212 sets "min _ cu _ size _ idc _ intra _ slice _ chroma" and "min _ cu _ size _ idc _ inter _ interrupt _ slice _ chroma" of the SPS. The setting section 212 outputs the set parameter sets of SPS, PPS, VUI, SEI, and the like to the encoding section 13.

The image in units of frames is input to the encoding unit 13. The encoding unit 13 encodes the input image by a method conforming to the VVC method in accordance with the instruction of the control unit 211. The encoding section 13 generates an encoded stream from the parameter set and encoded data obtained as a result of the encoding, and outputs the encoded stream to the transmitting section 14.

The transmitting section 14 transmits the encoded stream supplied from the encoding section 13 to a decoding apparatus 410 described later.

(operation of the encoding apparatus)

Fig. 33 is a flowchart showing a stream generation process of the encoding apparatus of fig. 32.

In step S511 of fig. 33, the setting section 212 of the encoding device 210 sets a parameter set. At this time, the setting unit 212 sets "min _ cu _ size _ idc _ intra _ slice _ chroma" and "min _ cu _ size _ idc _ inter _ slice _ chroma" of the SPS. The setting section 212 outputs the set parameter set to the encoding section 13.

In step S512, the encoding unit 13 executes encoding processing for encoding an image in units of frames. This encoding process is basically the same process as the process except for the CU partition structure determination process (S33) in fig. 9 and 10 described above, and therefore, the description of the encoding process is omitted. Encoded data is generated by an encoding process. The encoded data is accumulated in the accumulation buffer 37.

Note that the partition structure, prediction mode, and the like of the CU are determined based on the RD cost calculated by the encoding process in step S512. The partition structure and prediction mode of the CU based on the RD cost are used in the encoding process of step S512.

In step S513, a coded stream is generated from the parameter set supplied from the setting section 212 and the accumulated coded data, and the coded stream is output to the transmission section 14.

In step S514, the transmission section 14 transmits the encoded stream supplied from the encoding section 13 to the decoding apparatus 410 described later, and then ends the processing.

Fig. 34 to 36 are flowcharts showing another example of the CU division structure determination process in step S33 of fig. 9.

Note that fig. 34 depicts processing in the case of I slices.

In step S601 of fig. 34, the control section 211 determines whether or not the CTUs of Y have been divided by QT.

In the case where it is determined in step S601 that the CTU has been divided by QT, the process proceeds to step S602.

In step S602, the encoding unit 13 calculates the RD cost for each CU divided by QT.

In contrast, in the case where it is determined in step S601 that the CTU has not been divided by QT, the process proceeds to step S603.

In step S603, the encoding unit 13 performs the RD cost calculation process for each CU that is not divided.

In step S604, the encoding section 13 performs the calculation process of the RD cost per CU divided by BT. Since the process of calculating the RD cost per CU divided by BT is substantially the same process as the process of fig. 15 described above, a description thereof is omitted. By step S604, the RD cost per CU divided by BT is calculated.

In step S605, the encoding unit 13 performs the calculation process of the RD cost per CU divided by TT. The calculation processing of the RD cost per CU divided by TT is substantially the same processing as the processing of fig. 16 described above, and therefore description thereof is omitted. In step S605, the RD cost per CU divided by TT is calculated.

In step S606, the control unit 211 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

In step S607 of fig. 35, the control section 211 determines whether the CTUs of Cb/Cr have been divided by QT.

In the case where it is determined in step S607 that the CTU has been divided by QT, the processing proceeds to step S608.

In step S608, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 when "min _ CU _ size _ idc _ intra _ slice _ chroma" is 2.

In the case where it is determined in step S608 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S609.

In step S609, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2 when "min _ CU _ size _ idc _ intra _ slice _ chroma" is 1.

In the case where it is determined in step S609 that the size of each CU of Cb/Cr after division is not 2 × 2, the processing proceeds to step S610.

In step S610, the encoding unit 13 calculates the RD cost for each CU divided by QT.

In contrast, in the case where it is determined in step S607 that the CTUs of Cb/Cr have not been divided by QT, or in the case where it is determined in step S608 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S611. Also in the case where it is determined in step S609 that the size of each CU of Cb/Cr after QT division is 2 × 2, the processing proceeds to step S611. In these cases, no RD cost is calculated, and no partitioning flag is sent.

In step S611, the encoding unit 13 performs the RD cost calculation process for each CU that is not divided.

In step S612, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 when "min _ CU _ size _ idc _ intra _ slice _ chroma" ═ 2.

In the case where it is determined in step S612 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S613.

In step S613, the control unit 211 determines whether or not the size of each CU of Cb/Cr is not 2 × 2 when "min _ CU _ size _ idc _ intra _ slice _ chroma" is 1.

In a case where it is determined in step S613 that the size of each CU of Cb/Cr after division is not 2 × 2, the process proceeds to step S614.

In step S614, the encoding section 13 performs the calculation process of the RD cost per CU divided by BT. The RD cost calculation process for each CU divided by BT is basically the same process as the RD cost calculation process divided by BT of fig. 15 described above. By step S614, the RD cost of each CU divided by BT is calculated.

In contrast, in the case where it is determined in step S613 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S615 of fig. 36. Also in the case where it is determined in step S614 that the size of each CU of Cb/Cr after QT division is 2 × 2, the process proceeds to step S615 of fig. 36. In those cases, no RD cost is calculated, and no division flag is set.

In step S615 of fig. 36, the control unit 211 determines whether the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 in the case of "min _ CU _ size _ idc _ intra _ slice _ chroma ═ 2".

In the case where it is determined in step S615 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S616.

In step S616, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2 when "min _ CU _ size _ idc _ intra _ slice _ chroma" is 1.

In the case where it is determined in step S616 that the size of each CU of Cb/Cr after division is not 2 × 2, the processing proceeds to step S617.

In step S617, the encoding unit 13 performs the calculation process of the RD cost per CU divided by TT. The calculation process of the RD cost per CU divided by TT is basically the same process as the calculation process of the RD cost divided by TT of fig. 16 described above. By step S617, the RD cost of each CU divided by TT is calculated.

In contrast, when it is determined in step S615 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S618. Also in the case where it is determined in step S616 that the size of each CU of Cb/Cr after QT division is 2 × 2, the processing proceeds to step S618. In these cases, no RD cost will be calculated, nor the partition flag will be set.

In step S618, the control unit 11 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

It should be noted that, in fig. 34 to 36 described above, steps S601 to S606 are directed to the CU partition structure determination processing of Y. Steps S607 to S618 are directed to CU partition structure determination processing for Cr/Cb.

Fig. 37 to 39 are flowcharts showing another example of the CU division structure determination process in step S33 of fig. 9.

It should be noted that in fig. 37, the processing is described with respect to the case of P/B slices.

In step S651 of fig. 37, the control section 211 determines whether the CTUs of Y have been divided by QT.

In the case where it is determined in step S651 that the CTU of Y has been divided by QT, the process proceeds to step S652.

In step S652, the encoding section 13 calculates the RD cost per CU divided by QT.

In contrast, in the case where it is determined in step S651 that the CTU of Y has not been divided by QT, the process proceeds to step S653.

In step S653, the encoding unit 13 performs the RD cost calculation process for each CU that is not divided.

In step S654, the encoding section 13 performs the calculation process of the RD cost per CU divided by BT. Since the calculation process of the RD cost per CU divided by BT is substantially the same process as the process in fig. 15 described above, a description thereof is omitted. By step S654, the RD cost per CU divided by BT is calculated.

In step S655, the encoding unit 13 performs the calculation process of the RD cost per CU divided by TT. Since the calculation process of the RD cost per CU divided by TT is basically the same process as the process in fig. 16 described above, a description thereof is omitted. By step S655, the RD cost per CU passing through the TT is calculated.

In step S656, the control unit 211 controls the encoding unit 13 to encode the CTU using the partition structure with the lowest RD cost among the calculated RD costs.

In step S657 of fig. 38, the control section 211 determines whether the CTUs of Cb/Cr have been divided by QT.

In the case where it is determined in step S657 that the CTU has been divided by QT, the process proceeds to step S658.

In step S658, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 if min _ CU _ size _ idc _ inter _ slice _ chroma is 2.

In the case where it is determined in step S658 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S659.

In step S659, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2 if min _ CU _ size _ idc _ inter _ slice _ chroma is 1.

In the case where it is determined in step S659 that the size of each CU of Cb/Cr is not 2 × 2, the processing proceeds to step S660.

In step S660, the encoding unit 13 calculates the RD cost for each CU divided by QT.

Meanwhile, in the case where it is determined in step S657 that the CTUs of Cb/Cr are not divided by QT, or in the case where it is determined in step S658 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S661. Also in the case where it is determined in step S659 that the size of each CU of Cb/Cr after QT division is 2 × 2, the process proceeds to step S661. In these cases, no RD cost is calculated, nor is the partition flag sent.

In step S661, the encoding section 13 performs the RD cost per CU calculation process without division.

In step S662, the control unit 211 determines whether the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma is 2.

In a case where it is determined in step S662 that the size of each CU of Cb/Cr after division is not other than 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S663.

In step S663, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma is 1.

In a case where it is determined in step S663 that the size of each CU of Cb/Cr after division is not 2 × 2, the process proceeds to step S664.

In step S664, the encoding unit 13 performs the calculation processing of the RD cost per CU divided by BT. The RD cost calculation process for each CU divided by BT is basically the same process as the RD cost calculation process divided by BT in fig. 15 described above. By step S664, the RD cost per CU divided by BT is calculated.

Further, in the case where it is determined in step S663 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S665 of fig. 39. Also in the case where it is determined in step S664 that the size of each CU of Cb/Cr after QT division is 2 × 2, the process proceeds to step S665 of fig. 39. In those cases, no RD cost is calculated, nor is a partition flag sent.

In step S665 in fig. 39, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma ═ 2.

In a case where it is determined in step S665 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S666.

In step S666, the control unit 211 determines whether or not the size of each CU of Cb/Cr after division is not 2 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma is 1.

In the case where it is determined in step S666 that the size of each CU of Cb/Cr after division is not 2 × 2, the process proceeds to step S667.

In step S667, the encoding unit 13 performs RD cost calculation processing for each CU divided by TT. The calculation process of the RD cost per CU divided by TT is basically the same process as the calculation process of the RD cost by TT of fig. 16 described above. In step S667, the RD cost per CU divided by TT is calculated.

In contrast, in a case where it is determined in step S665 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S668. Also in the case where it is determined in step S666 that the size of each CU of Cb/Cr after QT division is 2 × 2, the process proceeds to step S668. In these cases, no RD cost is calculated, nor is the partition flag sent.

In step S668, the control unit 211 controls the encoding unit 13 to encode the CTU using the division structure with the lowest RD cost among the calculated RD costs.

It should be noted that, in fig. 37 to 39 described above, steps S651 to S656 are directed to the CU partition structure determination processing of Y. Steps S657 to S668 are directed to CU partition structure determination processing for Cr/Cb.

As described above, the CU of the minimum size is included in and transmitted together with the SPS. Since the minimum-sized CU is not used on the decoding side, the delay of intra prediction can be reduced.

At this time, since it is not necessary to calculate the RD cost of a CU of an unnecessary size in the CU division structure determination process, the processing time can be reduced. In addition, since it is not necessary to transmit a partition flag (CU partition structure information) of a CU of an unnecessary size, it is possible to improve coding efficiency.

(example of configuration of embodiment of decoding apparatus)

Fig. 40 is a block diagram depicting an example of the configuration of an embodiment of a decoding apparatus that decodes a coded stream transmitted from the encoding apparatus 210 of fig. 32 and to which the present disclosure is applied.

The decoding device 410 of fig. 40 includes a control unit 411, a receiving unit 112, an extracting unit 113, and a decoding unit 114.

Among the components shown in fig. 40, the same components as those described above with reference to fig. 18 are denoted by the same reference numerals. Duplicate descriptions are appropriately omitted.

The control section 411 of the decoding device 410 is configured similarly to the control section 111 of fig. 18, and operates as a central processing device of the decoding device 410.

The control section 411 extracts information included in the parameter set. The control section 411 controls the decoding section 114 based on information included in the parameter set. Specifically, the control section 411 outputs information included in the parameter set of fig. 28 and information newly set in the SPS and specifying the minimum size of the CU to the decoding section 114 to perform the CU partition structure determination process.

The receiving unit 112 receives the coded stream transmitted from the encoding device 210 of fig. 32, and outputs the coded stream to the extracting unit 113.

The extraction section 113 extracts a parameter set and encoded data from the encoded stream supplied from the reception section 112. The extraction section 113 outputs the parameter set to the control section 411. The extraction unit 113 outputs the encoded data to the decoding unit 114.

The decoding unit 114 decodes the encoded data supplied from the extracting unit 113 by a method conforming to the VVC method in accordance with an instruction from the control unit 411. At this time, the decoding section 114 executes CU partition structure determination processing according to an instruction of the control section 411, and the CU of the determined partition structure is used by the decoding section 114. The decoding unit 114 outputs an image after decoding.

(operation of decoding apparatus)

Fig. 41 is a flowchart illustrating an image generation process of the decoding apparatus of fig. 40.

In step S701 in fig. 41, the receiving unit 112 of the decoding apparatus 410 receives the coded stream transmitted from the encoding apparatus 210 in fig. 32, and outputs the coded stream to the extracting unit 113.

In step S702, the extraction section 113 extracts encoded data and parameter sets from the encoded stream supplied from the reception section 112. The extraction unit 113 outputs the encoded data to the decoding unit 114. The extraction section 113 outputs the parameter set to the control section 411.

In step S703, the control unit 411 extracts "min _ CU _ size _ idc _ intra _ slice _ chroma" and "min _ CU _ size _ idc _ inter _ slice _ chroma", which are information specifying the minimum size of the CU, from the SPS.

The control section 411 controls the decoding section 114 based on information included in the parameter set and information specifying the minimum size of the CU.

In step S704, the decoding unit 114 decodes the encoded data supplied from the extraction unit 113 in accordance with the instruction of the control unit 411. This decoding process is basically the same process as the process in fig. 21 described above except for the CU division structure determination process (S333), and therefore, the description thereof is omitted.

Fig. 42 is a flowchart illustrating an example of the CU division structure determination process in step S333 of fig. 21.

Note that in the description of fig. 42, reference is made to the tree shown in fig. 43 as appropriate.

Fig. 43 is a diagram depicting an example of a tree used in the CU partition structure determination process.

On the tree of fig. 43, branch F1 through branch F3 are depicted. Flags corresponding to the branch F1 through the branch F3 are included in the bitstream from the encoding side as division flags, and are transmitted together with the bitstream from the encoding side.

In step S711 of fig. 42, the lossless decoding unit 132 of the decoding unit 114 determines whether or not the CTUs have been divided by QT.

In the case where it is determined in step S711 that the CTU has been divided by QT, the process proceeds to step S712.

In step S712, the lossless decoding unit 132 determines whether the size of each CU of Cb/Cr after the division is not 2 × 2, 2 × 4, or 4 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma ═ 2.

In the case where it is determined in step S712 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S713.

In step S713, the lossless decoding unit 132 determines whether or not the size of each CU of Cb/Cr after the division is not 2 × 2 if min _ CU _ size _ idc _ inter _ slice _ chroma is 1.

In the case where it is determined in step S713 that the size of each CU of Cb/Cr after division is not 2 × 2, the processing proceeds to step S714.

In step S714, the lossless decoding unit 132 reads the flag of the branch F1 shown in fig. 43, out of the division flags.

In the case where it is determined in step S713 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S715. In the case where it is determined in step S714 that the size of each CU of Cb/Cr after QT division is 2 × 2, the process proceeds to step S715.

In step S715, the lossless decoding unit 132 does not read the flag of the branch F1, and interprets the flag as 0. Thereafter, the process proceeds to step S716.

In contrast, in the case where it is determined in step S711 that the CTU has not been divided by QT, the process proceeds to step S716.

In step S716, the lossless decoding unit 132 determines whether or not the flag of branch F1 is 0.

In the case where it is determined in step S716 that the flag of branch F1 is 0, the processing proceeds to step S717.

In step S717, the lossless decoding unit 132 determines whether the size of each CU of Cb/Cr after the division is not 2 × 2, 2 × 4, or 4 × 2 when min _ CU _ size _ idc _ inter _ slice _ chroma ═ 2.

In the case where it is determined in step S717 that the size of each CU of Cb/Cr after division is not 2 × 2, 2 × 4, or 4 × 2, the processing proceeds to step S718.

In step S718, the lossless decoding unit 132 determines whether or not the size of each CU of Cb/Cr after the division is not 2 × 2 when min _ CU _ size _ idc _ intra _ slice _ chroma is 1.

In the case where it is determined in step S718 that the size of each CU of Cb/Cr after division is not 2 × 2, the processing proceeds to step S719.

In step S719, the lossless decoding unit 132 reads the flag of branch F2 shown in fig. 43 out of the division flags. After that, the process proceeds to step S721.

In contrast, when it is determined in step S717 that the size of each CU of Cb/Cr after QT division is 2 × 2, 2 × 4, or 4 × 2, the process proceeds to step S720. Also in the case where it is determined in step S718 that the size of each CU of Cb/Cr after QT division is 2 × 2, the processing proceeds to step S720.

In step S720, the lossless decoding unit 132 does not read the flag of branch F2, and interprets the flag as 0. Thereafter, the process proceeds to step S721.

In step S721, the lossless decoding unit 132 determines whether or not the flag of branch F2 is 1.

In the case where it is determined in step S721 that the flag of branch F2 is 1, the processing proceeds to step S722.

In step S722, the lossless decoding unit 132 reads out the flag of the branch F3 shown in fig. 43 from within the CU partition structure information, and the CU partition structure determination process ends.

In the case where it is determined in step S716 that the flag of the branch F1 is 1 or in the case where it is determined in step S721 that the flag of the branch F2 is 0, the CU partition structure determination process ends.

As described above, since the CU of the minimum size is included in and transmitted together with the SPS, and the CU of the minimum size is not used on the decoding side, the delay of the intra prediction can be reduced.

It is to be noted that although fig. 42 is directed to the case of I slice, the processing in the case of P/B slice differs only in the type of slice, and similar processing to that in the case of I slice is performed, and therefore description thereof is omitted.

Further, although in the foregoing description, the minimum-sized CU is included in the SPS, the minimum-sized CU may be included in some other syntax such as the PPS.

<4. fourth embodiment (determination of minimum CU by criterion) >

In the fourth embodiment, a CU of the minimum size is not added to the high level syntax, and a CU of 2 × 2size cannot be used for Cr/Cb according to the standard.

Specifically, it is determined by a standard so that min _ cu _ size _ idc _ intra _ slice _ chroma is used — 2, and decoding is performed according to the standard.

It is to be noted that the method of determining the minimum size of the CU is not restrictive, and any other design may be applied as long as Cb/Cr of 2 × 2size cannot be used.

By determining the minimum-sized CU by the criteria as described above, a large delay that may occur in the case of Cb/Cr of 2 × 2 can be avoided.

In the foregoing description, a VVC-compliant method is used as an encoding method. However, the present technology is not limited thereto, and other encoding/decoding methods may be applied.

It should be noted that although the above-described third embodiment is also directed to an example in which not only 2 × 2 but also 2 × 4 or 4 × 2 is excluded, in the first, second, and fourth embodiments, not only 2 × 2 but also 2 × 4 or 4 × 2 may be excluded.

In addition, although the case of the 4:2:0 chroma format is described in the foregoing description, it can be applied to other chroma formats.

It is to be noted that the present disclosure can be applied to an image encoding apparatus and an image decoding apparatus used when image information (bit stream) compressed by orthogonal transformation such as discrete cosine transformation and motion compensation is received through a network medium such as satellite broadcasting, cable television, the internet, or mobile phones (e.g., VVC method). Also, the present disclosure may be applied to an image encoding apparatus and an image decoding apparatus used when processing image information on a storage medium such as an optical disk or a magnetic disk or a flash memory.

<5 > fifth embodiment

(description of computer to which the present disclosure applies)

Although the series of processes described above may be executed by hardware, it may be executed by software in other ways. In the case where a series of processes is executed by software, a program configuring the software is installed into a computer. The computer herein includes a computer incorporated in dedicated hardware and a general-purpose personal computer that performs various types of functions by, for example, installing various types of programs into a personal computer or the like.

Fig. 44 is a block diagram showing an example of a hardware configuration of a computer that executes a series of processing described hereinabove according to a program.

In the computer, a CPU (central processing unit) 501, a ROM (read only memory) 502, and a RAM (random access memory) 503 are connected to each other by a bus 504.

Further, an input/output interface 505 is connected to the bus 504. The input section 506, the output section 507, the storage section 508, the communication section 509, and the drive 510 are connected to the input/output interface 505.

The input unit 506 includes, for example, a keyboard, a mouse, a microphone, and the like. The output unit 507 includes a display, a speaker, and the like. The storage unit 508 includes, for example, a hard disk, a nonvolatile memory, and the like. The communication unit 509 includes a network interface and the like. The drive 510 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured in the manner as described above, the CPU 501 loads a program stored in the storage section 508, for example, into the RAM 503 through the input/output interface 505 and the bus 504, and executes the program to realize the series of processing described above.

A program to be executed by a computer (CPU 501) may be recorded on a removable medium 511, for example, as a package medium or the like, and provided as the removable recording medium 511. Further, the program may be provided through a wired or wireless transmission medium such as a local area network, the internet, or digital satellite broadcasting.

In the computer, the program can be installed into the storage section 508 via the input/output interface 505 by installing the removable medium 511 on the drive 510. Further, the program may be received by the communication section 509 via a wired or wireless transmission medium and installed into the storage section 508. Further, the program may be installed in advance in the ROM 502 or the storage section 508.

Note that the program to be executed by the computer may be a program that performs processing in time series in the order as described in this specification, or may be a program that performs processing in parallel or at a necessary time (for example, at the time of calling the processing).

It is to be noted that, in this specification, an example in which various types of information are multiplexed in encoded data and transmitted from an encoding side to a decoding side is described. However, the technique for transmitting such information is not limited to this example. For example, such information may be transmitted or recorded as separate data associated with the encoded data without being multiplexed with the encoded data. Here, the term "associated" means that an image (which may be a part of an image such as a slice or a block) included in a bitstream and information corresponding to the image can be linked when decoding. In particular, the information may be transmitted on a different transmission line than the transmission line encoding the data. Further, the information may be recorded on a recording medium different from the recording medium used to encode the data (or another recording area of the same recording medium). Furthermore, the information and the encoded data may be associated with each other in freely selected units (e.g. a plurality of frames, one frame or a part of a frame).

Further, in the present specification, the term system is used to denote a set of a plurality of components (a device, a module (a component), and the like), and it does not matter whether all the components are accommodated in the same housing. Therefore, a plurality of devices accommodated in separate housings and connected to each other through a network is a system, and one device in which a plurality of modules are accommodated in a single housing is also a system.

The advantageous effects described in this specification are only illustrative and not restrictive. Other benefits may also exist.

The embodiments of the present disclosure are not limited to the embodiments described above, but various changes may be made without departing from the subject matter of the present disclosure.

It is to be noted that the present disclosure may be applied to an image encoding apparatus and an image decoding apparatus used when an encoded stream is received through a network medium such as satellite broadcasting, cable television, the internet, or a mobile phone, or when the encoded stream is processed on a storage medium such as an optical disk, a magnetic disk, or a flash memory.

Further, the present disclosure may employ a configuration of cloud computing by which one function is shared and co-processed by a plurality of devices through a network.

Further, each step of the above-described flowcharts may be executed by a single apparatus, or may be shared and executed by a plurality of apparatuses.

Further, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one apparatus or may be shared and executed by a plurality of apparatuses.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is obvious that a person having common general knowledge in the technical field to which the present disclosure relates can make various changes or modifications within the scope of the technical idea described in the claims, and it should be recognized that such changes or modifications naturally fall within the technical scope of the present disclosure.

Note that the present technology can also adopt the following configuration.

(1) An image processing apparatus comprising:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for the chrominance, a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of the chrominance; and

an encoding section configured to encode an image with the CU partition structure of the luminance and the CU partition structure of the chrominance to generate an encoded stream.

(2) The image processing apparatus according to the above (1),

the size of the CU partition structure, which is the smallest size in at least one of the vertical direction and the horizontal direction, is 2 × 2, 4 × 2, or 2 × 4.

(3) The image processing apparatus according to the above (1) or (2), further comprising:

a cost calculation unit configured to calculate a cost of the CU partition structure for the luminance and a cost of the CU partition structure for the chrominance, respectively,

for the luminance, the control portion determines a CU division structure of the luminance based on a cost of the CU division structure for the luminance, and

for the chroma, the control portion suppresses calculation of a CU partition structure cost for the chroma for a minimum CU partition structure, and determines the CU partition structure for the chroma based on the cost of the CU partition structure for the chroma.

(4) The image processing apparatus according to any one of the above (1) to (3), further comprising:

a setting section configured to set a syntax describing the minimum CU partition structure, wherein,

the control portion determines, for the chroma, a CU partition structure of the chroma by excluding the minimum CU partition structure based on the syntax.

(5) The image processing apparatus according to the above (4), further comprising:

a transmitting section configured to transmit the syntax and the coded stream.

(6) The image processing apparatus according to the above (4), wherein,

the syntax includes SPS.

(7) The image processing apparatus according to any one of the above (1) to (3),

the minimum CU partition structure is defined by a standard.

(8) An image processing method performed by an image processing apparatus, comprising:

excluding, for the chroma, a CU partition structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine a CU partition structure for the chroma, when the CU partition structure for the luminance and the CU partition structure for the chroma are to be determined, respectively; and

and encoding an image by using the luminance CU partition structure and the chrominance CU partition structure to generate an encoding stream.

(9) A program that causes a computer to function as:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude, for the chrominance, a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction to determine the CU division structure of the chrominance; and

an encoding section configured to encode an image with the CU partition structure of the luminance and the CU partition structure of the chrominance to generate an encoded stream.

(10) An image processing apparatus comprising:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the CU division structure of chrominance, and determine the CU division structure of luminance and the CU division structure of chrominance based on an encoding stream generated by encoding an image using the CU division structure of luminance and the CU division structure of chrominance; and

a decoding section configured to decode the encoded stream based on the CU partition structure of the luminance and the CU partition structure of the chrominance to generate the image.

(11) An image processing method performed by an image processing apparatus, comprising:

when a luminance CU division structure and a chrominance CU division structure are to be determined, respectively, excluding a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the chrominance CU division structure, and determining the luminance CU division structure and the chrominance CU division structure based on an encoding stream generated by encoding an image using the luminance CU division structure and the chrominance CU division structure; and

decoding the encoded stream based on the luma CU partition structure and the chroma CU partition structure to generate the image.

(12) A program that causes a computer to function as:

a control section configured to, when a CU division structure of luminance and a CU division structure of chrominance are to be determined, respectively, exclude a CU division structure that is the smallest size in at least one of a vertical direction and a horizontal direction for the chrominance to determine the CU division structure of chrominance, and determine the CU division structure of luminance and the CU division structure of chrominance based on an encoding stream generated by encoding an image using the CU division structure of luminance and the CU division structure of chrominance; and

a decoding section configured to decode the encoded stream based on the CU partition structure of the luminance and the CU partition structure of the chrominance to generate the image.

[ list of reference numerals ]

10: encoding device

11: control unit

12: setting part

13: encoding unit

14: transmitting part

46: intra prediction unit

47: motion prediction and compensation unit

36: encoding unit

110: decoding device

111: control unit

112: receiving part

113: extraction section

114: decoding unit

132: lossless decoding unit

143: intra prediction unit

144: motion compensation unit

210: encoding device

211: control unit

212: setting part

410: decoding device

411: control unit