Loop filtering implementation method, device and computer storage medium

文档序号：196265 发布日期：2021-11-02 浏览：34次中文

阅读说明：本技术 环路滤波实现方法、装置及计算机存储介质 (Loop filtering implementation method, device and computer storage medium ) 是由万帅王铭泽马彦卓霍俊彦杨付正于 2019-03-07 设计创作，主要内容包括：本申请实施例公开了一种环路滤波实现方法、装置及计算机存储介质,该方法包括：获取待滤波图像；其中,所述待滤波图像是待编码视频中的原始图像在视频编码过程中所生成的,所述待编码视频包括原始图像帧,所述原始图像帧包括所述原始图像；确定所述待滤波图像的融合信息；其中,所述融合信息是所述待滤波图像的至少两个图像分量以及对应的辅助信息融合得到的；基于所述融合信息对所述待滤波图像进行环路滤波处理,得到所述待滤波图像滤波后的至少一个图像分量。(The embodiment of the application discloses a loop filtering implementation method, a loop filtering implementation device and a computer storage medium, wherein the method comprises the following steps: acquiring an image to be filtered; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image; determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information; and performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered.)

1. A loop filtering implementation method comprises the following steps:

acquiring an image to be filtered; the image to be filtered is generated in the process of encoding an original image in a video to be encoded;

determining fusion information of the image to be filtered; wherein the fusion information comprises at least two image components of the image to be filtered;

and acquiring loop filtering output of the image to be filtered based on the fusion information to obtain at least one image component in the loop filtering output of the image to be filtered.

2. The method of claim 1, wherein the acquiring an image to be filtered comprises:

and carrying out video coding processing on an original image in a video to be coded, and taking the generated reconstructed image as the image to be filtered.

3. The method according to claim 2, wherein, the original image in the video to be coded is processed by video coding to generate a reconstructed image; and carrying out preset filtering processing on the reconstructed image, and taking the image subjected to preset filtering as the image to be filtered.

4. The method of claim 1, wherein prior to the determining fusion information for the image to be filtered, the method further comprises:

determining auxiliary information corresponding to the image to be filtered; wherein the side information at least includes block division information, the block division information including a block size and a block position.

5. The method according to claim 4, wherein the determining the auxiliary information corresponding to the image to be filtered comprises:

and carrying out coding unit CU partition on an original image in a video to be coded to obtain CU partition information, and taking the CU partition information as block partition information corresponding to the image to be filtered.

6. The method according to claim 5, wherein the regarding the CU partition information as the block partition information corresponding to the image to be filtered comprises:

filling a first value in each pixel point position corresponding to a CU boundary and filling a second value in other pixel point positions to obtain a first matrix corresponding to the CU partition information; wherein the first value is different from the second value;

and taking the first matrix as block division information corresponding to the image to be filtered.

7. The method according to any one of claims 4 to 6, wherein the determining auxiliary information corresponding to the image to be filtered includes:

and acquiring a quantization parameter corresponding to an original image in a video to be coded, and taking the quantization parameter as quantization parameter information corresponding to the image to be filtered.

8. The method according to claim 7, wherein the using the quantization parameter as quantization parameter information corresponding to the image to be filtered comprises:

establishing a second matrix with the same size as the original image; filling the normalized value of the quantization parameter corresponding to the original image in each pixel point position in the second matrix;

and taking the second matrix as the quantization parameter information corresponding to the image to be filtered.

9. The method according to any one of claims 1 to 8, wherein the determining of the fusion information of the image to be filtered includes:

and carrying out fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

10. The method according to any one of claims 1 to 9, wherein prior to said determining fusion information of the image to be filtered, the method further comprises:

selecting image components with low resolution according to at least two image components of the image to be filtered;

and performing up-sampling processing on the low-resolution image component.

11. A loop filter implementation apparatus includes: an acquisition unit, a determination unit and a filtering unit, wherein,

the acquisition unit is configured to acquire an image to be filtered; the image to be filtered is generated in the process of encoding an original image in a video to be encoded;

the determining unit is configured to determine fusion information of the image to be filtered; wherein the fusion information comprises at least two image components of the image to be filtered;

the filtering unit is configured to obtain a loop filtering output of the image to be filtered based on the fusion information, and obtain at least one image component in the loop filtering output of the image to be filtered.

12. The in-loop filtering implementation apparatus according to claim 11, wherein the obtaining unit is specifically configured to perform video coding processing on an original image in a video to be coded, and use the generated reconstructed image as the image to be filtered.

13. The in-loop filtering implementation apparatus according to claim 12, wherein the obtaining unit is specifically configured to perform video coding processing on an original image in a video to be coded, so as to generate a reconstructed image; and carrying out preset filtering processing on the reconstructed image, and taking the image subjected to preset filtering as the image to be filtered.

14. The in-loop filtering implementation apparatus according to claim 11, wherein the determining unit is further configured to determine auxiliary information corresponding to the image to be filtered; wherein the side information at least includes block division information, the block division information including a block size and a block position.

15. The apparatus according to claim 14, wherein the apparatus further comprises a dividing unit configured to perform CU division on an original image in the video to be encoded, so as to obtain CU division information, and use the CU division information as block division information corresponding to the image to be filtered.

16. The apparatus according to claim 15, wherein the determining unit is specifically configured to, for the CU partition information, fill a first value in each pixel position corresponding to a CU boundary, and fill a second value in other pixel positions, so as to obtain a first matrix corresponding to the CU partition information; wherein the first value is different from the second value; and taking the first matrix as the block division information corresponding to the image to be filtered.

17. The in-loop filtering implementation apparatus according to any one of claims 14 to 16, wherein the obtaining unit is further configured to obtain a quantization parameter corresponding to an original image in the video to be encoded, and use the quantization parameter as quantization parameter information corresponding to the image to be filtered.

18. The in-loop filtering implementation apparatus of claim 17, wherein the determining unit is specifically configured to establish a second matrix having the same size as the original image; filling the normalized value of the quantization parameter corresponding to the original image in each pixel point position in the second matrix; and taking the second matrix as quantization parameter information corresponding to the image to be filtered.

19. The in-loop filtering implementation apparatus according to any one of claims 11 to 18, wherein the in-loop filtering implementation apparatus further includes a fusion unit configured to perform fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

20. The in-loop filtering implementation apparatus according to any one of claims 11 to 19, wherein the in-loop filtering implementation apparatus further includes a sampling unit configured to select, for at least two image components of the image to be filtered, an image component with a low resolution; and upsampling the low-resolution image component.

21. A loop filtering implementation method comprises the following steps:

acquiring an image to be filtered; the image to be filtered is generated in the process of encoding an original image in a video to be encoded;

determining fusion information of the image to be filtered; wherein the fusion information comprises at least two image components of the image to be filtered;

and acquiring loop filtering output of the image to be filtered based on the fusion information to obtain at least one image component in the loop filtering output of the image to be filtered.

22. The method of claim 21, wherein the acquiring an image to be filtered comprises:

and taking a reconstructed image generated after the original image in the video to be coded is coded as the image to be filtered.

23. The method according to claim 22, wherein the reconstructed image is subjected to a preset filtering process, and the image after the preset filtering process is taken as the image to be filtered.

24. The method of claim 21, wherein prior to the determining fusion information for the image to be filtered, the method further comprises:

25. The method of claim 24, wherein the determining the auxiliary information corresponding to the image to be filtered comprises:

and taking CU division information obtained after the CU division of an encoding unit is carried out on the original image in the video to be encoded as block division information corresponding to the image to be filtered.

26. The method according to claim 25, wherein the regarding the CU partition information as block partition information corresponding to the image to be filtered comprises:

and taking the first matrix as block division information corresponding to the image to be filtered.

27. The method according to claim 26, wherein the quantization parameter information corresponding to the image to be filtered is a second matrix with the same size as the original image; and filling the normalized value of the quantization parameter corresponding to the original image in each pixel point position in the second matrix.

28. The method according to any one of claims 21 to 27, wherein the determining of the fusion information of the image to be filtered comprises:

and carrying out fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

29. The method according to any one of claims 21 to 28, wherein prior to said determining fusion information of the image to be filtered, the method further comprises:

selecting image components with low resolution according to at least two image components of the image to be filtered;

and performing up-sampling processing on the low-resolution image component.

30. A loop filter implementation apparatus includes: an acquisition unit and a determination unit, wherein,

the acquisition unit is configured to acquire an image to be filtered; the image to be filtered is generated in the process of encoding an original image in a video to be encoded;

the determining unit is configured to determine fusion information of the image to be filtered; wherein the fusion information comprises at least two image components of the image to be filtered;

the obtaining unit is further configured to obtain a loop filter output of the image to be filtered based on the fusion information, and obtain at least one image component in the loop filter output of the image to be filtered.

31. The in-loop filtering implementation apparatus according to claim 30, wherein the determining unit is specifically configured to use a reconstructed image generated after encoding an original image in the video to be encoded as the image to be filtered.

32. The apparatus for implementing loop filtering according to claim 31, wherein the determining unit is specifically configured to perform a preset filtering process on the reconstructed image, and use a preset filtered image as the image to be filtered.

33. The in-loop filtering implementation apparatus of claim 30, wherein the determining unit is further configured to determine auxiliary information corresponding to the image to be filtered; wherein the side information at least includes block division information, the block division information including a block size and a block position.

34. The in-loop filtering implementation apparatus of claim 33, wherein the determining unit is further configured to perform Coding Unit (CU) partition on an original image in the video to be coded to obtain CU partition information as the block partition information corresponding to the image to be filtered.

35. The apparatus according to claim 34, wherein the determining unit is specifically configured to, for the CU partition information, fill a first value in each pixel position corresponding to a CU boundary, and fill a second value in other pixel positions, so as to obtain a first matrix corresponding to the CU partition information; wherein the first value is different from the second value; and taking the first matrix as the block division information corresponding to the image to be filtered.

36. The in-loop filtering implementation apparatus of claim 35, wherein the quantization parameter information corresponding to the image to be filtered is a second matrix having the same size as the original image; and filling the normalized value of the quantization parameter corresponding to the original image in each pixel point position in the second matrix.

37. The in-loop filtering implementation apparatus according to any one of claims 30 to 36, wherein the in-loop filtering implementation apparatus further includes a fusion unit configured to perform fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

38. The in-loop filtering implementation apparatus according to any one of claims 30 to 37, wherein the in-loop filtering implementation apparatus further includes a sampling unit configured to select, for at least two image components of the image to be filtered, an image component with a low resolution; and upsampling the low-resolution image component.

39. A loop filter implementation device, wherein the loop filter implementation device comprises: a memory and a processor;

the memory for storing a computer program operable on the processor;

the processor, when executing the computer program, is adapted to perform the steps of the method according to any of claims 1-10 or to perform the steps of the method according to any of claims 21-29.

40. A computer storage medium, wherein the computer storage medium stores a loop filter implementation program that when executed by at least one processor implements the steps of the method of any one of claims 1-10 or the steps of the method of any one of claims 21-29.

Technical Field

The embodiment of the application relates to the technical field of image processing, in particular to a loop filtering implementation method and device and a computer storage medium.

Background

In video Coding systems, most video Coding uses a hybrid Coding framework based on block-shaped Coding Units (CUs), since adjacent CUs use different Coding parameters, such as: different transformation processes, different Quantization Parameters (QPs), different prediction modes, different reference image frames and the like, and the sizes of errors introduced by the CUs and the distribution characteristics thereof are mutually independent, and the discontinuity of the boundary of the adjacent CUs generates a block effect, thereby influencing the subjective and objective quality of the reconstructed image and even influencing the prediction accuracy of the subsequent encoding and decoding.

Thus, during the encoding and decoding process, the loop filter is used to improve the subjective and objective quality of the reconstructed image. The traditional loop filter usually artificially reduces the characteristics of distorted images, artificially designs the structure of the filter and configures the coefficients of the filter, such as block-removing filtering, sample adaptive compensation, adaptive loop filtering, and the like, the filter depending on the artificial design has poor fitting to the optimal filter, the adaptive capacity and the filtering effect are poor, and the encoding end needs to write filter related parameters depending on local statistical information into a code stream to ensure the consistency of the encoding and decoding end, which increases the number of encoding bits.

With the rapid development of the deep learning theory, a Convolutional Neural Network (CNN) is proposed in the industry to perform filtering processing on a reconstructed image to remove image distortion, so that the objective quality is obviously improved compared with that of a traditional loop filter. However, the current CNN filter does not fully utilize the related information, so that the improvement of subjective and objective quality of the reconstructed image is limited.

Disclosure of Invention

The embodiment of the application provides a loop filtering implementation method, a loop filtering implementation device and a computer storage medium, wherein coding parameters such as block division information and/or QP information are used as auxiliary information to be fused with a plurality of input image components, so that not only is the relation among the image components fully utilized, but also the computational complexity is reduced, and the coding code rate is saved; and meanwhile, the subjective and objective quality of the video reconstruction image in the encoding and decoding process is further improved.

The technical scheme of the embodiment of the application can be realized as follows:

in a first aspect, an embodiment of the present application provides a method for implementing loop filtering, where the method includes:

acquiring an image to be filtered; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image;

determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information;

and performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered.

In a second aspect, an embodiment of the present application provides a loop filter implementation apparatus, where the loop filter implementation apparatus includes: an acquisition unit, a determination unit and a filtering unit, wherein,

the acquisition unit is configured to acquire an image to be filtered; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image;

the determining unit is configured to determine fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information;

the filtering unit is configured to perform loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered.

In a third aspect, an embodiment of the present application provides a loop filter implementation apparatus, where the loop filter implementation apparatus includes: a memory and a processor, wherein,

the memory for storing a computer program operable on the processor;

the processor is adapted to perform the steps of the method according to the first aspect when running the computer program.

In a fourth aspect, embodiments of the present application provide a computer storage medium storing a loop filter implementation program, which when executed by at least one processor implements the steps of the method according to the first aspect.

The embodiment of the application provides a loop filtering implementation method, a loop filtering implementation device and a computer storage medium, wherein an image to be filtered is obtained firstly, and the image to be filtered is generated in the video coding process of an original image in a video to be coded; then determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information; finally, performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered; therefore, the coding parameters such as the block division information and/or the QP information are used as auxiliary information to be fused with the input multiple image components, so that the relation among the multiple image components is fully utilized, the problem that the multiple image components need to be subjected to multiple complete network forward calculations is effectively solved, the calculation complexity is further reduced, and the coding code rate is saved; in addition, auxiliary information such as block division information and/or QP information is fused, filtering can be further assisted, and the subjective and objective quality of video reconstruction images in the encoding and decoding process is improved.

Drawings

Fig. 1 is a schematic diagram of a composition structure of a conventional encoding block diagram provided in a related art scheme;

fig. 2 is a schematic structural diagram of an improved encoding block diagram provided in an embodiment of the present application;

fig. 3 is a schematic flowchart of a loop filtering implementation method according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a block partition matrix according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a conventional CNN filter according to an embodiment of the present disclosure;

fig. 6A and 6B are schematic structural diagrams of another conventional CNN filter according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a loop filter frame according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of another loop filter framework provided in the embodiment of the present application;

fig. 9 is a schematic structural diagram of a loop filter implementation apparatus according to an embodiment of the present disclosure;

fig. 10 is a schematic diagram of a specific hardware structure of a loop filter implementation apparatus according to an embodiment of the present application.

Detailed Description

So that the manner in which the features and elements of the present embodiments can be understood in detail, a more particular description of the embodiments, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings.

In a video codec system, a video to be encoded includes an original image frame including an original image, and the original image is subjected to various processes such as prediction, transformation, quantization, reconstruction, filtering, and the like, during which a pixel value shift of a processed video image with respect to the original image may have occurred, resulting in visual impairment or artifacts. In addition, under the mixed coding framework based on block-shaped CUs adopted by most video coding and decoding systems, since the adjacent coding blocks adopt different coding parameters (such as different transformation processes, different QPs, different prediction modes, different reference image frames, and the like), the sizes of errors introduced by the respective coding blocks and the distribution characteristics thereof are independent from each other, and the discontinuity of the boundaries of the adjacent coding blocks generates a blocking effect. These distortions affect not only the subjective and objective quality of the reconstructed image, but also the prediction accuracy of the subsequent encoding and decoding if the reconstructed image is used as a reference image of the subsequent encoded pixels, thereby affecting the size of bits in the video code stream. Therefore, In a video coding and decoding system, an In-Loop Filter (In-Loop Filter) is often added to improve the subjective and objective quality of a reconstructed image.

Referring to fig. 1, a schematic diagram of a conventional encoding block diagram 10 according to a related art is shown. As shown in fig. 1, the conventional coding block diagram 10 may include components such as a transform and quantization unit 101, an inverse transform and inverse quantization unit 102, a prediction unit 103, a filtering unit 104, and an entropy coding unit 105; among them, the prediction unit 103 further includes an intra prediction unit 1031 and an inter prediction unit 1032. For an input original image, a Coding Tree Unit (CTU) may be obtained through preliminary partitioning, and a CTU is continuously subjected to content adaptive partitioning, so that a CU may be obtained, where the CU generally includes one or more Coding Blocks (CBs). Performing intra prediction of the intra prediction unit 1031 or inter prediction of the inter prediction unit 1032 on the coding block, so as to obtain residual information; transforming the residual information into the coding block by a transforming and quantizing unit 101, including transforming the residual information from a pixel domain into a transform domain, and quantizing the obtained transform coefficients to further reduce the bit rate; after determining the prediction mode, the prediction unit 103 is further configured to provide the selected intra-prediction data or inter-prediction data to the entropy coding unit 105; furthermore, the inverse transform and inverse quantization unit 102 is used for reconstruction of the encoded block, reconstructing a residual block in the pixel domain, which removes blocking artifacts by the filtering unit 104, and then adding the reconstructed residual block to the decoded image buffer unit for generating a reconstructed reference image; the entropy Coding unit 105 is used to code various Coding parameters and quantized transform coefficients, for example, the entropy Coding unit 105 adopts header information Coding and Context-based Adaptive Binary arithmetic Coding (CABAC) algorithm, and may be used to code Coding information indicating the determined prediction mode and output a corresponding code stream.

For the conventional encoding block diagram 10 shown In fig. 1, the filtering unit 104 is a Loop Filter, also called an In-Loop Filter (In-Loop Filter), and may include a De-Blocking Filter (DBF) 1041, a Sample Adaptive Offset (SAO) Filter 1042, an Adaptive Loop Filter (ALF) 1043, and the like. The deblocking filter 1041 is configured to implement deblocking filtering, and in next generation Video Coding standard h.266/multifunctional Video Coding (VVC), for all coded block boundaries in an original image, first, according to Coding parameters on both sides of the boundary, and according to a block boundary texture value obtained by calculation, it is determined whether to perform a deblocking filtering decision, and finally, according to the boundary strength and the filtering decision, pixel information on both sides of the coded block boundary is modified. In VVC, after deblocking filtering is performed, in order to reduce quantization distortion of high-frequency ac coefficients, an SAO technique, i.e., a sample adaptive compensation filter 1042, is also introduced; further, starting from the pixel domain, a negative value is added to the pixel at the peak and a positive value is added to the pixel at the trough for compensation processing. In the VVC, after the deblocking filtering and the sample adaptive compensation filtering are performed, a further filtering process using an adaptive loop filter 1043 is required; for adaptive loop filtering, an optimal filter in the mean square sense is calculated according to the pixel values of the original image and the pixel values of the distorted image. However, these filters (such as deblocking filter 1041, sample adaptive compensation filter 1042, adaptive loop filter 1043, etc.) not only require fine manual design, extensive decision making; moreover, at the encoding end, filter related parameters (such as a filter coefficient, a Flag value of whether the filter is selected or not, and the like) depending on local statistical information need to be written into the code stream so as to ensure the consistency of the encoding end and the decoding end, and the encoding bit number is increased; meanwhile, the manually designed filter has low fitting degree to a complex function of a real optimization target, and the filtering effect needs to be enhanced.

The embodiment of the application provides a loop filtering implementation method, which is applied to an improved coding block diagram; compared with the conventional encoding block diagram 10 shown in fig. 1, the main difference is that a modified loop filter is used to replace the deblocking filter 1041, the sample adaptive compensation filter 1042, and the adaptive loop filter 1043 in the related art. In this embodiment of the present application, the improved loop filter may be a Convolutional Neural Network (CNN) filter, or may be a filter established by other deep learning, and this embodiment of the present application is not limited specifically.

Taking a convolutional neural network filter as an example, refer to fig. 2, which shows a schematic structural diagram of an improved coding block diagram 20 provided by an embodiment of the present application. As shown in fig. 2, compared to the conventional encoding block diagram 10, the filtering unit 104 in the modified encoding block diagram 20 includes a convolutional neural network filter 201. The convolutional neural network filter 201 may be used in place of all of the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in fig. 1, may also be used in place of any one or two of the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in fig. 1, or may even be used in combination with any one or more of the deblocking filter 1041, the sample adaptive compensation filter 1042 and the adaptive loop filter 1043 in fig. 1. It should be further noted that, for each component shown in fig. 1 or fig. 2, such as the transformation and quantization unit 101, the inverse transformation and inverse quantization unit 102, the prediction unit 103, the filtering unit 104, the entropy coding unit 105, or the convolutional neural network filter 201, these components may be virtual modules or hardware modules. In addition, those skilled in the art will appreciate that these elements do not constitute limitations on the encoded block diagrams, which may include more or fewer components than illustrated, or some components in combination, or a different arrangement of components.

In the embodiment of the application, the convolutional neural network filter 201 can be directly deployed at the encoding end and the decoding end after filter network training, so that no filter related parameters need to be transmitted; the convolutional neural network filter 201 may also perform fusion processing on auxiliary information such as block division information and/or QP information with the input plurality of image components; therefore, the method not only makes full use of the relationship among a plurality of image components, but also reduces the calculation complexity and saves the coding rate; and meanwhile, the subjective and objective quality of the video reconstruction image in the encoding and decoding process is further improved.

It should be noted that the loop filtering implementation method in the embodiments of the present application may be applied to not only an encoding system but also a decoding system. Generally, in order to save the coding rate and ensure that the decoding system can perform correct decoding processing, the loop filter of the embodiment of the present application must be deployed synchronously in the coding system and the decoding system. The following will describe in detail an application in an encoding system as an example.

Referring to fig. 3, a schematic flowchart of a loop filtering implementation method provided in an embodiment of the present application is shown, where the method may include:

s301: acquiring an image to be filtered; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image;

s302: determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information;

s303: and performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered.

It should be noted that, the original image may be divided into CTUs, or divided into CUs by CTUs; that is, the block partition information in the embodiment of the present application may refer to CTU partition information or CU partition information; in this way, the loop filtering implementation method in the embodiment of the present application may be applied to not only the loop filtering at the CU level but also the loop filtering at the CTU level, and the embodiment of the present application is not particularly limited.

In the embodiment of the application, the image to be filtered is obtained; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image; determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information; therefore, the relation among a plurality of image components is fully utilized, and the problem that the plurality of image components need to be subjected to multiple times of complete network forward calculation can be avoided, so that the calculation complexity is reduced, and the coding code rate is saved; finally, performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered; auxiliary information such as block division information and/or QP information is blended into the fusion information, so that the filtering can be further assisted, and the subjective and objective quality of video reconstruction images in the encoding and decoding process is improved.

In some embodiments, the image components include a first image component, a second image component, and a third image component; wherein the first image component represents a luminance component, the second image component represents a first chrominance component, and the third image component represents a second chrominance component.

It should be noted that, in a video image, a first image component, a second image component, and a third image component are generally adopted to characterize an original image or an image to be filtered. Wherein, under the bright-chroma component representation method, the three image components are respectively a luminance component, a blue chroma (color difference) component and a red chroma (color difference) component; specifically, the luminance component is typically represented using the symbol Y, the blue chrominance component is typically represented using the symbol Cb, and may also be represented by U; the red chrominance component is usually denoted by the symbol Cr, but may also be denoted by V. In the embodiment of the present application, the first image component may be a luminance component Y, the second image component may be a blue chrominance component U, and the third image component may be a red chrominance component V, but the embodiment of the present application is not particularly limited. At least one image component represents one or more of a first image component, a second image component and a third image component, and at least two image components may be the first image component, the second image component and the third image component, and may also be the first image component and the second image component, and may even be the second image component and the third image component, which are not particularly limited in the embodiments of the present application.

In the next generation of the video coding standard VVC, the corresponding Test Model is a VVC Test Model (VTM). When testing is performed on VTM, the current standard test sequence adopts YUV 4:2:0 format, and each frame of image in the video to be encoded in the format can be composed of three image components: one luminance component (denoted by Y) and two chrominance components (denoted by U and V). Assuming that the height of an original image in the video to be encoded is H and the width is W, the size information corresponding to the first image component is H × W, and the size information corresponding to the second image component or the third image component is H × WIt should be noted that, the YUV format 4:2:0 will be described as an example in the embodiment of the present application, but the loop filtering implementation method in the embodiment of the present application is also applicable to other sampling formats.

Taking YUV as a 4:2:0 format as an example, since the size information of the first image component is different from that of the second image component or the third image component, in order to input the first image component and/or the second image component and/or the third image component to the improved loop filter at one time, sampling or recombination processing needs to be performed on the three image components at this time, so that the spatial size information of the three image components is the same.

In some embodiments, a pixel rearrangement process (which may also be referred to as a downsampling process) may be performed on the high-resolution image components so that spatial size information of the three image components is the same. Specifically, before the fusing the at least two image components of the image to be filtered and the auxiliary information, the method further includes:

selecting high-resolution image components aiming at least two image components of the image to be filtered;

and performing pixel rearrangement processing on the high-resolution image component.

It should be noted that, if the first image component is a luminance component, the second image component is a first chrominance component, and the third image component is a second chrominance component; the high resolution image component is the first image component, and the first image component needs to be subjected to the pixel rearrangement process. Exemplarily, taking an original image with a size of 2 × 2 as an example, the original image is converted into 4 channels, that is, 2 × 2 × 1 tensors are arranged into 1 × 1 × 4 tensors; then when the size information of the first image component of the original image is H × W, it can be converted into a pixel rearrangement process before loop filtering is performedIn the form of (a); because the size information of the second image component and the third image component are bothThus, the spatial dimension information of the three image components can be the same; then combining the first image component, the second image component and the third image component after the pixel rearrangement processing, and transforming the combined image component into the imageIs input into the improved loop filter.

In some embodiments, the low resolution image components may also be upsampled such that the spatial dimension information of the three image components is the same. Specifically, before the fusing the at least two image components of the image to be filtered and the auxiliary information, the method further includes:

selecting image components with low resolution according to at least two image components of the image to be filtered;

and performing up-sampling processing on the low-resolution image component.

It should be noted that, in addition to the pixel rearrangement processing (i.e., down adjustment) of the size information for the image component with high resolution, in the embodiment of the present application, the up sampling processing (i.e., up adjustment) may be performed for the image component with low resolution. In addition, for the image component with low resolution, not only the up-sampling process, but also the deconvolution process, and even the super-resolution process, etc. may be performed, and the effects of these three processes are the same, and the embodiment of the present application is not particularly limited.

It should be further noted that, if the first image component is a luminance component, the second image component is a first chrominance component, and the third image component is a second chrominance component; the image component with low resolution is the second image component or the third image component, and at this time, the second image component or the third image component needs to be up-sampled. Illustratively, when the size information of the second image component and the third image component of the original image are bothBefore loop filtering, the loop filter can be converted into H multiplied by W form through up-sampling processing; since the size information of the first image component is H × W, the spatial size information of the three image components can be the same, and the second image component after the up-sampling process and the third image component after the up-sampling process keep the same resolution as the first image component.

In some embodiments, the obtaining the image to be filtered includes:

carrying out video coding processing on an original image in a video to be coded, and taking a generated reconstructed image as the image to be filtered; alternatively, the first and second electrodes may be,

carrying out video coding processing on an original image in a video to be coded to generate a reconstructed image; and carrying out preset filtering processing on the reconstructed image, and taking the image subjected to preset filtering as the image to be filtered.

It should be noted that, based on the modified coding block diagram 20, in the process of video coding an original image in a video to be coded, when the original image is subjected to video coding processing, CU division, prediction, transformation, quantization and other processing are performed on the original image, and in order to obtain a reference image for video coding a subsequent image to be coded, inverse transformation, inverse quantization, reconstruction, filtering and other processing may also be performed on the reference image. In this way, the image to be filtered in the embodiment of the present application may be a reconstructed image generated after reconstruction processing in a video encoding process, or may also be a preset filtered image obtained by performing preset filtering on the reconstructed image by using another preset filtering method (for example, a deblocking filtering method), and the embodiment of the present application is not limited specifically.

In some embodiments, prior to the determining the fusion information of the image to be filtered, the method further comprises:

determining auxiliary information corresponding to the image to be filtered; wherein the side information at least includes block division information and/or quantization parameter information.

It is understood that the auxiliary information may be used to assist in filtering, improving the filtering quality. In the embodiment of the present application, the auxiliary information may be not only block partition information (e.g., CU partition information and/or CTU partition information), but also quantization parameter information, and even Motion Vector (MV) information, prediction direction information, and the like; these pieces of information may be used alone as auxiliary information, or may be used in any combination as auxiliary information, for example, the block division information may be used alone as auxiliary information, the block division information and the quantization parameter information may be used together as auxiliary information, or the block division information and the MV information may be used together as auxiliary information, and the embodiments of the present application are not particularly limited.

It can also be understood that since the original picture can be divided into CTUs, or divided into CUs by CTUs; therefore, the loop filter implementation method in the embodiment of the present application may be applied to not only the CU-level loop filter (at this time, the block partition information is CU partition information), but also the CTU-level loop filter (at this time, the block partition information is CTU partition information), and the embodiment of the present application is not limited specifically. The following will describe CU partition information as an example of block partition information.

In some embodiments, the determining auxiliary information corresponding to the image to be filtered includes:

and performing CU division on an original image in a video to be coded to obtain CU division information, and taking the CU division information as block division information corresponding to the image to be filtered.

Further, in some embodiments, the taking the CU partition information as the block partition information corresponding to the image to be filtered includes:

and taking the first matrix as block division information corresponding to the image to be filtered.

The first value may be a preset numerical value, a preset letter, or the like, and the second value may be a preset numerical value, a preset letter, or the like, where the first value is different from the second value; for example, the first value may be set to 2, and the second value may be set to 1, but the embodiment of the present application is not particularly limited.

In the embodiment of the present application, CU partition information may be used as auxiliary information to assist the image to be filtered in performing filtering processing. That is to say, in the process of video encoding of the original image in the video to be encoded, CU partition information can be fully utilized, and the CU partition information is fused with at least two image components of the image to be filtered to guide filtering.

Specifically, CU partition information is converted into a Coding Unit Map (CUmap), and is represented by a two-dimensional matrix, that is, a CUmap matrix, that is, a first matrix in the embodiment of the present application; that is, for an original image, it may be divided into a plurality of CUs; and filling the positions of the pixel points corresponding to each CU boundary with a first value, and filling the positions of other pixel points with a second value, so that a first matrix reflecting CU partition information can be constructed. For example, refer to fig. 4, which shows a schematic structural diagram of a block division matrix provided in an embodiment of the present application. As shown in fig. 4, if the diagram represents one CTU, the CTU may be divided into 9 CUs; assuming that the first value is set to 2 and the second value is set to 1; in this way, the pixel positions corresponding to each CU boundary are filled with 2, and the pixel positions corresponding to other pixel positions are filled with 1, that is, the pixel positions filled with 2 represent the CU boundaries, so that CU partition information, that is, auxiliary information corresponding to an image to be filtered can be determined.

In some embodiments, the determining auxiliary information corresponding to the image to be filtered includes:

Further, in some embodiments, the taking the quantization parameter as quantization parameter information corresponding to the image to be filtered includes:

and taking the second matrix as the quantization parameter information corresponding to the image to be filtered.

It should be noted that, the distortion degrees of the images to be filtered corresponding to different quantization parameters are different. If the quantization parameter information is integrated, the filtering network can be made to have the capability of processing any quantization parameter in a self-adaptive manner in the training process.

In the embodiment of the present application, the quantization parameter information may also be used as auxiliary information to assist the image to be filtered in performing filtering processing. That is to say, in the process of video encoding of the original image in the video to be encoded, the quantization parameter information can be fully utilized, and the quantization parameter information and at least two image components of the image to be filtered are fused to guide filtering. The quantization parameter information may be subjected to normalization processing, and the quantization parameter information may also be subjected to non-normalization processing (such as classification processing, interval division processing, and the like); the quantization parameter normalization process will be described in detail below as an example.

Specifically, the quantization parameter information is converted into a second matrix reflecting the quantization parameter information; that is, taking an original image as an example, a matrix with the same size as the original image is established, and the positions of all pixel points in the matrix are filled with normalized values of quantization parameters corresponding to the original image; wherein the normalization value of quantization parameter is QP_max(x, y) denotes, i.e.:

in the formula (1), QP represents a quantization parameter value corresponding to an original image, x represents an abscissa value of each pixel point position in the CU block, and y represents an ordinate value of each pixel point position in the CU block; QP_maxRepresents the maximum value of the quantization parameter, generally QP_maxIs 51, but QP_maxOther values, such as 29, 31, etc., are also possible, and the embodiments of the present application are not particularly limited.

In some embodiments, the improved loop filter comprises a convolutional neural network filter.

It should be noted that the improved loop filter is used to implement the loop filtering process for the image to be filtered. The improved loop filter may be a convolutional neural network filter, or may be a filter established by other deep learning, and the embodiment of the present application is not particularly limited. Here, a convolutional neural network filter, also referred to as a CNN filter, is a type of feed-forward neural network that includes convolution calculations and has a deep structure, and is one of the representative algorithms of deep learning. The input layer of the CNN filter may process multi-dimensional data such as three image component (Y/U/V) channels of an original image in the video to be encoded.

Referring to fig. 5, a schematic diagram of a composition structure of a conventional CNN filter 50 provided in an embodiment of the present application is shown. As shown in fig. 5, the conventional CNN filter 50 is an improvement on the basis of the h.265/High Efficiency Video Coding (HEVC) standard, and includes a 2-layer convolutional network structure, which can replace a deblocking filter and a sample adaptive compensation filter. Image to be filtered (with F)_inRepresentation) are input to the input layers of the conventional CNN filter 50, and sequentially pass through the first layer convolutional network F₁(assuming a convolution kernel size of 3X 3, containing 64 signatures) and a second convolutional network F₂(assuming that the convolution kernel has a size of 5 × 5 and contains 32 feature maps), a residual information F is obtained₃(ii) a Then the image F to be filtered_inAnd residual information F₃A summation operation is performed to finally obtain a filtered image (using F) output by the conventional CNN filter 50_outRepresentation). The convolution network structure is also called as a residual error neural network and is used for outputting residual error information corresponding to the image to be filtered. In this conventional CNN filter 50, the three image components (Y/U/V) of the image to be filtered are processed independently, but share the same filter network and related parameters of the filter network.

Referring to fig. 6A and 6B, schematic diagrams of a composition structure of another conventional CNN filter 60 provided in an embodiment of the present application are shown; this conventional CNN filter 60 uses two filter networks, the filter network shown in fig. 6A being dedicated to output the first image component, and the filter network shown in fig. 6B being dedicated to output the second image component or the third image component. Assuming that the height of the original image in the video to be encoded is H and the width is W, the size information corresponding to the first image component is H × W,the first image component may be subjected to a pixel rearrangement process to convert it intoIn the form of (a); the size information corresponding to the second image component or the third image component isThe three image components are combined and transformed intoIs input to a conventional CNN filter 60. Based on the filter network shown in fig. 6A, the input layer network receives the image F to be filtered_in(assuming the size of the convolution kernel is N, the number of channels is 6) and then the first layer of convolution network F is passed_1-Y(assuming the size of the convolution kernel is L1 XL 1, the number of convolution kernels is M, and the number of channels is 6) and a second layer convolution network F_2-Y(assuming that the size of the convolution kernel is L2 × L2, the number of convolution kernels is 4, and the number of channels is M), then a residual information F is obtained_3-Y(assuming the size of the convolution kernel is N × N and the number of channels is 4); then the input image F to be filtered_inAnd residual information F_3-YA summation operation is performed to finally obtain a filtered first image component (using F) output by the conventional CNN filter 60_out-YRepresentation). Based on the filter network shown in fig. 6B, the input layer network receives the image F to be filtered_in(assuming the size of the convolution kernel is N, the number of channels is 6) and then the first layer of convolution network F is passed_1-U(assuming the size of the convolution kernel is L1 XL 1, the number of convolution kernels is M, and the number of channels is 6) and a second layer convolution network F_2-U(assuming that the size of the convolution kernel is L2 × L2, the number of convolution kernels is 2, and the number of channels is M), then a residual information F is obtained_3-U(assuming the size of the convolution kernel is N × N and the number of channels is 2); then the input image F to be filtered_inAnd residual information F_3-UA summation operation is performed to finally obtain the filtered second image component or the filtered third image component (using F) output by the conventional CNN filter 60_out-URepresentation).

With respect to the conventional CNN filter 50 shown in fig. 5, or the conventional CNN filter 60 shown in fig. 6A and 6B, it is not reasonable to independently process each image component because the relationship between different image components is not considered; in addition, coding parameters such as block partition information and QP information are not fully utilized at the input end, however, the distortion of the reconstructed image mainly comes from the block effect, and the boundary information of the block effect is determined by the CU partition information; that is, the filter network in the CNN filter should focus on the boundary region; besides, the quantization parameter information is integrated into the filter network, so that the generalization capability of the filter network can be improved, and the filter network can filter distorted images with any quality. Therefore, the loop filtering implementation method provided by the embodiment of the application has the advantages that the CNN filtering structure is reasonably arranged, the same filtering network can simultaneously receive a plurality of image components, the relationship among the image components is fully considered, and the enhanced images of the image components can be simultaneously output after filtering; in addition, the loop filtering implementation method can also perform auxiliary filtering by using encoding parameters such as the fused block division information and/or QP information as auxiliary information, thereby improving the filtering quality.

In some embodiments, the determining fusion information of the image to be filtered includes:

and carrying out fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

It should be noted that, in the embodiment of the present application, the fusion information may be obtained by fusing a first image component, a second image component, a third image component, and auxiliary information of an image to be filtered; or the first image component, the second image component and the auxiliary information of the image to be filtered are fused to obtain fused information; or the first image component, the third image component and the auxiliary information of the image to be filtered are fused to obtain fused information; even the second image component, the third image component and the auxiliary information of the image to be filtered can be fused to obtain fusion information; the embodiments of the present application are not particularly limited. It should be further noted that "performing fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information" may be to fuse at least two image components of the image to be filtered first, and then fuse the auxiliary information; or, each image component of at least two image components of the image to be filtered and the corresponding auxiliary information are merged firstly, and then the at least two processed image components are merged; that is, the embodiment of the present application is not particularly limited to the specific mode of the fusion process.

In addition, in this embodiment of the application, "performing loop filtering processing on the image to be filtered based on the fusion information to obtain the at least one image component filtered by the image to be filtered", specifically, after fusing and inputting the plurality of image components (such as the first image component, the second image component, and the third image component) of the image to be filtered and the auxiliary information into the filtering network, only the first image component filtered by the image to be filtered, or the second image component filtered by the image to be filtered, or the third image component filtered by the image to be filtered may be output, the first image component filtered by the image to be filtered and the second image component filtered by the image to be filtered, or the second image component filtered by the image to be filtered and the third image component filtered by the image to be filtered, or even the first image component filtered by the image to be filtered, or the third image component filtered by the image to be filtered, The filtered second image component and the filtered third image component; the embodiments of the present application are not particularly limited.

Taking the example that three image components of the image to be filtered are simultaneously input into the filter network, refer to fig. 7, which shows a schematic structural diagram of a loop filter framework 70 provided in the embodiment of the present application. As shown in fig. 7, the loop filtering framework 70 may include three image components (respectively indicated by Y, U, V) 701 of an image to be filtered, side information 702, an input fusion unit 703, a joint processing unit 704, a first adder 705, a second adder 706, a third adder 707, and three filtered image components (respectively indicated by Out _ Y, Out _ U, Out _ V) 708. The input fusion unit 703, the joint processing unit 704, the first adder 705, the second adder 706, and the third adder 707 collectively form an improved loop filter in the embodiment of the present application; the input fusion unit 703 is used for fusing all the three image components 701 of the image to be filtered and the auxiliary information 702 together, and then inputting the fused image to the joint processing unit 704; the joint processing unit 704 includes a multi-layer convolution filter network for performing convolution calculation on the input information, and since a specific convolution calculation process is similar to that of the related art, a specific execution step of the joint processing unit 704 is not described. After passing through the joint processing unit 704, residual information of the Y image component, residual information of the U image component, and residual information of the V image component can be obtained respectively; inputting the Y image component of the three image components 701 of the image to be filtered and the residual information of the obtained Y image component into a first adder 705 together, wherein the output of the first adder 705 is the filtered Y image component (denoted by Out _ Y); inputting the U image component in the three image components 701 of the image to be filtered and the residual information of the obtained U image component into a second adder 706 together, wherein the output of the second adder 706 is the filtered U image component (denoted by Out _ U); the V image component of the three image components 701 of the image to be filtered and the residual information of the resulting V image component are input together to a third adder 707, and the output of the third adder 707 is the filtered V image component (denoted by Out _ V). Here, for the output component, if only the filtered Y image component needs to be output, the loop filtering framework 70 may not include the second adder 706 and the third adder 707; the loop filtering framework 70 may not include the first adder 705 and the third adder 707 if only the filtered U-picture component needs to be output; the loop filter framework 70 may not include the third adder 707 if it is desired to output the filtered Y image component and the filtered U image component; the embodiments of the present application are not particularly limited.

Taking the example that two image components of an image to be filtered are input into a filter network at the same time, refer to fig. 8, which shows a schematic structural diagram of another loop filter framework 80 provided in the embodiment of the present application. As shown in fig. 8, the loop filtering framework 80 includes two image components (denoted by Y and U, respectively) 801 of an image to be filtered, auxiliary information 702, an input fusion unit 703, a joint processing unit 704, a first adder 705, a second adder 706, and two image components (denoted by Out _ Y and Out _ U, respectively) 802 after filtering. The difference from the loop filtering framework 70 shown in fig. 7 is that the loop filtering framework 80 fuses the two image components 801 of the image to be filtered and the auxiliary information 702 together and then inputs them to the joint processing unit 704; after passing through the joint processing unit 704, residual information of the Y image component and residual information of the U image component can be obtained respectively; inputting the Y image component in the two image components 801 of the image to be filtered and the residual information of the obtained Y image component into a first adder 705 together, wherein the output of the first adder 705 is the filtered Y image component (denoted by Out _ Y); the U image component of the two image components 801 of the image to be filtered and the residual information of the obtained U image component are input to a second adder 706 together, and the output of the second adder 706 is the filtered U image component (denoted by Out _ U). Here, for the output component, if only the filtered Y image component needs to be output, the loop filtering framework 70 may not include the second adder 706; the loop filtering framework 70 may not include the first adder 705 if only the filtered U-picture component needs to be output; the embodiments of the present application are not particularly limited. It should be noted that, if only a single image component of an image to be filtered is considered to be subjected to filtering processing at a time, fusion among a plurality of image components is not considered at this time, which is the same as the filtering processing method of a conventional CNN filter in the related art, and detailed description is not given in this application embodiment.

Taking the loop filtering framework 70 shown in fig. 7 as an example, it employs a deep learning network (e.g., CNN) to perform loop filtering, and the difference from the conventional CNN filter is that the improved loop filter in the embodiment of the present application can simultaneously input three image components of an image to be filtered into the filtering network, and also incorporate other coding-related auxiliary information (e.g., block division information, quantization parameter information, MV information, etc.) into the filtering network, and input all these information into the filtering network at one time after incorporating them; therefore, the relation among the three image components is fully utilized, and other auxiliary information related to coding is used for assisting the filtering, so that the filtering quality is improved; in addition, the three image components are processed simultaneously, so that the problem that three times of complete network forward calculation is needed for the three image components is effectively solved, the calculation complexity is further reduced, and the coding code rate is saved. For example, based on VTM3.0, it is found in a certain experimental test that, compared with the related art, the loop filtering implementation method in the embodiment of the present application can simultaneously implement a bit rate reduction of 6.4% for the Y image component, a bit rate reduction of 9.8% for the U image component, and a bit rate reduction of 11.6% for the V image component on the premise of the same recovered video quality, thereby saving the encoding rate.

The above embodiment provides a loop filtering implementation method, where an image to be filtered is obtained, where the image to be filtered is generated in a video encoding process of an original image in a video to be encoded; determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information; performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component of the image to be filtered; therefore, the encoding parameters such as the block division information and/or the QP information are used as auxiliary information to be fused with the input multiple image components, so that the relationship among the multiple image components is fully utilized, the calculation complexity is reduced, and the encoding code rate is saved; and meanwhile, the subjective and objective quality of the video reconstruction image in the encoding and decoding process is further improved.

Based on the same inventive concept of the foregoing embodiment, refer to fig. 9, which shows a schematic structural diagram of a loop filter implementation apparatus 90 provided in the embodiment of the present application. The loop filter implementation apparatus 90 may include: an acquisition unit 901, a determination unit 902 and a filtering unit 903, wherein,

the acquiring unit 901 is configured to acquire an image to be filtered; the image to be filtered is generated in the video coding process of an original image in a video to be coded, the video to be coded comprises an original image frame, and the original image frame comprises the original image;

the determining unit 902 is configured to determine fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information;

the filtering unit 903 is configured to perform loop filtering processing on the image to be filtered based on the fusion information, so as to obtain at least one image component of the image to be filtered.

In the above scheme, the obtaining unit 901 is specifically configured to perform video coding processing on an original image in a video to be coded, and use a generated reconstructed image as the image to be filtered; alternatively, the first and second electrodes may be,

the acquiring unit 901 is specifically configured to perform video coding processing on an original image in a video to be coded, so as to generate a reconstructed image; and carrying out preset filtering processing on the reconstructed image, and taking the image subjected to preset filtering as the image to be filtered.

In the above scheme, the determining unit 902 is further configured to determine auxiliary information corresponding to the image to be filtered; wherein the side information at least includes block division information and/or quantization parameter information.

In the above scheme, referring to fig. 9, the apparatus 90 for implementing loop filtering further includes a dividing unit 904, configured to perform CU division on an original image in a video to be encoded to obtain CU division information, and use the CU division information as block division information corresponding to the image to be filtered.

In the above scheme, the determining unit 902 is specifically configured to, for the CU partition information, fill a first value in each pixel position corresponding to a CU boundary, and fill a second value in other pixel positions to obtain a first matrix corresponding to the CU partition information; wherein the first value is different from the second value; and taking the first matrix as the block division information corresponding to the image to be filtered.

In the foregoing solution, the obtaining unit 901 is further configured to obtain a quantization parameter corresponding to an original image in a video to be encoded, and use the quantization parameter as quantization parameter information corresponding to the image to be filtered.

In the above solution, the determining unit 902 is specifically configured to establish a second matrix having the same size as the original image; filling the normalized value of the quantization parameter corresponding to the original image in each pixel point position in the second matrix; and taking the second matrix as quantization parameter information corresponding to the image to be filtered.

In the above scheme, referring to fig. 9, the loop filter implementing apparatus 90 further includes a fusion unit 905 configured to perform fusion processing on at least two image components of the image to be filtered and corresponding auxiliary information to obtain fusion information of the image to be filtered.

In the above solution, referring to fig. 9, the loop filter implementing apparatus 90 further includes a sampling unit 906, configured to select, for at least two image components of the image to be filtered, an image component with a low resolution; and upsampling the low-resolution image component.

It is understood that in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and may also be a module, or may also be non-modular. Moreover, each component in the embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware or a form of a software functional module.

Based on the understanding that the technical solution of the present embodiment essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method of the present embodiment. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Accordingly, the present embodiment provides a computer storage medium storing a loop filter implementation program that, when executed by at least one processor, implements the steps of the method described in the preceding embodiments.

Based on the composition of the loop filter implementing apparatus 90 and the computer storage medium, referring to fig. 10, which shows a specific hardware structure example of the loop filter implementing apparatus 90 provided in the embodiment of the present application, the hardware structure example may include: a network interface 1001, a memory 1002, and a processor 1003; the various components are coupled together by a bus system 1004. It is understood that the bus system 1004 is used to enable communications among the components. The bus system 1004 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for the sake of clarity the various busses are labeled in fig. 10 as the bus system 1004. The network interface 1001 is used for receiving and transmitting signals during information transmission and reception with other external network elements;

a memory 1002 for storing a computer program capable of running on the processor 1003;

a processor 1003 configured to, when running the computer program, perform:

and performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered.

It is to be appreciated that the memory 1002 in the subject embodiment can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (ddr Data Rate SDRAM, ddr SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1002 of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

And the processor 1003 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be implemented by integrated logic circuits of hardware or instructions in the form of software in the processor 1003. The Processor 1003 may be a general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 1002, and the processor 1003 reads the information in the memory 1002 and performs the steps of the above method in combination with the hardware thereof.

It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the Processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, other electronic units configured to perform the functions described herein, or a combination thereof.

For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.

Optionally, as another embodiment, the processor 1003 is further configured to execute the steps of the method in the foregoing embodiments when running the computer program.

It should be noted that: the technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of the application, firstly, an image to be filtered is obtained, wherein the image to be filtered is generated in the video coding process of an original image in a video to be coded; then determining fusion information of the image to be filtered; the fusion information is obtained by fusing at least two image components of the image to be filtered and corresponding auxiliary information; finally, performing loop filtering processing on the image to be filtered based on the fusion information to obtain at least one image component after the image to be filtered is filtered; therefore, the coding parameters such as the block division information and/or the QP information are used as auxiliary information to be fused with the input multiple image components, so that the relation among the multiple image components is fully utilized, the problem that the multiple image components need to be subjected to multiple complete network forward calculations is effectively solved, the calculation complexity is further reduced, and the coding code rate is saved; in addition, auxiliary information such as block division information and/or QP information is fused, filtering can be further assisted, and the subjective and objective quality of video reconstruction images in the encoding and decoding process is improved.

25页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：一种编码树单元的参数确定方法及系统

Loop filtering implementation method, device and computer storage medium

相关技术

网友询问留言