阅读说明：本技术 局部色调映射电路、图像感测装置及其操作方法 (Local tone mapping circuit, image sensing apparatus and method of operating the same ) 是由 闵昌基 金镇寿 于 2021-02-20 设计创作，主要内容包括：本申请公开了局部色调映射电路、图像感测装置及其操作方法。局部色调映射电路可以包括：平均值计算电路,其适于计算从包括多个像素的像素阵列输出的像素数据当中的中央像素数据与中央像素数据的相邻像素数据之间的差的绝对值的平均值；范围计算电路,其适于计算中央像素数据与相邻像素数据之间的差的绝对值中的最大值和最小值之间的范围；以及局部色调映射强度调整电路,其适于基于由范围计算电路计算出的最大值和最小值之间的范围和由平均值计算电路计算出的平均值根据中央像素与中央像素的相邻像素当中的边缘像素之间的距离来不同地分配权重值,并根据所分配的权重值来调整局部色调映射强度。(The application discloses a local tone mapping circuit, an image sensing device and an operation method thereof. The local tone mapping circuit may include: an average value calculation circuit adapted to calculate an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from a pixel array including a plurality of pixels; a range calculation circuit adapted to calculate a range between a maximum value and a minimum value among absolute values of differences between the center pixel data and the adjacent pixel data; and a local tone mapping intensity adjustment circuit adapted to assign weight values differently according to distances between the center pixel and edge pixels among adjacent pixels of the center pixel based on a range between the maximum value and the minimum value calculated by the range calculation circuit and an average value calculated by the average value calculation circuit, and to adjust the local tone mapping intensity according to the assigned weight values.)

1. A local tone mapping circuit, the local tone mapping circuit comprising:

an average value calculation circuit configured to calculate an average value of absolute values of differences between central pixel data in respective cores of a pixel array and adjacent pixel data of the central pixel data among pixel data output from the pixel array including a plurality of pixels;

a range calculation circuit configured to calculate a range between a maximum value and a minimum value of the absolute values; and

a local tone mapping intensity adjustment circuit configured to assign a weight value to the pixel data according to a distance between a center pixel and an edge pixel among neighboring pixels of the center pixel based on the range and the average value, and to adjust a local tone mapping intensity according to the assigned weight value.

2. The local tone mapping circuit according to claim 1, wherein the local tone mapping intensity adjustment circuit sets the weight value based on a set threshold according to the distance between the center pixel and the edge pixel.

3. The local tone mapping circuit according to claim 2, wherein the weight value is calculated by:

wherein "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes the average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes the maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes the minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

4. The local tone mapping circuit according to claim 1, wherein the local tone mapping intensity adjustment circuit decreases the local tone mapping intensity when the range is larger than a set value and the average value is smaller than the set value.

5. The local tone mapping circuit according to claim 1, wherein the local tone mapping intensity adjustment circuit increases the local tone mapping intensity when the range is smaller than a set value and the average value is larger than the set value.

6. An image sensing device, comprising:

an image sensor including a pixel array having a plurality of pixels;

an image signal processor configured to process an output signal of the image sensor; and

a local tone mapping circuit, the local tone mapping circuit comprising:

an average value calculation circuit configured to calculate an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from the pixel array;

a range calculation circuit configured to calculate a range between a maximum value and a minimum value of the absolute values; and

a local tone mapping intensity adjustment circuit configured to assign a weight value to the pixel data according to a distance between a center pixel and an edge pixel among neighboring pixels of the center pixel based on the range and the average value, and to adjust a local tone mapping intensity according to the assigned weight value.

7. The image sensing apparatus according to claim 6, wherein the local tone mapping intensity adjustment circuit sets the weight value based on a set threshold according to the distance between the center pixel and the edge pixel.

8. The image sensing device according to claim 7, wherein the weight value is calculated by:

wherein "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes the average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes the maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes the minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

9. The image sensing device according to claim 6, wherein the local tone mapping intensity adjustment circuit decreases the local tone mapping intensity when the range is larger than a set value and the average value is smaller than the set value.

10. The image sensing device according to claim 6, wherein the local tone mapping intensity adjustment circuit increases the local tone mapping intensity when the range is smaller than a set value and the average value is larger than the set value.

11. An operating method of an image sensing device, the operating method comprising the steps of:

calculating an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from a pixel array including a plurality of pixels;

calculating a range between a maximum value and a minimum value of the absolute values;

assigning a weight value to pixel data according to a distance between a central pixel and an edge pixel among the neighboring pixels of the central pixel based on the range and the average value; and

the local tone mapping strength is adjusted according to the assigned weight value.

12. The method of operation of claim 11, wherein assigning the weight values comprises: setting the weight value based on a set threshold according to a distance between the center pixel and the edge pixel.

13. The operating method of claim 12, wherein the weight value is calculated by:

wherein "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes the average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes the maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes the minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

14. The method of operation of claim 11, wherein adjusting the local tone mapping strength comprises: when the range is greater than a set value and the average value is less than the set value, the local tone mapping strength is decreased.

15. The method of operation of claim 11, wherein adjusting the local tone mapping strength comprises: increasing the local tone mapping intensity when the range is less than a set value and the average value is greater than the set value.

16. An image sensing device, comprising:

an image sensor including a pixel array having a plurality of pixels; and

a local tone mapping circuit configured to:

receiving pixel data from the pixel array, the pixel data including a plurality of pixel entries in a setup kernel;

calculating an absolute value of a difference between a central pixel item among the plurality of pixel items and each of neighboring pixel items neighboring the central pixel item;

assigning a weight value to the center pixel item based on a range between a maximum value and a minimum value of the absolute values and an average value of the absolute values; and

adjusting the local tone mapping strength of the center pixel term according to the weight value.

Technical Field

Various embodiments of the present invention generally relate to semiconductor devices. In particular, various embodiments relate to a local tone (local tone) mapping circuit, an image sensing apparatus, and an operating method thereof.

Background

Recently, computer environment paradigms have moved to ubiquitous computing, which enables computer systems to be used almost anytime and anywhere. As a result, the use of portable electronic devices such as mobile phones, digital cameras, notebook computers, and the like has been rapidly increased.

Recently, due to rapid development of display devices, development of image photographing devices having image sensors, such as cameras and camcorders, has been accelerated. The image capturing apparatus can capture an image and record the captured image in a recording medium, and at the same time, can reproduce the image at any time. Therefore, as the number of users of the image photographing device increases, the demand for more functions in the image photographing device also increases. In particular, in addition to the compact size, reduced weight, and lower power consumption of the image capturing apparatus, more functions for high performance and multiple functions are desired.

Disclosure of Invention

Embodiments of the present invention relate to a local tone mapping circuit, an image sensing apparatus, and an operating method thereof, which are capable of reducing a distortion phenomenon. The distortion phenomenon may be halo artifacts, such as may occur in a local tone mapping operation. The operation may use an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from a pixel array including a plurality of pixels.

In an embodiment, the local tone mapping circuit may include: an average value calculation circuit adapted to calculate an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data in respective kernels (kernel) of a pixel array among pixel data output from the pixel array including a plurality of pixels; a range calculation circuit adapted to calculate a range between a maximum value and a minimum value of the absolute values; and a local tone mapping intensity adjusting circuit adapted to assign a weight value to the pixel data according to a distance between the center pixel and an edge pixel among adjacent pixels of the center pixel based on the range and the average value, and to adjust the local tone mapping intensity according to the assigned weight value.

The local tone mapping intensity adjustment circuit may set the weight value based on a set threshold according to a distance between the center pixel and the edge pixel.

The weight value is calculated by:

where "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes an average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes a maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes a minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

The local tone mapping strength adjustment circuit may decrease the local tone mapping strength when the range is greater than the set value and the average value is less than the set value.

The local tone mapping strength adjustment circuit may increase the local tone mapping strength when the range is less than the set value and the average value is greater than the set value.

In another embodiment, an image sensing apparatus may include: an image sensor including a pixel array having a plurality of pixels; an image signal processor adapted to process an output signal of the image sensor; and a local tone mapping circuit, comprising: an average value calculation circuit adapted to calculate an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from the pixel array; a range calculation circuit adapted to calculate a range between a maximum value and a minimum value of the absolute values; and a local tone mapping intensity adjusting circuit adapted to assign a weight value to the pixel data according to a distance between the center pixel and an edge pixel among adjacent pixels of the center pixel based on the range and the average value, and to adjust the local tone mapping intensity according to the assigned weight value.

The local tone mapping intensity adjustment circuit may set the weight value based on a set threshold according to a distance between the center pixel and the edge pixel.

The local tone mapping strength adjustment circuit may calculate the weight values by the following equation,

where "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes an average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes a maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes a minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

The local tone mapping strength adjustment circuit may decrease the local tone mapping strength when the range is greater than the set value and the average value is less than the set value.

The local tone mapping strength adjustment circuit may increase the local tone mapping strength when the range is less than the set value and the average value is greater than the set value.

In another embodiment, a method of operating a sensing device may include the steps of: calculating an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from a pixel array including a plurality of pixels; calculating a range between a maximum value and a minimum value in the absolute values; assigning a weight value to the pixel data according to a distance between the central pixel and an edge pixel among neighboring pixels of the central pixel based on the range and the average; and adjusting the local tone mapping strength according to the assigned weight value.

The step of assigning a weight value may comprise: the weight value is set based on a set threshold according to the distance between the center pixel and the edge pixel.

The weight value is calculated by:

where "dweight" denotes a weight value according to a distance between the center pixel and the edge pixel, "dif _ aver" denotes an average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes a maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes a minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

The step of adjusting the local tone mapping intensity may comprise: when the range is greater than the set value and the average value is less than the set value, the local tone mapping strength is decreased.

The step of adjusting the local tone mapping intensity may comprise: when the range is smaller than the set value and the average value is larger than the set value, the local tone mapping strength is increased.

In another embodiment, an image sensing apparatus may include: an image sensor including a pixel array having a plurality of pixels; and a local tone mapping circuit adapted to: receiving pixel data from a pixel array, the pixel data including a plurality of pixel terms in a setup kernel; calculating an absolute value of a difference between a central pixel item among the plurality of pixel items and each of neighboring pixel items neighboring the central pixel item; assigning a weight value to the central pixel item based on a range between a maximum value and a minimum value of the absolute values and an average value of the absolute values; and adjusting the local tone mapping strength of the central pixel item according to the weight value.

These and other features and advantages of the present invention will be appreciated by those of ordinary skill in the art from the following drawings and detailed description.

Drawings

The description herein makes reference to the accompanying drawings wherein like reference numerals refer to like parts throughout the several views.

Fig. 1 is a block diagram illustrating an image sensing apparatus employing a local tone mapping circuit according to an embodiment of the present invention.

Fig. 2 is a block diagram illustrating an image sensing apparatus employing a local tone mapping circuit according to an embodiment of the present invention.

Fig. 3 is a block diagram illustrating a local tone mapping circuit according to an embodiment of the present invention.

Fig. 4 is a diagram illustrating a kernel of a size of 5 × 5 pixels in the local tone mapping circuit shown in fig. 3.

Fig. 5 is a flowchart illustrating an operation of the image sensing apparatus according to the embodiment of the present invention.

Fig. 6 is a block diagram illustrating a system configured to implement an image sensing apparatus according to an embodiment of the present invention.

Detailed Description

Various examples of the invention are described in more detail below with reference to the accompanying drawings. The invention may be embodied in other embodiments, forms and variations and should not be construed as limited to the embodiments set forth herein. Rather, the described embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the invention to those skilled in the art. Reference throughout the specification to "one embodiment," "another embodiment," and so forth, does not necessarily mean only one embodiment, and different references to any such phrases are not necessarily referring to the same embodiment.

It will be understood that, although the terms first, second, third, etc. may be used herein to identify various elements, these elements are not limited by these terms. These terms are used to distinguish one element from another element which might otherwise have the same or similar name. Thus, a first element in one instance may be termed a second element or a third element in another instance, without any change in the elements themselves.

The drawings are not necessarily to scale and, in some instances, the proportions may have been exaggerated to clearly illustrate features of the embodiments. When an element is referred to as being connected or coupled to another element, it will be understood that the former may be directly connected or coupled to the latter, or electrically connected or coupled to the latter via one or more intervening elements. Unless the context indicates otherwise, whether two elements are connected/coupled directly or indirectly, communication between the two elements may be wired or wireless. In addition, it will also be understood that when an element is referred to as being "between" two elements, it can be the only element between the two elements, or one or more intervening elements may also be present.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms and vice versa unless the context clearly indicates otherwise. The articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form.

It will be further understood that the terms "comprises," "comprising," "includes" and "including," when used in this specification, specify the presence of stated elements, and do not preclude the presence or addition of one or more other elements. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present invention and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. In other instances, well known process structures and/or processes have not been described in detail in order to not unnecessarily obscure the present invention.

It is also noted that, in some instances, features or elements described in connection with one embodiment may be used alone or in combination with other features or elements of another embodiment, unless specifically stated otherwise, as will be apparent to one skilled in the relevant art.

Embodiments of the present invention are described in detail with reference to the accompanying drawings.

Fig. 1 is a block diagram illustrating an image sensing apparatus 10 according to an embodiment of the present invention.

Referring to fig. 1, an image sensing apparatus 10 may include an image sensor 100 and an Image Signal Processor (ISP) 400.

The image sensing device 10 may be implemented in any suitable electronic device that can receive and process image data, such as a Personal Computer (PC) or mobile computing device.

More specifically, the image sensing device 10 may be implemented in a laptop computer, a mobile phone, a smart phone, a tablet, a Personal Digital Assistant (PDA), an Enterprise Digital Assistant (EDA), a digital camera, a digital video camera, a Portable Multimedia Player (PMP), a Mobile Internet Device (MID), a wearable computer, an internet of things (IoT), or an internet of everything (IoE).

The image sensor 100 may include a pixel array 200 and a local tone mapping circuit 300.

The pixel array 200 may include a plurality of pixels. Herein, a pixel may represent pixel data, and may have an RGB data format, a YUV data format, or a YCbCr data format. Note, however, that the present invention is not limited to the above data format.

The local tone mapping circuit 300 may calculate an average value of absolute values of differences between central pixel data and adjacent pixel data among pixel data output from the pixel array 200. Further, the local tone mapping circuit 300 may calculate a range between the maximum value and the minimum value among the absolute values of the differences between the center pixel data and the adjacent pixel data, and adjust the local tone mapping intensity based on a weight value assigned according to the distance between the center pixel and the edge pixel among the adjacent pixels of the center pixel.

The local tone mapping circuit 300 may be implemented as shown in fig. 3-5.

The detailed configuration and operation of the local tone mapping circuit 300 will be described in detail later with reference to fig. 3 to 5.

The image signal processor 400 may be implemented in an integrated circuit, a system on a chip (SoC), or a mobile application processor. The image signal processor 400 may process an output signal of the image sensor 100. That is, the image signal processor 400 may receive and process an image output signal output from the local tone mapping circuit 300 of the image sensor 100.

More specifically, the image signal processor 400 may generate RGB image data from a Bayer (Bayer) pattern corresponding to the pixel data. For example, the image signal processor 400 may process a bayer pattern so that image data is displayed in a display, and may transfer the processed image data to an interface.

In an embodiment, each of the image sensor 100 and the image signal processor 400 may be implemented as a multi-chip package (MCP). In another embodiment, the image sensor 100 and the image signal processor 400 may be implemented as a single chip.

Fig. 2 is a block diagram illustrating the image sensing apparatus 10 according to an embodiment of the present invention.

Referring to fig. 2, the image sensing apparatus 10 may include an image sensor 100 and an Image Signal Processor (ISP) 400. The image signal processor 400 may include a local tone mapping circuit 300.

The local tone mapping circuit 300 may be implemented as shown in fig. 3-5.

The structure and operation of the image sensing apparatus 10 shown in fig. 2 may be substantially the same as those of the image sensing apparatus 10 shown in fig. 1, except that the local tone mapping circuit 300 is implemented not inside the image sensor 100 but in the image signal processor 400. Therefore, a detailed description of the image sensor device 10 of fig. 2 may be omitted.

Hereinafter, the local tone mapping circuit 300 is described with reference to fig. 3 to 5.

Fig. 3 is a block diagram illustrating a local tone mapping circuit 300 according to an embodiment of the present invention. Fig. 4 is a diagram illustrating a kernel of a size of 5 × 5 pixels in the local tone mapping circuit 300 shown in fig. 3. Fig. 5 is a flowchart illustrating an operation of the image sensing apparatus according to the embodiment of the present invention.

Referring to fig. 3, the local tone mapping circuit 300 may include an average value calculation circuit 310, a range calculation circuit 320, and a local tone mapping intensity adjustment circuit 330.

The average value calculation circuit 310 may calculate the center pixel data P among the pixel data output from the pixel array including a plurality of pixels_center(e.g., P of FIG. 4)₃₃) Neighboring pixel data adjacent to the central pixel data (e.g., divide by P in FIG. 4)₃₃Other than P₁₁To P₅₅) The average of the absolute values of the differences therebetween.

Herein, the central pixel data P may be calculated according to equation 1_centerAnd adjacent pixel data P₁₁To P₅₅The average of the absolute values of the differences therebetween.

[ formula 1]

Where "dif _ aver" represents an average value of absolute values of differences between central pixel data and adjacent pixel data, "N" represents a kernel size, "abs (P)_center-P_x，_y) "represents an absolute value of a difference between the central pixel data and one of the adjacent pixel data," P_center"denotes pixel data of a central pixel located in a kernel having a size of (n × n), and" P_x，_y"denotes pixel data having coordinates (x, y) in a kernel having a size of (n × n).

In an embodiment of the present invention, "N" may be "5" and "P_x，_y"denotes P in a kernel having a size of (n × n)₁₁To P₅₅Corresponding pixel data.

The range calculation circuit 320 may calculate the central pixel data P_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween.

Herein, the central pixel data P may be calculated according to equation 2_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween.

[ formula 2]

dif_max＝max(abs(P_center-P_x，y))

dif_min＝_min(abs(P_center-P_x，y))，

Where "dif _ max" represents the maximum value of the absolute value of the difference between the central pixel data and the adjacent pixel data, "dif _ min" represents the minimum value of the absolute value of the difference between the central pixel data and the adjacent pixel data, "abs (P)_center-P_x，y) "represents the absolute value of the difference between the central pixel data and the adjacent pixel data," P_center"represents pixel data located in the center pixel of a kernel having a size of (n × n) and" P_x，y"denotes pixel data of coordinates (x, y) in a kernel having a size of (n × n).

The local tone mapping intensity adjustment circuit 330 may assign the weight values to the pixel data differently according to the distances between the center pixel in the corresponding kernel and the edge pixels among the adjacent pixels of the center pixel, based on the range between the maximum value and the minimum value calculated by the range calculation circuit 320 and the average value calculated by the average value calculation circuit 310. Further, the local tone mapping strength adjustment circuit 330 may adjust the local tone mapping strength according to the assigned weight value.

Herein, the assigned weight value may be calculated according to equation 3.

[ formula 3]

Where "dweight" denotes a weight value according to a distance between a center pixel in the kernel and an edge pixel among adjacent pixels of the center pixel, "dif _ aver" denotes an average value of absolute values of differences between the center pixel data and the adjacent pixel data, "dif _ max" denotes a maximum value of absolute values of differences between the center pixel data and the adjacent pixel data, and "dif _ min" denotes a minimum value of absolute values of differences between the center pixel data and the adjacent pixel data.

Further, the local tone mapping intensity adjustment circuit 330 may set a weight value for the pixel data based on a predetermined threshold value according to a distance between the center pixel and the edge pixel among the adjacent pixels of the center pixel.

When the central pixel data P_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween is larger than a predetermined value and the central pixel data P_centerAnd adjacent pixel data P₁₁To P₅₅The local tone mapping strength adjustment circuit 330 may reduce the local tone mapping strength when the average of the absolute values of the differences therebetween is smaller than a predetermined value. Herein, it may be determined that the edge pixel and the corresponding pixel are disposed relatively far away from each other. If the local tone mapping strength is reduced, the effect of edge pixel image data on low frequencies may be reduced.

When the central pixel data P_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween is smaller than a predetermined value and the central pixel data P_centerAnd adjacent pixel data P₁₁To P₅₅The local tone mapping strength adjustment circuit 330 may increase the local tone mapping strength when the average of the absolute values of the differences therebetween is greater than a predetermined value. Herein, the edge pixel and the adjacent pixel data P may be determined₁₁To P₅₅Are arranged relatively close to each other. Therefore, distortion such as halo artifact can be preventedLike widespread distribution.

Hereinafter, an operation of the image sensing device according to the embodiment of the present invention will be described with reference to fig. 5. Fig. 5 is a flowchart illustrating an operation of an image sensing device (e.g., the image sensing device 10 in fig. 1 and 2) according to an embodiment of the present invention.

Referring to fig. 5, the operation of the image sensing apparatus according to another embodiment of the present invention may include an average value calculation operation S1000, a range calculation operation S2000, a weight value assignment operation S3000, and a local tone mapping intensity adjustment operation S4000.

In operation S1000, central pixel data P among pixel data output from a pixel array including a plurality of pixels may be calculated_centerAnd central pixel data P_centerAdjacent pixel data P₁₁To P₅₅The average of the absolute values of the differences therebetween.

Herein, the central pixel data P may be calculated according to equation 1_centerAnd adjacent pixel data P₁₁To P₅₅The average of the absolute values of the differences therebetween.

In operation S2000, the central pixel data P may be calculated_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween.

Herein, the central pixel data P may be calculated according to equation 2_centerAnd adjacent pixel data P₁₁To P₅₅The range between the maximum value and the minimum value of the absolute value of the difference therebetween.

In operation S3000, a weight value according to a distance between the center pixel and the edge pixel may be differently assigned based on a range between the maximum value and the minimum value calculated in operation S2000 and the average value calculated in operation S1000.

Herein, a weight value according to a distance between the center pixel and the edge pixel may be assigned according to equation 3.

In operation S4000, the local tone mapping strength may be adjusted according to the differently assigned weight values.

When a range between the maximum value and the minimum value of the absolute value of the difference between the center pixel data and the adjacent pixel data is larger than a set value and the average value of the absolute value of the difference between the center pixel data and the adjacent pixel data is smaller than the set value, the local tone mapping intensity may be reduced.

When a range between the maximum value and the minimum value of the absolute value of the difference between the center pixel data and the adjacent pixel data is smaller than a set value and an average value of the absolute value of the difference between the center pixel data and the adjacent pixel data is larger than a predetermined value, the local tone mapping intensity may be increased.

Hereinafter, a system configured to implement an image sensing apparatus according to an embodiment of the present invention is described in detail with reference to fig. 6.

FIG. 6 illustrates a system configured to implement an image sensing device according to an embodiment of the invention.

In various embodiments, the system of FIG. 6 may be any of various types of computing devices, including, but not limited to: personal computer systems, desktop, laptop or notebook computers, mainframe computer systems, handheld computing devices, cellular telephones, smart phones, mobile telephones, workstations, network computers, consumer devices, application servers, storage devices, smart displays, peripheral devices such as switches, modems, routers, and the like, or generally any type of computing device. According to an embodiment, the system of fig. 6 may represent a system on chip (SoC). The circuitry of SoC 1000 may be integrated onto a single semiconductor substrate as an integrated circuit "chip". In some embodiments, the circuitry may be implemented on two or more separate chips in a system. SoC 1000 will be used as an example herein.

In the illustrated embodiment, the circuitry of SoC 1000 includes a Central Processing Unit (CPU) complex 1020, on-chip peripheral circuits 1040A-1040B (more simply, "peripherals"), a Memory Controller (MC)1030, a communication fabric 1010, and an image signal processor 400. SoC 1000 may also be coupled to additional circuitry such as memory 1800 and image sensor 100. The circuits 1020, 1030, 1040A-1040B, and 400 may all be coupled to a communication structure 1010. Memory controller 1030 may be coupled to memory 1800, and peripherals 1040B may be coupled to external interface 1900. In addition, an image signal processor 400 may be coupled to the image sensor 100.

Peripherals 1040A-1040B may be any set of additional hardware functionality in SoC 1000. For example, peripherals 1040A-1040B may include a display controller, a Graphics Processing Unit (GPU), a video encoder/decoder, a scaler, a rotator, a mixer (blender), etc., configured to display video data on one or more display devices.

In some implementations, the image signal processor 400 may be part of another video peripheral configured to process image capture data from the image sensor 100 (or other image sensor). The image signal processor 400 and the image sensor 100 may be configured to implement the image sensor 400 and the image sensor 100 shown in fig. 1 to 5.

Peripherals 1040A-1040B may also include audio peripherals such as a microphone, a speaker, an interface to a microphone and a speaker, an audio processor, a digital signal processor, a mixer, and the like. Peripherals 1040A-1040B (e.g., peripheral 1040B) may include peripheral interface controllers for various interfaces 1900 external to SoC 1000, including interfaces such as Universal Serial Bus (USB), Peripheral Circuit Interconnect (PCI) including PCI-express (PCIe), serial and parallel ports, and the like. Peripherals 1040A-1040B may also include networking peripherals such as Media Access Controllers (MACs). In general, any hardware suite may be included according to various embodiments.

The CPU complex 1020 may include one or more processors (P)1024 that function as CPUs of the SoC 1000. Processor 1024 may execute the main control software of the system, such as an operating system. Generally, software executed by the CPU may control the other circuits of the system to achieve the desired functionality of the system. The processor 1024 may also execute other software such as application programs. The application may provide user functionality and may rely on the operating system for underlying device control. Thus, the processor 1024 may also be referred to as an application processor. The CPU complex 1020 may further include other hardware such as an L2 cache 1022 and/or interfaces to other circuitry of the system (e.g., interfaces to the communication fabric 1010).

In general, a processor may include any circuitry and/or microcode configured to execute instructions defined in an instruction set architecture implemented by the processor. Instructions and data that are manipulated by the processor in response to executing the instructions may generally be stored in the memory 1800, although some instructions may also be defined for direct processor access to peripherals. A processor may encompass a processor core implemented on an integrated circuit as a system on a chip (SOC 1000) or at other levels of integration, along with other circuitry. A processor may also encompass a discrete microprocessor, a processor core and/or microprocessor integrated into a multi-chip module implementation, a processor implemented as a plurality of integrated circuits, and the like.

Memory controller 1030 may generally include circuitry for receiving memory operations from other circuitry of SoC 1000 and for accessing memory 1800 to complete the memory operations. The memory controller 1030 may be configured to access any type of memory 1800. For example, the memory 1800 may be a Static Random Access Memory (SRAM) or a Dynamic Ram (DRAM) such as a synchronous DRAM (sdram) including double data rate (DDR, DDR2, DDR3, etc.) DRAM. Low power/mobile versions of DDR DRAM (e.g., LPDDR, mDDR, etc.) may be supported. The memory controller 1030 may include queues for memory operations to order (and possibly reorder) the operations and present the operations to the memory 1800. Memory controller 1030 may also include a data buffer to store write data waiting to be written to memory and read data waiting to be returned to the source of the memory operation. In some implementations, memory controller 1030 may include a memory cache to store recently accessed memory data. For example, in a SoC implementation, memory caching may reduce power consumption in the SoC by avoiding re-accessing data from memory 1800 (if it is expected to be accessed again soon). In some cases, a memory cache may also be referred to as a system cache, as opposed to a dedicated cache, such as an L2 cache 1022 or a cache in processor 1024 for only some circuits. Additionally, in some embodiments, the system cache need not be located within memory controller 1030.

In an embodiment, memory 1800 may be packaged with SoC 1000 in a chip-on-chip or package-on-package configuration. A multi-chip module configuration of SoC 1000 and memory 1800 may also be used. Such a configuration may be relatively more secure (in terms of data observability) than transmissions to other circuits in the system (e.g., to an endpoint). Thus, protected data may reside unencrypted in memory 1800, while protected data may be encrypted to be exchanged between SoC 1000 and an external endpoint.

The communication fabric 1010 may be any communication interconnect and protocol for communicating between the circuits of the SoC 1000. The communication fabric 1010 may be bus-based, including shared bus fabrics, crossbar fabrics, and hierarchical buses with bridges. The communication fabric 1010 may also be packet-based and may be a hierarchy with bridges, a crossbar, point-to-point, or other interconnect.

Note that the number of circuits of SoC 1000 (and the number of sub-circuits within CPU complex 1020) may vary from implementation to implementation. Each circuit/sub-circuit may be more or less than the number shown in fig. 6.

In some embodiments, the methods described herein may be implemented by a computer program product or software. In some embodiments, a non-transitory computer-readable storage medium may have stored thereon instructions that may be used to program a computer system (or other electronic devices) to perform some or all of the techniques described herein. A computer-readable storage medium may include any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). The machine-readable medium may include, but is not limited to, magnetic storage media (e.g., floppy diskettes); optical storage media (e.g., CD-ROM); a magneto-optical storage medium; read Only Memory (ROM); random Access Memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory or an electronic or other type of media suitable for storing program instructions. Additionally, program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).

As described above, the local tone mapping circuit, the image sensing apparatus, and the operating method thereof according to the embodiments of the present invention may reduce a distortion phenomenon such as a halo artifact occurring in a local tone mapping operation using an average value of absolute values of differences between central pixel data and adjacent pixel data of the central pixel data among pixel data output from a pixel array including a plurality of pixels.

While particular embodiments of the present invention have been illustrated and described, it would be obvious to those skilled in the art that, based upon the teachings of the present invention, that various changes and modifications can be made without departing from the spirit and scope of the invention as defined in the following claims. The present invention is intended to embrace all such alternatives and modifications as fall within the scope of the appended claims.

Cross Reference to Related Applications

This patent application claims priority from korean patent application No.10-2020-0059164, filed on 18/5/2020, which is incorporated herein by reference in its entirety.