阅读说明：本技术 编码装置和方法、图像捕获装置以及存储介质 (Encoding apparatus and method, image capturing apparatus, and storage medium ) 是由 村田贵史 望月成记 铃木辽太 于 2021-05-25 设计创作，主要内容包括：本公开涉及编码装置和方法、图像捕获装置以及存储介质。一种编码装置包括：生成部件,该生成部件用于从获得自图像传感器的RAW数据针对相应曝光时间生成多个RAW数据片,该图像传感器能够以针对每个像素不同的曝光时间执行拍摄；以及编码部件,该编码部件用于对由生成部件生成的多个RAW数据片进行编码。(The present disclosure relates to an encoding apparatus and method, an image capturing apparatus, and a storage medium. An encoding apparatus includes: a generation section for generating a plurality of pieces of RAW data for respective exposure times from RAW data obtained from an image sensor capable of performing photographing with different exposure times for each pixel; and an encoding means for encoding the plurality of RAW slices generated by the generation means.)

1. An encoding apparatus comprising:

a generation section for generating a plurality of pieces of RAW data for respective exposure times from RAW data obtained from an image sensor capable of performing photographing with different exposure times for each pixel; and

encoding means for encoding the plurality of RAW slices generated by the generating means.

2. The encoding apparatus according to claim 1, wherein the generating means generates RAW data in a bayer arrangement for each exposure time.

3. The encoding apparatus according to claim 1, wherein the exposure time different for each pixel is constituted of two types, a first exposure time and a second exposure time longer than the first exposure time.

4. The encoding apparatus according to claim 3, wherein,

wherein the generation means generates two pieces of RAW data corresponding to the first exposure time and two pieces of RAW data corresponding to the second exposure time, and

the encoding means encodes the two pieces of RAW data corresponding to the first exposure time and the two pieces of RAW data corresponding to the second exposure time.

5. The encoding apparatus according to claim 3, wherein,

wherein the generation means generates one piece of RAW data corresponding to the first exposure time by adding a plurality of pieces of pixel data of the first exposure time and generates one piece of RAW data corresponding to the second exposure time by adding a plurality of pieces of pixel data of the second exposure time, and

the encoding means encodes the one piece of RAW data corresponding to the first exposure time and the one piece of RAW data corresponding to the second exposure time.

6. The encoding device according to claim 5, wherein the generation means generates the one piece of RAW data corresponding to the first exposure time and the one piece of RAW data corresponding to the second exposure time by calculating an addition average of a plurality of pieces of pixel data.

7. The encoding apparatus according to claim 3, wherein,

wherein the generation means generates differential RAW data from a difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time and RAW data corresponding to the other of the first exposure time and the second exposure time, and

the encoding means encodes the differential RAW data and RAW data corresponding to the other of the first exposure time and the second exposure time.

8. The encoding device according to claim 7, wherein the generating means applies a gain to the RAW data corresponding to one of the first exposure time and the second exposure time so as to be closer to the RAW data corresponding to the other of the first exposure time and the second exposure time.

9. The encoding device according to claim 1, wherein the generating means generates the plurality of pieces of RAW data by calculating an addition average of signals of pixels of the same exposure time and the same color existing in the vicinity.

10. The encoding device according to claim 1, further comprising control means for controlling an exposure time of each pixel of the image sensor,

wherein the generating means generates the plurality of pieces of RAW data if the exposure time of each pixel is changed, and generates one piece of RAW data if the exposure time of each pixel is not changed.

11. The encoding device according to claim 10, wherein the generating means generates one piece of RAW data by calculating an average value of image data of pixels of the same color existing in the vicinity if the exposure time of each pixel does not change.

12. The encoding device according to claim 3, wherein the encoding means determines the quantization parameter of the RAW data corresponding to one of the first exposure time and the second exposure time using the quantization parameter of the RAW data corresponding to the other of the first exposure time and the second exposure time as a reference.

13. The encoding device according to claim 12, wherein the encoding means determines the quantization parameter of the RAW data corresponding to the second exposure time using the quantization parameter of the RAW data corresponding to the first exposure time as a reference.

14. The encoding apparatus according to claim 1, wherein,

wherein the generating means generates first RAW data for a first exposure time and second RAW data for a second exposure time different from the first exposure time,

the encoding apparatus further includes quantizing means for quantizing the first RAW data and the second RAW data,

the encoding means encodes the first RAW data and the second RAW data that have been quantized by the quantizing means, and

the quantization means determines a quantization parameter for the first RAW data and a quantization parameter for the second RAW data for respective regions classified by luminance of the first RAW data.

15. The encoding apparatus according to claim 14, wherein the quantizing means

Determining which of the first RAW data and the second RAW data has a correct exposure,

determining a quantization parameter for the first RAW data and a quantization parameter for the second RAW data for a corresponding region classified by brightness of the first RAW data if the first RAW data has a correct exposure, and

determining a quantization parameter for the first RAW data and a quantization parameter for the second RAW data for a corresponding region classified by brightness of the second RAW data if the second RAW data has a correct exposure.

16. The encoding device of claim 15, wherein the first exposure time is shorter than the second exposure time.

17. The encoding device according to claim 16, wherein the quantizing means determines a quantization parameter of an area classified as dark as a quantization parameter larger than a quantization parameter of an area classified as light for the first RAW data, and determines a quantization parameter of an area classified as light as a quantization parameter larger than a quantization parameter of an area classified as dark for the second RAW data.

18. The encoding apparatus as set forth in claim 15,

wherein, when capturing is performed by the image sensor with the same exposure time without performing capturing with a different exposure time for each pixel,

the generation section obtains third RAW data obtained by averaging pieces of pixel data of pixels of the same color existing in the vicinity,

the quantization section determines a quantization parameter for the third RAW data for a corresponding region classified by brightness of the third RAW data and quantizes the third RAW data, an

The encoding means encodes the quantized third RAW data.

19. The encoding apparatus as set forth in claim 18,

wherein the content of the first and second substances,

the quantization component determines: the quantization parameter for the first RAW data classified as a bright area is a quantization parameter corresponding to a quantization parameter for an area classified as bright in the third RAW data if the first RAW data has a correct exposure, and

the quantization component determines: the quantization parameter for the second RAW data classified as a dark area is a quantization parameter corresponding to a quantization parameter for an area classified as dark in the third RAW data if the second RAW data has a correct exposure.

20. The encoding device as set forth in claim 19,

wherein the content of the first and second substances,

the quantization component determines: the quantization parameter for the first RAW data classified as a bright area is a quantization parameter larger than a quantization parameter to be used in the third RAW data if the first RAW data has a correct exposure, an

The quantization component determines: the quantization parameter for the second RAW data classified as a dark area is a larger quantization parameter than the quantization parameter to be used in the third RAW data if the second RAW data has a correct exposure.

21. The encoding apparatus according to claim 18, wherein the quantization means determines: the quantization parameter for the second RAW data classified as a dark area is a quantization parameter that is greater than or equal to a quantization parameter to be used for an area classified as dark in the third RAW data and is less than a quantization parameter to be used for an area classified as light in the third RAW data if the first RAW data has a correct exposure.

22. The encoding apparatus according to claim 18, wherein the quantization means determines: the quantization parameter for the first RAW data classified as a bright area is a quantization parameter that is less than or equal to a quantization parameter to be used for an area classified as bright in the third RAW data and is greater than a quantization parameter to be used for an area classified as dark in the third RAW data if the second RAW data has a correct exposure.

23. The encoding device according to claim 14, wherein the quantizing means performs classification by brightness for each area of the first or second RAW data using a first threshold for determining whether it is a dark portion and a second threshold for determining whether it is a bright portion.

24. The encoding device according to claim 14, wherein the region is a square region of one or more pixels.

25. An image capture device comprising:

an image sensor capable of controlling an exposure time of each pixel; and

the encoding device of claim 1.

26. An encoding method, comprising:

generating a plurality of pieces of RAW data for respective exposure times from RAW data obtained from an image sensor capable of performing photographing with different exposure times for each pixel; and

encoding the plurality of RAW slices generated in the generating.

27. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the steps of the encoding method according to claim 26.

Technical Field

The present invention relates to a technique for encoding and recording an image obtained by an image sensor capable of controlling an exposure time of each pixel.

Background

In a known image capturing apparatus, RAW image information (RAW data) obtained by capturing performed by an image sensor is converted into signals composed of luminance and color differences by applying a de-bayer process (demosaic process), and a so-called development process such as noise removal, optical distortion correction, and image optimization is performed on each signal. Also, in general, the luminance signal and the color difference signal that have been subjected to the development processing are compression-encoded and recorded in a recording medium.

On the other hand, there is also an image capturing apparatus that stores image capturing data (RAW data) that is in a state immediately after output from an image sensor and has not been subjected to development processing in a recording medium. When recording the RAW data, data saving can be performed in a state where a large number of tones are maintained without deteriorating color information from the image sensor, and thus high-degree-of-freedom editing can be performed. However, there are problems in that the recording data amount of the RAW data is huge and a large amount of free space is required in the recording medium. Therefore, it is desirable that RAW data also undergo compression encoding and be recorded while suppressing the amount of data.

Incidentally, as an apparatus for obtaining a high dynamic range image, there is known an image capturing apparatus with which an image having a wide dynamic range can be obtained with one shot, as disclosed in japanese patent laid-open publication No.2013 and 21660, since pixels different in exposure time are arranged on the same plane. A synthesis method for generating a high dynamic range image at the time of development when using such an image capturing apparatus is disclosed in japanese patent laid-open No. 2013-21660.

However, in the known art disclosed in the above-mentioned japanese patent laid-open No.2013-21660, a method of encoding RAW data before undergoing synthesis is not disclosed.

In addition, when the image capturing apparatus as described in japanese patent laid-open No.2013-21660 is used, if encoding of RAW data before undergoing synthesis is attempted, since the level difference between pixels whose exposure times are different and which are arranged on the same plane is large, a large amount of high-frequency components are generated, and thus the encoding efficiency is lowered. Therefore, there is a problem that the data amount when recording the RAW data increases.

Disclosure of Invention

The present invention has been made in view of the above-described problems, and provides a technique for reducing the amount of data when RAW data in which pixel signals different in exposure time are mixed is encoded and recorded.

According to a first aspect of the present invention, there is provided an encoding apparatus comprising: a generation section for generating a plurality of pieces of RAW data (piece of RAW data) for respective exposure times from RAW data obtained from an image sensor capable of performing photographing with a different exposure time for each pixel; and an encoding means for encoding the plurality of RAW slices generated by the generation means.

According to a second aspect of the present invention, there is provided an image capturing apparatus comprising: an image sensor capable of controlling an exposure time of each pixel; and the encoding device described above.

According to a third aspect of the present invention, there is provided an encoding method comprising: generating a plurality of pieces of RAW data for respective exposure times from RAW data obtained from an image sensor capable of performing photographing with different exposure times for each pixel; and encoding the plurality of RAW slices generated in the generating.

According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program for causing a computer to execute the steps of the above-described encoding method.

Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

Drawings

Fig. 1 is a block diagram illustrating a functional configuration of a digital camera as a first embodiment of an encoding apparatus of the present invention.

Fig. 2 is a diagram illustrating a pixel array of an image capturing unit.

Fig. 3 is a diagram illustrating the pixel array of the image capturing unit and the setting of the exposure time.

Fig. 4A is a diagram illustrating a separation method of RAW data in the first embodiment.

Fig. 4B is a diagram illustrating a separation method of RAW data in the first embodiment.

Fig. 4C is a diagram illustrating a separation method of RAW data in the first embodiment.

Fig. 4D is a diagram illustrating a separation method of RAW data in the first embodiment.

Fig. 5 is a diagram illustrating RAW data output when the exposure time of a pixel is the same.

Fig. 6 is a block diagram illustrating a configuration of a RAW coding unit.

Fig. 7A and 7B are diagrams illustrating an example of frequency transform (sub-band division).

Fig. 8 is a diagram illustrating an example of a unit for generating a quantization parameter.

Fig. 9A is a diagram illustrating an exemplary generation of a quantization parameter.

Fig. 9B is a diagram illustrating an exemplary generation of a quantization parameter.

Fig. 9C is a diagram illustrating an exemplary generation of a quantization parameter.

Fig. 10A and 10B are diagrams illustrating a separation method of RAW data in the second embodiment.

Fig. 11 is a diagram illustrating a separation method of RAW data in the third embodiment.

Fig. 12 is a diagram illustrating the pixel array and the setting of exposure time in the fourth embodiment.

Fig. 13 is a diagram illustrating the rearrangement of the pixel array in the fourth embodiment.

Fig. 14 is a block diagram illustrating the configuration of a RAW encoding unit in the fifth embodiment.

Fig. 15A and 15B are diagrams illustrating frequency transform (subband division).

Fig. 16 is a process block diagram for describing the HDR composition process.

Fig. 17A to 17C are diagrams illustrating a combining ratio in HDR combining processing when a long-exposure image has a correct exposure.

Fig. 18A to 18C are diagrams illustrating a combining ratio in HDR combining processing when a short-exposure image has a correct exposure.

Fig. 19A to 19C are diagrams illustrating exemplary settings of quantization parameters.

Fig. 20A to 20C are flowcharts illustrating a quantization processing procedure of the fifth embodiment.

Fig. 21A to 21C are flowcharts illustrating a quantization processing procedure of the sixth embodiment.

Detailed Description

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following examples are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention is not limited to the invention requiring all of such features, and a plurality of such features may be combined as appropriate. Further, in the drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.

(first embodiment)

Fig. 1 is a block diagram illustrating a functional configuration of a digital camera 100 as a first embodiment of an encoding apparatus of the present invention. The digital camera 100 includes an image capturing unit 101, a separation unit 102, a RAW encoding unit 103, a recording processing unit 104, a recording medium 105, a memory I/F (memory interface) 106, and a memory 107.

The image capturing unit 101 includes a lens optical system including an optical lens, an aperture, a focus controller, and a lens driving unit and capable of optical zooming, and an image sensor in which a plurality of pixels each including a photoelectric conversion element are two-dimensionally arranged.

The image sensor performs photoelectric conversion on an object image formed by the lens optical system in each pixel, and also performs analog/digital conversion with an a/D conversion circuit and outputs digital signals (pixel data, RAW data) in units of pixels. A CCD image sensor, a CMOS image sensor, or the like is used as the image sensor.

Note that, in the present embodiment, each pixel of the image sensor is provided with one of R (red), G1/G2 (green), and B (blue) color filters, as shown in fig. 2. Note that the RAW data output from the image capturing unit 101 is stored in the memory 107 via the memory I/F106.

The separation unit 102 is a circuit or a module for separating the RAW data obtained by the image capturing unit 101 into pieces of RAW data for respective exposure times. The RAW data stored in the memory 107 is read out via the memory I/F106 and separated into RAW data pieces for respective exposure times, and the RAW data pieces are output to the RAW encoding unit 103.

The RAW encoding unit 103 is a circuit or a module that performs a calculation operation on RAW data and encodes the RAW data input from the separation unit 102. The RAW encoding unit 103 stores encoded data generated by encoding in the memory 107 via the memory I/F106.

The recording processing unit 104 reads out various types of data such as encoded data stored in the memory 107 via the memory I/F106, and records the read-out data in the recording medium 105. The recording medium 105 is a recording medium constituted by a large-capacity random access memory such as a nonvolatile memory.

The memory I/F106 mediates memory access requests from the processing units and performs read/write control for the memory 107. The memory 107 is a volatile memory such as an SDRAM, and functions as a storage section. The memory 107 provides a storage area for storing various types of data such as image data and sound data mentioned above or various types of data output from a processing unit constituting the digital camera 100.

Next, a pixel arrangement structure of the image capturing unit 101 will be described with reference to fig. 2. As shown in fig. 2, the image capturing unit 101 is characterized in that R pixels, G1 pixels, G2 pixels, and B pixels are arranged in units of 2 × 2 pixels, and the same color is arranged in each 2 × 2 pixel. The image capturing unit 101 has a structure in which 4 × 4 pixels in total are a minimum unit and the minimum unit is repeatedly arranged.

The setting of the exposure time in the image sensor having the pixel arrangement structure shown in fig. 2 and in which the exposure time can be controlled for each pixel (photographing is possible in the case where the exposure time is different for each pixel) will be described with reference to fig. 3. As shown in fig. 3, the horizontal direction is represented by x, the vertical direction by y, the column number by the x coordinate and the row number by the y coordinate. The numbers with parentheses indicate coordinates indicating the position of each pixel on the image sensor. In addition, white pixels represent short-exposure pixels, and gray pixels represent long-exposure pixels. In the present embodiment, the short-exposure pixels that perform short exposure and the long-exposure pixels that perform long exposure are arranged in a zigzag manner in the column direction, as shown in fig. 3.

For example, the setting regarding the exposure time of four R pixels at the upper left end in fig. 3 is as follows. R (1,1) is a short-exposure pixel, R (2,1) is a long-exposure pixel, R (1,2) is a long-exposure pixel, and R (2,2) is a short-exposure pixel. In this way, the short-exposure pixels and the long-exposure pixels are alternately arranged in each column, and the short-exposure pixels and the long-exposure pixels are alternately arranged in each row. When only the short-exposure pixels are followed in the y direction, in the first column and the second column, in the first row from above, the first column is the short-exposure pixels, in the second row, the second column is the short-exposure pixels, in the third row, the first column is the short-exposure pixels, and in the fourth row, the second column is the short-exposure pixels. Similarly, when only the long-exposure pixels are followed in the y direction, in the first column and the second column, the second column is the long-exposure pixels in the first row from above, the first column is the long-exposure pixels in the second row, the second column is the long-exposure pixels in the third row, and the first column is the long-exposure pixels in the fourth row.

As described above, the pixel arrangement structure and the setting of the exposure time are set such that the pixels of the same color are set in units of 2 × 2 pixels, and two short-exposure pixels (one of two exposure times) and two long-exposure pixels (the other of two exposure times) are arranged in these 4 pixels.

Here, if encoding is attempted to be performed in a state where RAW data is obtained by the image capturing unit 101 (i.e., in a state where pixels different in exposure time are mixed), a large amount of high-frequency components are generated because a level difference between pixels different in exposure time is large, and the amount of recorded data of the RAW data increases. Therefore, in the present embodiment, the RAW data is separated into pieces of RAW data for respective exposure times by the separation unit 102, and generation of high-frequency components is suppressed by matching the levels between pixels, whereby the recording data amount of the RAW data is reduced.

Next, the separation method will be described with reference to fig. 4A to 4D. As shown in fig. 4A to 4D, the separation unit 102 separates the RAW data input from the image capturing unit 101 into bayer arrangement RAW data composed of only short-exposure pixels and bayer arrangement RAW data composed of only long-exposure pixels, and outputs the two separated pieces of RAW data to the RAW encoding unit 103.

Specifically, the RAW data composed of only the short-exposure pixels is separated into two planes of the short-exposure RAW data illustrated by the RAW data 401a in fig. 4A and the RAW data 401B in fig. 4B. The RAW data 401a is short-exposure RAW data configured by extracting short-exposure pixels each marked with a diamond shape in odd rows and odd columns, as shown in fig. 4A. In addition, the RAW data 401B is short-exposure RAW data configured by extracting short-exposure pixels each marked with a diamond shape in even-numbered rows and even-numbered columns, as shown in fig. 4B.

Similarly, the RAW data composed of only the long-exposure pixels is separated into two planes of the long-exposure RAW data illustrated by the RAW data 401C in fig. 4C and the RAW data 401D in fig. 4D. The RAW data 401C is long-exposure RAW data configured by extracting long-exposure pixels each marked with a diamond shape in odd-numbered rows and even-numbered columns, as shown in fig. 4C. In addition, the RAW data 401D is long-exposure RAW data configured by extracting long-exposure pixels each marked with a diamond shape in even rows and odd columns, as shown in fig. 4D. The RAW encoding unit 103 encodes RAW data 401a, 401b, 401c, and 401d input from the separation unit 102 in a bayer arrangement, respectively.

Note that the separation method of the separation unit 102 when the exposure time is different between pixels arranged on the same plane has been described above using the pixel array in fig. 2. Next, processing to be performed by the separation unit 102 when the exposure times of the pixels are all the same will be described with reference to fig. 5.

In this case, for the RAW data obtained by the image capturing unit 101, the separation unit 102 configures the RAW data 501 by calculating the pixel average value of every four pixels of the same color component marked by a gray diamond (as shown in fig. 5), and outputs the RAW data 501 to the RAW encoding unit 103. Specifically, as shown in the following equations 1 to 4, the separation is performed by calculating the additive average of each color component.

Next, a detailed configuration of the RAW encoding unit 103 that performs processing on the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d and a processing flow will be described with reference to a block diagram shown in fig. 6.

The RAW encoding unit 103 includes a channel transform unit 601, a frequency transform unit 602, a quantization parameter generation unit 603, a quantization unit 604, an encoding unit 605, and a quantization parameter encoding unit 606.

The channel conversion unit 601 converts RAW data in a bayer arrangement configuration input from the separation unit 102 into a plurality of channels. For example, the conversion to four channels is performed separately for R, G1, G2, and B in a bayer arrangement. Alternatively, for R, G1, G2, and B, the transformation into four channels is performed by further performing calculations using the following transformation equations 5 to 8.

Y ═ R + G1+ G2+ B)/4 formula 5

C0 ═ R-B equation 6

C1 ═ G0+ G1)/2 ═ R + B)/2 formula 7

C2-G0-G1 formula 8

Note that an exemplary configuration for conversion to four channels is shown here, but the number of channels and the conversion method are not limited thereto.

The frequency transform unit 602 performs a frequency transform process by discrete wavelet transform at a predetermined resolution level (hereinafter, denoted as "lev") for each channel, and outputs the generated sub-band data (transform coefficients) to the quantization parameter generation unit 603 and the quantization unit 604.

Fig. 7A shows a filter bank configuration for discrete wavelet transform that implements the subband splitting process with respect to lev-1. When the discrete wavelet transform process is performed in the horizontal direction and the vertical direction, segmentation into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH) is performed as shown in fig. 7B.

Transfer functions of the low-pass filter (hereinafter, denoted as "lpf") and the high-pass filter (hereinafter, denoted as "hpf") illustrated in fig. 7A are shown in equation 9 and equation 10, respectively.

lpf(z)＝(-z^-2+2z^-1+6+2z¹-z²) Equation 8 equation 9

hpf(z)＝(-z^-1+2-z¹) Equation 210

When lev is greater than 1, subband segmentation is performed hierarchically for the low frequency subband (LL). Note that here, as shown in equation 9 and equation 10, the discrete wavelet transform is configured by five-tap lpf and three-tap hpf, but is not limited thereto, and a filter configuration different in the number of taps and coefficients may be employed.

The quantization parameter generation unit 603 generates a quantization parameter for performing quantization processing on the sub-band data (transform coefficient) generated by the frequency transform unit 602 for each specific predetermined sub-band data unit. The generated quantization parameter is input to the quantization parameter encoding unit 606 and is also supplied to the quantization unit 604.

The quantization unit 604 performs quantization processing on the sub-band data (transform coefficient) output from the frequency transform unit 602 based on the quantization parameter supplied from the quantization parameter generation unit 603, and outputs the quantized sub-band data (transform coefficient) to the encoding unit 605.

The encoding unit 605 performs predictive differential entropy encoding of the quantized sub-band data (transform coefficients) output from the quantization unit 604 for each sub-band in raster scan order, and stores the generated encoded RAW data to the memory 107. Note that other methods may be used as the prediction method and the entropy coding method.

The quantization parameter encoding unit 606 is a processing unit for performing encoding on the quantization parameter input from the quantization parameter generation unit 603. The quantization parameter is encoded using an encoding method common to the encoding unit 605, and the generated encoded quantization parameter is stored to the memory 107.

Next, the relationship between the subband data, the channel data, and the RAW data when the quantization parameter is generated assuming that the above-mentioned predetermined subband unit is 4 × 4 will be described with reference to fig. 8.

As shown in fig. 8, the 4 × 4 sub-band corresponds to 8 × 8 pixels of each channel, and also corresponds to a block corresponding to 16 × 16 pixels of each RAW data. Therefore, in this case, in the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d, the quantization parameter needs to be stored in the memory 107 for each RAW data block corresponding to 16 × 16 pixels.

Note that it is effective to apply the same quantization parameter to the short-exposure RAW data 401a and 401b and to the long-exposure RAW data 401c and 401d in order to reduce the data amount of the quantization parameter. In this case, the data amount can be reduced to half. In addition, in the present embodiment, a quantization parameter generated with an exposure time closer to the correct exposure is used as a reference, and other quantization parameters are calculated so as to further reduce the data amount. Thereby, the data amount of the quantization parameter can be reduced to one fourth. Here, the reason why the quantization parameter generated with an exposure time closer to the correct exposure is used as a reference is because the quantization parameter cannot be generated according to the accurate characteristics of the subject in the case where an overexposed (overexposed) or underexposed (underexposed) image of overexposed highlight (brown out high) or blackened (blacked out) occurs.

When the short exposure is closer to the correct exposure, as a specific example, a calculation formula for calculating a quantization parameter for the long-exposure RAW data with reference to a quantization parameter generated with respect to the short-exposure RAW data is shown in formula 11.

Formula 11 where L _ Qp is α × S _ Qp + β

Here, the first and second liquid crystal display panels are,

l _ Qp: quantization parameters for long exposure RAW data

S _ Qp: quantization parameters for short exposure RAW data

α: slope of

Beta: and (4) intercept.

Note that, in the present embodiment, the quantization parameter for the long-exposure RAW data is calculated with reference to the quantization parameter generated for the short-exposure RAW data. However, the quantization parameter for the short-exposure RAW data may be calculated with reference to the quantization parameter generated for the long-exposure RAW data. In addition, the quantization parameter may be calculated by setting α and β for each of the short exposure and the long exposure while not using both the short exposure and the long exposure as a reference.

Next, a determination method of α and β shown in formula 11 will be described. Although α and β may be any values, in the present embodiment, a detailed parameter determination method will be described. When it is assumed that the short exposure is closer to the correct exposure, as in the above-described example, in the long exposure, since the exposure time is longer than the short exposure, the overexposure is achieved. Therefore, regarding a region where the luminance is medium to bright at the time of short exposure, the possibility that the pixel value reaches the saturation level and the pixel value according to the subject luminance cannot be output at the time of long exposure is high. On the other hand, with regard to the dark area, it is possible to obtain detailed information with respect to the short exposure. Therefore, regarding the region determined as the region having the middle to bright luminance in the short-exposure RAW data, the quantization parameter for the long-exposure RAW data is increased with respect to the short-exposure. In addition, the same parameters are set with respect to the area determined to be dark, and therefore, the data amount of the quantization parameter can be reduced while ensuring the image quality.

A detailed description will be given with reference to fig. 9A to 9C. Fig. 9A shows one exemplary setting of a quantization parameter according to the luminance of the short-exposure RAW data in the short-exposure RAW data. In addition, fig. 9B shows an exemplary setting of a quantization parameter according to the luminance of the short-exposure RAW data in the long-exposure RAW data. Note that the luminance index may be evaluated using the 1LL subband corresponding to the quantization parameter generation unit described above. The magnitude relationship between the quantization parameters is shown in equations 12 to 14.

Q0< Q1< Q2 formula 12

Q1< Q3 formula 13

Q2< Q4 formula 14

First, considering the visual characteristics (Q0< Q1< Q2), the quantization parameter in the short-exposure RAW data is set so that the quantization parameter decreases as the darkness increases. In contrast, in the long-exposure RAW data, the quantization parameter is set so that Q0 is set to be the same as that in the short-exposure RAW data in the region corresponding to the dark portion in the short-exposure RAW data, and the quantization parameter is set to be increased relative to the short-exposure RAW data in the region corresponding to the medium to bright portion (Q1< Q3, Q2< Q4).

Fig. 9C shows a graph for calculating a quantization parameter for long-exposure RAW data with reference to a quantization parameter generated for short-exposure RAW data. The horizontal axis represents the quantization parameter (S _ Qp) for the short-exposure RAW data, and the vertical axis represents the quantization parameter (L _ Qp) for the long-exposure RAW data. α and β shown in equation 11 may be set to implement the relationship of equation 12 to equation 14.

Note that α and β are stored in the memory 107 similarly to the encoded data, and are recorded in the recording medium 105 via the memory I/F106 together with the encoded data. In addition, a flag indicating which of the short exposure and the long exposure has a quantization parameter to be a reference is stored in the memory 107, and is recorded in the recording medium 105 via the memory I/F106 together with the encoded data. Note that when α and β are set for each exposure time while not using both short exposure and long exposure as a reference, the flag may not be included.

In addition, when any of the above cases is handled, the configuration may be such that a flag indicating whether the exposure time is to be referred to is included, and next, if there is an exposure time to be referred to, a flag indicating which of the short exposure and the long exposure is to be referred to is included. In this case as well, each flag information is stored in the memory 107 and recorded in the recording medium 105 via the memory I/F106 together with the encoded data.

As described above, in the present embodiment, the separation unit 102 separates the RAW data into pieces of data for respective exposure times, the level difference between pixels to be encoded is eliminated, and thereby, high frequency components are suppressed, and therefore, the amount of recording data of the RAW data can be reduced. In addition, the quantization parameter for other RAW data of different exposure times is determined using the quantization parameter calculated for one type of RAW data as a reference, and therefore, the recording data amount of the RAW data can be reduced.

(second embodiment)

Next, a second embodiment of the present invention will be described. In the second embodiment, the separation method of the RAW data in the separation unit 102 is different from that of the first embodiment. Note that the configuration of the digital camera of the second embodiment is the same as that of the digital camera of the first embodiment, and therefore redundant description will be omitted, and differences will be described.

In the first embodiment, a piece of RAW data obtained by separating pixels into groups of pixels of the same exposure time in the separation unit 102 (i.e., specifically, into two planes of RAW data constituted only by short-exposure pixels and two planes of RAW data constituted only by long-exposure pixels) is output to the RAW encoding unit 103.

In contrast, in the second embodiment, the following method will be described: to further reduce the data amount, in the separation unit 102, pixel values of pixels of the same exposure time and the same color component existing in the vicinity are added, and an average pixel value is calculated and output to the RAW encoding unit 103.

The process of the separation unit 102 in the present embodiment will be described with reference to fig. 10A and 10B. For the RAW data of the pixels that are input from the image capturing unit 101 and in which different exposure times are mixed, the separation unit 102 calculates the addition average of the pixel values of the pixels surrounded by each rectangle shown in fig. 10A (i.e., the pixels that are short-exposure pixels and have the same color component), and separates into the short-exposure RAW data 1001 a. Specifically, as shown in equations 15 to 18 below, the separation is performed by calculating an addition average for each color component.

.

.

Similarly, the long-exposure RAW data 1001B is separated by calculating the addition ties of pixels surrounded by each rectangle shown in fig. 10B and being long-exposure pixels and having the same color component. Specifically, as shown in equations 19 to 22 below, the separation is performed by calculating an addition average for each color component.

.

.

As described above, in the second embodiment, the RAW data obtained by the image capturing unit 101 is separated by calculating the addition average in the separation unit 102, and therefore, the amount of data to be output to the RAW encoding unit 103 can be reduced to half with respect to the first embodiment.

(third embodiment)

Next, a third embodiment of the present invention will be described. In the third embodiment, the method of separating the RAW data in the separation unit 102 is different from the method of separating the RAW data of the first and second embodiments. Note that the configuration of the digital camera of the present embodiment is the same as that of the digital cameras of the first and second embodiments, and therefore redundant description will be omitted, and differences will be described.

In the second embodiment, the addition average of pixel values of pixels of the same exposure time and the same color component existing in the vicinity is calculated in the separation unit 102, and the addition average is output to the RAW encoding unit 103. In the third embodiment, in order to further reduce the data amount with respect to the second embodiment, a gain is applied to the RAW data of one exposure time from the RAW data of the other exposure time, and the difference between them is output to the RAW encoding unit 103. That is, the RAW encoding unit 103 encodes the RAW data of the addition average for one exposure time and encodes the RAW data of the difference (differential RAW data) for the other exposure time.

Processing in the separation unit 102 in the present embodiment will be described with reference to fig. 11. First, the separation unit 102 adds pixel values of pixels of the same exposure time and the same color component existing nearby, and obtains RAW data 1001a and 1001B by calculating the average thereof similarly to the second embodiment as shown in fig. 10A and 10B. Next, a difference between the first line in the long-exposure RAW data 1001b and a value obtained by multiplying the first line of the short-exposure RAW data 1001a by a gain γ corresponding to the long-exposure RAW data plus an offset ∈ is obtained. Here, the gain γ and the offset ∈ may be determined by performing calculation in reverse from the exposure time in advance, or may be determined using histograms of pixel values of the obtained short-exposure pixels and long-exposure pixels.

Specifically, as shown in the following equations 23 to 26, the difference in the first row is calculated.

This operation is similarly performed for the second, third, and fourth rows in addition to the first row, and the calculated difference value is output to the RAW encoding unit 103. Note that, in the present embodiment, correction is performed for short-exposure RAW data, but correction may be performed for long-exposure RAW data. However, from the viewpoint of rounding processing, poor accuracy is better when the gain γ is applied to the short-exposure RAW data.

As described above, in the third embodiment, instead of outputting the RAW data as it is to the RAW encoding unit 103, the RAW data is output as a difference value, and therefore, the recording data amount of the RAW data can be further reduced with respect to the second embodiment.

(fourth embodiment)

Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, a pixel array different from those of the first to third embodiments (i.e., specifically, the pixel array shown in fig. 12) is applied to the image capturing unit 101.

In the first to third embodiments described above, the image capturing unit 101 having the following structure has been described: wherein the minimum unit includes 4 × 416 pixels composed of four different pixels R, G1, G2, and B, and the minimum unit is repeatedly arranged as shown in fig. 2.

In contrast, fig. 12 shows the pixel array of the image capturing unit 101 and the setting of the exposure time in the fourth embodiment. The horizontal direction is represented by x, the vertical direction by y, the column number by the x coordinate and the row number by the y coordinate. The numbers with parentheses indicate coordinates indicating the position of each pixel on the image sensor. In addition, white pixels represent short-exposure pixels, and gray pixels represent long-exposure pixels. In this way, in fig. 12, in the pixel array of the bayer arrangement constituted by the array of R, G1, G2, and B pixels, short-exposure pixels and long-exposure pixels are alternately arranged in units of two columns.

Also, in the pixel array and exposure time setting in fig. 12, as shown in fig. 13, as a result of performing the processing while performing rearrangement on the pixel arrangement structure shown in fig. 2, the processing described in the first to third embodiments may be performed.

As described above, in the fourth embodiment, even if the pixel array is changed, the similar processing to that described in the first to third embodiments can be performed.

(fifth embodiment)

Next, a detailed configuration and a process flow of the RAW encoding unit 103 that performs an encoding process on the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d in the fifth embodiment will be described with reference to a block diagram shown in fig. 14. Note that the configuration shown in fig. 1 to 5 is similar to that of the first embodiment.

The RAW encoding unit 103 mainly includes a channel transforming unit 1601, a frequency transforming unit 1602, a quantization parameter generating unit 1603, a quantizing unit 1604, and an encoding unit 1605.

The channel conversion unit 1601 converts the RAW data in the bayer arrangement configuration input from the separation unit 102 to a plurality of channels. Here, the conversion to four channels is performed separately for R, G1, G2, and B in the bayer arrangement.

The frequency transform unit 1602 performs a frequency transform process by discrete wavelet transform at a predetermined resolution level (hereinafter, denoted as "lev") for each channel, and outputs generated sub-band data (transform coefficients) to a quantization parameter generation unit 1603 and a quantization unit 1604.

Fig. 15A shows a filter bank configuration for discrete wavelet transform that implements the subband splitting process with respect to lev-1. When the discrete wavelet transform process is performed in the horizontal direction and the vertical direction, segmentation into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH) is performed as shown in fig. 15B.

Transfer functions of the low-pass filter (hereinafter, denoted as "lpf") and the high-pass filter (hereinafter, denoted as "hpf") illustrated in fig. 15A are shown in equation 27 and equation 28, respectively.

lpf(Z)＝(-Z^-2+2Z^-1+6+2Z¹-Z²) Equation 8 formula 27

hpf(Z)＝(-Z^-1+2-Z¹) Equation 28

When lev is greater than 1, subband segmentation is performed hierarchically for the low frequency subband (LL). Note that here, as shown in equation 27 and equation 28, the discrete wavelet transform is configured by five-tap lpf and three-tap hpf, but is not limited thereto, and a filter configuration different in the number of taps and coefficients may be employed.

Quantization parameter generating section 1603 calculates a luminance characteristic amount for each predetermined coefficient (square block of one or more coefficients, square region of one or more pixels) of the sub-band data (transform coefficient) generated by frequency transforming section 1602, and generates a quantization parameter based on the characteristic amount. Similarly, quantization is performed in units of predetermined coefficients (square blocks of one or more coefficients), but is desirably the same as the unit of calculating the feature amount in consideration of controllability of image quality. Subsequently, a method of setting a quantization parameter according to luminance and a flow of generating the quantization parameter will be described in detail. Then, the generated quantization parameter is output to quantization unit 1604.

Quantization unit 1604 performs quantization processing on the sub-band data (transform coefficients) input from frequency transform unit 1602 using the quantization parameter supplied from quantization parameter generation unit 1603, and outputs the quantized sub-band data (transform coefficients) to encoding unit 1605.

The encoding unit 1605 performs predictive differential entropy encoding of the quantized sub-band data (transform coefficients) input from the quantization unit 1604 for each sub-band in raster scan order, and stores the generated encoded RAW data to the memory 107. Note that other methods of prediction and entropy coding may be used.

Here, an HDR (high dynamic range) composition processing method will be described using fig. 16. Fig. 16 is a block diagram of a process for performing HDR compositing. The digital camera 100 is configured to record two RAW data tables (sheet of RAW data) different in exposure amount, and thus a description is given assuming that HDR composition processing in the present embodiment performs HDR composition on the two RAW data tables. Note that one of the exposure RAW data tables is RAW data obtained by capturing performed with correct exposure. Another RAW data table is RAW data obtained with an exposure time that causes overexposure or underexposure, the RAW data being auxiliary data of DR extension.

The development processing unit 801 performs development processing on the long-exposure RAW data. Then, the generated developed long-exposure image is output to the gain correction unit 803. The development processing unit 802 performs development processing on the short-exposure RAW data. Then, the generated developed short-exposure image is output to the gain correction unit 804.

The gain correction unit 803 performs gain correction on the long-exposure image using a gain value based on a predetermined combination ratio. Subsequently, the synthesis ratio will be described. The gain correction unit 804 performs gain correction on the short-exposure image using a gain value based on a predetermined combination ratio. Subsequently, the synthesis ratio will be described. The addition processing unit 805 performs addition processing of pixels at the same coordinate position for the long-exposure image and the short-exposure image.

In this way, in the HDR combining processing, the gain correction processing and the addition processing are performed on the image generated by performing the development processing on the two RAW data tables different in exposure amount. Note that this HDR combining process is similarly performed on each color component (R, G, B) constituting the image data. In addition, the development processing includes a de-bayer processing, a luminance-color difference conversion processing, a noise removal processing, an optical distortion correction processing, and the like.

Next, the composition ratio between the short-exposure image data and the long-exposure image data will be described. The idea of the composition ratio differs based on which exposure image data has correctly exposed image data. The case in which the long-exposure image data has the correct exposure and the case in which the short-exposure image data has the correct exposure will be described separately.

First, the composition ratio in the case where the long-exposure image data has a correct exposure will be described. When long-exposure image data is obtained by capturing performed with correct exposure, the exposure time of the short-exposure image data is relatively shorter than that of the long-exposure image data, and thus the short-exposure image data has underexposure.

An example of a histogram of image data when capturing is performed under this exposure condition is shown in fig. 17A. The histogram shown in fig. 17A is a histogram of a specific color component constituting image data. The horizontal axis of the histogram represents a pixel value indicating the luminance of the image data, and the vertical axis represents the number of pixels. In addition, Ta and Tb denote pixel threshold values, and Tc denotes a pixel upper limit value. The definition is as follows: a region satisfying the condition of pixel value ≦ Ta is referred to as a dark portion, a region satisfying the condition of Ta < pixel value ≦ Tb is referred to as a medium portion, and a region satisfying the condition of Tb < pixel value is referred to as a bright portion. In this histogram, the long-exposure image data correctly expresses the color tones in the dark portion region and the middle portion region, but in the bright portion region, there are many pixels in a region above Tc, which is the upper pixel limit, and therefore the long-exposure image data is in a state in which the color tone information is lost due to the occurrence of overexposed highlight. In the HDR combining process, short-exposure image data at the same coordinate position is combined in order to expand the tone range in which overexposed highlight appears. In the HDR combining process under this exposure condition, the addition process is performed by performing gain correction so that the combining ratio of the long-exposure image data is large in the dark portion region and the middle portion region where DR can be ensured at the time of correct exposure, and the combining ratio of the short-exposure image data is increased in the light portion region where it is difficult to ensure DR at the time of correct exposure.

An example of the composition ratio is shown in fig. 17B. The horizontal axis represents the pixel value of the long-exposure image data (correct exposure), and the vertical axis represents the composition ratio. The graph in fig. 17B represents the composition ratio of the pieces of exposure image data according to the pixel values, and the composition ratio of the pieces of exposure image data is changed so that the sum thereof is constantly 100%. As described in fig. 17A, since the bright portion includes many pixels where the overexposed highlight occurs, in the graph in fig. 17B, the synthesis ratio of the long-exposure image data is decreased from the pixel value at the threshold value Tb to 0% at the pixel upper limit value Tc, and the synthesis ratio of the short-exposure image data is increased from the pixel value at the threshold value Tb to 100% at the pixel upper limit value Tc. With such a combination ratio, it is possible to expand the DR in the combined image while reducing the influence of overexposed highlights. Note that, in order to make the description easier to understand, an example has been described in which the combination ratio is changed with the threshold Tb as a boundary, but the combination ratio of the pieces of exposure image data is not limited to this.

Based on the above description, the magnitude relation between the combination ratio of the long-exposure image data and the short-exposure image data is shown in fig. 17C. A0 in the figure indicates a composition ratio in a dark portion of the long-exposure pixel, a1 indicates a composition ratio in a middle portion of the long-exposure pixel, and a2 indicates a composition ratio in a light portion of the long-exposure pixel. In addition, A3 in the figure indicates a composition ratio in a dark portion of the short-exposure pixel, a4 indicates a composition ratio in a middle portion of the short-exposure pixel, and a5 indicates a composition ratio in a light portion of the short-exposure pixel. The size relationship between the composition ratios for the respective luminance regions is a0> A3 in the dark portion, a1> a4 in the middle portion, and a2< a5 in the bright portion.

Next, the composition ratio in the case where the short-exposure image data has a correct exposure will be described. When the short-exposure image data is obtained by the capturing performed with the correct exposure, the exposure time of the long-exposure image data is relatively longer than that of the short-exposure image data, and thus the long-exposure image data is overexposed.

An example of a histogram of image data when capturing is performed under this exposure condition is shown in fig. 18A. The histogram shown in fig. 18A is a histogram of a specific color component constituting image data. The horizontal axis of the histogram represents a pixel value indicating the luminance of the image data, and the vertical axis represents the number of pixels. In addition, Ta and Tb denote pixel thresholds, and Td denotes a pixel lower limit value. The definition is as follows: a region satisfying the condition of pixel value ≦ Ta is referred to as a dark portion, a region satisfying the condition of Ta < pixel value ≦ Tb is referred to as a medium portion, and a region satisfying the condition of Tb < pixel value is referred to as a bright portion. In this histogram, the short-exposure image data correctly expresses the hue in the middle-and light-portion regions, but in the dark-portion region, many pixels exist in a region below Td, Td being a pixel lower limit, and thus the short-exposure image data is in a state in which hue information is lost due to the occurrence of a blocked shadow (blocked up shadow). In the HDR combining process, in order to expand the tone range in which the occluding shade occurs, long-exposure image data at the same coordinate position is combined. In the HDR combining process under this exposure condition, the addition process is performed by performing gain correction so that the combining ratio of the short-exposure image data is large in the medium portion region and the light portion region where DR can be ensured at the time of correct exposure, and the combining ratio of the long-exposure image data is increased in the dark portion region where it is difficult to ensure DR at the time of correct exposure.

Next, an example of the combination ratio is shown in fig. 18B. The horizontal axis represents the pixel value of the short-exposure image data (correct exposure), and the vertical axis represents the composition ratio. The graph in fig. 18B represents the combination ratio of the pieces of exposure image data according to the pixel values, and the combination ratio of the pieces of exposure image data is changed so that the sum thereof is constantly 100%. As described in fig. 18A, since the dark portion includes many pixels where the shading occurs, in the graph in fig. 18B, the composition ratio of the long-exposure image data is changed to 100% at the pixel lower limit value Td, and the composition ratio of the short-exposure image data is changed to 0% at the pixel lower limit value Td. With such a composition ratio used, it is possible to expand DR in the synthesized image while reducing the influence of occlusion shading. Note that, in order to make the description easier to understand, an example has been described in which the combination ratio is changed with the threshold Ta as a boundary, but the combination ratio of the pieces of exposure image data is not limited to this.

Based on the above description, the magnitude relation between the combination ratio of the long-exposure image data and the short-exposure image data is shown in fig. 18C. B0 in the figure represents the synthesis ratio in the dark portion of the short-exposure pixel, B1 represents the synthesis ratio in the middle portion of the short-exposure pixel, and B2 represents the synthesis ratio in the light portion of the short-exposure pixel. In addition, B3 in the figure indicates a composition ratio in a dark portion of the long-exposure pixel, B4 indicates a composition ratio in a middle portion of the long-exposure pixel, and B5 indicates a composition ratio in a light portion of the long-exposure pixel. The size relationship between the composition ratios for the respective luminance regions is B0< B3 in the dark portion, B1> B4 in the middle portion, and B2> B5 in the light portion.

As described above, in the HDR composition processing, the composition ratio of the pieces of exposed image data is changed depending on whether or not there is a correct exposure and the size of the pixel value (luminance). The size of the composition ratio indicates the degree of influence on the image quality, and in a region where the composition ratio is large, the influence on the image quality is large, and in a region where the composition ratio is small, the influence on the image quality is small. Therefore, for RAW data to be compressed for recording, it is necessary to optimally distribute the code amount according to the degree of influence on the image quality based on the synthesis ratio in the HDR synthesis processing. That is, it is important to set the quantizing parameter so that the image quality is ensured by assigning a larger amount of code to a region where the synthesis ratio is larger, and the amount of code is reduced with respect to a region where the synthesis ratio is small and the influence on the image quality is small.

Next, a basic idea in quantization parameter generation performed by the quantization parameter generation unit 1603 will be described. As described above, it is assumed that weighting of the quantization parameter is performed in accordance with the synthesis ratio obtained by the intended HDR synthesis process. To which an idea of weighting the quantization parameter according to luminance in consideration of the visual characteristics of the image is added.

In post-processing after development, RAW data is subjected to adjustment of luminance levels such as gamma (gamma) correction processing and tone curve correction processing. When a dark portion where the original luminance level is small is compared with a bright portion where the original luminance level is large, the change ratio of the pixel value in the dark portion is large even if adjustment is performed for the same luminance level. If the quantization process is performed with the same quantization parameter for the dark portion and the bright portion, the variation ratio of the pixel value in the dark portion is large, and thus the quantization error due to the quantization process is amplified, and the image quality deterioration becomes noticeable. On the other hand, in a bright portion where the variation ratio of the luminance level is small, the variation ratio of the pixel value is also small, and therefore the degree of amplification of the quantization error is small, and the image quality deterioration is not significant. In order to secure image quality after post-processing, it is necessary to perform quantization of RAW data in consideration of a quantization error amplified by the post-processing. In addition, in the dark portion, the contrast is small relative to that in the light portion, and the signal level of the sub-band data is small. Therefore, if coarse quantization is performed with respect to a dark portion, the sub-band data after quantization is likely to be 0. Once the coefficient becomes 0, the signal cannot be restored in the inverse quantization process, and significant image quality deterioration occurs.

For these reasons, control is performed such that the quantization parameter is decreased in a dark portion region where image quality degradation is likely to be significant, and the quantization parameter is increased in a bright portion region where image quality degradation is unlikely to be significant. In the present embodiment, the following configuration will be described: wherein quantization tables in which quantization parameters for respective subbands are compiled are prepared in advance, and the quantization tables to be referred to are switched according to a synthesis ratio and a luminance feature amount. These quantization tables are constructed from quantization parameters for the corresponding sub-band data slices according to lev. The quantization parameter for each sub-band is set so that the quantization parameter is smaller in lower sub-bands where image quality degradation is likely to be significant. If lev is 1, the size relationship between quantization parameters of the corresponding subbands is 1LL <1 HL-1 LH <1 HH.

Based on the idea of weighting the quantization parameters according to luminance, exemplary settings of the quantization tables for the RAW data pieces obtained by capturing at the respective exposure times will be described with respect to the following three conditions, respectively. Note that, in the present embodiment, an example will be described in which the luminance feature amount is classified into three feature regions of a dark portion, a middle portion, and a bright portion. Note that the definition of the features to be classified is similar to those in the histograms in fig. 17 and fig. 18.

[ Exposure time is the same between short-exposure RAW data and long-exposure RAW data ]

Under this condition, one RAW data piece is generated by calculating the pixel average of every adjacent four pixels of the same color component (see fig. 5). One piece of RAW data is to be quantized, and since HDR synthesis processing will not be performed, luminance feature classification is performed using RAW data generated by calculating pixel averages, and quantization is performed using a quantization table according to the classification result. An exemplary setting of the quantization table is shown in fig. 19A. Q0 denotes a quantization table for guaranteeing image quality in the dark portion, Q1 denotes a quantization table for guaranteeing image quality in the middle portion, and Q2 denotes a quantization table for guaranteeing image quality in the bright portion. The size relationship between quantization tables is as follows.

Q0<Q1<Q2

In this way, a quantization table according to brightness based on visual characteristics is set.

[ when the exposure time is different between the short-exposure RAW data and the long-exposure RAW data and the short-exposure RAW data has a correct exposure ]

Under such conditions, the image data is separated into short-exposure RAW data and long-exposure RAW data (see fig. 4A to 4D). An exemplary setting of the quantization table is shown in fig. 19B. Since a blocking shadow is likely to occur in the short-exposure RAW data obtained by the capturing performed with the correct exposure, the DR in the dark portion is extended using the long-exposure RAW data obtained by the capturing performed under the condition of the overexposure. Quantization tables indicated by Q1 and Q2 in the figure are similar to those in fig. 19A. Here, two quantization tables indicated by Q3 and Q4 are newly added. Q3 indicates a table aimed at suppressing the generated code amount on the assumption that the region is a region in which the synthesis ratio in the HDR synthesis processing is small and the influence on the image quality is small. Q4 indicates a quantization table aimed at allocating a large amount of code in order to expand DR in a dark portion where occlusion shading is likely to occur in the HDR combining process. The size relationship between quantization tables is as follows.

Q0≤Q4<Q1<Q2<Q3

Alternatively, the first and second electrodes may be,

Q0<Q4≤Q1<Q2<Q3

the quantization parameter in Q4 is greater than or equal to the quantization parameter in Q0 and less than the quantization parameter in Q2. In this way, it becomes possible to ensure image quality after HDR combining processing by setting a quantization table in which the quantization parameter is relatively small for a dark portion in long-exposure RAW data for which the combining ratio is large, in addition to a quantization table according to luminance based on visual characteristics. On the other hand, since the quantization table in which the quantization parameter is large is set for the dark portion in the short-exposure RAW data for which the synthesis ratio is small and for the medium portion and the bright portion in the long-exposure RAW data, the data amount can be effectively reduced without degrading the image quality after the HDR synthesis processing.

[ when the exposure time is different between the short-exposure RAW data and the long-exposure RAW data has a correct exposure ]

Also under such conditions, the image data is separated into RAW data composed of short-exposure pixels and RAW data composed of long-exposure pixels (refer to fig. 4A to 4D). An exemplary setting of the quantization table is shown in fig. 19C. As described above, it is possible that overexposed highlight occurs in long-exposed pixels for capturing with correct exposure, and DR in a bright portion is expanded using short-exposed pixels for capturing with underexposure. Quantization tables indicated by Q0, Q1, and Q3 are similar to those in fig. 19A and 19B. Here, a quantization table indicated by Q5 is newly added. Q5 indicates a quantization table aimed at allocating a large amount of code to expand DR in a bright portion where overexposed highlight is likely to occur in the HDR combining process. The size relationship between quantization tables is as follows.

Q0<Q1≤Q5<Q2<Q3

Alternatively, the first and second electrodes may be,

Q0<Q1<Q5≤Q2<Q3

the quantization parameter in Q5 is less than or equal to the quantization parameter in Q2 and greater than the quantization parameter in Q0. In this way, it becomes possible to ensure image quality after HDR synthesis processing by setting a quantization table in which the quantization parameter is relatively small with respect to a bright portion in which the synthesis ratio is large in short-exposure RAW data, in addition to a quantization table according to luminance based on visual characteristics. On the other hand, since the quantization table in which the quantization parameter is large is set with respect to the light portion in the long-exposure RAW data in which the combination ratio is small and the dark portion and the middle portion in the short-exposure RAW data, the data amount can be effectively reduced without degrading the image quality after the HDR combination processing.

Next, the quantization processing procedure will be described using the flowcharts shown in fig. 20A to 20C. In the present embodiment, in order to make the description easier to understand, let 1 is assumed, and the luminance feature amount is calculated using subband data constituting RAW data obtained by capturing at an exposure time that will be correct exposure.

It is assumed that the calculation and quantization processing of the luminance feature quantity are performed in units of one coefficient, and an operation is performed to uniquely determine the quantization table to be applied to the respective pieces of RAW data for different exposure times in accordance with the luminance feature quantity of the respective coefficient (see fig. 19 for details).

In the present embodiment, an operation mode in which capturing is performed while changing the exposure time for each pixel is referred to as an HDR mode, and an operation mode in which capturing is performed without changing the exposure time is referred to as a normal mode. As described above, in the HDR mode, the horizontal size and the vertical size of the RAW data to be recorded are doubled with respect to the normal mode (refer to fig. 4A to 5), and thus the amount of data to be subjected to quantization processing is different between these modes.

In step S1201, the controller 108 determines whether the operation mode of the digital camera 100 is the HDR mode. If it is determined to be the HDR mode, the process proceeds to step S1202, and if not, the process proceeds to step S1219.

In step S1202, the controller 108 determines whether the short-exposure RAW data has a correct exposure. If the short-exposure RAW data has correct exposure, the process proceeds to step S1203, and if not, the process proceeds to step S1211.

In step S1203, the controller 108 calculates a luminance feature amount using the short-exposure subband data with correct exposure. The magnitude of the coefficient of the 1LL subband of the G1 (green) component is used as the luminance feature quantity. This is because the LL subband is a DC component and thus can represent luminance, and the reason why the G1 component is used is because human visual characteristics are sensitive to variations of the G component, and the G1 component is important visual information.

In step S1204, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1203 and a predetermined threshold. If it is determined to be a dark portion, the process proceeds to step S1205, and if not, the process proceeds to step S1206.

In step S1205, the controller 108 determines the quantization table for the color component sub-band data piece constituting the short-exposure RAW data to be Q3, and determines the quantization table for the color component sub-band data piece constituting the long-exposure RAW data to be Q4, and performs quantization processing.

In step S1206, the controller 108 determines whether the region of interest is a middle section based on the magnitude relationship between the luminance feature amount calculated in step S1203 and a predetermined threshold value. If the determination is the middle section, the process proceeds to step S1207, and if not, the process proceeds to step S1208.

In step S1207, it is determined that the quantization table for the color component sub-band data piece constituting the short-exposure RAW data is Q1, and the quantization table for the color component sub-band data piece constituting the long-exposure RAW data is Q3, and quantization processing is performed.

In step S1208, the controller 108 determines that the quantization table for the color component sub-band data piece constituting the short-exposure RAW data is Q2 and the quantization table for the color component sub-band data piece constituting the long-exposure RAW data is Q3, and performs quantization processing.

In step S1209, the controller 108 determines whether or not the quantization process is completed for all the sub-band data pieces in the image plane. If the quantization process is completed for all the sub-band data pieces, the process is ended, and if not, the process proceeds to step S1210.

In step S1210, the controller 108 updates the quantization processing target coefficient. After the update of the coefficient is completed, the controller 108 returns the process to step S1203.

In step S1211, the controller 108 calculates a luminance feature amount using the long-exposure sub-band data with correct exposure. The magnitude of the coefficient of the 1LL subband of the G1 component is used as the luminance feature amount, similarly to step S1203.

In step S1212, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1211 and a predetermined threshold value. If it is determined to be a dark portion, the process proceeds to step S1213, and if not, the process proceeds to step S1214.

In step S1213, the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data is determined to be Q3, and the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data is determined to be Q0, and quantization processing is performed.

In step S1214, the controller 108 determines whether the region of interest is a moderate portion based on the magnitude relationship between the luminance feature amount calculated in step S1211 and a predetermined threshold value. If the determination is the middle portion, the process proceeds to step S1215, and if not, the process proceeds to step S1216.

In step S1215, the controller 108 determines that the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data is Q3, and the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data is Q1, and performs quantization processing.

In step S1216, the controller 108 determines that the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data is Q5 and the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data is Q3, and performs quantization processing.

In step S1217, the controller 108 determines whether the quantization process is completed for all the sub-band data pieces in the image plane. If the quantization process is completed for all the sub-band data pieces, the process is ended, and if not, the process proceeds to step S1218.

In step S1218, the controller 108 updates the quantization processing target coefficient. The controller 108 returns the process to step S1211 after the update of the coefficient is completed.

In step S1219, since it is determined to be the normal mode, the controller 108 calculates the luminance feature amount using the sub-band data obtained by performing frequency transform on the RAW data generated by performing the addition average. The size of the coefficient of the 1LL subband of the G1 component obtained by performing the addition averaging is used as the luminance feature amount, similarly to step S1203.

In step S1220, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1219 and a predetermined threshold value. If it is determined to be a dark portion, the process proceeds to step S1221, and if not, the process proceeds to step S1222.

In step S1221, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the RAW data to be Q0, and performs quantization processing.

In step S1222, the controller 108 determines whether the region of interest is a middle portion based on the magnitude relationship between the luminance feature amount calculated in step S1219 and a predetermined threshold value. If the determination is a medium portion, the process proceeds to step S1223, and if not, the process proceeds to step S1224.

In step S1223, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the RAW data to be Q1, and performs quantization processing.

In step S1224, the controller 108 determines the quantization table used for the color component sub-band data pieces constituting the RAW data to be Q2, and performs quantization processing.

In step S1225, the controller 108 determines whether or not the quantization process is completed for all pieces of sub-band data in the image plane. If the quantization process is completed for all the sub-band data pieces, the process is ended, and if not, the process proceeds to step S1226.

In step S1226, the controller 108 updates the quantization processing target coefficient. The controller 108 returns the process to step S1219 after the updating of the coefficient is completed.

As described above, in the present embodiment, the separation unit 102 separates the RAW data into pieces of data for respective exposure times, eliminates the level difference between pixels different in exposure time, and thereby, suppresses high-frequency components, and therefore, the amount of recording data of the RAW data can be reduced. In addition, since weighting of the quantization parameter is performed in consideration of the composition ratio while the HDR composition processing after the development processing is envisioned, the recording data amount of the RAW data can be effectively reduced.

Note that, in the present embodiment, an example in which the luminance features are classified into three stages has been described, but the number of stages in which classification is performed is not limited thereto, and the number of stages may be further increased. In addition, in the flowcharts shown in fig. 20A to 20C, the following configurations have been described: wherein the quantization tables for the sub-band data pieces of the other color components are uniquely determined based on the feature amounts calculated using the 1LL sub-band data of the G1 component. However, an operation may be performed such that the quantization table is determined by calculating the feature amount independently for each color component.

In addition, an example has been described in which the calculation unit of the feature amount and the processing unit of the quantization are each coefficient, but the processing unit may be a coefficient block (two or more coefficients).

In addition, an example of lev-1 is described in the flowcharts shown in fig. 20A to 20C, but in the case of lev-2 or more, the horizontal size and the vertical size of the sub-band data differ according to lev. Therefore, the calculation unit of the feature amount cannot be the same as the processing unit of the quantization. It is assumed that the feature amount is calculated in units of one coefficient of 2LL subband data having lev-2. In this case, due to the characteristic of subsampling (subsampling) of frequency resolution, a2 × 2 block needs to be set as a processing unit for quantization of subband data having lev of 1.

In addition, the size of the coefficient of the 1LL subband data is used as the luminance feature amount, but the feature amount representing luminance may be generated using other methods, such as using a pixel or an average value calculated from the coefficients of the 1LL subband data of a plurality of color components, and the above-described method is not limited.

In addition, the channel conversion unit 1601 has been described using an example in which conversion into four channels is performed for each color element in R, G1, G2, and B in a bayer arrangement, but the color elements of R, G1, G2, and B may also be converted into four channels using the following conversion formulae 29 to 32.

Y ═ R + G1+ G2+ B)/4 formula 29

C0 ═ R-B equation 30

C1 ═ G0+ G1)/2- (R + B)/2 formula 31

C2-G0-G1 formula 32

The above transformation equations illustrate an exemplary transformation to four channels consisting of luminance and color difference. In this case, if control is performed such that the quantization parameter for the luminance component is decreased and the quantization parameter for the other color difference components is increased, the encoding efficiency is improved by utilizing the human visual characteristic. Note that the number of channels and the conversion method are not limited thereto.

(sixth embodiment)

Next, a sixth embodiment will be described. In the sixth embodiment, a method of determining a quantization table for a feature region in which the synthesis ratio is large for RAW data that does not have correct exposure is different from that of the first embodiment. In the first embodiment, a fixed pattern (pattern) prepared in advance is set as a quantization table for a feature region in which the synthesis ratio is large of RAW data having no correct exposure. Therefore, if each exposure RAW data is obtained by capturing performed at an exposure time extremely different from that of the correct exposure, the most appropriate quantization table cannot be selected according to the brightness, and there is a possibility that deterioration in image quality is incurred, or the code amount is unnecessarily increased. Therefore, in the present embodiment, a method of further improving the coding efficiency will be described. In this method, for a feature region in which the synthesis ratio is large, luminance feature determination is also performed on RAW data that does not have correct exposure, and a quantization table that is most suitable according to the feature is selected. Note that the configuration of the image capturing apparatus of the sixth embodiment is similar to that of the fifth embodiment, and therefore the description thereof is omitted.

Fig. 21A to 21C show the quantization processing procedure of the present embodiment. The difference from the fifth embodiment is that processing steps S1301 to S1312 are added. Description of processing steps similar to those of the fifth embodiment is omitted, and only the differences will be described.

In step S1301, the controller 108 calculates a luminance feature amount using the long-exposure subband data having overexposure. The magnitude of the coefficient of the 1LL subband of the G1 component is used as the luminance feature amount, similarly to the fifth embodiment.

In step S1302, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1301 and a predetermined threshold value. If it is determined to be a dark portion, the process proceeds to step S1303, and if not, the process proceeds to step S1304.

In step S1303, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data to be Q0, and performs quantization processing.

In step S1304, the controller 108 determines whether the region of interest is a middle portion based on the magnitude relationship between the luminance feature amount calculated in step S1301 and a predetermined threshold value. If the determination is the middle portion, the processing proceeds to step S1305, and if not, the processing proceeds to step S1306.

In step S1305, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data to be Q1, and performs quantization processing.

In step S1306, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the long-exposure RAW data to be Q2, and performs quantization processing.

In step S1307, the controller 108 calculates a luminance feature amount using the short-exposure subband data having underexposure. The magnitude of the coefficient of the 1LL subband of the G1 component is used as the luminance feature amount, similarly to the fifth embodiment.

In step S1308, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1307 and a predetermined threshold value. If it is determined to be a dark portion, the process proceeds to step S1309, and if not, the process proceeds to step S1310.

In step S1309, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data to be Q0, and performs quantization processing.

In step S1310, the controller 108 determines whether the region of interest is a middle section based on the magnitude relationship between the luminance feature amount calculated in step S1307 and a predetermined threshold. If the determination is the middle section, the process proceeds to step S1311, and if not, the process proceeds to step S1312.

In step S1311, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data to be Q1, and performs quantization processing.

In step S1312, the controller 108 determines the quantization table for the color component sub-band data pieces constituting the short-exposure RAW data to be Q2, and performs quantization processing.

As described above, since the most appropriate quantization table is set according to the luminance for the RAW data obtained by the capturing performed with the exposure time not having the correct exposure, the encoding efficiency can be further improved.

OTHER EMBODIMENTS

The embodiment(s) of the present invention may also be implemented by a computer reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (also may be more fully referred to as a "non-transitory computer readable storage medium") to perform the functions of one or more of the above-described embodiment(s) and/or a system or apparatus comprising one or more circuits (e.g., Application Specific Integrated Circuits (ASICs)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by a computer of a system or apparatus, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may include one or more processors (e.g., Central Processing Unit (CPU), Micro Processing Unit (MPU)), and may include a separate computerOr a separate network of processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or from a storage medium. The storage medium may include, for example, one or more of the following: hard disk, Random Access Memory (RAM), read-only memory (ROM), storage device for distributed computing systems, optical disk (such as Compact Disk (CD), Digital Versatile Disk (DVD), blu-ray disk (BD)^TM) Flash memory devices, memory cards, and the like.

OTHER EMBODIMENTS

The embodiments of the present invention can also be realized by a method in which software (programs) that perform the functions of the above-described embodiments are supplied to a system or an apparatus through a network or various storage media, and a computer or a Central Processing Unit (CPU), a Micro Processing Unit (MPU) of the system or the apparatus reads out and executes the methods of the programs.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.