阅读说明：本技术 图像处理设备、图像处理方法和存储介质 (Image processing apparatus, image processing method, and storage medium ) 是由 荒川谅治 于 2021-05-18 设计创作，主要内容包括：本发明涉及图像处理设备、图像处理方法和存储介质。图像处理设备包括：检测单元,其被配置为从拍摄图像中检测特定区域；以及判断单元,其被配置为基于所检测到的特定区域通过使用利用机器学习预先训练的学习单元来判断包括特征颜色的光源颜色。(The invention relates to an image processing apparatus, an image processing method, and a storage medium. The image processing apparatus includes: a detection unit configured to detect a specific area from a captured image; and a determination unit configured to determine a light source color including the characteristic color by using a learning unit trained in advance with machine learning based on the detected specific region.)

1. An image processing apparatus comprising:

a detection unit configured to detect a specific area from a captured image; and

a determination unit configured to determine a light source color including a characteristic color by using a learning unit trained in advance with machine learning based on the detected specific region.

2. The image processing apparatus according to claim 1, further comprising a correction unit configured to calculate a white balance correction value based on a result of the determination by the determination unit and perform white balance correction on the captured image.

3. The image processing apparatus according to claim 2,

wherein the determination unit is further configured to determine the light source color based on the color included in the specific region, an

Wherein the correction unit calculates a first white balance correction value corresponding to a light source color determined based on a color included in the specific region.

4. The image processing apparatus according to claim 3,

wherein the determination unit further determines the light source color based on a color of a region including a region other than the specific region, an

Wherein the correction unit calculates a second white balance correction value corresponding to a light source color determined based on a color of a region including a region other than the specific region.

5. The image processing apparatus according to claim 4, wherein the correction unit calculates a second white balance correction value corresponding to a light source color determined based on a color of a white area.

6. The image processing apparatus according to claim 4 or 5, wherein the correction unit generates a third white balance correction value based on the first white balance correction value and the second white balance correction value.

7. The image processing apparatus according to claim 6, wherein the correction unit generates the third white balance correction value by mixing the first white balance correction value and the second white balance correction value at a mixing ratio based on the determined reliability of the light source color.

8. The image processing apparatus according to any one of claims 1 to 5, wherein the specific region is a region that includes a characteristic color corresponding to a known subject and has a characteristic color distribution.

9. The image processing apparatus according to any one of claims 1 to 5, wherein the characteristic color is a human skin color.

10. The apparatus according to claim 1, wherein the learning data for the learning unit is data having a correlation between a light source whose color distribution is preliminarily distributed at a certain distance from a blackbody radiation locus and a subject having the characteristic color.

11. The image processing device according to any one of claims 1 to 5, further comprising a processing unit configured to perform conversion processing to reduce an influence of an individual difference of an image capturing device configured to capture an image,

wherein the signal value to be input to the learning unit is the signal value subjected to the conversion processing.

12. The image processing apparatus according to claim 11, wherein the processing unit converts a red value, i.e., an R value, a green value, i.e., a G value, and a blue value, i.e., a B value, for the specific region, which correspond to coordinates of a space having the individual difference, into an R value, a G value, and a B value, which correspond to coordinates of a space having no individual difference, based on a distance between two straight lines obtained by linear interpolation between two light sources, from among the plurality of light sources, which are determined for each space based on the R value, the G value, and the B value for the space having no individual difference and for which the R value, the G value, and the B value are known, which correspond to the space having the individual difference.

13. An image processing method performed by an image processing apparatus, the method comprising:

detecting a specific region from the captured image; and

the light source color including the characteristic color is judged by using a learning unit trained in advance with machine learning based on the detected specific region.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 13 when executed.

Technical Field

The present invention relates to a technique for processing a captured image.

Background

In image capturing with a digital camera, white balance (hereinafter abbreviated as WB) processing is generally performed in the camera so that white under image capturing illumination has equal red (R), green (G), and blue (B) signal values. Japanese patent laid-open No. 2006-319830 discusses a technique for estimating a light source based on an image signal obtained by capturing white and adjusting RGB values based on light source parameters corresponding to the estimated light source so that white under illumination becomes achromatic.

In the case of taking an image of a close-up face of a person, or in the case of taking a group photograph including a plurality of faces, the white detection target region may be very small, which makes WB processing difficult. In this regard, japanese patent laid-open No. 2009-159496 discusses a technique for performing WB processing based on a skin color region. In the WB processing, it is necessary to identify the type of external factor (for example, the type of ambient light) having an influence in the shooting scene, and perform color correction processing corresponding to the identified type of external factor (for example, whether ambient light is sunlight or fluorescent light).

In the technique discussed in japanese patent laid-open No. 2006-319830, a light source is estimated based on an image signal obtained by capturing white, and WB correction corresponding to the estimated light source is adjusted. Therefore, if the area of achromatic color is small, the light source cannot be estimated, which makes it difficult to perform adjustment to obtain the optimal WB.

Further, in the technique discussed in japanese patent laid-open No. 2009-159496, the detected skin color of each face is converted into a target skin color suitable as the skin color. However, the conversion process is not performed based on the judgment of the light source. Therefore, for example, if image capturing is performed in an environment where there are a plurality of light sources to be estimated, such as outdoors (in sunlight, in a cloudy condition, in shading, or the like), WB correction cannot be appropriately performed, and the original color cannot be reproduced in some cases.

Disclosure of Invention

According to an aspect of the present invention, an image processing apparatus includes: a detection unit configured to detect a specific area from a captured image; and a determination unit configured to determine a light source color including a characteristic color by using a learning unit (e.g., a learning model) trained in advance with machine learning based on the detected specific region.

Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

Drawings

Fig. 1 is a block diagram showing a basic configuration of an image capturing apparatus including an image processing apparatus according to an exemplary embodiment.

Fig. 2 is a block diagram showing a functional structure of the image processing apparatus.

Fig. 3A and 3B are diagrams for explaining balance (WB) correction value calculation processing.

Fig. 4 is a diagram for explaining a relationship between individual differences of signal values.

Fig. 5A and 5B each show a relationship between the blackbody radiation locus and the light source on the XY chromaticity diagram.

Fig. 6 is a flowchart showing a process for determining WB correction values to be used.

Fig. 7A, 7B, and 7C are diagrams showing various types of reliability tables to be used during mixing.

Detailed Description

Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings. The following exemplary embodiments are not intended to limit the present invention, and all combinations of features described in the exemplary embodiments are not essential to the present invention. In the specification, the same components are denoted by the same reference numerals.

Fig. 1 is a block diagram schematically showing the configuration of an image capturing apparatus 100 as an application example of an image processing apparatus according to the present exemplary embodiment.

The image capturing apparatus 100 is a camera such as a digital still camera or a digital video camera. The image capturing apparatus 100 may be any electronic device having a camera function, such as a mobile phone having a camera function or a computer equipped with a camera.

The optical system 101 includes a lens, a shutter, a diaphragm, and the like, and forms an optical image of a subject on an imaging plane of the image sensor 102. The optical system 101 transmits information on the focal length, shutter speed, and aperture value to a Central Processing Unit (CPU) 103. The image sensor 102 is a Charge Coupled Device (CCD) image sensor, a Complementary Metal Oxide Semiconductor (CMOS) image sensor, or the like, and includes, for example, red (R), green (G), and blue (B) sensor elements in a Bayer array (Bayer array). The image sensor 102 converts an optical image formed by the optical system 101 into luminance information of each pixel. Digital data obtained via an analog-to-digital (AD) converter (not shown), so-called RAW data before development processing, is stored in the main storage 104 via the CPU 103. The electrical gain (hereinafter referred to as international organization for standardization (ISO) sensitivity) of the image sensor 102 is set by the CPU 103.

The photometric sensor 105 is divided into a plurality of photometric areas (for example, 96 areas composed of 12 areas in the horizontal direction by 8 areas in the vertical direction), and detects the object luminance of each divided area based on the amount of light incident from the optical system 101. The luminance signal of each photometric area output from the photometric sensor 105 is converted into a digital signal by an AD converter (not shown), and the digital signal is sent to the CPU 103. The number of divided regions is not limited to 96 described above, but may be any positive number.

The CPU103 serving as a control unit controls units constituting the image capturing apparatus 100 based on input data and a program stored in advance to realize the functions of the image capturing apparatus 100. In the following description, at least some of the functions realized by the CPU103 executing the programs may be realized by dedicated hardware such as an Application Specific Integrated Circuit (ASIC) or the like.

The main storage 104 is a volatile storage such as a Random Access Memory (RAM) or the like, and serves as a work area of the CPU 103. The information stored in the main storage 104 is used by the image processing apparatus 107 or recorded on the storage medium 108.

The auxiliary storage 109 is a nonvolatile storage such as an Electrically Erasable Programmable Read Only Memory (EEPROM), and stores a program (firmware) for controlling the image capturing apparatus 100 and various setting information. The CPU103 uses a program and various setting information stored in the secondary storage device 109.

The storage medium 108 records image data obtained by, for example, photographing an image with the image sensor 102 and temporarily stored in the main storage 104. The storage medium 108 is, for example, a semiconductor memory card. The storage medium 108 is detachable from the image capturing apparatus 100, and the recorded data can be read by other apparatuses such as a personal computer and the like. In other words, the image capturing apparatus 100 includes a mechanism for attaching and detaching the storage medium 108 and a read/write function for the storage medium 108.

The display unit 110 displays a viewfinder image during image capturing, a captured image, and a Graphical User Interface (GUI) image for interactive operations.

The operation unit 111 is an input device group that receives a user operation and transmits operation input information to the CPU 103. Examples of the operation unit 111 include buttons, levers, and touch panels. The operation unit 111 may include an input device using voice or line of sight. The operation unit 111 also includes therein a release button for obtaining a user operation to start image capturing in the image capturing apparatus 100. In the image capturing apparatus 100 according to the present exemplary embodiment, the image processing apparatus 107 includes various image processing modes to be applied to capturing images, and these modes can be set to an image capturing mode using the operation unit 111.

The recognition device 106 detects a specific subject region (for example, a region corresponding to a human face or a sky region) from a captured image, and transmits data on the detected region to the CPU 103. In the present exemplary embodiment, the recognition device 106 detects a specific subject region by using machine learning and a Support Vector Machine (SVM) or the like. However, the method for detecting a specific subject region is not limited to machine learning and SVM.

The image processing apparatus 107 performs various types of image processing on captured image data obtained by capturing an image with the image sensor 102. Examples of the image processing include so-called development processing (such as white balance processing or the like), color interpolation processing for converting signals corresponding to RGB bayer arrays into three plane signals for RGB colors, gamma correction processing, saturation correction, and color phase correction. As described in detail below, the image processing apparatus 107 according to the present exemplary embodiment also detects a specific region from the specific subject region detected by the recognition apparatus 106, and determines a light source color including a characteristic color based on the detected specific region by using a learning unit trained in advance with machine learning. Further, the image processing apparatus 107 performs white balance correction based on the light source color obtained as a result of the determination. For example, if a person is detected as a specific subject region, the image processing apparatus 107 further detects a human skin region as a specific region from a region corresponding to the person, estimates (determines) a light source color based on a signal value corresponding to the human skin region by using a learning unit, and performs white balance correction based on the estimated light source color. At least part of the processing to be performed by the image processing apparatus 107 can be implemented as software modules by the CPU103 executing the program according to the present exemplary embodiment. The white balance is hereinafter abbreviated as WB as needed.

The processing performed by the image processing apparatus 107 according to the present exemplary embodiment will be described below. The present exemplary embodiment shows the following processing: for example, under natural light (sunlight), an image of a person is captured at a position where a small number of achromatic objects exist, such as a lawn area, and appropriate white balance correction can be performed on a skin color area of the person in the captured image.

Fig. 2 is a block diagram showing the processing of the image processing apparatus 107 as functional blocks. Each processing unit of the image processing apparatus 107 may be realized by a hardware component such as a circuit or the like, or some or all of the processing units may be realized by a software module by executing a program according to the present exemplary embodiment.

The image signal generation unit 201 receives image data obtained by capturing an image with the image sensor 102 and converting an analog signal into a digital signal by an AD converter (not shown). The image signal generation unit 201 generates image data including RGB colors in each pixel by performing synchronization processing on the received R, G, B image data formed in the bayer array. Further, the image signal generation unit 201 outputs the generated image data to each of the region detection unit 202 and the WB control unit 203.

The area detection unit 202 detects a skin color area from an area corresponding to a person detected as a specific subject area by the recognition device 106.

The WB control unit 203 calculates WB correction values based on the image data output from the image signal generation unit 201 and information about the skin color regions detected by the region detection unit 202. Then, the WB control unit 203 performs white balance correction processing on the image data by using the calculated WB correction value. The WB correction value calculation process performed by the WB control unit 203 will be described in detail later.

A color conversion Matrix (MTX) unit 204 multiplies the color gains so that the image data subjected to WB correction processing by the WB control unit 203 can be reproduced in the optimum color, and converts the image data into two color difference data R-Y and B-Y.

A Low Pass Filter (LPF) unit 205 limits the bandwidth of the color difference data R-Y and B-Y output from the color conversion MTX unit 204.

A Chroma Suppression (CSUP) unit 206 suppresses a false color component in a saturated portion of the color difference data R-Y and B-Y obtained after the bandwidth is limited by the LPF unit 205.

The Y generation unit 211 generates luminance data Y from the image data on which the WB correction processing has been performed by the WB control unit 203.

The edge enhancement unit 212 generates edge-enhanced luminance data Y from the luminance data Y generated by the Y generation unit 211.

The RGB conversion unit 207 generates RGB data from the color difference data R-Y and B-Y output from the CSUP unit 206 and the luminance data Y output from the edge enhancement unit 212.

The gamma (γ) correction unit 208 applies gradation correction based on a predetermined γ characteristic to the RGB data output from the RGB conversion unit 207.

The color luminance conversion unit 209 converts the RGB data subjected to the gamma correction into YUV data.

A Joint Photographic Experts Group (JPEG) compression unit 210 compression-encodes the YUV data output from the color luminance conversion unit 209. The compression-encoded image data is recorded as an image data file on the storage medium 108.

Next, WB correction value calculation processing performed by the WB control unit 203 will be described in detail.

First, the WB control unit 203 performs block division processing of dividing an image into a plurality of blocks in the horizontal direction and the vertical direction on the image data output from the image signal generation unit 201. In the block division processing, for example, an image is divided into 96 blocks composed of 12 blocks in the horizontal direction by 8 blocks in the vertical direction. The number of divided blocks is not limited to 96, but may be any positive number. Then, the WB control unit 203 calculates the R integrated value, G integrated value, and B integrated value within the blocks for each block, and calculates the R/G value and B/G value based on the R integrated value, G integrated value, and B integrated value.

Further, the WB control unit 203 integrates the R value, the G value, and the B value in each block included in the white region 302 set on the R/G axis and the B/G axis as shown in fig. 3A, thereby obtaining an integrated R value Rinteg, G value gineg, and B value bineg.

In fig. 3A, a solid line indicates the blackbody radiation locus 301. The white area 302 is set so that the R/G value and the B/G value can be plotted within the area indicated by the broken line in fig. 3A when the image capturing apparatus 100 captures an image of an achromatic subject under various types of light sources (e.g., in sunlight, in shadow, and under a Light Emitting Diode (LED) lamp, a tungsten lamp, a mercury lamp, a fluorescent lamp, and a flash lamp). A circle 303 shown in fig. 3A and 3B indicates a position where a single integrated value is converted into coordinates corresponding to the R/G axis and the B/G axis. The present exemplary embodiment describes an example in which WB calculation processing is performed by extracting pixels that may have achromatic colors from a region of a subject by using the R/G axis and the B/G axis and estimating light source colors. However, pixels that may have achromatic colors may be extracted from the region of the subject, and WB calculation processing may be performed by a method other than the above-described method.

Circles 304 shown in fig. 3A represent the results on the R/G axis and the B/G axis of estimating the light source color (correlated color temperature) from the signal values in the skin color region detected by the region detection unit 202 by using a pre-trained learning unit and calculating WB correction values based on the correlated color temperature. In the present exemplary embodiment, the training data used by the learning unit for estimating the light source color (correlated color temperature) is data having a correlation between a light source whose color distribution is preliminarily distributed at a certain distance from the blackbody radiation locus and a subject having a characteristic color.

Further, in the present exemplary embodiment, a signal value subjected to conversion processing for reducing the individual difference of the image sensor 102 of the image capturing apparatus 100, for example, is used as a signal value to be input to the learning unit. This makes it possible to use a general-purpose learning unit without preparing a learning unit for each unit, and to suppress variations in estimation accuracy caused by individual variations.

The following equations (1) and (2) are conversion equations representing conversion processing for reducing individual differences, and are conversion equations from an adjustment space with individual differences to a reference space without individual differences. Fig. 4 is a diagram showing a relationship between the adjustment space and the reference space. A solid line 420 shown in fig. 4 represents a relationship between RGB values of different types of light sources (i.e., light source a, light source B, and light source C) in the reference space. The solid line 410 represents the relationship between the RGB values of the light sources a, B, C in the adjustment space.

In fig. 4, a circle 401 represents RGB values (RrefA, GrefA, BrefA) of the light source a in the reference space. The circle 402 represents the RGB values (RrefB, GrefB, BrefB) of the light source B in the reference space. The circle 403 represents the RGB values (RrefC, GrefC, BrefC) of the light source C in the reference space. In fig. 4, a circle 404 represents RGB values (RadjA, GadjA, BadjA) of the light source a in the adjustment space. Circle 405 represents the RGB values (RadjB, GadjB, BadjB) of light source B in the adjustment space. The circle 406 represents the RGB values (RadjC, GadjC, BadjC) of the light source C in the adjustment space.

RGB values of the light source a, the light source B, and the light source C in each of the reference space and the adjustment space are measured in advance. Specifically, the light source a, the light source B, and the light source C are a plurality of light sources of which R, G and B values corresponding to a reference space having no individual difference and R, G and B values corresponding to an adjustment space having an individual difference are known.

In this case, the WB control unit 203 forms two straight lines by linear interpolation between two light sources determined from a plurality of known light sources for each space based on R, G and B values for a specific region in the captured image. Then, the WB control unit 203 performs conversion processing of converting R, G and B values corresponding to the coordinates of the adjustment space for a specific region in the captured image into R, G and B values corresponding to the coordinates of the reference space, based on the distance between the two straight lines.

More specifically, in the present exemplary embodiment, the WB control unit 203 determines that the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color region are located closer to which one of the light source a, the light source B, and the light source C. In the example shown in fig. 4, a case where the coordinates of the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color area correspond to the coordinates in the adjustment space indicated by the circle 407 shown in fig. 4 is enumerated. Further, the WB control unit 203 determines whether the Bi value among the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color region is located at a position closer to the light source a than the light source B or a position closer to the light source C. In the example shown in fig. 4, the Bi value among the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color area is located closer to the light source a than the light source B. Therefore, the WB control unit 203 performs linear interpolation between coordinates corresponding to RGB values of two light sources (i.e., the light source B and the light source a). Then, the WB control unit 203 converts the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color region from the coordinate values of the adjustment space to the coordinate values of the reference space based on the distance between two straight lines obtained by linear interpolation in each of the reference space and the adjustment space. In the example shown in fig. 4, the coordinates of the RGB values (Ri, Gi, Bi) of the signal values obtained from the skin color area correspond to the coordinates in the adjustment space indicated by the circle 407 shown in fig. 4. Therefore, the WB control unit 203 converts the coordinate values in the adjustment space into coordinate values in the reference space indicated by a circle 408 shown in fig. 4.

When the Ri value satisfies RadjB < Ri,

R＝(RrefA-RrefB)/(RadjA-RadjB)*(Ri-RadjB)+RrefB

B＝(BrefA-BrefB)/(BadjA-BadjB)*(Bi-BadjB)+BrefB (1)

when the Ri value satisfies RadjB ≧ Ri,

R＝(RrefB-RrefC)/(RadjB-RadjC)*(Ri-RadjC)+RrefC

B＝(BrefB-BrefC)/(BadjB-BadjC)*(Bi-BadjC)+BrefC (2)

in the present exemplary embodiment, as described above, the estimated light source color (correlated color temperature) is obtained from the learning unit having the correlation characteristics between the human skin region and the natural light source and the artificial light source distributed in advance at a certain distance from the blackbody radiation locus. Fig. 5A shows a relationship between the blackbody radiation locus on the XY chromaticity diagram and the light source. FIG. 5B is a graph showing the relationship between the blackbody radiation locus converted to values on the R/G axis and the B/G axis and the light source. The color distribution of natural light sources is distributed on the black body radiation locus, while the color distribution of artificial light sources (fluorescent lamps, LED light sources, etc.) is distributed at a certain distance from the black body radiation locus. In fig. 5A and 5B, a region 501 represents distribution in shading, a region 502 represents distribution under cloudy conditions, a region 503 represents distribution under sunlight, and each region 504 represents distribution under a bulb.

In the case of calculating WB correction values based on the estimated light source colors (correlated color temperatures), conversion from the reference space to the adjustment space is necessary so that the reverse conversion of the above conversion method is performed to calculate point coordinates (RSg, BSg) corresponding to color temperatures on the blackbody radiation locus. Then, the WB control unit 203 calculates a first WB correction value from the estimated light source color (correlated color temperature) by the following equation (3). In the present exemplary embodiment, WB correction values calculated from the estimated light source color (correlated color temperature) are used as the first WB correction values. WRgain in equation (3) represents the R gain of the first WB correction values calculated from the estimated light source colors. Likewise, in equation (3), WGgain represents a G gain of the first WB correction value, and WBgain represents a B gain of the first WB correction value.

WRgain＝1/RSg

WGgain＝1

WBgain＝1/BSg (3)

Further, in the present exemplary embodiment, the WB control unit 203 calculates WB correction values corresponding to light source colors estimated based on colors of regions including regions other than the specified region, that is, regions including regions other than the specified subject region. In the present exemplary embodiment, WB correction values calculated based on light source colors estimated from colors of regions including regions other than the specific region are used as the second WB correction values. Further, the WB control unit 203 calculates coordinates of a point on the blackbody radiation locus corresponding to the color temperature of the second WB correction value (for example, coordinates indicated by a circle 303 shown in fig. 3B). Then, the WB control unit 203 calculates coordinates of a third WB correction value obtained by mixing a first WB correction value (represented by a circle 304 shown in fig. 3B, for example) based on the estimated light source color and a second WB correction value (represented by a circle 303 shown in fig. 3B) based on the estimated light source color at a mixing ratio corresponding to the color temperature on the blackbody radiation locus. The circle 305 shown in fig. 3B represents the coordinates of the third WB correction values on the horizontal axis R/G and the vertical axis B/G in a simplified manner.

Third WB correction value generation processing to be executed by the WB control unit 203 by mixing the first WB correction values and the second WB correction values and processing to determine WB correction values to be used for white balance correction will be described below with reference to a flowchart shown in fig. 6.

In step S601, the WB control unit 203 calculates the reliability of the specific region based on the ratio of the specific region to the captured image. Specifically, the WB control unit 203 calculates the reliability of the skin-color region based on the ratio of the skin-color region detected by the region detection unit 202 to the whole captured image.

Fig. 7A is a diagram showing a table for calculating the reliability of the skin color area. In fig. 7A, the horizontal axis represents the number of pixels detected as a skin color area, and the vertical axis represents the reliability Sratio of the skin color area. In the present exemplary embodiment, the minimum value Smin, the threshold value Sth, and the maximum value Smax on the horizontal axis may be freely set. Since the number of pixels of the entire captured image is known, the ratio of the skin-color area to the entire captured image can be obtained based on the number of pixels detected as the skin-color area. Therefore, the minimum value Smin, the threshold value Sth, and the maximum value Smax on the horizontal axis may be set to values at which the reliability Sratio increases with an increase in the number of pixels detected as the skin color area.

In the present exemplary embodiment, referring to the table shown in fig. 7A, the WB control unit 203 acquires the reliability Sratio of the skin color area based on the number of pixels in the skin color area (the number of pixels in the specific area). In step S602, the WB control unit 203 determines whether the reliability Sratio of the skin color region corresponds to the reliability obtained when the number of pixels in the skin color region is greater than or equal to a threshold (greater than or equal to a threshold Sth), that is, whether the ratio of the skin color region to the entire image is sufficiently large. In a skin-color region having reliability Sratio obtained when the number of pixels in the skin-color region is greater than or equal to a threshold (greater than or equal to a threshold Sth), it is considered that the reliability of the light source color estimated based on the color of the skin-color region is high, and the first WB correction value calculated based on the light source color is a WB correction value that can be used to make appropriate WB correction. If the WB control unit 203 determines that the number of pixels in the skin-color region is greater than or equal to the threshold value (greater than or equal to the threshold value Sth) and the reliability of the skin-color region is high (yes in step S602), the processing proceeds to step S606. On the other hand, if the WB control unit 203 determines that the number of pixels in the skin-color region is less than the threshold Sth and the reliability of the skin-color region is not high (no in step S602), the processing proceeds to step S603.

In step S606, a first WB correction value calculated based on light source colors estimated from the colors of the skin color region (i.e., light source colors with high reliability) is set for WB correction.

In step S603, the WB control unit 203 calculates a distance between the blackbody radiation locus and coordinates corresponding to a second WB correction value corresponding to a light source color estimated based on colors of regions including regions other than the specific region. Then, the WB control unit 203 obtains the reliability Tdist of the distance with reference to the table shown in fig. 7B.

In fig. 7B, the horizontal axis represents the shortest distance between the blackbody radiation locus and the coordinates corresponding to the second WB correction values, and the vertical axis represents the reliability Tdist of the distance between the blackbody radiation locus and the coordinates corresponding to the second WB correction values. Further, the WB control unit 203 determines whether the distance is greater than or equal to a threshold (greater than or equal to a threshold Dth). If the WB control unit 203 determines that the distance is smaller than the threshold value (no in step S603), the reliability Tdist of the distance is high, and thus the processing proceeds to step S604. On the other hand, if the WB control unit 203 determines that the distance is greater than or equal to the threshold value (yes in step S603), the reliability Tdist of the distance is low, and thus the processing proceeds to step S605. That is, the WB control unit 203 determines the reliability of the light source color based on the white region by using the reliability of the distance Tdist. If the reliability of the light source color is high, the process advances to step S604. On the other hand, if the reliability is low, the process advances to step S605.

If the WB control unit 203 determines that the distance between the blackbody radiation locus and the coordinates corresponding to the second WB correction value is greater than or equal to the threshold Dth and the processing proceeds to step S605, there is a possibility that the processing is performed outdoors, and thus it may be difficult to estimate the light source. Therefore, the third WB correction value Wc is calculated by using the following equation (4). In equation (4), W304 represents the first WB correction value, and W303 represents the second WB correction value. Equation (4) is an arithmetic equation for mixing the first WB correction values and the second WB correction values at a mixing ratio based on the reliability Tdist of the distance. Since the reliability of distance Tdist is the reliability of the distance between the blackbody radiation locus and the coordinates corresponding to the second WB correction values, it can be considered that the mixing ratio based on the reliability of distance Tdist is the mixing ratio based on the reliability of the light source color estimated from the colors of the regions including the regions other than the specific region.

Wc＝W304*(1-Tdist/100)+W303*Tdist/100 (4)

The WB control unit 203 calculates the reliability of the ratio of the white area shown in fig. 7C during block division. In step S604, referring to the table shown in fig. 7C, the WB control unit 203 calculates the reliability ratio of the white region based on the information indicating the ratio of the block corresponding to the white region to the entire image.

In fig. 7C, the horizontal axis represents the number of blocks in the extracted white region, and the vertical axis represents the reliability ratio of the ratio. In the present exemplary embodiment, the block area is divided into 96 blocks composed of 12 blocks in the horizontal direction by 8 blocks in the vertical direction. The values on the horizontal axis shown in fig. 7C are merely examples, and are not limited to these values. In other words, these values may be set such that the reliability ratio of the ratio increases as the ratio of the blocks in the white area increases.

In step S604, the WB control unit 203 determines whether a white region is detected in a region including the specific subject region when calculating the second WB correction value. If the WB control unit 203 determines that a white region is detected (yes in step S604), the processing proceeds to step S607. In step S607, the second WB correction value is used for WB correction. On the other hand, if the WB control unit 203 determines that a white region is not detected (no in step S604), the processing proceeds to step S605.

If the process advances from step S604 to step S605, the WB control unit 203 determines that there is a possibility that the light source estimation accuracy is low, and thus determines not to calculate the second WB correction value as the optimal WB correction value. In step S605, the WB control unit 203 calculates a third WB correction value Wc by mixing the first WB correction value and the second WB correction value with equation (5). In equation (5), W304 represents the first WB correction value, and W303 represents the second WB correction value. Equation (5) is an arithmetic equation for mixing the first WB correction value and the second WB correction value at a mixing ratio based on the reliability ratio of the ratio.

Wc＝W304*(1-Tratio/100)+W303*Tratio/100 (5)

As described above, the WB control unit 203 according to the present exemplary embodiment detects a specific region including a characteristic color from a captured image, estimates a light source color based on the characteristic color of the specific region by using a learning unit trained with machine learning, and calculates a first WB correction value. Further, the WB control unit 203 calculates a second WB correction value based on the light source color estimated from a white region in a region including a region other than the specific region. Further, in the present exemplary embodiment, the third WB correction values are calculated by mixing the first WB correction values and the second WB correction values at a mixing ratio based on the estimated reliability of the light source colors. Then, the WB control unit 203 determines which of the first WB correction values to the third WB correction values is to be used for the WB correction processing, based on the reliability of the specific region, the reliability of the white region, and information indicating whether the white region is detected. With this structure, the image processing apparatus 107 according to the present exemplary embodiment can realize optimum WB correction with reduced influence of ambient light. The image processing apparatus 107 according to the present exemplary embodiment can realize the optimum WB correction processing by estimating the light source based on the skin area or the like in which the hue difference is distributed in a certain range regardless of the race in the case where there is a person who is likely to be the main subject in image capturing.

Although the present exemplary embodiment shows an example in which a human skin region is detected as a specific region, any region (such as a region corresponding to a green plant) may be detected as a specific region.

OTHER EMBODIMENTS

The embodiments of the present invention can also be realized by a method in which software (programs) that perform the functions of the above-described embodiments are supplied to a system or an apparatus through a network or various storage media, and a computer or a Central Processing Unit (CPU), a Micro Processing Unit (MPU) of the system or the apparatus reads out and executes the methods of the programs.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.