阅读说明：本技术 图像处理装置、信息生成装置及其方法 (Image processing apparatus, information generating apparatus, and method thereof ) 是由 孙乐公 平泽康孝 近藤雄飞 大野大志 于 2020-01-15 设计创作，主要内容包括：确定环境信息获取单元31获取材料确定环境中的光源的入射偏振光信息。确定目标信息获取单元32从通过捕获材料确定环境中的材料确定目标的图像而获得的偏光图像中获取出射偏振光信息。确定处理单元34可以基于在确定环境信息获取单元31处获取的入射偏振光信息、在确定目标信息获取单元32处获取的出射偏振光信息以及预先存储在信息存储单元33中的材料偏光特性信息——即针对每种材料指示入射偏振光的每个入射方向和反射光的每个出射方向的偏光和反射特性的材料偏光特性信息——来确定材料确定目标的材料。(The determination environment information acquisition unit 31 acquires incident polarized light information of a light source in the material determination environment. The determination target information acquisition unit 32 acquires the outgoing polarized light information from the polarized light image obtained by capturing the image of the material determination target in the material determination environment. The determination processing unit 34 may determine the material of the material determination target based on the incident polarized light information acquired at the determination environment information acquisition unit 31, the exit polarized light information acquired at the determination target information acquisition unit 32, and the material polarization characteristic information stored in advance in the information storage unit 33, that is, the material polarization characteristic information indicating the polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light for each material.)

1. An image processing apparatus comprising:

a determination environment information acquisition unit configured to acquire incident polarized light information of a light source in a material determination environment;

a determination target information acquisition unit configured to acquire outgoing polarized light information from a polarized light image obtained by capturing an image of a material determination target in the material determination environment; and

a determination processing unit configured to determine a material of the material determination target based on the incident polarized light information acquired at the determination environment information acquiring unit, the exit polarized light information acquired at the determination target information acquiring unit, and material polarization characteristic information indicating polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light and generated in advance.

2. The image processing apparatus according to claim 1,

wherein the determination processing unit calculates an error of one of the incident polarized light information and the outgoing polarized light information estimated using the material polarization characteristic information selected from the incident direction of the incident polarized light on the material determination target, the outgoing direction of the reflected light from the material determination target, and the other of the incident polarized light information acquired at the determination environment information acquisition unit and the outgoing polarized light information acquired at the determination target information acquisition unit, and determines the material of the material determination target based on the calculated error.

3. The image processing apparatus according to claim 2,

wherein the determination processing unit generates estimated exit polarized light information using the selected material polarization characteristic information and the incident polarized light information, and determines the material of the material determination target based on an error between the estimated exit polarized light information and the exit polarized light information acquired at the determination target information acquiring unit.

4. The image processing apparatus according to claim 2,

wherein the determination processing unit calculates estimated incident polarized light information using the selected material polarization characteristic information and the outgoing polarized light information acquired at the determination target information acquisition unit, and determines the material of the material determination target based on an error between the estimated incident polarized light information and the incident polarized light information acquired at the determination environment information acquisition unit.

5. The image processing apparatus according to claim 2,

wherein the material polarization characteristic information is generated for each of a plurality of materials, and

the determination processing unit selects material polarization characteristic information corresponding to the incident direction of the incident polarized light and the exit direction of the reflected light among the material polarization characteristic information according to each material, calculates an error for each material, and determines a material having the smallest error as a material of the material determination target.

6. The image processing apparatus according to claim 5,

wherein the material polarization characteristic information is generated for each of a plurality of materials, and

in a case where a minimum error of the calculated errors is smaller than a threshold value set in advance, the determination processing unit determines a material having the minimum error as a material of the material determination target.

7. The image processing apparatus according to claim 2,

wherein the determination environment information acquisition unit acquires incident polarized light information of each of a plurality of light sources in the material determination environment, and

the determination processing unit calculates the error using incident polarized light information of each of the plurality of light sources, and determines a material having the smallest error as a material of the material determination target.

8. The image processing apparatus according to claim 2,

wherein the determination environment information acquisition unit acquires incident polarized light information of each of a plurality of light sources in the material determination environment, and

the determination processing unit calculates the error using incident polarized light information of a light source selected from the incident polarized light information of each of the plurality of light sources, and determines a material having the smallest error as the material of the material determination target.

9. The image processing apparatus according to claim 1, further comprising:

a detection region setting unit configured to set a target object detection region from a polarization image obtained by capturing an image of the material determination target; and

a region detection unit configured to detect a target subject region from the target subject detection region set at the detection region setting unit based on a material determination result at the determination processing unit.

10. The image processing apparatus according to claim 1,

wherein the material polarization characteristic information and the incident polarization information acquired at the determination environment information acquisition unit are stored in an information storage unit in advance, and the determination processing unit determines the material of the material determination target using the incident polarization information and the material polarization characteristic information stored in the information storage unit.

11. The image processing apparatus according to claim 1,

wherein the determination environment information acquisition unit divides the material determination environment into a plurality of regions, and sets the average incident direction and the average incident polarization information for each region as the incident direction and the incident polarization information for the region.

12. The image processing apparatus according to claim 1,

wherein the determination processing unit determines the material of the material determination target based on the normalized incident polarized light information, the exit polarized light information, and the material polarization characteristic information.

13. An image processing method comprising:

acquiring incident polarized light information of a light source in a material determination environment at a determination environment information acquisition unit;

acquiring, at a determination target information acquisition unit, outgoing polarized light information from a polarized light image obtained by capturing an image of a material determination target in the material determination environment; and

determining, at a determination processing unit, a material of the material determination target based on the incident polarized light information acquired at the determination environment information acquiring unit, the exit polarized light information acquired at the determination target information acquiring unit, and material polarization characteristic information indicating polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light for each material and generated in advance.

14. An information generating apparatus comprising:

a light source information acquisition unit configured to acquire incident polarized light information of incident polarized light with respect to an information generation target that is material-conspicuous, for each incident direction, from a light source in a measurement environment in which the information generation target is set;

an exit polarized light information acquisition unit configured to acquire exit polarized light information of reflected light from the information generation target for each exit direction; and

a material polarization characteristic information generation unit configured to generate material polarization characteristic information indicating polarization and reflection characteristics in an incident direction of the incident polarized light and an exit direction of the reflected light for each direction using the incident polarized light information acquired at the light source information acquisition unit and the exit polarized light information acquired at the exit polarized light information acquisition unit.

15. The information generating apparatus according to claim 14,

wherein, a plurality of materials are arranged,

the light source information acquiring unit acquires incident polarized light information of each material,

the outgoing polarized light information acquiring unit acquires outgoing polarized light information of each material, and

the material polarization characteristic information generation unit generates material polarization characteristic information indicating polarization and reflection characteristics for each incident direction and each exit direction for each material.

16. The information generating apparatus according to claim 14,

wherein the outgoing polarized light information acquisition unit acquires the outgoing polarized light information based on an information generation target imaging unit configured to generate polarized light images in a plurality of polarization directions by capturing an image of the information generation target and an observation value of the polarized light image generated at the information generation target imaging unit.

17. The information generating apparatus according to claim 14,

wherein the material polarization characteristic information generation unit generates normalized material polarization characteristic information.

18. The information generating apparatus according to claim 14,

wherein the incident polarized light information indicates a Stokes vector of the incident polarized light, the exit polarized light information indicates a Stokes vector of the reflected light, and the material polarization characteristic information indicates a Mueller matrix.

19. An information generating method, comprising:

acquiring, at a light source information acquisition unit, incident polarized light information of incident polarized light with respect to an information generation target for each incident direction from a light source in a measurement environment in which the information generation target is set to be material-conspicuous;

acquiring, at an outgoing polarized light information acquisition unit, outgoing polarized light information of reflected light from the information generation target for each outgoing direction; and

at a material polarization characteristic information generation unit, material polarization characteristic information indicating polarization and reflection characteristics in an incident direction of the incident polarized light and an exit direction of the reflected light is generated for each direction using the incident polarized light information acquired at the light source information acquisition unit and the exit polarized light information acquired at the exit polarized light information acquisition unit.

Technical Field

The present technology relates to an image processing apparatus, an information generating apparatus, and a method thereof, which are capable of easily specifying a material based on a polarization image.

Background

As a method of determining a material in the related art, patent document 1 discloses specifying a type of an unknown material by capturing light that has passed through or has been reflected from the material, scattering the light, and comparing signal information generated based on the scattered light with signal information of a material registered in advance.

Reference list

Patent document

Patent document 1: japanese patent application laid-open No. 2002-243639

Disclosure of Invention

Problems to be solved by the invention

Incidentally, the signal information of the material registered in advance is a measurement result under certain environments and conditions. Therefore, it is necessary to provide signal information of the material in advance for each environment and for each condition to specify the type of the material from the signal information generated based on the scattered light. Further, in the case where the environment or condition changes, it is necessary to measure the environment or condition and generate signal information every time the environment or condition changes, and therefore, it is not easy to determine the type of material. .

Therefore, the present technology aims to provide an image processing apparatus, an information generating apparatus, and methods thereof that are capable of easily determining a material based on a polarization image.

Solution to the problem

In accordance with a first aspect of the present technique,

there is provided an image processing apparatus including:

a determination environment information acquisition unit configured to acquire incident polarized light information of a light source in a material determination environment;

a determination target information acquisition unit configured to acquire outgoing polarized light information from a polarized image obtained by capturing an image of a material determination target in a material determination environment; and

a determination processing unit configured to determine a material of the material determination target based on the incident polarized light information acquired at the determination environment information acquiring unit, the exit polarized light information acquired at the determination target information acquiring unit, and material polarization characteristic information indicating polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light and generated in advance.

In this technique, a determination environment information acquisition unit acquires incident polarized light information, such as an incident stokes vector of a light source in a material determination environment. Further, the determination target information acquisition unit acquires outgoing polarized light information, such as an outgoing stokes vector, from a polarized light image obtained by capturing an image of the material determination target in the material determination environment. The determination processing unit uses the incident polarized light information, the exit polarized light information, and material polarization characteristic information that is an indication Mueller (Mueller) matrix for each incident direction of the incident polarized light and each exit direction of the reflected light and is generated in advance and stored in the information storage unit. Further, the determination processing unit may use the normalized incident polarized light information, the exit polarized light information, and the material polarization characteristic information.

The determination processing unit calculates an error of one of the outgoing polarized light information and the incoming polarized light information estimated using material polarization characteristic information selected from an incident direction of the incoming polarized light on the material determination target, an outgoing direction of reflected light from the material determination target, and the other of the incoming polarized light information and the outgoing polarized light information. For example, the determination processing unit generates estimated exit polarized light information using the selected material polarization characteristic information and the incident polarized light information, and calculates an error between the estimated exit polarized light information and the exit polarized light information acquired at the determination target information acquiring unit. Further, the determination processing unit may calculate estimated incident polarized light information using the selected material polarization characteristic information and the outgoing polarized light information acquired at the determination target information acquisition unit, and may calculate an error between the estimated incident polarized light information and the incident polarized light information acquired at the determination environment information acquisition unit.

The determination processing unit determines the material of the material determination target based on the calculated error. For example, material polarization characteristic information is generated for each of a plurality of materials, and the determination processing unit determines a material having the smallest error as the material of the material determination target in the case where the smallest error among the calculated errors or the smallest error is smaller than a threshold value set in advance. The determination environment information acquisition unit divides the material determination environment into a plurality of regions, and sets the average incident direction and the average incident polarization information of each region as the incident direction and the incident polarization information of the region. Further, the determination environment information acquisition unit may acquire incident polarized light information of each of the plurality of light sources in the material determination environment, and the determination processing unit may calculate an error using the incident polarized light information of each of the plurality of light sources, and may calculate an error using incident polarized light information of a light source selected from the incident polarized light information of each of the plurality of light sources.

Further, the image processing apparatus may further include: a detection region setting unit configured to set a target object detection region from a polarization image obtained by determining an image of a target by a capture material; and a region detection unit configured to detect a target subject region from the target subject detection region set at the detection region setting unit based on the material determination result at the determination processing unit.

In accordance with a second aspect of the present technique,

there is provided an image processing method including:

acquiring incident polarized light information of a light source in a material determination environment at a determination environment information acquisition unit;

acquiring, at a determination environment information acquisition unit, outgoing polarized light information from a polarized light image obtained by capturing an image of a material determination target in a material determination environment; and

the material determination target is determined at the determination processing unit based on the incident polarized light information acquired at the determination environment information acquiring unit, the exit polarized light information acquired at the determination target information acquiring unit, and the material polarization characteristic information that indicates, for each material, the polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light and is generated in advance.

According to a third aspect of the present technique,

there is provided an information generating apparatus including:

a light source information acquisition unit configured to acquire incident polarized light information of incident polarized light on an information generation target for each incident direction from a light source in a measurement environment in which the material-distinct information generation target is set;

an exit polarized light information acquisition unit configured to acquire exit polarized light information of reflected light from the information generation target for each exit direction; and

a material polarization characteristic information generation unit configured to generate material polarization characteristic information indicating polarization and reflection characteristics in an incident direction of the incident polarized light and an exit direction of the reflected light for each direction using the incident polarized light information acquired at the light source information acquisition unit and the exit polarized light information acquired at the exit polarized light information acquisition unit.

In the present technology, for each incident direction and each material, a light source information acquisition unit acquires incident polarized light information, for example, an incident stokes vector of incident polarized light on an information generation target, from a light source in a measurement environment in which the information generation target whose material is obvious is set. The exit polarized light information acquisition unit acquires exit polarized light information of each exit direction and each material, for example, an exit stokes vector of reflected light from the information generation target, based on an observation value of the polarization image generated at the information generation target imaging unit and the polarization image configured to generate polarization images in a plurality of polarization directions by capturing an image of the information generation target. The material polarization characteristic information generation unit generates material polarization characteristic information indicating polarization characteristics and reflection characteristics of an incident direction of the incident polarized light and an exit direction of the reflected light, such as a mueller matrix, using the incident polarized light information and the exit polarized light information for each direction and each material. Further, the material polarization characteristic information generation unit may generate normalized material polarization characteristic information.

According to a fourth aspect of the present technique

Provided is an information generation method including:

acquiring, at a light source information acquisition unit, incident polarized light information of incident polarized light on an information generation target for each incident direction from a light source in a measurement environment in which the material-distinct information generation target is set;

acquiring, at an outgoing polarized light information acquisition unit, outgoing polarized light information of reflected light from the information generation target for each outgoing direction; and

at the material polarization characteristic information generation unit, material polarization characteristic information indicating polarization and reflection characteristics in the incident direction of the incident polarized light and the exit direction of the reflected light is generated for each direction using the incident polarized light information acquired at the light source information acquisition unit and the exit polarized light information acquired at the exit polarized light information acquisition unit.

Drawings

Fig. 1 is a view for explaining polarization and reflection characteristics.

Fig. 2 is a view showing an example of the configuration of the material determination system.

Fig. 3 is a view showing an example of the configuration of the information generating apparatus.

Fig. 4 is a flowchart showing an example of the operation of the information generating apparatus.

Fig. 5 is a flowchart (No.1) showing the operation of the information generating apparatus in detail.

Fig. 6 is a flowchart (No.2) showing the operation of the information generating apparatus in detail.

Fig. 7 is a view showing an example of material polarization characteristic information.

Fig. 8 is a view showing an example of the configuration of the image processing apparatus.

Fig. 9 is a view showing an example of the configuration of the environment imaging unit 311.

Fig. 10 is a view showing an example of a polarization image generated at the environment imaging unit 311.

Fig. 11 is a view showing an example of division of a polarization image.

Fig. 12 is a flowchart showing an example of the operation of the image processing apparatus.

Fig. 13 is a flowchart showing a first operation example.

Fig. 14 is a flowchart showing a second operation example.

Fig. 15 is a flowchart showing a third operation example.

Fig. 16 is a view showing a third operation example.

Fig. 17 is a view showing an example of a case where objects having substantially the same appearance are distinguished.

Fig. 18 is a view showing an example of a case where the determination result is presented in units of pixels.

Fig. 19 is a view showing an example of another method for acquiring a polarization image.

Detailed Description

Embodiments for implementing the present technology will be described below. Note that the description will be provided in the following order.

1. Polarization and reflection characteristics

2. Configuration of a material determination system

3. Information generating apparatus

3-1. configuration of information generating apparatus

3-2. operation of information generating apparatus

4. Image processing apparatus

4-1. configuration of image processing apparatus

4-2. operation of image processing apparatus

4-2-1. determining a first operation of a processing unit

4-2-2. determining a second operation of the processing unit

5. Example of operation of image processing apparatus

5-1. first example of operation

5-2. second example of operation

5-3. third example of operation

5-4 examples of other operations

6. Other configurations and operations

7. Application example

<1. polarization and reflection characteristics >

Fig. 1 is a view for explaining polarization and reflection characteristics. The light emitted from the light source LT irradiates the measurement object OB via a polarizer such as a linear polarizer PL1, and the imaging device CM captures an image of the measurement object OB via a linear polarizer PL2, for example. Note that the Z direction indicates the zenith direction, and the angle θ is the zenith angle.

In the case where the polarization directions of the linear polarizers (PL1 and PL2) are set to, for example, 0 °, 45 °, 90 °, and 135 °, and the pixel values obtained by capturing an image of the measurement object using the imaging device CM are set to the observation values I, the observation value I (0 °) in the case where the polarization direction is 0 °, the observation value I (45 °) in the case where the polarization direction is 45 °, the observation value I (45 °) in the case where the polarization direction is 0 °, and the observation value IA relationship between observation value I (90 °) in the case where the direction is 90 ° and observation value I (135 °) in the case where the polarization direction is 135 ° may be expressed as a Stokes (Stokes) vector VS ═ s⁰,s¹,s²]^T. Note that the relationship between the stokes vector and the observed value is as shown in expression (1).

[ mathematical formula 1]

In the Stokes vector, the component s⁰Indicating unpolarized brightness or average brightness. In addition, the component s¹Intensity difference, component s, indicating polarization direction between 0 ° and 90 °²Indicating the difference in intensity of the polarization directions between 45 ° and 135 °. In other words, the Stokes vector of 0 ° becomes [1,1,0 ]]^TThe Stokes vector of 45 DEG becomes [1,0, 1]]^TThe Stokes vector of 90 DEG becomes [1, -1,0 ]]^T135 ° into [1,0, -1%]^T。

Here, the expression (2) is established in the case where the stokes vector of the light in the incident direction ω i irradiated on the measurement object OB is set to "VSi", the stokes vector of the light in the exit direction ω o measured at the imaging device CM is set to "VSo", and the mueller matrix in the case of the incident direction ω i and the exit direction ω o is set to M (ω o, ω i). Note that expression (3) is a determinant representation of expression (2).

M(ωo,ωi)·VSi＝VSo...(2)

[ mathematical formula 2]

In the case where the polarization direction of the incident light irradiated onto the measurement object OB is 0 °, expression (3) becomes expression (4). Further, in the case where the polarization direction of incident light is 45 °, expression (3) becomes expression (5), in the case where the polarization direction of incident light is 90 °, expression (3) becomes expression (6), and in the case where the polarization direction of incident light is 135 °, expression (3) becomes expression (7).

[ mathematical formula 3]

Therefore, the mueller matrix M (ω o, ω i) indicated in expression (8) can be calculated based on expressions (4) to (7). Further, normalization is performed to eliminate the influence of the luminance in the mueller matrix M (ω o, ω i). Expression (9) indicates the normalized mueller matrix M (ω o, ω i).

[ mathematical formula 4]

The mueller matrix calculated in this way indicates polarization and reflection characteristics specific to the material of the measurement object. Further, the polarization and reflection characteristics are independent of the external environment, and therefore, the characteristics can be utilized at any position without repeatedly acquiring the polarization and reflection characteristics as long as the characteristics are measured once.

<2. configuration of Material determination System >

The material determination system determines the material of the material determination target based on the material polarization characteristic information indicating the polarization and reflection characteristics of each incident direction ω i of the incident polarization light and each exit direction ω o of the reflected light for each material, the exit polarization information calculated based on the polarization image of the reflected light from the exit direction ω o of the material determination target, and the incident polarization information of the incident polarization light of the incident direction ω i.

Fig. 2 shows an example of a configuration of a material determination system. The material determination system 10 includes an information generation apparatus 20 and an image processing apparatus 30. The information generating device 20 generates material polarization characteristic information for each material using a known material. Note that the information generating device 20 may register the generated material polarization characteristic information in the database unit 50 or may output the generated polarization characteristic information to the image processing device 30. The image processing device 30 captures an image of the material determination target to acquire a polarization image, and calculates polarization and reflection characteristics based on the polarization image. Further, the image processing device 30 determines the material of the material determination target based on the material polarization characteristic information acquired from the information generating device 20 or the database unit 50, the calculated polarization and reflection characteristics, the incident direction ω i, and the exit direction ω o.

<3. information generating apparatus >

<3-1. arrangement of information generating apparatus >

Fig. 3 shows an example of the configuration of the information generating apparatus. The information generating device 20 includes a light source information acquiring unit 21, an outgoing polarized light information acquiring unit 22, and a material polarization characteristic information generating unit 23.

The light source information acquisition unit 21 acquires incident polarized light information about the light source. The light source information acquisition unit 21 includes a light source imaging unit 211 and an incident polarized light information calculation unit 212.

The light source imaging unit 211 includes an imaging unit and a polarizing plate that can change a polarization direction and is disposed in front of the imaging unit. The light source imaging unit 211 captures an image of the light source for each polarization direction at the imaging unit to generate polarization images of a plurality of polarization directions while setting a plurality of predetermined polarization directions as polarization directions (linear polarizations) of the polarizing plates, and outputs the polarization images to the incident polarization information calculation unit 212.

The incident polarized light information calculation unit 212 calculates an incident stokes vector for each incident direction based on the polarized light image generated at the light source imaging unit 211. The incident polarized light information calculation unit 212 calculates an incident stokes vector (for example, an incident direction ω i) for each incident direction¹To ω i^mIncident stokes vector VSi on¹To VSi^m) Output to the material polarization characteristic information generation unit 23 as incident polarization information. Note that the incident direction may be determined by the position of the light source controlled at the information generating device 20, or the position information may be acquired from the light source to determine the direction of the light source with respect to the light source imaging unit 211. Further, the incident polarized light information calculation unit 212 may adopt a configuration in which information indicating the incident direction is input from the outside.

The outgoing polarized light information acquiring unit 22 acquires outgoing polarized light information on reflected light from the information generation target whose material is conspicuous in advance. The exit polarized light information acquisition unit 22 includes a known material imaging unit 221 and an exit polarized light information calculation unit 222.

The known material imaging unit 221 includes an imaging unit and a polarizing plate that can change a polarization direction and is disposed in front of the imaging unit. The known material imaging unit 221 captures an image of an information generation target to generate a polarization image for each polarization direction while setting a plurality of predetermined polarization directions as polarization directions (linear polarizations) of the polarizing plates. Further, the known material imaging unit 221 performs calibration before imaging, and obtains the direction (the outgoing direction ω o) of reflected light incident on the pixels of the known material imaging unit 221 in the imaging device coordinate system. Note that the calibration may be performed using any method, and is performed so that the exit direction ω o in the imaging device coordinate system can be obtained by using the external parameters and the internal parameters generated by the calibration. Further, the known material imaging unit 221 may generate an image of the target from the information captured from different directions, for example, from directions of different zenith angles, and generate a polarization image of a different exit direction ω o for each of the plurality of polarization directions.

The exit polarized light information calculation unit 222 calculates an exit stokes vector VSo using observed values (pixel values) indicated by the polarized light image generated at the known material imaging unit 221. Further, the outgoing polarized light information calculation unit 222 calculates a reflection stokes vector for each outgoing direction using the polarized light image in the different polarized light direction generated for each outgoing direction. The exit polarized light information calculating unit 222 calculates an exit stokes vector for each exit direction, for example, the exit direction ω o¹To ω oⁿUpper exit stokes vector VSo¹To VSoⁿOutput to the material polarization characteristic information generation unit 23 as outgoing polarized light information.

The material polarization characteristic information generation unit 23 generates material polarization characteristic information indicating the polarization characteristic and the reflection characteristic of the information generation target. The polarization and reflection characteristic calculation unit 231 and the material polarization characteristic information generation unit 232 are included.

For example, the polarization and reflection characteristic calculation unit 231 calculates the polarization and reflection characteristics based on the incident stokes vector VSi in the incident directions ω i1 to ω im indicated in the incident polarized light information supplied from the light source information acquisition unit 21¹To VSi^mAnd an exit direction ω o indicated in the exit polarized light information supplied from the exit polarized light information obtaining unit 22¹To ω oⁿUpper exit stokes vector VSo¹To VSoⁿPolarization and reflection characteristics are calculated, and a mueller matrix is calculated based on expression (2) for the incident direction and the exit direction. For example, the polarization and reflection characteristic calculation unit 231 is based on the incident direction ω i¹Incident stokes vector VSi¹And an emission direction ω o¹Of the exit stokes vector VSo¹To calculate the Mueller matrix M (ω o)¹,ωi¹). Further, the polarization and reflection characteristic calculation unit 231 calculates the polarization and reflection characteristic based on the incident direction ω i^mIncident stokes vector VSi^mAnd an emission direction ω oⁿOf the exit stokes vector VSoⁿTo calculateMueller matrix M (ω o)ⁿ,ωi^m). The material polarization characteristic information generation unit 23 normalizes the calculated mueller matrices, and outputs the normalized mueller matrices to the material polarization characteristic information generation unit 232.

The material polarization characteristic information generation unit 232 generates material polarization characteristic information by correlating the incident direction and the exit direction with the polarization and reflection characteristic information (e.g., normalized mueller matrix) calculated at the polarization and reflection characteristic calculation unit 231. Further, the light source information acquisition unit 21 and the exit polarized light information acquisition unit 22 acquire the incident polarized light information and the exit polarized light information of each known material, and the polarized light and reflection characteristic calculation unit 231 generates the polarized light and reflection characteristic information of each known material, and generates the material determination information in which the incident direction and the exit direction are associated with the polarized light and reflection characteristic information of each known material. As described above, the material polarization characteristic information generation unit 232 outputs the generated material polarization characteristic information to, for example, the database unit 50. Note that the database unit 50 may be provided at the information generating apparatus 20 or may be provided at the image processing apparatus 30. Further, the database unit 50 may be provided at an external device different from the information generating device 20 and the image processing device 30.

<3-2. operation of information creation means >

Subsequently, the operation of the information generating apparatus 20 will be described. Fig. 4 is a flowchart showing an example of the operation of the information generating apparatus. In step ST1, the information generating apparatus 20 initializes the imaging unit. The information generating apparatus 20 initializes the light source imaging unit 211 of the light source information acquiring unit 21 and the known material imaging unit 221 of the outgoing polarized light information acquiring unit 22. The information generating apparatus 20 calibrates the light source imaging unit 211 and the known material imaging unit 221 so that the coordinate systems match each other, and the process proceeds to step ST 2.

In step ST2, the information generating device 20 acquires incident polarized light information of the measurement environment. The light source information acquiring unit 21 of the information generating apparatus 20 acquires the incident directions and the incident polarized light information indicating the incident stokes vector of each incident direction, and the process proceeds to step ST 3.

In step ST3, the information generating apparatus 20 acquires the outgoing polarization information. The exit polarized light information acquisition unit 22 of the information generating apparatus 20 acquires exit polarized light information indicating an exit stokes vector for each exit direction for the incident direction of each light source, and the process proceeds to step ST 4.

In step ST4, the information generating device 20 generates material polarization characteristic information. The material polarization characteristic information generating unit 23 of the information generating device 20 generates polarization and reflection characteristics for each combination of the incident direction and the exit direction based on the incident polarization information acquired in step ST2 and the exit polarization information acquired in step ST 3. For example, the material polarization characteristic information generation unit 23 calculates the mueller matrix M (ω o, ω i) for each incident direction and each exit direction using the incident stokes vector VSi in the incident direction ω i and the exit stokes vector VSo in the transmission direction ω o. And (4) direction. Further, the material polarization characteristic information generation unit 23 generates material polarization characteristic information by associating the incident direction and the exit direction with the polarization and reflection characteristic information, and the processing proceeds to step ST 5.

In step ST5, the information generating device 20 determines whether generation of material polarization characteristic information of each material is completed. In the case where there is a material for which material polarization characteristic information has not been generated, the process proceeds to step ST6, and in the case where the generation of the material polarization characteristic information of each material has been completed, the information generating apparatus 20 ends the process.

In step ST6, the information generating apparatus 20 updates the material. The information generating apparatus 20 polarizes the subject for which an image is to be captured at the known-material imaging unit 221 of the exit polarized-light information acquiring unit 22 into a material for which material polarization characteristic information has not been generated yet, and the process returns to step ST 3.

Further, fig. 5 and 6 are flowcharts showing in detail the operation of the information generating apparatus. Fig. 5 and 6 indicate the case where the incident direction and the incident stokes vector are used at the position of each angle θ a in the azimuth direction and each angle θ b in the zenith direction. Further, the imaging direction of the known material imaging unit 221 moves for each angle θ c in the zenith direction. Further, the light source imaging unit 211 and the known material imaging unit 221 switch the polarization directions to "0 °, 45 °, 90 °, and 135 °. Note that the incident polarized light information is acquired in advance.

In step ST11, the information generating apparatus 20 initializes the known-material imaging unit. The exit polarized light information acquisition unit 22 of the information generation apparatus 20 calibrates the known material imaging unit 221 to set the azimuth angle and the zenith angle to 0 °, and the process proceeds to step ST 12.

In step ST12, the information generating apparatus 20 initializes the light source zenith angle. The light source information acquiring unit 21 of the information generating apparatus 20 initializes the light source imaging unit 211, and sets the direction in which the zenith angle of the known material imaging unit 221 is 0 ° to the direction in which the zenith angle of the light source imaging unit 211 is 0 °, and the process proceeds to step ST 13.

In step ST13, the information generating apparatus 20 initializes the light source azimuth. The light source information acquisition unit 21 of the information generation apparatus 20 initializes the light source imaging unit 211, and sets the direction in which the direction of the known material imaging unit 221 is 0 ° as the direction in which the azimuth angle of the light source imaging unit 211 is 0 °, and the process proceeds to step ST 14.

In step ST14, the information generating apparatus 20 initializes the polarizing plate on the light source side. The light source information acquiring unit 21 of the information generating apparatus 20 sets the polarization direction of the polarizing plate used at the light source imaging unit 211 to 0 °, and the process proceeds to step ST 15.

In step ST15, the information generating apparatus 20 initializes the polarizing plate on the known material imaging side. The exit polarized light information acquisition unit 22 of the information generation apparatus 20 sets the polarization direction of the polarizing plate used at the known material imaging unit 221 to 0 °, and the processing proceeds to step ST 16.

In step ST16, the information generating apparatus 20 captures an image of the information generation target. The known material imaging unit 221 captures information whose material is conspicuous to generate an image of the target to generate a polarization image, and the process proceeds to step ST 17.

In step ST17, the information generating apparatus 20 rotates the polarizing plate on the known material imaging side by 45 °. The outgoing polarized light information acquisition unit 22 of the information generating device 20 rotates the polarization direction of the polarizing plate by 45 °, and the processing proceeds to step ST 18.

In step ST18, the information generating apparatus 20 determines whether the polarization direction on the image side of the known material is less than 180 °. The process of the exit polarized light information acquiring unit 22 of the information generating device 20 returns to step ST16 in the case where the polarization direction after rotation is less than 180 °, and proceeds to step ST19 in the case where the polarization direction after rotation is equal to or greater than 180 °.

In step ST19, the information generating apparatus 20 acquires the outgoing polarization information. Each of the polarization images in the case where the polarization directions are "0 °, 45 °, 90 °, and 135 °" is generated by performing the processing from step ST16 to step ST18, and therefore, the information generating device 20 calculates the stokes vector based on the generated polarization image, and the processing proceeds to step ST 20.

In step ST20, the information generating apparatus 20 rotates the polarizing plate on the light source side by 45 °. The light source information acquiring unit 21 of the information generating apparatus 20 rotates the polarization direction of the polarizing plate by 45 °, and the processing proceeds to step ST 21.

In step ST21, the information generating apparatus 20 determines whether the polarization direction of the light source side is less than 180 °. The process of the light source information acquiring unit 21 of the information generating device 20 returns to step ST15 in the case where the polarization direction after rotation is less than 180 °, and proceeds to step ST22 in the case where the polarization direction after rotation is equal to or greater than 180 °.

In step ST22, the information generating apparatus 20 calculates polarization and reflection characteristics. The light source information acquisition unit 21 of the information generation device 20 calculates the mueller matrix based on the exit stokes vectors in the cases where the directions of the polarization of the incident polarized light of the information generation target are "0 °, 45 °, 90 °, and 135 °. In other words, the light source information acquiring unit 21 calculates the mueller matrix indicated in expression (8) based on expressions (4) to (7) described above, and the process proceeds to step ST 23.

In step ST23, the information generating apparatus 20 stores the material polarization characteristic information. The light source information acquiring unit 21 of the information generating apparatus 20 generates the material polarization characteristic information in which the incident direction ω i indicating the direction of the light source and the exit direction ω o indicating the direction of the known material imaging unit 221 are associated with the mueller matrix calculated in step ST22, and stores the material polarization characteristic information in the database unit or the like, and the processing proceeds to step ST 24.

In step ST24, the information generating apparatus 20 shifts the light source azimuth by θ a °. The light source information acquiring unit 21 of the information generating apparatus 20 moves the azimuth angle of the light source imaging unit 211 by θ a °, and the process proceeds to step ST 25.

In step ST25, the information generating apparatus 20 determines whether the light source azimuth is less than 360 °. The process of the light source information acquiring unit 21 of the information generating apparatus 20 returns to step ST14 in the case where the light source azimuth is less than 360 °, and proceeds to step ST26 in the case where the light source azimuth is equal to or greater than 360 °.

In step ST26, the information generating apparatus 20 moves the light source zenith angle by θ b °. The light source information acquiring unit 21 of the information generating apparatus 20 moves the zenith angle of the light source imaging unit 211 by θ b °, and the process proceeds to step ST 27.

In step ST27, the information generating apparatus 20 determines whether the light source zenith angle is less than 90 °. The process of the light source information acquiring unit 21 of the information generating apparatus 20 returns to step ST13 in the case where the light source zenith angle is less than 90 °, and proceeds to step ST28 in the case where the light source zenith angle is equal to or greater than 90 °. In other words, by performing the processing from step ST13 to step ST27, the material polarization characteristic information for each incident direction having the resolution of θ a ° in the azimuth direction and θ b ° in the zenith direction is stored in the database unit or the like for one exit direction.

In step ST28, the information generating apparatus 20 moves the zenith angle of the known material imaging unit by θ c °. The exit polarized light information acquisition unit 22 of the information generating apparatus 20 moves the zenith angle of the known material imaging unit 221 by θ c °, and the process proceeds to step ST 29.

In step ST29, the information generating apparatus 20 determines whether the zenith angle of the known material imaging unit is less than 90 °. The process of the exit polarized light information acquisition unit 22 of the information generating apparatus 20 returns to step ST12 in the case where the zenith angle of the known material imaging unit 221 is less than 90 °, and the exit polarized light information acquisition unit 22 ends the process when the zenith angle is equal to or greater than 90 °. Therefore, the material polarization characteristic information for each incident direction in which the resolution of the azimuth direction is the angle θ a, the resolution of the zenith direction is the angle θ b, and for each exit direction in which the resolution of the zenith direction is the angle θ c is stored in the database unit or the like.

Further, by performing the processing shown in fig. 5 and 6 for each information generation target whose material is different, it is possible to store, for each material, material polarization characteristic information for each incident direction in which the resolution of the azimuth direction is the angle θ a and the resolution of the zenith direction is the angle θ b and for each exit direction in which the resolution of the zenith direction is the angle θ c in the database unit or the like.

In this way, according to the information generating apparatus of the present technology, it is possible to generate material polarization characteristic information indicating polarization and reflection characteristics independent of the external environment and specific to the material. FIG. 7 shows an example of material polarization property information, and shows, for example, the incident direction ω i of the material MT1¹To ω i^mAnd an emission direction ω o¹To ω oⁿOf each combined mueller matrix M (ω o)¹,ωi¹) To M (ω o)ⁿ,ωi^m). Further, fig. 7 shows the incident direction ω i of the material MT2¹To ω i^mAnd an emission direction ω o¹To ω oⁿOf each combined mueller matrix M (ω o)¹,ωi¹) To M (ω o)ⁿ,ωi^m). Note that fig. 7 shows the material polarization characteristics of q types of different materials for each combination of the incident direction and the exit direction.

<4. image processing apparatus >

<4-1. configuration of image processing apparatus >

The configuration of the image processing apparatus will be described next. Fig. 8 shows an example of the configuration of the image processing apparatus. The image processing apparatus 30 includes a determination environment information acquisition unit 31, a determination target information acquisition unit 32, an information storage unit 33, and a determination processing unit 34.

The determination environment information acquisition unit 31 acquires incident polarized light information of a light source in the material determination environment. The determined environment information acquisition unit 31 includes an environment imaging unit 311 and an incident polarized light information calculation unit 312.

Fig. 9 shows an example of the configuration of the environment imaging unit 311. The environment imaging unit 311 includes, for example, a plurality of imaging units 3111 having different imaging directions and a polarizing plate 3112 which can change a polarization direction and is disposed in front of each imaging unit. Note that each polarizing plate 3112 has the same polarization direction. The environment imaging unit 311 captures an image of an environment to generate, for example, a global plane polarization image for each of a plurality of polarization directions at the time of capturing an image of a material determination target. Note that the environment imaging unit 311 may acquire a global plane polarization image of each of a plurality of polarization directions through one imaging unit 3111 and one polarizing plate 3112 using a fish-eye lens or the like. Further, the environment imaging unit 311 does not always generate a global plane polarization image. For example, in the case where the light source is provided only in a limited range, the environment imaging unit 311 may generate a polarized image in the limited range.

Fig. 10 shows an example of a polarization image generated at the environment imaging unit 311. Note that (a) of fig. 10 shows an example of a fisheye image indicating a holosphere, and (b) of fig. 10 shows an example of an expanded image obtained by expanding the fisheye image on a cylindrical surface.

The incident polarized light information calculation unit 312 divides the polarized light image generated at the environment imaging unit 311 in the zenith direction and the azimuth direction, calculates the average incident stokes vector and the average incident direction of the light beam within the area for each divided area to set the average incident stokes vector and the average incident direction as incident polarized light information, and outputs the incident polarized light information to the information storage unit 33.

Fig. 11 shows an example of division of a polarization image. Note that fig. 11 (a) shows an example of segmentation of the fisheye image shown in fig. 10 (a), and fig. 11 (b) shows an example of segmentation of the expanded image shown in fig. 10 (b).

Here, as shown in (c) of fig. 11, in the case where the light source position of the incident polarized light of the incident direction ω i is included in the area ARi, the incident polarized light information calculation unit 312 calculates the average incident direction ω i¹Average incident Stokes vector VSi in sum region ARi¹. The incident polarized light information calculation unit 312 calculates the average incident direction and the average incident stokes vector of the other areas in a similar manner. The incident polarized light information calculation unit 312 outputs the calculated incident stokes vector for each incident direction to the information storage unit 33.

Further, the determination environment information acquisition unit 31 calculates an incident stokes vector of each light source at the time of capturing an image of the material determination target, and outputs incident polarized light information indicating the incident stokes vector of each incident direction of each light source to the information storage unit 33.

The determination target information acquiring unit 32 acquires outgoing polarized light information on reflected light from the material determination target. The determination target information acquisition unit 32 includes a determination target imaging unit 321 and an exit polarized light information calculation unit 322.

The determination target imaging unit 321 includes an imaging unit and a polarizing plate that can change a polarization direction and is disposed in front of the imaging unit. The determination target imaging unit 321 captures an image of the material determination target from the exit direction ω o to generate a polarization image of each of the plurality of polarization directions. Further, the determination target imaging unit 321 performs calibration or the like before imaging in a similar manner to the above-described known material imaging unit 221, so that the exit direction ω o in the image pickup device coordinate system can be obtained by using the external parameters and the internal parameters generated by the calibration.

The exit polarized light information calculation unit 322 calculates the stokes vector VSo using the observed value (pixel value) represented by the polarized light image generated at the determination target imaging unit 321. Further, the exit polarized light information calculation unit 222 outputs exit polarized light information indicating the exit direction ω o and the exit stokes vector VSo to the determination processing unit 34.

The information storage unit 33 stores the material polarization characteristic information generated at the information generating device 20 and the incident polarized light information acquired at the determination environment information acquiring unit 31. Note that the information storage unit 33 may store the material polarization characteristic information acquired from the information generating apparatus 20, or may store the material polarization characteristic information acquired from the database unit.

The determination processing unit 34 is based on the incident polarized light information acquired at the determination environment information acquiring unit 31, the exit polarized light information acquired at the determination target information acquiring unit 32, and material polarization characteristic information that is generated in advance and indicates polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected polarized light. The determination processing unit 34 includes an estimation processing unit 341, an error calculation unit 342, and a material determination processing unit 343.

The estimation processing unit 341 estimates one of the incident polarized light information and the exit polarized light information using the material polarization characteristic information selected from the incident direction of the incident polarized light on the material determination target, the exit direction of the reflected light from the material determination target, and the other of the incident polarized light information acquired at the determination environment information acquisition unit and the exit polarized light information acquired at the determination target information acquisition unit.

The estimation processing unit 341 acquires the incident polarized light information acquired at the determination environment information acquisition unit 31 from the information storage unit 33. Further, the estimation processing unit 341 acquires the material polarization characteristic information corresponding to the incident direction ω i of the incident polarized light information acquired at the determination environment information acquisition unit 31 and the exit direction ω o of the reflected light incident on the determination target imaging unit 321 from the information storage unit 33. The estimation processing unit 341 estimates the outgoing polarized light information based on the acquired incident polarized light information and material polarization characteristic information, or estimates the incident polarized light information based on the acquired material polarization characteristic information and the outgoing polarized light information acquired at the determination target information acquisition unit 32.

For example, the estimation processing unit 341 acquires the mueller matrix M (ω o, ω i) of each material corresponding to the incident direction ω i and the exit direction ω o of the exit polarized light information acquired at the determination target information acquiring unit 32 from the information storage unit 33. Further, the estimation processing unit 341 acquires the incident stokes vector VSi (ω i) in the incident direction ω i from the information storage unit 33. The estimation processing unit 341 calculates, for each material, an estimated outgoing stokes vector VSEo (ω o) whose incident direction is not limited to a specific direction based on the mueller matrix M (ω o, ω i) and the incoming stokes vector VSi (ω i), or calculates an estimated incoming stokes vector VSEi (ω i) based on the inverse matrix of the mueller matrix M (ω o, ω i) and the outgoing stokes vector VSo (ω o), and outputs the estimated outgoing stokes vector VSEo (ω o) and the estimated incoming stokes vector VSEi (ω i) to the error calculation unit 342.

The error calculation unit 342 calculates, for each material, an error between the outgoing polarized light information estimated at the estimation processing unit 341 and the outgoing polarized light information obtained at the determination target information acquisition unit 32 or an error between the incident polarized light information estimated at the estimation processing unit 341 and the incident polarized light information in the incident direction ω i acquired at the determination environment information acquisition unit 31. For example, the error calculation unit 342 calculates, for each material, an integral value of an error between the estimated exit stokes vector VSEo (ω o) estimated at the estimation processing unit 341 and the exit stokes vector VSo (ω o) acquired at the determination target information acquisition unit 32 or an error between the estimated incident stokes vector VSEi (ω i) estimated at the estimation processing unit 341 and the incident stokes vector VSi (ω i) for each incident direction, and outputs the integral value to the material determination processing unit 343.

The material determination processing unit 343 determines the material of the material determination target based on the error (or the integrated value of the error) calculated at the error calculation unit 342. For example, the material determination processing unit 343 determines the material, which makes the error calculated for each material at the error calculation unit 342 become the smallest error, as the material of the material determination target. Further, in a case where the minimum error among the calculated errors is smaller than a threshold value set in advance, the determination processing unit may determine the material having the minimum error as the material of the material determination target. Further, the determination processing unit may calculate an error of the specific material, and in a case where the calculated error is smaller than a threshold value set in advance, may determine the specific material as the material of the material determination target. In other words, the determination processing unit may also determine whether the material is a desired material and determine the material.

<4-2. operation of image processing apparatus >

Fig. 12 is a flowchart showing an example of the operation of the image processing apparatus. In step ST31, the image processing apparatus 30 initializes the determination target imaging unit. The image processing apparatus 30 initializes the determination target imaging unit 321 of the determination target information acquisition unit 32. The image processing apparatus 30 calibrates the determination target imaging unit 321 so that the coordinate systems match, and the process proceeds to step ST 32.

In step ST32, the image processing apparatus 30 acquires the incident polarized light information of the material determination environment. The determination environment information acquiring unit 31 of the image processing device 30 acquires the incident polarized light information of the light source based on the polarized image obtained by capturing the image of the material determination environment, and the process proceeds to step ST 33.

In step ST33, the image processing apparatus 30 acquires the outgoing polarized light information. The determination target information acquisition unit 32 of the image processing apparatus 30 acquires the exit stokes vector and the exit polarized light information of the exit direction indicating the material determination target, and the process proceeds to step ST 34.

In step ST34, the image processing apparatus 30 executes determination processing. The determination processing unit 34 of the image processing device 30 determines the material of the material determination target based on the incident direction detected in step ST32, the exit polarized light information acquired in step ST33, and the material polarization characteristic information and the incident polarized light information stored in advance in the information storage unit 33.

<4-2-1. first operation of determining processing Unit >

In the first operation of the determination processing unit 34, the outgoing polarization information is estimated based on the polarization and reflection characteristic information corresponding to the plurality of incoming directions ω i when the image of the target is determined by the captured material, the outgoing direction ω o of the reflected light supplied to the determination target imaging unit 321, and the incoming polarization information in the incoming direction ω i acquired at the determination environment information acquisition unit 31, and the material of the target is determined based on the estimated outgoing polarization information and the outgoing polarization information acquired at the determination target information acquisition unit 32.

The determination processing unit 34 performs the operation of expression (10) using the incident stokes vector VSi (ω i) in the incident direction ω i and the mueller matrix M (ω o, ω i) acquired from the information storage unit 33, and calculates the estimated exit stokes vector VSeo (ω o) in the exit direction ω o.

M(ωo,ωi)·VSi(ωi)＝VSeo(ωo)...(10)

Further, the determination processing unit 34 calculates an estimated exit stokes vector VSeo (ω o) whose incident direction is not limited to a specific direction by performing an operation shown in expression (11) and integrating the estimated exit stokes vector VSeo (ω o) of each of the plurality of incident directions ω i.

[ math figure 5]

The determination processing unit 34 calculates an error E (ω o) between the estimated exit stokes vector VSEo (ω o) and the exit stokes vector VSo (ω o) acquired at the determination target information acquiring unit 32 based on expression (12). Further, the normalized stokes vector may be used in the calculation of the error E (ω o) to prevent the influence of the luminance component. In addition, expression (13) indicates a stokes vector before normalization, and expression (14) indicates a stokes vector after normalization.

[ mathematical formula 6]

E(ω_o)＝||VSo(ω_o)-VSEo(ω_o)||…(12)

The determination processing unit 34 performs the above-described processing using the mueller matrix for each material, compares the error E (ω o) -1 calculated for each material with E (ω o) -q (where q is the number of materials), and determines the minimum error E (ω o) min. Further, in the case where the error E (ω o) min is smaller than the threshold value Tho set in advance, the determination processing unit 34 determines the material corresponding to the mueller matrix used to calculate the error E (ω o) min as the material determination target. Further, in the case where the error E (ω o) min is equal to or larger than the threshold value Tho, the determination processing unit 34 determines the material for which the material determination target cannot be determined. Note that the determination accuracy of the material can be adjusted by adjusting the threshold value Tho. By the above processing, the material of the material determination target can be determined.

<4-2-2. second operation of determining processing Unit >

In the second operation of the determination processing unit 34, the incident polarized light information is estimated based on the polarized light and reflection characteristic information corresponding to the plurality of incident directions ω i at the time of capturing the image of the material determination target and the exit direction ω o of the reflected light supplied to the determination target imaging unit 321 and the exit polarized light information acquired at the determination target information acquiring unit 32, and the material of the material determination target is determined based on the estimated incident polarized light information and the incident polarized light information of each incident direction acquired at the determination environment information acquiring unit 31.

The determination processing unit 34 uses an inverse matrix M of the mueller matrix corresponding to the incident direction ω i and the exit direction ω o^-1(ω o, ω i) and the exit stokes vector VSo (ω o) acquired at the determination target information acquiring unit 32 to perform the operation of expression (15),and calculates an estimated incident stokes vector VSEi (ω i) in the incident direction ω i.

VSEi(ωi)＝M^-1(ωo,ωi)·VSo(ωo)...(15)

The determination processing unit 34 calculates an error e (ω o, ω i) between the incident stokes vector VSi (ω i) in the incident direction ω i and the estimated incident stokes vector VSEi (ω i) based on expression (16). Further, the normalized stokes vector is used in the calculation of the error e (ω o, ω i) to prevent the influence of the luminance component. Further, in a case where the incident direction is not limited to a specific direction, the determination processing unit 34 calculates the error E (ω i) by performing the operation shown in expression (17) and integrating the errors E (ω o, ω i) calculated for each of the plurality of incident directions ω i.

[ math figure 7]

e(ω_o,ω_i)＝||VSi(ω_i)-VSEi(ω_i)||…(16)

The determination processing unit 34 performs the above-described processing using the mueller matrix for each material, compares the error E (ω i) -1 calculated for each material with E (ω i) -q (where q is the number of materials), and determines the minimum error E (ω i) min. Further, in the case where the error E (ω i) min is smaller than the threshold value Thi set in advance, the determination processing unit 34 determines a known material corresponding to the mueller matrix used to calculate the error E (ω i) min as the material of the material determination target. Further, in the case where the error E (ω i) min is equal to or larger than the threshold value Thi, the determination processing unit 34 determines the material for which the material determination target cannot be determined. Note that the determination accuracy of the material can be changed by adjusting the threshold value Thi. By the above processing, the material of the material determination target can be determined.

In this way, according to the image processing device of the present technology, the material of the material determination target can be determined based on the material polarization characteristic information generated at the information generating device and indicating the polarization and reflection characteristics specific to the material, the incident polarization information of the light source in the material determination environment, and the exit polarization information acquired from the polarization image of the material determination target.

<5. example of operation of image processing apparatus >

<5-1. first operation example >

In the first operation example, the information storage unit 33 stores therein mueller matrices of plural kinds (q kinds) of materials and incident stokes vectors of plural kinds (r kinds) of light sources.

Fig. 13 is a flowchart showing a first operation example. In step ST41, the image processing apparatus initializes the determination target imaging unit. The determination target information acquisition unit 32 of the image processing apparatus 30 calibrates the determination target imaging unit 321, and the processing proceeds to step ST 42.

In step ST42, the image processing apparatus calculates an error of the determination target pixel u. The determination processing unit 34 of the image processing device 30 calculates an error eij for each combination of the i (where i is 0, 1.. q) th type of material and the j (where j is 0, 1.. r) th type of light source, and the processing proceeds to step ST 43.

In step ST43, the image processing apparatus detects the minimum error Emin. The determination processing unit 34 of the image processing apparatus 30 calculates a combination x, y of the material and the light source, which minimizes the error, based on expression (18), and sets the error of the combination x, y of the material and the light source as the minimum error Emin, as shown in expression (19).

[ mathematical formula 8]

Emin＝e_x,y...(19)

In step ST44, the image processing apparatus determines whether the minimum error Emin is smaller than the threshold value Tha. The processing of the determination processing unit 34 of the image processing apparatus 30 proceeds to step ST45 in the case where it is determined that the minimum error Emin detected in step ST43 is less than the threshold Tha, and proceeds to step ST46 in the case where the minimum error Emin is equal to or greater than the threshold Tha.

In step ST45, the image processing apparatus determines the material having the smallest error Emin as the material of the material determination target, and the process proceeds to step ST 47.

In step ST46, the image processing apparatus determines that the material of the material determination target is an unknown material, and the process proceeds to step ST 47.

In step ST47, the image processing apparatus determines whether the determination of all pixels has been completed. The process of the image processing apparatus 30 proceeds to step ST48 if the determination of all the pixels is not completed, and proceeds to step ST49 if the determination of all the pixels is completed.

In step ST48, the image processing apparatus updates the determination target pixel. The image processing apparatus 30 selects a new pixel on which material determination has not been performed as a determination target pixel, and the process returns to step ST 42.

In step ST49, the image processing apparatus outputs the determination result. For example, the image processing apparatus 30 displays the image areas in which the materials are determined to be the same material with the same color, the same brightness, or the like based on the determination result of the material. Further, the image processing apparatus 30 displays the material as the region of the different material with the different color or brightness.

In this way, according to the first operation example, the materials of the respective subjects included in the imaging range of the determination target imaging unit 321 can be determined. In addition, the difference in material is indicated as a difference in display property, and thus the difference in material can be easily determined.

<5-2. second operation example >

In the second operation example, a plurality of (q) mueller matrices of materials and incident stokes vectors of a plurality of (r) light sources are stored in the information storage unit 33. Further, unlike the first operation example, a light source is selected according to the condition when the material is determined, and the material is determined using incident polarized light information of the selected light source.

For example, in the case of being determined to be indoor, incident polarized light information of the indoor illumination light source is used. Further, in the case of outdoor determination of materials, incident polarized light information in the case where the light source is the sun is used. Further, in the case of determining a material outdoors at night, incident polarized light information of a light source irradiating a position of a polarized light determination target is used.

Fig. 14 is a flowchart showing a second operation example. In step ST51, the image processing apparatus initializes the determination target imaging unit. The determination target information acquisition unit 32 of the image processing apparatus 30 calibrates the determination target imaging unit 321, and the processing proceeds to step ST 52.

In step ST52, the material determination unit selects a light source. The determination environment information acquisition unit 31 of the image processing apparatus 30 selects a light source according to the condition when the captured material determines the image of the target, and the process proceeds to step ST 53.

In step ST53, the image processing apparatus calculates an error of the determination target pixel u. The determination processing unit 34 of the image processing apparatus 30 calculates an error ei for each material of the i-th (where i ═ 0, 1.. q) type, and the processing proceeds to step ST 54.

In step ST54, the image processing apparatus detects the minimum error Emin. The determination processing unit 34 of the image processing apparatus 30 detects the material x whose error becomes minimum based on expression (20), and sets the error of the material x to the minimum error Emin shown in expression (21). Note that "y" indicates the selected light source.

[ mathematical formula 9]

Emin＝e_xy...(21)

In step ST55, the image processing apparatus determines whether the minimum error Emin is smaller than the threshold value Tha. The processing of the determination processing unit 34 of the image processing apparatus 30 proceeds to step ST56 in the case where it is determined in step ST54 that the detected minimum error Emin is smaller than the threshold Tha, and proceeds to step ST57 in the case where the minimum error Emin is equal to or larger than the threshold Tha.

In step ST56, the image processing apparatus determines the material having the smallest error Emin as the material of the material determination target, and the process proceeds to step ST 58.

In step ST57, the image processing apparatus determines that the material of the material determination target is an unknown material, and the process proceeds to step ST 58.

In step ST58, the image processing apparatus determines whether the determination of all pixels has been completed. The process of the image processing apparatus 30 proceeds to step ST59 if the determination of all the pixels has not been completed, and proceeds to step ST60 if the determination of all the pixels has been completed.

In step ST59, the image processing apparatus updates the determination target pixel. The image processing apparatus 30 selects a new pixel for which material determination has not been made as a determination target pixel, and the process returns to step ST 53.

In step ST60, the image processing apparatus outputs the determination result. For example, the image processing apparatus 30 displays the image areas in which the materials are determined to be the same material with the same color, the same brightness, or the like based on the determination result of the material. Further, the image processing apparatus 30 displays the material as the region of the different material with the different color or brightness.

In this way, according to the second operation example, similarly to the first operation example, the materials of the respective subjects included in the imaging range of the determination target imaging unit 321 can be determined. Further, the difference in material is indicated as a difference in display property, so that the difference in material can be easily determined. Further, in the second operation example, the light source is specified, so that the material can be determined more easily than in the first operation example.

<5-3. third operational example >

In the third operation example, description will be made with respect to a case where the image processing apparatus 30 further includes a detection region setting unit configured to set a target object detection region from a polarization image obtained by determining an image of a target by a capture material. The region detection unit is configured to detect a target subject region from the target subject detection region set at the detection region setting unit based on the material determination result at the determination processing unit.

Fig. 15 is a flowchart showing a third operation example. In step ST71, the image processing apparatus initializes the determination target imaging unit. The determination target information acquisition unit 32 of the image processing apparatus 30 calibrates the determination target imaging unit 321, and the processing proceeds to step ST 72.

In step ST72, a target object detection area is set. The detection region setting unit of the image processing apparatus 30 sets a target subject detection region including a material determination target (target subject) from the polarization image generated at the determination target imaging unit 321 or a non-polarization image generated based on the polarization image obtained using the method in the related art. For example, the detection region setting unit detects a background region and sets a region different from the background region as a target object detection region, and the processing proceeds to step ST 73.

In step ST73, the image processing apparatus calculates an error of the determination target pixel u in the target object detection region. The determination processing unit 34 of the image processing device 30 calculates an error eij for each combination of the i (where i is 0, 1.. q) th type of material and the j (where j is 0, 1.. r) th type of light source, and the processing proceeds to step ST 74.

In step ST74, the image processing apparatus detects the minimum error Emin. The determination processing unit 34 of the image processing apparatus 30 detects the minimum error Emin in a similar manner to the first operation example, and the processing proceeds to step ST 75.

In step ST75, the image processing apparatus determines whether the minimum error Emin is smaller than the threshold value Tha. The processing of the determination processing unit 34 of the image processing apparatus 30 proceeds to step ST76 in the case where it is determined in step ST74 that the detected minimum error Emin is smaller than the threshold Tha, and proceeds to step ST77 in the case where the minimum error Emin is equal to or larger than the threshold Tha.

In step ST76, the image processing apparatus determines the material having the smallest error Emin as the material of the subject corresponding to the determination target pixel u, and the processing proceeds to step 78.

In step ST77, the image processing apparatus determines that the material of the subject corresponding to the determination target pixel u is an unknown material, and the process proceeds to step ST 78.

In step ST78, the image processing apparatus determines whether the determination of the material in the target subject detection area has been completed. The process of the image processing apparatus 30 proceeds to step ST79 in the case where there is a pixel in the target object detection region for which material determination is not performed, and proceeds to step ST80 in the case where determination of each pixel within the target object detection region has been completed.

In step ST79, the image processing apparatus updates the determination target pixel. The image processing apparatus 30 selects a new pixel on which material determination has not been performed as a determination target pixel, and the process returns to step ST 73.

In step ST80, the image processing apparatus outputs the determination result. For example, the image processing apparatus 30 displays the image areas in which the materials are determined to be the same material with the same color, the same brightness, or the like based on the determination result of the material. Further, the image processing apparatus 30 displays the material as the region of the different material with the different color or brightness.

Fig. 16 shows a third operation example in which, for example, the target object is a vehicle Ga, and the road Gb includes a shadow Gc of the vehicle Ga. Fig. 16 (a) indicates the imaging range AP of the determination target imaging unit 321 with a broken line. Fig. 16 (b) shows the following case: the target subject detection area ARa is extracted from the image of the imaging range AP while excluding the background area using a related art method, and the target subject detection area ARa includes the shadow Gc of the vehicle generated on the road Gb and the vehicle Ga. Fig. 16 (c) shows the material determination result. In the case where the mueller matrix unique to the road is stored in the information storage unit 33, the road Gb in the target object detection area ARa can be detected. Therefore, a region obtained by excluding a portion of the shadow Gc of the road Gb from the target object detection region ARa can be determined as an image region of the vehicle as the target object. Therefore, for example, as shown in (d) of fig. 16, the rectangular area ARb indicating the vehicle can be detected with high accuracy.

In this way, according to the present technology, even in a case where it is difficult to determine the region of the desired object based on the color, brightness, and the like of the object, the material can be divided based on the material polarization characteristic information indicating the polarization and reflection characteristics unique to the material that are not related to the external environment, so that the region of the desired object can be determined.

<5-4. other operation examples >

In other operation examples, objects having substantially the same appearance are distinguished from each other. Fig. 17 shows an example of a case where, for example, soap, toothpaste, salt, and dairy products are used as objects having substantially the same appearance. In this case, the material polarization characteristic information of each object generated at the information generating device 20 is stored in the information storage unit 33 in advance. Further, incident polarized light information at the time of capturing an image of a material determination target is stored in advance in the information storage unit 33.

The image processing device 30 captures an image of the material determination target after calibration, and determines a material, which has the smallest error of the estimation result using the mueller matrix of each material as described above, as the material of the material determination target. Therefore, although objects having substantially the same appearance cannot be distinguished from each other from an image captured with the imaging unit 3211 in the related art as shown in (a) of fig. 17, the use of the present technology makes it possible to distinguish objects having substantially the same appearance from each other according to the difference in material based on a polarization image acquired with the determination target imaging unit 321 including the imaging unit 3211 and the polarizing plate 3212, as shown in (b) of fig. 17. Further, although not shown, even in the case where the appearances are different due to the influence of shading or the like, objects formed of substantially the same material can be distinguished from each other.

Further, the image processing device 30 may determine the material for each pixel of the polarization image acquired at the determination target imaging unit 321, and thus, the determination result may be presented in units of images. Fig. 18 shows an example of a case where the determination result is presented in units of pixels. Fig. 18 (a) shows an example of a case where the polarization image acquired at the determination target imaging unit 321 including the imaging unit 3211 and the polarizing plate 3212 includes the objects OBa, OBb, and OBc. Fig. 18(b) shows an example of the determination result, and shows only the objects OBa, OBc of which the material is determined. Further, the error is calculated as described above, and therefore, as shown in (c) of fig. 18, the gradation or the like of the object can be set according to the error so that the determination accuracy of the material can be recognized.

Further, in the case where a target to be determined on a material is set indoors, the image processing apparatus may use a light source suitable for the material. In this case, in the case of using a light source suitable for the material, the information generating device can determine the material with high accuracy by generating the material polarization characteristic information.

<6. other configurations and operations >

Incidentally, although an example of a case where a polarizing plate is disposed in front of the imaging unit and a plurality of polarization images having different polarization directions are acquired by performing imaging while rotating the polarizing plate in the above-described information generating device and image processing device has been described, other methods may be used to acquire the polarization images.

Fig. 19 shows an example of another method for acquiring a polarization image. For example, as shown in (a) of fig. 19, a polarization image is generated by performing imaging in a polarization image acquisition unit in which a polarization filter 502 including an arrangement of pixels of a plurality of polarization directions is set at an image sensor 501. Note that (a) of fig. 19 shows an example of a case where a polarization filter 502 including pixels in four different polarization directions (the polarization directions are indicated by arrows) is arranged in front of the image sensor 501. Further, as shown in fig. 19(b), the polarization image acquiring unit may generate a plurality of polarization images of different polarization directions using the configuration of the multi-lens array. For example, a plurality of (four in the drawing) lenses 503 are provided on the front surface of the image sensor 501, and optical images of an object are respectively formed on the imaging surface of the image sensor 501 by the respective lenses 503. Further, a polarizing plate 504 is provided on the front surface of each lens 503, and a plurality of polarized images of different polarization directions are generated while setting different directions as the polarization directions of the polarizing plate 504. This configuration of the polarization image acquisition unit enables a plurality of polarization images to be acquired by one imaging. Note that the polarized light image acquiring unit may generate polarized light images of three primary colors by providing a color filter at the image sensor 501. Further, a plurality of polarized light images may be generated by the imaging unit provided for each polarization direction.

<7. application example >

The technology according to the present disclosure may be applied to various fields. For example, the technology according to the present disclosure may also be implemented as a device installed in any type of moving body, such as an automobile, an electric automobile, a hybrid automobile, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, or a robot. For example, the environment around the driver is displayed to the driver using the determination result of the material, so that the driver can grasp the environment more easily, and the fatigue of the driver can be reduced. Further, safer automatic driving or the like can be realized. Further, by applying the present technology to an apparatus or the like used in a factory production process, it is possible to prevent components or the like of different materials from being mixed in. Further, for the monitoring field, by applying the present technology, it is possible to realize a monitoring operation for a material and, for example, a shape and a motion of an object by using a determination result of the material.

The series of processes described in this specification may be performed by hardware, software, or a combined configuration of hardware and software. In the case where the processing is performed by software, a program in which a processing sequence is recorded may be installed in a memory embedded in a computer of dedicated hardware to execute the program. Alternatively, the program may be installed in a general-purpose computer capable of executing various processes to be executed.

For example, the program may be recorded in advance on a hard disk, a Solid State Drive (SSD), or a Read Only Memory (ROM) as a recording medium. Alternatively, the program may be temporarily or permanently stored (recorded) in (on) a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto-optical disk (MO) disk, a Digital Versatile Disc (DVD), a blu-ray disc (registered trademark) (BD), a magnetic disk, or a semiconductor memory card. Such a removable recording medium may be provided as so-called package software.

In addition, the program may not only be installed on the computer from a removable recording medium, but also be delivered to the computer wirelessly or by wire from a download site via a network such as a LAN (local area network) or the internet. In such a computer, the program transferred in the above-described manner may be received and installed on a recording medium such as a built-in hardware.

Further, the effects described in this specification are not restrictive but merely examples, and there may be additional effects not described above. Furthermore, the present technology should not be construed as being limited to the above-described embodiments of the present technology. The present technology is disclosed in an embodiment thereof by way of example, and it is apparent to those skilled in the art that modifications or substitutions may be made thereto without departing from the spirit of the present technology. The claims should be considered to determine the present technology.

The image processing apparatus according to the present technology may be configured as follows.

(1) An image processing apparatus comprising:

a determination environment information acquisition unit configured to acquire incident polarized light information of a light source in a material determination environment;

a determination target information acquisition unit configured to acquire outgoing polarized light information from a polarized light image obtained by capturing an image of a material determination target in the material determination environment; and

a determination processing unit configured to determine a material of the material determination target based on the incident polarized light information acquired at the determination environment information acquiring unit, the exit polarized light information acquired at the determination target information acquiring unit, and material polarization characteristic information indicating polarization and reflection characteristics of each incident direction of the incident polarized light and each exit direction of the reflected light and generated in advance.

(2) The image processing apparatus according to (1), wherein the determination processing unit calculates an error of one of the outgoing polarized light information and the incoming polarized light information estimated using the material polarization characteristic information selected from the incoming direction of the incoming polarized light on the material determination target, the outgoing direction of the reflected light from the material determination target, and the other of the incoming polarized light information acquired at the determination environment information acquisition unit and the outgoing polarized light information acquired at the determination target information acquisition unit, and determines the material of the material determination target based on the calculated error.

(3) The image processing apparatus according to (2), wherein the determination processing unit generates estimated exit polarized light information using the selected material polarization characteristic information and the incident polarized light information, and determines the material of the material determination target based on an error between the estimated exit polarized light information and the exit polarized light information acquired at the determination target information acquiring unit.

(4) The image processing apparatus according to (2), wherein the determination processing unit calculates estimated incident polarized light information using the selected material polarization characteristic information and the outgoing polarized light information acquired at the determination target information acquisition unit, and determines the material of the material determination target based on an error between the estimated incident polarized light information and the incident polarized light information acquired at the determination environment information acquisition unit.

(5) The image processing apparatus according to any one of (2) to (4), wherein the material polarization characteristic information is generated for each of a plurality of materials, and

the determination processing unit selects material polarization characteristic information corresponding to the incident direction of the incident polarized light and the exit direction of the reflected light among the material polarization characteristic information according to each material, calculates an error for each material, and determines a material having the smallest error as a material of the material determination target.

(6) The image processing apparatus according to (5), wherein the material polarization characteristic information is generated for each of a plurality of materials, and

in a case where a minimum error of the calculated errors is smaller than a threshold value set in advance, the determination processing unit determines a material having the minimum error as a material of the material determination target.

(7) The image processing apparatus according to any one of (2) to (6), wherein the determination environment information acquisition unit acquires incident polarized light information of each of a plurality of light sources in the material determination environment, and

the determination processing unit calculates the error using incident polarized light information of each of the plurality of light sources, and determines a material having the smallest error as a material of the material determination target.

(8) The image processing apparatus according to any one of (2) to (6), wherein the determination environment information acquisition unit acquires incident polarized light information of each of a plurality of light sources in the material determination environment, and

the determination processing unit calculates the error using incident polarized light information of a light source selected from the incident polarized light information of each of the plurality of light sources, and determines a material having the smallest error as the material of the material determination target.

(9) The image processing apparatus according to any one of (1) to (6), further comprising: a detection region setting unit configured to set a target object detection region from a polarization image obtained by capturing an image of the material determination target; and

a region detection unit configured to detect a target subject region from the target subject detection region set at the detection region setting unit based on a material determination result at the determination processing unit.

(10) The image processing apparatus according to any one of (1) to (9), wherein the material polarization characteristic information and incident polarization information acquired at the determination environment information acquisition unit are stored in an information storage unit in advance, and the determination processing unit determines the material of the material determination target using the incident polarization information and the material polarization characteristic information stored in the information storage unit.

(11) The image processing apparatus according to any one of (1) to (10), wherein the determination environment information acquisition unit divides the material determination environment into a plurality of regions, and sets an average incident direction and average incident polarization information of each region as the incident direction and incident polarization information of the region.

(12) The image processing apparatus according to any one of (1) to (11), wherein the determination processing unit determines the material of the material determination target based on the normalized incident polarized light information, exit polarized light information, and material polarization characteristic information.

In addition, the information generating apparatus of the present technology may also be configured as follows.

(1) An information generating apparatus comprising:

a light source information acquisition unit configured to acquire incident polarized light information of incident polarized light with respect to an information generation target that is material-conspicuous, for each incident direction, from a light source in a measurement environment in which the information generation target is set;

an exit polarized light information acquisition unit configured to acquire exit polarized light information of reflected light from the information generation target for each exit direction; and

a material polarization characteristic information generation unit configured to generate material polarization characteristic information indicating polarization and reflection characteristics in an incident direction of the incident polarized light and an exit direction of the reflected light for each direction using the incident polarized light information acquired at the light source information acquisition unit and the exit polarized light information acquired at the exit polarized light information acquisition unit.

(2) The information generating apparatus according to (1), wherein a plurality of materials are provided,

the light source information acquiring unit acquires incident polarized light information of each material,

the outgoing polarized light information acquiring unit acquires outgoing polarized light information of each material, and

the material polarization characteristic information generation unit generates material polarization characteristic information indicating polarization and reflection characteristics for each incident direction and each exit direction for each material.

(3) The information generation device according to (1) or (2), wherein the outgoing polarized light information acquisition unit acquires the outgoing polarized light information based on an observation value of a polarized image generated at an information generation target imaging unit and an information generation target imaging unit configured to generate polarized images in a plurality of polarization directions by capturing an image of the information generation target

(4) The information generating apparatus according to any one of (1) to (3), wherein the material polarization characteristic information generating unit generates normalized material polarization characteristic information.

(5) The information generating apparatus according to any one of (1) to (4), wherein the incident polarized light information indicates a stokes vector of the incident polarized light, the exit polarized light information indicates a stokes vector of the reflected light, and the material polarization characteristic information indicates a mueller matrix.

List of reference numerals

10 material determination system

20 information generating device

21 light source information acquisition unit

22 emergent polarized light information acquisition unit

23 Material polarization characteristic information generating Unit

30 image processing device

31 determining environment information acquiring unit

32 determination target information acquisition unit

33 information storage unit

34 determining processing unit

50 database unit

211 light source imaging unit

212 incident polarized light information calculating unit

221 known material imaging unit

222 emergent polarized light information calculating unit

231 polarization and reflection characteristic calculation unit

232 material polarization characteristic information generation unit

311 ambient imaging unit

312 incident polarized light information calculating unit

321 determine a target imaging unit

322 emergent polarized light information calculating unit

341 estimation processing unit

342 error calculation unit

343 material determination processing unit

501 image sensor

502 polarizing filter

503 lens

504 polarizing plate

3111. 3211 image forming unit

3112. 3212 polarizing plate