阅读说明：本技术 摄像装置、图像处理方法和存储介质 (Image pickup apparatus, image processing method, and storage medium ) 是由 井田义明 井上智晓 楠美祐一 池本圣雄 于 2019-06-27 设计创作，主要内容包括：本发明提供摄像装置、图像处理方法和存储介质。该装置包括：摄像元件,其用于通过不同像素接收通过光学系统的光瞳的不同区域的多个光束,来获得多个视差图像；以及处理器,其用于对通过使用摄像元件获得的图像进行图像处理。处理器包括第一获得单元、第二获得单元和第三获得单元,第一获得单元基于通过使用摄像元件在包括至少三个不同光源条件的光源条件下摄影的多个光度立体图像,来获得第一形状信息,第二获得单元通过使用所述多个视差图像来获得第二形状信息,第三获得单元通过使用第一形状信息和第二形状信息来获得第三形状信息。(The invention provides an imaging apparatus, an image processing method, and a storage medium. The device includes: an image pickup element for receiving a plurality of light fluxes passing through different regions of a pupil of an optical system by different pixels to obtain a plurality of parallax images; and a processor for performing image processing on an image obtained by using the image pickup element. The processor includes a first obtaining unit that obtains first shape information based on a plurality of photometric stereo images photographed under a light source condition including at least three different light source conditions by using an image pickup element, a second obtaining unit that obtains second shape information by using the plurality of parallax images, and a third obtaining unit that obtains third shape information by using the first shape information and the second shape information.)

1. An apparatus, comprising:

an image pickup element for receiving a plurality of light fluxes passing through different regions of a pupil of an optical system by different pixels to obtain a plurality of parallax images; and

a processor for performing image processing on an image obtained by using the image pickup element,

wherein the processor includes a first obtaining unit that obtains first shape information of the object based on a plurality of photometric stereo images photographed under a light source condition including at least three different light source conditions by using the image pickup element, a second obtaining unit that obtains second shape information of the object by using the plurality of parallax images, and a third obtaining unit that obtains third shape information of the object by using the first shape information and the second shape information.

2. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the first shape information is surface normal information and the second shape information is distance information.

3. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the region of the pupil through which the light beam passes when photographing an image included in the plurality of photometric stereo images includes at least a part of the region through which the light beam passes when photographing at least one of the plurality of parallax images.

4. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein an area of a pupil through which the light flux passes when photographing each image included in the plurality of parallax images and an area through which the light flux passes when photographing an image included in the plurality of photometric stereoscopic images have a common portion.

5. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the images included in the plurality of photometric stereo images are obtained by adding all pixels related to the plurality of parallax images.

6. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein an area of a pupil through which the light beam passes when an image included in the plurality of photometric stereo images is photographed is the same as an area through which the light beam passes when at least one of the plurality of parallax images is photographed.

7. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the second shape information is distance information, and

wherein each image included in the plurality of photometric stereo images is obtained by a pixel associated with a reference image of the plurality of parallax images on which distance information is based.

8. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the light source condition is a condition of turning on a plurality of light sources or turning off a combination of the plurality of light sources, and

wherein all of the plurality of light sources are turned off when the plurality of parallax images are photographed.

9. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the light source condition is a condition of turning on a plurality of light sources or turning off a combination of the plurality of light sources, and

wherein all of the plurality of light sources are turned on when the plurality of parallax images are photographed.

10. The apparatus of claim 8, wherein the first and second electrodes are disposed on opposite sides of the substrate,

wherein the apparatus comprises a light emitter comprising the plurality of light sources.

11. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the plurality of parallax images include images obtained by applying a weighted average of images included in the plurality of photometric stereo images.

12. The apparatus of claim 1, wherein the first and second electrodes are disposed on opposite sides of the housing,

wherein the plurality of parallax images include images obtained by using one of the plurality of photometric stereo images.

13. The apparatus of any one of claims 1 to 12,

wherein an exposure condition when an image included in the plurality of photometric stereo images is photographed is different from an exposure condition when the plurality of parallax images is photographed.

14. The apparatus of any one of claims 1 to 13,

wherein the third shape information is surface normal information or distance information.

15. An image processing method, the image processing method comprising:

obtaining first shape information based on a plurality of photometric stereo images photographed under a light source condition including at least three different light source conditions by using an image pickup element for receiving a plurality of light fluxes passing through different regions of a pupil of an optical system by different pixels to obtain a plurality of parallax images;

obtaining second shape information by using the plurality of parallax images; and

the third shape information is obtained by using the first shape information and the second shape information.

16. The image processing method according to claim 15,

wherein the region of the pupil through which the light beam passes when photographing an image included in the plurality of photometric stereo images includes at least a part of the region through which the light beam passes when photographing at least one of the plurality of parallax images.

17. The image processing method according to claim 15,

wherein an area of a pupil through which the light flux passes when photographing each image included in the plurality of parallax images and an area through which the light flux passes when photographing an image included in the plurality of photometric stereoscopic images have a common portion.

18. A storage medium in which a program that causes a computer to perform a method comprising:

obtaining first shape information based on a plurality of photometric stereo images photographed under a light source condition including at least three different light source conditions by using an image pickup element for receiving a plurality of light fluxes passing through different regions of a pupil of an optical system by different pixels to obtain a plurality of parallax images;

obtaining second shape information by using the plurality of parallax images; and

the third shape information is obtained by using the first shape information and the second shape information.

19. The storage medium of claim 18 wherein the storage medium,

wherein the region of the pupil through which the light beam passes when photographing an image included in the plurality of photometric stereo images includes at least a part of the region through which the light beam passes when photographing at least one of the plurality of parallax images.

20. The storage medium of claim 18 wherein the storage medium,

wherein an area of a pupil through which the light flux passes when photographing each image included in the plurality of parallax images and an area through which the light flux passes when photographing an image included in the plurality of photometric stereoscopic images have a common portion.

Technical Field

Background

When capturing an object, image processing can be performed based on the physical model after image capturing in such a manner that the object is captured so that shape information (e.g., distance information and surface normal information) can be obtained. For this reason, it is desired to develop a technique that enables accurate shape information to be easily obtained.

Japanese patent laid-open No. s 61-198015 discloses that the calculation speed of distance information can be increased in such a manner that a photometric stereo method (which is known as a method of directly obtaining a surface normal of an object) is used together with the distance measurement of a stereo camera.

In the method disclosed in japanese patent laid-open No. s 61-198015, an image pickup region is divided into a plurality of regions based on surface normal information obtained by a photometric stereo method, distances between the centers of gravity of the divided regions are obtained from a parallax image by a stereo camera, and relative distances in an object located in each region are obtained from the surface normal information obtained by the photometric stereo method.

However, with this method, a plurality of optical systems are used to obtain a parallax image. Therefore, the device is complicated, and the device size increases. In addition, it is difficult to calculate the matching between parallax images due to the aberration and shading of the optical system. Further, a structure that provides calibration of each optical system of the stereo camera and a relationship between the optical systems is required.

Disclosure of Invention

Drawings

Fig. 1 illustrates an image pickup apparatus according to an embodiment.

Fig. 2 illustrates an image pickup apparatus according to an embodiment.

Fig. 3 is a block diagram of an image pickup apparatus according to an embodiment.

Fig. 4 is a flowchart illustrating image processing performed by the image processor of the image pickup apparatus according to the embodiment.

Fig. 5 illustrates an image capturing apparatus and an image processing apparatus according to another embodiment.

Fig. 6 is a block diagram of an image processing apparatus according to another embodiment.

Fig. 7 is a flowchart illustrating image processing performed by an image processing apparatus according to another embodiment.

Fig. 8 illustrates a specular reflection component.

Fig. 9 illustrates a dual view photographing method.

Fig. 10 illustrates a relationship between a light receiving unit of an image pickup element and a pupil of an image pickup optical system according to the embodiment.

Fig. 11 illustrates a relationship between a light receiving unit of an image pickup element and an object according to an embodiment.

Aspects of the embodiments relate to an image processing method and an image pickup apparatus capable of calculating shape information of an object.