阅读说明：本技术 纵深取得装置、纵深取得方法以及程序 (Depth acquisition device, depth acquisition method, and program ) 是由 吾妻健夫 佐藤智 若井信彦 吉冈康介 清水规贵 河合良直 天田高明 川合阳子 村井猛 于 2019-09-11 设计创作，主要内容包括：能正确取得到被摄体的纵深的纵深取得装置(1)具备存储器(200)和处理器(110a)。处理器(110a),取得红外光的强度,该红外光的强度通过基于被摄体的红外光的摄像而被测量,并被保存在存储器(200),通过基于该红外光的强度算出到被摄体的距离作为纵深,来生成纵深图像,取得BW图像,该BW图像通过基于红外光的强度的与IR图像实质相同的场景、相同视点以及相同时刻的摄像而生成,并被保持在存储器(200),从该BW图像检测包含沿着与BW图像的变动的方向垂直的方向的边缘在内的区域即边缘区域,对与该边缘区域对应的纵深图像内的区域即补正对象区域的纵深进行补正。(A depth acquisition device (1) capable of accurately acquiring the depth of an object is provided with a memory (200) and a processor (110 a). A processor (110a) acquires the intensity of infrared light, which is measured by imaging with infrared light from an object and stored in a memory (200), generates a depth image by calculating the distance to the object as the depth based on the intensity of infrared light, acquires a BW image, which is generated by imaging with the same scene, the same viewpoint, and the same time as the IR image based on the intensity of infrared light and is stored in the memory (200), detects an edge region, which is a region including an edge in a direction perpendicular to the direction of fluctuation of the BW image, from the BW image, and corrects the depth of a correction target region, which is a region in the depth image corresponding to the edge region.)

1. A depth acquisition device includes a memory and a processor,

the processor is used for processing the data to be processed,

acquiring intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging in which the infrared light is received by a plurality of pixels included in an imaging element, respectively, and being stored in the memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

detecting an edge region including an edge in a direction perpendicular to a direction of fluctuation of the visible light image from the visible light image,

and correcting the depth of a correction target region which is a region in the depth image corresponding to the edge region.

2. The depth retrieval device according to claim 1,

in the detection of the edge region in question,

detecting the edge having an intensity above a 1 st threshold,

the edge region is a region from the edge having the intensity of the 1 st threshold or more to a distance corresponding to a variation in the visible light image.

3. The depth retrieval device according to claim 1,

in the detection of the edge region in question,

and detecting a region in which a difference between pixel values of the visible light image and a previous visible light image is equal to or greater than a 2 nd threshold as the edge region.

4. The depth access device according to any one of claims 1 to 3,

in the correction of the correction target region, the depth of the correction target region is corrected using the depth of a peripheral region that is a region located around the correction target region in the depth image.

5. The depth retrieval device according to claim 4,

the peripheral region is a region which is in contact with a lower side of the correction target region,

in the correction of the depth of the correction target region, the depth of the correction target region is replaced with the depth of the peripheral region.

6. The depth access device according to any one of claims 1 to 3,

in the correction of the depth of the correction target region, the depth of the correction target region is corrected by filtering the depth image using the visible light image as a reference image.

7. The depth access device according to any one of claims 1 to 3,

in the correction of the depth of the correction target region,

the depth of the correction target region in the depth image is corrected by inputting the infrared light image, the visible light image, the depth image, and the edge region into a learning model.

8. A depth acquisition device includes a memory and a processor,

the processor is used for processing the data to be processed,

acquiring intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging in which the infrared light is received by a plurality of pixels included in an imaging element, respectively, and being stored in the memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

the depth of the depth image is corrected by inputting the depth image, the infrared light image, and the visible light image to a learning model.

9. A depth acquisition device is provided with:

an infrared light camera having an image pickup element that measures an intensity of infrared light irradiated from a light source and reflected by an object by performing image pickup in which a plurality of pixels included in the image pickup element respectively receive the infrared light;

a depth calculating unit that calculates, for each of a plurality of pixels included in the image pickup device, a distance to the subject as a depth based on the intensity of the infrared light received at the pixel, and generates a depth image;

a visible light camera that generates a visible light image by performing visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element of the infrared light camera and imaging at a viewpoint and a time substantially the same as the infrared light image;

an edge region detection unit that detects an edge region, which is a region including an edge along a direction perpendicular to a direction of fluctuation of the visible light image, from the visible light image; and

and a depth correction unit that corrects a depth of a correction target region that is a region in the depth image corresponding to the edge region.

10. A depth acquisition device is provided with:

an infrared light camera having an image pickup element that measures an intensity of infrared light irradiated from a light source and reflected by an object by performing image pickup in which a plurality of pixels included in the image pickup element respectively receive the infrared light;

a depth calculating unit that calculates, for each of a plurality of pixels included in the image pickup device, a distance to the subject as a depth based on the intensity of the infrared light received at the pixel, and generates a depth image;

a visible light camera that generates a visible light image by performing visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element of the infrared light camera and imaging at a viewpoint and a time substantially the same as the infrared light image; and

and a depth correction unit configured to correct a depth of the depth image by inputting the depth image, the infrared light image, and the visible light image into a learning model.

11. A method for obtaining depth of a semiconductor wafer,

acquiring intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging the infrared light received by a plurality of pixels included in an imaging element, respectively, and being stored in a memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

detecting an edge region including an edge in a direction perpendicular to a direction of fluctuation of the visible light image from the visible light image,

and correcting the depth of a correction target region which is a region in the depth image corresponding to the edge region.

12. A method for obtaining depth of a semiconductor wafer,

acquiring intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging the infrared light received by a plurality of pixels included in an imaging element, respectively, and being stored in a memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

the depth of the depth image is corrected by inputting the depth image, the infrared light image, and the visible light image to a learning model.

13. A program for taking a distance of an object as a depth, the program causing a computer to execute:

acquiring intensity of infrared light irradiated from a light source and reflected by the object, the intensity of the infrared light being measured by imaging in which the infrared light is received by a plurality of pixels included in an imaging element, respectively, and being stored in a memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

detecting an edge region including an edge in a direction perpendicular to a direction of fluctuation of the visible light image from the visible light image,

and correcting the depth of a correction target region which is a region in the depth image corresponding to the edge region.

14. A program for taking a distance of an object as a depth, the program causing a computer to execute:

acquiring intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging the infrared light received by a plurality of pixels included in an imaging element, respectively, and being stored in a memory,

calculating a distance to the subject as a depth for each of a plurality of pixels included in the image pickup element based on the intensity of the infrared light received at the pixel to generate a depth image,

acquiring a visible light image generated by visible light-based imaging of a scene substantially the same as an infrared light image formed based on the intensity of the infrared light received by each of a plurality of pixels included in the imaging element and imaging at a viewpoint and time substantially the same as the infrared light image, and being held in the memory,

the depth of the depth image is corrected by inputting the depth image, the infrared light image, and the visible light image to a learning model.

Technical Field

The present disclosure relates to a depth acquisition device and the like that acquire a depth as a distance of an object.

Background

Conventionally, a distance measuring device for measuring a distance to an object has been proposed (for example, see patent document 1). The distance measuring device includes a light source and an imaging unit. The light source irradiates light to the subject. The imaging unit images the reflected light reflected by the subject. Then, the distance measuring device converts each pixel value of the image obtained by the image capturing into a distance to the object, thereby measuring the distance to the object. That is, the distance measuring device acquires the depth of the image obtained by the imaging unit.

Documents of the prior art

Patent document

Patent document 1: JP 2011-

Disclosure of Invention

Problems to be solved by the invention

However, the distance measuring device of patent document 1 has a problem that the depth cannot be accurately obtained.

Therefore, the present disclosure provides a depth acquisition device capable of accurately acquiring the depth, which is the distance to the subject.

Means for solving the problems

A depth acquisition device according to an aspect of the present disclosure includes a memory and a processor, the processor acquiring an intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging in which the infrared light is received by a plurality of pixels included in an imaging element, the infrared light being stored in the memory, calculating a distance to the object as a depth for each of the plurality of pixels included in the imaging element based on the intensity of the infrared light received by the pixel, and acquiring a depth image, the visible light image being generated by imaging in a visible light manner and at a viewpoint and a time substantially the same as those of the infrared light image based on a scene substantially the same as the infrared light image formed based on the intensity of the infrared light received by the plurality of pixels included in the imaging element, and an edge region, which is a region including an edge along a direction perpendicular to a direction of fluctuation (control き) of the visible light image, is detected from the visible light image, and a depth of a correction target region, which is a region in the depth image corresponding to the edge region, is corrected.

These general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. In addition, the recording medium may be a non-transitory recording medium.

ADVANTAGEOUS EFFECTS OF INVENTION

The depth acquisition device of the present disclosure can accurately acquire the depth, which is the distance to the subject. Further advantages and effects in an aspect of the present disclosure will be apparent from the description and the accompanying drawings. The advantages and/or effects are provided by the embodiments and the features described in the specification and the drawings, respectively, but not necessarily all of them are provided to obtain 1 or more of the same features.

Drawings

Fig. 1 is a block diagram showing a hardware configuration of a depth acquisition device according to an embodiment.

Fig. 2 is a schematic diagram showing a pixel array included in the solid-state imaging device according to the embodiment.

Fig. 3 is a timing chart showing a relationship between light emission timing of a light emitting element of a light source and exposure timing of the 1 st pixel of a solid-state imaging element in the embodiment.

Fig. 4 is a block diagram showing an example of a functional configuration of the depth acquisition device according to the embodiment.

Fig. 5 shows an example of a BW image and an IR image.

Fig. 6 is a flowchart showing the overall processing operation of the depth acquisition device in the embodiment.

Fig. 7 is a flowchart showing an example of the processing operation of the boundary area detection unit in the embodiment.

Fig. 8 is a diagram for explaining edge detection in the embodiment.

Fig. 9 is a flowchart showing another example of the processing operation of the boundary area detection unit in the embodiment.

Fig. 10 is a block diagram showing an example of a functional configuration of a depth acquisition device in a modification of the embodiment.

Fig. 11A is a diagram showing an example of a simulation result of the depth acquisition device according to the embodiment.

Fig. 11B is a diagram showing another example of the simulation result of the depth acquisition device according to the embodiment.

Detailed Description

(insight underlying the present disclosure)

The inventors of the present invention have found that the following problems occur with the distance measuring device of patent document 1 described in the section of "background art".

As described above, the distance measuring device of patent document 1 irradiates a subject with light from a light source, captures the subject irradiated with the light, acquires an image, and measures the depth of the image. ToF (Time Of Flight) is used for the depth measurement. In such a distance measuring device, imaging under different imaging conditions is performed to improve the distance measuring accuracy. That is, the distance measuring device performs imaging under a predetermined imaging condition, and sets an imaging condition different from the predetermined imaging condition according to the imaging result. Then, the distance measuring device performs imaging again in accordance with the set imaging conditions.

However, when imaging the boundary of 2 objects having different light reflectances, the distance measuring device of patent document 1 may have difficulty in accurately measuring the depth around the boundary even if the imaging conditions are changed.

In order to solve the above-described problem, a depth acquisition device according to one aspect of the present disclosure includes a memory and a processor, the processor acquiring an intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging in which the infrared light is received by a plurality of pixels included in an imaging element, the intensity being stored in the memory, calculating a distance to the object as a depth based on the intensity of the infrared light received by the pixel for each of the plurality of pixels included in the imaging element, and acquiring a depth image, the visible light image being generated by imaging in a substantially same viewpoint and time as the infrared light image and based on imaging in a substantially same scene as the infrared light image formed based on the intensity of the infrared light received by the pixels included in the imaging element, and an edge region, which is a region including an edge along a direction perpendicular to a direction of fluctuation of the visible light image, is detected from the visible light image, and the depth of a correction target region, which is a region in the depth image corresponding to the edge region, is corrected. The region in the depth image corresponding to the edge region is a region in the depth image that is located at the same position as the edge region in the visible light image and has the same shape and size as the edge region.

For example, when imaging a boundary of 2 objects having different reflectances of infrared light, if the camera used for the imaging is changed, the reflectance at the measurement target position around the boundary may be greatly changed in the ToF-based measurement. In such a case, noise is obtained at the measurement target position, and a depth image indicating an incorrect depth is acquired.

In the depth acquisition device according to the above-described aspect, however, a region including an edge along a direction perpendicular to the direction of the fluctuation of the visible light image is detected from the visible light image as the edge region. The edge included in the edge area corresponds to the boundary of the 2 objects described above, for example. That is, the edge region is a region around the above-described boundary where the possibility that the reflectance of infrared light greatly changes in ToF measurement is high. Then, a region in the depth image corresponding to the edge region is set as a correction target region, and the depth of the correction target region is corrected to reduce noise, so that the depth around the boundary can be accurately obtained.

Here, an example of an image of a substantially identical scene captured at substantially the same viewpoint and time is an image captured by different pixels of the same imaging device. Such an image has substantially the same viewing angle, viewpoint, and imaging timing as those of each of red, green, and blue channel images of a color image captured by a color filter of Bayer (Bayer) arrangement. That is, the positions of images of substantially the same scene captured at substantially the same viewpoint and time on the image of the subject in each captured image do not differ by 2 pixels or more. For example, when a point light source having visible light and infrared components exists in a scene and only 1 pixel is imaged with high luminance in a visible light image, the point light source is also imaged closer to 2 pixels of a pixel corresponding to the pixel position imaged in the visible light image in an infrared light image. The imaging at substantially the same time point means that the difference between the imaging time points is equal to 1 frame or less.

The edge region corresponds to a light/dark boundary region. That is, the processor detects, from the visible light image, a bright-dark boundary region that is a region including a boundary along a direction perpendicular to a direction of variation of the visible light image, the boundary being a boundary of 2 regions having mutually different luminances. In this case, the processor corrects the depth of the correction target region which is a region in the depth image corresponding to the bright-dark boundary region.

In the detection of the edge region, the edge having an intensity equal to or higher than a 1 st threshold may be detected, and the edge region may be a region from the edge having an intensity equal to or higher than the 1 st threshold to a distance corresponding to a variation in the visible light image.

This enables the edge region to be appropriately detected. As a result, a region in which noise is likely to occur can be appropriately detected in the depth image.

In the detection of the edge region, a region in which a difference between pixel values of the visible light image and a previous visible light image is equal to or greater than a 2 nd threshold may be detected as the edge region.

This enables simple detection of the edge region.

In the correction of the correction target region, the depth of the correction target region may be corrected using the depth of a peripheral region that is a region located around the correction target region in the depth image. For example, the peripheral region may be a region that is in contact with a lower side of the correction target region, and the depth of the correction target region may be replaced with the depth of the peripheral region in correcting the depth of the correction target region.

The depth of the peripheral region is likely to be approximated to the correct depth of the region to be corrected. Therefore, the depth of the correction target region can be accurately corrected by using the depth of the peripheral region.

In the correction of the depth of the correction target region, the depth of the correction target region may be corrected by filtering the depth image using the visible light image as a reference image.

For example, a spreading Filter such as a pilot Filter (Guided Filter) may be used for the filtering. Thus, the depth of the correction target region can be corrected accurately.

In the correction of the depth of the correction target region, the depth of the correction target region in the depth image may be corrected by inputting the infrared light image, the visible light image, the depth image, and the edge region into a learning model.

Thus, if the learning model is learned in advance so that the corrected depth image having a correct solution is output for the input of the depth image, the infrared light image, the visible light image, and the edge region, an appropriate depth image can be obtained easily without detecting the edge region.

The depth acquisition device according to another aspect of the present disclosure may further include a memory and a processor, the processor acquiring an intensity of infrared light irradiated from a light source and reflected by an object, the intensity of the infrared light being measured by imaging in which the infrared light is received by each of a plurality of pixels included in an imaging element, the processor being stored in the memory, calculating a distance to the object as a depth based on the intensity of the infrared light received by each of the pixels for each of the plurality of pixels included in the imaging element, and acquiring a depth image, the visible light image being generated by imaging in a substantially same scene as an infrared light image based on the intensity of the infrared light received by each of the plurality of pixels included in the imaging element, the visible light image being generated by imaging in a substantially same viewpoint and time as the infrared light image based on the visible light image, and a correction unit for correcting the depth of the depth image by inputting the depth image, the infrared light image, and the visible light image into a learning model.

Thus, if the learning model is learned in advance so that the corrected depth image having a correct solution is output for the input of the depth image, the infrared light image, and the visible light image, an appropriate depth image can be obtained easily without detecting an edge region.

These general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of a system, a method, an integrated circuit, a computer program, or a recording medium. In addition, the recording medium may be a non-transitory recording medium.

The embodiments are described below in detail with reference to the accompanying drawings.

The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of the steps, and the like shown in the following embodiments are examples, and do not limit the present disclosure. In addition, among the components in the following embodiments, components not described in the independent claims representing the highest concept are described as arbitrary components.

The drawings are schematic and not necessarily strictly illustrated. In the drawings, the same constituent elements are denoted by the same reference numerals.

(embodiment mode)

[ hardware configuration ]

Fig. 1 is a block diagram showing a hardware configuration of a depth acquisition apparatus 1 according to an embodiment. The depth acquisition device 1 in the present embodiment has a hardware configuration capable of acquiring an image based on infrared light (or near-infrared light) and an image based on visible light by imaging in substantially the same scene and at substantially the same viewpoint and time. The term "substantially the same" means the same as the degree that the effects of the present disclosure can be obtained.

As shown in fig. 1, the depth acquisition device 1 includes a light source 10, a solid-state imaging element 20, a processing circuit 30, a diffusion plate 50, a lens 60, and a band-pass filter 70.

The light source 10 irradiates irradiation light. More specifically, the light source 10 emits irradiation light to be irradiated to the object at a timing indicated by the light emission signal generated in the processing circuit 30.

The light source 10 is configured to include, for example, a capacitor, a driving circuit, and a light emitting element, and emits light by driving the light emitting element with electric energy stored in the capacitor. The light emitting element is realized by a laser diode, a light emitting diode, or the like, as an example. The light source 10 may include 1 type of light-emitting element, or may include a plurality of types of light-emitting elements according to the purpose.

Hereinafter, the light emitting element is, for example, a laser diode that emits near-infrared light, a light emitting diode that emits near-infrared light, or the like. However, the irradiation light irradiated from the light source 10 is not necessarily limited to near-infrared light. The irradiation light irradiated by the light source 10 may be, for example, infrared light (also referred to as infrared light) in a frequency band other than near-infrared light. In the present embodiment, the irradiation light irradiated from the light source 10 is described as infrared light, but the infrared light may be near-infrared light or infrared light in a frequency band other than near-infrared light.

The solid-state imaging element 20 images an object and outputs an imaging signal indicating an exposure amount. More specifically, the solid-state imaging element 20 performs exposure at a timing indicated by an exposure signal generated in the processing circuit 30, and outputs an imaging signal indicating an exposure amount.

The solid-state imaging element 20 includes a pixel array in which 1 st pixels for imaging using reflected light of irradiation light reflected by an object and 2 nd pixels for imaging the object are arranged in an array. The solid-state imaging element 20 may have logic functions such as a cover glass and an AD converter as necessary.

In the following description, the reflected light is infrared light as in the case of the irradiation light, but the reflected light is not necessarily limited to infrared light as long as the irradiation light is reflected by the object.

Fig. 2 is a schematic diagram showing the pixel array 2 included in the solid-state imaging element 20.

As shown in fig. 2, the pixel array 2 is configured by being arranged in an array form such that 1 st pixels 21(IR pixels) for performing imaging using reflected light in which irradiation light is reflected by an object and 2 nd pixels 22(BW pixels) for imaging the object are alternately arranged in units of columns.

In fig. 2, the 2 nd pixel 22 and the 1 st pixel 21 are arranged in the pixel array 2 so as to be adjacent to each other in the row direction and arranged in a stripe shape in the row direction, but the present invention is not limited thereto, and may be arranged every plural rows (every 2 rows as an example). That is, the 1 st and 1 st pixels 22 and 21 adjacent in the row direction may be arranged such that the 2 nd rows adjacent in the row direction are alternately arranged every M rows (M is a natural number). Further, the 2 nd pixels 22 may be arranged so that the 1 st row and the 1 st row adjacent to each other in the row direction are arranged so that the 2 nd row adjacent to each other in the row direction is separated by different rows (the 1 st row N and the 2 nd row L are alternately repeated (N and L are different natural numbers)).

The 1 st pixel 21 is realized by, for example, an infrared light pixel having sensitivity to infrared light as reflected light. The 2 nd pixel 22 is implemented, for example, with a visible light pixel having sensitivity to visible light.

The infrared pixel includes, for example, an optical filter (also referred to as an IR filter) that transmits only infrared light, a microlens, a light receiving element that is a photoelectric conversion unit, an accumulation unit that accumulates charges generated by the light receiving element, and the like. Therefore, an image representing the luminance of infrared light is represented by image pickup signals output from a plurality of infrared light pixels (i.e., the 1 st pixel 21) included in the pixel array 2. The image of the infrared light is also referred to as an IR image or an infrared light image hereinafter.

The visible light pixel includes, for example, an optical filter (also referred to as BW filter) that transmits only visible light, a microlens, a light receiving element that is a photoelectric conversion unit, and an accumulation unit that accumulates electric charge converted by the light receiving element. Therefore, the 2 nd pixel 22 which is a visible light pixel outputs an image pickup signal indicating luminance and color difference. That is, a color image representing the luminance and color difference of visible light is expressed by the image pickup signals output from the plurality of 2 nd pixels 22 included in the pixel array 2. The optical filter of the visible light pixel may transmit both visible light and infrared light, or may transmit only light in a specific wavelength band such as red (R), green (G), or blue (B) among visible light.

In addition, the visible light pixels may detect only the luminance of visible light. In this case, the 2 nd pixel 22 which is a visible light pixel outputs an image pickup signal indicating luminance. Therefore, a black-and-white image representing the luminance of visible light, in other words, a monochromatic image is expressed by the image pickup signals output from the plurality of 2 nd pixels 22 included in the pixel array 2. This monochrome image will hereinafter also be referred to as BW image. The color image and the BW image are also collectively referred to as a visible light image.

Referring back to fig. 1 again, the description of the depth acquisition apparatus 1 is continued.

The processing circuit 30 calculates object information relating to an object using the image pickup signal output from the solid-state image pickup element 20.

The processing circuit 30 includes an arithmetic processing device such as a microcomputer. The microcomputer includes a processor (microprocessor), a memory, and the like, and generates a light emission signal and an exposure signal by the processor executing a drive program stored in the memory. The processing circuit 30 may include 1 piece of hardware, or may include a plurality of pieces of hardware, using an FPGA, an ISP, or the like.

The processing circuit 30 calculates the distance to the object by a ToF distance measurement method using the image pickup signal from the 1 st pixel 21 of the solid-state image pickup element 20, for example.

The calculation of the distance to the object by the processing circuit 30 based on the ToF ranging method will be described below with reference to the drawings.

Fig. 3 is a timing chart of the relationship between the light emission timing of the light emitting element of the light source 10 and the exposure timing of the 1 st pixel 21 of the solid-state imaging element 20 when the processing circuit 30 calculates the distance to the subject using the ToF ranging method.

In fig. 3, Tp is a light emission period during which the light emitting element of the light source 10 emits the irradiation light, and Td is a delay time from when the light emitting element of the light source 10 emits the irradiation light to when the reflected light of the irradiation light reflected by the object returns to the solid-state imaging device 20. The 1 st exposure period is the same timing as the light emission period during which the light source 10 emits the irradiation light, and the 2 nd exposure period is the timing from the end time point of the 1 st exposure period to the elapse of the light emission period Tp.

In fig. 3, q1 represents the total amount of exposure amount in the 1 st pixel 21 of the solid-state imaging element 20 caused by the reflected light in the 1 st exposure period, and q2 represents the total amount of exposure amount in the 1 st pixel 21 of the solid-state imaging element 20 caused by the reflected light in the 2 nd exposure period.

When the light emission of the irradiation light by the light emitting element of the light source 10 and the exposure of the 1 st pixel 21 of the solid-state imaging element 20 are performed at the timing shown in fig. 3, and the light velocity is c, the distance d to the object can be shown by the following (expression 1).

d ═ c × Tp/2 × q2/(q1+ q2) … (formula 1)

Therefore, the processing circuit 30 can calculate the distance to the object by using (expression 1) the image pickup signal from the 1 st pixel 21 of the solid-state image pickup device 20.

In addition, the plurality of 1 st pixels 21 of the solid-state imaging element 20 may be exposed only during the 3 rd exposure period Tp after the 1 st exposure period and the 2 nd exposure period have ended. The 1 st pixels 21 can detect noise other than reflected light from the exposure amount obtained in the 3 rd exposure period Tp. That is, the processing circuit 30 can calculate the distance d to the subject more accurately by removing noise from the exposure amount q1 in the 1 st exposure period and the exposure amount q2 in the 2 nd exposure period in the above (expression 1).

Referring back to fig. 1 again, the description of the depth acquisition apparatus 1 is continued.

The processing circuit 30 can detect an object and calculate a distance to the object using, for example, an image pickup signal from the 2 nd pixel 22 of the solid-state image pickup element 20.

That is, the processing circuit 30 may perform detection of an object and calculation of a distance to the object based on the visible light images captured by the plurality of 2 nd pixels 22 of the solid-state imaging element 20. Here, the detection of the object may be realized by, for example, discriminating the shape by pattern recognition using edge detection of singular points of the object, or may be realized by processing such as Deep Learning (Deep Learning) using a Learning model that is learned in advance. The distance to the subject can be calculated using world coordinate transformation. Of course, the detection of the object may be realized by a multi-modal learning process using not only the visible light image but also the luminance and distance information of the infrared light captured by the 1 st pixel 21.

The processing circuit 30 generates a light emission signal indicating a timing of light emission and an exposure signal indicating a timing of exposure. Then, the processing circuit 30 outputs the generated light emission signal to the light source 10 and outputs the generated exposure signal to the solid-state imaging device 20.

The processing circuit 30 can cause the depth acquisition device 1 to realize continuous imaging at a predetermined frame rate by generating and outputting an emission signal so that the light source 10 emits light at a predetermined cycle, and generating and outputting an exposure signal so that the solid-state imaging element 20 is exposed at a predetermined cycle, for example. The processing circuit 30 includes, for example, a processor (microprocessor), a memory, and the like, and generates a light emission signal and an exposure signal by the processor executing a drive program stored in the memory.

The diffusion plate 50 adjusts the intensity distribution and angle of the irradiated light. In the adjustment of the intensity distribution, the diffusion plate 50 makes the intensity distribution of the irradiation light from the light source 10 uniform. In the example shown in fig. 1, the depth acquisition device 1 includes the diffusion plate 50, but the diffusion plate 50 may not be provided.

The lens 60 is an optical lens for condensing light entering from the outside of the depth acquisition device 1 on the surface of the pixel array 2 of the solid-state imaging element 20.

The band-pass filter 70 is an optical filter that transmits infrared light and visible light, which are reflected light. In the example shown in fig. 1, the depth acquisition device 1 includes the band pass filter 70, but the band pass filter 70 may not be provided.

The depth acquisition device 1 having the above-described configuration is mounted on a transportation facility and used. For example, the depth acquisition device 1 is mounted on a vehicle traveling on a road surface and used. The transport facility on which the depth acquisition device 1 is mounted is not necessarily limited to a vehicle. The depth acquisition device 1 may be mounted on a transportation facility other than a vehicle such as a motorcycle, a ship, or an airplane, for example, and used.

[ functional Structure of depth acquisition device ]

The depth acquisition apparatus 1 according to the present embodiment acquires an IR image and a BW image by imaging in substantially the same scene and at substantially the same viewpoint and time using the hardware configuration shown in fig. 1. Here, the IR image is formed based on the intensity of the infrared light received by each of the plurality of 1 st pixels 21 included in the solid-state imaging element 20. Therefore, the depth acquisition device 1 acquires the intensity of the infrared light in each of the plurality of 1 st pixels 21 by imaging the IR image. The depth acquisition device 1 acquires a depth image representing the distance to the subject reflected in the IR image as the depth based on the intensity of the infrared light of the 1 st pixel 21. Then, the depth acquisition device 1 detects an edge region in the BW image, and corrects the depth of a region in the depth image corresponding to the edge region.

In the present disclosure, the 2 nd region in the 2 nd image corresponding to the 1 st region in the 1 st image is a region in the 2 nd image which is located at the same position as the 1 st region in the 1 st image and has the same shape and size as the 1 st region. The 1 st image and the 2 nd image are arbitrary images, and the 1 st region and the 2 nd region are arbitrary regions.

In addition, the edge in the present disclosure is a boundary of 2 regions whose luminance is different from each other. The 2 regions are light and dark regions. In addition, the average luminance in the bright area is higher than that in the dark area. Therefore, the depth acquisition device 1 in the present disclosure detects, as the edge area, a bright-dark boundary area that is an area including a boundary between a bright area and a dark area in a visible light image such as a BW image.

Fig. 4 is a block diagram showing an example of a functional configuration of the depth acquisition apparatus 1.

The depth acquisition device 1 includes a light source 101, an IR camera 102, a BW camera 103, a processor 110a, and a memory 200. Further, the depth acquisition device 1 in the present embodiment includes the light source 101, the IR camera 102, and the BW camera 103, but may include only the processor 110a and the memory 200 without including these components.

The light source 101 may include the light source 10 and the diffusion plate 50 shown in fig. 1, and emits light to irradiate infrared light to the object.

The IR camera 102 may be referred to as an infrared camera, and may include the plurality of 1 st pixels 21 of the solid-state imaging element 20 shown in fig. 1, the lens 60, and the band-pass filter 70. The IR camera 102 acquires an IR image by performing infrared light-based imaging of a scene including an object in accordance with the timing at which the light source 101 irradiates infrared light on the object. In addition, the IR camera 102 measures the intensity of infrared light by imaging an IR image. That is, the IR camera 102 has the solid-state imaging element 20, and measures the intensity of infrared light irradiated from the light source 101 and reflected by an object by performing imaging in which each of the 1 st pixels 21 included in the solid-state imaging element 20 receives the infrared light.

The BW camera 103 may be referred to as a visible light camera, and may include the plurality of 2 nd pixels 22 of the solid-state image pickup element 20 shown in fig. 1, the lens 60, and the band pass filter 70. Such a BW camera 103 obtains a visible light image (specifically, a BW image) by performing visible light-based imaging of a scene substantially the same as the IR image and imaging at substantially the same viewpoint and the same time as the IR image. That is, the BW camera 103 generates a BW image by performing imaging based on visible light of a scene substantially the same as an IR image formed based on the intensity of infrared light received by each of the plurality of 1 st pixels 21 included in the solid-state imaging device 20 and imaging at substantially the same viewpoint and time as the IR image.

The memory 200 is a recording medium for storing an IR image obtained by image capturing by the IR camera 102 and a BW image obtained by image capturing by the BW camera 103. As described above, the IR image is formed based on the intensity of the infrared light received by each of the 1 st pixels 21 included in the solid-state imaging element 20. Therefore, the IR image shows the intensity of infrared light for each pixel. That is, the memory 200 can be said to store the intensity of the infrared light. In addition, such a Memory 200 may be specifically a ROM (Read Only Memory), a RAM (Random access Memory), an SSD (solid state drive), or the like, and may be nonvolatile or volatile. In addition, the memory 200 may be a hard disk.

The processor 110a acquires the IR image and the BW image from the memory 200, calculates a depth image from the IR image, and detects a bright-dark boundary region in the BW image based on the IR image and the BW image. Then, the processor 110a corrects the depth of the area in the depth image corresponding to the bright-dark boundary area.

The bright-dark boundary region of the BW image is a region including a boundary between a bright region and a dark region and extending along a boundary in a direction perpendicular to the direction of fluctuation of the BW image. In a region in the depth image corresponding to the bright-dark boundary region, there is a possibility that an inappropriate depth is indicated.

The reason for this is as follows. The brightness in the bright and dark boundary regions of the BW image varies greatly in a short period of time due to the variation of the BW image. That is, in a short period of time, an object reflected in the bright-dark boundary region is replaced with another object having a light reflectance larger than that of the object. The BW image fluctuation is caused by, for example, the BW camera 103 fluctuation, and when the IR camera 102 and the BW camera 103 fluctuate together, for example, when the depth acquisition device 1 is mounted on a vehicle, the BW image fluctuation and the IR image fluctuation are equal to each other. Therefore, the same phenomenon as that of the BW image occurs in the IR image. That is, an object reflected in an area within the IR image corresponding to the bright-dark boundary area is replaced with another object having a reflectance of infrared light greatly different from that of the object in a short period of time. As a result, the depth measured by ToF using the intensity of infrared light corresponding to the bright-dark boundary region becomes incorrect. That is, the depth obtained by such measurement appears as noise in the depth image.

As described above, the region in the depth image in which noise appears corresponds to the region in the BW image including the boundary. Therefore, by detecting a bright-dark boundary region, which is a region including such a boundary, a region to be corrected can be appropriately found in the depth image. The depth acquisition device 1 in the present embodiment corrects the depth of the region to be corrected in the depth image, that is, the correction target region that is a region corresponding to the bright-dark boundary region.

The processor 110a includes a light emission control unit 113, an IR acquisition unit 114, a BW acquisition unit 115, a depth calculation unit 111a, a boundary region detection unit 112, and a depth correction unit 111 b.

The light emission control section 113 controls the light source 101. That is, the light emission control unit 113 outputs the light emission signal to the light source 101 to cause the light source 101 to emit light. Thus, infrared light is irradiated from the light source 101 to the object, and light reflected by the object, that is, reflected light enters the IR camera 102.

The IR acquisition unit 114 acquires an IR image from the IR camera 102 via the memory 20. Here, each pixel of the IR image represents the intensity of the infrared light received at the position of the pixel as a pixel value (specifically, luminance). Therefore, the IR acquisition unit 114 acquires the intensity of the infrared light by acquiring the IR image. That is, the IR obtaining section 114 obtains the intensity of infrared light irradiated from the light source 101 and reflected by the object, and the intensity of the infrared light is measured by the plurality of 1 st pixels 21 included in the solid-state imaging device 20 receiving the image pickup of the infrared light, and is stored in the memory 200.

The BW acquisition unit 115 acquires a BW image from the BW camera 103 via the memory 200. That is, the BW acquiring unit 115 acquires a BW image generated by visible light-based imaging of a scene substantially the same as an IR image formed based on the intensity of infrared light received by each of the 1 st pixels 21 included in the solid-state imaging device 20 and imaging at substantially the same viewpoint and time as the IR image, and holds the BW image in the memory 200.

The depth calculating unit 111a calculates a distance to the subject as a depth based on the intensity of the infrared light received by each of the 1 st pixels 2 for each of the 1 st pixels 21 included in the solid-state imaging device 20, and generates a depth image.

The boundary area detection unit 112 detects a bright-dark boundary area from the BW image, the bright-dark boundary area including a boundary between a bright area and a dark area and being along a boundary in a direction perpendicular to a direction of fluctuation of the BW image.

The depth correction unit 111b corrects the depth of the correction target region, which is a region in the depth image corresponding to the bright-dark boundary region.

For example, when imaging a boundary of 2 objects having different reflectances of infrared light, if the IR camera 102 used for the imaging is changed, the reflectance at a measurement target position around the boundary may be greatly changed in the ToF-based measurement. In such a case, noise is obtained at the measurement target position, and a depth image indicating an incorrect depth is acquired.

However, in the depth acquisition device 1 according to the present embodiment, as described above, an area including a boundary between a bright area and a dark area and a boundary along a direction perpendicular to the direction of fluctuation of the visible light image is detected as a bright-dark boundary area from the BW image. The bright-dark boundary region is a region around the above-described boundary where the possibility that the reflectance of infrared light greatly changes in ToF measurement is high. By setting the region in the depth image corresponding to such a bright-dark boundary region as a correction target region, the depth of the correction target region is corrected, so that noise can be reduced and the depth around the boundary can be accurately obtained.

Therefore, even if the boundaries of 2 objects having different reflectances with respect to infrared light are along the direction perpendicular to the direction of the fluctuation of the image, the depth acquisition device 1 in the present embodiment can accurately acquire the depth, which is the distance around the boundaries of these objects.

[ example of respective images ]

Fig. 5 shows an example of a BW image and an IR image.

For example, when the vehicle is traveling on a road surface, the IR camera 102 and the BW camera 103 of the depth acquisition device 1 mounted on the vehicle perform imaging at substantially the same timing. At this time, the IR camera 102 and the BW camera 103 are exposed at 3 timings different from each other. Thus, 3 IR images are generated by the imaging of the IR camera 102, and 3 BW images are generated by the imaging of the BW camera 103.

The IR image generated by the image pickup by the IR camera 102 and acquired by the IR acquisition unit 114 shows the road surface and trees around it, for example, as shown in (a) to (c) of fig. 5.

In addition, in the BW image generated by the imaging by the BW camera 103 and acquired by the BW acquiring unit 115, as shown in (d) to (f) of fig. 5, substantially the same scenes as the IR images shown in (a) to (c) of fig. 5 are reflected. The BW image shown in (d) to (f) of fig. 5 is an image obtained by imaging at substantially the same viewpoint and at substantially the same timing as the imaging of the IR image shown in (a) to (c) of fig. 5. Therefore, the IR images shown in fig. 5 (a) to (c) and the BW images shown in fig. 5 (d) to (f) show the same objects in the corresponding regions. The mutually corresponding regions are regions having the same position, size, and shape in each image.

In the BW image, the luminance of the tree leaf image is low, and the luminance of the background sky image is high. Therefore, a region including a boundary in a direction perpendicular to the direction of the image fluctuation, among boundaries between the leaf image and the sky image, corresponds to a bright-dark boundary region. In such an area in the IR image corresponding to the bright-dark boundary area, an inappropriate infrared light intensity is obtained, and thus the image tends to be blurred. Further, the reliability of the depth calculated from such inappropriate infrared light intensity is also low. The depth correcting unit 111b in the present embodiment corrects the depth to improve the reliability of the depth.

[ treatment procedure ]

Fig. 6 is a flowchart showing the overall processing operation of the depth acquisition device 1 in the present embodiment.

(step S110)

First, the BW camera 103 generates a BW image by performing image pickup based on visible light. Then, the BW acquiring unit 115 acquires the BW image via the memory 200.

(step S120)

Next, the IR camera 102 measures the intensity of the infrared light by performing imaging based on the infrared light. Then, the IR obtaining unit 114 obtains the intensity of the infrared light via the memory 200. Here, the intensity of the acquired infrared light is the intensity of the infrared light received by each of the 1 st pixels 21 of the solid-state imaging device 20. Therefore, the IR acquisition unit 114 acquires the intensity of the infrared light to acquire an IR image formed based on the intensity of the infrared light.

Specifically, the intensity of the infrared light acquired from the memory 200 includes at least 3 intensities measured by exposure of the solid-state imaging element 20 at timings different from each other by at least 3 times when the infrared light irradiated from the light source 101 and reflected by the object is received by the solid-state imaging element 20. For example, the intensity of the infrared light includes 3 intensities measured through the 1 st exposure period, the 2 nd exposure period, and the 3 rd exposure period as shown in fig. 3. For example, each pixel value included in the IR image of 1 frame is represented as an accumulated value of the intensity of infrared light measured by exposure at the at least 3 times of timing.

(step S130)

Next, the boundary area detection unit 112 detects a bright-dark boundary area from the BW image acquired in step S110. Then, the boundary area detection unit 112 outputs a mask image having a 2 value in which 1 is shown for each pixel in the detected bright-dark boundary area and 0 is shown for each pixel in the other areas.

(step S140)

Next, the depth calculating unit 111a calculates the depth based on the intensity of the infrared light acquired in step S120. That is, the depth calculating unit 111a calculates the distance to the subject as the depth based on the intensity of the infrared light received by each of the 1 st pixels 2 of the 1 st pixels 21 included in the solid-state imaging device 20, and generates the depth image. Specifically, the depth calculating unit 111a calculates the depth of each of the 1 st pixels 21 included in the solid-state imaging device 20 based on the at least 3 intensities measured in the 1 st pixel 21, and generates a depth image. For example, the depth calculating unit 111a calculates the depth using the above (expression 1) in accordance with the ToF ranging method.

(step S150)

Next, the depth correction unit 111b corrects the depth of the correction target region, which is a region in the depth image corresponding to the bright-dark boundary region.

Specifically, the depth correction unit 111b acquires the BW image, the depth image, and the mask image, and corrects the depth (i.e., the pixel value in the depth image) in the region indicated as 1 from the mask image among the depth images. The depth is corrected by, for example, an extension Filter of a Guided Filter (Guided Filter) that uses the BW image as a guide image. The expansion filter is disclosed in the non-patent literature (Jiangbo Lu, Keyang Shi, Dongbo Min, Liang Lin, and Minh N.Do, "Cross-Based Local Multipoint Filtering", 2012 IEEE Conference on Computer Vision and Pattern Recognition). Alternatively, the correction of the depth can be realized by a method disclosed in non-patent documents (Dongbo Min, Sunghwan Choi, Jiangbo Lu, Bumsub Ham, Kwanghon Sohn, and Minh N.Do., "Fast Global Image Smoothing Based on Weighted Least Squares". IEEE Transactions on Image Processing, Vol.23, No.12, December 2014).

As described above, the depth correction unit 111b according to the present embodiment corrects the depth of the correction target area by filtering the depth image using the BW image as the guide image or the reference image. Thus, the depth of the correction target region can be corrected accurately.

The depth correction is not limited to the above, and for example, interpolation may be performed with emphasis on the depth in a region adjacent to the lower portion of the bright-dark boundary region. That is, the depth correction unit 111b can correct the depth of the correction target region by using the depth of the peripheral region, which is a region located around the correction target region in the depth image. The peripheral region may be a region that is in contact with the lower side of the correction target region. In this case, the depth correction unit 111b may replace the depth of the correction target region with the depth of the peripheral region. The depth of the peripheral region is likely to be approximated to the correct depth of the region to be corrected. Therefore, by using the depth of the peripheral region, the depth of the region to be corrected can be corrected accurately.

In addition, the IR image may be used as an input in addition to the BW image, the depth image, and the mask image. For example, the depth correction unit 111b may correct the depth of the correction target region in the depth image by inputting the IR image, the BW image, the depth image, and the bright-dark boundary region to the learning model. In addition, the mask image described above may be used instead of the light and dark boundary regions. For example, the learning model is a neural network constructed by deep learning. In the learning of the learning model, the IR image, the BW image, the depth image, and the bright-dark boundary region are provided as inputs to the learning model. Then, the learning model is learned so that the output of the learning model for the input matches the corrected depth image of the correct solution. By using such a learned learning model, the depth correction unit 111b can easily acquire a corrected depth image.

Fig. 7 is a flowchart showing an example of the processing operation of the boundary area detection unit 112. Fig. 7 shows the processing in step S130 of fig. 6 in detail.

(step S131)

First, the boundary area detection unit 112 detects a variation from the BW image. The variation is detected, for example, by performing a variation search of block matching between the BW image and a past BW image.

(step S132)

Next, the boundary area detection unit 112 detects an edge along the direction perpendicular to the direction of the fluctuation detected in step S131 as the boundary from the BW image. In this case, the boundary area detection unit 112 may detect an edge having an intensity equal to or higher than the 1 st threshold.

(step S133)

The boundary area detection unit 112 detects an area including the edge detected in step S132 as a bright-dark boundary area. In this case, the boundary area detection unit 112 extracts, for example, an area located at a certain distance from the edge as a light-dark boundary area. Specifically, the bright-dark boundary region is a region from an edge having an intensity equal to or higher than the 1 st threshold to a distance corresponding to the fluctuation of the BW image.

Here, the distance is equal to the length of a fluctuation vector indicating the fluctuation of the BW image, or is a length obtained by multiplying the length of the fluctuation vector by a constant (for example, 1.1 times the length of the fluctuation vector). Alternatively, the distance may be a distance obtained by adding a constant to the length of the variation vector (for example, the length of the variation vector is +3 pixels or +5 pixels). In addition, the direction of the variation vector may be considered in this distance. For example, the distance from the edge to the direction of the variation vector is equal to or longer than the length of the variation vector, and the distance from the direction opposite to the variation vector may be, for example, 0.1 times the length of the variation vector, or may be a constant such as 3 pixels or 5 pixels.

As described above, the boundary area detection unit 112 in the present embodiment detects an edge along a direction perpendicular to the direction of fluctuation in the BW image as a boundary from the BW image, and detects an area including the edge as a bright-dark boundary area. Specifically, the boundary area detection unit 112 detects an edge having an intensity equal to or higher than the 1 st threshold. Then, the area including the edge is an area from the edge having the intensity equal to or higher than the 1 st threshold to a distance corresponding to the fluctuation of the BW image. This enables appropriate detection of the bright-dark boundary region. As a result, a region in which noise is likely to occur can be appropriately detected in the depth image.

Fig. 8 is a diagram for explaining the edge detection in step S132 in fig. 7.

For example, the object shown in the BW image moves with the passage of time as shown by times t0, t1, and t2 in fig. 8. Such a variation in the object is generated by, for example, a movement of a vehicle on which the depth acquisition device 1 is mounted. Such a variation in the pixel (x, y) within the BW image is represented as a variation vector (u (x, y), v (x, y)).

The boundary area detection unit 112 calculates the intensity of the edge as the amount of change in the pixel value by differentiating the distribution of the pixel values in the spatial direction. The differential for an arbitrary direction n is characterized by (equation 2).

[ mathematical formula 1]

Here, the direction n is considered as a unit vector in the direction of the fluctuation vector (u (x, y), v (x, y)) of the image, that is, in the direction parallel to the direction of the fluctuation vector. That is, the x component cos θ and the y component sin θ of n are characterized by (equation 3). The variation vector of the image is obtained by a known method.

[ mathematical formula 2]

The luminance gradients in the horizontal direction and the vertical direction in (expression 2) are calculated by (expression 4) and (expression 5), respectively.

[ mathematical formula 3]

[ mathematical formula 4]

The calculation of the luminance gradients in the horizontal direction and the vertical direction is not limited to the calculation based on the difference between both sides shown in (equation 4) and (equation 5), and may be based on the calculation based on the difference between the front side shown in (equation 6) and (equation 7), or may be based on the calculation based on the difference between the rear side shown in (equation 8) and (equation 9). The same results can be obtained with any of these calculations.

[ math figure 5]

[ mathematical formula 6]

[ math figure 7]

[ mathematical formula 8]

In addition, although the calculations are performed only between pixels in the same row or the same column in (equations 4) to (9), the present invention is not limited to this, and a stable luminance gradient calculation for reducing the influence of noise may be performed by performing the calculations using pixel values in adjacent rows or columns as shown in (equations 10) to (15).

[ mathematical formula 9]

[ mathematical formula 10]

[ mathematical formula 11]

[ mathematical formula 12]

[ mathematical formula 13]

[ mathematical formula 14]

Further, when the luminance gradient is calculated using the pixel values of a plurality of rows or a plurality of columns, the weight of the attention row or the attention column may be increased as compared with the weight of the adjacent row or the adjacent column as in (expressions 16) to (expression 21).

[ mathematical formula 15]

[ mathematical formula 16]

[ mathematical formula 17]

[ mathematical formula 18]

[ math figure 19]

[ mathematical formula 20]

The boundary region detection unit 112 uses the horizontal luminance gradient and the vertical luminance gradient shown in (expression 4) to (expression 21), and the horizontal component cos θ and the vertical component sin θ of the direction vector n of the fluctuation shown in (expression 3), and appliesDifferentiating the direction along the direction n of the variation by equation 2 Calculated as the intensity of the edge.

The boundary area detection unit 112 may calculate the direction differential by using not only the method of (expression 2) but also another method. For example, the boundary region detection unit 112 calculates the pixel value of at least 1 pixel on a straight line defined by the target point and the differential direction by interpolation. Then, the boundary region detection unit 112 calculates a direction differential from a difference value between the calculated pixel value of at least 1 pixel and any 2 pixel values out of the pixel values of the attention point. The same results as in the above (formula 2) can be obtained by this method.

The boundary area detection unit 112 detects pixels having an absolute value of the direction differential equal to or greater than the 1 st threshold as edges.

Fig. 9 is a flowchart showing another example of the processing operation of the boundary area detection unit 112. Fig. 9 shows the processing in step S130 of fig. 6 in detail.

(step S135)

The boundary area detection unit 112 calculates the difference between the BW image and the BW image immediately before.

(step S136)

Then, the boundary area detection unit 112 detects an area in the BW image where the difference is equal to or greater than the 2 nd threshold as a bright-dark boundary area.

As described above, the boundary area detection unit 112 in the present embodiment detects an area in which the difference between the pixel values of the BW image and the past BW image is equal to or greater than the 2 nd threshold as a bright-dark boundary area. This enables the bright-dark boundary region to be easily detected.

(modification example)

In the above embodiment, the light/dark boundary region is detected to correct the depth, but the depth may be corrected using a learning model instead of detecting the light/dark boundary region.

Fig. 10 is a block diagram showing an example of a functional configuration of the depth acquisition apparatus 1 according to the present modification. Among the components shown in fig. 10, the same components as those shown in fig. 4 are denoted by the same reference numerals as those shown in fig. 4, and detailed description thereof is omitted.

The depth acquisition apparatus 1 in the present modification includes a processor 110b instead of the processor 110a shown in fig. 4, and further includes a learning model 104.

The learning model 104 is, for example, a neural network, and is formed by deep learning. For example, the BW image, the IR image, and the depth image are used as input data to the learning model 104. The learning model 104 has already performed learning so that a corrected depth image having a positive solution is output for a combination of these input data.

The processor 110b does not include the boundary region detection unit 112 shown in fig. 4, and includes a depth correction unit 111c instead of the depth correction unit 111b shown in fig. 4.

The depth correction unit 111c inputs the input data to the learning model 104. As a result, the depth correction unit 111c obtains the corrected depth image from the learning model 104 as output data for the input data.

That is, the depth acquisition device 1 shown in fig. 10 includes a memory 200 and a processor 110 b. The processor 110b obtains the intensity of infrared light irradiated from the light source 101 and reflected by the object, and the intensity of the infrared light is measured by the imaging in which the plurality of 1 st pixels 21 included in the solid-state imaging element 20 receive the infrared light, and is stored in the memory 200. The processor 110b calculates the distance to the subject as the depth for each of the 1 st pixels 21 included in the solid-state imaging device 20 based on the intensity of the infrared light received by the 1 st pixel 21, and generates a depth image. Further, the processor 110b obtains a BW image generated by imaging with visible light of a scene substantially the same as an IR image formed based on the intensity of infrared light received by each of the plurality of 1 st pixels 21 included in the solid-state imaging device 20 and imaging at substantially the same viewpoint and time as the IR image, and holds the BW image in the memory 200. Then, the processor 110b corrects the depth of the depth image by inputting the depth image, the IR image, and the BW image to the learning model 104.

Thus, if the learning model 104 is learned in advance so that the corrected depth image having a positive solution is output for the input of the depth image, the IR image, and the BW image, the depth of the depth image can be appropriately corrected without detecting the bright-dark boundary region.

Fig. 11A shows an example of a simulation result of the depth acquisition apparatus 1 according to the present modification.

The depth acquisition device 1 acquires a BW image shown in fig. 11A (a) by imaging with the BW camera 103, and further acquires an IR image shown in fig. 11A (b) by imaging with the IR camera 102. The BW image and the IR image are images obtained by imaging the same scene at the same viewpoint and the same time.

The depth calculating unit 111A generates a depth image shown in fig. 11A (c) based on the intensity of the infrared light forming the IR image. In the bright-dark boundary region in the BW image shown in fig. 11A (a), the contour of an object, for example, the contour of a white line of a crosswalk drawn on a road surface, is clearly reflected. However, in the depth image shown in fig. 11A (c), the depth of the region corresponding to the bright-dark boundary region is not clearly represented.

The depth correction unit 111b inputs the BW image, the IR image, and the depth image shown in (a) to (c) of fig. 11A to the learning model 104, and thereby obtains the corrected depth image shown in (d) of fig. 11A from the learning model 104.

As a result, in the depth acquisition device 1 according to the present modification, the corrected depth image can be brought close to the depth image of the positive solution shown in fig. 11A (e).

Fig. 11B shows another example of the simulation result of the depth acquisition device 1 according to the present modification.

The depth acquisition device 1 acquires a BW image shown in fig. 11B (a) by imaging with the BW camera 103, and further acquires an IR image shown in fig. 11B (B) by imaging with the IR camera 102. In the example shown in fig. 11B, as in the example shown in fig. 11A, the BW image and the IR image are images obtained by imaging the same scene from the same viewpoint and at the same time.

The depth calculating unit 111a generates a depth image shown in fig. 11B (c) based on the intensity of the infrared light forming the IR image. In the bright and dark boundary region in the BW image shown in fig. 11B (a), the contour of an object, for example, the contour of a white line of a crosswalk drawn on a road surface, is clearly reflected. However, in the depth image shown in fig. 11B (c), the depth of the region corresponding to the bright-dark boundary region is not clearly represented.

The depth correction unit 111B inputs the BW image, the IR image, and the depth image shown in (a) to (c) of fig. 11B to the learning model 104, and thereby obtains the corrected depth image shown in (d) of fig. 11B from the learning model 104.

As a result, in the depth acquisition device 1 according to the present modification, the corrected depth image can be brought close to the depth image of the positive solution shown in fig. 11B (e).

As described above, in the depth acquisition device 1 according to the present embodiment and the modification thereof, even when the boundary of 2 objects having different reflectances of infrared light is imaged, the depth around the boundary can be accurately acquired.

In the above embodiments, each component may be implemented by a dedicated hardware configuration or by executing a software program suitable for each component. Each component can be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the depth acquisition device and the like of the above-described embodiment and the modification causes the computer to execute the steps included in any one of the flowcharts of fig. 6, 7, and 9.

The depth acquisition device according to one or more embodiments has been described above based on the embodiments and the modifications thereof, but the present disclosure is not limited to the embodiments and the modifications thereof. The present invention is not limited to the embodiments described above, and various modifications and variations can be made without departing from the spirit and scope of the present invention.

For example, the depth calculating unit 111a calculates the depth based on the intensities of 3 infrared lights measured at 3 exposure timings different from each other. However, the number of exposure timings and the number of intensities of infrared light are not limited to 3, and may be 4 or more.

In the modification of the above embodiment, the depth acquisition device 1 includes the learning model 104, but the learning model 10 may not be included. In this case, the depth acquisition device 1 inputs the above-mentioned input data to the learning model 104 via, for example, a communication network, and acquires the corrected depth image, which is the output data from the learning model 104, via the communication network.

In the present disclosure, all or a part of components and devices, or all or a part of functional blocks of the block diagrams shown in fig. 1, 4, and 10 may be implemented by one or more electronic circuits including a semiconductor device and a semiconductor Integrated Circuit (IC) or an LSI (large scale integration). The LSI or IC may be integrated on one chip, or may be configured by combining a plurality of chips. For example, functional blocks other than the memory element may be integrated into one chip. Here, the term LSI and IC are used, but may be called system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration) depending on the degree of integration. For the same purpose, a Field Programmable Gate Array (FPGA) that is programmed after manufacturing of the LSI, or a reconfigurable logic device (reconfigurable logic device) that can perform reconfiguration of a bonding relationship inside the LSI or setting of circuit sections inside the LSI can be used.

Further, a part, all or a part of functions or operations of a component, a device, or an apparatus can be performed by a software process. In this case, the software is recorded in a non-transitory recording medium such as one or more ROMs, optical disks, or hard disk drives, and when the software is executed by the processing device (processor), the software causes the processing device (processor) and peripheral devices to execute a specific function in the software. The system or apparatus may be provided with one or more non-transitory recording media recording software, a processing device (processor), and a required hardware device such as an interface.

Industrial applicability

The present disclosure can be applied to a depth acquisition device that acquires a depth that is a distance to an object, and can be used as, for example, an in-vehicle device.

Description of reference numerals

1 depth acquisition device

10. 101 light source

20 solid-state imaging element

21 st pixel (IR)

22 nd pixel 2 (BW)

30 processing circuit

50 diffusion plate

60 lens

70 band-pass filter

102 IR camera

103 BW camera

104 learning model

110a, 110b processor

111a depth calculating unit

111b, 111c depth correction unit

112 boundary region detection unit

113 light emission control unit

114 IR acquisition unit

115 BW acquisition unit

200 memory.