Image pickup apparatus and image processing system
阅读说明:本技术 摄像装置和图像处理系统 (Image pickup apparatus and image processing system ) 是由 佐藤泰史 坂本顺一 见潮充 马场裕一郎 于 2019-03-08 设计创作,主要内容包括:一种摄像装置,包括:像素阵列部,在像素阵列部中排列有多个像素组,每个像素组由以2×2矩阵布置的四个像素构成。作为由四个像素构成的像素组,形成有由三个接收红色光的像素和一个接收红外光的像素构成的第一像素组、由三个接收蓝色光的像素和一个接收红外光的像素构成的第二像素组、由三个接收绿色光的像素和一个接收红外光的像素构成的第三像素组以及由三个接收绿色光的像素和一个接收红外光的像素构成的第四像素组。由第一像素组、第二像素组、第三像素组和第四像素组构成的四个像素组布置并形成一组的2×2个单元,其中,第一像素组和第二像素组对角地定位,并且第三像素组和第四像素组对角地定位。(An image pickup apparatus comprising: a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 × 2 matrix are arranged. As a pixel group composed of four pixels, a first pixel group composed of three pixels receiving red light and one pixel receiving infrared light, a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light, a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, and a fourth pixel group composed of three pixels receiving green light and one pixel receiving infrared light are formed. Four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.)
1. An image pickup apparatus comprising:
a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 × 2 matrix are arranged,
among them, as a pixel group constituted by four pixels, there are formed:
A first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group which is composed of three pixels receiving green light and one pixel receiving infrared light, and
four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
2. The image pickup apparatus according to claim 1,
in each pixel group, pixels that receive infrared light are disposed at the same position.
3. The image pickup apparatus according to claim 2,
an on-chip lens is provided for each pixel.
4. The image pickup apparatus according to claim 1,
any one of the red filter, the green filter, and the blue filter is arranged to correspond to a pixel that receives red light, green light, or blue light, and
The infrared light transmitting filter is arranged to correspond to a pixel that receives infrared light.
5. The image pickup apparatus according to claim 4,
the infrared light transmission filter is formed by laminating at least two of a red filter, a green filter, and a blue filter.
6. The image pickup apparatus according to claim 5,
the infrared light transmission filter is formed by laminating both a red filter and a blue filter.
7. The image pickup apparatus according to claim 1,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that these pixels receiving infrared light are arranged in a 2 × 2 matrix in every four adjacent pixel groups.
8. The image pickup apparatus according to claim 7,
an on-chip lens is provided for each pixel.
9. The image pickup apparatus according to claim 7,
an on-chip lens is arranged in each of the pixels receiving red, green and blue light, and
common on-chip lenses are arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light.
10. The image pickup apparatus according to claim 7,
Any one of the red filter, the green filter and the blue filter is arranged to correspond to a pixel receiving red light, green light or blue light, and
a common infrared light transmitting filter is arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light.
11. The image pickup apparatus according to claim 10,
the common infrared light transmission filter is configured by laminating at least two of a red filter, a green filter, and a blue filter.
12. The image pickup apparatus according to claim 1,
image data having a red component, a green component, and a blue component is generated for each pixel.
13. The image pickup apparatus according to claim 1,
data of pixels of the same color in each pixel group are added to generate image data.
14. The image pickup apparatus according to claim 7,
data of four pixels arranged in a 2 × 2 matrix and receiving infrared light is added to generate image data.
15. The image pickup apparatus according to claim 1,
three pixels other than the pixel that receives infrared light in each pixel group are exposed under different exposure conditions according to the pixel.
16. The image pickup apparatus according to claim 15,
in each pixel group, pixels receiving infrared light are disposed at the same position,
a pair of scanning lines are arranged in each pixel row, an
The pixels in each image row are alternately connected to one scanning line and another scanning line in units of one pixel.
17. The image pickup apparatus according to claim 15,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that, in every four adjacent pixel groups, these pixels receiving infrared light are arranged in a 2 x 2 matrix,
a pair of scanning lines are arranged in each pixel row, an
The pixels in each image row are alternately connected to one scanning line and the other scanning line in units of two pixels.
18. An image processing system comprising:
an image pickup device for picking up an image of an object and a signal processing section for processing a signal from the image pickup device,
wherein the image pickup device includes a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 x 2 matrix are arranged,
as a pixel group constituted by four pixels, there are formed:
A first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group which is composed of three pixels receiving green light and one pixel receiving infrared light, and
four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
19. The image processing system of claim 18, further comprising:
and a light source unit that irradiates infrared light to an object.
20. The image processing system of claim 18, further comprising:
an authentication processing section that executes an authentication process based on the infrared light image.
21. The image processing system according to claim 18,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that, in every four adjacent pixel groups, these pixels receiving infrared light are arranged in a 2 x 2 matrix,
Common on-chip lenses are arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light, and
the authentication processing section is configured to perform authentication processing by using at least one of an infrared light image and a depth map generated based on an image plane phase difference of pixels receiving infrared light.
Technical Field
The invention relates to an imaging apparatus and an image processing system. In particular, the present invention relates to an image pickup apparatus and an image processing system suitable for capturing a visible light image and an infrared light image.
Background
With the progress of semiconductor processing, there has been proposed an image pickup apparatus configured such that one pixel in a bayer array is divided into a plurality of pixels (for example, see fig. 14 of patent document 1). The image pickup apparatus having such a configuration is advantageous in that an image having an excellent S/N ratio can be obtained by adding and reading pixels of the same color, a high-resolution image can be obtained by performing full resolution demosaic processing (full resolution demosaic processing), and a High Dynamic Range (HDR) image can be obtained without requiring multiple image pickup by setting different exposure conditions for a plurality of pixels corresponding to one pixel in a bayer array.
Reference list
Patent document
Patent document 1: japanese patent application laid-open No. 2011-
Disclosure of Invention
Problems to be solved by the invention
In recent years, it has been proposed to perform sensing processing such as measurement processing on image information and use information such as the size of an object or the position of the center of gravity of the object, the distance to the object, and the amount of movement of the object as the sensing information.
Preferably, the two kinds of image information may be acquired in a state where there is no parallax between the image information for sensing processing and the image information for image display (viewing). Therefore, it is preferable that an infrared light image for sensing processing or the like and a visible light image for viewing processing or the like are obtained from one image pickup device. However, in a configuration in which one pixel in the bayer array is divided into a plurality of pixels, it is impossible to obtain an infrared light image and a visible light image from one image pickup device.
An object of the present invention is to provide an image pickup apparatus having an advantage of a configuration in which one pixel of a bayer array is divided into a plurality of pixels and an infrared light image can also be acquired, and an image processing system using the image pickup apparatus.
Problem solving scheme
An image pickup apparatus according to the present invention for achieving the above object includes:
a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 × 2 matrix are arranged,
as a pixel group constituted by four pixels, there are formed:
a first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group which is composed of three pixels receiving green light and one pixel receiving infrared light, and
four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
An image processing system according to the present invention for achieving the above object includes:
an image pickup device for picking up an image of an object and a signal processing section for processing a signal from the image pickup device,
wherein the image pickup device includes a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 x 2 matrix are arranged,
as a pixel group composed of four pixels, there is formed
A first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
Four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
Drawings
Fig. 1 is a basic configuration diagram of an image processing system using an image pickup apparatus according to a first embodiment.
Fig. 2 is a schematic plan view for explaining the configuration of the image pickup apparatus.
Fig. 3 is a schematic plan view for explaining a pixel array in the image pickup apparatus of the reference example.
Fig. 4 is a schematic plan view for explaining a pixel array in the image pickup device according to the first embodiment.
Fig. 5 is a schematic plan view for explaining a relationship between a pixel array and an on-chip lens array.
Fig. 6 is a schematic partial end view of the image pickup device.
Fig. 7 is a schematic diagram of the spectral transmittance of the filter.
Fig. 8 is a schematic process diagram showing a procedure of image processing accompanying full resolution demosaicing processing.
Fig. 9A and 9B are schematic diagrams for explaining the relationship of pixels used for interpolation in the full-resolution demosaic processing.
Fig. 10 is a schematic process diagram showing the procedure of image processing accompanying pixel addition processing.
Fig. 11 is a schematic process diagram showing a process of image processing accompanying HDR processing.
Fig. 12A is a schematic plan view for explaining an array of pixels having different exposure times. Fig. 12B is a schematic plan view for explaining the relationship between the scanning line connected to the pixel and the control target pixel.
Fig. 13 is a schematic diagram for explaining the spectral transmittance of the dual band pass filter.
Fig. 14 is a schematic diagram for explaining spectral transmittance when a dual band-pass filter and a color filter are overlapped.
Fig. 15 is a schematic plan view for explaining a pixel array in the image pickup device according to the second embodiment.
Fig. 16 is a schematic plan view for explaining the relationship between the pixel array and the on-chip lens array.
Fig. 17 is a schematic plan view of the relationship between the pixel array and the on-chip lens array for explanation after fig. 16.
Fig. 18 is a schematic partial end view of an image pickup device including the on-chip lens shown in fig. 17.
Fig. 19 is a schematic process diagram showing the procedure of image processing accompanying full-resolution demosaicing processing.
Fig. 20 is a schematic process diagram showing the procedure of image processing accompanying pixel addition processing.
Fig. 21 is a schematic operation diagram for explaining the influence of ambient infrared light when an infrared light image is captured during a rolling shutter operation.
Fig. 22 is a schematic operation diagram for explaining the influence of surrounding infrared light when an infrared light image is captured during a rolling shutter operation, following fig. 21.
Fig. 23 is a schematic process diagram showing a process of image processing accompanying HDR processing.
Fig. 24A is a schematic plan view for explaining an array of pixels having different exposure times. Fig. 24B is a schematic plan view for explaining the relationship between the scanning line connected to the pixel and the control target pixel.
Fig. 25 is a basic configuration diagram of an image processing system that performs authentication processing and viewing processing according to the third embodiment.
Fig. 26 is a schematic process diagram showing image processing accompanying the authentication processing and the viewing processing.
Fig. 27 is a schematic process diagram showing the procedure of image processing accompanying authentication processing and viewing processing when the image pickup apparatus according to the first embodiment is used.
Fig. 28 is a schematic partial end view of the image pickup apparatus for explaining a filter configuration capable of reducing the influence of infrared light on pixels receiving visible light.
Fig. 29 is a schematic plan view for explaining a first modification of the pixel array.
Fig. 30 is a schematic plan view for explaining a second modification of the pixel array.
Fig. 31 is a schematic plan view for explaining a third modification of the pixel array.
Fig. 32 is a schematic plan view for explaining a fourth modification of the pixel array.
Fig. 33 is a schematic plan view for explaining a fifth modification of the pixel array.
Fig. 34 is a schematic plan view for explaining a sixth modification of the pixel array.
Fig. 35 is a block diagram showing an example of a schematic configuration of the vehicle control system.
Fig. 36 is an explanatory view showing an example of the mounting positions of the vehicle external information detection unit and the imaging unit.
Detailed Description
Hereinafter, the present invention will be explained based on embodiments with reference to the drawings. The present invention is not limited to these embodiments, and various numerical values and materials in the embodiments are examples. In the following description, the same elements or elements having the same function will be denoted by the same reference numerals without repeated description. Note that the description will be given in the following order.
1. General description of image pickup apparatus and image processing system of the present invention
2. First embodiment
3. Second embodiment
4. Third embodiment
5. Various modifications
6. Application example
7. Structure of the invention
[ general description of imaging apparatus and image processing System of the present invention ]
As described above, the image pickup apparatus of the present invention and the image pickup apparatus used in the image processing system of the present invention (hereinafter, these image pickup apparatuses are collectively referred to as "the image pickup apparatus of the present invention" in some cases) include:
A pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 × 2 matrix are arranged.
As a pixel group consisting of four pixels, there are formed:
a first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
And a fourth pixel group including three pixels receiving green light and one pixel receiving infrared light.
Four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
In the image pickup apparatus of the present invention, pixels that receive infrared light may be disposed at the same position in each pixel group. In this case, an on-chip lens may be arranged in each pixel.
In the image pickup apparatus of the present invention having the above-described various preferred configurations, any one of the red filter, the green filter, and the blue filter may be arranged to correspond to a pixel that receives red light, green light, or blue light, and the infrared light transmission filter may be arranged to correspond to a pixel that receives infrared light. The infrared light transmission filter may be formed by laminating at least two of a red filter, a green filter, and a blue filter. More preferably, the infrared light transmission filter is preferably formed by laminating both a red filter and a blue filter.
Alternatively, in the image pickup apparatus of the present invention, the pixels that receive infrared light may be disposed adjacent to the pixels that receive infrared light in adjacent pixel groups such that in every four adjacent pixel groups, these pixels that receive infrared light are arranged in a 2 × 2 matrix.
Even in this case, an on-chip lens may be arranged in each pixel. Alternatively, an on-chip lens may be disposed in each of the pixels that receive red, green, and blue light, and a common on-chip lens may be disposed in the pixels that are disposed in a 2 × 2 matrix and receive infrared light.
In the configuration in which the common on-chip lens is arranged, the image plane phase difference is detected by the pixels arranged in a 2 × 2 matrix and receiving infrared light, so that distance information can be obtained while obtaining an infrared light image.
Further, any one of the red, green, and blue filters may be arranged to correspond to pixels that receive red, green, or blue light, and the common infrared light transmission filter may be arranged in pixels that are arranged in a 2 × 2 matrix and receive infrared light. In this case, the common infrared light transmission filter may be configured by laminating at least two of a red filter, a green filter, and a blue filter.
In the image pickup apparatus of the present invention including the various preferred configurations described above, image data having a red component, a green component, and a blue component can be generated for each pixel. With this configuration, the resolution of the image can be improved.
Alternatively, the data of the same color pixels in each pixel group may be added to generate image data. With this configuration, the S/N ratio can be improved by adding pixels of the same color. In the case where pixels receiving infrared light are arranged in a 2 × 2 matrix, data of four pixels may be added to generate image data. In particular, if the pixel array section is configured to add and read accumulated charges of a plurality of pixels, the reading speed can be improved.
Alternatively, three pixels other than the pixel that receives infrared light in each pixel group may be exposed under different exposure conditions according to the pixel. With this configuration, a high dynamic range image can be acquired by using image information having different exposure conditions.
In this case, it is preferable that the air conditioner,
in each pixel group, the pixels receiving infrared light may be disposed at the same position,
a pair of scanning lines may be arranged in each pixel row, an
The pixels in each image row may be alternately connected to one scan line and another scan line in units of one pixel.
Alternatively, in this case,
the pixels receiving infrared light may be disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups, so that, in every four adjacent pixel groups, the pixels receiving infrared light are arranged in a 2 x 2 matrix,
a pair of scanning lines may be arranged in each pixel row, an
The pixels in each image row are alternately connected to one scanning line and the other scanning line in units of two pixels.
The image processing system using the image pickup apparatus of the present invention having various preferred configurations described above may further include a light source section that irradiates infrared light to the subject.
Further, the image processing system of the present invention having the above-described preferred configuration may further include an authentication processing portion that performs an authentication process based on the infrared light image.
In this case, the pixels receiving infrared light may be disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups, so that, in every four adjacent pixel groups, these pixels receiving infrared light are arranged in a 2 × 2 matrix,
a common on-chip lens may be arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light, and
The authentication processing portion may be configured to perform the authentication processing by using at least one of the infrared light image and a depth map generated based on an image plane phase difference of the pixels receiving the infrared light.
Examples of the image pickup apparatus of the present invention including the above-described preferred forms and configurations may include a CMOS image sensor or the like. The image pickup device may be of a front-side illumination type or a back-side illumination type. Then, examples of the image pickup apparatus and the image processing system of the present invention may include a smartphone, a user interface camera for games, a biometric authentication camera, and the like. Various devices including the image pickup device of the present invention can acquire a normal visible light image and an infrared light image without parallax, and can appropriately perform image processing.
Examples of the substrate on which the image pickup element array section is formed may include a semiconductor substrate, particularly a silicon semiconductor substrate. The silicon semiconductor substrate absorbs light having a wavelength of about 1 μm in addition to visible light. Therefore, a photoelectric conversion element such as a photodiode or a phototransistor formed on a silicon substrate can perform photoelectric conversion on infrared light in addition to visible light.
Examples of the color filter and the infrared light transmission filter may include a filter layer that allows transmission of specific wavelengths of red, green, blue, and the like. Various filters may be composed of, for example, a material layer based on an organic material using an organic compound such as a pigment and a dye.
In addition to the color filter and the infrared light transmission filter, an interlayer insulating layer, a planarization layer, and the like constituting the image pickup device can be formed based on known methods such as various chemical vapor deposition methods (CVD methods), coating methods, and various physical vapor deposition methods (PVD methods). Further, examples of the patterning method may include known methods such as a combination of a photolithography technique and an etching technique, and a lift-off method.
In the image pickup apparatus of the present invention, an on-chip lens (OCL) may be arranged above the pixels to improve light collection efficiency. The on-chip lens preferably includes a material that is a transparent material and has a high refractive index.
In the case where the interlayer insulating layer or the planarizing layer used in the image pickup device is formed of a transparent material, for example, an insulating material having no light absorption property, specifically, based on SiOXA material of (a constituent material of a silicon-based oxide film), a low dielectric constant insulating material (such as SiN, SiON, SiOC, SiOF, SiCN, and organic SOG), a polyimide-based resin, a fluorine-based resin, or the like can be used as the material. Basically, the on-chip lens is similar.
The image processing system of the present invention may include an optical portion including a lens that focuses light from an object or the like, a light source portion that irradiates infrared light to the object, and necessary components. The configuration of the light source section is not particularly limited, and a known light emitting element such as an infrared light emitting diode may be used.
The signal processing section that processes a signal from the image pickup device may be configured to operate based on a physical connection of hardware, or may be configured to operate based on a program. The authentication processing section, the viewing processing section, and the control section that controls the entire image processing system and the like are also similar. Further, the image processing system may be integrated into one unit or may be constructed as a separate body.
In the various requirements described in the present specification, for example, various differences caused by design or manufacturing are allowed. In addition, the drawings used in the following description are schematic. For example, fig. 6 described later shows an end face structure of the image pickup apparatus, but the ratios of the width, height, thickness, and the like are not shown.
[ first embodiment ]
The first embodiment relates to an image pickup apparatus and an image processing system using the same according to the present invention.
Fig. 1 is a basic configuration diagram of an image processing system using an image pickup apparatus according to a first embodiment. Fig. 2 is a schematic plan view for explaining the configuration of the image pickup apparatus.
The image processing system 1 shown in fig. 1 includes:
an
an optical portion (imaging lens) 300 that focuses light from an object, and
and a
The operation of the entire image processing system 1 is controlled by a control section (not shown) or the like.
In the
The pixel 101 includes, for example, a photoelectric conversion portion using a Photodiode (PD) or the like, a pixel transistor, and the like. The pixel transistor includes, for example, various transistors such as a transfer transistor, a selection transistor, a reset transistor, and an amplification transistor, although depending on the configuration of the pixel.
Next, the basic operation of the driving part 110 will be explained.
The vertical driving section 113 is configured by, for example, a shift register, selects a predetermined scanning line 104, and drives the pixels 101 connected to the selected scanning line 104 in units of a row. The vertical driving section 113 selectively scans the
The column processing section 111 is arranged for each pixel column. The column processing section 111 processes a signal for one row output from the pixel 101, and performs processing such as Correlated Double Sampling (CDS) and a/D conversion.
The horizontal driving section 112 is also constituted by, for example, a shift register, similarly to the vertical driving section 113. The horizontal driving section 112 sequentially selects each column processing section 111 arranged in each pixel column, and outputs a pixel signal.
In the above, the basic operation of the driving part 110 is explained.
Note that the pixel 101 may be configured so that a binning operation (combining operation) for collectively processing a plurality of adjacent pixels can be performed depending on the operation mode of the
Further, fig. 2 shows that one scanning line 104 and one signal line 103 are arranged in each pixel row and each pixel column, but this is merely an example. For example, in the case of acquiring an HDR image by one image capturing, it is necessary to control the exposure time in accordance with pixels. In this case, it is necessary to arrange a plurality of scanning lines in at least some pixel rows.
Next, the pixel arrangement of the pixel array section is explained. To facilitate understanding of the present invention, first, a pixel arrangement of an image pickup apparatus of a reference example is explained, and then the first embodiment is explained.
Fig. 3 is a schematic plan view for explaining a pixel array in the image pickup apparatus of the reference example.
In the pixel array section 902 of the reference example, pixels are arranged in a form in which one pixel in a regular bayer array is replaced with four pixels arranged in a 2 × 2 matrix. In the drawing, reference symbol R denotes a pixel that receives red light, reference symbol G denotes a pixel that receives green light, and reference symbol B denotes a pixel that receives blue light. The other figures are similar.
The pixel array has advantages in that an image having an excellent S/N ratio can be obtained by adding and reading pixels of the same color, a high-resolution image can be obtained by performing a full-resolution demosaic process, and a high dynamic range can be obtained without requiring multiple image captures by setting different exposure conditions for a plurality of pixels corresponding to one pixel in a bayer array. However, an infrared light image and a visible light image cannot be obtained from one image pickup device.
Fig. 4 is a schematic plan view for explaining a pixel array in the image pickup device according to the first embodiment.
The
Specifically, as a pixel group constituted by four pixels, there are formed:
a first pixel group GP1, which is composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group GP2, which is composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group GP3 composed of three pixels receiving green light and one pixel receiving infrared light, an
And a fourth pixel group GP4 composed of three pixels receiving green light and one pixel receiving infrared light.
Four pixel groups consisting of the first pixel group GP1, the second pixel group GP2, the third pixel group GP3, and the fourth pixel group GP4 are arranged and form a group of 2 × 2 cells, wherein the first pixel group GP1 and the second pixel group GP2 are diagonally positioned, and the third pixel group GP3 and the fourth pixel group GP4 are diagonally positioned. The second embodiment described later is also similar. Note that the pixel receiving infrared light is denoted by reference numeral IR. The other figures are similar.
In each pixel group, pixels that receive infrared light are disposed at the same position. Fig. 4 shows an example of the following case: the pixels receiving infrared light are disposed on the lower right side of the 2 × 2 rows in each pixel group.
In the above, a pixel array is explained. Next, the relationship between the pixel array and the on-chip lens array and the arrangement relationship between the various components will be explained.
Fig. 5 is a schematic plan view for explaining a relationship between a pixel array and an on-chip lens array. Fig. 6 is a schematic partial end view of the image pickup device.
As shown in fig. 5 and 6, in the
In fig. 6, components for red light, green light, blue light, and infrared light are shown arranged in a row for convenience of explanation. The
The
The
Fig. 7 is a schematic diagram of spectral transmittance of the filter. As shown in the figure, color filters for image pickup devices distributed on the market generally exhibit a transmittance close to 100% on the longer wavelength side than 800 nm. Note that, as shown in fig. 28 to be described later, the infrared light transmission filter may be formed by laminating at least two of a red filter, a green filter, and a blue filter.
As described above, in the
In the image processing system 1 shown in fig. 1, a high-resolution image can be obtained by generating image data having a red component, a green component, and a blue component for each pixel. Hereinafter, a description is given with reference to fig. 8.
Fig. 8 is a schematic process diagram showing the procedure of image processing accompanying full-resolution demosaicing processing. For convenience of explanation, processing for a total of 16 pixels in 4 × 4 rows including the first pixel group GP1, the second pixel group GP2, the third pixel group GP3, and the fourth pixel group GP4 is schematically shown. The other figures described later are also similar.
The pixel signal from the
The RGB information is appropriately subjected to interpolation processing to generate high resolution information (step S12: RGB information interpolation). Next, Bayer process is performed (step S14: Bayer process).
As described with reference to fig. 7, the optical filter exhibits transparency to wavelengths in the infrared region in addition to the target light. In fig. 8, bayer information is schematically represented as (R + IR, G + IR, B + IR). Here, a description such as "+ IR" means that an infrared light component is included in addition to a visible light component. The other figures described later are also similar.
The IR information is appropriately subjected to interpolation processing so that the information corresponds to a regular bayer array (step S13: IR information interpolation).
The interpolation processing in steps S12 and S13 is not particularly limited, and may be performed using a known method. Here, an example of the interpolation processing in step S13 will be explained with reference to the drawings.
Fig. 9A and 9B are schematic diagrams for explaining the relationship of pixels used for interpolation in the full-resolution demosaic processing.
In FIG. 9A, for example, the pixel R can be formed by using a pixel R14Ambient infrared pixels (reference IR)2、IR3、IR4And IR23) Is interpolated by the calculation of the following formula and is denoted by reference numeral R 14IR information corresponding to the represented pixels (denoted by reference sign IR)14Is shown)
IR14=(IR2+IR3+IR4+IR23)/4。
Further, in fig. 9B, for example, it is possible to form a pixel by using a pixel R14Upper and lower infrared pixels (reference IR)4And IR23) Is calculated to interpolate the information of (a) and is denoted by reference sign R24IR information corresponding to the represented pixels (denoted by reference sign IR)24Is shown)
IR24=(IR4+IR23)/2。
In the above, the interpolation processing is explained. Next, the processes after step S14 and step S13 are explained with reference to fig. 8.
The infrared light component contained in the interpolated RGB information is removed by separating the interpolated IR information from the interpolated RGB information (step S15: IR component removal). More specifically, the processing is performed in the following state: the IR information after interpolation is multiplied by a predetermined coefficient according to the specifications of the
The RGB information after step S15 is information of a normal bayer array, and is information in which the influence of infrared light is also reduced. Based on this, high-resolution image information can be obtained by performing a normal bayer process (step S16: bayer process).
In the above, the process of obtaining a high resolution image is explained.
Further, in the image processing system 1 shown in fig. 1, an image having excellent S/N can be obtained by adding data of pixels of the same color in each pixel group to generate image data. Hereinafter, a description is given with reference to fig. 10.
Fig. 10 is a schematic process diagram showing the procedure of image processing accompanying pixel addition processing.
The pixel signal from the
Regarding the RGB information, a process of adding pixel information of the same color is performed for each pixel group (step S22: RGB information addition). By adding the information of a plurality of pixels, the S/N is increased.
With respect to the IR information, a process of adding the information included in the four pixel groups is performed (step S23: IR information addition).
The infrared light components contained in the added RGB information are removed by separating the added IR information from the added RGB information (step S24: IR component removal). More specifically, the processing is performed in the following state: the added IR information is multiplied by a predetermined coefficient according to the specifications of the
The RGB information after step S24 is information of a regular bayer array whose resolution is reduced, and is information in which the influence of infrared light is also reduced. Based on this, image information can be obtained by performing normal bayer processing (step S25: bayer processing).
In the above, the process of obtaining an image having excellent S/N is explained.
Further, in the image processing system 1 shown in fig. 1, when three pixels other than the pixel that receives infrared light in each pixel group are exposed under different exposure conditions according to the pixel, an image excellent in dynamic range can be obtained. Hereinafter, a description is given with reference to fig. 11.
Fig. 11 is a schematic process diagram showing a process of image processing accompanying HDR processing.
In the
Fig. 12A is a schematic plan view for explaining an array of pixels having different exposure times. Fig. 12B is a schematic plan view for explaining the relationship between the scanning line connected to the pixel and the control target pixel.
As shown in fig. 12A, in each pixel group, pixels that receive infrared light are disposed at the same position. Then, as shown in fig. 12B, a pair of scanning lines is arranged in each pixel row. The pixels in each image row are alternately connected to one scanning line and another scanning line in units of one pixel. Specifically, the first scan line is connected to the short-time-exposed pixel through the contact point, the second scan line is connected to the medium-time-exposed pixel through the contact point, the third scan line pixel is connected to the long-time-exposed pixel through the contact point, and the fourth scan line is connected to the IR pixel through the contact point. Among the exposure times of the IR pixels, an exposure time suitable for the IR pixels is set. As one example, the exposure time is set to be longer than the exposure time of the medium-time exposure and shorter than the exposure time of the long-time exposure. Note that the relationship between each scan line and each contact point is not limited to the configuration of fig. 12B, and only a configuration in which the same scan line is connected to pixels having the same exposure time through contact points is required.
The pixel signal from the
Regarding the RGB information, RGB information having an expanded dynamic range is synthesized in each pixel group based on pixel information having different exposure times (step S32: HDR information synthesis).
With respect to the IR information, a process of adding information included in the four pixel groups is performed (step S33: IR information addition).
The infrared light component contained in the synthesized RGB information is removed by separating the added IR information from the synthesized RGB information (step S34: IR component removal). More specifically, the processing is performed in the following state: the IR information after the addition is multiplied by a predetermined coefficient according to the specifications of the
The RGB information after step S34 is information of a normal bayer array whose resolution is reduced and is RGB information whose dynamic range is expanded. Based on this, image information having an expanded dynamic range can be obtained by performing a normal bayer process (step S35: bayer process).
In the above, the first embodiment is explained. For example, in the image processing system of the first embodiment, a plurality of pixels may be added at a low illuminance, and a full-resolution signal may be generated by performing re-mosaic processing (re-mosaic) at a high illuminance. Alternatively, by separately controlling the exposure time of the IR pixel and the other pixels, the S/N reduction caused by the IR color mixture can be prevented. Further, for example, in low-illuminance imaging, high-image-quality processing, such as NR processing on a visible light image by using the signal of an IR pixel as a guide, may be performed.
Note that in the above description, the visible light pixels are described as being used for capturing an image containing an infrared light component. In some cases, a dual bandpass filter having transmission bands for visible light and infrared light within a predetermined range may be used to reduce the effect of the infrared light component on the visible light pixels. Fig. 13 is a schematic diagram for explaining the spectral transmittance of the dual band pass filter. Fig. 14 is a schematic diagram for explaining spectral transmittance when a dual band-pass filter and a color filter are overlapped.
[ second embodiment ]
The second embodiment also relates to an image pickup apparatus according to the present invention and an image processing system using the same.
The second embodiment has a configuration similar to that explained in the first embodiment, but the pixel array and the on-chip lens array in the pixel array section are different. In the schematic configuration diagram of the image processing system 2 of the second embodiment, it suffices to replace the
Fig. 15 is a schematic plan view for explaining a pixel array in the image pickup device according to the second embodiment.
The
In the first embodiment, in each pixel group, pixels that receive infrared light are disposed at the same position. On the other hand, in the second embodiment, the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups, so that in every four adjacent pixel groups, the pixels receiving infrared light are arranged in a 2 × 2 matrix.
In the above, a pixel array is explained. Next, the relationship between the pixel array and the on-chip lens array and the arrangement relationship between the various components are explained.
In the second embodiment, similarly to the first embodiment, the on-
Alternatively, unlike the second embodiment, an on-chip lens may be arranged in each of the pixels that receive red, green, and blue light, and a common on-chip lens may be arranged in the pixels that are arranged in a 2 × 2 matrix and receive infrared light. Fig. 17 shows this example, and shows the arrangement relationship between the conventional on-
The image pickup apparatus arranged with the common on-chip lens has an advantage in that distance measurement can be performed by using an image plane phase difference in the IR pixels in addition to photographing an infrared light image. Hereinafter, in the specification, a common on-chip lens is arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light. In general, when a color filter is formed, when a color array is different between adjacent pixels, there is a possibility that: the color filters overlap between adjacent pixels of different colors, and the portion blocks transmission of light and becomes an ineffective area. In the second embodiment, four IR pixels are arranged adjacent to each other, and thus the boundary between different color filters is reduced. Therefore, the overlap of the color filters between adjacent pixels is reduced, and the sensitivity can be improved. More preferably, as shown in fig. 18, pixels arranged in a 2 × 2 matrix and receiving infrared light are formed by a single color filter, and the sensitivity can be further improved.
Also in the image processing system 2 according to the second embodiment, a high-resolution image can be obtained by generating image data having a red component, a green component, and a blue component for each pixel.
Fig. 19 is a schematic process diagram showing the procedure of image processing accompanying full-resolution demosaicing processing.
In the first embodiment, an image processing procedure accompanying the full-resolution demosaic processing is explained with reference to fig. 8. The process in fig. 19 is basically similar except that the calculation formulas for interpolation of RGB information and interpolation of IR information, etc., are different due to the difference in pixel array. Therefore, the description is omitted.
Further, also in the image processing system 2 according to the second embodiment, an image having excellent S/N can be obtained by generating image data by adding data of pixels of the same color in each pixel group. Hereinafter, a description is given with reference to fig. 20.
Fig. 20 is a schematic process diagram showing the procedure of image processing accompanying pixel addition processing.
In the first embodiment, an image processing procedure accompanying the pixel addition processing is explained with reference to fig. 10. The process in fig. 20 is also substantially similar, but differs in that a pixel array level merge operation may be performed for infrared light pixels as well as visible light pixels.
That is, in the first embodiment, the IR pixels are discretely arranged. Therefore, it is difficult to perform a pixel array level merge operation for collectively processing a plurality of adjacent pixels on the IR pixel. Therefore, an operation of separately extracting IR pixel information and then externally adding the information is required, and a processing time is required.
On the other hand, in the second embodiment, the pixel array level merge operation may be performed on the visible light pixels and the infrared light pixels. Therefore, the operation speed can be improved. Illustrating the advantages obtained by increasing the speed of operation.
Fig. 21 is a schematic operation diagram for explaining the influence of ambient infrared light when an infrared light image is captured during a rolling shutter operation.
In the rolling shutter operation, the start and end of the exposure period are changed for each pixel row. Therefore, even in the case where infrared light is actively emitted to capture an image from the shutter of the last row of the previous frame to the readout of the current frame, exposure is performed in a state where only ambient light is emitted during a period according to the readout.
Fig. 22 is a schematic operation diagram for explaining the influence of ambient infrared light when an infrared light image is captured during a rolling shutter operation, following fig. 21.
Fig. 22 shows a case where the reading speed is higher than that in fig. 21. The shutter setting period is the same in fig. 21 and 22. However, the length of the period in which only the ambient light is emitted and exposed is shorter than that in fig. 21. Therefore, by increasing the operation speed, the influence of the ambient infrared light at the time of capturing the infrared light image can be reduced.
In the explanation of the examples of fig. 21 to 22, infrared light is actively emitted from the shutter of the last row of the previous frame to the readout of the current frame. In fig. 21 and 22, if even if only infrared light is actively emitted to a portion shown only as ambient light, the influence of the ambient infrared light can be reduced. In this case, the time of infrared light irradiation can be shortened by increasing the operation speed, and therefore power consumption associated with infrared light irradiation can be reduced.
In the above, the advantages of the pixel array level merge operation are explained.
Further, in the image processing system 2, when three pixels other than the pixel that receives infrared light in each pixel group are exposed under different exposure conditions according to the pixel, an image excellent in dynamic range can be obtained. Hereinafter, a description is given with reference to fig. 23.
Fig. 23 is a schematic process diagram showing a process of image processing accompanying HDR processing.
In the
In the first embodiment, an image processing procedure accompanying HDR processing is explained with reference to fig. 11. The process in fig. 23 is substantially similar, except that the pixel array and the like are different. Therefore, the description is omitted.
Fig. 24A is a schematic plan view for explaining an array of pixels having different exposure times. Fig. 24B is a schematic plan view for explaining the relationship between the scanning line connected to the pixel and the control target pixel.
As shown in fig. 24A, the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups so that the pixels receiving infrared light are arranged in a 2 × 2 matrix in every four adjacent pixel groups. Next, as shown in fig. 24B, a pair of scanning lines is arranged in each pixel row. The pixels in each image row are alternately connected to one scanning line and the other scanning line in units of two pixels. Specifically, the first scan line is connected to the middle-time exposed pixel through the contact point, the second scan line is connected to the long-time exposed pixel through the contact point, the third scan line is connected to the IR pixel through the contact point, and the fourth scan line is connected to the long-time exposed pixel through the contact point. Among the exposure times of the IR pixels, an exposure time suitable for the IR pixels is set. As one example, the exposure time is set to be longer than that of the medium-time exposure and shorter than that of the long-time exposure. Note that the relationship between each scanning line and each contact point is not limited to the configuration of fig. 24B, and any configuration may be made as long as the same scanning line is connected to pixels having the same exposure time through the contact point.
In the above, the second embodiment is explained. In addition to the effects of the image processing system of the first embodiment, the image processing system of the second embodiment has an effect of obtaining an easy pixel array level binning operation.
[ third embodiment ]
The third embodiment relates to an image processing system using the image pickup apparatus according to the present invention.
In the description of the second embodiment, in the configuration in which the common on-chip lens is arranged for the IR pixels, distance measurement can be performed using an image plane phase difference (image plane phase difference) in the IR pixels in addition to capturing an infrared light image. The third embodiment is an image processing system that uses the image pickup apparatus according to the second embodiment and executes authentication processing and viewing processing.
Fig. 25 is a basic configuration diagram of an image processing system that executes authentication processing and viewing processing according to the third embodiment.
The image processing system 3 shown in fig. 25 includes an
Fig. 26 is a schematic process diagram showing an image processing procedure accompanying the authentication processing and the viewing processing.
The pixel signal from the
The separated IR information is used as an infrared light image. Further, a phase difference is detected based on the separated IR information, and information of a range image is generated (step S53: phase difference detection). Then, the authentication processing section 210 performs authentication processing by using at least one of the infrared light image and a depth map generated based on the image plane phase difference of the pixels receiving the infrared light. For example, comprehensive authentication such as 3D face authentication and iris authentication may be performed based on the information of the infrared light image and the distance image (step S54).
Note that it is sufficient if the image processing accompanying the full-resolution demosaic processing, the image processing using pixel addition, and the image processing accompanying the HDR processing described with reference to fig. 19, 20, and 23 are appropriately performed in addition to the above-described steps.
In a system that emits infrared light having a specific pattern as structured light in order to obtain information of a range image, it is difficult to satisfactorily image an iris. That is, in such a system, it is difficult to adopt a configuration in which iris information is used for authentication.
On the other hand, the image processing system 3 does not need to emit infrared light having a specific pattern to obtain information of the range image. That is, it is sufficient if the light source section irradiates planar infrared light irradiation to the object, and thus the iris can be satisfactorily imaged, and the iris information can be used for authentication.
In the above, the third embodiment is explained. In the image processing system of the third embodiment, distance measurement may be performed under irradiation of planar infrared light, and sensing processing such as iris authentication using an infrared light image, 3D face authentication using distance information from an infrared light image, and gesture recognition may be performed while acquiring a viewing image having a monocular configuration.
In face detection and face authentication by capturing visible light, it is difficult to determine spoofing by photography or the like. In this application, the image and the depth map can be acquired in a monocular configuration, and hence spoofing can be determined by photography or the like without additionally adding an IR sensor.
Note that in the case of the image processing system using the image pickup apparatus according to the second embodiment, although a depth map cannot be generated, iris information may be used for authentication. Fig. 27 is a schematic process diagram showing an image processing procedure accompanying authentication processing and viewing processing when the image pickup apparatus according to the first embodiment is used.
[ various modifications ]
Next, various modifications will be described.
In the above description, the visible light pixels are affected by the infrared light due to the characteristics of the color filter. In some cases, the extent to which visible light pixels are affected by infrared light may be reduced.
Fig. 28 is a schematic partial end view of the image pickup apparatus for explaining a filter configuration capable of reducing the influence of infrared light on pixels receiving visible light.
In fig. 28, the filter configuration of the visible light pixel has a two-layer structure. More specifically, the configuration has an infrared
In the example of fig. 28, the filter of the infrared pixel is formed by laminating two types of visible light filters. Therefore, it is not necessary to form the infrared light transmission filter separately from the color filter. As the two kinds of visible light laminated filters used in the infrared light pixel, any two kinds of filters of R, G and B transmission filters may be used. The most suitable combination of any combination is a combination of an R transmission filter and a B transmission filter shown in the example of fig. 28. The reason is that, as shown in the spectral characteristics in fig. 7, the transmission regions of the R transmission filter and the B transmission filter have the smallest overlap in the visible light region. Therefore, by combining the R and B transmission filters, it is possible to more effectively block visible light and transmit infrared light. Note that the filter for visible light used in the infrared light pixel is not limited to two types of stacked filters, and three or more types of filters for visible light may be stacked.
Next, a modification of the pixel arrangement will be explained.
Fig. 29 is a schematic plan view for explaining a first modification of the pixel array.
Fig. 29 is a modification of the pixel array of the first embodiment in which a common on-chip lens is provided in each pixel group. Note that although 8 × 8 pixels are shown in the drawing, only the upper 8 × 4 pixels in the pixel arrangement are shown, and the lower 8 × 4 pixels are shown with the on-chip lens arrangement overlapping the pixel arrangement. Fig. 30, 31, 32, 33, and 34, which will be described later, are also similar.
The configuration of fig. 30 has an advantage in that the image plane phase difference can be detected when capturing visible light. Thus, for example, 3D face authentication using distance information of visible light and a face image of infrared light is possible. Furthermore, similarly, iris authentication is also possible.
Fig. 30 is a schematic plan view for explaining a second modification of the pixel array.
Fig. 30 is a modification of the pixel array of the first embodiment, which is configured such that one pixel includes two photoelectric conversion portions. An on-chip lens is disposed in each pixel.
The configuration of fig. 30 is advantageous in that one pixel includes two photoelectric conversion portions and an image plane phase difference can be detected.
Fig. 31 is a schematic plan view for explaining a third modification of the pixel array.
Fig. 31 shows the following configuration: a pixel group of 2 × 2 pixels each having the same color is arranged, and a part of the pixel group is used for infrared light. An on-chip lens is arranged in each pixel.
Fig. 32 is a schematic plan view for explaining a fourth modification of the pixel array.
Fig. 32 also shows the following configuration: a pixel group of 2 × 2 pixels each having the same color is arranged, and a part of the pixel group is used for infrared light. For visible light pixels, an on-chip lens is arranged in each pixel, and for infrared light pixels, a common on-chip lens is arranged in 2 × 2 pixels.
Fig. 33 is a schematic plan view for explaining a fifth modification of the pixel array.
Fig. 33 also shows the following configuration; a pixel group of 2 × 2 pixels each having the same color is arranged, and a part of the pixel group is used for infrared light. For the visible light pixels and the infrared light pixels, a common on-chip lens is arranged in 2 × 2 pixels.
Fig. 34 is a schematic plan view for explaining a sixth modification of the pixel array.
Fig. 34 also shows the following configuration: a pixel group of 2 × 2 pixels each having the same color is arranged, and a part of the pixel group is used for infrared light. For the visible light pixels and the infrared light pixels, a common on-chip lens is arranged in 2 × 2 pixels.
The configurations of various image pickup apparatuses according to the present invention as described above are advantageous in that one pixel of the bayer array is divided into a plurality of pixels, for example, the S/N ratio is increased by adding and reading pixels of the same color, and the resolution is increased by performing the full-resolution demosaic processing. Information of an infrared light image and information of a visible light image can be obtained from one image pickup device. Further, according to the image processing system using the image pickup apparatus according to the present invention, the sensing process can be performed simultaneously with the acquisition of the viewing image in the monocular configuration.
[ application example ]
The technique according to the present invention can be applied to various products. For example, the technology according to the present invention can be implemented as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobile device, an airplane, an unmanned aerial vehicle, a ship, a robot, a construction machine, and an agricultural machine (tractor).
Fig. 35 is a block diagram showing a schematic configuration example of a
Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage section that stores the programs executed by the microcomputer or parameters and the like used for various arithmetic operations, and a drive circuit that drives various devices as control targets. Each control unit includes a network I/F for communicating with other control units via the
The drive
The vehicle
The vehicle body
The
Vehicle exterior
The environmental sensor may be, for example, at least one of a raindrop sensor for detecting rainy weather, a fog sensor for detecting fog, a sunshine sensor for detecting illuminance of the day, and a snow sensor for detecting snowfall. The environmental information detection sensor may be at least one of an ultrasonic sensor, a radar device, or a light detection and ranging or laser camera ranging (LIDAR) device. The
Here, fig. 36 shows an example of the mounting positions of the
Note that fig. 36 shows an example of the imaging ranges of the
The vehicle exterior
Returning to fig. 35, the description will be continued. The vehicle exterior
Further, the vehicle exterior
The in-vehicle
The
The
The generic communication I/F7620 is a generic communication I/F that mediates communications with various devices present in the
The dedicated communication I/F7630 is a communication I/F supporting a communication protocol defined for use in a vehicle. The dedicated communication I/F7630 may, for example, implement a standard protocol such as wireless access in a vehicular environment (WAVE), which is a combination of lower IEEE802.11p and upper IEEE1609, Dedicated Short Range Communication (DSRC), or cellular communication protocol. The dedicated communication I/F7630 generally performs V2X communication, and V2X communication is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.
For example,
The
The in-vehicle device I/F7660 mediates connection between the
The in-vehicle network I/F7680 is an interface that mediates communication between the
The
Based on information obtained by at least one of the general communication I/F7620, the dedicated communication I/F7630, the
The audio
Note that in the example shown in fig. 35, at least two control units connected via the
The technique according to the present invention can be applied to, for example, the image pickup portion of the vehicle exterior information detection unit in the above-described configuration.
Note that the technique of the present invention may also have the following configuration.
[A1] An image pickup apparatus comprising:
a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 × 2 matrix are arranged,
Among them, as a pixel group constituted by four pixels, there are formed:
a first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
a third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group which is composed of three pixels receiving green light and one pixel receiving infrared light, and
four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, wherein the first pixel group and the second pixel group are diagonally positioned, and the third pixel group and the fourth pixel group are diagonally positioned.
[A2] The image pickup apparatus according to [ a1], wherein,
in each pixel group, pixels that receive infrared light are disposed at the same position.
[A3] The image pickup apparatus according to [ a2], wherein,
an on-chip lens is provided for each pixel.
[A4] The image pickup apparatus according to [ A1] or [ A2], wherein,
any one of the red filter, the green filter, and the blue filter is arranged to correspond to a pixel that receives red light, green light, or blue light, and
The infrared light transmitting filter is arranged to correspond to a pixel that receives infrared light.
[A5] The image pickup apparatus according to [ a4], wherein,
the infrared light transmission filter is formed by laminating at least two of a red filter, a green filter, and a blue filter.
[A6] The image pickup apparatus according to [ a5], wherein,
the infrared light transmission filter is formed by laminating both a red filter and a blue filter.
[A7] The image pickup apparatus according to [ a1], wherein,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that these pixels receiving infrared light are arranged in a 2 × 2 matrix in every four adjacent pixel groups.
[A8] The image pickup apparatus according to [ a7], wherein,
this provides an on-chip lens for each pixel.
[A9] The image pickup apparatus according to [ a7], wherein,
an on-chip lens is arranged in each of the pixels receiving red, green and blue light, and
common on-chip lenses are arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light.
[A10] The image pickup apparatus according to any one of [ A7] to [ A9], wherein,
any one of the red filter, the green filter, and the blue filter is arranged to correspond to a pixel that receives red light, green light, or blue light, and
A common infrared light transmitting filter is arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light.
[A11] The image pickup apparatus according to [ a10], wherein,
the common infrared light transmission filter is configured by laminating at least two of a red filter, a green filter, and a blue filter.
[A12] The image pickup apparatus according to any one of [ A1] to [ A11], wherein,
image data having a red component, a green component, and a blue component is generated for each pixel.
[A13] The image pickup apparatus according to any one of [ A1] to [ A11], wherein,
data of pixels of the same color in each pixel group are added to generate image data.
[A14] The image pickup apparatus according to any one of [ A7] to [ A11], wherein,
data of four pixels arranged in a 2 × 2 matrix and receiving infrared light is added to generate image data.
[A15] The image pickup apparatus according to [ a1], wherein,
three pixels other than the pixel that receives infrared light in each pixel group are exposed under different exposure conditions according to the pixel.
[A16] The image pickup apparatus according to [ a15], wherein,
in each pixel group, pixels receiving infrared light are disposed at the same position,
a pair of scanning lines are arranged in each pixel row, an
The pixels of each image row are alternately connected to one scanning line and another scanning line in units of one pixel.
[A17] The image pickup apparatus according to [ a15], wherein,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that, in every four adjacent pixel groups, these pixels receiving infrared light are arranged in a 2 x 2 matrix,
a pair of scanning lines are arranged in each pixel row, an
The pixels in each image row are alternately connected to one scanning line and the other scanning line in units of two pixels.
[B1] An image processing system comprising:
an image pickup device for picking up an image of an object and a signal processing section for processing a signal from the image pickup device,
wherein the image pickup device includes a pixel array section in which a plurality of pixel groups each composed of four pixels arranged in a 2 x 2 matrix are arranged,
as a pixel group including four pixels, there are formed:
a first pixel group composed of three pixels receiving red light and one pixel receiving infrared light,
a second pixel group composed of three pixels receiving blue light and one pixel receiving infrared light,
A third pixel group composed of three pixels receiving green light and one pixel receiving infrared light, an
A fourth pixel group which is composed of three pixels receiving green light and one pixel receiving infrared light, and
four pixel groups consisting of a first pixel group, a second pixel group, a third pixel group, and a fourth pixel group are arranged and form a group of 2 × 2 cells, and of the 2 × 2 cells of the group, the first pixel group and the second pixel group are located diagonally, and the third pixel group and the fourth pixel group are located diagonally.
[B2] The image processing system according to [ B1], further comprising:
and a light source unit that irradiates infrared light to an object.
[B3] The image processing system according to [ B1] or [ B2], further comprising:
an authentication processing section that executes an authentication process based on the infrared light image.
[B4] The image processing system according to any one of [ B1] to [ B3], wherein,
the pixels receiving infrared light are disposed adjacent to the pixels receiving infrared light in the adjacent pixel groups such that, in every four adjacent pixel groups, these pixels receiving infrared light are arranged in a 2 x 2 matrix,
common on-chip lenses are arranged in pixels arranged in a 2 × 2 matrix and receiving infrared light, and
The authentication processing section is configured to perform authentication processing by using at least one of an infrared light image and a depth map generated based on an image plane phase difference of pixels receiving infrared light.
List of reference numerals
1. 2, 3 image processing system
10 semiconductor substrate
11 photoelectric conversion part
20 planarization layer
21 light-shielding layer
30 optical filter
30RRed filter
30GGreen filter
30BBlue filter
30IRInfrared light transmission filter
30IRAInfrared light absorption (reflection) filter
40 layer of transparent material
41 on-chip lens
41A common on-chip lens
100. 100A image pickup apparatus
101 pixel
102. 102A, 102B, 102C, 102D, 102E, 102F, 102G, 902 pixel array section
10 signal line
104 scan line
110 driving part
111-column processing part
112 horizontal driving part
113 vertical driving part
200 signal processing part
210 authentication processing unit
220 watching processing part
300 optical part
400 infrared light source unit