阅读说明：本技术 利用运动相关像素结合的摄像装置 (Image pickup apparatus using motion-dependent pixel combination ) 是由 曾田岳彦 虎岛和敏 赤堀博男 小林秀央 于 2019-08-23 设计创作，主要内容包括：一种摄像装置包括：像素单元,其具有布置成形成多行和多列的多个像素；读出单元,该读出单元根据划分图案将像素单元划分为多个像素块,各个像素块包括多个像素,并且该读出单元组合来自所述多个像素块中的一个像素块中所包括的多个像素的信号,以生成所述多个像素块中的各个像素块的一个信号；检测单元,其检测由所述像素块的读出单元顺序地生成的多个信号之间的信号值的变化；以及控制单元,其控制读取单元。控制单元响应于检测单元检测到信号值的变化来控制读出单元,以从像素块中所包括的多个像素中的各个像素分别输出信号,并且控制读出单元,使得划分图案在帧之间是不同的。(An image pickup apparatus includes: a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns; a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including a plurality of pixels, and combines signals from the plurality of pixels included in one of the plurality of pixel blocks to generate one signal for each of the plurality of pixel blocks; a detection unit that detects a change in signal value among a plurality of signals sequentially generated by the readout unit of the pixel block; and a control unit that controls the reading unit. The control unit controls the readout unit to output signals from respective ones of a plurality of pixels included in the pixel block, respectively, in response to the detection unit detecting a change in the signal value, and controls the readout unit such that the division pattern is different between frames.)

1. An image pickup apparatus comprising:

a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns;

a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks;

a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and

a control unit that controls the readout unit to output signals from respective pixels among a plurality of pixels included in at least the one pixel block in response to the detection unit detecting a change in the signal value,

wherein the control unit controls the readout unit such that the division pattern is different between at least two frames.

2. The image pickup apparatus according to claim 1,

wherein the plurality of pixel blocks in the division pattern have a first pixel block and a second pixel block, each of the first pixel block and the second pixel block includes a predetermined number of pixels, and

wherein the control unit changes the number of pixels included in the first pixel block or the second pixel block.

3. An image pickup apparatus comprising:

a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns;

a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks;

a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and

a control unit that controls the readout unit to output signals from respective pixels among a plurality of pixels included in at least the one pixel block in response to the detection unit detecting a change in the signal value,

wherein the control unit controls the readout unit such that, in a frame in which the division pattern includes the first pixel block and the second pixel block, the number of pixels included in the first pixel block and the number of pixels included in the second pixel block are different from each other.

4. The image pickup apparatus according to claim 2 or 3,

wherein the number of pixels included in the first pixel block is smaller than the number of pixels included in the second pixel block, and

wherein the control unit changes the number or arrangement of the respective pixel blocks in the first pixel block and the second pixel block by frame.

5. The image pickup apparatus according to claim 4, wherein the first pixel block and the second pixel block are adjacent to each other.

6. The image pickup apparatus according to claim 4 or 5, wherein the second pixel block is arranged closer to a periphery of the pixel unit than the first pixel block.

7. The image pickup apparatus according to claim 4 or 5, wherein the second pixel block is arranged closer to a center of the pixel unit than the first pixel block.

8. The image pickup apparatus according to claim 4 or 5, wherein the first pixel block is arranged between a plurality of the second pixel blocks.

9. The image pickup apparatus according to any one of claims 2 to 8, wherein the control unit reduces the number of pixels included in each pixel block when a change in the signal value is not detected within a predetermined period.

10. The image pickup apparatus according to any one of claims 4 to 8, wherein the control unit changes the number or arrangement of each of the first pixel block and the second pixel block by frame when a change in the signal value is not detected within a predetermined period.

11. The image pickup apparatus according to any one of claims 1 to 10, wherein the readout unit reads out the signal by pixel when a change in the signal value is detected.

12. The image pickup apparatus according to any one of claims 1 to 10,

wherein the readout unit is configured to:

sequentially performing a first mode of reading out signals from each of the plurality of pixel blocks divided by a first division pattern and a second mode of reading out signals from each of the plurality of pixel blocks divided by a second division pattern different from the first division pattern, and

in the first mode and the second mode, when a change in the signal value is detected, a third mode in which signals are read out on a pixel-by-pixel basis is performed.

13. The image pickup apparatus according to claim 12,

wherein when a change in the signal value is detected in the first mode, the first mode transitions to the third mode and then to the first mode, and

wherein when a change in the signal value is detected in the second mode, the second mode is shifted to the third mode and then to the second mode.

14. The image pickup apparatus according to claim 12, wherein after the transition to the third mode, the third mode transitions to a mode in which a frequency of detecting a change in the signal value is highest among the first mode and the second mode.

15. The image pickup apparatus according to claim 12,

wherein when a change in the signal value is not detected within a predetermined period in the first mode, the first mode is switched to the second mode,

wherein when a change in the signal value is not detected within a predetermined period in the second mode, the second mode is switched to the third mode, and

wherein the predetermined period of time of one mode, among the first mode and the second mode, in which a frequency of detecting a change in the signal value is the highest is longer than the predetermined period of time of the other mode.

16. The image pickup apparatus according to any one of claims 1 to 15, wherein each signal of one of the pixel blocks is an added value of signals of a plurality of pixels included in one of the pixel blocks.

17. The image pickup apparatus according to any one of claims 1 to 15, wherein each signal of one of the pixel blocks is an average value of signals of a plurality of pixels included in one of the pixel blocks.

18. The image pickup apparatus according to any one of claims 1 to 15, wherein each signal of one of the pixel blocks is a maximum value of signals of a plurality of pixels included in one of the pixel blocks.

19. The image pickup apparatus according to any one of claims 1 to 18,

wherein each pixel includes: a photoelectric conversion unit that accumulates charges based on incident light; a transfer transistor that transfers charge to the floating diffusion region; an amplifying transistor that outputs a signal based on the charge in the floating diffusion region to a column signal line; and a reset transistor that resets the floating diffusion region,

wherein the pixel unit includes a first switch electrically connecting or disconnecting the floating diffusion regions on the plurality of rows to each other, and

wherein the readout unit combines signals of a plurality of pixels on a plurality of rows in the pixel block by turning on the first switch.

20. The image pickup apparatus according to claim 19,

wherein the pixel unit includes a second switch which electrically connects or disconnects the column signal lines on the plurality of columns to each other, and

wherein the readout unit combines signals of a plurality of pixels on a plurality of columns in the pixel block by turning on the second switch.

21. The image pickup apparatus according to claim 19,

wherein the pixel unit includes a third switch electrically connecting or disconnecting the floating diffusion regions on the plurality of columns to each other, and

wherein the readout unit combines signals of a plurality of pixels on a plurality of columns in the pixel block by turning on the third switch.

22. The image pickup apparatus according to claim 21,

wherein the pixel unit includes a fourth switch electrically connecting or disconnecting the first switch and the third switch with or from the floating diffusion region, and

wherein the readout unit turns off the third switch when reading out the signal per pixel.

23. The image pickup apparatus according to claim 21,

wherein each pixel includes a plurality of photoelectric conversion units on which color filters of different colors from each other are arranged, and

wherein the readout unit reads out signals of photoelectric conversion units provided with color filters of the same color on each row when reading out signals per pixel.

24. The image pickup apparatus according to any one of claims 1 to 22,

wherein the plurality of pixels include color filters having a plurality of colors, and

wherein the readout unit combines signals of a plurality of pixels including color filters of the same color among a plurality of pixels included in one of the pixel blocks.

25. The image pickup apparatus according to any one of claims 1 to 22,

wherein the plurality of pixels include color filters having a plurality of colors, and

wherein the readout unit combines signals of a plurality of pixels including color filters of different colors among a plurality of pixels included in one of the pixel blocks.

26. The image pickup apparatus according to any one of claims 1 to 25, further comprising:

a first substrate on which a pixel unit is formed; and

a second substrate stacked on the first substrate and formed with a readout unit.

27. The image capture device of claim 26, further comprising: and a third substrate which is stacked on the first substrate and the second substrate and is formed with a memory cell which holds a signal read out by the readout unit.

28. The image pickup apparatus according to any one of claims 1 to 27, wherein the control unit sets the number of pixels included in the pixel block or the arrangement of the pixel block based on a learning model that has learned in advance a relationship between the signals from the plurality of pixels or the signals from the pixel block and the number of pixels included in the pixel block or the arrangement of the pixel block.

29. The image pickup apparatus according to claim 28, wherein the control unit learns the learning model by updating weights between nodes of a neural network that is input with signals from the plurality of pixels or signals from the pixel block and outputs the number of pixels included in the pixel block or an arrangement of the pixel blocks, in the neural network.

30. An image pickup apparatus comprising:

a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns;

a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks;

a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and

a control unit that controls the readout unit to output signals from respective pixels among a plurality of pixels included in at least the one pixel block in response to the detection unit detecting a change in the signal value,

wherein the control unit controls the readout unit to output signals from a plurality of pixels included in at least the one pixel block, respectively, when the detection unit does not detect a change in the signal value within a predetermined number of frames.

31. A camera system, comprising:

the image pickup apparatus according to any one of claims 1 to 29; and

and a signal processing unit that processes a signal output from the image pickup device.

32. The imaging system of claim 31, wherein,

wherein the pixel includes a plurality of photoelectric conversion units, and

wherein the signal processing unit processes the signals generated by the plurality of photoelectric conversion units, respectively, and acquires distance information on a distance from the image pickup device to the object.

Technical Field

The present invention relates to an imaging apparatus, an imaging system, and a driving method of the imaging apparatus.

Background

An image pickup apparatus having a function of detecting a motion of an object has been conventionally proposed. The image pickup apparatus disclosed in non-patent document 1 aims to reduce current consumption in a motion detection period by dividing a pixel array into a plurality of pixel blocks and adding and reading out signals within the pixel blocks.

[ list of references ]

[ non-patent document ]

Non-patent document 1: kumagai, et al, "A1/4-inch 3.9Mpixel Low-Power Event-Driven Back-Illuminated Stacked CMOS Image Sensor" ISSCC dig. Tech. papers, pp.86-87, Feb 2018.

Disclosure of Invention

However, in non-patent document 1, the number, arrangement, and the like of pixels forming a pixel block during a motion detection period are not considered, and there is a problem of lowering the motion detection accuracy of the specific object. Alternatively, in the operation flow disclosed in non-patent document 1, when a dark subject moves, when a small subject moves, or the like, a moving object may not be detected. In this case, the opportunity to photograph the subject at high resolution may be lost.

An image pickup apparatus according to one disclosure of the present specification includes: a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns; a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks; a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and a control unit that controls the readout unit to output signals from respective pixels of a plurality of pixels included in at least the one pixel block in response to the detection unit detecting a change in the signal value, respectively, and controls the readout unit so that the division pattern is different in at least two frames.

Another disclosed imaging device according to the present specification includes: a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns; a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks; a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and a control unit that controls the readout unit in response to the detection unit detecting a change in the signal value to output signals from respective pixels of a plurality of pixels included in at least the one pixel block, respectively, and controls the readout unit so that, in a frame in which the division pattern includes the first pixel block and the second pixel block, the number of pixels forming the first pixel block and the number of pixels forming the second pixel block are different from each other.

Another disclosed imaging device according to the present specification includes: a pixel unit having a plurality of pixels arranged to form a plurality of rows and a plurality of columns; a readout unit that divides a pixel unit into a plurality of pixel blocks according to a division pattern, each pixel block including at least two pixels of the plurality of pixels, and combines signals from the at least two pixels of the plurality of pixels included in one of the plurality of pixel blocks to generate one signal of each of the plurality of pixel blocks; a detection unit that detects a change in signal value between the plurality of signals sequentially generated by the readout unit of the one pixel block; and a control unit that controls the readout unit to output signals from respective ones of a plurality of pixels included in at least the one pixel block in response to the detection unit detecting a change in the signal value, and controls the readout unit to output signals from the plurality of pixels included in at least the one pixel block, respectively, when the detection unit does not detect a change in the signal value for a predetermined number of frames.

According to the present invention, it is possible to appropriately capture an object while suppressing current consumption. For example, the accuracy of motion detection of the object can be improved. Alternatively, the subject may be photographed independently of the motion detection.

Drawings

Fig. 1 is a block diagram of an image pickup system in a first embodiment of the present invention.

Fig. 2 is a block diagram of an image pickup apparatus in a first embodiment of the present invention.

Fig. 3 is a conceptual diagram of a pixel block in the first embodiment of the present invention.

Fig. 4 is a diagram showing the arrangement of pixels in the first embodiment of the present invention.

Fig. 5 is a block diagram of a pixel unit in the first embodiment of the present invention.

Fig. 6 is a diagram showing a readout method of a pixel cell in the first embodiment of the present invention.

Fig. 7 is a flowchart showing a driving method of an image pickup apparatus in the first embodiment of the present invention.

Fig. 8 is a flowchart showing a driving method of an image pickup apparatus in a second embodiment of the present invention.

Fig. 9 is a diagram showing a pixel block in a third embodiment of the present invention.

Fig. 10 is a diagram showing a pixel block in a third embodiment of the present invention.

Fig. 11 is a block diagram of an image pickup apparatus in a fourth embodiment of the present invention.

Fig. 12 is a block diagram of an image pickup apparatus in a fifth embodiment of the present invention.

Fig. 13 is a block diagram of an image pickup apparatus in a sixth embodiment of the present invention.

Fig. 14 is a flowchart showing a driving method of an image pickup apparatus in a seventh embodiment of the present invention.

Fig. 15 is a flowchart showing a driving method of an image pickup apparatus in an eighth embodiment of the present invention.

Fig. 16 is a diagram showing a readout method of a pixel cell in a ninth embodiment of the present invention.

Fig. 17 is a diagram showing a readout method of a pixel cell in a ninth embodiment of the present invention.

Fig. 18 is a diagram of a machine learning model in a ninth embodiment of the present invention.

Fig. 19 is a block diagram of a pixel unit in a tenth embodiment of the present invention.

Fig. 20 is a block diagram of a pixel unit in an eleventh embodiment of the present invention.

Fig. 21A is a block diagram of an imaging system in an in-vehicle camera in a twelfth embodiment of the present invention.

Fig. 21B is a block diagram of an image pickup system in an in-vehicle camera of a twelfth embodiment of the present invention.

Detailed Description

Embodiments of the present invention will be described below. An image pickup apparatus in an embodiment described later includes a pixel unit, a readout unit, a detection unit, and a control unit. The pixel unit has a plurality of pixels arranged over a plurality of rows and a plurality of columns. The readout unit divides a pixel unit into a plurality of pixel blocks each including a plurality of pixels according to a predetermined division pattern, and combines signals from the plurality of pixels included in the pixel blocks to generate one signal in each of the pixel blocks. The detection unit detects a change in signal values of a plurality of signals sequentially output from one pixel block. In response to the detection unit detecting a change in the signal value, the control unit controls the readout unit to output signals from respective ones of a plurality of pixels included in the at least one pixel block, respectively. The control unit controls the readout unit such that the division pattern is different between at least two frames.

Specifically, each of the plurality of pixel blocks in the division pattern may include a predetermined number of pixels. The control unit changes the number of pixels included in the pixel block based on the motion of the object detected in the image pickup signals of the plurality of frames. Further, the plurality of pixel blocks in the division pattern may include a first pixel block including a predetermined number of pixels and a second pixel block including a number of pixels greater than the predetermined number of pixels. In this case, the control unit may change the number or arrangement of the respective pixel blocks in the first pixel block and the second pixel block by frame.

By changing the division pattern, such as the number or arrangement of pixels of a pixel block, on a frame-by-frame basis, during a period in which the motion of the object is detected, it is possible to improve the motion detection accuracy while suppressing the current consumption.

Embodiments of the present invention will be described below by using the drawings. The present invention is not limited to the following examples. For example, features of a portion of any of the embodiments described below can be added to or replaced with features of a portion of other embodiments.

[ first embodiment ]

Fig. 1 is a block diagram of the image pickup system of the present embodiment. The camera system may be a digital still camera, a digital camera, a camera head, a surveillance camera, a copier, a facsimile machine, a mobile terminal, a smartphone, an in-vehicle camera, an observation satellite, an artificial intelligence robot, or the like.

The image pickup system shown in fig. 1 has a barrier 101, a lens 102, an aperture 103, an image pickup device 100, a signal processing unit 104, a memory unit 105, an external I/F unit 106, a storage medium control I/F unit 107, a storage medium 108, a machine device 109, and a control unit 110. The barrier 101 protects the lens 102, and the lens 102 forms an optical image of an object on the image pickup apparatus 100. The aperture 103 may change the amount of light that has passed through the lens 102. The image pickup apparatus 100 is a Complementary Metal Oxide Semiconductor (CMOS) type solid-state image pickup apparatus, and converts an optical image formed by the lens 102 into image data. The image pickup apparatus 100 may include a semiconductor substrate on which a pixel circuit, a signal processing circuit, and the like are formed, a package storing the semiconductor substrate, a connection terminal connected to an external circuit, and the like. An analog-to-digital (AD) converter unit is formed on a semiconductor substrate of the image pickup apparatus 100. The signal processing unit 104 performs image processing such as gradation correction, noise removal, and the like on the image data output by the image pickup apparatus 100.

The memory unit 105 has a volatile memory such as a dynamic memory or a nonvolatile memory such as a flash memory, and functions as a frame memory that stores image data. The external I/F unit 106 is a wired or wireless interface for communicating with an external computer, network, server, or the like. The storage medium control I/F unit 107 is an interface that performs storage or readout of image data on the storage medium 108, and the storage medium 108 is a removable storage medium such as a memory card having a semiconductor memory that stores image data. The machine device 109 may include a driving device of an optical mechanism such as the lens 102 and the diaphragm 103, a mechanism device that performs attitude control of the camera head, and the like. The control unit 110 has a CPU, ROM, RAM, and the like, and controls the entire image pickup system according to a predetermined program. Further, the control unit 110 may detect the motion of the subject in the image data and perform predetermined processing thereon. In fig. 1, the signal processing unit 104, the memory 105, and the control unit 110 are provided separately from the image pickup apparatus 100, but may be formed on the same semiconductor substrate as the image pickup apparatus 100.

Fig. 2 is a block diagram of the image pickup apparatus of the present embodiment. In the present embodiment, the circuit elements of the image pickup apparatus 100 are formed on two stacked semiconductor substrates 1A and 1B. The pixel unit 2 is formed on a semiconductor substrate (first substrate) 1A, and readout units such as a vertical scanning circuit 3, an analog-to-digital converter circuit (ADC circuit) 4, a horizontal scanning circuit 5, a signal processing circuit 6, and a control circuit 7 are formed on a semiconductor substrate (second substrate) 1B. For example, the respective wiring layers of the semiconductor substrates 1A and 1B are electrically connected to each other by a metal bond (metallic bond) such as Cu — Cu.

The pixel unit 2 has a plurality of pixels 10 arranged over a plurality of rows and columns, and each pixel 10 has a photoelectric conversion unit that generates and accumulates electric charges based on irradiation light. Note that in this specification, the row direction indicates the horizontal direction in the drawings, and the column direction indicates the vertical direction in the drawings. Microlenses and color filters may be arranged on the pixels 10. The color filters are primary color filters of, for example, red, blue, and green, and are provided on each pixel 10 according to a Bayer (Bayer) arrangement. Some of the pixels 10 are light-shielded as optical black pixels (OB pixels). A column signal line L1 is provided for each column of the pixels 10, and a signal based on incident light is output from the pixels 10 to a column signal line L1.

The vertical scanning circuit 3 is formed of a shift register, a gate circuit, a buffer circuit, and the like, and outputs a driving pulse by a line based on a vertical synchronizing signal, a horizontal synchronizing signal, a clock signal, and the like. A drive pulse is supplied to the pixels 10 on the respective rows. The drive pulses may be provided in rows sequentially or randomly.

The ADC circuit 4 is disposed on each column of the pixels 10, reads out signals from the pixels 10, and performs analog-to-digital conversion on the signals. The ADC circuit 4 has a comparator, a pulse generation circuit, and a digital memory. The comparator is formed of a differential amplifier circuit, and outputs a high-level signal or a low-level signal according to a comparison result between the analog signal on the column signal line L1 and the ramp signal that changes with time. When the output of the comparator is inverted, the pulse generation circuit outputs a one-shot pulse, and the digital memory holds the count value of the counter in response to detection of the trigger pulse. A period from a time when the potential of the ramp signal starts to decrease to a time when the output of the comparator is inverted varies depending on the potential of the signal input to the comparator. The count value held in the digital memory indicates the amplitude of the signal potential.

The horizontal scanning circuit 5 is formed of a shift register, a gate circuit, or the like, and sequentially scans the plurality of ADC circuits 4. That is, the horizontal scanning circuit 5 sequentially reads out digital image data from the digital memory of the ADC circuit 4. The signal processing circuit 6 performs various signal processes such as correlated double sampling, gradation correction, noise reduction, white balance, and the like on the digital image data. The image data from the signal processing circuit 6 is output to the outside of the image pickup apparatus 100.

The control circuit 7 functions as a timing generator that generates various control signals and drive signals based on a clock, a synchronization signal, and the like. The control circuit 7 controls the vertical scanning circuit 3, the ADC circuit 4, the horizontal scanning circuit 5, and the signal processing circuit 6. Further, as described below, the control circuit 7 may divide the pixel unit 2 into a plurality of pixel blocks and control to read out signals by the pixel blocks.

Fig. 3 is a conceptual diagram of a pixel block in the present embodiment. The pixel unit 2 is divided into a plurality of pixel blocks BL, and each pixel block BL has a plurality of pixels 10 formed of m rows and n columns. The image pickup apparatus 100 in this embodiment can read out signals by pixel blocks BL and change the number of pixel blocks BL by frame. Further, the number of pixels 10 forming the image pickup pixel block BL and the shape and size of the pixel block BL may also be changed.

Fig. 4 is a diagram showing the arrangement of pixels in the present embodiment. Color filters of red (R), blue (B), and green (G) are formed on the pixels 10 according to the bayer arrangement. For example, a red color filter R11 is disposed on the first row, first column of pixels 10, and a blue color filter B22 is disposed on the second row, second column of pixels 10. Further, for example, the color filter G12 is arranged on the pixels 10 of the first row and the second column, and the green color filter G21 is arranged on the pixels 10 of the second row and the first column. Note that the color filters are not necessarily required to be formed according to the bayer arrangement, and may be formed using color filters of complementary colors such as magenta, cyan, yellow, and green.

Fig. 5 is a block diagram of a pixel unit of the present embodiment. The pixel unit 2 has a plurality of pixels 10 arranged in a matrix, a column signal line L1, switches M5 and M6, and a constant current source 11. Each pixel 10 includes a photoelectric conversion unit PD, a transfer transistor M1, a reset transistor M2, an amplification transistor M3, a selection transistor M4, and a floating diffusion FD. The following description shows an example in which the transistors forming the pixel 10 are N-channel MOS transistors. The photoelectric conversion unit PD is formed of, for example, a photodiode, and performs photoelectric conversion using the accumulation of incident light and electric charges. Note that the photoelectric conversion unit PD is not limited to a photodiode, and may be a material that generates a photoelectric effect. The photoelectric conversion unit PD is provided with a microlens, and light collected by the microlens enters the photoelectric conversion unit PD.

A driving pulse is input from the vertical scanning circuit 3 to the gate of the transfer transistor M1. When the drive pulse transitions to the high level, the transfer transistor M1 is turned on (on state), and the electric charge of the photoelectric conversion unit PD is transferred to the floating diffusion FD as the input node of the amplification transistor M3. Further, when the drive pulse transitions to the low level, the transfer transistor M1 is turned off (non-conductive state). By turning on or off the transfer transistor M1, the charges of the photoelectric conversion unit PD can be transferred to the floating diffusion FD. The amplification transistor M3 outputs a signal based on the electric charges transferred to the floating diffusion FD to the column signal line L1 via the selection transistor M4.

The source of the reset transistor M2 is connected to the floating diffusion FD, and a drive pulse is input from the vertical scanning circuit 3 to the gate. When the drive pulse transitions to the high level, the reset transistor M2 is turned on, and a reset voltage is supplied to the floating diffusion FD. A selection transistor M4 is disposed between the amplification transistor M3 and the column signal line L1, and a drive pulse is input from the vertical scanning circuit 3 to the gate of the selection transistor M4. When the driving pulse transitions to the high level, the amplifying transistor M3 and the column signal line L1 are electrically connected to each other.

The general operation of the pixel 10 formed as described above will be described. The vertical scanning circuit 3 resets the charge of the floating diffusion FD by turning on the selection transistor M4 and turning on the reset transistor M2. The vertical scanning circuit 3 turns off the reset transistor M2 and ends the reset operation. A signal of a reset state of the pixel 10 is output to the column signal line L1 and input to the ADC circuit 4. After the reset transistor M2 of the pixel 10 is turned off, the voltage of the floating diffusion FD includes reset noise. The vertical scanning circuit 3 turns on the transfer transistor M1, and transfers the charges accumulated in the photoelectric conversion unit PD to the floating diffusion FD. The potential of the floating diffusion FD changes by a predetermined potential according to the amount of charge. A signal based on the electric charges of the photoelectric conversion unit PD is output from the amplifying transistor M3 to the column signal line L1 and input to the ADC circuit 4. By calculating the difference between the signal in the reset state described above and the signal based on the photoelectrically converted electric charges, so-called correlated double sampling is performed, and image data from which noise is removed is obtained.

In the present embodiment, the pixel cell 2 also has switches M5 and M6. A plurality of switches (first switches) M5 connect or disconnect the floating diffusions FD of the pixels 10 on a plurality of rows on the same column to each other. The source of one of the switches M5 is connected to the floating diffusion FD, and the drain is connected to the drain of the other switch M5. A driving pulse is input from the vertical scanning circuit 3 to the gate of the switch M5, and when the driving pulse transitions to a high level, the plurality of floating diffusions FD are electrically connected by the switch M5. For example, when the switch M5 of the first row and the second column is turned on, the charges of the floating diffusion FD of the two pixels 10 having the color filters R11 and G21 are added and a signal based on the added charges is output to the column signal line L1. Further, when the switch M5 on the first to fourth rows is turned on, the charges of the floating diffusion FD on the four pixels 10 having the color filters R11, G21, R31, and G41 are added. In this way, by turning on the switch M5 on a desired row, the number of pixels 10 to be combined (bin) in the column direction (vertical direction) can be arbitrarily changed.

A plurality of switches (second switches) M6 are disposed between adjacent column signal lines L1 and connect or disconnect the column signal lines L1 to each other. For example, the switch M6 has a source connected to the column signal line L1 on the first column and a drain connected to the column signal line L1 on the second column. When a driving pulse is input from the vertical scanning circuit 3 to the gate of the switch M6 and the driving pulse is turned to be high level, the switch M6 makes the column signal lines L1 on the first column and the second column electrically connected to each other. By turning on the switch M5 on a desired column, the number of pixels 10 to be combined in the row direction (horizontal direction) can be arbitrarily changed.

In the present embodiment, by appropriately turning on or off the plurality of switches M5 and M6, the pixel unit 2 can be divided, and the respective numbers of pixels 10 in the row direction and the column direction of the pixel block BL can be arbitrarily changed. For example, assume that the pixel unit 2 is divided into a plurality of pixel blocks BL of m rows and n columns. In this case, the respective pixel blocks BL formed by the M rows and n columns of pixels 10 can be combined to read out one signal by turning on (M-1) switches M5 and turning on (n-1) switches M6. The common signal is output to a plurality of column signal lines L1 connected to one pixel block BL. Thus, it is sufficient to read out signals in n columns and also in m rows and scan. Therefore, the current consumption of the image pickup apparatus 100 can be reduced by increasing the size of the pixel block BL and increasing the number of pixels to be combined. On the other hand, in order to improve the motion detection accuracy between frames, it is preferable to reduce the number of pixels in the pixel block BL and perform high-resolution readout. In the present embodiment, as described below, it is possible to reduce current consumption while improving the motion detection accuracy by dynamically changing the division pattern (such as the number and arrangement of pixels of the pixel block BL).

Note that in fig. 5, although an example in which pixels 10 having a plurality of different color filters (R11, G12, G21, B22, and the like) are combined is shown, pixels 10 having color filters (G12, G21, and the like) of the same color may be combined.

Fig. 6 is a diagram showing a readout method of a pixel unit in the present embodiment, which shows pixel blocks of respective frames and the number of pixels included in the pixel blocks. The pixel unit 2 is divided into a predetermined division pattern including a plurality of pixel blocks BL, and each pixel block BL includes a predetermined number of pixels. In fig. 6, frames are read out in the order of first, second,. In the first frame and the second frame, the pixel block BL is formed of m rows and n columns, i.e., m × n pixels 10. In the nth frame and the (N +1) th frame, the pixel block BL is formed of k rows and L columns, i.e., k × L pixels 10. In the X-th frame and the (X +1) -th frame, the pixel block BL is formed of p rows and q columns, i.e., p × q pixels 10. In the present embodiment, it is desirable to reduce the number of pixels forming the pixel block BL in accordance with the readout time of the frame. Alternatively, it is desirable to satisfy the expression m × n > k × L > p × q. Alternatively, at least one of the expressions m > k > p and n > L > q may be satisfied.

In the first frame and the second frame, the number of divided pixel blocks BL is the smallest, and the number of pixels combined in one pixel block BL is the largest. Since it is sufficient to read out signals in m rows or n columns, current consumption in the image pickup apparatus 100 can be minimized. In the nth and (N +1) th frames, the number of pixel blocks BL is larger than that in the first and second frames, and the motion detection accuracy can be improved. In the X-th and (X +1) -th frames, the number of pixel blocks BL is larger than that in the N-th and (N +1) -th frames, and the motion detection accuracy can be improved.

In the present embodiment, as with the nth frame and the (N +1) th frame, a plurality of frames having the same number of pixel blocks BL are continuous. When the motion detection accuracy is prioritized, it is desirable to increase the number of frames having a larger number of pixel blocks BL. Further, when the reduction of power consumption is prioritized, the number of frames having a smaller number of pixel blocks BL can be increased. Note that although three division patterns of the pixel block BL are shown in fig. 6, two division patterns or four or more division patterns may be used.

Fig. 7 is a flowchart illustrating a driving method of the image pickup apparatus in the present embodiment, which shows a readout method of a signal in a motion detection period (motion detection mode). The image pickup apparatus 100 can sequentially perform a plurality of different modes of division patterns of pixel blocks.

In step S100, the control unit 110 in the image pickup system starts operating according to a predetermined program. The control unit 110 activates the image pickup apparatus 100, and the image pickup apparatus 100 starts accumulation of electric charges based on incident light.

In step S101, the control unit 110 sets the operation mode of motion detection and block readout to mode 1 (first mode), and supplies a signal indicating mode 1 to the image pickup apparatus 100. The control circuit 7 of the image pickup apparatus 100 starts block readout in accordance with the signal of mode 1. Here, for example, block readout in mode 1 is a readout operation by the first division pattern of the pixel blocks BL divided into m rows and n columns. In the mode 1, the number of pixel blocks BL in one frame is the smallest and the number of pixels combined in one pixel block BL is the largest. Therefore, current consumption in the image pickup apparatus 100 can be reduced.

In step S102, the control unit 110 performs motion detection on image data of a plurality of frames. That is, the control unit 110 compares signals of a specific pixel block BL between a plurality of frames, and determines whether a signal difference (change in signal value) between the plurality of frames exceeds a predetermined threshold TH. It is preferable that the plurality of frames to be compared are a plurality of consecutive frames of the pixel blocks BL having the same division pattern, for example, the first frame and the second frame of fig. 6. In the first frame, the difference is not calculated, and the determination result is no. If the signal difference between the frames exceeds the threshold TH (yes in step S102), the control unit 110 determines that the relative position of the object with respect to the background has changed, and the operation of the image pickup apparatus 100 shifts to the all-pixel readout mode (third mode) (step S110). In the all-pixel readout mode, the image pickup apparatus 100 reads out signals for each pixel, and outputs high-definition image data without performing addition readout on the plurality of pixels 10. Note that, instead of reading out the respective pixels 10, addition reading out may be performed in a pixel block BL smaller than the pixel block BL in mode 1. Further, in the all-pixel readout mode, the resolution (the number of bits) and the frame rate of the ADC circuit 4 can be improved as compared with the mode 1.

In the full-pixel readout mode, the image system outputs high-definition image data in which an object is photographed to the storage medium 108 or an external device. Note that, under a predetermined condition, such as after a predetermined period from the transition to the all-pixel readout mode or when an object is no longer detected, the control unit 110 may transition the operation mode to mode 1 in which motion detection is performed (step S101).

If the signal difference between frames does not exceed the threshold TH, i.e., no motion is detected (no at step S102), the control unit 110 determines whether motion detection in mode 1 has been performed for a predetermined number of frames, i.e., within a predetermined period of time (step S103). If the motion detection in mode 1 has not been performed for a predetermined number of frames (no at step S103), the control unit 110 repeats the block readout (step S101) and the motion detection in the next frame (step S102).

If the motion detection in mode 1 has been performed for a predetermined number of frames and the motion of the subject has not been detected within a predetermined period (yes in step S103), the control unit 110 shifts the motion detection and block readout operation mode to mode 2 (second mode). The image pickup apparatus 100 starts block readout according to the mode 2 (step S104). Here, for example, the block readout in the mode 2 is a readout operation by the second division pattern of the pixel blocks BL having k rows and L columns shown in the nth frame and the (N +1) th frame of fig. 6. The number of pixels of the pixel block BL in the mode 2 is smaller than that in the mode 1, and the number of pixel blocks BL included in one frame is larger. Therefore, the motion detection can be performed with higher accuracy than in mode 1.

In step S105, the control unit 110 determines whether the signal difference between the plurality of frames exceeds a predetermined threshold TH. It is preferable that the plurality of frames to be compared are consecutive frames of the pixel blocks BL having the same pattern, for example, the nth frame and the (N +1) th frame of fig. 6. If the signal difference between frames exceeds the threshold TH, that is, if motion is detected (yes at step S105), the control unit 110 shifts the operation of the image pickup apparatus 100 to the full-pixel readout mode (step S110).

If the signal difference between the frames does not exceed the threshold TH, that is, no motion is detected (no in step S105), the control unit 110 determines whether motion detection in mode 2 has been performed for a predetermined number of frames (step S106). If the motion detection in mode 2 has not been performed for a predetermined number of frames (no at step S106), the control unit 110 repeats the block readout (step S104) and the motion detection (step S105) in the next frame. If the control unit 110 has performed motion detection for a predetermined number of frames in mode 2 (yes at step S106), the control unit 110 further reduces the number of pixels of the pixel block BL and performs motion detection.

Subsequently, the control unit 110 and the image pickup apparatus 100 repeat the above-described processing while gradually reducing the size of the pixel block BL. If the motion of the subject has not been detected within a predetermined number of frames (predetermined period of time) ("no" in step S105 and "yes" in step S106), the control unit 110 sets the operation mode of motion detection and block readout to the pattern P (step S107). The mode P is an operation mode in which block readout and motion detection are performed in a predetermined minimum pixel block BL. If no motion has been detected even with the minimum pixel block BL within a predetermined number of frames (no at step S108 and yes at step S109), the control unit 110 sets the operation mode to mode 1 (step S101), and repeats the above-described processing.

As described above, the image pickup apparatus 100 of the present embodiment can change the division pattern, such as the number or arrangement of pixels of the pixel block BL, on a frame basis based on the result of the motion detection. In this embodiment, since block readout can be performed inside the image pickup apparatus 100, current consumption can be reduced. For example, when the pixel block BL is formed of m rows and n columns of pixels 10, it is sufficient to make only the ADC circuits 4 connected to any of the column signal lines L1 on n columns operate in m rows and perform reading and scanning. Therefore, the current consumption of the image pickup apparatus 100 can be reduced by increasing the size of the pixel block BL and increasing the number of pixels to be combined. On the other hand, in order to improve the motion detection accuracy between frames, the size of the pixel block BL can be reduced, and high-resolution readout can be performed. In the present embodiment, by performing motion detection while gradually reducing the pixel blocks BL, both reduction in current consumption and improvement in motion detection accuracy can be achieved.

Note that in steps S102, S105, and S108, although the threshold TH for motion detection may be the same for all modes, it may be set to an optimum value for each mode. Similarly, in steps S103, S106, and S109, although the number of frames used as the determination criterion may be the same for all modes, it may be set to an optimum value for each mode.

Further, in the pixel blocks BL of m rows and n columns, one of the addition in the row direction and the addition in the column direction may be performed inside the image pickup apparatus 100, and the other addition may be performed outside the image pickup apparatus 100. Also in this case, by performing addition inside the image pickup apparatus 100, an advantage of reducing current consumption can be obtained. Further, instead of adding the signals of all the pixels 10 forming the pixel block BL, an average value of the signals of any of the pixels 10 or signals of some of the pixels 10 in the pixel block BL may be used as the representative value of the pixel block BL. Further, in fig. 5, when the switch M5 is turned off, the selection transistors M4 on a plurality of rows may be simultaneously turned on to read out signals on a plurality of rows. When voltages of signals of a plurality of rows simultaneously output to the column signal line L1 are close to each other, an average value of the plurality of signals is output to the column signal line L1. When a certain signal is particularly large in a simultaneously selected row, the maximum pixel value will be output. The average value or the maximum value of the pixel values on a plurality of lines can be used as a representative value of the pixel block BL for motion detection between frames.

[ second embodiment ]

Fig. 8 is a flowchart illustrating a driving method of the image pickup apparatus in the present embodiment, which shows a method of reading out a signal in a motion detection period (motion detection mode). Features different from those of the first embodiment will be mainly described below. The features other than the above have the same configuration as that of the first embodiment. All the description about the first embodiment is applied to the parts of the same configuration as that of the first embodiment.

If no motion has been detected for a predetermined number of frames with the smallest pixel block BL (no at step S108 and yes at step S109), the control unit 110 shifts the operation mode to the full-pixel readout mode (step S110). After a predetermined time has elapsed from the transition to the full-pixel readout mode, and the control unit 110 shifts the operation mode to mode 1 in which motion detection is performed (step S101). At this time, it is possible to transition to any one of the mode 2 to the mode P other than the mode 1. Alternatively, which mode to transition to may be selected according to a predetermined condition.

Note that, in fig. 8, the all-pixel readout mode (step S110) is performed every time one processing cycle from mode 1 to mode P (steps S101 to S109) is performed. In contrast, the all-pixel readout mode may be performed for each of the plural times of processing from mode 1 to mode P (step S110). That is, if the total number of frames from the start of the operation after the processing of the pattern P does not reach the predetermined number of frames (no in step S109), the control unit 110 may transit to the pattern 1 (step S101) without performing the all-pixel readout in step S110 (step S110). On the other hand, if the total number of frames from the start of the operation reaches a predetermined number of frames, the control unit 110 may transit to the mode 1 after performing the all-pixel readout (step S110).

As described above, even if no motion is detected within a predetermined number of frames, transition to the all-pixel readout mode is made, so that even a moving object whose motion detection is difficult can be photographed at high resolution. Note that in the present embodiment, it is not necessary to change the size, the number, and the like of the pixel blocks BL as shown in the first embodiment. For example, the image pickup apparatus 100 performs block reading while always fixing the division pattern of the pixel block, and transitions to the full-pixel readout mode when no motion is detected within a predetermined number of frames. In such a configuration, the following advantages are obtained: high-resolution photographing can be performed even for a moving object whose motion detection is difficult.

[ third embodiment ]

Fig. 9 and 10 are diagrams showing pixel blocks in the present embodiment. In the present embodiment, a plurality of kinds of pixel blocks having different numbers of pixels and different shapes are arranged within one frame. Features different from those of the first embodiment will be mainly described below.

In fig. 9, the pixel unit 2 is divided into a plurality of pixel blocks BL1, BL2, BL3, BL4, … …. For example, the pixel block (first pixel block) BL1 is formed in a substantially square shape and is formed of a smaller number of pixels 10 than the other pixel blocks (second pixel blocks) BL2, BL3, and BL 4. The pixel block BL4 is formed in a substantially square shape like the pixel block BL1, but is formed of a larger number of pixels 10 than the other pixel blocks BL1, BL2, and BL 3. The pixel blocks BL2 and BL3 may be formed of the same number of pixels 10, but have different shapes from each other. By arranging a plurality of kinds of pixel blocks having different numbers of pixels or different shapes in one frame, the motion detection accuracy can be improved in any region. For example, when it is necessary to increase the motion detection accuracy in a portion closer to the center of the imaging region, it is preferable to reduce the number of pixels of the pixel block BL in the portion closer to the center as compared with a portion closer to the periphery.

Further, the division pattern, such as the number, shape, arrangement, and the like of the pixels of the pixel block BL, may be changed by frame. For example, when the subject moves from a portion closer to the center to a portion closer to the periphery, the sizes of the pixel blocks BL2 and BL4 on the portion closer to the periphery may be smaller than the size of the pixel block BL1 at the center. Further, as shown in fig. 10, a pixel block BL6 having a different number of pixels from that of the pixel block BL may be arranged between the pixel blocks BL5 having the same number of pixels. In the case where pixel blocks having different numbers of pixels are adjacently arranged, it is more likely that an object is detected in more pixel blocks BL. Thus, it is possible to reduce the number of pixel blocks BL in one frame and effectively increase the motion detection accuracy while suppressing the current consumption.

[ fourth embodiment ]

Fig. 11 is a block diagram of the image pickup apparatus of the present embodiment. Features different from those of the first embodiment will be mainly described below. In the present embodiment, circuit elements forming the image pickup apparatus 100 are formed on a single semiconductor substrate 1. That is, the pixel unit 2, the vertical scanning circuit 3, the ADC circuit 4, the horizontal scanning circuit 5, the signal processing circuit 6, and the control circuit 7 are formed on the semiconductor substrate 1. When the semiconductor substrate 1 has a sufficient area, the manufacturing cost can be suppressed as compared with the first embodiment in which semiconductor substrates are stacked. Also in the present embodiment, the motion detection accuracy can be increased while reducing the current consumption, in a manner similar to the first embodiment.

[ fifth embodiment ]

Fig. 12 is a block diagram of the image pickup apparatus of the present embodiment. Features different from those of the first embodiment will be mainly described below. In the present embodiment, the image pickup device 100 is formed across three stacked semiconductor substrates 1A, 1B, and 1C. The pixel unit 2 is formed in the semiconductor substrate 1A, and the vertical scanning circuit 3, the ADC circuit 4, the horizontal scanning circuit 5, the signal processing circuit 6, and the control circuit 7 are formed in the semiconductor substrate 1B. A memory circuit such as a Dynamic Random Access Memory (DRAM) is formed in the semiconductor substrate (third substrate) 1C. The DRAM temporarily stores the digitally converted image data. When the rate is limited on the signal path from the image pickup apparatus 100 to the signal processing unit 104, by storing image data in the DRAM, signals can be read out from the pixel unit 2 at a high rate. This enables shooting at a high frame rate, and accurate detection of the motion of a rapidly moving object.

[ sixth embodiment ]

Fig. 13 is a block diagram of the image pickup apparatus of the present embodiment. Features different from those of the first embodiment will be mainly described below. In the present embodiment, the ADC circuit 4 is provided for each pixel 10. Although the circuit size is larger, signals can be read out at a higher rate compared to the first embodiment. This enables shooting at a high frame rate and accurate detection of a fast-moving object.

[ seventh embodiment ]

Fig. 14 is a flowchart illustrating a driving method of the image pickup apparatus in the present embodiment. When motion detection is performed after the all-pixel readout mode, the image pickup apparatus 100 in the present embodiment shifts to the same readout mode as that in which motion detection is performed. Features different from those of the first embodiment will be mainly described below.

In step S200, the control unit 110 starts operation according to a predetermined program. The control unit 110 activates the image pickup apparatus 100, and the image pickup apparatus 100 starts accumulation of electric charges based on incident light.

In step S201, the control unit 110 sets the operation mode of motion detection and block readout to mode 1, and the image pickup apparatus 100 starts block readout according to mode 1. In a similar manner to the first embodiment, in the block readout according to this mode 1, the number of pixels included in one pixel block BL is the largest.

In step S202, the control unit 110 determines whether the signal difference between a plurality of frames exceeds a predetermined threshold TH. If the signal difference between the frames exceeds the threshold TH, that is, if motion is detected (yes in step S202), the control unit 110 shifts the operation of the image pickup apparatus 100 to the full-pixel readout mode (step S210). In the full-pixel readout mode, the image pickup apparatus 100 reads out signals for each pixel and outputs high-definition image data. Under a predetermined condition, such as after a predetermined time has elapsed from the transition to the all-pixel readout mode or when the subject is no longer detected, the control unit 110 makes a transition from the all-pixel readout mode to the motion mode of mode 1 to perform motion detection (step S201).

If the signal difference between frames does not exceed the threshold TH, that is, no motion is detected (no in step S202), the control unit 110 determines whether motion detection in mode 1 has been performed for a predetermined number of frames (step S203). If the motion detection in mode 1 has not been performed for a predetermined number of frames (no at step S203), the control unit 110 repeats the block readout (step S201) and the motion detection (step S202) in the next frame.

If the motion of the object is not detected and the motion detection in mode 1 is performed for a predetermined number of frames (yes in step S203), the control unit 110 shifts the operation mode to mode 2 (step S204). The image pickup apparatus 100 starts block readout and motion detection in mode 2 by using a division pattern that reduces the number of pixels of the pixel block BL.

If the signal difference between frames exceeds the threshold TH, that is, if motion is detected (yes in step S205), the control unit 110 shifts the operation of the image pickup apparatus 100 to the full-pixel readout mode (step S211). Then, under a predetermined condition, the control unit 110 shifts the operation mode from the full-pixel readout mode to mode 2 (step S204).

If the signal difference between the frames does not exceed the threshold TH, that is, no motion is detected (no in step S205), the control unit 110 determines whether motion detection in mode 2 has been performed for a predetermined number of frames (step S206). If the motion detection in mode 2 has not been performed for a predetermined number of frames (no at step S206), the control unit 110 repeats the block readout (step S204) and the motion detection (step S205) in the next frame. If the motion detection in mode 2 is performed for a predetermined number of frames (yes at step S206), the control unit 110 performs block readout and motion detection by using a division pattern that reduces the number of pixels of the pixel block BL.

Then, the control unit 110 and the image pickup apparatus 100 repeat the above-described processing while gradually reducing the size of the pixel block BL. If no motion is detected, the control unit 110 sets the operation mode of motion detection and block readout to the mode P (step S207). The mode P is an operation mode for block readout and motion detection with a predetermined minimum pixel block BL.

If the signal difference between the frames exceeds the threshold TH, that is, if motion is detected (yes at step S208), the control unit 110 shifts the operation of the image pickup apparatus 100 to the full-pixel readout mode (step S212). Then, under a predetermined condition, the control unit 110 shifts the operation mode from the full-pixel readout mode to the mode P (step S207). Also in the minimum pixel block BL, if no motion is detected for a predetermined number of frames (no in step S208 and yes in step S209), the control unit 110 sets the operation mode to mode 1 (step S201).

In the present embodiment, when the full-pixel readout mode is shifted to the block readout mode, the mode is again shifted to the same mode as the mode in which motion detection has been performed. For example, when motion is detected in mode 2 and the full-pixel readout mode is entered, the block readout mode to be performed subsequently is mode 2. In this way, when motion detection is performed after the all-pixel readout mode, the same readout mode as that at the time of motion detection is performed. If the motion of the subject is repeatedly detected in the same area within the image, the motion detection can be efficiently performed by performing the motion detection using the same readout mode. As a result, the speed and accuracy of motion detection can be increased.

[ eighth embodiment ]

Fig. 15 is a flowchart illustrating a driving method of the image pickup apparatus in the present embodiment. When the motion detection mode is resumed from the all-pixel readout mode, the image pickup system in the present embodiment can change the number of frames and the pattern of pixel blocks for motion determination based on the previous motion detection result. Features different from those of the first embodiment and the sixth embodiment will be mainly described below.

The processing from step S300 to step S309 is substantially the same as that of the first and sixth embodiments. When the signal difference between the plurality of frames exceeds the threshold TH (yes in step S302, step S305, and step S308), the control unit 110 shifts the operation of the image pickup apparatus 100 to the full-pixel readout mode (step S310). After a predetermined time has elapsed from the transition to the full-pixel readout mode or under a predetermined condition, the control unit 110 performs the processing of and after step S311 to perform motion detection again.

In step S311, the control unit 110 resets a division pattern (such as the number of pixels, arrangement, or the like of a pixel block) based on the motion detection result (step S302, step S305, or step S308). For example, it is assumed that if the division pattern in pattern 2 is used as a result of repeating the motion detection of steps S301 to S309 by the control unit 110 (step S304), the frequency at which the motion of the object is detected increases. In this case, the control unit 110 changes the division patterns in some frames of the pattern 1 and the pattern P in the same manner as in the pattern 2 (steps S301, S307). Further, the pixel blocks in mode 2 can be divided more finely. Note that the motion detection result may be stored in a memory of the image pickup system or the image pickup apparatus 100.

In step S312, the control unit 110 resets the number of frames at the time of determining the number of frames (step S303, S306, or S309) based on the motion detection result (S302, S305, or S308). For example, it is assumed that if the motion detection in mode 2 is performed as a result of the motion detection of steps S301 to S309 being repeatedly performed by the control unit 110 (step S305), the frequency at which the motion of the object is detected increases. In this case, the control unit 110 increases the number of determination frames in the pattern 2 (step S306). That is, by increasing the rate (number of times) of processing of a mode in which the frequency of motion detection is high, the speed and accuracy of motion detection can be increased.

Subsequently, the control unit 110 repeats the motion detection of steps S301 to S309 by using the set value and the number of frames of the division pattern of the pixel block. Further, the control unit 110 may learn the optimum setting value while repeating the motion detection. As described above, by resetting the division pattern of the pixel block and determining the number of frames based on the result of the motion detection, and by increasing the proportion of the step of signal readout in which the frequency of motion detection is high, the speed and accuracy of motion detection can be increased.

[ ninth embodiment ]

In the present embodiment, an example further extended from the eighth embodiment will be described mainly with respect to features different from the eighth embodiment. The driving method of the present embodiment is substantially the same as the driving method shown in the flowchart of fig. 15 described in the eighth embodiment, but differs in the division pattern of the pixel blocks. The division patterns simplified to the patterns 1 to 3 will be described below. When the motion detection of steps S301 to S309 is performed for the first time, the control unit 110 performs motion detection in the order of mode 1, mode 2, and mode 3.

Fig. 16 is a diagram showing a readout method of a pixel unit in the present embodiment, which shows an initial division pattern of a pixel block. The respective frames are read out in the order of first, second,. -, nth, (N +1),. -, xth, and (X + 1). For the purpose of illustration, the region a, the region B, and the region C are pixel regions of the same size. In the first frame and the second frame corresponding to mode 1, the number of pixel blocks forming the upper left region a in the pixel unit 2 is larger than the number of pixel blocks forming the center region B and the lower right region C. In other words, the number of pixels included in a single pixel block in the region a is smaller than the number of pixels included in a single pixel block in the region B or the region C. Therefore, the motion detection accuracy in the area a is higher than that in the area B or the area C. Similarly, in the nth frame and the (N +1) th frame corresponding to the pattern 2, the number of pixel blocks forming the center region B in the pixel unit 2 is larger than the number of pixel blocks forming the upper left region a and the lower right region C. Therefore, the motion detection accuracy in the region B is higher than that in the region a or the region C. In the X-th frame and the (X +1) -th frame corresponding to pattern 3, since the number of pixel blocks forming the lower right region C is larger than the number of pixel blocks forming the upper left region a and the center region B in the pixel unit 2, the motion detection accuracy in the region C is higher than that in the region a and the region B. Note that although only the regions a to C are representatively indicated in fig. 16, more regions may be set as the image pickup region.

Here, a case where a moving object is not detected in fig. 16 in any mode as a result of the motion detection repeating steps S301 to S309 (yes in step S309) will be described. In this case, the moving object detection may be continued without changing the setting of the division pattern of the pixel block and the number of frames in step S311 and step S312. Alternatively, a setting of changing the division pattern or the number of frames of the pixel block may be employed. The method of such change may be set in advance, or such change may be set randomly.

Next, a case where a moving object is detected in any mode as a result of the motion detection of repeating steps S301 to S309 will be described. If a moving object is detected (yes in any of steps S302, S305, and S308), the control unit 110 shifts to the full-pixel readout mode (step S310). Then, in the same manner as in the case where no moving object is detected, in steps S311 and S312, the moving object detection can be continued without changing the setting of the division pattern and the number of frames of the pixel block. Alternatively, the division pattern or the number of frames of the pixel block may be set to be changed.

The control unit 110 may set an optimal division pattern and the number of frames based on statistical data of the detection result while repeating the motion detection. For example, during repetitive motion detection, when the frequency of motion detection in pattern 1 is high, the proportion of pattern 1 occurrence may increase in the order of pattern 1, pattern 2, pattern 1, pattern 3, pattern 1, pattern 2, pattern 1, … …. When the probability of detecting a moving object in a specific region is high, it is possible to increase the detection frequency in the region and to detect a moving object more efficiently.

Further, during the repetitive motion detection, when the order of the pattern of detecting motion is characterized, the order in which the pattern appears may be changed by resetting the division pattern (such as the number, arrangement, and the like of pixels in the pixel block) in step S311.

For example, it is assumed that the next mode in which the frequency at which the motion of the object is detected is high after the motion of the object is detected in mode 3 as a result of the control unit 110 repeating the motion detection of steps S301 to S309 is mode 2. Further, it is assumed that, after the motion of the object is detected in mode 2, the next mode in which the frequency at which the motion of the object is detected is high is mode 1. In this case, the control unit 110 may reset the division pattern, such as the number, arrangement, and the like of the pixels in the pixel block, so that the division pattern of the pixel block has the order of mode 3, mode 2, and then mode 1, which are initially set.

Fig. 17 is a diagram showing a readout method of a pixel unit in the present embodiment, which shows a division pattern obtained after resetting the division pattern of a pixel block. The frames are read out in the order of first, second, N +1, X, and (X + 1). In the first frame and the second frame corresponding to mode 1, since the number of pixel blocks forming the lower right region C in the pixel unit 2 is larger than the number of pixel blocks forming the upper left region a and the center region B, the motion detection accuracy in the region C is higher than that in the region a and the region B. In the nth frame and the (N +1) th frame corresponding to the pattern 2, since the number of pixel blocks forming the center region B is larger than the number of pixel blocks forming the upper left region a and the lower right region C in the pixel unit 2, the motion detection accuracy in the region B is higher than that in the region a and the region C. In the X-th frame and the (X +1) -th frame corresponding to the pattern 3, since the number of pixel blocks forming the upper left region a is larger than the number of pixel blocks forming the center region B and the lower right region C in the pixel unit 2, the motion detection accuracy in the region a is higher than that in the region B and the region C.

As described above, by changing the order of the division patterns of the pixel blocks so that the frequency of motion detection increases, the speed and accuracy of motion detection can be increased. For example, when the subject moves from the lower right of the screen to the upper left of the screen, the division pattern shown in fig. 17 may be reset.

Further, by classifying the subject based on the image information obtained at the time of motion detection and performing machine learning, it is also possible to predict the motion of the subject and to reset the division pattern such as the number of pixels, arrangement, and the like of the pixel block. Fig. 18 shows a schematic diagram of a neural network of the machine learning model in the present embodiment. The machine learning model may be learned, for example, by the control unit 110 and stored in the memory unit 105. The neural network includes an input layer having a plurality of nodes, an intermediate layer having a plurality of nodes, and an output layer having a single node. The image photographed in the full-pixel mode may be input to each node of the input layer. The respective nodes of the intermediate layer are connected to the respective nodes of the input layer. The respective elements of the input values input to the nodes of the intermediate layer are used in the calculations in the respective nodes of the intermediate layer. For example, each node of the intermediate layer calculates an operation value by using an input value input from each node of the input layer, a predetermined weighting coefficient, and a predetermined bias value. Each node of the intermediate layer is connected to the output layer, and outputs the calculated operation value to the node of the output layer. Operation values are input from the respective nodes of the intermediate layer to the nodes of the output layer. The machine learning model (middle layer) classifies moving objects included in the image. For example, by distinguishing the difference of moving objects recognized such as a person, an animal, a vehicle, or the like and predicting the size, the moving range, or the speed of the subject, it is also possible to reset a division pattern (such as the number of pixels of a pixel block, arrangement, or the like) and perform an output operation from the output layer. Note that information about pixel blocks where moving objects are detected may be added as input to the machine learning model. Thus, a region in which a moving object may exist within the image is identified, and the classification accuracy of the moving object can be improved.

Further, the output of the machine learning model can be used to identify subjects having close relationships with moving objects. For example, roads (on which vehicles may appear), passages, doors, or windows (into or out of which people may enter) are identified as outputs of the machine learning model. The mode is selected or switched so that the pixel block of the region in which the subject exists is reduced.

When the classified subject is not the desired subject, the accuracy of detecting the desired subject can be improved by shifting to setting the pixel block division pattern without performing the all-pixel readout mode (step S310) (step S311).

Note that when an image captured in the all-pixel readout mode is input to a neural network of a machine learning model or the like and control is performed based on the output of the neural network, it is not necessary to sequentially change the size of pixel blocks from mode 1 to mode P, nor to dispose a plurality of pixel blocks having different sizes within one frame. Various methods for setting the pixel blocks are included in the present invention. For example, the output-based control is not limited to the above-described mode selection, but may include control of the timing of transition to the all-pixel readout mode when no moving object is detected. Alternatively, the above control may be applied to control of an exposure period of the image pickup apparatus, gain control inside the image pickup apparatus, control of a frame rate, and the like. Further, the above-described control may be applied to control of the period of output of the full-pixel mode when a moving object is detected.

The information on the input and output of the machine learning model is not limited to the above example. In addition to the image, various information (conditions) such as a shooting time, a shooting place, and the like may be input, and the optimal division pattern and the number of pixels of the pixel block may be output under the respective conditions. By feeding back the result of moving object detection to the machine learning model and updating the respective weighting coefficients between the nodes, learning of the division pattern and the number of pixels of the pixel block that can most effectively detect a moving object under various conditions can be performed. The nodes of the output layer calculate the values of the output layer by using the calculated values, the weighting coefficients, and the offset values input from the respective nodes of the intermediate layer. Note that learning of the neural network may be performed by, for example, an error back propagation method. Specifically, an output value obtained when data is input to the input layer and an output obtained from the teaching data are compared with each other, and an error resulting from the comparison is fed back to the intermediate layer. By repeating this operation until the error becomes lower than a predetermined threshold, learning of a neural network (learning model) can be performed.

[ tenth embodiment ]

Fig. 19 is a block diagram of a pixel unit of the present embodiment. Although the signals of the plurality of columns are added by using the switches provided between the adjacent column signal lines L1 in the first embodiment, in the present embodiment, the signals of the pixels of the plurality of columns are added by using the switches that electrically connect or disconnect the floating diffusion regions of the plurality of columns to each other. Features different from those of the first embodiment will be mainly described below.

The pixel 10 includes a photoelectric conversion unit PD, a transfer transistor M1, a reset transistor M2, an amplification transistor M3, a selection transistor M4, a switch (fourth switch) M10, and a floating diffusion FD. Further, the plurality of pixels 10 are connected to each other via a switch (first switch) M50 and a switch (third switch) M60. In the pixel 10, the source of the switch M10 is electrically connected to the floating diffusion FD, and the drain of the switch M10 is electrically connected to the source of the switch M50, the source of the reset transistor M2, and the source of the switch M60. The drain of one switch M50 is electrically connected to the drain of the other switch M50. A drive pulse is input from the vertical scanning circuit 3 to the gate of the switch M50.

In the block readout mode, i.e., the motion detection mode in which signals are added, signals of the floating diffusion FD of any number of pixels can be added by controlling the on or off of the switches M50 and M60 while the switch M10 is turned on. Specifically, first, the reset transistor M2 is turned on and off, and the floating diffusion FD is reset. Then, the transfer transistor M1 is turned on and off, and the photo-charges of the photoelectric conversion unit PD are transferred to the floating diffusion FD. On the plurality of rows, in response to the switches M10 and M50 being turned on, the floating diffusions FD on the plurality of rows are electrically connected to each other via the switch M10 and the switch M50. Thus, addition readout can be performed in the column direction (vertical direction). Further, on the plural columns, in response to the switches M10 and M60 being turned on, the floating diffusions FD on the plural columns are electrically connected to each other via the switch M10 and the switch M60. Thus, the addition readout can be performed in the row direction (horizontal direction).

In the first embodiment, the addition readout in the row direction is performed by electrically connecting a plurality of column signal lines L1 to each other through a switch M6. In this case, when the difference between the plurality of signals is large, the maximum signal may be output instead of the added value of the signals. For example, a state is assumed in which high-intensity light enters only a certain pixel 10 and substantially no light enters other pixels. In the pixel 10 having high intensity light, the potential of the floating diffusion FD is significantly lowered, the potential difference between the gate and the source of the amplifying transistor M3 is reduced, and thus the amplifying transistor M3 will be turned off. On the other hand, in each low-intensity pixel 10, the potential of the floating diffusion FD is kept in a high state, and the potential of the source of the amplifying transistor M3 is increased. Therefore, the potential of the column signal line L1 will be defined only by the low-intensity pixels 10, and therefore deviates from the ideal added value (average value) of the signals. This tendency is more pronounced for a larger number of column signal lines L1 for addition, and it may be more difficult to detect a motion of a small high-intensity object.

In contrast, according to the present embodiment, by adding the charges in the plurality of floating diffusion regions FD also in the row direction, ideal signal addition can be performed. In particular, when the number of pixels used for addition is large in the row direction, the advantage of the present embodiment is significant. According to the present embodiment, ideal signal addition can be performed in the horizontal direction, and the number of pixels used for addition in the horizontal direction can be increased.

During normal shooting, that is, in the all-pixel readout mode in which signal addition is not performed, the switch M50 and the switch M60 are in an off state. Further, the switch M10 is in an off state during the readout operation except for the reset operation in each pixel 10. Specifically, during the reset, the reset transistor M2 and the switch M10 are simultaneously turned on and off. After the floating diffusion FD is reset, the switch M10 is turned off. In response to the transfer transistor M1 being turned on and off, electric charges are transferred from the photoelectric conversion unit PD to the floating diffusion FD. At this time, since the switch M10 is turned off, the floating diffusion FD is electrically isolated from the switch M50 and the switch M60. Thus, it is possible to prevent signals on the adjacent column signal line L1 from being mixed to the floating diffusion FD due to capacitive coupling or the like, and to avoid occurrence of color mixing. Further, it is also possible to prevent parasitic capacitance of the switch M50 or the switch M60 from attaching to the floating diffusion FD and improve the SN ratio.

[ eleventh embodiment ]

Fig. 20 is a block diagram of a pixel unit in the present embodiment, which shows a modification of the pixel unit in the eighth embodiment. Features different from those of the eighth embodiment will be mainly described below.

The pixel 10 includes photoelectric conversion units PD1 and PD2, transfer transistors M11 and M12, a reset transistor M2, an amplification transistor M3, a selection transistor M4, and a floating diffusion FD. The photoelectric conversion units PD1 and PD2 share a single floating diffusion FD. The photoelectric conversion units PD1 and PD2 are provided with different color filters, respectively. For example, in the pixel 10 in the first row and the first column, a red color filter (R11) is arranged on the photoelectric conversion unit PD1, and a green color filter (G12) is arranged on the photoelectric conversion unit PD 2. By independently turning on or off the transfer transistors M11 and M12, the charges of the photoelectric conversion units PD1 and PD2 can be independently read out, and the pixel 10 can function as a unit pixel including two pixels 10(R11, G12). The floating diffusions FD of the plurality of pixels 10 are connected to each other via the switches M50 and M60. However, unlike the tenth embodiment, no other switch is provided between the floating diffusion FD and the switches M50 and M60.

In the motion detection mode, that is, in the block readout mode in which signals are added, by simultaneously turning on the transfer transistors M11 and M12, the charges of the photoelectric conversion units PD1 and PD2 can be added and read out. Further, as in the tenth embodiment, the floating diffusion regions FD of any number of pixels may be connected to each other by controlling the on or off of the switches M50 and M60.

During normal shooting, that is, in the all-pixel readout mode in which signal addition is not performed, the switch M50 and the switch M60 are in an off state. In the present embodiment, although no switch is provided between the floating diffusion FD and the switches M50 and M60, the colors of the pixels 10 simultaneously read out on the respective rows are the same, and therefore color mixing does not occur. For example, in readout of the pixels 10 of the first row, signals of the red pixels R11, R13, … … are simultaneously read out first, and then signals of the green pixels G12, G14, … … are simultaneously read out. That is, the color components of the signals read out simultaneously on the respective rows are the same. Therefore, in the present embodiment, color mixing can be prevented. Note that a switch may be provided between the floating diffusion FD and the switches M50 and M60 in the same manner as in the tenth embodiment. In this case, it is possible to prevent interference of signals of the same color, and to reduce parasitic capacitance attached to the floating diffusion FD and improve the SN ratio.

[ twelfth embodiment ]

Fig. 21A and 21B show an example in which the image pickup apparatus in any one of the first to eleventh embodiments is applied to an image pickup system relating to an in-vehicle camera. In the present embodiment, the pixel 10 forming the image pickup apparatus 100 may include a first photoelectric conversion unit and a second photoelectric conversion unit. The signal processing unit 104 may be configured to process a signal based on the electric charges generated by the first photoelectric conversion unit and a signal based on the electric charges generated by the second photoelectric conversion unit, and acquire distance information on a distance from the image pickup apparatus 100 to the object.

The imaging system 2000 has an image processing unit 2030 which performs image processing on a plurality of image data acquired by the imaging device 100, and a parallax calculation unit 2040 which calculates a parallax (phase difference of parallax image) from the plurality of image data acquired by the imaging system 2000. Further, the image capturing system 2000 has a distance measuring unit 2050 that calculates a distance to the object based on the calculated parallax, and a collision determining unit 2060 that determines whether there is a possibility of collision based on the calculated distance. Here, the parallax calculation unit 2040 and the distance measurement unit 2050 are examples of a distance information acquisition unit that acquires distance information on the distance to the object. That is, the distance information is information on parallax, defocus amount, distance to the subject, and the like. The collision determination unit 2060 may determine the collision possibility using any distance information. The distance information acquisition unit may be realized by specially designed hardware, or may be realized by a software module. Further, the distance information acquiring unit may be implemented by a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), or by a combination thereof.

The camera system 2000 is connected to the vehicle information acquisition device 2310, and can acquire vehicle information such as a vehicle speed, a yaw rate (yaw rate), a steering angle (steering angle), and the like. The imaging system 2000 is connected to a control ECU 2410, and the control ECU 2410 is a control device that outputs a control signal for causing the vehicle to generate a braking force based on the determination result of the collision determination unit 2060. Further, the camera system 2000 is also connected to a warning device 2420 that gives a warning to the driver based on the determination result of the collision determination unit 2060. For example, when the collision probability is high as a result of the determination by the collision determination unit 2060, the control ECU 2410 performs vehicle control to avoid a collision or reduce damage by applying a brake, pushing back an accelerator, suppressing engine power, or the like. The warning device 2420 warns the user by emitting an alarm such as a sound, displaying warning information on a display of a car navigation system or the like, providing vibration to a seat belt or a steering wheel, or the like. The camera system 2000 functions as a control unit that controls the operation of the control vehicle as described above.

In the present embodiment, an area around the vehicle, for example, a front area or a rear area, is photographed by using the camera system 2000. Fig. 21B shows the imaging system when the front area of the vehicle (imaging area 2510) is imaged. The vehicle information acquisition device 2310 as a photographing control unit transmits an instruction to the image pickup system 2000 or the image pickup device 100 to perform the operations described in the above first to eleventh embodiments. Since the operation of the image pickup apparatus 100 is the same as that in the first to eleventh embodiments, the description thereof will be omitted here. Such a configuration can further improve the ranging accuracy.

Although the example of the control for avoiding a collision with another vehicle has been described above, the embodiment is applicable to the automatic driving control following another vehicle, the automatic driving control without deviating from the lane, and the like. Further, the imaging system is not limited to a vehicle such as a subject vehicle, and may be applied to a mobile unit (mobile device) such as a ship, an airplane, or an industrial robot, for example. In addition, the camera system can be widely applied to devices using object recognition, such as an Intelligent Transportation System (ITS), without being limited to mobile units.

[ thirteenth embodiment ]

Although the signal processing unit 104 and the control unit 110 that perform the motion detection processing and the determination processing of the mode of the readout block are disposed outside the image pickup apparatus in the above-described embodiment, the signal processing unit 104 and the control unit 110 may be disposed inside the image pickup apparatus. For example, the signal processing unit 104 and the control unit 110 may be mounted on a semiconductor substrate (third substrate) 1C shown in fig. 12. When the rate is limited on the signal path from the image pickup apparatus 100 to the signal processing unit 104, by disposing the signal processing unit 104 and the control unit 110 inside the image pickup apparatus 100, the signal transmission path to the signal processing unit 104 and the control unit 110 can be shortened. Thus, the signal processing unit 104 can read out signals at a high rate that enables shooting at a high frame rate, and the control unit 110 can accurately detect the motion of a rapidly moving object.

[ fourteenth embodiment ]

The signal processing unit 104 and the control unit 110 may be mounted on a semiconductor substrate (second substrate) 1B shown in fig. 13. Also in the present embodiment, when the rate is limited on the signal path from the image pickup apparatus 100 to the signal processing unit 104, by arranging the signal processing unit 104 and the control unit 110 inside the image pickup apparatus 100, the signal transmission path to the signal processing unit 104 and the control unit 110 can be shortened. This enables shooting at a high frame rate and accurate detection of the motion of a rapidly moving object.

[ other examples ]

The present invention is not limited to the above-described embodiments, and may be implemented in various forms without departing from the technical concept of the present invention or the main features thereof. For example, an example in which a part of the configuration of any embodiment is added to or replaced with a part of the configuration of another embodiment is one of the embodiments of the present invention.

Embodiments of the present invention may be implemented by a computer of an image pickup system or an image pickup apparatus, which reads out and executes computer-executable instructions (e.g., one or more programs) stored in a storage medium. Furthermore, an Application Specific Integrated Circuit (ASIC) may be used as the non-transitory computer readable storage medium. A storage medium storing program codes for realizing the above-described functions may be supplied to the image pickup system or the image pickup apparatus. Further, the camera system or the camera apparatus may download a program to perform some or all of the above functions through a network or a server.

A processor (e.g., a Central Processing Unit (CPU), a Micro Processing Unit (MPU)) may be included in the image pickup system or the image pickup apparatus. The computer-executable instructions may be provided to the computer from, for example, a network or a storage medium. For example, the storage medium may be a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), a storage device of a distributed computing system, an optical disk (e.g., a Compact Disk (CD), a Digital Versatile Disk (DVD), a blu-ray disk (BD) (registered trademark)), a flash memory device, a memory card, or the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

The present application claims the benefits of japanese patent application No. 2018-163852, filed on 31.8.2018 and japanese patent application No. 2019-108210, filed on 10.6.2019, which are hereby incorporated by reference in their entirety.

[ list of reference numerals ]

BL, BL1, BL2 pixel block

2 pixel unit

3 vertical scanning circuit

4 ADC circuit

5 Signal processing circuit

6 horizontal scanning circuit

7 control circuit

10 pixels

100 image pickup device

110 control unit

PD1 and PD2 photoelectric conversion units

M1 transfer transistor

M2 reset transistor

M3 amplifying transistor

M4 selection transistor

FD floating diffusion region

M5, M6, M10, M50 and M60 switches

L1 column signal line