阅读说明：本技术 一种3d堆叠式图像传感器 (3D heap image sensor ) 是由 李琛 段杰斌 郭奥 郭令仪 左青云 沈灵 于 2019-08-19 设计创作，主要内容包括：本发明公开的一种3D堆叠式图像传感器,包括上层像素衬底和下层处理衬底,其中,所述上层像素衬底包括像素阵列,所述下层处理衬底包括人工智能算法模块,且所述上层像素衬底和下层处理衬底连通；所述人工智能算法模块包括逻辑算法单元和乘加矩阵加速处理单元,且所述乘加矩阵加速处理单元的输出端口连接所述逻辑算法单元的输入端口；所述像素阵列中的像元连接所述逻辑算法单元的输入端口或乘加矩阵加速处理单元的输入端口,所述逻辑算法单元的输出端口输出分析信息。本发明提供的一种3D堆叠式图像传感器,通过将乘加矩阵加速处理单元所在的人工智能算法模块与像素阵列集成在一起,使得图像传感器集成度高,功耗小,运算速度快,能效高。(The invention discloses a 3D stacked image sensor, which comprises an upper layer pixel substrate and a lower layer processing substrate, wherein the upper layer pixel substrate comprises a pixel array, the lower layer processing substrate comprises an artificial intelligence algorithm module, and the upper layer pixel substrate is communicated with the lower layer processing substrate; the artificial intelligence algorithm module comprises a logic algorithm unit and a multiplication and addition matrix acceleration processing unit, and an output port of the multiplication and addition matrix acceleration processing unit is connected with an input port of the logic algorithm unit; and the pixels in the pixel array are connected with the input port of the logic algorithm unit or the input port of the multiplication and addition matrix acceleration processing unit, and the output port of the logic algorithm unit outputs analysis information. According to the 3D stacked image sensor provided by the invention, the artificial intelligence algorithm module where the multiplication and addition matrix acceleration processing unit is located is integrated with the pixel array, so that the image sensor is high in integration level, low in power consumption, high in operation speed and high in energy efficiency.)

1. A3D stacked image sensor is characterized by comprising an upper pixel substrate and a lower processing substrate, wherein the upper pixel substrate comprises a pixel array, the lower processing substrate comprises an artificial intelligence algorithm module, and the upper pixel substrate is communicated with the lower processing substrate;

The artificial intelligence algorithm module comprises a logic algorithm unit and a multiplication and addition matrix acceleration processing unit, and an output port of the multiplication and addition matrix acceleration processing unit is connected with an input port of the logic algorithm unit; pixels in the pixel array are connected with an input port of the logic algorithm unit or an input port of the multiplication and addition matrix acceleration processing unit, and an output port of the logic algorithm unit outputs analysis information;

The pixel signals generated in the pixel array are transmitted to a logic algorithm unit or a multiplication and addition matrix acceleration processing unit for logic operation or AI operation, the pixel signals subjected to AI operation are transmitted to the logic algorithm unit, and the logic algorithm unit is used for intelligently analyzing the pixel signals and outputting analysis information.

2. The 3D stacked image sensor as claimed in claim 1, wherein the artificial intelligence algorithm module further comprises a storage unit, and the storage unit is connected with the logic algorithm unit and the multiplication and addition matrix acceleration processing unit at the same time.

3. The 3D stacked image sensor as claimed in claim 2, wherein the memory unit is a non-volatile magnetic random access memory.

4. A 3D stacked image sensor as claimed in claim 1, wherein said logic algorithm unit comprises a functional subunits, a being an integer greater than 0.

5. The 3D stacked image sensor as claimed in claim 1, wherein the multiplier-adder matrix acceleration processing unit is a cross matrix formed by a resistive random access memory or a phase change memory.

6. The 3D stacked image sensor according to claim 5, wherein the multiplier-adder matrix acceleration processing unit comprises M layers of neural networks, and the output port of the neural network of the previous layer is connected with the input port of the neural network of the next layer until the output port of the neural network of the last layer is connected with the logic algorithm unit; m is an integer greater than 0.

7. A 3D stacked image sensor as claimed in claim 1, wherein said upper pixel substrate further comprises a pad, said pad and lower handle substrate being in communication by deep hole through silicon 3D bonding or hybrid 3D bonding.

8. the 3D stacked image sensor as claimed in claim 7, wherein the size of the deep hole through silicon 3D bonding is 0.5-10 um, and the size of the hybrid 3D bonding is 0.5-5 um.

9. The 3D stacked image sensor as claimed in claim 1, wherein the lower processing substrate further comprises a row and column address decoding module, a programmable gain amplifier module, an analog-to-digital converter module, a digital-to-analog converter module, an on-chip temperature sensor module, a power-on reset module, a current-voltage reference module, a phase-locked loop module, a power control module, a small array memory module, a control chip operation and timing generation module, an artificial intelligence algorithm module, and a mobile industry processor interface MIPI module.

10. The 3D stacked image sensor of claim 1, wherein the pixel array comprises active pixels, redundant pixels, dark pixels, and reference pixels.

Technical Field

The invention relates to the field of image sensors, in particular to a 3D stacked image sensor.

Background

With the continuous improvement of automobile intellectualization and safety, various vehicle-mounted sensors are greatly developed, and the demand is increasing day by day. Currently, vehicle-mounted image sensors are divided into three fields according to applications: 1. and the automobile control realizes gesture recognition, driving state monitoring and the like of a driver by using a 3D camera or a 2D camera. The Advanced Driver Assistance System (ADAS) utilizes various sensors installed on a vehicle to sense the surrounding environment at any time during the driving process of the vehicle, collects data, identifies, detects and tracks static and dynamic objects, and combines with map data of a navigator to perform systematic operation and analysis, thereby enabling a Driver to detect possible dangers in advance and effectively increasing the comfort and safety of vehicle driving. 3. The inside and outside real-time monitoring of the car shows, utilize camera system to realize functions such as scene display outside the car, help the driver to know the environment outside the car in real time, eliminate outer blind area of car, dead angle, improve driving safety. In recent years, due to the continuous development of the automobile market and the continuous progress of the automobile electronic technology, the vehicle-mounted sensor market is rapidly increased, and the composite annual average growth rate of the vehicle-mounted image sensor market is predicted to keep 25.6% of acceleration in the future.

In recent years, with the development of the AI technology in a well-spraying manner, image sensor chips mounted with the AI technology have become a focus of research in the industry. In 2017, a paper published by KAIST on ISSCC provides a single-chip image sensor chip with an AI face recognition function. The chip integrates the face recognition processing chip and the image sensor into one chip, so that the speed and the precision of face recognition are greatly improved, in addition, because the face recognition judgment is finished at the original data level, the output data is greatly reduced, the chip output pins are reduced, the application difficulty of the face recognition chip is greatly reduced, and the chip is also very suitable for the miniaturization of products. In 2018, a paper published by japan CIS manufacturer SONY on ISSCC proposes a monolithic image sensor chip with an AI mode recognition function using a 3D chip stacking technique. The intelligent shooting system has the function of intelligently identifying shooting scenes in a mode, and parameters such as frame rate, resolution and exposure time of the image sensor can be adjusted according to the identified scenes. By adopting the technology, the mode judgment can be completed on the image sensor, the backward output data is greatly reduced, and the application difficulty of the subsequent processing chip is reduced. In addition, by adopting a 3D stacking technology, the AI processor and the image sensor are fused into a chip, so that the speed of the chip is effectively improved, the power consumption is reduced, and the volume of a chip application product is greatly reduced.

At present, an additional AI processing chip is needed for fatigue driving early warning and gesture recognition by using an AI technology, which inevitably affects the volume of a final product and the design difficulty of the product. If the image sensor and the AI processing chip can be integrated on one chip, the competitiveness of the product can be greatly improved.

Disclosure of Invention

The invention aims to provide a 3D stacked image sensor, which integrates an artificial intelligence algorithm module where a multiplication and addition matrix acceleration processing unit is positioned with a pixel array to form a 3D integrated chip, so that the image sensor has high integration level, low power consumption, high operation speed and high energy efficiency.

In order to achieve the purpose, the invention adopts the following technical scheme: A3D stacked image sensor comprises an upper pixel substrate and a lower processing substrate, wherein the upper pixel substrate comprises a pixel array, the lower processing substrate comprises an artificial intelligence algorithm module, and the upper pixel substrate and the lower processing substrate are communicated;

The artificial intelligence algorithm module comprises a logic algorithm unit and a multiplication and addition matrix acceleration processing unit, and an output port of the multiplication and addition matrix acceleration processing unit is connected with an input port of the logic algorithm unit; pixels in the pixel array are connected with an input port of the logic algorithm unit or an input port of the multiplication and addition matrix acceleration processing unit, and an output port of the logic algorithm unit outputs analysis information;

The pixel signals generated in the pixel array are transmitted to a logic algorithm unit or a multiplication and addition matrix acceleration processing unit for logic operation or AI operation, the pixel signals subjected to AI operation are transmitted to the logic algorithm unit, and the logic algorithm unit is used for intelligently analyzing the pixel signals and outputting analysis information.

furthermore, the artificial intelligence algorithm module also comprises a storage unit which is simultaneously connected with the logic algorithm unit and the multiplication and addition matrix acceleration processing unit.

Further, the storage unit is a nonvolatile magnetic random access memory.

Further, the logic algorithm unit comprises a functional subunits, wherein a is an integer greater than 0.

Furthermore, the multiplication and addition matrix acceleration processing unit is a cross matrix formed by a resistance change type memory or a phase change memory.

furthermore, the multiplication and addition matrix acceleration processing unit comprises M layers of neural networks, and an output port of a neural network of an upper layer is connected with an input port of a neural network of a lower layer until an output port of a neural network of a last layer is connected with the logic algorithm unit; m is an integer greater than 0.

Further, the upper layer pixel substrate further comprises a bonding pad, and the bonding pad and the lower layer processing substrate are communicated through deep hole 3D bonding or mixed 3D bonding penetrating through silicon.

Further, the size of the deep hole 3D bonding penetrating through the silicon is 0.5-10 um, and the size of the mixed 3D bonding is 0.5-5 um.

Furthermore, the lower-layer processing substrate further comprises a row address decoding module, a programmable gain amplifier module, an analog-to-digital converter module, a digital-to-analog converter module, an on-chip temperature sensor module, a power-on reset module, a current-voltage reference module, a phase-locked loop module, a power control module, a small array storage module, a control chip working and time sequence generating module, an artificial intelligence algorithm module and a mobile industry processor interface MIPI module.

Further, the pixel array includes active pixels, redundant pixels, dark pixels, and reference pixels.

The invention has the beneficial effects that: the pixel array and the processing substrate are integrated together to form a 3D integrated chip, the 3D integrated chip can realize image sensing, and can integrate an artificial intelligence algorithm module on the chip, so that the invention has the advantage of high integration level; the storage unit is a nonvolatile magnetic random access memory and has high compatibility with a CMOS (complementary metal oxide semiconductor) process; the multiplication and addition matrix acceleration processing unit adopts RRAM or PCM, is compatible with a CMOS (complementary metal oxide semiconductor) process, and has the advantages of high integration level, low power consumption, high operation speed and high energy efficiency compared with the traditional AI multiplication and addition matrix operation based on a CMOS cell library.

Drawings

Fig. 1 is an integrated schematic diagram of an image sensor of the present invention.

Fig. 2 is a schematic diagram of an upper pixel substrate.

fig. 3 is a schematic diagram of a pixel array.

Fig. 4 is a schematic view of an underlying handle substrate.

FIG. 5 is a schematic diagram of a cross matrix formed by RRAM or PCM.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in detail below with reference to the accompanying drawings.

The 3D stacked image sensor provided by the present invention, as shown in fig. 1, includes an upper pixel substrate (Top layer) and a lower processing substrate (Bottom layer), wherein the upper pixel substrate includes a pixel array, the lower processing substrate includes an artificial intelligence algorithm module, and the upper pixel substrate and the lower processing substrate are connected. The artificial intelligence algorithm module comprises a logic algorithm unit and a multiplication and addition matrix acceleration processing unit, and an output port of the multiplication and addition matrix acceleration processing unit is connected with an input port of the logic algorithm unit; the pixels in the pixel array are connected with the input port of the logic algorithm unit or the input port of the multiplier-addition matrix acceleration processing unit, and the output port of the logic algorithm unit outputs analysis information. The pixel signals generated in the pixel array are transmitted to the logic algorithm unit or the multiplication and addition matrix acceleration processing unit to be respectively subjected to logic operation or AI operation, the pixel signals subjected to the AI operation are transmitted to the logic algorithm unit, and the logic algorithm unit is used for intelligently analyzing the initial pixel signals (the pixel signals directly generated by the pixel array) and the pixel signals subjected to the AI operation and outputting analysis information. Wherein, the flow rate of the water is controlled by the control unit.

As shown in fig. 2, the upper pixel substrate (upper pixel Wafer) further includes a PAD (PAD), and the PAD and the lower process substrate in the upper pixel substrate communicate; pixel signals generated by all pixels in the pixel array are led out through a PAD, wherein part of the pixel signals need to be subjected to AI operation and are transmitted to an input port of the multiplication and addition matrix acceleration processing unit, and part of the pixel signals do not need to be subjected to AI operation and can be subjected to traditional operation in the logic algorithm unit and are transmitted to an input port of the logic algorithm unit. And the pixel signals after AI operation are transmitted to a logic algorithm unit, and the logic algorithm unit is used for intelligently analyzing the pixel signals and outputting analysis information. Input ports of a PAD in an upper layer pixel substrate and a logic algorithm unit or a multiplication and addition matrix acceleration processing unit in a lower layer processing substrate are communicated through deep hole 3D bonding (TSV PAD 3D bonding) or Hybrid 3D bonding (Hybrid 3D bonding) of through silicon. Wherein the size of TSV (Through Silicon Via) PAD 3D bonding is 0.5-10 um, preferably 2 um; the dimension of the Hybrid 3D bonding is 0.5-5 um, preferably 1.5um, the dimension here refers to the length and width of the area occupied by the TSV PAD 3D bonding or the Hybrid 3D bonding in the upper layer pixel substrate and the lower layer processing substrate, the TSV PAD 3D bonding or the Hybrid 3D bonding is preferably square, and at this time, the dimension refers to the side length of the square. The dimensions shown in fig. 2 are preferred dimensions for one embodiment, and do not limit the upper pixel substrate of the present invention.

As shown in fig. 3, the Pixel array includes Active pixels (Active pixels), redundant pixels (Dummy pixels), Dark pixels (Dark pixels), Reference pixels (Reference pixels), and the like, wherein the arrangement position of each Pixel can be set with Reference to the related art; preferably, the pixels are arranged in a standard RGGB manner. The arrangement of the pixel elements in fig. 3 is only one arrangement, and does not limit the scope of the present invention.

As shown in fig. 4, the schematic diagram of the lower processing substrate (lower pixel Wafer) is shown, the lower processing substrate includes a Row and column address decoder module (Row decoder/column decoder), a programmable gain amplifier module (PGA), an analog-to-digital converter module (ADC), a digital-to-analog converter module (DAC), an on-chip temperature sensor module, a power-on reset module, a current-voltage reference module, a phase-locked loop module (PLL) for generating a clock for a chip, a power control module (PM), a small-array memory module (Efuse), a timing generation module for controlling the chip operation and simulating IP timing generation, an artificial intelligence algorithm module for processing image data, a mobile industry processor interface MIPI Module (MIPI), etc., wherein the on-chip temperature sensor module, the power-on reset module, the current-voltage reference module, the timing generation module for controlling the chip operation and simulating IP timing generation are too small in size, not shown in the figure. All modules in the underlying processing substrate except the artificial intelligence algorithm module are existing modules, and the implementation manner and functions thereof belong to the prior art and are not described in detail herein. The dimensions shown in fig. 4 are preferred dimensions for one embodiment and are not intended to limit the underlying handle substrate of the present invention.

Referring to fig. 4, the artificial intelligence algorithm module of the present invention includes a logic algorithm unit, a storage unit and a multiplication and addition matrix acceleration processing unit, and an output port of the multiplication and addition matrix acceleration processing unit is connected to an input port of the logic algorithm unit; the pixels in the pixel array are connected with the input port of the logic algorithm unit or the input port of the multiply-add matrix acceleration processing unit, and the output port of the logic algorithm unit outputs analysis information; the storage unit is simultaneously connected with the logic algorithm unit and the multiplication and addition matrix acceleration processing unit.

The logic algorithm unit is used for providing various artificial intelligence algorithms, for example, intelligent acquisition and processing of near-infrared 3D information of a cockpit can be realized through an intelligent CMOS image sensor, and intelligent analysis and control in aspects of driver fatigue driving early warning, dangerous behavior early warning, face recognition, gesture recognition and the like can be realized through the artificial intelligence algorithms. When the image sensor is a vehicle-mounted image sensor, the number of the functional subunits in the logic algorithm unit can be 4, and the functional subunits comprise a face recognition subunit, a fatigue driving early warning subunit, a gesture recognition subunit and a dangerous behavior early warning subunit, and all the subunits can be realized through a unit library provided by a process plant. When the image sensor is used in other scenes, the functional subunits in the logic algorithm unit can be replaced according to specific requirements so as to adapt to different scenes. The functional subunits are directly provided by the existing process to carry out simple logic analysis.

Because the artificial intelligence algorithm needs to store a part of image or buffer a part of intermediate data when processing data, usually the storage function of this part is completed by DRAM (Dynamic Random Access Memory), but because the DRAM process is not compatible with the CMOS standard process, in the present invention, MRAM (Magnetic Random Access Memory) is adopted to realize, and MRAM is a Non-Volatile (Non-Volatile) Magnetic Random Access Memory. It possesses the high-speed read-write capability of Static Random Access Memory (SRAM) and the high integration of dynamic random access memory, and can be written repeatedly basically for infinite times, so that it is very suitable for storing or caching data when processing data as artificial intelligence algorithm. Importantly, the MRAM process is compatible with the CMOS process.

The multiplication and addition matrix acceleration processing unit is a cross matrix formed by a Resistive Random Access Memory (RRAM) or a Phase-Change Memory (PCM). The RRAM or the PCM can form a cross matrix, the multiplication and addition matrix acceleration processing unit comprises M layers of neural networks, an output port of the first layer of neural network is connected with an input port of the second layer of neural network, an output port of the second layer of neural network is connected with an input port of the third layer of neural network until an output port of the M-1 layer of neural network is connected with an input port of the M layer of neural network, and an output port of the M layer of neural network is connected with a logic algorithm unit. As shown in fig. 5, the von neumann architecture oriented separation bottleneck is realized by very naturally realizing vector and matrix multiplication based on memristor cross matrix, and realizing signal parallel computation with extremely low power consumption, thereby providing very high data throughput rate. In fig. 5, V is data input by an input port of a certain layer of neural network, I is data output by an output port of a certain layer of neural network, W is a weight corresponding to the neural network, the weight corresponding to the neural network can be stored in the RRAM or the PCM, and the RRAM or the PCM can store a single value (1bit) or multiple values (2 to 10 bits), preferably 6 bits. The weight corresponding to the neural network can be stored in the RRAM or the PCM, or can be stored in the MRAM first, and is imported from the MRAM to the RRAM or the PCM before calculation, and the storage position of the specific weight can be set according to actual needs. Since neural network computations have different levels, each level has a corresponding input value, output value, and weight value. The multiplication and addition matrix acceleration processing unit carries out AI operation on pixel signals needing AI operation through a neural network, and transmits the result after the operation to the logic algorithm unit, wherein the specific input, operation and output are as follows:

For the first layer of neural network, the input value is the original pixel signal which needs to be subjected to AI operation, the weight value of the first layer of neural network is stored in the MRAM in advance, when the pixel signal is calculated in the first layer of neural network, the weight value is led into the RRAM or the PCM from the MRAM, and the output value is transmitted to the second layer of neural network;

for the second layer of neural network, the input value is the output value of the first layer of neural network, the weight value of the second layer of neural network is pre-stored in the MRAM, the weight value is led into the RRAM or the PCM from the MRAM when the second layer is calculated, and the output value is transmitted to the third layer of neural network;

For the third layer of neural network, the input value is the output value of the second layer of neural network, the weight value of the third layer of neural network is pre-stored in the MRAM, and is imported from the MRAM to the RRAM or PCM when the third layer is calculated, and the output value is output to the fourth layer of neural network;

by analogy, until the last layer of neural network is processed, the final output value (calculation result) is output to the logic algorithm unit, and the AI operation result is combined with the pixel signal output by the pixel array and each functional subunit in the logic algorithm unit, so that the intelligent analysis of the pixel signal can be realized.

The invention connects the upper layer pixel substrate containing the pixel array and the lower layer processing substrate together through the deep hole 3D bonding or the mixed 3D bonding penetrating through silicon, and integrates the image generation and processing on one chip to form a 3D integrated chip, the 3D integrated chip can realize the image sensing, and can integrate the artificial intelligence algorithm module on the chip, and the invention has the advantage of high integration level; the storage unit is a nonvolatile magnetic random access memory and has high compatibility with a CMOS (complementary metal oxide semiconductor) process; the multiplication and addition matrix acceleration processing unit adopts RRAM or PCM, is compatible with a CMOS (complementary metal oxide semiconductor) process, and has the advantages of high integration level, low power consumption, high operation speed and high energy efficiency compared with the traditional AI multiplication and addition matrix operation based on a CMOS cell library.

The above description is only a preferred embodiment of the present invention, and the embodiment is not intended to limit the scope of the present invention, so that all equivalent structural changes made by using the contents of the specification and the drawings of the present invention should be included in the scope of the appended claims.