阅读说明：本技术 影像处理电路 (Image processing circuit ) 是由 杨得炜 于 2019-01-02 设计创作，主要内容包括：本发明提出一种影像处理电路,其中存储器控制单元先将参考影像与参数从第一存储器移动至第三存储器,卷积神经网络电路从第三存储器取得参数。存储器控制单元也将至少一部分的参考影像从第三存储器移动至第二存储器,其中第三存储器的储存空间大于第二存储器的储存空间。深度解码器从第二存储器取得参考影像,根据此参考影像与结构化影像计算出深度信息,并将深度信息储存至第二存储器。(The invention provides an image processing circuit, wherein a memory control unit firstly moves a reference image and parameters from a first memory to a third memory, and a convolutional neural network circuit obtains the parameters from the third memory. The memory control unit also moves at least a part of the reference image from the third memory to the second memory, wherein the storage space of the third memory is larger than that of the second memory. The depth decoder obtains the reference image from the second memory, calculates the depth information according to the reference image and the structured image, and stores the depth information into the second memory.)

1. An image processing circuit, comprising:

a first memory for storing a reference image and a plurality of parameters;

a second memory, wherein the storage space of the second memory is smaller than that of the first memory;

a memory control unit electrically connected to the first memory and the second memory;

a convolution neural network circuit electrically connected to the second memory; and

a depth decoder electrically connected to the second memory for receiving a structured image from an infrared sensor,

wherein in a first mode:

the memory control unit moves the reference image and the plurality of parameters from the first memory to a third memory, wherein the third memory is electrically connected to the convolutional neural network circuit, the memory control unit and the second memory, and the storage space of the third memory is larger than that of the second memory;

the convolutional neural network circuit obtains the plurality of parameters from the third memory, and the memory control unit moves at least one part of the reference image from the third memory to the second memory; and

the depth decoder obtains the at least one portion of the reference image from the second memory, calculates depth information according to the at least one portion of the reference image and the structured image, and stores the depth information in the second memory.

2. The image processing circuit of claim 1, further comprising:

a bus electrically connected to the memory control unit, the second memory and the convolutional neural network circuit; and

a multiplexer electrically connected to the bus, the third memory and the convolutional neural network circuit,

in the first mode, the convolutional neural network circuit obtains the plurality of parameters from the third memory through the multiplexer.

3. The image processing circuit of claim 2 wherein in the first mode:

the memory control unit also moves a portion of the plurality of parameters from the third memory to the second memory via the bus; and

the convolutional neural network circuit retrieves the portion of the plurality of parameters directly from the second memory while the memory control unit moves the at least a portion of the reference image from the third memory to the second memory via the bus.

4. The image processing circuit of claim 3 wherein the portion of the plurality of parameters corresponds to a layer of a convolutional neural network.

5. The image processing circuit as claimed in claim 1, wherein the second memory further receives a gray-scale image from the infrared sensor,

the convolutional neural network circuit also obtains the depth information and the gray-scale image from the second memory, and executes an object detection program or an object identification program according to the gray-scale image, the depth information and the parameters.

6. The image processing circuit of claim 1 wherein the at least a portion of the reference image comprises at least one column of pixels in the reference image.

7. The image processing circuit as claimed in claim 1, wherein the second memory is a static random access memory and the third memory is a dynamic random access memory.

8. The image processing circuit as claimed in claim 7, wherein the first memory is a flash memory.

9. The image processing circuit as claimed in claim 1, wherein the memory control unit switches between the first mode and a second mode in which:

the memory control unit moves the reference image and the plurality of parameters from the first memory to the second memory;

the convolutional neural network circuit retrieves the plurality of parameters from the second memory; and

the depth decoder obtains the reference image from the second memory, calculates the depth information according to the reference image and the structured image, and stores the depth information into the second memory.

10. The image processing circuit as claimed in claim 9, wherein the memory control unit switches between the first mode and the second mode according to a physical switch, a detection circuit or a firmware setting.

Technical Field

The present invention relates to an image processing circuit, and more particularly, to a memory architecture of a convolutional neural network circuit.

Background

In recent years, the technology of convolutional neural networks has received a lot of attention from both academic and industrial fields, and has been developed in a breakthrough in many fields, especially in image processing. When the convolutional neural network is used for image processing, a large amount of memory is often required to store the image and parameters in the convolutional neural network, and therefore, how to design an appropriate hardware architecture to store or transport the data is a concern for those skilled in the art.

Disclosure of Invention

An embodiment of the invention provides an image processing circuit, which includes a first memory, a second memory, a memory control unit, a convolutional neural network circuit, and a depth decoder. The first memory stores a reference image and a plurality of parameters, and the storage space of the second memory is smaller than that of the first memory. The memory control unit is electrically connected to the first memory and the second memory, the convolutional neural network circuit is electrically connected to the second memory, and the depth decoder is electrically connected to the second memory. The depth decoder is used for receiving the structured image from the infrared sensor. In a first mode: the memory control unit moves the reference image and the parameters from the first memory to a third memory, wherein the third memory is electrically connected to the convolutional neural network circuit, the memory control unit and the second memory, and the storage space of the third memory is larger than that of the second memory; the convolutional neural network circuit obtains the parameters from the third memory, and the memory control unit moves at least one part of the reference image from the third memory to the second memory; and the depth decoder acquires a part of the reference image from the second memory, calculates depth information according to the part of the reference image and the structured image, and stores the depth information into the second memory.

In some embodiments, the image processing circuit further includes a bus and a multiplexer. The bus is electrically connected to the memory control unit, the second memory and the convolutional neural network circuit, and the multiplexer is electrically connected to the bus, the third memory and the convolutional neural network circuit. In the first mode, the convolutional neural network circuit obtains the parameters from the third memory through the multiplexer.

In some embodiments, in the first mode, the memory control unit also moves a portion of the parameter from the third memory to the second memory over the bus; and when the memory control unit moves the part of the reference image from the third memory to the second memory through the bus, the convolutional neural network circuit directly obtains the parameters from the second memory.

In some embodiments, the parameters of the portion are layers corresponding to a convolutional neural network.

In some embodiments, the second memory further receives the gray-scale image from the infrared sensor. The convolutional neural network circuit also obtains the depth information and the gray-scale image from the second memory, and executes an object detection program or an object identification program according to the gray-scale image, the depth information and the parameters.

In some embodiments, the portion of the reference image includes at least one row of pixels.

In some embodiments, the second memory is a static random access memory, and the third memory is a dynamic random access memory.

In some embodiments, the first memory is a flash memory.

In some embodiments, the memory control unit switches between a first mode and a second mode. In a second mode: the memory control unit moves the reference image and the parameters from the first memory to the second memory; the convolutional neural network circuit retrieves the parameters from the second memory; and the depth decoder acquires the reference image from the second memory, calculates the depth information according to the reference image and the structured image, and stores the depth information into the second memory.

In some embodiments, the memory control unit switches between the first mode and the second mode according to a physical switch, a detection circuit, or a firmware setting.

In the image processing circuit, the space requirement of the memory can be reduced, or different memory architectures can be adopted under different modes.

In order to make the aforementioned and other features and advantages of the invention more comprehensible, embodiments accompanied with figures are described in detail below.

Drawings

FIG. 1 is a circuit diagram illustrating an electronic device according to an embodiment.

FIG. 2 is a circuit diagram illustrating an electronic device according to an embodiment.

Description of reference numerals:

100: electronic device

110: infrared sensor

111: infrared projector

120: image processing circuit

121: first memory

122: second memory

123: third memory

130: memory control unit

140: convolutional neural network circuit

150: depth decoder

160: bus line

170: multiplexer

Detailed Description

With respect to the terms "first", "second", … …, etc., herein, no particular order or sequential meaning is implied and should be taken merely to distinguish one element or operation from another element or operation described in similar technical language.

[ first embodiment ]

FIG. 1 is a circuit diagram illustrating an electronic device according to an embodiment. Referring to fig. 1, the electronic device 100 includes an infrared sensor 110, an infrared projector 111 and an image processing circuit 120. The electronic device 100 may be implemented as a mobile phone, a tablet computer, a notebook computer, or other suitable electronic device. In some embodiments, the electronic device 100 may further include other elements or components, such as a camera, and the like, which are not limited herein. The image processing circuit 120 includes a first memory 121, a second memory 122, a third memory 123, a memory control unit 130, a convolutional neural network circuit 140, a depth decoder 150, a bus 160, and a multiplexer 170. The second memory 122 is electrically connected to the infrared sensor 110, the depth decoder 150, and the convolutional neural network circuit 140. The bus 160 is electrically connected to the second memory 122, the memory control unit 130, and the multiplexer 170. The first memory 121 is electrically connected to the memory control unit 130, and the multiplexer 170 is electrically connected to the convolutional neural network circuit 140 and the third memory 123.

In this embodiment, the image processing circuit 120 is configured to execute an image processing procedure, such as an object detection procedure or an object identification procedure. When executing the image processing procedure, a depth map of the scene is required, which is calculated by comparing the displacement (disparity) between specific patterns in two infrared images, which are referred to herein as a reference image and a structured image. The pattern in the reference image is a predetermined pattern, also called gold pattern (gold pattern), such as a plurality of dots or stripes arranged randomly, and the invention is not limited to this predetermined pattern. The structured image is projected by the infrared projector 111 to form the predetermined pattern into the scene, and then the infrared sensor 110 senses the obtained image, i.e. the structured image also has the predetermined pattern. However, due to the depth of the scene, the predetermined pattern in the structured image is shifted accordingly, so that the depth of field at a certain position in the reference image and the structured image can be calculated by calculating the shift of the position. In this embodiment, the reference image is stored in the first memory 121.

Convolutional neural network circuitry 140, on the other hand, is used to implement a convolutional neural network. Generally, a convolutional neural network has a plurality of layers, and each layer has a plurality of parameters, which are determined in advance through training. In this embodiment, the parameters are also stored in the first memory 121.

In some embodiments, the storage space of the first memory 121 and the third memory 123 are larger than the storage space of the second memory 122, but the reading speed of the second memory 122 is higher than the reading speed of the first memory 121 and the third memory 123. For example, the first memory 121 is a flash memory, the second memory 122 is a static random access memory, and the third memory 123 is a dynamic random access memory, but the invention is not limited thereto. Specifically, the parameters and the reference image are first transferred from the first memory 121 to the third memory 123, and then transferred from the third memory 123 to the second memory 122.

Specifically, the memory control unit 130 moves the reference image and the parameter from the first memory 121 to the third memory 123, and the multiplexer 170 selects the input terminal "0". Next, the multiplexer 170 may continue to select the input "0", and the memory control unit 130 moves at least part of the reference image from the third memory 123 to the second memory 122. The depth decoder 150 obtains the reference image of the portion from the second memory 122, calculates depth information according to the reference image and the structured image received from the infrared sensor 110, and finally stores the depth information in the second memory 122. In some embodiments, since the depth decoder 150 calculates the depth information row by row (row), the reference image of the portion only needs to include one or more rows of pixels, in other words, the memory control unit 130 moves one or more rows of pixels from the third memory 123 to the second memory 122 at a time, so that the space requirement of the second memory 122 is reduced because the entire reference image does not need to be loaded into the second memory 122 at a time.

On the other hand, the convolutional neural network circuit 140 has two ways to retrieve the parameters from the third memory 123. The first method is that the multiplexer 170 selects the input terminal "1", and the convolutional neural network circuit 140 reads the parameter from the third memory 123 through the multiplexer 170. In the second method, the multiplexer 170 selects the input terminal "0", the memory control unit 130 first moves the parameter from the third memory 123 to the second memory 122, and then the convolutional neural network circuit 140 directly reads the parameter from the second memory 122, at the same time, the memory control unit 130 can move the reference image from the third memory 123 to the second memory 122 through the bus 160, so that the parameter and the reference image can be read simultaneously. In some embodiments, the parameters moved to the second memory 122 correspond to one layer (but not all layers) of the convolutional neural network, which also reduces the space requirement of the second memory 122.

In some embodiments, the infrared sensor 110 senses a gray-scale image in addition to the structured image. The gray-scale image has no projected pattern, that is, the pixels of the gray-scale image only reflect the objects in the scene, and the gray-scale image is transmitted to the second memory 122. The convolutional neural network circuit 140 obtains the parameters, the depth information and the grayscale images from the second memory 122, and executes an object detection procedure or an object recognition procedure, such as a face recognition procedure or a face recognition procedure, according to the grayscale images, the depth information and the parameters.

[ second embodiment ]

FIG. 2 is a circuit diagram illustrating an electronic device according to an embodiment. In the embodiment of fig. 2, the image processing circuit 120 does not have the third memory 123, and thus the reference images and parameters are moved from the first memory 121 to the second memory 122. The convolutional neural network circuit 140 may retrieve the parameters from the second memory. The depth decoder 150 obtains the reference image from the second memory 122, calculates depth information according to the reference image and the structured image, and stores the depth information in the second memory 122.

Compared to the first embodiment of fig. 1, the second embodiment of fig. 2 is better for the image with smaller resolution, so that all the parameters and the entire reference image can be directly stored in the second memory 122. However, the invention is not limited thereto, and the first embodiment and the second embodiment can be applied to images with any resolution. It should be noted that the difference between fig. 1 and fig. 2 is only the third memory 123, and other elements are the same, so that the image processing circuit 120 can be applied to different products without much modification. The first embodiment described above is referred to as a first mode, the second embodiment is referred to as a second mode, and the memory control unit 130 can be switched between the first mode and the second mode. In some embodiments, a detection circuit (not shown) may be disposed in the image processing circuit 120, so that the detection circuit determines whether the third memory 123 exists, if the third memory 123 exists, the memory control unit 130 executes the first mode, otherwise, the second mode is executed. Alternatively, the memory control unit 130 may be switched between the first mode and the second mode according to a physical switch or firmware setting, the physical switch may be disposed on a circuit board or any other position, and the firmware setting may be burned into the memory control unit 130 or other controllers in advance, which is not limited in this disclosure.

Although the present invention has been described with reference to the above embodiments, it should be understood that the invention is not limited thereto, and that various changes and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention.