阅读说明：本技术 一种图像矩阵卷积的实现方法、计算设备及储存介质 (Method for realizing image matrix convolution, computing equipment and storage medium ) 是由 王正阳 张勇 刘明航 于 2021-08-02 设计创作，主要内容包括：本发明公开了一种图像矩阵卷积的实现方法、计算设备及储存介质,方法包括：获取待卷积图像矩阵和卷积核,待卷积图像矩阵根据卷积核的大小将待卷积图像矩阵展开为行矩阵,得到第一展开图像矩阵,将第一展开图像矩阵转换为列矩阵,得到第二展开图像矩阵,将卷积核展开为以第二展开图像矩阵的列数作为列数、以卷积核大小作为行数的矩阵,得到第三展开矩阵,第二展开图像矩阵中每一行数据大小、第三展开矩阵中每一行数据大小均为向量寄存器的大小,将第二展开图像矩阵和第三展开矩阵进行卷积运算,得到图像矩阵的特征矩阵。本发明利用计算设备中CPU的向量寄存器,实现在同一时间内执行多次浮点数据运算,显著提高图像矩阵的卷积计算速度。(The invention discloses a method for realizing image matrix convolution, a computing device and a storage medium, wherein the method comprises the following steps: the method comprises the steps of obtaining an image matrix to be convolved and a convolution kernel, expanding the image matrix to be convolved into a row matrix according to the size of the convolution kernel to obtain a first expanded image matrix, converting the first expanded image matrix into a column matrix to obtain a second expanded image matrix, expanding the convolution kernel into a matrix with the number of columns of the second expanded image matrix as the number of columns and the size of the convolution kernel as the number of rows to obtain a third expanded matrix, wherein the size of each row of data in the second expanded image matrix and the size of each row of data in the third expanded matrix are the size of a vector register, and performing convolution operation on the second expanded image matrix and the third expanded matrix to obtain a feature matrix of the image matrix. The invention uses the vector register of CPU in the computing device to realize the execution of a plurality of times of floating point data operation in the same time, and obviously improves the convolution computing speed of the image matrix.)

1. A method for implementing image matrix convolution, executed in a computing device, the method comprising:

acquiring an image matrix to be convolved and a convolution kernel, wherein each element of the image matrix to be convolved is a pixel value of an image;

expanding the image matrix to be convolved into a row matrix to obtain a first expanded image matrix;

allocating continuous memory addresses to the first expanded image matrix to serve as a first memory space;

saving the first expanded image matrix to the first memory space;

converting the first expanded image matrix into a column matrix to obtain a second expanded image matrix, wherein the size of each row of data in the second expanded image matrix is the size of a vector register of a CPU (Central processing Unit) in the computing equipment;

expanding the convolution kernel into a matrix with the number of columns of the second expanded image matrix as the number of columns and the size of the convolution kernel as the number of rows to obtain a third expanded matrix, wherein the size of each row of data in the third expanded matrix is the size of the vector register;

and performing convolution operation on the second expanded image matrix and the third expanded matrix to obtain a convolution result matrix of the image matrix, wherein the convolution result matrix is a characteristic matrix of the image matrix.

2. The method of claim 1, wherein the step of expanding the image matrix to be convolved into a row matrix to obtain a first expanded image matrix comprises:

and according to the size of the convolution kernel, expanding the image matrix to be convolved into a row matrix through an im2col algorithm to serve as the first expanded image matrix.

3. The method of claim 1, further comprising the step of:

and normalizing the image matrix to be convolved to enable the size of each element in the obtained image matrix to be convolved to be the size of floating point data of the vector register supporting floating point data operation.

4. The method of claim 1, wherein converting the first unfolded image matrix into a column matrix to obtain a second unfolded image matrix comprises:

determining the times of floating point data operation which can be simultaneously executed by the vector register, and taking the times of executing the floating point data operation as a first numerical value;

taking the elements of each column in the adjacent rows of the first expanded image matrix as a group of data, and taking the group of data as a row of elements of the matrix, wherein the number of rows in the adjacent rows of the first expanded image matrix is equal to the first numerical value;

when the remaining number of rows of the first expanded image matrix is less than the first numerical value, the missing elements in the first expanded image matrix are filled up through zero elements until the remaining number of rows is equal to the first numerical value, and the second expanded image matrix is obtained.

5. The method of claim 4, further comprising the step of:

allocating continuous memory addresses to the second expanded image matrix to serve as a second memory space;

and storing the second expanded image matrix into the second memory space.

6. The method according to any one of claims 1 to 5, further comprising the step of:

forcibly converting the size of each element in the convolution kernel into the size of floating point data of which the vector register supports floating point data operation;

wherein the step of unfolding the convolution kernel into a matrix with the number of columns of the second unfolded image matrix as the number of columns and the size of the convolution kernel as the number of rows to obtain a third unfolded matrix comprises:

copying each element in the converted convolution kernel into a plurality of columns of the second unfolded image matrix according to a first preset rule to obtain a group of same data, wherein the first preset rule is a sequential rule from left to right and from top to bottom;

and sequentially using the obtained group of same data as a row of elements of the matrix according to the copying sequence to obtain the third expansion matrix.

7. The method of claim 6, further comprising the step of:

allocating continuous memory addresses to the third expansion matrix as a third memory space;

and saving the third expansion matrix to the third memory space.

8. The method of any one of claims 1 to 7, wherein the step of convolving the second unwrapped image matrix with the third unwrapped matrix to obtain a matrix of convolution results for the image matrix comprises:

determining the size of the convolution result matrix according to the image matrix to be convolved and the convolution kernel;

dividing the second expanded image matrix into a plurality of groups of matrixes according to the size of the third expanded matrix, wherein the number of rows and the number of columns of each group of matrixes are the same as those of the third expanded matrix, and elements of each row in each group of matrixes are not overlapped;

performing convolution operation on each group of matrixes and the third expansion matrix to obtain a convolution result of each element in each group of matrixes;

and writing the convolution result of the last row of elements of each group of matrixes into the convolution result matrix of the image matrix according to the first preset rule.

9. A computing device, comprising:

at least one processor; and

a memory storing program instructions, wherein the program instructions are configured to be adapted to be executed by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 1-8.

10. A readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the method of any of claims 1-8.

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method for implementing image matrix convolution, a computing device, and a storage medium.

Background

In the information age, the requirements for real-time performance and high efficiency of data processing (for example, image data) are increasing, and image matrix convolution is a very important data processing means, so that improving the computational efficiency of image matrix convolution is a major concern at present.

In the prior art, the computational efficiency of the image matrix convolution is improved by multiple algorithms, such as im2col algorithm. The im2col algorithm is understood to be that a two-dimensional image matrix is expanded into rows from left to right and from top to bottom in advance according to the size of a convolution kernel, and new rows are continuously distributed in a memory, which is equivalent to expanding an original matrix into a row matrix. When convolution operation is carried out, the probability of cache miss is greatly reduced and the speed of accessing the memory is obviously improved due to continuous memory distribution. However, in the image matrix convolution process, only the function of software is used, but only a compiler is used to realize the image matrix convolution, so that the calculation speed is limited, and if the calculation efficiency is to be improved, additional equipment needs to be accessed, so that the conventional image matrix convolution is complex to realize and low in efficiency.

For this reason, it is desirable to provide an efficient implementation of the convolution of the image matrix to solve the above-mentioned problems.

Disclosure of Invention

To this end, the present invention provides a method of implementing a convolution of an image matrix in an attempt to solve, or at least alleviate, the problems identified above.

According to one aspect of the invention, there is provided a method for implementing convolution of an image matrix, the method being executed in a computing device and comprising:

acquiring an image matrix to be convolved and a convolution kernel, wherein each element of the image matrix to be convolved is a pixel value of an image;

expanding the image matrix to be convolved into a row matrix to obtain a first expanded image matrix;

converting the first expanded image matrix into a column matrix to obtain a second expanded image matrix, wherein the size of each row of data in the second expanded image matrix is the size of a vector register of a CPU (Central processing Unit) in the computing equipment;

expanding the convolution kernel into a matrix with the column number of the second expanded image matrix as the column number and the size of the convolution kernel as the row number to obtain a third expanded matrix, wherein the size of each row of data in the third expanded matrix is the size of the vector register;

and performing convolution operation on the second expanded image matrix and the third expanded matrix to obtain a convolution result matrix of the image matrix, wherein the convolution result matrix is a characteristic matrix of the image matrix.

Optionally, the step of expanding the image matrix to be convolved into a row matrix to obtain a first expanded image matrix includes:

and according to the size of the convolution kernel, expanding the image matrix to be convolved into a row matrix through an im2col algorithm to be used as a first expanded image matrix.

Optionally, the method further comprises the steps of:

and normalizing the image matrix to be convolved to ensure that the size of each element in the obtained image matrix to be convolved is the size of floating point data of which the vector register supports floating point data operation.

Optionally, the method further comprises the steps of:

allocating continuous memory addresses to the first expanded image matrix as a first memory space;

and storing the first expanded image matrix into a first memory space.

Optionally, the step of converting the first expanded image matrix into a column matrix to obtain a second expanded image matrix includes:

determining the times that the vector register can simultaneously execute floating-point data operation, and taking the times of executing the floating-point data operation as a first numerical value;

taking elements of each column in adjacent rows of the first expanded image matrix as a group of data, and taking the group of data as a row of elements of the matrix, wherein the row number of the adjacent rows of the first expanded image matrix is equal to the first numerical value;

and when the remaining number of rows of the first expanded image matrix is less than the first numerical value, filling missing elements in the first expanded image matrix by zero elements until the remaining number of rows is equal to the first numerical value, and obtaining a second expanded image matrix.

Optionally, the method further comprises the steps of:

distributing continuous memory addresses for the second expanded image matrix to serve as a second memory space;

and storing the second expanded image matrix into a second memory space.

Optionally, the method further comprises the steps of:

forcibly converting the size of each element in the convolution kernel into the size of floating point data of which the vector register supports floating point data operation;

wherein, the step of unfolding the convolution kernel into a matrix with the column number of the second unfolded image matrix as the column number and the convolution kernel size as the row number to obtain a third unfolded matrix comprises:

copying each element in the converted convolution kernel into a plurality of columns of a second expanded image matrix according to a first preset rule to obtain a group of same data, wherein the first preset rule is a sequence rule from left to right and from top to bottom;

and taking the obtained group of same data as a row of elements of the matrix in sequence according to the copying sequence to obtain a third expansion matrix.

Optionally, the method further comprises the steps of:

allocating continuous memory addresses to the third expansion matrix as a third memory space;

and saving the third expansion matrix to a third memory space.

Optionally, the step of performing convolution operation on the second expanded image matrix and the third expanded matrix to obtain a convolution result matrix includes:

determining the size of a convolution result matrix according to the image matrix to be convolved and the convolution kernel;

dividing the second expanded image matrix into a plurality of groups of matrixes according to the size of the third expanded matrix, wherein the number of rows and columns of each group of matrixes is the same as that of the third expanded matrix, and elements of each row in each group of matrixes are not overlapped;

performing convolution operation on each group of matrixes and the third expansion matrix to obtain a convolution result of each element in each group of matrixes;

and writing the convolution result of the last row of elements of each group of matrixes into a convolution result matrix according to a first preset rule.

Optionally, the step of writing the convolution result of the last row of elements of each group of matrices into the convolution result matrix according to a first preset rule includes:

and if the total number of the convolution results of the last row of elements of each group of matrixes is larger than the size of the convolution result matrix, selecting the convolution results with the size of the former convolution kernel from the convolution results of the last row of elements of each group of matrixes, and writing the convolution results into the convolution result matrix according to a first preset rule.

Optionally, the step of performing convolution operation on each group of matrices and the third expansion matrix to obtain a convolution result of each element in each group of matrices includes:

reading a row of elements from any one of the plurality of sets of matrices as a first set of elements;

reading a row of elements equal to the number of rows of the first set of elements from the third expanded matrix as a second set of elements;

and multiplying each element in the first group of elements by an element at a corresponding position in the second group of elements, and summing the product result and the convolution results of the elements in the same column number and the previous row of the elements to obtain the convolution result of each element.

Optionally, the floating point data operation supported by the vector register has a floating point data size of any one of 16 bits, 32 bits, and 64 bits.

Optionally, the CPU of the computing device supports a SIMD instruction set, the CPU of the computing device comprising vector registers.

According to an aspect of the present invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method as described above.

According to an aspect of the present invention, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the method as described above.

According to the technical scheme of the invention, an implementation method of image matrix convolution is provided, the method comprises the steps of firstly obtaining an image matrix to be convolved and a convolution kernel, expanding the image matrix to be convolved into a row matrix according to the size of the convolution kernel to obtain a first expanded image matrix, converting the first expanded image matrix into a column matrix to obtain a second expanded image matrix, expanding each row of data in the second expanded image matrix into a vector register of a CPU (central processing unit) in a computing device, expanding the convolution kernel into a matrix with the column number of the second expanded image matrix as the column number and the convolution kernel as the row number to obtain a third expanded matrix, performing convolution operation on each row of data in the second expanded image matrix and the corresponding row of data in the third expanded image matrix to obtain a convolved image matrix, and the data storage addresses of the second expanded image matrix and the third expanded matrix are consecutive.

According to the method for realizing the matrix convolution, data with the size equal to that of the vector register are respectively read from the second expanded image matrix and the third expanded matrix once, so that the vector register of the CPU in the computing equipment is fully utilized, multiple floating point data operations can be executed at the same time, and the SIMD instruction can be realized.

Secondly, in the convolution operation process, because the data storage addresses of the second expanded image matrix and the third expanded matrix are continuous, the data reading and writing speed from the second expanded image matrix and the third expanded matrix is high, and the convolution operation speed of the image matrix is further improved. In addition, in the process of inverse normalization, because the data storage addresses of the convolution result matrix of the image matrix are continuous, the speed of reading the convolution result matrix of the image matrix is high, and the speed of convolution calculation of the image matrix is further improved.

Drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.

FIG. 1 shows a schematic diagram of a computing device 100, according to one embodiment of the invention;

FIG. 2 shows a flow diagram of a method 200 for implementing image matrix convolution according to one embodiment of the present invention;

FIG. 3 shows a schematic diagram of a normalized image matrix to be convolved according to an embodiment of the invention;

FIG. 4 shows a schematic diagram of a preprocessed convolution kernel according to one embodiment of the present invention;

FIG. 5 shows a schematic diagram of a first unfolded image matrix according to one embodiment of the invention;

FIG. 6 illustrates a first unfolded image matrix formed after filling in 3 rows of elements of size 0, according to one embodiment of the present invention;

FIG. 7 shows a schematic diagram of a second unfolded image matrix according to one embodiment of the invention;

FIG. 8 shows a schematic diagram of a third expansion matrix according to one embodiment of the invention; and

FIG. 9 shows a schematic diagram of a convolution result matrix of an image matrix according to one embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Matrix convolution is a very effective and fast tool to get the primary effect of image processing. In image processing, an image is convolved in a matrix form, that is, an image matrix is convolved, the process of image matrix convolution can be regarded as a "sliding window" process, specifically, a smaller matrix is used as a convolution kernel to slide through the whole convolved matrix, each sliding place needs to perform a matrix inner product operation with the convolution kernel (that is, a small matrix of the convolved matrix corresponding to the sliding window is multiplied and accumulated with the convolution kernel), and the operation result forms a brand new matrix.

In the prior art, the calculation efficiency of the image matrix convolution is improved through algorithms, such as im2col algorithm, FFT algorithm, Winograd algorithm, and the like. The above algorithms are all for accelerating the convolution calculation process, and the calculation process of the matrix convolution is described by taking im2col algorithm as an example. The im2col algorithm can be understood as that a two-dimensional image matrix is expanded into rows from left to right and from top to bottom in advance according to the size of a convolution kernel, new rows are continuously distributed in a memory, which is equivalent to that the original matrix is expanded into a row matrix, and the memory distribution is continuous when convolution operation is performed, so that the cache miss probability is greatly reduced, and the memory access speed is obviously improved. However, the image matrix convolution process only depends on the function of software, but only depends on a compiler to realize the image matrix convolution, so that the calculation speed is limited, and if the calculation efficiency is to be improved, additional equipment needs to be accessed, so that the conventional image matrix convolution is complicated to realize and has low calculation efficiency.

To this end, the invention proposes a method for implementing a convolution of an image matrix, the method being implemented in a computing device. FIG. 1 shows a block diagram of a computing device 100, according to one embodiment of the invention. A block diagram of a computing device 100 As shown in FIG. 1, in a basic configuration 102, the computing device 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 may be used for communication between the processor 104 and the system memory 106.

Depending on the desired configuration, the processor 104 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 104 may include one or more levels of cache, such as a level one cache 110 and a level two cache 112, a processor core 114, and registers 116. The example processor core 114 may include an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a digital signal processing core (DSP core), or any combination thereof. The example memory controller 118 may be used with the processor 104, or in some implementations the memory controller 118 may be an internal part of the processor 104.

Depending on the desired configuration, system memory 106 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include an operating system 120, one or more applications 122, and program data 124. In some embodiments, application 122 may be arranged to operate with program data 124 on an operating system. The program data 124 comprises instructions, and in the computing device 100 according to the invention the program data 124 comprises instructions for performing the method 200 for implementing a convolution of an image matrix.

The computing device 100 also includes a storage device 132, the storage device 132 including removable storage 136 and non-removable storage 138, the removable storage 136 and the non-removable storage 138 each connected to the storage interface bus 134. In the present invention, the data related to each event occurring during the program execution process and the time information indicating the occurrence of each event may be stored in the storage device 132, and the operating system 120 is adapted to manage the storage device 132. The storage device 132 may be a magnetic disk.

Computing device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus/interface controller 130. The example output device 142 includes an image processing unit 148 and an audio processing unit 150. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 158. An example communication device 146 may include a network controller 160, which may be arranged to facilitate communications with one or more other computing devices 162 over a network communication link via one or more communication ports 164.

A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.

Computing device 100 may be implemented as a server, such as a file server, a database server, an application server, a WEB server, etc., or as part of a small-form factor portable (or mobile) electronic device, such as a cellular telephone, a Personal Digital Assistant (PDA), a personal media player device, a wireless WEB-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. Computing device 100 may also be implemented as a personal computer including both desktop and notebook computer configurations. In some embodiments, the computing device 100 is configured to perform an implementation 200 of image matrix convolution according to the present invention.

FIG. 2 shows a flow diagram of a method 200 for implementing image matrix convolution according to an embodiment of the present invention. The method 200 is suitable for execution in a computing device 100, such as the computing device 100 described above. The method of the present invention can be applied to the convolution calculation process of the convolutional neural network in the field of image processing (including image recognition (e.g., AI image recognition), image enhancement, image segmentation, etc.), and can also be integrated into the convolution algorithm of image matrix convolution to increase the speed of convolution calculation.

It should be noted that, the CPU of the computing device in the present invention supports SIMD instruction sets, and the CPU of the computing device includes vector registers, all CPUs satisfying the above conditions are within the scope of the present invention, and the present invention is not limited thereto. For example, the CPU of the computing device may be a loongson 3a4000CPU, the loongson 3a4000CPU supports the MIPSMSA instruction set, and the MIPS MSA instruction set belongs to the SIMD instruction set, that is, the loongson 3a4000CPU supports the SIMD instruction set, the SIMD instruction set is single instruction multiple data, that is, a single CPU instruction operates multiple data at the same time, and the loongson 3a4000CPU includes a vector register, where the size of the vector register may be 128 bits, 256 bits, 512 bits, and the like, which is not limited in this disclosure.

As shown in fig. 2, the method 200 includes steps S201 to S210. The method 200 starts in step S201, and in step S201, an image matrix to be convolved and a preset convolution kernel are obtained. In the present invention, mainly for the convolution operation of the image matrix, the image matrix may be an image matrix in image processing (image processing such as image recognition (e.g. AI image recognition), image enhancement, image segmentation, etc.), and each element of the image matrix to be convolved is a pixel value of an image. The invention does not limit the size of the image matrix to be convolved and the size of the convolution kernel, for example, the size of the image matrix to be convolved is 5X5, and the size of the convolution kernel is 3X 3.

After the image matrix to be convolved is obtained, step S202 is executed to perform normalization processing on the image matrix to be convolved to obtain the normalized image matrix to be convolved, in one embodiment, each element in the image matrix to be convolved is uniformly processed into a uniform size in the normalization process, and the uniform size is the size of floating point data of which the vector register supports floating point data operation. The floating point data size of the floating point data operation supported by the vector register is one of 16 bits, 32 bits and 64 bits, that is, each element in the image matrix to be convolved can be uniformly processed into 16 bits, 32 bits or 64 bits in the normalization process. For example, fig. 3 shows a schematic diagram of a normalized image matrix to be convolved according to an embodiment of the present invention, and the size of the normalized image matrix to be convolved in fig. 3 is 5 × 5.

Then, step S203 is executed to perform preprocessing on the convolution kernel, specifically, to forcibly convert each element size in the convolution kernel into a floating point data size that supports floating point data operation with the vector register, for example, by using a compiler to forcibly convert each element size in the convolution kernel into a floating point data size that supports floating point data operation with the vector register, where the floating point data size is the same as the unified size determined in step S202, if the unified size is 32 bits, each element size in the convolution kernel is forcibly converted into 32 bits in step S203, and if the unified size is 64 bits, each element size in the convolution kernel is forcibly converted into 64 bits in step S203. For example, FIG. 4 shows a schematic diagram of a preprocessed convolution kernel, the preprocessed convolution kernel size of FIG. 4 being 3X3, according to one embodiment of the invention.

After the image matrix to be convolved is normalized and the convolution kernel is preprocessed, step S204 is executed to expand the normalized image matrix to be convolved into a row matrix to obtain a first expanded image matrix. In one embodiment, the normalized image matrix to be convolved is expanded into a row matrix by an im2col algorithm, depending on the size of the convolution kernel. Specifically, the normalized image matrix to be convolved is expanded into rows of a first expanded image matrix by sliding one element to the right once according to the size of a convolution kernel and the sequence from left to right and from top to bottom.

Taking the normalized size of the image matrix to be convolved as 5X5 (as shown in fig. 3), the size of the convolution kernel as 3X3 (as shown in fig. 4), the size of each element in the normalized image matrix to be convolved as 32 bits, and the size of each element in the preprocessed convolution kernel as 32 bits as an example, a process of obtaining the first expanded image matrix will be described. According to the size of the convolution kernel, taking the first row and first column of elements of the normalized image matrix to be convolved as the starting position, selecting the elements of the size of the convolution kernel, such as the elements included in the square frame 3-1 in fig. 3, and expanding the elements in the square frame 3-1 into a row of elements (i.e., 1, 1, 1, 0, 1, 0, 1, 0, 0) from left to right and from top to bottom, so as to obtain the first row of elements of the matrix shown in fig. 5, and fig. 5 shows a schematic diagram of the first expanded image matrix according to an embodiment of the present invention.

Sliding an element to the right, taking the first row and the second column of the normalized image matrix to be convolved as the starting positions, selecting the elements with the size of the convolution kernel, such as the elements included in the square frame 3-2 in fig. 3, and expanding the elements in the square frame 3-2 into a row of elements (i.e. 1, 1, 1, 0, 1, 0, 0, 0, 1) from left to right and from top to bottom, so as to obtain the second row of elements of the matrix shown in fig. 5. Then, sliding an element to the right, taking the third column element in the first row of the normalized image matrix to be convolved as the starting position, selecting the elements with the convolution kernel size, such as the elements included in the square box 3-3 in fig. 3, and expanding the elements in the square box 3-3 into a row of elements (i.e. 1, 1, 1, 0, 1, 1, 0) from left to right in the order from top to bottom, so as to obtain the third row of elements of the matrix shown in fig. 5.

After the first row of elements of the normalized image matrix to be convolved is expanded, one element is slid down from the first row and the first column of elements of the normalized image matrix to be convolved, that is, the second row and the first column of elements of the normalized image matrix to be convolved are expanded into the fourth row of elements (0, 1, 0, 1, 0, 0, 1, 0, 1) of the matrix shown in fig. 5 according to the above process with the second row and the first column of elements of the normalized image matrix to be convolved as the starting positions, such as the elements included in the square boxes 3-4 in fig. 3. And so on, a first expanded image matrix corresponding to the normalized image matrix to be convolved is obtained (as shown in fig. 5).

After the first expanded image matrix is obtained, step S205 is executed to allocate consecutive memory addresses to the first expanded image matrix as a first memory space, and store the first expanded image matrix in the first memory space. The starting address of the consecutive memory addresses is an integer multiple of the byte size (i.e. the ratio of the size of the vector register to 8) corresponding to the vector register. When the elements of the first expansion image matrix are read and written later, the reading and writing speed is high because the memory addresses of the first expansion space are continuous, thereby improving the speed of convolution calculation of the image matrix.

Then, step S206 is executed to convert the first expanded image matrix into a column matrix, and obtain a second expanded image matrix. Specifically, the number of times that the vector register can simultaneously perform floating point data operations is determined, the number of times that the floating point data operations are performed is used as a first numerical value, elements of each column in adjacent rows of the first expanded image matrix are used as a group of data, the group of data is used as a row of elements of the matrix, the number of rows of the adjacent rows of the first expanded image matrix is equal to the first numerical value, if the first numerical value is 4, the elements of each column in the adjacent 4 rows of the first expanded image matrix are used as a group of data, and the group of data is used as a row of elements of the matrix. And when the remaining number of lines of the first expanded image matrix is less than a first numerical value, filling missing elements in the first expanded image matrix to the extent that the remaining number of lines is equal to the first numerical value through zero elements, and if the first numerical value is 4, filling missing elements in the first expanded image matrix to 4 lines through zero elements to obtain a second expanded image matrix.

Since the element in each row of the first numerical value is taken as one row of the second expanded image matrix at a time, it is known that the first numerical value is the number of times that the vector register can simultaneously perform floating-point data operations, and the size of each element of the first expanded image matrix is the same as the size of the floating-point data that the vector register supports floating-point data operations, that is, the size of each row of the element in the second expanded image matrix is the size of the vector register, and the number of columns of the second expanded image matrix is the number of times that the vector register can simultaneously perform floating-point data operations.

A process of obtaining the second expanded image matrix will be described with an example of the first expanded image matrix shown in fig. 5, in which the loongson 3a4000CPU and the unified UOS V201030 are used as operating systems. As is known, the loongson 3a4000CPU includes a 128-bit vector register that supports 32-bit floating point data operations (in this case, each element of the first unfolded image matrix, the preprocessed convolution kernel is 32 bits), because the CPU includes a 128-bit vector register, the vector register supports 32-bit floating point data operations, then the first value is 128/32-4, that is, the elements in the first column in the first group of 4 rows adjacent to the first expanded image matrix shown in fig. 5 are taken as the first row elements of the second expanded image matrix, that is, the elements in the rectangular frame 5-1 in fig. 5 are taken as the first row elements of the second expanded image matrix, so as to obtain the first row elements (1, 1, 1, 0) of the second expanded image matrix shown in fig. 7, and fig. 7 shows a schematic diagram of the second expanded image matrix according to an embodiment of the present invention. That is, data of 128 bits at a time is read as one row element of the second expanded image matrix.

Then sliding an element to the right, taking the element in the second column in the first group of 4 rows adjacent to the first expanded image matrix as the second row element of the second expanded image matrix, that is, taking the element in the rectangular box 5-2 in fig. 5 as the second row element of the second expanded image matrix, to obtain the second row element (1, 1, 1, 1) of the second expanded image matrix as shown in fig. 7, and so on, until all the columns in the first group of 4 rows are taken as one row element of the second expanded image matrix, then taking the 5 th row to the 8 th row in fig. 5 as the 4 th row element of the second group, taking the element in the first column in the second group of 4 rows adjacent to the first expanded image matrix as the 10 th row element of the second expanded image matrix as shown in fig. 5, that is, the element in the rectangular box 5-3 in fig. 5 as the 10 th row element of the second expanded image matrix, the 10 th row element (1, 0, 1, 0) of the second unfolded image matrix shown in fig. 7 is obtained until all columns in the 4 rows of the second group are taken as one row element of the second unfolded image matrix.

Since the matrix in fig. 5 has only one row of elements left unread, not meeting the requirement for each set of 4 rows, in one embodiment, 3 rows of 0 size elements are filled to obtain the matrix shown in fig. 6, and fig. 6 shows the matrix formed after the first expanded image matrix according to one embodiment of the present invention is filled with 3 rows of 0 size elements, wherein the third set of 4 rows corresponding to 3 rows of 0 size elements are filled as the elements in the square box 6-1 in fig. 6. And similarly, the elements in the first column and in the third group of 4 rows adjacent to the first expanded image matrix are used as the 19 th row elements of the second expanded image matrix, so as to obtain the 19 th row elements (0, 0, 0, 0) of the second expanded image matrix shown in fig. 7, until all columns in the third group of 4 rows are used as one row elements of the second expanded image matrix, so as to obtain the complete second expanded image matrix shown in fig. 7.

Among the following are examples of key code for converting a first unfolded image matrix into a column matrix to obtain a second unfolded image matrix:

then, step S207 is executed to allocate consecutive memory addresses to the second expanded image matrix as a second memory space, and store the second expanded image matrix into the second memory space, where a start address of the consecutive memory addresses is an integer multiple of a byte size (i.e., a ratio of the size of the vector register to 8) corresponding to the vector register. And executing step S208 to expand the convolution kernel preprocessed in step S203 into a matrix with the number of columns of the second expanded image matrix as the number of columns and the size of the convolution kernel as the number of rows, so as to obtain a third expanded matrix. Specifically, each element in the converted convolution kernel is copied into a plurality of columns of the second expanded image matrix according to a first preset rule to obtain a group of same data, wherein the first preset rule is a sequence rule from left to right and from top to bottom, and the obtained group of same data are sequentially used as a row of elements of the matrix according to the copying sequence to obtain a third expanded matrix. Taking the convolution kernel shown in fig. 4 as an example, the obtained third expansion matrix is shown in fig. 8, and fig. 8 is a schematic diagram of the third expansion matrix according to an embodiment of the present invention. Because each element in the convolution kernel is the floating point data size of the floating point data operation supported by the vector register, the number of the expanded columns is the same as that of the second expanded image matrix, and the number of the columns of the second expanded image matrix is the number of times that the vector register can simultaneously execute the floating point data operation, the size of each row element in the third expanded matrix is also the size of the vector register.

After the third expanded matrix corresponding to the expanded convolution kernel is obtained, step S209 is executed to allocate continuous memory addresses to the third expanded matrix as a third memory space, and store the third expanded matrix in the third memory space, where similarly, the start address of the continuous memory addresses is an integer multiple of the byte size corresponding to the vector register. An example of critical code that expands the convolution kernel into a third expansion matrix and assigns consecutive memory addresses to it is as follows:

and then, executing step S210, performing convolution operation on the second expanded image matrix and the third expanded matrix to obtain a convolution result matrix of the image matrix, where the convolution result matrix is a feature matrix of the image matrix. In one embodiment, the size of the convolution result matrix is determined according to the image matrix to be convolved and the convolution kernel, the second unwrapped image matrix is divided into a plurality of groups of matrices according to the size of the third unwrapped matrix, wherein the number of rows and columns of each group of matrices is the same as that of the third unwrapped matrix, each row of elements in each group of matrices is not overlapped, each group of matrices and the third unwrapped matrix are subjected to convolution operation to obtain the convolution result of each element in each group of matrices, if the total number of the convolution results of the last row of elements in each group of matrices is greater than the size of the convolution result matrix, a convolution result with the size of the previous convolution kernel is selected from the convolution results of the last row of elements in each group of matrices, and the first preset rule is written into the convolution result matrix according to the first preset rule, and is as described above, and is not repeated here. And if the total number of the convolution results of the last row of elements of each group of matrixes is equal to the size of the convolution result matrix, writing the convolution results of the last row of elements of each group of matrixes into the convolution result matrix according to a first preset rule.

Taking the normalized image matrix to be convolved (5X5) shown in fig. 3 and the preprocessed convolution kernel (3X3) shown in fig. 4 as examples, the size of the convolution result matrix is calculated in the following manner: 5-round (3/2) ═ 3, thus determining the size of the final convolution result matrix to be 3X3, where round means rounded.

In one embodiment, the step of performing convolution operation on each set of matrices and the third expansion matrix to obtain a convolution result of each element in each set of matrices includes: reading a row of elements from any one group of matrixes of the multiple groups of matrixes to serve as a first group of elements, reading a row of elements which are equal to the row number of the first group of elements from the third expansion matrix to serve as a second group of elements, multiplying each element in the first group of elements by an element at a corresponding position in the second group of elements, and summing the product result and the convolution result of the elements which have the same column number and are arranged in the previous row of elements to obtain the convolution result of the image matrix of each element. And repeating the process for each row element of each group of matrixes to finally obtain the convolution result of the image matrixes of all the elements in the second expanded image matrix.

In the above process, since the data storage addresses of the second expanded image matrix and the third expanded matrix are continuous, the speed of reading and writing data from and into the second expanded image matrix and the third expanded matrix is fast, thereby further improving the speed of convolution calculation of the image matrix.

The procedure of the convolution calculation of the image matrix will be described in detail by taking the second expansion image matrix shown in fig. 7 and the third expansion matrix shown in fig. 8 as examples.

Reading the first 9 rows of elements from the second expanded image matrix of fig. 7 as a first set of matrices, such as the elements shown by the square box 7-1 in fig. 7, reading the first row of elements (1, 1, 1, 0) from the first set of matrices as a first set of elements, reading the first row of elements (0, 0, 0, 0) from the third expanded matrix as a first set of elements, multiplying each element in the first set of elements by the corresponding position element in the second set of elements (i.e., 1 × 0, 0 × 0) to obtain the result of 0, 0, respectively, summing the result of multiplication with the convolution result of the same column number and row element as each element in the first row, and the result of convolution of the same column number and row element as each element in the first row, thereby summing the result of convolution with the same column number and the same row number as each element in the second group of elements, The convolution results of the previous row of elements are summed to 0+ 0-0, resulting in the convolution results 0, 0 of the first row of elements of the first set of matrices.

Then, reading a second row element (1, 1, 1, 1) from the first set of matrix, reading a second row element (1, 1, 1, 1) from the third expanded matrix, multiplying each element of the second row element in the first set of elements by an element at a corresponding position of the second row element in the third expanded matrix (i.e., 1, 1, 1), respectively, to obtain results 1, 1, respectively, summing the product result with the convolution result of the previous row element with the same number of columns of each element, the convolution result of each element of the first row element being 0, respectively, summing the product result with each element at a corresponding position in the first row element of the convolution result matrix, to obtain results 1+0, 1+ 1, 1+0, and 1, thereby obtaining results 1, 1, 1, 0, 1, 1, and 1+0 of the second row element of the first set of the first matrix 1. 1 and 1. By analogy, the convolution results of the first set of matrices of the second expanded image matrix are (0, 0, 0, 0), (1, 1, 1, 1), (1, 2, 1, 2), (2, 2, 2, 2), (2, 3, 3, 2), (2, 3, 4, 2), respectively.

Then, the elements of the 10 th row to 18 th row are read from the second expanded image matrix of fig. 7 as the second group matrix, such as the elements shown by the square frame 7-2 in fig. 7, the first row element (1, 0, 1, 0) is read from the second group matrix, the first row element (0, 0, 0, 0) is read from the third expanded matrix, each element in the first row element read from the second group matrix is multiplied by the element at the corresponding position in the first row element read from the third expanded matrix (i.e., 1 × 0, 0 × 0), and the obtained results are 0, 0, 0, respectively. In this case, since the convolution calculation is performed for the first row element, the convolution result of the previous row element is 0, and the multiplication result is summed up with each element at the corresponding position in the previous row element to obtain 0+0 to 0, and 0+0 to 0, thereby obtaining the convolution results 0, and 0 of the first row element in the second group matrix. By analogy, the convolution results of the second group of matrices of the second expanded image matrix are (0, 0, 0, 0), (0, 1, 1, 0), (0, 2, 1, 1), (1, 2, 2, 1), (2, 2, 2, 2).

Then, the elements of the 19 th row to the 27 th row are read from the second expanded image matrix of fig. 7 as a third group matrix, as shown by the square frame 7-3 in fig. 7, the first row element (0, 0, 0, 0) is read from the third group matrix, the first row element (0, 0, 0, 0) is read from the third expanded matrix, each element in the first row read from the third group matrix is multiplied by the element at the corresponding position in the first row element read from the third expanded matrix (i.e., 0 × 0), and the obtained results are 0, 0, 0, respectively. In this case, since the convolution calculation is performed for the first row element, the convolution result of the previous row element is 0, and the multiplication result is summed up with each element at the corresponding position in the first row element, so that 0+0 is 0, and 0+0 is 0, thereby obtaining the convolution results 0, and 0 of the first row element in the third group matrix. By analogy, the convolution results of the third set of matrices of the second expanded image matrix are (0, 0, 0, 0), (1, 0, 0, 0), (2, 0, 0, 0), (3, 0, 0, 0), (4, 0, 0) respectively.

Then, the convolution result (2, 3, 4, 2) of the last row of the first set of matrix, the convolution result (2, 2, 2) of the last row of the second set of matrix, and the convolution result (4, 0, 0, 0) of the last row of the third set of matrix are extracted, it can be seen that the number of the extracted convolution results is 12, since the size of the convolution result matrix of the image matrix to be convolved (5X5) shown in fig. 3 and the convolution kernel (3X3) shown in fig. 4 has been calculated to be 3X3, the size of the convolution result matrix of the second expanded image matrix and the third expanded matrix is 3X3, which is equivalent to that the convolution result matrix includes 9 elements, and the number of the extracted convolution results 12 is greater than the size 9 of the convolution result matrix, therefore, the convolution result of the last row of the first set of matrix is written into the convolution result matrix from left to right and from top to bottom, specifically, the method comprises the following steps: taking the first convolution result (2) in the last row of convolution results of the first set of matrices as the first row and first column elements of the convolution result matrix, the second convolution result (3) as the first row and second column elements of the convolution result matrix, and so on, and taking the fourth convolution result (2) in the last row of convolution results of the first set of matrices as the second row and first column elements of the convolution result matrix.

Then, starting from the second row and the second column of the convolution result matrix, similarly according to the sequence from left to right and from top to bottom, the first convolution result (2) in the last row of the convolution results of the second group of matrices is used as the second row and the second column elements of the convolution result matrix, the second convolution result (2) in the last row of the convolution results of the second group of matrices is used as the second row and the third column elements of the convolution result matrix, and so on, and the fourth convolution result (2) in the last row of the convolution results of the second group of matrices is used as the third row and the second column elements of the convolution result matrix. Finally, starting from the third row and the third column of the convolution result matrix, the first convolution result (4) in the convolution result in the last row of the third group of matrices is used as the third row and the third column of the convolution result matrix, also from left to right and from top to bottom, so that the convolution result matrix shown in fig. 9 is obtained, and fig. 9 is a schematic diagram of the convolution result matrix of the image matrix according to an embodiment of the present invention.

The following is an example of a critical code for performing a convolution operation on the second expanded image matrix and the third expanded matrix to obtain a convolution result:

because the size of each row element in the second expanded image matrix is the size of the vector register, that is, data with the size of the vector register is respectively read from the second expanded image matrix and the third expanded matrix once, which is equivalent to the data size during each convolution operation being the size of the vector register, that is, the vector register of the CPU in the computing device is fully utilized, and multiple floating point data operations can be executed at the same time, therefore, the characteristics of the CPU can be fully utilized, the computing efficiency is maximized, and no additional equipment is needed to be added in the above process, so that the implementation is simple. In addition, in the convolution operation process, because the data storage addresses of the second expanded image matrix and the third expanded matrix are continuous, the data reading and writing speed from the second expanded image matrix and the third expanded matrix is high, and the convolution operation speed of the image matrix is further improved.

And finally, allocating continuous memory space for the convolution result matrix as a fourth memory space. And reading the convolution result matrix from the fourth memory space, performing inverse normalization processing on the convolution result matrix to obtain a convolution result, and writing the convolution result into the fourth memory space. In the above process, since the data storage addresses of the convolution result matrix are continuous, the speed of reading the convolution result matrix is fast in the process of inverse normalization, thereby further improving the speed of convolution calculation of the image matrix.

As can be seen from the above, in the method for implementing convolution of an image matrix according to the present invention, data having a size equal to that of the vector register is respectively read from the second expanded image matrix and the third expanded matrix at a single time, which is equivalent to fully utilizing the vector register of the CPU in the computing device, and a plurality of floating point data operations can be executed at the same time, i.e., the SIDM command can be implemented.

The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.

In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to perform the method of the invention for implementing image matrix convolution according to instructions in said program code stored in the memory.

By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.