阅读说明：本技术 大规模并行神经推理计算元件 (Large-scale parallel neural reasoning computing element ) 是由 泽田润 D·莫德哈 J·V·阿尔苏尔 S·K·艾瑟尔 R·奥普斯瓦密 B·S·塔巴 A· 于 2019-03-11 设计创作，主要内容包括：提供了大规模并行神经推理计算元件。多个乘法器被布置成多个相等大小的组。多个乘法器的每一个都适于并行地对输入激活施加权重以产生输出。多个加法器可操作地耦合到多组乘法器中的一个。多个加法器的每一个适于并行地将乘法器的输出加到其相关组,以产生部分和。多个功能块可操作地耦合到多个加法器中的一个。多个功能块的每一个适于并行地将函数应用于其相关的加法器的部分和以生成输出值。(Massively parallel neuro-inference computing elements are provided. The plurality of multipliers are arranged in a plurality of equally sized groups. Each of the plurality of multipliers is adapted to apply weights to input activations in parallel to produce an output. A plurality of adders is operatively coupled to one of the plurality of sets of multipliers. Each of the plurality of adders is adapted to add the output of the multiplier to its associated set in parallel to produce a partial sum. A plurality of functional blocks are operatively coupled to one of the plurality of adders. Each of the plurality of functional blocks is adapted to apply a function to its associated partial sum of adders in parallel to generate an output value.)

1. A system, comprising:

a plurality of multipliers arranged in a plurality of equally sized groups, each of the plurality of multipliers adapted to apply weights to input activations in parallel to produce an output; and

a plurality of adders, each of the plurality of adders operatively coupled to one of the plurality of sets of multipliers, each of the plurality of adders adapted to add the output of a multiplier to its associated set in parallel to produce a partial sum.

2. The system of claim 1, further comprising:

a plurality of functional blocks, each of the plurality of functional blocks operatively coupled to one of the plurality of adders, each of the plurality of functional blocks adapted to apply a function in parallel to a partial sum of its associated adder to generate an output value.

3. The system of claim 1, adapted to receive a weight matrix and an activation vector.

4. The system of claim 1, wherein each of the plurality of adders comprises an adder tree.

5. The system of claim 4, wherein the adder tree is a binary tree.

6. The system of claim 4, wherein the adder tree comprises a plurality of carry-save adders.

7. The system of claim 3, wherein each activation of an activation vector is broadcast to all multiplier banks.

8. The system of claim 3, further comprising a shrink pipeline operably coupled to each of the plurality of sets of multipliers.

9. The system of claim 1 wherein the plurality of sets of multipliers are pipelined.

10. The system of claim 1, wherein the weights are balanced ternary values.

11. The system of claim 1, wherein each of the plurality of multipliers comprises a multiplexer.

12. The system of claim 3, wherein the weight matrix is compressed, and wherein the system is adapted to decompress the compressed weight matrix.

13. The system of claim 2, wherein the plurality of function blocks are adapted to combine the output value with a subsequently calculated output value.

14. The system of claim 2, further comprising:

a plurality of shifters, each shifter being operatively connected to one of the functional blocks, each shifter being adapted to shift an output value of its respective functional block in parallel, and wherein

A plurality of functional blocks are adapted to combine the shifted values with subsequently calculated output values.

15. The system of claim 2, wherein the function of each function block is an activation function.

16. The system of claim 2, wherein the function of each functional block is programmable.

17. The system of claim 2, wherein the function of each functional block is an addition.

18. The system of claim 2, wherein the function of each functional block is a multiplication.

19. The system of claim 2, wherein the function of each function block is an identification function.

20. The system of claim 2, further comprising a look-up table, the function of each activated function block comprising a look-up from the look-up table.

21. The system of claim 20, wherein the look-up table is programmable.

22. The system of claim 2, wherein the function of each function block is a maximum function.

23. The system of claim 2, wherein the function of each function block is a minimum function.

24. The system of claim 2, further comprising:

a vector register operatively coupled to the functional block, the vector register adapted to store an output of the functional block;

a second plurality of functional blocks, each of the second plurality of functional blocks operatively coupled to the vector register, each of the second plurality of functional blocks adapted to apply a function to the stored output values in parallel.

25. The system of claim 24, wherein the function of each of the second plurality of function blocks is an identification function.

26. The system of claim 24, wherein the function of each of the second plurality of function blocks is an activation function.

27. The system of claim 24, adapted to perform a permutation operation.

28. The system of claim 24, adapted to perform a vector packing operation.

29. The system of claim 24, adapted to perform a max-pooling operation.

30. The system of claim 24, adapted to perform an average pooling operation.

31. The system of claim 24, adapted to perform element-by-element multiplication operations.

32. A method, comprising:

applying, by a plurality of equally sized multiplier groups, a plurality of weights in parallel to a plurality of input activations to generate a plurality of outputs for each group of multipliers;

the outputs from each set of multipliers are added in parallel to generate a partial sum from each set of multipliers.

33. The method of claim 32, further comprising:

a function is applied to the partial sums in parallel to generate a plurality of output values.

34. A system, comprising:

a plurality of multipliers arranged in a plurality of equally sized groups;

a plurality of adders, each of the plurality of adders operably coupled to one of the plurality of sets of multipliers;

a computer readable storage medium having program instructions embodied thereon, the program instructions being executable to perform a method comprising:

applying weights to the input activations in parallel by each of a plurality of multipliers to produce an output;

the outputs of the multipliers are added in parallel to their associated sets by each of a plurality of adders to produce partial sums.

35. The system of claim 34, further comprising:

a plurality of functional blocks, each of the plurality of functional blocks operatively coupled to one of the plurality of adders;

the function is applied in parallel by each of the plurality of functional blocks to its associated partial sum of adders to generate output values.

Background

Embodiments of the present disclosure relate to neural network inference and, more particularly, to massively parallel neuro-inference computation elements.

Disclosure of Invention

In accordance with embodiments of the present disclosure, systems, methods, and computer program products for massively parallel neuro-inference computation are provided. The plurality of multipliers are arranged in a plurality of equally sized groups. Each of the plurality of multipliers is adapted to apply weights to input activations in parallel to produce an output. A plurality of adders is operatively coupled to one of the plurality of sets of multipliers. Each of the plurality of adders is adapted to add the output of the multiplier to its associated set in parallel to produce a partial sum. A plurality of functional blocks are operatively coupled to one of the plurality of adders. Each of the plurality of functional blocks is adapted to apply a function to its associated partial sum of adders in parallel to generate an output value.

According to one aspect, there is provided a system comprising: a plurality of multipliers arranged in a plurality of equally sized groups, each of the plurality of multipliers being adapted to apply weights to input activations in parallel to produce an output; a plurality of adders, each of the plurality of adders operatively coupled to one of the plurality of sets of multipliers, each of the plurality of adders adapted to add the output of a multiplier to its associated set in parallel to produce a partial sum.

According to one aspect, there is provided a method comprising: applying, by a plurality of equally sized multiplier groups, a plurality of weights in parallel to a plurality of input activations to generate a plurality of outputs for each group of multipliers; the outputs from each set of multipliers are added in parallel to generate a partial sum from each set of multipliers.

According to one aspect, there is provided a system comprising: a plurality of multipliers arranged in a plurality of equally sized groups; a plurality of adders, each of the plurality of adders operably coupled to one of the plurality of sets of multipliers; a computer readable storage medium having program instructions embodied thereon, the program instructions being executable to perform a method comprising: applying, by each of a plurality of multipliers, weights to the input activations in parallel to produce outputs; the outputs of the multipliers are added in parallel to their associated sets by each of a plurality of adders to produce partial sums.

Drawings

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

fig. 1 depicts a plurality of neuro-nuclear inference processor architectures according to embodiments of the present disclosure.

Fig. 2 depicts a massively parallel vector matrix multiplier for computing partial sums according to an embodiment of the disclosure.

Fig. 3 depicts a dendritic circuit in accordance with an embodiment of the present disclosure.

FIG. 4 depicts a vector matrix multiplier with an adder array and a vector register in accordance with an embodiment of the present disclosure.

Fig. 5 depicts a vector matrix multiplier with an activation function according to an embodiment of the present disclosure.

FIG. 6 depicts a lookup table for an activation function according to an embodiment of the present disclosure.

7A-B depict input activation via a distribution broadcast in accordance with embodiments of the present disclosure.

FIG. 8 depicts a binary tree for an adder of a dendritic circuit according to an embodiment of the present disclosure.

FIG. 9 depicts a tree of carry-save adders for dendritic circuits in accordance with an embodiment of the present disclosure.

Fig. 10A-B depict ternary multipliers implemented by multiplexers in accordance with embodiments of the present disclosure.

FIG. 11 depicts a shifter for vector matrix multiplication according to an embodiment of the present disclosure.

Fig. 12 depicts a neural circuit in accordance with an embodiment of the present disclosure.

Fig. 13 depicts a shuffle (shuffle) operation using a vector matrix multiplication unit in accordance with an embodiment of the present disclosure.

14A-B illustrate example butterfly network computations according to embodiments of the present disclosure.

Fig. 15 depicts a data packing operation using a vector matrix multiplication unit in accordance with an embodiment of the present disclosure.

Fig. 16 depicts a maximum pooling (pooling) operation using a vector matrix multiplication unit in accordance with an embodiment of the present disclosure.

Fig. 17 depicts an average pooling operation using a vector matrix multiplication unit according to an embodiment of the present disclosure.

Fig. 18 illustrates an element-by-element multiplication operation using a vector matrix multiplication unit according to an embodiment of the present disclosure.

Fig. 19 depicts a programmable operation using a vector matrix multiplication unit to combine a partial sum with a plurality of element-wise operations, in accordance with an embodiment of the present disclosure.

Fig. 20 depicts a method for operating a neural processing core, in accordance with an embodiment of the present disclosure.

FIG. 21 depicts a compute node according to an embodiment of the present disclosure.

Detailed Description

An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of inputs. Two neurons are connected if the output of one neuron is the input of the other neuron. The weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

Neurons compute their outputs by applying a nonlinear activation function to a weighted sum of their inputs, called activations. The weighted sum is an intermediate result obtained by multiplying each input by a corresponding weight and accumulating the products. The partial sum is a weighted sum of the input subsets. The weighted sum of all inputs may be computed in stages by accumulating one or more partial sums.

A neural network is a collection of one or more neurons. Neural networks are typically divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, and typically perform similar functions. The input layer is a layer that receives input from a source external to the neural network. The output layer is a layer that sends output to a target outside the neural network. All other layers are intermediate process layers. A multi-layer neural network is a network of more than one layer. A deep neural network is a multi-layer neural network having many layers.

An array of neural processing units, called neural cores, provides an architecture for implementing various types of neural networks and deep neural networks. Various core architectures are suitable for implementing neural network computations. Typically, such calculations involve calculating neuron activations based on input values, synaptic weights, and neural activation functions. It will be appreciated that neuron activation may be distributed in various ways, including via a network on chip. Various examples herein relate to neuron activation transmitted over a network, however, it should be understood that the present disclosure applies more generally to any neurosynaptic system, including GPUs and application specific ASIC chips, as well as neuron activation independent of the transmission mechanism.

According to various embodiments of the present disclosure, the neural program represents a neural network. The neural program includes information related to a neural network. In certain embodiments, the information includes neuron characteristics and dynamics that determine the response of the electronic neuron to activation of the input neuron. For example, the neuron characteristics and dynamics may include a threshold parameter, a leakage parameter, a delay parameter, or a reset parameter. In some embodiments, the neural procedure information includes synaptic connections of the neurons (e.g., synaptic connections via a matrix of synaptic weights). In some embodiments, the neuro-procedure information includes axonal characteristics (e.g., axonal type). In some embodiments, the neural procedure information includes one or more destinations (e.g., target axons) to which the output activation of the neuron should be delivered.

According to various embodiments, the neural network represents an example of a neural procedure. The neural network may be instantiated in hardware, in simulation, or in both. For example, a neural program may generate one or more instances of a neural network, where the instances may reside on a single core, multiple cores, or multiple chips.

In accordance with embodiments of the present disclosure, a dendritic tree is provided that may be configured in a dedicated orientation to operate in parallel for each synaptic multiply-accumulate circuit, and in some cases. In addition, according to a preferred embodiment, the present disclosure provides dedicated sections and operators and nonlinear activation circuits for each neuron, which can operate in parallel as well. In some embodiments, a multi-function circuit is used for adder or max/min operations. In some embodiments, the system includes a pipelined dendritic tree.

According to embodiments of the present disclosure, a neuron activation function is included, which in some embodiments employs a look-up table to represent the neuron activation function. The advantage of this function is that it can enable an efficient and reconfigurable implementation of discrete models of continuous non-linearity, such as sigmoid, hyperbolic tangent (tanh), ReLU, etc. In some embodiments, the system distributes activation to all parallel neuron circuits in the core using, for example, a broadcast network or a systolic distribution. Further, in accordance with a preferred embodiment, the present disclosure includes circuitry for efficient ternary vector matrix multiplication. In some embodiments, m-term n-bit vector multiplication is performed using m-term ternary multiplier and accumulator circuits. In some embodiments, m-term n-bit vector multiplication is performed directly with m-term ternary compressed vectors (e.g., 8 bits for 5 weights) using an accumulation circuit.

Referring now to fig. 1, a baseline multi-core neuro-inference processor architecture is depicted, in accordance with an embodiment of the present disclosure. In some embodiments, the neural core 100 receives a neural activation vector as an axon input and multiplies the input activation with a matrix representing synaptic weights to produce a neural activation vector as an output. In some embodiments, the input and output activations are provided as 32-element vectors. In such an embodiment, the synaptic weights are represented by a 32 x 32 matrix having 1024 weight elements. In various embodiments, the weights are binary, ternary (-1,0,1), or small integers. In some embodiments, multiple nuclei are tiled on a chip, as shown in fig. 1.

In general, a plurality of neuro-inference cores 100 are connected by a network on chip (NoC) 102. As the labels in fig. 1 show, each neuro-inference core 100 includes an activation memory 104, a weight memory 106, a broadcast bus 108, an input activation register 110, a massively parallel vector matrix multiplier 112, an output activation register 114, a partial sum register 116, and a routing buffer 118.

In an exemplary embodiment, the baseline core operates according to the following protocol:

1. at initialization, the input activation is stored in the core activation memory.

2. Weight values are assigned to each core by a network on chip (NoC)102 and stored in a weight store 106.

3. The active block X is read from the active memory 104 and the weight block W is read from the weight memory 106 and sent to the vector matrix multiplier.

4. The vector matrix multiplier calculates a vector matrix product XW, the output Z being stored in a partial sum register 116.

5. The partial sum may be transmitted to or received from the neighboring core through the routing buffer 118 and then added to the other partial matrix multiplication result X 'W ═ Z'.

6. The activate function is applied to the final partial sum and the result Y is written to the output activate register 114.

7. The output activation Y is written back to the activation memory 104.

Thus, the various core configurations provide distributed logic for computation. In various embodiments, multiple distributed neural cores operate in parallel. This parallelism can increase the speed of neural network processing while reducing the latency between input representation and output computation. For a given problem, each neural core implements a portion of a larger neural network model. Each of the neural cores receives a portion of the entire chip input, as well as a portion of the entire neural network model. This enables modularity of the chip and core, simplifying system design, debugging and testing.

One of ordinary skill in the art will appreciate that the particular core embodiments disclosed herein are for purposes of illustration and not limitation, and that a variety of core embodiments may be incorporated into embodiments of the present disclosure.

In an exemplary embodiment, the following neural network calculations are performed for each neuron j, with the input activation being x_iWeight of w_ijThen the partial sum Z can be calculated in the following manner_j：

An output activation is then calculated from the partial sum z of each output neuron j

y_j＝f(z_j)

Where f (z) is a non-linear activation function.

For f (z), a sigmoid function, hyperbolic tangent tanh (x), or a simpler linear rectifier function may be used.

The neural accelerators, including but not limited to GPUs and ASICs, may perform distributed multiply-add operations over several multiply-accumulate (MAC) units. However, even though the underlying neural network fundamentally allows massively parallel computation, such architectures have inherent sequential properties when performing a series of multiply-add operations, which limits the degree of parallelism. Thus, the present disclosure preferably provides a low-precision neural network in which the hardware is capable of performing substantially more (e.g., orders of magnitude) computations in parallel. In particular, the present disclosure preferably provides a massively parallel computing element comprising a plurality of multiply-add operations for parallel computing neurons.

Referring now to FIG. 2, an exemplary massively parallel computing element for computing neuron activations is disclosed. The exemplary embodiment includes a massively parallel vector matrix multiplication unit for a neuro-inference processor with low-precision neurons. A plurality of adder trees 204 (each capable of summing n outputs from n multipliers) is provided in the calculation unit 202. A single multiplier is depicted at 206, where a single column 208 of the VMM depicts a dendritic tree (and is described in more detail with respect to fig. 3).

The system receives an X, n element input activation row vector (in some embodiments, each element is 4 bits, n 32). In addition, the system receives a W, n × m weight matrix (in some embodiments, each element is 2 bits and n ═ m ═ 32). The calculation unit 202 has n × m multipliers and adders to combine the products. They compute all the multiplications in parallel and accumulate the results in parallel (however, the exact location of the multipliers may vary and is not limited to the particular arrays described). In some embodiments, there are 1024 small multipliers by 32. The result Z ═ XW is the vector matrix product with m elements. In an exemplary embodiment, m is 32, but each of the m elements is 10 bits.

The output elements of the vector matrix multiplication unit of fig. 2 can be computed by n parallel multipliers followed by an adder tree, as shown in fig. 3. In particular, n-element vectors X and W are input_j(jth column of the matrix W) is assigned to n-way parallel multipliers 302, which n-way parallel multipliers 302 may be arranged in a parallel direction. Each multiplier computes a product and product tree 304 sums the products into a single sum. The output of the adder tree is the partial sum 306, which can be used to compute neuron activation. As set forth further below, the partial sum may be provided to an activation function, which in some embodiments may beProgrammed and may be set to a non-linear or linear function, such as an identification function.

In the exemplary embodiment of fig. 3, n multipliers and adder trees are provided. In addition, n-element row vectors X and W to be input_j(jth column of matrix W) to n multipliers; each multiplier calculates a product; the product tree adds the products into a sum; output Z_j＝∑_ix_iw_ijIn some embodiments, the multiplier is a 4 × 2 bit multiplier and the adder tree adds the outputs from n 32 multipliers, the final output being 10 bits.

In connection with the operation schematically shown in fig. 4, in an exemplary embodiment, the vector matrix multiplier result is added to the (partial sum) result of the previous calculation. In such an embodiment, the output result of the vector matrix multiplication unit (referred to as the partial sum vector) may be added to the previously calculated partial sum vector. As shown in FIG. 4, the partial sum register may store a partial sum vector of m elements, and m parallel adders to sum the new partial sum (Z) with the previously computed partial sum (V)^t-1) And (4) adding. Also, the partial sum (V) calculated previously^t-1) The communication channel may be from other cores by way of the core. In the illustrated embodiment, the partial sum vector addition occurs in parallel with m parallel adders 407. In addition, further optimization can be achieved when the output of the vector matrix multiplier is represented by two sets of vectors (sum and carry vectors). Then, m parallel adders can be implemented using carry-save adders (similar to floating-point multiply adders).

As shown in FIG. 5, an array of activation functions is disclosed in which each output element from the vector matrix unit is Z_j＝∑_ix_iw_ij. An activation function f may be applied to obtain neuron activation: y is_j＝f(∑_ix_iw_ij). Exemplary activation functions include binary threshold functions, sigmoid functions, ReLU (rectified linear unit) and linear functions, and any non-linear function that may use a look-up table (LUT). Next, an array of m activation function calculators is implemented, coupled to the outputs of the vector matrix unit. This results in m activation values, each of which can be expressed as: y is_j＝f(∑_ix_iw_ij). To simplify the architectural design, the output of the activation function (Y) may have the same precision as the input activation (X). In some embodiments, the input and output activations are both 4 bits. An activation function calculator may be used to reduce the accuracy of the vector. The local sum vector (Z) typically has a higher precision (10 bits), while the output activation vector (Y) has a lower precision (4 bits). Various linear and non-linear functions can achieve this reduction in accuracy. Also in the exemplary embodiment, m-n-32, so the input and output activation vectors have the same length. Thus, the output activation vector (Y) can be immediately used as the input activation vector (X).

A typical neuron activation uses an n-element vector X, an n × m weight matrix W, computing Y_j＝f(∑_ix_iw_ij) And produces an m-element output vector Y as shown. Z_j＝∑_ix_iw_ijPart of which is computed by a vector matrix multiplier 502, the vector matrix multiplier 502 being composed of an array of multipliers 504 and an array of adder trees 506. The activation function 508 is applied in parallel to the output of the vector matrix multiplication. In some embodiments, n-m-32 or n-m-64; thereby making the input and output size the same.

In connection with the operation schematically illustrated in fig. 6, in an exemplary embodiment, the programmable nonlinear monotonic activation function is implemented by a table lookup. In this exemplary embodiment, the programmable table entries T0, T1, T2 … specify the fraction of a particular output value and PS range:

·z_j<t0 when f is 0

·T0<z_j<T1Then f is 1

·T1<z_j<T2 then f 2 and so on.

In the implementation shown in fig. 6, the programmable lookup table contains T0, T1, T2. The comparator array compares the table entry with the part and the element z_jA comparison is made. The selector logic selects a function value output based on the comparator result. It should be understood that the configuration of fig. 6 may be applied to any size lookup table.

Only the higher order bits or only the lower order bits may be compared to reduce the logic size. This is because the precision of the output activation is limited (e.g., has 4 bits, and the lookup table only requires 15 entries). This allows to implement arbitrary non-linear functions, such as sigmoid functions, without the need for complex arithmetic calculation units.

According to another embodiment of the present disclosure, the vector matrix multiplication is pipelined, and the combinatorial circuit with the activate function circuit is also pipelined. Pipelining may be performed to improve clock frequency and performance of logic circuits. Since the vector matrix multiplication circuit is a relatively large circuit, it is very suitable for pipelining.

Fig. 7A-B depict exemplary embodiments in which the distribution of input activations is accomplished by broadcasting or by shrinking (systolic) pipelines. The vector matrix multiplication unit consists of m dendritic trees, which are medium sized circuits that can optimize how the input is broadcast. Additionally, the delivery input activation may be accomplished by broadcasting. In some embodiments, all dendritic trees begin to compute together and all output results are produced simultaneously. Additionally or alternatively, the transfer input activation may be accomplished by shrinking the pipeline. For example, the activation input may be passed from one latch to another latch each clock cycle and to the dendritic tree at a different clock cycle. Some delay may be incurred in this case, but throughput may be improved by providing a faster clock frequency.

In connection with the operation schematically illustrated in fig. 8, an exemplary embodiment is shown in which the adder tree is configured as a binary tree of adders. According to embodiments of the present disclosure, the adder tree may be implemented in various ways. One exemplary implementation of an adder tree is to use a binary tree.

This is advantageous because current circuit synthesis tools can be used to generate a fully optimized circuit. For low precision adder trees, this approach may provide adequate results.

In connection with the operations schematically illustrated in FIG. 9, in an exemplary embodiment, the adder tree is implemented with carry-save adders. The advantage of a carry-save adder is that the cycle time is short, 3 inputs and produces 2 outputs, the sum of which equals the sum of the inputs, and is often used in multiplier circuits. For example, if s₀And c₀Are two outputs of the CSA:

and vector: s₀＝CSA-S(a₀,a₁,a₂)

Carry vector: c. C₀＝CSA-C(a₀,a₁,a₂)

Then, s₀+c₀＝a₀+a₁+a₂

In the exemplary embodiment shown, the Wallace adder tree is comprised of carry-save adders, and the outputs are two vectors whose sum is equal to the sum of all inputs. In order to obtain the final summation result, the two output vectors have to be added for the last time by a normal adder. There may be an additional CSA plus a previously computed partial sum before the final addition is performed. The advantage of this technique is that the run time is shorter than a simple adder tree.

In some embodiments, an n m weight matrix W is employed that has only { -1,0,1} as possible values. Ternary weight matrices with 4-bit neurons can achieve high precision by appropriate training algorithms. An example 4 x 4 matrix is as follows:

fig. 10A-B depict exemplary embodiments in which the ternary multipliers are implemented by multiplexers. Here, when the weight is a ternary value w_ij∈ { -1,0,1}, a multiplexer may be provided in place of the multiplier_iw_ij。

w_ijOutput x when 1_i

w_ijWhen is-1, output-x_i

If neither control signal is on, the output is a default 0 value.

The optimization can be done as follows, as shown in fig. 10B: use of(

One complement of) instead of-x_i(two complements). Due to the fact thatSo we need to add 1 at other locations. This 1 is very suitable if the CSA tree is used as an adder tree, since the CSA-C output of the CSA is shifted to the left by 1. In addition, w_ijCan be represented by two bits [ b ]₁b₀].(1:[01],0:[00],-1[10]A.) then, b₀And b₁May be used as inputs to the multiplexer.

According to another embodiment of the present disclosure, the weight matrix may be stored in a compressed format and decoded as needed. Here, the ternary weight values may be compressed, since the ternary representation carries less information (less than 2 binary bits). In some embodiments, an 8-bit binary may represent 5 ternary values. For example, the compressed 8-bit value ω is decoded into 5 ternary values w using the following equation_0:4。

w₀＝ω％3-1

w₁＝(ω/3)％3-1

w₂＝(ω/9)％3-1

w₃＝(ω/27)％3-1

w₄＝(ω/81)％3-1

Where a% b is the remainder of a divided by b.

Decoding of the compression weights is performed just before the vector matrix multiplication is performed. This results in less storage and communication overhead (e.g., only 8 bits are needed to represent 5 ternary values instead of 10 bits). It should be understood that this exemplary compression algorithm is provided for illustration, and that other compression algorithms may be employed for weight matrix compression. In addition, the sparse matrix may be compressed in order to obtain a better compression rate.

FIG. 11 depicts an exemplary embodiment in which a shifter is employed to obtain higher precision vector matrix multiplication. In some cases, it is desirable to use higher accuracy calculations in evaluating neural networks. For example, the input activation data may be represented by an 8-bit value. The 8-bit activation value x can be expressed as:

x[7:0]＝x_L+(x_M<<4)

wherein x_LIs the least significant 4 bits of the data,

and x_MIs the most significant 4 bits.

XW can be calculated by iterative vector matrix multiplication because:

W＝X_LW+X_MW<<4

a calculation section and V^t-1＝X_LW and shifted 4 bits to the left, which is then added to the new portion and Z ═ X_MAnd W is above. A shifter array in the path from the partial sum register to the adder array may be used to add the shifted partial sum to the new vector matrix multiplication result.

Thus, there are many vector matrix multipliers that apply multiply-add element by element and compute in turn. According to a preferred embodiment, the present disclosure relates to low precision vector multiplication, for which a parallel architecture can be implemented to compute the entire vector multiplication all at once. Thus, the novel parallel aspect disclosed herein is particularly advantageous over the prior art.

In various embodiments, the circuit shown in FIG. 5 includes m circuits, as shown in FIG. 12. In particular, n element vectors X and W will be input_jThe n multipliers 1202 (j-th column of the matrix W) are assigned (may be arranged in parallel). Each multiplierProducts are computed and product tree 1204 adds the products to a single sum. The output of the adder tree enters an activation function 1206, which produces neuron activations. In some embodiments, the function is programmable and may be set to a non-linear or linear function, e.g., an identification function.

According to embodiments of the present disclosure, the proposed system and method calculates a vector multiplication, which may be followed by a nonlinear or linear activation function. In particular, embodiments of the present disclosure provide for calculation of neural activation. In addition, embodiments of the present disclosure provide for "pooling" operations in Neural Networks (NN), which is indispensable and time consuming. Further, embodiments of the present disclosure provide data shuffling operations in Neural Network (NN) reasoning. These calculations are achieved by extending the dendritic tree circuit described above.

An exemplary data shuffle operation is shown in conjunction with the operation schematically illustrated in FIG. 13. For example, the butterfly permutation is as follows: the input vector X is (0123), and the permuted output result is Y (1032). In some embodiments, the shuffle operation may be implemented by: setting an activation function as an identification function, (id (x) ═ x); the weight matrix W is set as a permutation matrix. For example, only one element in a row and column is 1 and all other elements are 0. An implementation of an exemplary butterfly network is shown in fig. 14A. When performing matrix multiplication using the matrix W shown in fig. 14A, the input vector X is (0,1,2,3) replaced with the output vector Y being (1,0,3, 2). An exemplary butterfly network for level 2 is shown in fig. 14B.

When performing matrix multiplication using the matrix W shown in fig. 14B, the input vector X is changed to (0,1,2,3) and the output vector Y is changed to (2,3,0, 1).

In connection with the operations schematically illustrated in fig. 15, an exemplary embodiment is illustrated in which data from multiple vectors is packed into a single vector, in accordance with embodiments of the present disclosure. This function is particularly useful for tensor down-sampling. For purposes of illustration and not limitation, vector X is applied by alternately selecting values^A(0123) and X^BPacked (4567) as a single vector Y (0246). In this exemplary embodimentThe following vector matrix multiplication unit 1502 is employed, in which the activation function is set to an identification function (id (x) ═ x).

First, perform: z ═ X^AW₀And V ═ Z +0

Then perform: v is V + X^BW₁

The vector register holds the partial sum (V) of the previous calculation¹). The partial sums are fed back to the adder array 1504, which then adds the new partial sum (X) to the adder array 1504^BW₁) And (4) adding. The data stream shown on the right side of fig. 15 includes an array of adders, registers, feedback paths, and an array of activation functions 1506 set to identify functions.

In connection with the operation schematically illustrated in fig. 16, an exemplary embodiment is illustrated in which a maximum pooling operation is performed using a vector matrix multiplication unit, according to an embodiment of the present disclosure. This function is particularly useful in neural networks to scale down large images to smaller sizes. For purposes of illustration and not limitation, for X^A＝(0 3 2 1),X^B＝(3 1 2 2),X^C(4010) and X^DIn the exemplary embodiment shown, the element-by-element maximum is taken to obtain Y ═ 4342.

Set the activation function as the identification function.

Use of identity matrices

Using the feedback path, the current partial sum (Z) is merged with the previously computed partial sum (V) using an element-wise maximum operation.

These operations can be described as follows:

1.V＝max(X^AW_id,0)

2.V＝max(X^BW_id,V)

3.V＝max(X^CW_id,V)

4.V＝max(X^DW_id,V)

5.Y＝id(V^t-1)

in connection with the operation schematically illustrated in fig. 17, an exemplary embodiment according to an embodiment of the present disclosure is illustrated, wherein an average pooling operation is performed using a vector matrix multiplication unit. This function is also particularly useful when scaling down large images to smaller sizes in neural networks. For purposes of illustration and not limitation, for X^A＝(0 3 2 1),X^B＝(3 1 2 2),X^C(4010) and X^DWe use the element averaging method to find Y ═ 2121.

Set the activation function to multiply 1/4

Using the identity matrix W_id

Using the feedback path, only the previously calculated partial sum (V) needs to be added to the new vector matrix product (Z ═ XW).

In the formula, the following operations are performed:

1.V＝X^AW_id+0

2.V＝X^BW_id+V

3.V＝X^CW_id+V

4.V＝X^DW_id+V

5.Y＝(V*1/4)

in connection with the operation schematically illustrated in fig. 18, an exemplary embodiment is shown in which a vector matrix multiplication unit is used to perform an element-by-element multiplication operation. This function is critical in certain neural networks, such as long-short term memory (LSTM). For purposes of illustration and not limitation, for X^A(0321) and X^BMultiplication element by element (3122), and final multiplicationThe fruit is Y ═ (0942). In this exemplary embodiment, a vector matrix multiplication unit may be used as follows:

setting the activation function as an identification function

Using the identity matrix W_id

The feedback is combined by an array of multiplier units to calculate the product of the previously calculated partial sum (V) and the new vector matrix product (Z ═ XW).

In the formula, the following operations are performed:

1.V＝X^AW_id*1

2.V＝X^BW_id*V

3.Y＝id(V)

in connection with the operation schematically illustrated in fig. 19, an exemplary embodiment is illustrated in which the previous partial sum programmable operation is combined with a plurality of element-by-element operations. For purposes of illustration and not limitation, a vector unit may perform element-wise functions such as Add (Add), Max (Max), Multiply (Multiply), Shift (Shift), Bypass (Bypass). Additionally or alternatively, the functions may be provided in conjunction with shifting (on an input), including:

·add(shift(A),B)，max(shift(A),B)，multiply(shift(A),B)，shift(A)

SIMD (Single instruction, multiple data) operations

Performing the same operation in parallel on all inputs of the vector unit

The vector unit is programmable and can perform any operation every cycle.

Accordingly, the vector unit has a plurality of selectable element-by-element operators, and the input path has a selectable shift operator.

Accordingly, embodiments of the present disclosure provide an apparatus that includes a vector matrix multiplication unit with a programmable element-by-element operator and an element-by-element shift operator. In some embodiments, the data logic is pipelined.

Referring to fig. 20, a method for operating a neural processing core is shown, in accordance with an embodiment of the present disclosure. At 2001, multiple equal sized multiplier banks apply multiple weights in parallel to multiple input activations to generate multiple outputs for each set of multipliers. At 2002, the plurality of outputs from each set of multipliers are added in parallel to generate a partial sum from each set of multipliers. In some embodiments, at 2003, functions are applied to the partial sums in parallel to generate a plurality of output values. In some embodiments, the function applied to the partial sums includes vector addition, multiplication, shift, and max operations, which are used to generate a composite partial sum. As shown, in some embodiments, additional partial sums may then be calculated at 2001. At 2004, an activation function is applied to the partial sums in parallel to generate a plurality of output activations.

Referring now to FIG. 21, a schematic diagram of an example of a compute node is shown. The computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. In any event, computing node 10 is capable of being implemented and/or performing any of the functions set forth above.

The computing node 10 has a computer system/server 12 that is operational with numerous other general purpose or special purpose computing system environments or configurations. As is well known, examples of computing systems, environments, and/or configurations that may be suitable for operation with computer system/server 12 include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

The computer system/server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system/server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in FIG. 21, computer system/server 12 in computing node 10 is shown in the form of a general purpose computing device. Components of computer system/server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processors 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, peripheral component interconnect express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

Computer system/server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The computer system/server 12 may also include other removable/non-removable devices, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable, nonvolatile magnetic media (not shown and commonly referred to as "hard disk drives"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may provide a magnetic disk such as a CD-ROM, DVD-ROM, or other optical media. In which case each may be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set of program modules (e.g., at least one) configured to carry out the functions of embodiments of the disclosure.

A program/utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data may be stored in memory 28 by way of example, and not limitation. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networked environment. Program modules 42 generally perform the functions and/or methodologies of the embodiments described herein.

The computer system/server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with computer system/server 12; and/or any device (e.g., network card, modem, etc.) that enables computer system/server 12 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 22. However, the computer system/server 12 may communicate with one or more networks, such as a Local Area Network (LAN), a general Wide Area Network (WAN), and/or a public network (e.g., the Internet) via the network adapter 20. As shown, network adapter 20 communicates with the other components of computer system/server 12 via bus 18. It should be understood that although not shown, other hardware and/or software components may be used in conjunction with the computer system/server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

In various embodiments, one or more neurosynaptic cores (not shown) are coupled to the bus 18. In such embodiments, the neurosynaptic core may receive data from the memory 28 or write data to the memory 28 via the bus 18. Also, as described herein, the neurosynaptic core may interact with other components via the bus 18. In various embodiments, the neurosynaptic core may include one or more local controllers, memories, or clocks, e.g., as described elsewhere herein.

The present disclosure may include systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.