阅读说明：本技术 神经网络训练系统、方法和计算机可读存储介质 (Neural metwork training system, method and computer readable storage medium ) 是由 费旭东 于 2018-03-19 设计创作，主要内容包括：一种神经网络训练系统、方法和计算机可读存储介质,属于机器学习算法领域。所述神经网络训练系统(300)包括：第一处理设备(301)和第二处理设备(302),其中,第一处理设备(301)用于获取目标神经网络的权向量和训练集中的N个数据向量,并基于N个数据向量中的每一个数据向量和权向量进行第一运算得到N组输出值,第一运算包括向量点积运算；第二处理设备(302),用于获取根据N组输出值计算得到的至少一个修正值,并根据至少一个修正值对第二处理设备(302)中存储的神经网络的权向量中的向量元素进行修正,得到修正权向量,并将修正权向量发送至第一处理设备(301)。提供的神经网络训练系统能够提高神经网络训练的效率。(A kind of neural metwork training system, method and computer readable storage medium, belong to machine learning algorithm field.The neural metwork training system (300) includes: the first processing equipment (301) and second processing equipment (302), wherein, first processing equipment (301) is used to obtain N number of data vector in the weight vector and training set of target nerve network, and the first operation is carried out based on each of N number of data vector data vector and weight vector and obtains N group output valve, the first operation includes dot product operation；Second processing equipment (302), for obtaining at least one correction value being calculated according to N group output valve, and the vector element in the weight vector of the neural network stored in second processing equipment (302) is modified according at least one correction value, amendment weight vector is obtained, and amendment weight vector is sent to the first processing equipment (301).The neural metwork training system of offer can be improved the efficiency of neural metwork training.)

A neural network training system, comprising a first processing device and a second processing device, the first processing device and the second processing device being different;

the first processing device is configured to:

acquiring a weight vector of a target neural network;

acquiring N data vectors in a training set, wherein the training set comprises a plurality of data vectors, and N is a positive integer greater than or equal to 1;

performing a first operation on the basis of each of the N data vectors and the weight vector to obtain N groups of output values, wherein the first operation comprises a vector dot product operation;

the second processing device is configured to:

obtaining at least one correction value, wherein each correction value in the at least one correction value is used for correcting one vector element in the weight vector, and each correction value is obtained by calculation according to the N groups of output values;

correcting vector elements in the weight vector according to the at least one correction value to obtain a corrected weight vector;

sending the modified weight vector to the first processing device, where the modified weight vector is used to instruct the first processing device to perform the first operation based on N other data vectors in the training set and the modified weight vector, and the N other data vectors are data vectors in the training set except the N data vectors.

The system of claim 1, wherein the first processing device is further configured to calculate the at least one correction value according to the N sets of output values and send the calculated at least one correction value to the second processing device;

the second processing device is specifically configured to receive the at least one correction value sent by the first processing device.

The system of claim 1, wherein the first processing device is further configured to send the N sets of output values to the second processing device;

the second processing device is specifically configured to calculate the at least one correction value according to the N sets of output values.

The system according to claim 1, wherein the second processing device is further configured to perform a first preset process on the correction weight vector to obtain a processed correction weight vector, where a data amount of the processed correction weight vector is smaller than a data amount of the correction weight vector;

the second processing device is specifically configured to send the processed correction weight vector to the first processing device;

the first processing device is further configured to receive the processed correction weight vector sent by the second processing device, and perform a second preset process on the processed correction weight vector to obtain the correction weight vector, where the second preset process is an inverse process of the first preset process.

The system according to claim 4, wherein the first preset processing includes at least one of compression processing and quantization processing.

The system according to claim 1, wherein the first processing device is specifically configured to:

performing neural network forward operation based on a first data vector and the weight vector to obtain a forward output vector of each layer of the target neural network, wherein the neural network forward operation comprises vector dot product operation and nonlinear transformation operation, and the first data vector is any one of the N data vectors;

obtaining an error vector, wherein the error vector is a difference vector between a forward output vector of an output layer of the target neural network and an ideal output vector corresponding to the first data vector in the training set;

performing neural network reverse operation based on the error vector and the weight vector to obtain a reverse output vector of each layer of the target neural network, wherein the neural network reverse operation comprises vector dot product operation;

obtaining the forward output vector of each layer of the target neural network and the backward output vector of each layer of the target neural network as a set of output values corresponding to the first data vector.

The system of claim 6, wherein the first processing device is further configured to calculate the at least one correction value from the N sets of output values via a formula, wherein the formula is:

wherein, Δ w_ijA correction value, X, representing a weight vector element corresponding to a directed arc in the target neural network from the ith node in the p-th layer to the jth node in the p + 1-th layer_pibRepresenting an ith vector of forward output vectors of a p-th layer of the target neural network in a b-th set of output values of the N sets of output valuesElement, E_(p+1)jbA j-th vector element in a reverse output vector of a p + 1-th layer of the target neural network in a b-th group of output values representing the N groups of output values, wherein i, j and p are positive integers greater than or equal to 1;

the first processing device is further configured to send the calculated at least one correction value to the second processing device;

the second processing device is specifically configured to receive the at least one correction value sent by the first processing device.

The system of claim 6, wherein the first processing device is further configured to send the N sets of output values to the second processing device;

the second processing device is specifically configured to calculate the at least one correction value according to the N sets of output values through a formula, where the formula is:

wherein, Δ w_ijA correction value, X, representing a weight vector element corresponding to a directed arc in the target neural network from the ith node in the p-th layer to the jth node in the p + 1-th layer_pibAn i-th vector element in a forward output vector of a p-th layer of the target neural network in a b-th set of output values representing the N sets of output values, E_(p+1)jbAnd j vector elements in a reverse output vector of a p +1 layer of the target neural network in the b group of output values representing the N groups of output values, wherein i, j and p are all positive integers greater than or equal to 1.

The system of claim 1, wherein the first processing device and the second processing device are integrated into one device.

The system according to claim 1, wherein the first processing device is a processing device composed of a plurality of operation nodes in a preset operation network;

the second processing device is a processing device deployed at the cloud or at the edge of a preset communication network.

The system of any of claims 1-10, wherein the first operation further comprises a nonlinear transformation operation.

A neural network training method, the method comprising:

the method comprises the steps that a first processing device obtains a weight vector of a target neural network;

the first processing device acquires N data vectors in a training set, wherein the training set comprises a plurality of data vectors, and N is a positive integer greater than or equal to 1;

the first processing device performs a first operation based on each of the N data vectors and the weight vector to obtain N groups of output values, wherein the first operation comprises a vector dot product operation;

the second processing device obtains at least one correction value, each correction value in the at least one correction value is used for correcting one vector element in the weight vector, each correction value is obtained through calculation according to the N groups of output values, and the second processing device is different from the first processing device;

the second processing device corrects vector elements in the weight vector according to the at least one corrected value to obtain a corrected weight vector;

the second processing device sends the modified weight vector to the first processing device, where the modified weight vector is used to instruct the first processing device to perform the first operation based on N other data vectors in the training set and the modified weight vector, and the N other data vectors are data vectors in the training set except the N data vectors.

The method of claim 12, further comprising:

the first processing equipment calculates the at least one correction value according to the N groups of output values and sends the calculated at least one correction value to the second processing equipment;

the second processing device obtaining at least one correction value, comprising:

and the second processing device receives the at least one correction value sent by the first processing device.

The method of claim 12, further comprising:

the first processing device sends the N groups of output values to the second processing device;

the second processing device obtaining at least one correction value, comprising:

the second processing device calculates the at least one correction value from the N sets of output values.

The method of claim 12, further comprising:

the second processing device performs first preset processing on the correction weight vector to obtain a processed correction weight vector, wherein the data volume of the processed correction weight vector is smaller than that of the correction weight vector;

the second processing device sending the modified weight vector to the first processing device, including:

the second processing device sends the processed correction weight vector to the first processing device;

the method further comprises the following steps:

and the first processing device receives the processed correction weight vector sent by the second processing device, and performs second preset processing on the processed correction weight vector to obtain the correction weight vector, wherein the second preset processing is the inverse processing of the first preset processing.

The method according to claim 15, wherein the first preset process includes at least one of a compression process and a quantization process.

The method of claim 12, wherein the first processing device performs a first operation based on each of the N data vectors and the weight vector to obtain N sets of output values, comprising:

the first processing device performs neural network forward operation on the basis of a first data vector and the weight vector to obtain a forward output vector of each layer of the target neural network, wherein the neural network forward operation comprises vector dot product operation and nonlinear transformation operation, and the first data vector is any one of the N data vectors;

the first processing device obtains an error vector, wherein the error vector is a difference vector of a forward output vector of an output layer of the target neural network and an ideal output vector corresponding to the first data vector in the training set;

the first processing device carries out neural network inverse operation based on the error vector and the weight vector to obtain an inverse output vector of each layer of the target neural network, wherein the neural network inverse operation comprises vector dot product operation;

the first processing device obtains a forward output vector of each layer of the target neural network and a backward output vector of each layer of the target neural network as a set of output values corresponding to the first data vector.

The method of claim 17, further comprising:

the first processing device calculates the at least one correction value according to the N sets of output values by a formula, wherein the formula is:

wherein, Δ w_ijA correction value, X, representing a weight vector element corresponding to a directed arc in the target neural network from the ith node in the p-th layer to the jth node in the p + 1-th layer_pibAn i-th vector element in a forward output vector of a p-th layer of the target neural network in a b-th set of output values representing the N sets of output values, E_(p+1)jbRepresenting an inverse of a p +1 layer of the target neural network in a b-th set of output values of the N sets of output valuesTo the jth vector element in the output vector, i, j and p are all positive integers greater than or equal to 1;

the first processing device sends the at least one correction value obtained through calculation to the second processing device;

the second processing device obtaining at least one correction value, comprising:

and the second processing device receives the at least one correction value sent by the first processing device.

The method of claim 17, further comprising:

the first processing device sends the N groups of output values to the second processing device;

the second processing device obtaining at least one correction value, comprising:

the second processing device calculates the at least one correction value according to the N sets of output values by a formula, wherein the formula is:

wherein, Δ w_ijA correction value, X, representing a weight vector element corresponding to a directed arc in the target neural network from the ith node in the p-th layer to the jth node in the p + 1-th layer_pibAn i-th vector element in a forward output vector of a p-th layer of the target neural network in a b-th set of output values representing the N sets of output values, E_(p+1)jbAnd j vector elements in a reverse output vector of a p +1 layer of the target neural network in the b group of output values representing the N groups of output values, wherein i, j and p are positive integers which are greater than or equal to 1.

The method of claim 12, wherein the first processing device and the second processing device are integrated into one device.

The method according to claim 12, wherein the first processing device is a processing device composed of a plurality of operation nodes in a preset operation network;

the second processing device is a processing device deployed at the cloud or at the edge of a preset communication network.

The method of any of claims 12-21, wherein the first operation further comprises a nonlinear transformation operation.

A computer-readable storage medium, in which a computer program is stored, which, when being executed by a first processing device according to any one of claims 1 to 11, is capable of performing operations performed by the first processing device according to any one of claims 1 to 11 during neural network training; alternatively, the first and second electrodes may be,

the stored computer program, when executed by the second processing device of any of claims 1-11, is capable of performing the operations performed by the second processing device of any of claims 1-11 during neural network training.