阅读说明：本技术 使用神经网络 (Using neural networks ) 是由 V·门科夫斯基 A·拉赫曼 C·D·F·雷诺 B·康罗伊 D·马夫里厄杜斯 E·布雷施 于 2018-10-16 设计创作，主要内容包括：一种用于识别针对神经网络的神经网络输入的方法、系统和计算机程序产品,该神经网络输入可能已经神经网络不正确地处理了。获得与神经网络输入相关联的(单个层的神经元的子集的)激活值集合。还获得与神经网络输入相关联的神经网络输出。确定第一神经网络输入与第二神经网络输入是否共享相似的激活值集合,但神经网络输出不相似,反之亦然。以这种方式,能够预测神经网络是否已经不正确地处理了第一神经网络输入和第二神经网络输入中的一个。(A method, system, and computer program product for identifying neural network inputs to a neural network that may have been incorrectly processed by the neural network. A set of activation values (of a subset of neurons of a single layer) associated with a neural network input is obtained. A neural network output associated with the neural network input is also obtained. It is determined whether the first neural network input and the second neural network input share a similar set of activation values but the neural network outputs are dissimilar, and vice versa. In this way, it can be predicted whether the neural network has incorrectly processed one of the first neural network input and the second neural network input.)

1. A computer-implemented method for using a neural network formed from one or more layers of neurons, the method comprising:

processing a first neural network input using the neural network to produce a first neural network output;

processing a second neural network input using the neural network to produce a second neural network output;

determining a first similarity indicator indicative of a level of similarity between the first neural network output and the second neural network output;

selecting a set of one or more neurons of the neural network, the set of neurons being a subset of neurons of a single layer of the neural network;

determining an activation value for each selected neuron, thereby producing a first set of activation values, when processing the first neural network input using the neural network;

determining an activation value for each selected neuron, thereby producing a second set of activation values, when processing the second neural network input using the neural network;

determining a second similarity indicator indicative of a level of similarity between the first set of activation values and a corresponding second set of activation values; and is

The determination is made as to which of two cases:

the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or

The first similarity indicator indicating a level of similarity below a third predetermined threshold and the second similarity indicator indicating a level of similarity above a fourth predetermined threshold,

thereby predicting whether the neural network has inaccurately processed one of the first neural network input and the second neural network input.

2. The computer-implemented method of claim 1, wherein:

the step of processing a first neural network input using the neural network comprises: classifying the first neural network input into a class, the class being the first neural network output;

the step of using the neural network to process a second neural network input comprises: classifying the second neural network input into a class, the class being the second neural network output;

the step of determining a first similarity indicator comprises: generating a first similarity indicator identifying whether the class of the first neural network input is the same as the class of the second neural network input; and is

The step of predicting whether one of the first neural network input and the second neural network input has been processed inaccurately comprises: the determination is made as to which of two cases:

the first similarity indicator indicates that the class of the first neural network input is the same as the class of the second neural network input and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or

The first similarity indicator indicating that the class of the first neural network input is different from the class of the second neural network input and the second similarity indicator indicating a level of similarity above a fourth predetermined threshold,

thereby predicting whether the neural network has incorrectly classified one of the first neural network input and the second neural network input.

3. The computer-implemented method of any of claims 1 or 2, wherein the set of neurons comprises at least two neurons of the same layer.

4. The computer-implemented method of claim 3, wherein the step of determining a second similarity indicator comprises calculating a multidimensional distance between the first set of values and the second set of values.

5. The computer-implemented method of claim 5, wherein the second predetermined threshold and the fourth predetermined threshold are predetermined threshold distances.

6. The computer-implemented method of any of claims 1 to 5, wherein selecting a set of one or more neurons comprises: selecting one or more neurons having activation values outside a predetermined range of activation values in the same layer when the neural network is used to process the first neural network input.

7. The computer-implemented method of any of claims 1 to 5, wherein selecting a set of one or more neurons comprises: selecting one or more neurons in the same layer that have activation values outside a predetermined range of activation values when the neural network is used to process the second neural network input.

8. The computer-implemented method of any of claims 1 or 2, wherein:

the step of using the neural network to process a second neural network input comprises: processing a plurality of second neural network inputs using the neural network, thereby generating a plurality of second neural network outputs;

the step of determining a first similarity indicator comprises: determining a first similarity indicator for each of the plurality of second neural network inputs, the first similarity indicator indicating a level of similarity between the first neural network output and the second neural network output, thereby producing a plurality of first similarity indicators;

the step of generating a second set of activation values comprises: determining, for each of the plurality of second neural network inputs, an activation value for each selected neuron, thereby producing a second set of activation values for each second neural network input, thereby producing a plurality of second sets of activation values, when the neural network is used to process the second neural network input;

the step of determining a second similarity indicator comprises: determining, for each of the plurality of second neural network inputs, a second similarity indicator indicating a level of similarity between the first set of activation values and a corresponding second set of activation values, thereby producing a plurality of second similarity indicators; and is

Predicting whether the neural network has inaccurately processed one of the first neural network input and the second neural network input comprises:

determining for each second neural network input which of two cases is: the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or the first similarity indicator indicates a level of similarity below a third predetermined threshold and the second similarity indicator indicates a level of similarity above a fourth predetermined threshold, predicting, for each second neural network input, whether the neural network has inaccurately processed one of the first and second neural network inputs; and is

Identifying all second neural network inputs for which the neural network is predicted to have inaccurately processed one of the first and second neural network inputs.

9. The method of claim 8, wherein the step of determining a second similarity indicator for each second neural network input comprises:

performing multidimensional clustering on the first set of activation values and each of a plurality of second sets of activation values; and is

Generating, for each second neural network input, a second similarity indicator indicating whether the first and second neural network inputs are associated with a set of activation values in the same cluster.

10. The computer-implemented method of any of the preceding claims, further comprising: performing a training process on the neural network in response to predicting that the neural network has inaccurately processed one of the first neural network input and the second neural network input.

11. A computer program product comprising a non-transitory computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any preceding claim.

12. A system (500) for using a neural network formed from one or more layers of neurons, comprising:

a memory (506) comprising instruction data representing an instruction set;

a processor (502) configured to communicate with the memory (506) and execute the set of instructions, wherein the set of instructions, when executed by the processor (502), cause the processor (502) to:

processing a first neural network input using the neural network to produce a first neural network output;

processing a second neural network input using the neural network to produce a second neural network output;

determining a first similarity indicator indicative of a level of similarity between the first neural network output and the second neural network output;

selecting a set of one or more neurons of the neural network, the set of neurons being a subset of neurons of a single layer of the neural network;

determining an activation value for each selected neuron, thereby producing a first set of activation values, when processing the first neural network input using the neural network;

determining an activation value for each selected neuron, thereby producing a second set of activation values, when processing the second neural network input using the neural network;

determining a second similarity indicator indicative of a level of similarity between the first set of activation values and a corresponding second set of activation values; and is

The determination is made as to which of two cases:

the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or

The first similarity indicator indicating a level of similarity below a third predetermined threshold and the second similarity indicator indicating a level of similarity above a fourth predetermined threshold,

thereby predicting whether the neural network has inaccurately processed one of the first neural network input and the second neural network input.

Technical Field

Various embodiments described herein relate to the field of using neural networks. More particularly, but not exclusively, various embodiments relate to systems and methods using a neural network formed of one or more layers.

Background

The user must be confident about the new technology, and if the user employs the new technology, especially in a healthcare scenario, the user may, for example, make a healthcare decision using the new technology. Users tend to trust traditional systems because their operation is very transparent and their behavior is therefore reasonably predictable.

In machine learning, a trade-off between accuracy and intelligibility is typically required. Deep neural networks have proven particularly successful in learning useful semantic representations of data. However, deep neural networks have complex architectures and processes, which make it difficult for users to understand the decision process performed in the neural network. Thus, deep learning based systems tend to lack transparency and predictability due to their complex structure and data-driven design process.

Due to the lack of transparency in the processing of the input data, users may not trust the output of the neural network because they cannot easily verify the validity of the method behind the output or process. Furthermore, the user may not agree on the output, and this may lead to distrust of the system without knowing how the neural network makes its decisions. As a result, this may result in decommissioning or reduced adoption of high performance technologies that may be very accurate in nature.

Accordingly, there is a need for methods and systems that ameliorate the above-mentioned problems.

Neural networks are formed by a plurality of cells (called artificial neurons, or simply "neurons") arranged in layers, the cells being configured to mimic neurons of the brain. Each layer in the neural network model may perform a different transformation on its inputs. A neuron in a neural network is connected to one or more other neurons in the network by one or more connections or edges. Each neuron or edge may have an associated weight (which may be referred to as a bias) that is applied to the output from the neuron or the connection from one neuron to a connected neuron. The weights can be updated with training of the neural network (e.g., using training data). A signal (e.g., data) travels from a first input layer of the neural network to a last output layer of the network, and may traverse one or more intermediate "hidden" layers. The weights are updated with training so that the output of the neural network becomes closer to the expected output.

Disclosure of Invention

According to a first aspect, there is a computer-implemented method for using a neural network formed from one or more layers of neurons, the method comprising: processing a first neural network input using the neural network to produce a first neural network output; processing a second neural network input using the neural network to produce a second neural network output; determining a first similarity indicator indicative of a level of similarity between the first neural network output and the second neural network output. The method further comprises the following steps: selecting a set of one or more neurons of the neural network, the set of neurons being a subset of neurons of a single layer of the neural network; determining an activation value for each selected neuron, thereby producing a first set of activation values, when processing the first neural network input using the neural network; determining an activation value for each selected neuron, thereby producing a second set of activation values, when processing the second neural network input using the neural network; and determining a second similarity indicator indicative of a level of similarity between the first set of activation values and a corresponding second set of activation values. The method further comprises the following steps: the determination is made as to which of two cases: the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or the first similarity indicator indicates a level of similarity below a third predetermined threshold and the second similarity indicator indicates a level of similarity above a fourth predetermined threshold, thereby predicting whether the neural network has inaccurately processed one of the first and second neural network inputs.

Thus, embodiments advantageously identify or predict whether one of the first neural network input or the second neural network input may have been inaccurately processed. In other words, it can be predicted whether the neural network generates inaccurate neural network outputs. An inaccurate neural network output is an output that does not reflect an expected or expected output of the neural network based on the neural network input (e.g., misclassification of the neural network input or an inadequate optimal solution to the problem).

In other words, embodiments enable identification of inconsistencies between the processing of two neural network inputs, enabling an assessment of the accuracy or errors of the neural network to be performed. Thus, it can be predicted whether the neural network is accurately performing the desired task on the neural network input(s), i.e., producing the correct or accurate neural network output.

This can be advantageously used to prompt retraining of the neural network or to draw the user's attention that the neural network output may be inaccurate (e.g., to avoid the user thinking that the neural network is accurate).

The similarity indicator can be any (discrete, categorical or continuous) metric that indicates a level of similarity between two values or sets of values. For example, a binary indicator may indicate whether two values (sets) are considered similar or dissimilar. In another example, the distance between two values (sets) may form a similarity indicator, since the smaller the (usual) distance, the greater the similarity.

Drawings

For a better understanding of the embodiments and to show more clearly how the same may be put into effect, reference will now be made, by way of example only, to the accompanying drawings in which:

fig. 1 and 2 illustrate neural networks for context understanding embodiments;

FIG. 3 illustrates a computer-implemented method according to an embodiment;

FIG. 4 illustrates an example output of a method according to an embodiment as displayed to a user; and is

Fig. 5 illustrates a system according to an embodiment.

The examples rely on the understanding that: using only a subset of the neurons of the neural network to generate a set of activation values for comparison between two neural network inputs enables comparison of the unique and specific information contained in the layers. For example, a first subset of neurons in a layer may be associated with a particular task of a neural network, while a second subset may be associated with a different task. Thus, it is possible to identify whether a particular task will cause inaccuracies in the processing of neural network inputs, thereby enabling a better understanding of the limitations and/or underlying processes of the neural network, or enabling directional retraining of the neural network. In particular, the user may be provided with a more thorough understanding of the decision-making process of the neural network.

In an embodiment, the step of processing a first neural network input using the neural network comprises: classifying the first neural network input into a class, the class being the first neural network output; the step of using the neural network to process a second neural network input comprises: classifying the second neural network input into a class, the class being the second neural network output; the step of determining a first similarity indicator comprises: generating a first similarity indicator identifying whether the class of the first neural network input is the same as the class of the second neural network input; and the step of predicting whether one of the first neural network input and the second neural network input has been processed inaccurately comprises: the determination is made as to which of two cases: the first similarity indicator indicates that the class of the first neural network input is the same as the class of the second neural network input and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or the first similarity indicator indicates that the class of the first neural network input is different from the class of the second neural network input and the second similarity indicator indicates a level of similarity above a fourth predetermined threshold, thereby predicting whether the neural network has incorrectly classified one of the first and second neural network inputs.

In other words, the neural network may be adapted to perform a classification of the neural network inputs and determining whether the classes are similar comprises determining whether they are identical. Thus, determining or predicting whether the neural network has inaccurately processed one of the neural network inputs may include predicting whether the neural network has incorrectly classified one of the neural network inputs.

Preferably, the set of neurons comprises at least two neurons of the same layer.

The step of determining a second similarity indicator may comprise calculating a multidimensional distance between the first set of values and the second set of values. This provides a reliable way to determine a value representing the similarity between two sets of values (since the size of the multidimensional distance represents the similarity). Thus, in such embodiments, the second predetermined threshold and the fourth predetermined threshold can be predetermined threshold distances.

The step of selecting a set of one or more neurons preferably comprises: selecting one or more neurons having activation values outside a predetermined range of activation values in the same layer when the neural network is used to process the first neural network input. Alternatively, the step of selecting a set of one or more neurons may comprise: selecting one or more neurons in the same layer that have activation values outside a predetermined range of activation values when the neural network is used to process the second neural network input.

Thus, an abnormal activation value may be defined as a set of neurons selected for comparison purposes. This increases the likelihood of identifying potentially important neurons of the layer.

Preferably, the step of using the neural network to process a second neural network input comprises: processing a plurality of second neural network inputs using the neural network, thereby generating a plurality of second neural network outputs; the step of determining a first similarity indicator comprises: determining a first similarity indicator for each of the plurality of second neural network inputs, the first similarity indicator indicating a level of similarity between the first neural network output and the second neural network output, thereby producing a plurality of first similarity indicators; the step of generating a second set of activation values comprises: determining, for each of the plurality of second neural network inputs, an activation value for each selected neuron, thereby producing a second set of activation values for each second neural network input, thereby producing a plurality of second sets of activation values, when the neural network is used to process the second neural network input; the step of determining a second similarity indicator comprises: determining, for each of the plurality of second neural network inputs, a second similarity indicator indicating a level of similarity between the first set of activation values and a corresponding second set of activation values, thereby producing a plurality of second similarity indicators; and predicting whether the neural network has inaccurately processed one of the first neural network input and the second neural network input comprises: determining for each second neural network input which of two cases is: the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or the first similarity indicator indicates a level of similarity below a third predetermined threshold and the second similarity indicator indicates a level of similarity above a fourth predetermined threshold, predicting, for each second neural network input, whether the neural network has inaccurately processed one of the first and second neural network inputs; and identifying all second neural network inputs for which it is predicted that the neural network has inaccurately processed one of the first and second neural network inputs.

In other words, more than one second neural network input may be evaluated with respect to the first neural network input to identify a set or group of second neural network inputs having at least one difference (e.g., a dissimilar neural network output or a dissimilar set of activation values) from the first neural network. This enables identification of large sets of neural network inputs that are potentially incorrectly processed.

Of course, it should be understood that this may be the first neural network input that is not accurately processed. Such a determination may be made if a large number or majority of the second neural network input(s) are identified as being potentially processed inaccurately.

The step of determining a second similarity indicator for each second neural network input may comprise: performing multidimensional clustering on the first set of activation values and each of a plurality of second sets of activation values; and generating, for each second neural network input, a second similarity indicator indicating whether the first and second neural network inputs are associated with a set of activation values in the same cluster.

Thus, the second similarity indicator may indicate whether the first set of activation values and the second set of activation values are in the same cluster. To perform clustering, a large total set of second activation values may be required, and thus a large total (e.g., greater than 50 or greater than 100) second neural network input may be required.

In some embodiments, the method further comprises: performing a training process on the neural network in response to predicting that the neural network has inaccurately processed one of the first neural network input and the second neural network input.

Thus, identifying that one of the first neural network input and the second neural network input has been inaccurately processed may prompt that a training process is to be performed. The training process may be based on, for example, user input (e.g., indicating correct neural network output for the first/second neural network input) or on additional training data for the neural network.

In other examples, potentially inaccurately processed neural network inputs (and/or associated neural network outputs) are marked for viewing by a user. This may include, for example, generating an alarm.

According to a second aspect, there is a computer program product comprising a non-transitory computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any of the embodiments described herein.

According to a third aspect, there is a system comprising: a memory including instruction data representing an instruction set; and a processor configured to communicate with the memory and execute the set of instructions. The set of instructions, when executed by the processor, cause the processor to: processing a first neural network input using the neural network to produce a first neural network output; processing a second neural network input using the neural network to produce a second neural network output; determining a first similarity indicator indicative of a level of similarity between the first neural network output and the second neural network output; selecting a set of one or more neurons of the neural network, the set of neurons being a subset of neurons of a single layer of the neural network; determining an activation value for each selected neuron, thereby producing a first set of activation values, when processing the first neural network input using the neural network; determining an activation value for each selected neuron, thereby producing a second set of activation values, when processing the second neural network input using the neural network; determining a second similarity indicator indicative of a level of similarity between the first set of activation values and a corresponding second set of activation values; and determines which of the following two cases: the first similarity indicator indicates a level of similarity above a first predetermined threshold and the second similarity indicator indicates a level of similarity below a second predetermined threshold; or the first similarity indicator indicates a level of similarity below a third predetermined threshold and the second similarity indicator indicates a level of similarity above a fourth predetermined threshold, thereby predicting whether the neural network has inaccurately processed one of the first and second neural network inputs.