阅读说明：本技术 对抗样本检测方法、装置、计算设备及计算机存储介质 (Confrontation sample detection method and device, computing equipment and computer storage medium ) 是由 王艺 黄波 王炜 于 2019-08-14 设计创作，主要内容包括：本申请涉及机器学习技术领域,公开了一种对抗样本检测方法、装置、计算设备及计算机存储介质。该方法包括：获取训练样本及其对应的训练样本标签,训练样本标签包括正常样本和对抗样本；将训练样本输入目标模型,得到训练样本的第一预测得分向量；将训练样本添加N次随机扰动,得到N组对比训练样本；将N组对比训练样本分别输入目标模型,得到每一组对比训练样本的第二预测得分向量；根据第一预测得分向量和每一组对比训练样本的第二预测得分向量构建特征数据；根据特征数据及特征数据对应的训练样本标签,训练分类模型,得到检测器；根据该检测器对输入的测试数据进行检测。本发明实施例可以根据该检测器实现对抗样本的可靠检测。(The application relates to the technical field of machine learning, and discloses a confrontation sample detection method, a device, computing equipment and a computer storage medium.)

1, challenge sample detection method, characterized in that, the method comprises:

acquiring a training sample and a training sample label corresponding to the training sample label, wherein the training sample label comprises a normal sample and a confrontation sample;

inputting the training sample into a target model to obtain th predicted score vector of the training sample;

adding N times of random disturbance to the training sample to obtain an N-pair comparison training sample, wherein N is a natural number greater than 0;

inputting the N groups of comparison training samples into a target model respectively to obtain a second prediction score vector of each groups of comparison training samples;

constructing feature data from the predicted score vector and the second predicted score vector for each pair of comparative training samples;

training a classification model according to the feature data and training sample labels corresponding to the feature data to obtain a detector;

and detecting the input test data according to the detector.

2. The method of claim 1, wherein inputting the training samples into a target model to obtain an th predictive score vector for the training samples comprises:

inputting the training samples into a target model to obtain a confidence coefficient vector corresponding to each training sample;

obtaining the maximum value in the confidence coefficient vector to obtain the prediction score of each training sample;

and taking a vector formed by the predictive scores of each training sample as a predicted score vector of the training sample.

3. The method of claim 1, wherein the adding N random perturbations to the training samples to obtain N pairs of comparative training samples comprises:

generating random disturbance according to a preset distribution function, wherein the preset distribution function is a distribution function which has an average value of 0 and is symmetrically distributed;

and adding the random disturbance to the training sample for N times to obtain N groups of comparison training samples.

4. The method according to claim 3, wherein the predetermined distribution function is a Gaussian distribution function with a mean value of 0.

5. The method of claim 1, wherein said constructing feature data from said predictive score vector and said second predictive score vector for each set of comparative training samples comprises:

calculating a difference vector between the th prediction score vector and the second prediction score vector for each pair of comparative training samples;

and constructing characteristic data according to the difference vectors of the N pairs of comparison training samples.

6. The method of claim 5, wherein said computing a difference vector between said th predictor score vector and said second predictor score vector for each pair of comparative training samples comprises:

calculating a rate of change vector for the second prediction score vector for each pair of comparative training samples relative to the prediction score vector;

and taking the change rate vector as the difference vector.

7. The method of claim 5 or 6, wherein the constructing the feature data from the difference vectors of the N pairs of comparative training samples comprises:

and denoising and reducing the dimension of the difference vector of the N groups of comparison training samples to obtain the characteristic data.

8. The method according to claim 7, wherein the denoising and dimensionality reduction processing is performed on the difference vector of the N sets of comparative training samples to obtain the feature data, and includes:

forming difference matrixes with N columns by using the difference vectors of the N pairs of comparison training samples;

sequencing elements of each rows in the difference matrix from small to large to obtain a sequenced difference matrix;

extracting preset quantiles of each rows in the sorted difference matrix;

and taking the preset quantiles obtained from all the rows as the characteristic data.

9. The method of claim 1, wherein when the number of normal samples and the number of confrontation samples are the same, training a classification model according to the feature data and the training sample labels corresponding to the feature data to obtain a detector comprises:

and training a binary model according to the characteristic data and the training sample label corresponding to the characteristic data to obtain the detector.

10. The method of claim 1, wherein the detecting the input test data according to the detector comprises:

acquiring test data;

inputting the test data into the detector to obtain a detection result;

and when the label corresponding to the detection result is the countermeasure sample, determining that the test data is the countermeasure sample.

11, A challenge sample test device, comprising:

the acquisition module is used for acquiring training samples and corresponding training sample labels thereof, wherein the training sample labels comprise normal samples and confrontation samples;

an input module, configured to input the training sample into a target model, so as to obtain a predicted score vector of the training sample;

the adding module is used for adding N times of random disturbance to the training sample to obtain N pairs of comparison training samples, wherein N is a natural number greater than 0;

the second input module is used for respectively inputting the N groups of comparison training samples into a target model to obtain a second prediction score vector of each groups of comparison training samples;

a construction module for constructing feature data from the th predictive score vector and the second predictive score vector for each pair of comparative training samples;

the training module is used for training a classification model according to the characteristic data and training sample labels corresponding to the characteristic data to obtain a detector;

and the detection module is used for detecting the input test data according to the detector.

kinds of computing devices, characterized by a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete the communication with each other through the communication bus;

the memory is configured to store at least executable instructions that cause the processor to perform operations corresponding to the challenge sample detection method of any of of claims 1-10.

13, computer storage media having stored therein at least executable instructions for causing a processor to perform operations corresponding to the method of training a detector of as claimed in any of of claims 1-10.

Technical Field

The present application relates to the field of machine learning technologies, and in particular, to a method, an apparatus, a computing device, and a computer storage medium for detecting confrontation samples.

Background

The machine learning is used as important data analysis tools and is widely applied to multiple application fields such as biological feature recognition, automobile automatic driving, machine vision and the like, and when great convenience is brought to people, safety problems are also exposed, a countermeasure sample is generated by adding tiny disturbance which is imperceptible to human beings to an original sample, and the machine learning model is easily attacked by the countermeasure sample.

Under the adaptive counterattack, an attacker not only knows all information of a target model but also knows the strategy for coping with the counterattack, so that the attacker can purposefully construct counterattack samples which can successfully attack the target model and bypass the defense, therefore, the two methods can both fail under the adaptive counterattack.

Disclosure of Invention

The embodiment of the application aims to provide countermeasure sample detection methods, devices, computing equipment and computer storage media.

In order to solve the technical problem, an embodiment of the application provides confrontation sample detection methods, which include the steps of obtaining training samples and corresponding training sample labels thereof, inputting the training samples into a target model to obtain th predicted score vectors of the training samples, adding N times of random disturbance to the training samples to obtain N groups of comparison training samples, wherein N is a natural number greater than 0, respectively inputting the N groups of comparison training samples into the target model to obtain second predicted score vectors of each groups of comparison training samples, constructing feature data according to the th predicted score vectors and the second predicted score vectors of each groups of comparison training samples, training a classification model according to the feature data and the training sample labels corresponding to the feature data to obtain a detector, and detecting input test data according to the detector.

In optional modes, inputting the training samples into a target model to obtain the th predicted score vector of the training samples comprises inputting the training samples into the target model to obtain the confidence coefficient vector corresponding to each training sample, obtaining the maximum value in the confidence coefficient vectors to obtain the predicted score of each training sample, and taking the vector formed by the predicted scores of each training sample as the th predicted score vector of the training samples.

In optional modes, adding N times of random disturbance to a training sample to obtain N pairs of comparison training samples comprises generating random disturbance according to a preset distribution function, wherein the preset distribution function is a symmetrically distributed distribution function with a mean value of 0, and adding N times of random disturbance to the training sample to obtain N pairs of comparison training samples.

In alternative, the preset distribution function is a gaussian distribution function with a mean value of 0.

In , constructing feature data from the predicted score vector and the second predicted score vector for each pairs of comparative training samples includes computing a difference vector between the predicted score vector and the second predicted score vector for each pairs of comparative training samples and constructing feature data from the difference vector for each N pairs of comparative training samples.

In , the calculating the difference vector between the predicted score vector and the second predicted score vector for each sets of comparative training samples includes calculating a rate of change vector for the second predicted score vector for each sets of comparative training samples relative to the predicted score vector, and using the rate of change vector as the difference vector.

In optional modes, constructing feature data according to the difference vector of the N-pair of comparison training samples includes denoising and dimensionality reduction processing the difference vector of the N-pair of comparison training samples to obtain the feature data.

In optional modes, denoising and dimensionality reduction are performed on the difference vectors of the N sets of comparison training samples to obtain the feature data, and the method comprises the steps of forming the difference vectors of the N sets of comparison training samples into difference matrixes with N columns, sorting elements of each rows in the difference matrixes from small to large to obtain the sorted difference matrixes, extracting preset quantiles of each rows in the sorted difference matrixes, and taking the preset quantiles obtained by all the rows as the feature data.

In optional modes, when the number of the normal samples is the same as that of the confrontation samples, training a classification model according to the feature data and the training sample labels corresponding to the feature data to obtain a detector, including training a binary classification model according to the feature data and the training sample labels corresponding to the feature data to obtain the detector.

In optional modes, the detecting the input test data according to the detector includes obtaining test data, inputting the test data into the detector to obtain a detection result, and determining that the test data is a challenge sample when a label corresponding to the detection result is the challenge sample.

The embodiment of the application also provides confrontation sample detection devices, which comprise an acquisition module, an adding module, a construction module, a training module and a detection module, wherein the acquisition module is used for acquiring training samples and corresponding training sample labels thereof, the training sample labels comprise normal samples and confrontation samples, the input module is used for inputting the training samples into a target model to obtain th predicted score vectors of the training samples, the adding module is used for adding N times of random disturbance to the training samples to obtain N groups of comparative training samples, N is a natural number larger than 0, the second input module is used for respectively inputting the N groups of comparative training samples into the target model to obtain second predicted score vectors of each group of comparative training samples, the construction module is used for constructing characteristic data according to the th predicted score vectors and the second predicted score vectors of each group of comparative training samples, the training module is used for training classification models according to the characteristic data and the training sample labels corresponding to the characteristic data to obtain detectors, and the detection module is used for detecting input test data according to the detectors.

In optional modes, the input module is further configured to input the training samples into a target model to obtain confidence vectors corresponding to each training sample, obtain a maximum value in the confidence vectors to obtain a predicted score of each training sample, and use a vector formed by the predicted scores of each training sample as a th predicted score vector of the training sample.

In optional modes, the adding module is further configured to generate random disturbance according to a preset distribution function, where the preset distribution function is a symmetrically distributed distribution function with an average value of 0, and add the random disturbance to the training sample N times to obtain N pairs of comparison training samples.

In alternative, the predetermined distribution function is a gaussian distribution function with a mean value of 0.

In alternative, the construction module is further configured to calculate a difference vector between the th predicted score vector and the second predicted score vector for each pairs of comparative training samples, and construct feature data from the difference vectors for the N pairs of comparative training samples.

In , the calculating the difference vector between the predicted score vector and the second predicted score vector per sets of comparative training samples comprises calculating a rate of change vector of the second predicted score vector per sets of comparative training samples relative to the th predicted score vector, and using the rate of change vector as the difference vector.

In optional modes, constructing feature data according to the difference vector of the N-pair of comparison training samples includes denoising and dimensionality reduction processing the difference vector of the N-pair of comparison training samples to obtain the feature data.

In optional modes, denoising and dimensionality reduction are performed on the difference vectors of the N sets of comparison training samples to obtain the feature data, and the method comprises the steps of forming the difference vectors of the N sets of comparison training samples into difference matrixes with N columns, sorting elements of each rows in the difference matrixes from small to large to obtain the sorted difference matrixes, extracting preset quantiles of each rows in the sorted difference matrixes, and taking the preset quantiles obtained by all the rows as the feature data.

In alternative, when the number of normal samples and the number of challenge samples are the same, the training module is further configured to train a binary model according to the feature data and the training sample labels corresponding to the feature data, so as to obtain a detector.

In optional modes, the detection module is further configured to obtain test data, obtain a detection result when the test data is input into the detection, and determine that the test data is an antagonistic sample when a label corresponding to the detection result is the antagonistic sample.

The embodiment of the application also provides kinds of computing equipment, which comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;

the memory is used for storing at least executable instructions which enable the processor to execute the operations corresponding to the confrontation sample detection method.

The embodiment of the application also provides nonvolatile computer readable storage media, which store computer executable instructions for causing a computer to execute operations corresponding to the confrontation sample detection methods.

Embodiments of the present application also provide computer program products comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform operations corresponding to the challenge sample detection methods described above.

The method comprises the steps of adding N times of random disturbance to a training sample to obtain N pairs of comparative training samples, constructing feature data according to a second predicted score vector of the N pairs of comparative training samples and an th predicted score vector of the training sample, training a classification model according to the feature data to obtain a detector, and detecting an antagonistic sample through the detector, wherein the training sample comprises a normal sample and an antagonistic sample, the N pairs of comparative training samples obtained after the N times of random disturbance comprise comparative training samples corresponding to the normal sample and comparative training samples corresponding to the antagonistic sample, respectively obtaining a predicted score vector and a second predicted score vector after the training sample and the comparative training sample are input into a target model, and for the normal sample, the corresponding predicted score vector and the second predicted score vector are slightly different, for the antagonistic sample, the corresponding predicted score vector and the second predicted score vector are greatly different, for the antagonistic sample, the feature data constructed according to the predicted vector and the second predicted score vector can obviously distinguish the normal sample from the antagonistic sample, so that the detector for the antagonistic sample can effectively detect the type of the training sample, and the detection method of the invention can be applied to the detection of the type of the antagonistic sample which is not dependent on the training sample.

Drawings

are illustrated by way of example and not limitation in the figures of the accompanying drawings in which elements having the same reference numeral designations represent like elements and in which the figures are not to scale unless otherwise specified.

FIG. 1 is a flow chart of a challenge sample detection method according to embodiment of the present application;

FIG. 2 is a flow chart of a method for detecting challenge samples according to a second embodiment of the present application;

FIG. 3 is a flow chart of a method for detecting challenge samples according to a third embodiment of the present application;

FIG. 3a is a graph of the ROC curve for correct challenge sample detection in the challenge sample detection methods according to the third embodiment of the present application;

FIG. 4 is a functional block diagram of an challenge sample testing device 40 according to a fourth embodiment of the present application;

fig. 5 is a schematic structural diagram of kinds of computing devices according to a fifth embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in various embodiments of the present application in order to provide a better understanding of the present application. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.

The application scenario of the embodiment of the present invention is countermeasure sample detection of a classification type target model, wherein the classification type target model is an arbitrary classification model in the existing machine learning, different target models are trained to obtain different detectors, but the process of training the detectors can be realized according to the method in the embodiment, the trained detectors are used for detecting input detection samples before the detection samples are input into the target model, and rejecting prediction classification of the countermeasure samples, wherein the countermeasure samples are samples formed by deliberately adding imperceptible noise to the detection samples input in a data set, so as to make the target model misclassify the detection samples with high confidence, for the countermeasure samples, after adding random disturbance, the response change of the target model is large, after adding random disturbance to normal samples, the response change of the target model is small, and according to the response of the target model, the following steps of are performed on each specific embodiment.

The th embodiment of the application relates to confrontation sample detection methods, the specific flow is shown in figure 1, and the embodiment specifically comprises the following steps:

step 101: and acquiring training samples and corresponding training sample labels thereof, wherein the training sample labels comprise normal samples and confrontation samples.

In this step, there are a plurality of training samples, and the content contained in each training samples is related to the target model, for example, the target model is a face recognition model, and the content contained in each training samples is face pictures, the training sample label is a sample type corresponding to the training sample, including a normal sample and a challenge sample, the normal sample is a sample without adding noise, the challenge sample is a sample after adding noise to the normal sample, wherein each training samples correspond to training sample labels, in the specific implementation process, the challenge sample is obtained by adding noise to the normal fgs sample through a challenge sample generation algorithm.

In a specific implementation mode, samples in a training set of a target model generate corresponding confrontation samples through a confrontation sample generation algorithm, the confrontation samples which can successfully attack the target model are collected, training samples with training sample labels as the confrontation samples in the step are obtained, and then a plurality of samples are selected from the training set of the target model to serve as training samples with the training sample labels as normal samples, wherein the number of the normal samples and the confrontation samples in the training samples can be or not , and when the number of the normal samples and the confrontation samples is the same, a two-classification model is used for training a detector, such as a binary model, so that the finally trained detector is not biased towards the normal samples or the confrontation samples, and the accuracy of the detector is improved.

And 102, inputting the training sample into the target model to obtain th predicted score vector of the training sample.

In the step, training samples are input into a target model to obtain confidence coefficient vectors corresponding to each training sample, the maximum value in the confidence coefficient vectors is obtained to obtain the prediction score of each training sample, and a vector formed by the prediction scores of each training sample is used as the th prediction score vector of the training sample.

The dimension of the confidence coefficient vector is related to the number of classes of the target model, for example, the target model is a ten-class classifier, the confidence coefficient vector corresponding to each training sample is ten-dimensional, and each -dimensional represents the probability that the training sample belongs to a class.

It should be noted that the element in the obtained -th predicted score vector corresponds to the element in the training sample , and the corresponding relationship with the training sample label is not changed.

Step 103: and adding N times of random disturbance to the training sample to obtain N groups of comparison training samples, wherein N is a natural number greater than 0.

In this step, the random perturbation may be random constants, and preferably, the random perturbation is generated according to a preset distribution function, where the preset distribution function is a distribution function with a mean value of 0 and is symmetrically distributed, so as to ensure that the random perturbation has randomness also on positive and negative values, and improve unbiased performance of the trained detector.

It should be noted that, when random perturbation is added to a training sample, every times of random perturbation is added to obtain pairs of comparative training samples, N times of random perturbation is added to obtain N pairs of comparative training samples, each pairs of comparative training samples correspond to the training sample , and the sample label is unchanged, for example, when N is 50, 50 times of random perturbation is added to obtain 50 pairs of comparative training samples.

And 104, respectively inputting the N groups of comparison training samples into the target model to obtain a second prediction score vector of each groups of comparison training samples.

In this step, the calculation process of the second prediction score vector may refer to the calculation process of the th prediction score vector in step 102, and will not be described herein again.

In other embodiments, in order to make the predicted score vector and the second predicted score vector have obvious difference, for a target models, when training the target models, the training samples are added to the training samples, and the labels corresponding to the training samples are obtained.

Step 105, constructing feature data according to the th prediction score vector and the second prediction score vector of each pairs of comparative training samples.

In embodiments, the difference vector between the th prediction score vector and the second prediction score vector of each th group of comparison training samples is calculated, the difference vector between the N th prediction score vector and the second prediction score vector is constructed, the number of feature data and the selection of feature data can be artificially defined by those skilled in the art in the course of implementing the embodiment of the present invention, the more the content of the difference vector contained in the feature data is, the better the detection effect of the trained detector is, wherein the difference vector can be obtained by subtracting the th prediction score vector from the second prediction score vector, or by subtracting the th prediction score vector from the second prediction score vector, and performing other operations based on the difference result obtained by dividing the difference vector obtained by th prediction score vector and the second prediction score vector, for example, by calculating the square of the difference, calculating the relative change of the second prediction score vector by 6323 th prediction score vector, and dividing the difference vector obtained by of the prediction score vector obtained by in the embodiment of the training samples, and calculating the change of the difference vector obtained by 637 th prediction score vector calculated by the average of the difference vector calculated average of the difference between the prediction score vector calculated by and the variation of the prediction score vector calculated results obtained by .

It should be noted that, in optional manners, the difference vector of the N sets of comparison training samples is denoised and subjected to dimensionality reduction to obtain the feature data, where the denoising of the difference vector of the N sets of comparison training samples is performed to remove noise interference in the difference vector, and it is ensured that the obtained feature data has a higher degree of distinction between the normal sample and the countermeasure sample.

In embodiments, the denoising and dimension reduction processing is performed on the difference vectors by combining N groups of difference vectors comparing training samples into a difference matrix of N columns, and extracting a plurality of statistical features as feature data for each row of the difference matrix, the statistical features reduce interference of noise interference on the difference vectors, and reduce dimensions of the N groups of difference vectors, so that the obtained feature data have a distinction degree between a normal sample and a countermeasure sample.

Step 106: and training a classification model according to the characteristic data and the training sample labels corresponding to the characteristic data to obtain the detector.

In this step, the feature data is obtained by comparing the th predicted score vector of the training sample with the th predicted score vector of the training sample, in the process of calculating the difference vector, the corresponding relationship between each training sample and the training sample label is not changed, after the feature data is constructed, the feature data also carries the training sample label of the corresponding training sample, and the classification model is trained according to the feature data and the corresponding training sample label to obtain the detector.

Step 107: the input test data is detected according to the detector.

In , the detection result is the confidence corresponding to the two types of samples, namely the normal sample and the confrontation sample, and the type of the sample with high confidence is the type of the test data.

The embodiment of the invention obtains N pairs of comparative training samples by adding N times of random disturbance to the training samples, constructs feature data according to second predicted score vectors of the N pairs of comparative training samples and th predicted score vectors of the training samples, trains a classification model according to the feature data, obtains a detector, and detects the confrontation samples through the detector, wherein the training samples comprise normal samples and the confrontation samples, the obtained N pairs of comparative training samples comprise comparative training samples corresponding to the normal samples and comparative training samples corresponding to the confrontation samples after adding N times of random disturbance, respectively obtains predicted score vectors and second predicted score vectors after inputting the training samples and the comparative training samples into a target model, for the normal samples, the corresponding predicted score vectors and the second predicted score vectors have small difference, for the confrontation samples, the corresponding predicted score vectors and the second predicted vectors have large difference, for the confrontation samples, the feature data constructed according to the predicted score vectors and the second predicted vectors can obviously distinguish the normal samples from the confrontation samples, thereby enabling the detector to effectively detect the confrontation samples and providing a reliable detection method for the classification model of the confrontation samples, which is not dependent on the training samples, and can be used for detecting the detection of the confrontation samples, and can be changed after the detection method of the confrontation samples.

The second embodiment of the present application relates to a detection method for antagonistic samples, the specific flow is shown in fig. 2, and the difference of this embodiment from the embodiment is that the step 105 specifically comprises the following steps:

step 201, a difference vector of the th prediction score vector and the second prediction score vector of each pairs of comparative training samples is calculated.

For the detailed description of this step, reference may be made to the description of step 105 in embodiment , and details are not repeated here.

In step 202, the difference vectors of the N pairs of comparison training samples are formed into difference matrixes of N columns.

In this step, N sets of comparison training samples are respectively subtracted from the training samples to obtain N sets of difference vectors, assuming that the number of the training samples is M, each sets of the N sets of difference vectors include M elements, a difference matrix of rows and N columns is formed for the N sets of difference vectors, each rows of the difference matrix correspond to N differences of training samples, each columns of the difference matrix correspond to difference vectors, that is, sets of difference vectors obtained by respectively subtracting corresponding samples of the comparison training samples and the training samples.

And step 203, sequencing the elements of each rows in the difference matrix from small to large to obtain a sequenced difference matrix.

The elements of each rows in the difference matrix represent the difference between the corresponding comparison training sample obtained after perturbations are added to training samples and the training sample.

And step 204, extracting preset quantiles of every rows in the sorted difference matrix.

The quantiles are the number of elements in the difference vector divided into several equal parts, the common number of elements includes a median, i.e., a dichotomy, a quartzity, a percentile, and the like, before the quantile is extracted, the difference vector needs to be sorted from small to large, the preset quantile is some number of values in every rows in the sorted difference matrix which needs to be extracted, the number of the preset quantiles is equal to of the feature data extracted in every rows in the difference matrix, wherein the embodiment of the invention does not limit the specific number and number of the preset quantiles in every rows, preferably, the number of the preset quantiles extracted in every rows is equal to , so as to ensure the unbiased training of disturbance signals of every 3963 by the detector, for example, if N is 50, i.e., 50 random disturbances are added to the training samples, the number of the difference vectors is 50, the formed difference matrix is 50 columns, if the training samples are M rows, the formed difference matrix is M rows, the difference matrix is extracted from the sorted difference matrices after 50, and the preset quantiles are extracted, and the number of the elements is 8617, respectively, and the preset elements is 8617%, and the number of the preset quantiles extracted.

Step 205: and taking the preset quantiles obtained from all the rows as characteristic data.

And summarizing the preset quantiles obtained by each rows to obtain the characteristic data, and assuming that the number of the preset quantiles extracted by each row is 17, the number of the extracted characteristic data is 17M for an M row difference value matrix consisting of M training samples.

According to the invention, the preset quantiles are extracted from the difference matrix composed of the difference vectors to serve as the feature data, so that the feature data contain more information in the difference vectors, and meanwhile, the selected feature data after sorting remove the larger or smaller data at the two ends of each rows in the difference matrix, so that the detector trained according to the preset quantiles as the feature data has higher robustness.

The third embodiment of the present application relates to the detection method of challenge samples, and in this example, the step 107 further includes the following steps as shown in fig. 3:

step 301: test data is acquired.

It should be noted that after the test data is obtained, the test data is preprocessed and the feature data is extracted, and the feature data is input to the detector to obtain the detection result. Wherein the type of feature data extracted is the same as the type of feature data extracted from the training sample when the detector was trained.

Step 302: and inputting the test data into a detector to obtain a detection result.

The detectors were trained by the methods described above for training the detectors in any of the embodiments.

Step 303: and when the label corresponding to the detection result is the countermeasure sample, determining the test data as the countermeasure sample.

In this step, the label corresponding to the test data is obtained by the detector, where the label includes an th label corresponding to the normal sample and a second label corresponding to the challenge sample, and when the output result of the detector is the th label, the detected sample is determined as the normal sample, and when the output result of the detector is the second label, the detected sample is determined as the challenge sample.

According to the embodiment of the invention, the acquired test data is detected by the detector, so that the sample type corresponding to the test data is obtained, and the reliable detection of the countermeasure sample is realized. Using the detector and the existing detector 1 and detector 2 to respectively detect the confrontation samples of three classifiers with target models of MNIST, CIFAR-10 and ImageNet, wherein the confrontation samples are respectively generated by the attack algorithms BIM, DeepFool and CW, and the success rate of the corresponding confrontation sample detection is shown in table 1:

TABLE 1

The method includes the steps of setting a countermeasure sample as a positive class, setting a normal sample as a negative class, detecting the countermeasure sample generated by the BIM for the detector 1, the detector 2 and the detector provided by the embodiment of the invention, and detecting a correct ROC curve of the countermeasure sample for the classifiers respectively with the target models of MNIST, CIFAR-10 and ImageNet as shown in fig. 3 a.

A fourth embodiment of the present application relates to countermeasure sample detection apparatuses, as shown in FIG. 4, a countermeasure sample detection apparatus 40 includes an obtaining module 401 for obtaining training samples and corresponding training sample labels thereof, where the training sample labels include normal samples and countermeasure samples, an input module 402 for inputting the training samples into a target model to obtain predicted score vectors of the training samples, an adding module 403 for adding N random perturbations to the training samples to obtain N pairs of comparative training samples, where N is a natural number greater than 0, a second input module 404 for inputting the N pairs of comparative training samples into the target model respectively to obtain second predicted score vectors of each pair of comparative training samples, a building module 405 for building feature data according to the predicted score vectors and the second predicted score vectors of each pair of comparative training samples, a training module 406 for training the model based on the feature data and the corresponding training sample labels of the feature data, and obtaining a detector, and a detection module 407 for detecting the input test data according to the detector.

In optional modes, the input module 401 further is configured to input the training samples into the target model to obtain confidence vectors corresponding to each training sample, obtain the maximum value in the confidence vectors to obtain the predicted score of each training sample, and use the vector formed by the predicted scores of each training sample as the th predicted score vector of the training sample.

In optional modes, the adding module 403 further is configured to generate random perturbation according to a preset distribution function, where the preset distribution function is a distribution function with a mean value of 0 and symmetric distribution, and add N times of random perturbation to the training sample to obtain N pairs of comparison training samples.

In alternative, the predetermined distribution function is a gaussian distribution function with a mean value of 0.

In alternative, the construction module 405 further steps to compute a difference vector between the th predicted score vector and the second predicted score vector for each pairs of comparative training samples and construct feature data from the difference vectors for the N pairs of comparative training samples.

In alternative, calculating the difference vector between the predicted score vector and the second predicted score vector for each sets of comparative training samples includes calculating a rate of change vector for the second predicted score vector for each sets of comparative training samples relative to the predicted score vector and treating the rate of change vector as the difference vector.

In optional modes, constructing feature data according to the difference vector of the N-pair of comparison training samples includes denoising and dimensionality reduction processing the difference vector of the N-pair of comparison training samples to obtain the feature data.

In optional modes, denoising and dimensionality reduction are performed on the difference vectors of the N sets of comparison training samples to obtain the feature data, and the method comprises the steps of forming the difference vectors of the N sets of comparison training samples into difference matrixes with N columns, sorting elements of each rows in the difference matrixes from small to large to obtain the sorted difference matrixes, extracting preset quantiles of each rows in the sorted difference matrixes, and taking the preset quantiles obtained by all the rows as the feature data.

In alternative, when the number of normal samples and challenge samples is the same, the training module 406 further is configured to train a binary model according to the feature data and the training sample labels corresponding to the feature data, so as to obtain a detector.

In alternative, the detecting module 407 further is configured to obtain the test data, input the test data to the detector to obtain the detection result, and determine that the test data is the challenge sample when the label corresponding to the detection result is the challenge sample.

It is worth to mention that each of the modules in this embodiment is a logical module, and in practical applications, logical units may be physical units, or portions of physical units, or may be implemented by a combination of multiple physical units.

The embodiment of the invention obtains N pairs of comparative training samples by adding random disturbance to training samples for N times through an adding module 403, constructs characteristic data according to a second predicted score vector of the N pairs of comparative training samples and an th predicted score vector of the training samples through a constructing module 405, trains a classification model according to the characteristic data through a training module 406, obtains a detector, and detects a confrontation sample according to the detector through a detecting module 407, wherein the training samples comprise normal samples and the confrontation samples, the N pairs of comparative training samples obtained after adding the random disturbance comprise comparative training samples corresponding to the normal samples and comparative training samples corresponding to the confrontation samples, after the training samples and the comparative training samples are input into a target model, a predicted score vector and a second predicted score vector are respectively obtained, for the normal samples, the corresponding predicted score vector and the second predicted score vector are small, for the confrontation samples, the predicted score vector and the second predicted vector corresponding to the confrontation samples and the second predicted score vector are large, for the confrontation samples, the characteristic data constructed according to the predicted score vector and the second predicted vector can distinguish the normal samples from the characteristic data, so that the confrontation samples and the characteristic data constructed by the training samples can be effectively used for the detection of the confrontation samples and the detection method, thereby, the invention can provide a detection method for detecting the detection of the confrontation samples which is not dependent on the detection of the detection method of the invention.

Fig. 5 is a schematic structural diagram of computing devices provided in a fifth embodiment of the present application, where as shown in fig. 5, the computing device includes:

, and a memory 602, processors 601 are shown in fig. 5 as an example.

The processor 601 and the memory 602 may be connected by a bus or other means, such as the bus connection in fig. 5.

The memory 602 is used as nonvolatile computer readable storage media and can be used for storing nonvolatile software programs, nonvolatile computer executable programs and modules, such as program instructions/modules (for example, the obtaining module 401, the input module 402, the adding module 403 shown in fig. 4, and the like) corresponding to the data submission method in the embodiment of the present application, the processor 601 executes various functional applications and data processing of the server by running the nonvolatile software programs, instructions and modules stored in the memory 602, namely, countermeasure sample detection methods in the above method embodiment are implemented.

The memory 602 may include a program storage area that may store an operating system, applications needed for at least functions, and a data storage area that may store data created from the use of the data submission device, etc. additionally, the memory 602 may include high speed random access memory, and may also include non-volatile memory, such as at least disk storage devices, flash memory devices, or other non-volatile solid state storage devices in some embodiments, the memory 602 may optionally include memory located remotely from the processor 601, which may be connected to the data submission device via a network.

The or more modules are stored in the memory 602 and, when executed by the or more processors 601, perform the method of training a detector and the method of challenge sample detection in any of the above-described method embodiments, e.g., performing the above-described method steps 101-107 of fig. 1, method steps 201-205 of fig. 2, method steps 301-303 of fig. 3, and implementing the functions of the modules 401-407 of fig. 4.

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details which are not described in detail in this embodiment, reference is made to the methods provided in the embodiments of the present application.

The computing devices of embodiments of the present application exist in a variety of forms, including but not limited to:

(1) mobile communication devices, which are characterized by mobile communication capabilities and are primarily targeted at providing voice and data communications. Such terminals include smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.

(2) The ultra-mobile personal computer device belongs to the category of personal computers, has calculation and processing functions, and also has mobile internet access characteristics like .

(3) Portable entertainment devices such devices may display and play multimedia content. Such devices include audio and video players (e.g., ipods), handheld game consoles, electronic books, as well as smart toys and portable car navigation devices.

(4) The server is similar to a general computer architecture, but has higher requirements on processing capability, stability, reliability, safety, expandability, manageability and the like because of the need of providing highly reliable services.

(5) And other electronic devices with data interaction functions.

The present embodiments provide non-transitory computer-readable storage media storing computer-executable instructions for execution by or more processors, such as performing method steps 101-107 of fig. 1, 201-205 of fig. 2, 301-303 of fig. 3, and implementing the functions of modules 401-407 of fig. 4, described above.

Embodiments of the present application further provide computer program products comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform operations corresponding to the challenge sample detection method embodiment, in particular to perform the above-described method steps 101 to 107 in fig. 1, method steps 201 to 205 in fig. 2, method steps 301 to 303 in fig. 3, and to implement the functions of modules 401 to 407 in fig. 4.

The above-described device embodiments are merely illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, that is, may be located in places, or may be distributed on a plurality of network units.

It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiments may be implemented by instructing the relevant hardware through a computer program, which may be stored in computer-readable storage medium, and when executed, may include the processes in the embodiments of the above methods.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present application, and not for limiting the same; within the context of the present application, the features of the above embodiments or of the different embodiments may also be combined, the steps may be carried out in any order, and there are many other variations of the different aspects of the present application as described above, which are not provided in detail for the sake of brevity; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may be modified or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the present disclosure.