阅读说明：本技术 一种基于Spark内存计算的粮食质量分类方法 (Grain quality classification method based on Spark memory calculation ) 是由 曹杰 王有权 马福民 申冬琴 于 2021-06-30 设计创作，主要内容包括：一种基于Spark内存计算的粮食质量分类方法,包括两个步骤,即粮库质量评价指标体系构建和基于Spark内存计算的粮食质量分类方法,具体内容如下：粮库质量评价指标体系构建,粮库中粮食(包括小麦、早籼稻谷、晚籼稻谷、籼糯稻谷、粳稻谷、粳糯稻谷)进行质量评价体系构建,然后,针对不同种类粮食存在等级严重不均衡以及数据量比较大的问题,提出了基于Spark内存计算的粮食质量分类方法,主要通过合成少数过采样技术解决每个Spark节点中类别的不均衡问题,并行计算则采用并行k最近邻方法；最后,通过不断的迭代实现每个未知粮库中不同仓中粮食的类别分类。(A grain quality classification method based on Spark memory calculation comprises two steps, namely grain depot quality evaluation index system construction and a grain quality classification method based on Spark memory calculation, and specifically comprises the following steps: the method comprises the following steps of constructing a grain depot quality evaluation index system, constructing the quality evaluation system on grains (including wheat, early indica type rice, late indica type rice, indica type glutinous rice, japonica rice and japonica glutinous rice) in a grain depot, then, aiming at the problems of serious unbalance of grades and large data quantity of different types of grains, providing a grain quality classification method based on Spark memory calculation, mainly solving the unbalance problem of categories in each Spark node by synthesizing a few oversampling technologies, and adopting a parallel k nearest neighbor method for parallel calculation; and finally, realizing classification of the grains in different bins in each unknown grain depot through continuous iteration.)

1. A grain quality classification method based on Spark memory calculation is characterized by comprising two steps, namely grain depot quality evaluation index system construction and a grain quality classification method based on Spark memory calculation, and specifically comprises the following steps:

1) construction of quality evaluation index system of grain depot

The grain in the grain depot comprises wheat, early indica type rice, late indica type rice, indica type glutinous rice, japonica rice and japonica glutinous rice, and is characterized in that a quality evaluation system is constructed, and the quality evaluation system comprises the following specific steps:

(1) quality index of grain

Mainly aiming at each grain bin in the grain depot: 1) volume weight; 2) imperfect grain proportion; 3) the proportion of the total content of impurities; 4) mineral content ratio 5) moisture ratio; 6) whether the color is normal or not; 7) whether the smell is normal or not is collected and uploaded to a grain information service cloud platform, and the grain quality is mainly classified into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

(2) Quality index

Aiming at each grain bin in the grain depot: 1) the roughness is generated; 2) the rice-conditioning rate; 3) percentage of impurity content; 4) percentage of moisture content; 5) the percentage content of the yellow rice; 6) the percentage of the content of the brown rice outside the grains; 7) percent of intermixing rate; 8) whether the color is normal or not; 9) whether the smell is normal or not is collected and uploaded to a grain information service cloud platform, and the quality is divided into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

2) Grain quality classification method based on Spark memory calculation

Respectively constructing Spark classifiers by the input wheat or rice, and storing results between the Map/Reduce steps in a main memory by Spark; spark can directly multiplex the intermediate result in the memory without repeatedly reading HDF; the Spark creates a Spark context object from the driving node, then the Spark context requests resources from the cluster manager and acquires an actuator on the cluster node, and the actuator process manages task calculation on one or more CPU threads; abstracting data with an elastic distributed dataset RDD data structure for parallel computation; the RDD stores the dataset as partitions, which are split across one or more cluster nodes; this abstraction allows users to perform the same operations regardless of how the data is distributed; map, Filter and Reduce functions run directly on RDD, and SQL table operations are implemented by further abstract data frames (dataframes) and data structures of datasets.

2. The Spark memory computing-based grain quality classification method as claimed in claim 2, wherein the grains comprise wheat, early indica rice, late indica rice, japonica rice and japonica rice.

3. The Spark memory computing-based grain quality classification method according to claim 1, wherein input data are stored on an HDFS (Hadoop distributed file system) of a grain information service cloud platform in a file Block (Block) form, and the data are loaded from the HDFS to an RDD (remote data device) to realize high-throughput distributed computing;

and taking the data set with the classification mark as an initialization category in the classification result. And then selecting the data from the RDDs by a K nearest neighbor method as distribution data, and broadcasting and distributing the data to each RDD data set.

4. The grain quality classification method based on Spark memory calculation as claimed in claim 1, wherein the parallel k nearest neighbors are executed first, and k is set to 5. The parallel k-nearest neighbor algorithm performs k-nearest neighbors on all RDDs separately, while excluding the query point itself from the returned neighbors. Each instance will give a class label based on the number of adjacent instances belonging to its class, and then a small number of types of labels generated for each instance are stored as data frames, which can also be stored in the RDD for future reuse.

5. The Spark memory calculation-based grain quality classification method of claim 4, wherein 6 grades of each grain have class imbalance, and the manual minority over-sampling method is used to supplement the minority samples, that is, each time a new sample is created, 5 samples of the current class are randomly selected, and then each feature is averaged, so as to generate a new composite sample, in such a way that the number of samples of the class is substantially balanced on each RDD.

6. The grain quality classification method based on Spark memory calculation as claimed in claim 4, wherein Spark algorithm is used to realize classification result summarization, and MapReduce idea is generally adopted, during each iteration, Map classes are used to execute k nearest neighbors in each RDD in a parallelization manner, then class Reduce operation is used to collect the best classes of each k nearest neighbor class so as to obtain more prediction classes of unlabeled data samples, all points with the same class are summarized to form a new RDD, finally the unlabeled samples with higher class confidence are labeled as prediction classes, and finally all samples with labels are broadcasted to each node again for iteration until a target function converges or an iteration end condition is satisfied.

Technical Field

The invention relates to the field of grain supervision of a grain big data cloud platform, in particular to a grain quality classification method based on Spark memory calculation.

Background

At present, quality level detection of grains in grain depots in China mostly comes from sampling detection, namely, the quality of different grains is judged based on grains with different properties, and judgment standards are different, so that the grains in the grain depots in various regions are difficult to be judged by a unified standard. Therefore, the manager of the grain depot can easily deceive the grain condition monitoring department by the ways of grain aging treatment, grain rotation, poor grain change, good grain change and the like. Therefore, how to classify the quality of the grains in the grain depot by the large grain data platform and various grain quality indexes of sampling inspection is an important component of the current national grain situation supervision.

The existing national and local grain detection methods are mostly defined according to the threshold values of different grain quality indexes, for example, the wheat quality standard judgment is usually carried out according to the volume weight, the proportion of imperfect grains, the proportion of total impurity content, the proportion of mineral content, the proportion of water, the color, the smell and the like, then different threshold division is adopted according to the grain with different grades, but the grain of each grain depot reaches a certain grade on certain quality standards, but does not reach the grade or exceed the grade on certain quality indexes, so that the threshold division method consumes a large amount of manpower and material resources, and simultaneously is difficult to accurately and comprehensively master the actual grain quality information of each grain depot in real time.

Starting from the quality of rice and wheat contained in each grain depot, the grain quality classification method based on Spark memory calculation is researched through the quality indexes of each grain collected by sampling. Specifically, grain condition index data are acquired by investigating and researching a strategic grain information service cloud platform in Jiangsu province, different grading standards are adopted for different grains, and a certain marked grain quality classification selection range is defined manually. At present, a small amount of grain quality classification methods exist, but a large amount of unmarked data and a small amount of artificially marked grain condition grade data generally exist in real grain condition big data. The quantity of the grains in each grade is usually greatly different, for example, the grains in the high grade and the grains in the low grade are present in a small proportion, most of the grains are grains in a middle proportion, namely, the quality classification of the grains in each grade has serious class imbalance. In addition, because the grain situation information of the grain depot is influenced by factors such as the year, the season and the like, the continuously-increased partitions of scale, complexity and the like and the imbalance of modern data sets need to be processed, and great troubles are brought to the quality classification of the grains in the grain depot.

Due to the fact that the data volume is too large, grain quality classification cannot be achieved by using a standard computer, and a high-performance computing platform is started, such as a GPU and a cluster. Therefore, the invention provides a grain quality classification method based on Spark memory calculation, which analyzes and understands the characteristics of multi-class unbalanced data sets by analyzing instance-class difficulty existing in each class. The method comprises the steps of firstly constructing an index construction work for realizing grain quality classification through a grain quality evaluation index system, and then solving the class imbalance problem and classifier learning construction work and the like through a large data learning method of multi-class imbalance.

The existing Spark memory model is as follows

1 in-heap and out-of-heap memory planning

Memory management of Spark Executor (executive) is built on memory management of JVM, Spark allocates space (onheat + Off-heat) of JVM in more detail to fully utilize memory. Meanwhile, an Off-Heap memory mode is introduced into Spark, so that a space can be directly opened in a system memory of a working node, and the use of the memory is further optimized (the memory is allocated from os by using malloc of c-style outside the jump independent of JVM hosting.

The Task running in the Executor can use both JVM (OnHeap + Off-heap) and Off-heap memory.

JVM OnHeap memory: the size is specified by a "- - -" exterior-memory "(i.e., spark. Concurrent tasks running in the Executor share the JVM in-heap memory.

JVM OffHeap memory: the size is specified by a parameter of "spark.

Off-beat mode: the Off-heap mode memory is not enabled by default, and can be started through a parameter of "spark.

The kernel abstraction of spark for data is an elastic Distributed data set (RDD), which is a Distributed set of elements.

In spark, all operations on data do not like creating RDD, transforming existing RDD and calling RDD operation for job hunting, and behind this, spark will automatically distribute data in RDD to cluster and execute the operation in parallel.

Disclosure of Invention

The invention aims to provide a grain quality classification method based on Spark memory calculation. Aiming at the problem that 6 grades of various grains have obvious category imbalance, the invention provides that local data oversampling is adopted on each RDD data to solve the problem, and sampling can be realized by adding or deleting some categories of samples, so that the sizes of a few categories are closer to those of a plurality of categories; thereby obtaining a more scientific classification.

The technical scheme includes that the grain quality classification method based on Spark memory calculation is provided through an implementation mode and comprises two parts, wherein the first part is constructed for a grain depot quality evaluation index system, and the second part is the grain quality classification method based on Spark memory calculation, and the unbalanced problem of grain quality classification is solved. The invention specifically comprises the following contents:

1) construction of quality evaluation index system of grain depot

The method comprises the following steps of (1) constructing a quality evaluation system for grains (including wheat, early indica rice, late indica rice, indica glutinous rice, japonica rice and japonica glutinous rice) in a grain depot, wherein the quality evaluation system specifically comprises the following steps:

(1) quality index of grain

Mainly aiming at each grain bin in the grain depot: 1) volume weight; 2) imperfect grain proportion; 3) the proportion of the total content of impurities; 4) mineral content ratio 5) moisture ratio; 6) whether the color is normal or not; 7) whether the smell is normal or not is collected and uploaded to a grain information service cloud platform, and the grain quality is mainly classified into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

(2) Quality index

Mainly aiming at each grain (rice) bin in the grain depot: 1) the roughness is generated; 2) the rice-conditioning rate; 3) percentage of impurity content; 4) percentage of moisture content; 5) the percentage content of the yellow rice; 6) the percentage of the content of the brown rice outside the grains; 7) percent of intermixing rate; 8) whether the color is normal or not; 9) whether the smell is normal or not is collected and uploaded to a grain information service cloud platform, and the quality is divided into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

2) Grain quality classification method based on Spark memory calculation

And (4) constructing a Spark classifier by respectively inputting wheat or rice, and storing results between Map/Reduce steps in a main memory as much as possible by using Spark. Spark can directly multiplex the intermediate result in the memory without repeatedly reading HDFS (Hadoop distributed file system), and the processing performance of the classification method can be improved. Spark creates a Spark context object from the driver node, then the Spark context requests resources from the cluster manager, and obtains the executor on the cluster node, and the executor process manages task computation on one or more CPU threads. The invention mainly adopts an elastic distributed data set (RDD) data structure to abstract data for parallel computation. RDDs store data sets as partitions, which may be split across one or more cluster nodes. This abstraction allows a user to perform the same operation regardless of how the data is distributed. The functions such as Map, Filter, Reduce and the like can be directly run on the RDD, and the table operation similar to SQL can be realized by further abstract data frames (dataframes) and data structures of data sets.

Has the advantages that: aiming at the problem that the 6 grades of various grains have obvious class imbalance, the invention provides that local data oversampling is adopted on each RDD data to solve the problem, and the sampling can be realized by adding or deleting samples of some classes, so that the sizes of a few classes are closer to those of a plurality of classes. In particular, this is achieved by synthesizing a few oversampling techniques, namely: by copying and adding a randomly selected instance to the current data set, it may be necessary to balance the classes using alternate resampling, and then concatenating the sampled instances to the existing data frames to form a balanced data set. The invention solves the problem of unbalance of grain quality classification.

Drawings

FIG. 1 is an overall frame diagram of the present invention;

Detailed Description

The overall frame diagram of the invention is shown in fig. 1, firstly, the quality indexes of various grains based on each grain depot are selected, and then, based on the characteristics, the grains in different grain depots are classified by using a grain quality classification method based on Spark memory calculation. The specific implementation mode is as follows:

step 1: constructing a quality evaluation index system of the grain depot, calculating quality index values of different types of grains contained in bins of each grain depot according to data of each grain depot, taking some sample data marked manually as a training data set (the quality types of the grains in the bins are known), and taking other unmarked data as a test data set (the quality types of the grains in the bins are unknown). For (1) wheat; (2) early indica rice, late indica rice, and indica glutinous rice; (3) and respectively establishing different data sets for the japonica rice and the japonica glutinous rice.

The invention mainly aims at (1) wheat existing in grain depots of Jiangsu province; (2) early indica rice, late indica rice, and indica glutinous rice; (3) the method comprises the following steps of (1) constructing a quality evaluation system of japonica rice and japonica glutinous rice, wherein the quality evaluation system comprises the following specific steps:

(1) quality index of wheat

Mainly aiming at each wheat bin in the grain depot: 1) volume weight; 2) imperfect grain proportion; 3) the proportion of the total content of impurities; 4) mineral content ratio 5) moisture ratio; 6) whether the color is normal or not; 7) whether the smell is normal or not is collected and uploaded to a grain information service cloud platform, and the wheat quality is mainly classified into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

(2) Quality indexes of early indica rice, late indica rice and indica glutinous rice

Mainly aims at the situation that each early indica rice, late indica rice and indica rice bin in the grain depot: 1) the roughness is generated; 2) the rice-conditioning rate; 3) percentage of impurity content; 4) percentage of moisture content; 5) the percentage content of the yellow rice; 6) the percentage of the content of the brown rice outside the grains; 7) percent of intermixing rate; 8) whether the color is normal or not; 9) whether the odor is normal or not is collected and uploaded to a grain information service cloud platform, and the early indica type rice, the late indica type rice and the indica type rice are mainly classified into 6 types according to the mass: grade one, grade two, grade three, grade four, grade five, grade outside.

(3) Quality index of japonica rice and japonica glutinous rice

Mainly aims at the following parts in each japonica rice and japonica glutinous rice granary in the grain depot: 1) the roughness is generated; 2) the rice-conditioning rate; 3) percentage of impurity content; 4) percentage of moisture content; 5) the percentage content of the yellow rice; 6) the percentage of the content of the brown rice outside the grains; 7) percent of intermixing rate; 8) whether the color is normal or not; 9) whether the odor is normal or not is collected and uploaded to a grain information service cloud platform, and the method mainly divides the quality of the japonica rice and the japonica glutinous rice into 6 types: grade one, grade two, grade three, grade four, grade five, grade outside.

Step 2: the input data of the invention is stored on the HDFS of the grain information service cloud platform in a file Block (Block) form, and the data is loaded to the RDD from the HDFS. Since Spark is only a computing framework and does not have a file system function, HDFS has high fault tolerance and security backup features, and can realize high-throughput distributed computing.

And step 3: and initializing a classification result, wherein the step is mainly completed at a main node, and a data set with a classification mark is mainly used as an initialization category. And then selecting the data from the RDDs by a K nearest neighbor method as distribution data, and broadcasting and distributing the data to each RDD data set.

And 4, step 4: because of the serious category imbalance of 6 grades of each grain, the invention simulates an artificial minority class oversampling method to supplement minority class samples, namely, every time a new sample is created, 5 samples of the current class are randomly selected, and then each characteristic is respectively averaged, so that a new synthesized sample is generated, and the number of the samples of the classes is basically balanced on each RDD through the method.

And 5: to compute the minority class state for each existing sample, the invention first performs parallel k nearest neighbors with k set to 5. The parallel k-nearest neighbor algorithm performs k-nearest neighbors on all RDDs separately, while excluding the query point itself from the returned neighbors. Each instance will give a class label based on the number of adjacent instances belonging to its class, and then a small number of types of labels generated for each instance are stored as data frames, which can also be stored in the RDD for future reuse.

Step 6: the classification result summarization is realized by utilizing a Spark algorithm, a MapReduce idea is generally adopted, when each iteration is performed, the k nearest neighbor is executed in each RDD in a parallelization mode by using a Map class, then the best class of each k nearest neighbor classification is collected together by using a class Reduce operation so as to obtain more prediction classes of unlabeled data samples, all points with the same class are summarized to form a new RDD, the unlabeled samples with high class confidence coefficients are labeled as the predicted classes, and all samples with the labels are broadcasted to each node again for iteration until a target function converges or an iteration ending condition is met.