Mass data processing method, device, medium and electronic equipment

文档序号:1952890 发布日期:2021-12-10 浏览:13次 中文

阅读说明:本技术 一种海量数据处理方法、装置、介质及电子设备 (Mass data processing method, device, medium and electronic equipment ) 是由 王涛 刘立兰 王梅 于 2021-09-18 设计创作,主要内容包括:本申请实施例公开了一种海量数据处理方法、装置、介质及电子设备。该方法包括:根据预先确定的标签规则对海量数据进行处理,得到标签数据;其中,所述海量数据用于表征数据量较大的数据;所述标签规则用于表征对所述海量数据进行打标的规则;通过归并模型对所述标签数据进行分类处理,得到分类数据,并对所述分类数据进行压缩处理,得到目标数据,以用于对所述目标数据进行对比处理;其中,所述归并模型是根据预先配置的模型文件生成的。本技术方案,能够提高海量数据处理效率高,优化大数据应用分析性能。(The embodiment of the application discloses a mass data processing method, a mass data processing device, a mass data processing medium and electronic equipment. The method comprises the following steps: processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data; classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file. According to the technical scheme, the mass data processing efficiency can be improved, and the application analysis performance of the big data is optimized.)

1. A mass data processing method is characterized by comprising the following steps:

processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file.

2. The method of claim 1, wherein processing the mass data according to a predetermined tag rule to obtain tag data comprises:

acquiring a label rule, analyzing the label rule and generating a label marking condition; the label marking conditions comprise a data set, a marking basis field, marking logic and a marking field;

marking the mass data according to the label marking conditions and a preset task scheduling plan to obtain label data.

3. The method of claim 2, wherein after parsing the label rule to generate label marking conditions, the method further comprises:

and verifying the label marking conditions, and marking the mass data according to a preset task scheduling plan according to the label marking conditions if the label marking conditions pass the verification.

4. The method of claim 1, wherein classifying the tag data through a merging model to obtain classified data comprises:

obtaining a model description file and model parameters from a pre-configured model directory;

adding the model parameters to the model description file to generate a model file;

and checking the model file, if the model file passes the checking, establishing a merging model according to the model file, executing the merging model according to a task scheduling plan, and classifying the tag data to obtain classified data.

5. The method of claim 1, wherein after compressing the classified data to obtain target data, the method further comprises:

responding to input operation, extracting data to be processed from the target data, responding to information configuration operation, and performing information configuration on the data to be processed to obtain comparison data;

determining at least two comparison fields according to the comparison data, and processing the at least two comparison fields according to a preset comparison condition to obtain a comparison result; wherein the comparison condition comprises cross comparison, and at least one of cross comparison and difference comparison.

6. The method of claim 5, wherein the information setting operation comprises at least one of a data name configuration operation, a field filtering operation, a field display operation, and a dictionary configuration operation.

7. The method according to claim 5, wherein processing the at least two comparison fields according to a preset comparison condition to obtain a comparison result comprises:

and responding to the input operation of the front-end page, moving the comparison fields to a preset comparison position, and processing the at least two comparison fields of the same type according to a preset comparison condition based on the preset comparison position to obtain a comparison result.

8. A mass data processing apparatus, comprising:

the tag data obtaining module is used for processing the mass data according to a predetermined tag rule to obtain tag data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

the target data obtaining module is used for carrying out classification processing on the tag data through a merging model to obtain classified data, and carrying out compression processing on the classified data to obtain target data for comparison processing on the target data; wherein the merging model is generated according to a pre-configured model file.

9. A computer-readable medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out a method for mass data processing according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the mass data processing method according to any of claims 1-7 when executing the computer program.

Technical Field

The embodiment of the application relates to the technical field of big data analysis, in particular to a method, a device, a medium and electronic equipment for processing mass data.

Background

With the development of the internet, various data are increased explosively, and the internet mass data is mainly characterized by low value density, so that information wanted by people appears in the data at any time, and how to efficiently extract the information becomes an urgent need for many people. The mass data can mean that the related data size is large enough to be incapable of being intercepted, managed, processed and arranged into data which can be interpreted by human beings in reasonable time through manpower.

At present, a big data technology is often adopted to analyze and process mass data, so that effective information is extracted. For example, the analysis and query of data are realized through Hive SQL by adopting a Hadoop + Hive architecture. Hive is a set of data warehouse analysis system constructed based on Hadoop, and provides rich SQL query modes to analyze data stored in a Hadoop distributed file system. The structured data file can be mapped into a database table, and the complete SQL query function is provided.

Data analysis and query are realized through Hive SQL, massive data are not preprocessed, and execution time is too long when the data amount is huge.

Disclosure of Invention

The embodiment of the application provides a mass data processing method, a mass data processing device, a mass data processing medium and electronic equipment, which can improve the mass data processing efficiency and optimize the application analysis performance of big data.

In a first aspect, an embodiment of the present application provides a method for processing mass data, where the method includes:

processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file.

In a second aspect, an embodiment of the present application provides a mass data processing apparatus, where the apparatus includes:

the tag data obtaining module is used for processing the mass data according to a predetermined tag rule to obtain tag data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

the target data obtaining module is used for carrying out classification processing on the tag data through a merging model to obtain classified data, and carrying out compression processing on the classified data to obtain target data for comparison processing on the target data; wherein the merging model is generated according to a pre-configured model file.

In a third aspect, an embodiment of the present application provides a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a mass data processing method according to an embodiment of the present application.

In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable by the processor, where the processor executes the computer program to implement the mass data processing method according to the embodiment of the present application.

According to the technical scheme provided by the embodiment of the application, mass data are processed according to a predetermined label rule to obtain label data; and classifying the label data through the merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data. According to the technical scheme, the mass data processing efficiency can be improved, and the application analysis performance of the big data is optimized.

Drawings

Fig. 1 is a flowchart of a mass data processing method provided in an embodiment of the present application;

FIG. 2 is a flowchart illustrating scheduling execution of a model instance according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram of a mass data processing process provided in the second embodiment of the present application;

fig. 4 is a schematic structural diagram of a visualization interface provided in the second embodiment of the present application;

fig. 5 is a schematic structural diagram of a mass data processing apparatus according to a third embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.

Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example one

Fig. 1 is a flowchart of a mass data processing method provided in an embodiment of the present application, where this embodiment is applicable to a situation of performing comparison processing on mass data, and the method may be executed by a mass data processing apparatus provided in an embodiment of the present application, where the apparatus may be implemented by software and/or hardware, and may be integrated in a device such as an intelligent terminal for data processing.

As shown in fig. 1, the method for processing mass data includes:

s110, processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

the mass data may refer to data with a large data size obtained based on the internet. For example, a user's purchase record in a shopping website, a user's phone call record, or a user's travel track, etc.

In this scenario, the tag rule may be a rule sent by another system for marking data. The tag rule may be a rule in json (javascript Object notification) format. The label marking can be carried out on the mass data according to the content of the label rule to obtain the label data.

In this technical solution, optionally, the processing the mass data according to the predetermined tag rule to obtain the tag data includes:

acquiring a label rule, analyzing the label rule and generating a label marking condition; the label marking conditions comprise a data set, a marking basis field, marking logic and a marking field;

marking the mass data according to the label marking conditions and a preset task scheduling plan to obtain label data.

Wherein, a data set may refer to a source of mass data. For example, a shopping record from the user, a phone call record of the user, or a travel track of the user.

In this embodiment, the marking reference field may refer to a field for distinguishing mass data of different types, and is used to classify the mass data. Each piece of data is composed of a plurality of fields, the marking basis field can be one field in the data, and the marking basis field in each piece of data can be predetermined.

Wherein, marking logic can refer to specific marking rules. For example, assuming that the data set is a shopping record, the marking basis field is clothing, and the marking logic may be 100 or more, i.e., marking clothing with a purchase amount of 100 or more.

In this scenario, the marking field may refer to a field in the data for adding a label. The marking field may be determined in advance.

In this embodiment, a task scheduling schedule may be generated according to operation control information such as a task operation period, a start time, and the like, in the process of accessing the mass data, marking is performed on the mass data according to the marking task to obtain tag data, the tag data is input to a specified position, and other warehousing programs are provided to realize data entry into the Hive library.

By marking the mass data, the labeling of the mass data can be realized, and the processing and analysis of the mass data are facilitated.

In this technical solution, optionally, after analyzing the tag rule and generating the tag marking condition, the method further includes:

and verifying the label marking conditions, and marking the mass data according to a preset task scheduling plan according to the label marking conditions if the label marking conditions pass the verification.

The data set or the marking basis field in the label marking condition can be verified, whether the data set or the marking basis field is empty or not is judged, and if the data set or the marking basis field is empty, the verification is not passed; if not, the check passes.

In the scheme, if the verification is passed, the label engine generates a marking task according to a legal verification rule, and in the process of accessing the mass data, the mass data is marked according to the marking task to obtain the label data.

By checking the marking conditions of the label, the marking efficiency of mass data can be improved, and the application analysis performance of the mass data can be optimized.

S120, classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file.

In this embodiment, the merging model may be used to classify the tag data, and classify the data of the same class into one class. For example, data labeled the same may be classified into the same class.

In the scheme, after the classification data is obtained, the occurrence days can be counted according to the data generation time and the frequency or day, the first occurrence time and the last occurrence time of the data are recorded, the classification data is compressed to form target data, and the target data is stored in the MPP database. For example, the marking of food, daily necessities, and clothing is performed based on the purchase record, and the compression processing is performed on the purchase record to form the purchase merge data based on the tag.

By processing the data, the classification and compression of the data can be realized, the processing and analysis of mass data are facilitated, and the application analysis performance of the big data is optimized.

In this technical solution, optionally, the classifying the tag data through a merging model to obtain classified data includes:

obtaining a model description file and model parameters from a pre-configured model directory;

adding the model parameters to the model description file to generate a model file;

and checking the model file, if the model file passes the checking, establishing a merging model according to the model file, executing the merging model according to a task scheduling plan, and classifying the tag data to obtain classified data.

In this embodiment, a set of massive data merging processing technology is provided, and the technology is composed of six parts, namely data source docking, model XML, model directory, model analysis service, model instance scheduling, and model execution engine.

The data source is in butt joint with the input of label data; the model XML is a flow process description file with a data analysis purpose, is a carrier of a specific model, namely a model description file, can be written by a user under line and uploaded to a model directory; the model directory is responsible for storing the XML of the model; the model analysis service provides an external model analysis service interface; the model instance scheduling is responsible for model XML analysis, model analysis instance generation and instance control; the model execution engine is responsible for the specific execution of the model.

In the scheme, by the data source docking module, Hive connection of the label data is set, and then docking of the label data is achieved. And manually compiling the model description file offline according to the merging model requirement, and uploading the model description file to a model directory. And the model instance scheduling is used for acquiring the model description file and the model parameters from the model directory, adding the model parameters to the model description file, generating the model file and writing the successfully verified model file into the scheduling plan. And the model execution engine executes the model file according to the scheduling plan and classifies the label data to obtain classified data.

Exemplarily, fig. 2 is a flowchart of scheduling and executing a model instance provided in an embodiment of the present application, and as shown in fig. 2, an external model analysis service interface is provided by a model analysis service, a model XML, that is, a model description file, is obtained from a model directory, and a model parameter is received, and the model parameter is added to the model description file to generate the model file, and the model file is parsed, and field information in the model file is checked, if the check is passed, a model instance is created based on the model file, and whether the model instance is repeatedly checked, and if the check is passed, a scheduling state is returned; if not, the model instance will be written to the execution scheduling plan. The scheduling plan comprises a plurality of model files, the model execution engine executes the model files according to the scheduling plan, and the label data are classified to obtain classified data.

By classifying the tag data to obtain classified data and compressing the classified data, the data amount of mass data can be reduced and the data processing efficiency can be improved.

According to the technical scheme provided by the embodiment of the application, mass data are processed according to a predetermined label rule to obtain label data; and classifying the label data through the merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data. By executing the technical scheme, the mass data processing efficiency can be improved, and the application analysis performance of the big data is optimized.

Example two

Fig. 3 is a schematic diagram of a mass data processing process provided in the second embodiment of the present application, and the second embodiment is further optimized based on the first embodiment. The concrete optimization is as follows: after the classified data is compressed to obtain target data, the method further comprises: responding to input operation, extracting data to be processed from the target data, responding to information configuration operation, and performing information configuration on the data to be processed to obtain comparison data; determining at least two comparison fields according to the comparison data, and processing the at least two comparison fields according to a preset comparison condition to obtain a comparison result; wherein the comparison condition comprises cross comparison, and at least one of cross comparison and difference comparison. The details which are not described in detail in this embodiment are shown in the first embodiment. As shown in fig. 3, the method comprises the steps of:

s310, processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

s320, classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file

S330, responding to input operation, extracting data to be processed from the target data, responding to information configuration operation, and performing information configuration on the data to be processed to obtain comparison data;

in the scheme, a set of visual interfaces is provided, the data to be processed can be extracted from the target data stored in the database by operating the visual interfaces, and the data to be processed is configured to obtain the comparison data. The comparison data may be one or more data of the data to be processed.

Exemplarily, fig. 4 is a schematic structural diagram of a visualization interface provided in the second embodiment of the present application, and as shown in fig. 4, the visualization interface includes five functions of a data selection sub-module, a field configuration sub-module, an index establishment sub-module, a filter item generation sub-module, and a filter interface generation sub-module. The extraction and the comparative analysis of the data to be processed can be realized through a visual interface.

In this technical solution, optionally, the information setting operation includes at least one of a data name configuration operation, a field screening operation, a field display operation, and a dictionary configuration operation.

The data name configuration operation can be used for modifying and setting the data name; the field name configuration operation may be used to rename all field information in the loaded data; the field filtering operation may be used to filter all field information in the data. The screening mode comprises text input, enumeration screening and the like, when the enumeration screening field is set, the data content in the duplication-removing field needs to be analyzed, the enumeration screening option is automatically generated, and the screening items selected by the user are ensured to be data; the field display operation is used for displaying fields, and specific data can be viewed in the screening result page; and the dictionary configuration operation is used for encoding information according to the dictionary in the dictionary translation field when the corresponding field is displayed.

In the scheme, firstly, data to be processed is extracted from target data in an MPP library through a data selection submodule, and the name of the data to be processed is set; and carrying out field configuration on the data to be processed, wherein the configuration process comprises the following steps: and loading all field information of the selected data to be processed by the visual interface, renaming the fields to control the field display names on the interface, setting field display, setting screening items and setting a dictionary to complete field configuration to obtain comparison data.

In this embodiment, after the field configuration is completed, the system automatically generates an index creation statement according to the configuration condition, and different index types can be selected according to the data characteristics. According to the scheme, a btree type index is selected. The MPP database executes the sentence to automatically create the index. And finally, automatically generating a front-end page by the system according to the field visual configuration, wherein the front-end page comprises a data element icon, a screening interface and a result interface. The data element icon may refer to an icon for representing data, for example, the clothing data may be represented by a clothing icon, and the information may be represented by a small person icon; the screening interface is used for screening the data; and the result interface is used for displaying the processing result of the data for the user to check and compare the result.

The data to be processed is processed through the visual interface, so that the data processing efficiency can be improved, the application analysis performance of the big data is optimized, and the user experience can be improved.

S340, determining at least two comparison fields according to the comparison data, and processing the at least two comparison fields according to preset comparison conditions to obtain comparison results; wherein the comparison condition comprises cross comparison, and at least one of cross comparison and difference comparison.

In this embodiment, the comparison fields may be two fields in the same comparison data, or two fields in different comparison data. Wherein, the comparison result can be common data, same data or different data in the two comparison fields. For example, the comparison field can be shopping data and user information, and by comparing and analyzing the shopping data and the user information, the people can be determined for each type of shopping behavior, and the people with various shopping behaviors can be determined. By carrying out comparative analysis on the data, accurate analysis and mining of user groups can be carried out so as to support marketing strategies.

Wherein, cross-comparison can be used for extracting common data in two comparison fields; and the comparison can be used to extract the same data in the two compared fields; the difference contrast can be used to extract different data in the two contrast fields.

In the scheme, the specific comparison execution process comprises the following steps: firstly establishing a comparison task; generating SQL sentences by the comparison tasks; screening data; if the screened data is not empty, performing data comparison and processing the data; if the comparison result is not null, storing the comparison result data; and if the comparison result is empty, returning to the comparison task state.

In this technical solution, optionally, the processing the at least two comparison fields according to a preset comparison condition to obtain a comparison result includes:

and responding to the input operation of the front-end page, moving the comparison fields to a preset comparison position, and processing the at least two comparison fields of the same type according to a preset comparison condition based on the preset comparison position to obtain a comparison result.

In the scheme, firstly, the contrast data is dragged to the process drawing interface. Wherein, the comparison data is displayed on the interface in the form of icons. And setting a screening condition, screening the data, and confirming the data range by checking the data result. Two selected data A and B are selected, and a comparison condition is set. Firstly, selecting a comparison mode, providing collision comparison in the intersection, union and difference 3, wherein the intersection is A ^ B, the union is A ^ B, and the difference is (A-A ^ B) or (B-A ^ B). Then, a collision field is selected, and a comparison field for an equal value comparison is selected in A, B from the two comparison data sets. And after the comparison condition is selected, the system performs data collision according to the comparison condition. After collision is finished, the result can be displayed on a drawing interface, the result can be used for collision with other data or comparison results again, the system can automatically record a collision logic process, a collision flow chart is formed, a business analysis thought is recorded, and the thought sharing and the use among users are facilitated.

By carrying out comparative analysis on the data, useful information in the data can be extracted, and the application analysis performance of the big data is optimized.

According to the technical scheme provided by the embodiment of the application, mass data are processed according to a predetermined label rule to obtain label data; and classifying the label data through the merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data. Responding to the input operation, extracting data to be processed from the target data, responding to the information configuration operation, and performing information configuration on the data to be processed to obtain comparison data; determining at least two comparison fields according to the comparison data, and processing the at least two comparison fields according to a preset comparison condition to obtain a comparison result; wherein the comparison condition comprises at least one of cross comparison, and comparison and difference comparison. By executing the technical scheme, the mass data processing efficiency can be improved, and the application analysis performance of the big data is optimized.

EXAMPLE III

Fig. 5 is a schematic structural diagram of a mass data processing apparatus according to an embodiment of the present application, and as shown in fig. 5, the mass data processing apparatus includes:

a tag data obtaining module 510, configured to process the mass data according to a predetermined tag rule to obtain tag data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

a target data obtaining module 520, configured to perform classification processing on the tag data through a merging model to obtain classification data, and perform compression processing on the classification data to obtain target data, so as to perform comparison processing on the target data; wherein the merging model is generated according to a pre-configured model file.

In this technical solution, optionally, the tag data obtaining module 510 includes:

the label marking condition generating unit is used for acquiring a label rule, analyzing the label rule and generating a label marking condition; the label marking conditions comprise a data set, a marking basis field, marking logic and a marking field;

and the label data obtaining unit is used for marking the mass data according to the label marking conditions and a preset task scheduling plan to obtain the label data.

In this technical solution, optionally, the tag data obtaining module 510 further includes:

and the verification unit is used for verifying the label marking conditions, and marking the mass data according to a preset task scheduling plan according to the label marking conditions if the verification is passed.

In this technical solution, optionally, the target data obtaining module 520 is specifically configured to:

obtaining a model description file and model parameters from a pre-configured model directory;

adding the model parameters to the model description file to generate a model file;

and checking the model file, if the model file passes the checking, establishing a merging model according to the model file, executing the merging model according to a task scheduling plan, and classifying the tag data to obtain classified data.

In this technical solution, optionally, the apparatus further includes:

the comparison data obtaining module is used for responding to input operation, extracting data to be processed from the target data, responding to information configuration operation, and performing information configuration on the data to be processed to obtain comparison data;

a comparison result obtaining module, configured to determine at least two comparison fields according to the comparison data, and process the at least two comparison fields according to a preset comparison condition to obtain a comparison result; wherein the comparison condition comprises cross comparison, and at least one of cross comparison and difference comparison.

In this technical solution, optionally, the information setting operation includes at least one of a data name configuration operation, a field screening operation, a field display operation, and a dictionary configuration operation.

In this technical solution, optionally, the comparison result obtaining module is specifically configured to:

and responding to the input operation of the front-end page, moving the comparison fields to a preset comparison position, and processing the at least two comparison fields of the same type according to a preset comparison condition based on the preset comparison position to obtain a comparison result.

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.

Example four

Embodiments of the present application also provide a medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a method for processing mass data, the method including:

processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file.

Media-any of various types of memory devices or storage devices. The term "media" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The medium may also include other types of memory or combinations thereof. In addition, the medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system through a network (such as the internet). The second computer system may provide the program instructions to the computer for execution. The term "media" may include two or more media that may reside in different locations, such as in different computer systems that are connected by a network. The media may store program instructions (e.g., embodied as computer programs) that are executable by one or more processors.

Of course, the medium provided in the embodiments of the present application and containing computer-executable instructions is not limited to the above-mentioned mass data processing operation, and may also perform related operations in the mass data processing method provided in any embodiment of the present application.

EXAMPLE five

The embodiment of the application provides electronic equipment, and the electronic equipment can be integrated with the mass data processing device provided by the embodiment of the application. Fig. 6 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application. As shown in fig. 6, the present embodiment provides an electronic device 600, which includes: one or more processors 620; the storage device 610 is configured to store one or more programs, and when the one or more programs are executed by the one or more processors 620, the one or more processors 620 are enabled to implement the mass data processing method provided in the embodiment of the present application, the method includes:

processing the mass data according to a predetermined label rule to obtain label data; the mass data is used for representing data with large data volume; the label rule is used for representing a rule for marking the mass data;

classifying the tag data through a merging model to obtain classified data, and compressing the classified data to obtain target data for comparison processing of the target data; wherein the merging model is generated according to a pre-configured model file.

Of course, those skilled in the art can understand that the processor 620 also implements the technical solution of the mass data processing method provided in any embodiment of the present application.

The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 6, the electronic device 600 includes a processor 620, a storage device 610, an input device 630, and an output device 640; the number of the processors 620 in the electronic device may be one or more, and one processor 620 is taken as an example in fig. 6; the processor 620, the storage device 610, the input device 630, and the output device 640 in the electronic apparatus may be connected by a bus or other means, and are exemplified by being connected by a bus 650 in fig. 6.

The storage device 610 is a computer-readable medium and can be used for storing software programs, computer-executable programs, and module units, such as program instructions corresponding to the mass data processing method in the embodiment of the present application.

The storage device 610 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. In addition, the storage 610 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 610 may further include memory located remotely from the processor 620, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input means 630 may be used to receive input numbers, character information, or voice information, and to generate key signal inputs related to user settings and function control of the electronic device. The output device 640 may include a display screen, a speaker, and other electronic devices.

The electronic equipment provided by the embodiment of the application can achieve the purposes of improving the data processing efficiency and optimizing the big data application analysis performance.

The mass data processing device, the medium and the electronic device provided in the above embodiments may execute the mass data processing method provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. For technical details that are not described in detail in the above embodiments, reference may be made to a mass data processing method provided in any embodiment of the present application.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.

16页详细技术资料下载
上一篇:一种医用注射器针头装配设备
下一篇:用户类别调整方法、装置、设备及其存储介质

网友询问留言

已有0条留言

还没有人留言评论。精彩留言会获得点赞!

精彩留言,会给你点赞!