阅读说明：本技术 异构数据的处理方法、装置以及计算机设备 (Heterogeneous data processing method and device and computer equipment ) 是由 赵云鹏 曾经 于 2020-05-22 设计创作，主要内容包括：本申请提供了一种异构数据的处理方法、装置以及计算机设备,涉及数据处理技术领域,缓解了对异构数据进行处理的工作难度较大的技术问题。该方法包括：基于所述数据结构建立所述异构数据在数据加载过程中使用的数据加载映射类；按照所述异构数据的实体内容属性生成若干个实体类,并确定每个所述实体类与所述数据加载映射类之间的计算关系；利用所述计算关系将所述异构数据转化为数据结构相同的同构数据。(The application provides a processing method and device of heterogeneous data and computer equipment, relates to the technical field of data processing, and solves the technical problem that the work difficulty of processing the heterogeneous data is high. The method comprises the following steps: establishing a data loading mapping class used by the heterogeneous data in a data loading process based on the data structure; generating a plurality of entity classes according to the entity content attributes of the heterogeneous data, and determining the calculation relationship between each entity class and the data loading mapping class; and converting the heterogeneous data into isomorphic data with the same data structure by utilizing the calculation relationship.)

1. The processing method of the heterogeneous data is characterized in that the data structures of the heterogeneous data are different; the method comprises the following steps:

establishing a data loading mapping class used by the heterogeneous data in a data loading process based on the data structure;

generating a plurality of entity classes according to the entity content attributes of the heterogeneous data, and determining the calculation relationship between each entity class and the data loading mapping class;

and converting the heterogeneous data into isomorphic data with the same data structure by utilizing the calculation relationship.

2. The method of claim 1, wherein the step of establishing a data loading mapping class for the heterogeneous data to be used in a data loading process based on the data structure comprises:

selecting a data loader plug-in based on a data format in the data structure;

and establishing a data loading mapping class used by the heterogeneous data in a data loading process through the data loader plug-in.

3. The method of claim 1, wherein the step of generating entity classes according to the entity content attributes of the heterogeneous data comprises:

generating a plurality of entity classes according to the upstream data interface of the heterogeneous data, wherein the entity classes correspond to the entity content attributes of the upstream data interface;

the entity class of the isomorphic data is generated according to the structure of a data table, the data table contains fields related to all services, and the data conversion process of the isomerous data is executed according to the structure of the data table.

4. The method of claim 1, wherein the computational relationship is automatically replicated by computing an expression or an attribute name of the entity class.

5. The method according to claim 1 or 4, wherein the step of converting the heterogeneous data into isomorphic data with the same data structure by using the calculation relationship comprises:

calculating the attribute value of the data loading mapping class;

and assigning the attribute values to the entity classes by utilizing the calculation relationship so as to convert the heterogeneous data into isomorphic data with the same data structure.

6. The method of claim 2, wherein the data sources of the heterogeneous data are not the same; the method further comprises the following steps:

selecting a data downloader plug-in according to the data source;

and downloading the heterogeneous data by using the data downloader plug-in.

7. The method according to claim 6, wherein the data reading modes of the heterogeneous data are different; after the step of downloading the heterogeneous data by using the data downloader plug-in, the method further comprises the following steps:

determining a reading analysis mode of the heterogeneous data based on the data reading mode;

and analyzing and reading the heterogeneous data downloaded by the data downloader plug-in by utilizing the data loader plug-in according to the reading analysis mode.

8. The processing device of the heterogeneous data is characterized in that the data structures of the heterogeneous data are different; the device comprises:

the establishing module is used for establishing a data loading mapping class used by the heterogeneous data in a data loading process based on the data structure;

the determining module is used for generating a plurality of entity classes according to the entity content attributes of the heterogeneous data and determining the calculation relationship between each entity class and the data loading mapping class;

and the conversion module is used for converting the heterogeneous data into isomorphic data with the same data structure by utilizing the calculation relationship.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer readable storage medium having stored thereon machine executable instructions which, when invoked and executed by a processor, cause the processor to execute the method of any of claims 1 to 7.

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing heterogeneous data, and a computer device.

Background

In the process of data processing, a large batch of data of a plurality of different data structures is often required to be processed. For example, in the financial insurance or banking industry, data from a plurality of different fund channels needs to be interfaced, so that the data supply modes and the data format structures of the different fund channels are different, and a large amount of heterogeneous data needs to be processed.

The existing heterogeneous data batch processing method generally needs to develop corresponding different batch processing programs according to a data format providing mode of heterogeneous batch data, so that unified updating or warehousing operation of data can be performed. However, the data traffic is large, the processing logic is complex, the development workload is large, and the work difficulty of heterogeneous data processing is large.

Disclosure of Invention

The invention aims to provide a method and a device for processing heterogeneous data and computer equipment, so as to relieve the technical problem of high difficulty in processing the heterogeneous data.

In a first aspect, an embodiment of the present application provides a method for processing heterogeneous data, where data structures of the heterogeneous data are different; the method comprises the following steps:

establishing a data loading mapping class used by the heterogeneous data in a data loading process based on the data structure;

generating a plurality of entity classes according to the entity content attributes of the heterogeneous data, and determining the calculation relationship between each entity class and the data loading mapping class;

and converting the heterogeneous data into isomorphic data with the same data structure by utilizing the calculation relationship.

In one possible implementation, the step of establishing, based on the data structure, a data loading mapping class used by the heterogeneous data in a data loading process includes:

selecting a data loader plug-in based on a data format in the data structure;

and establishing a data loading mapping class used by the heterogeneous data in a data loading process through the data loader plug-in.

In one possible implementation, the step of generating a plurality of entity classes according to the entity content attribute of the heterogeneous data includes:

generating a plurality of entity classes according to the upstream data interface of the heterogeneous data, wherein the entity classes correspond to the entity content attributes of the upstream data interface;

the entity class of the isomorphic data is generated according to the structure of a data table, the data table contains fields related to all services, and the data conversion process of the isomerous data is executed according to the structure of the data table.

In one possible implementation, the computational relationship performs an automatic copy operation by computing an expression or an attribute name of the entity class.

In one possible implementation, the step of converting the heterogeneous data into isomorphic data with the same data structure by using the calculation relationship includes:

calculating the attribute value of the data loading mapping class;

and assigning the attribute values to the entity classes by utilizing the calculation relationship so as to convert the heterogeneous data into isomorphic data with the same data structure.

In one possible implementation, the data sources of the heterogeneous data are not the same; the method further comprises the following steps:

selecting a data downloader plug-in according to the data source;

and downloading the heterogeneous data by using the data downloader plug-in.

In one possible implementation, the data reading modes of the heterogeneous data are different; after the step of downloading the heterogeneous data by using the data downloader plug-in, the method further comprises the following steps:

determining a reading analysis mode of the heterogeneous data based on the data reading mode;

and analyzing and reading the heterogeneous data downloaded by the data downloader plug-in by utilizing the data loader plug-in according to the reading analysis mode.

In a second aspect, an apparatus for processing heterogeneous data is provided, where data structures of the heterogeneous data are different; the device comprises:

the establishing module is used for establishing a data loading mapping class used by the heterogeneous data in a data loading process based on the data structure;

the determining module is used for generating a plurality of entity classes according to the entity content attributes of the heterogeneous data and determining the calculation relationship between each entity class and the data loading mapping class;

and the conversion module is used for converting the heterogeneous data into isomorphic data with the same data structure by utilizing the calculation relationship.

In a third aspect, an embodiment of the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program executable on the processor, and the processor implements the method of the first aspect when executing the computer program.

In a fourth aspect, this embodiment of the present application further provides a computer-readable storage medium storing machine executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method of the first aspect.

The embodiment of the application brings the following beneficial effects:

the method, the device and the computer equipment for processing heterogeneous data can establish a data loading mapping class used by heterogeneous data in a data loading process based on a data structure, generate a plurality of entity classes according to entity content attributes of the heterogeneous data, determine a calculation relationship between each entity class and the data loading mapping class, and convert the heterogeneous data into isomorphic data with the same data structure by using the calculation relationship, in the scheme, the heterogeneous data is converted into the isomorphic data with the same data structure by the calculation relationship between each entity class and the data loading mapping class, the service and the function are completely separated, the service coupling is reduced, the reusability of a module is increased, repeated development work is avoided, the processing of multi-source heterogeneous data in batches can be flexibly supported, and a simple and rapid access mode is provided at the same time, the work difficulty of carrying out data processing on the heterogeneous data is reduced.

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the detailed description of the present application or the technical solutions in the prior art, the drawings needed to be used in the detailed description of the present application or the prior art description will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic flowchart of a method for processing heterogeneous data according to an embodiment of the present application;

fig. 2 is an example of an entity class attribute in the method for processing heterogeneous data according to the embodiment of the present application;

fig. 3 is another schematic flow chart of a method for processing heterogeneous data according to an embodiment of the present disclosure;

fig. 4 is an example of a general interface of a data downloader in the method for processing heterogeneous data according to the embodiment of the present application;

fig. 5 is another schematic flow chart of a method for processing heterogeneous data according to an embodiment of the present disclosure;

fig. 6 is another example of an entity class attribute in the method for processing heterogeneous data according to the embodiment of the present application;

fig. 7 is a schematic structural diagram of a device for processing heterogeneous data according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram illustrating a computer device provided in an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "comprising" and "having," and any variations thereof, as referred to in the embodiments of the present application, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

With the popularization of informatization, the scale and complexity of a service system are obviously increased, and the system association is tighter. The micro-service of the system further aggravates the coupling degree between the systems, and the stability of the system is more challenging due to excessive data sources, so that the data closed loop is promoted. The data closed loop is realized, batch synchronization of different system data, namely multi-source heterogeneous data is needed, the data smoothly enters a corresponding business system, and the system can correctly provide data service to enable business. In addition, in the financial insurance or banking industry, because of large data traffic and complex processing logic, batch data file exchange is required among systems to unify data states. For example, in the financial insurance business, data file exchange needs to be performed by interfacing with a plurality of fund channels (different bank channels and different trust channels), and for different fund channels, data providing modes are different, and structures have no unified standard. Therefore, the processing of multi-source heterogeneous data becomes an indispensable ring of a complex business system.

At present, in a conventional heterogeneous data batch processing method, a corresponding system developer generally develops a corresponding batch processing program according to a supply mode of heterogeneous batch data, and performs unified update or warehousing operation of data, but the method has the following problems:

first, flexibility is poor. The traditional batch data processing steps are as follows: data reading, data processing and data warehousing are carried out, different batch programs need to be developed for multi-source heterogeneous batch data to carry out the steps, the development workload is large, and the system quality developed by different developers is different. The ETL tool solves the problem of development workload of developers, but the ETL tool is high in integration level, poor in flexibility when being integrated with other systems or frames, complex in steps, and high in working difficulty when processing logic of internal modules needs to be improved aiming at services.

Secondly, the service coupling degree is high. When the system performs data warehousing, the business processing is not limited to one step, for example, a certain batch process: reading data, calculating an overdue total amount (if the field is empty) according to a business formula, and then warehousing (the field business corresponding relation needs to be specified), wherein except the first step, other steps are related to business, and the business can be distributed in one or more steps in the process, so that the business coupling degree is high.

Thirdly, the degree of multiplexing is low. The high service coupling of the system causes the special processing of special service scenes to be filled in the system module, and the multiplexing of the functional modules cannot be carried out.

Based on this, the embodiment of the application provides a method and a device for processing heterogeneous data and computer equipment, and by the method, the technical problem that the work difficulty of processing the heterogeneous data is high can be solved.

Embodiments of the present invention are further described below with reference to the accompanying drawings.

Fig. 1 is a flowchart illustrating a method for processing heterogeneous data according to an embodiment of the present disclosure. Wherein, the data structures of the heterogeneous data are different. As shown in fig. 1, the method includes:

step S110, establishing a data loading mapping class used by the heterogeneous data in the data loading process based on the data structure.

Step S120, a plurality of entity classes are generated according to the entity content attributes of the heterogeneous data, and the calculation relationship between each entity class and the data loading mapping class is determined.

And step S130, converting the heterogeneous data into isomorphic data with the same data structure by using the calculation relationship.

Heterogeneous data are converted into isomorphic data with the same data structure through the calculation relation between each entity class and the data loading mapping class, services are completely separated from functions, service coupling is reduced, reusability of modules is improved, repeated development work is avoided, processing of multi-source heterogeneous batch data can be flexibly supported, and meanwhile, a simple and rapid access mode is provided.

The above steps are described in detail below.

In practical application, multi-source heterogeneous data is generated by combining a data source, a data structure and a data reading mode, for example:

the following description will be given taking data whose data source is the FTP protocol and data structure is the TXT delimiter type as an example. As shown in the table below, data sources: FTP; data structure: a separator; a data reading mode: block read.

In some embodiments, the step S110 may include the following steps:

step a), selecting a data loader plug-in based on a data format in a data structure;

and b), establishing a data loading mapping class used by the heterogeneous data in the data loading process through the data loader plug-in.

For the preparation process of the data loading mapping class, the mapping class used in the data loading is established according to the data format provided by the third-party system, for example, "1 | + | minuscule | + | male | + |" may set the entity class attribute as shown in fig. 2. The attributes need to correspond to the heterogeneous data one to one, the corresponding mode depends on the implementation of a data loader, and the data in the example is cut by using "+ |" as a separator and is assigned into the attributes in sequence. In addition, the data loading mapping class needs to be placed in a file directory special for the data loading mapping class, and the system automatically scans class files in the directory to automatically load the class files.

For the selection process of the data loader, if the corresponding loader exists, the data loading mapping class object is returned. As shown in fig. 3, the TXT file in the example may select a TXT file delimiter pattern loader, and the file will be read stripe-by-stripe (streamed) and parsed into entity mapping classes in delimiter and data order.

If the corresponding loader does not exist, the data loading method needs to be realized according to a basic interface of the data loader, namely, the data loading method is realized through a custom plug-in. Specifically, each module plug-in of the system is realized by using a unified principle, when a default plug-in provided by the system cannot meet the service requirement, a user-defined plug-in is required to be realized, and the realization flow of the user-defined plug-in is exemplified by a data downloader plug-in at present.

For the implementation process of the corresponding module plug-in interface class of the system, a general interface of the data downloader is shown in fig. 4, the data downloader initializes by the initialization parameter transmitted by the system, and when the downloader is used for downloading, the specified downloading information is read, and the system automatically downloads the file into the directory path (local working directory). If FTP downloads, the initialization information is FTP connection authority information, and the download information is a work directory path and a remote path of a target file on the FTP.

Configuration plug-in base information, an example of which is shown below:

{

“name”:“DemoLoader”

“mainClass”:“com.test.DemoLoader”

}

the system obtains the implementation class information of the data downloader by reading the basic information of the plug-in, and loads all library files under the lower-level lib directory in the directory.

And (3) putting the plug-in into a corresponding directory, such as a downloader directory/dataDownloader, when the downloader is found to be used in the running and batching process of the system, carrying out hot loading on the plug-in, and carrying out initialization according to the configuration file so as to use the plug-in the system.

The reusability of the module is increased by introducing a plug-in mechanism of the loader, so that one-time development can be reused in any other subsystem developed based on the system through copy and paste. The flexible adaptive access of multi-source heterogeneous data is realized by using a plug-in adaptation mechanism. The problem of isomorphic data conversion of heterogeneous data is solved through a plug-in adaptation mechanism, and the multi-source heterogeneous batch data processing method based on the plug-in adaptation mechanism can realize one-time development and multi-place use of modules and completely decouple service centralized configuration and functions.

In some embodiments, the step S120 may include the following steps:

and c), generating a plurality of entity classes according to the upstream data interface of the heterogeneous data, wherein the entity classes correspond to the entity content attributes of the upstream data interface.

The entity class of the isomorphic data is generated according to the structure of a data table, the data table comprises fields related to all services, and the data conversion process of the isomorphic data is executed according to the structure of the data table.

It should be noted that a data table may be established in the database, where the data table needs to perform domain modeling according to the service, specifically, all service-related fields are extracted to establish a corpus data field table, and the database is a corpus data field table, that is, the data table. The structure of the data table can be used as a subsequent isomorphic data entity class, and heterogeneous data are subjected to data conversion according to the structure of the data table and then are stored in a warehouse. It will also be understood that the heterogeneous data will be field information present within a subset of the data table or just the binned data table.

Illustratively, a database table can be designed according to the stored information, a corresponding database table is established, an initialization tool provided by the system is used for initializing the system, and database connection and entity mapping relations are initialized.

In some embodiments, the data sources of the heterogeneous data are not the same; the method may further comprise the steps of:

step e), selecting a data downloader plug-in according to the data source;

and f), downloading the heterogeneous data by using the data downloader plug-in.

As shown in fig. 3 and 5, a data downloader may also be selected prior to the selection of a data loader. For the selection process of the data downloader, for example, the file read in the FTP is block read, so that the file needs to be completely downloaded and then parsed. If no corresponding data downloader exists in the system, the data downloader is realized according to the basic interface of the data downloader, and a local working directory path is returned.

The reusability of the module is increased by introducing a plug-in mechanism of the downloader, so that one-time development can be reused in any other subsystem developed based on the system through copy and paste. The flexible adaptive access of multi-source heterogeneous data is realized by using a plug-in adaptation mechanism. The problem of isomorphic data conversion of heterogeneous data is solved through a plug-in adaptation mechanism, and the multi-source heterogeneous batch data processing method based on the plug-in adaptation mechanism can realize one-time development and multi-place use of modules and completely decouple service centralized configuration and functions.

Based on the step e) and the step f), the data reading modes of the heterogeneous data are different; after the step f), the method may further comprise the steps of:

step g), determining a reading analysis mode of the heterogeneous data based on a data reading mode;

and h), analyzing and reading the heterogeneous data downloaded by the data downloader plug-in by using the data loader plug-in according to the reading analysis mode.

It should be noted that, the downloader is used to obtain data, and the loader is used to parse and read data, and the combination of the data downloader and the loader can be more flexible. Specifically, according to flexible development support of modules such as a downloader and a loader, the method is also applicable to any multi-source heterogeneous data generated by combining a data source, a data structure and a data reading mode, and a plug-in adaptation mechanism can be applied to each module according to the requirement of system flexibility.

In practical applications, the above steps may also be combined with channel information configuration, for example, opening a channel information configuration interface, and configuring channel information, where the channel information includes a channel code, a channel name, and channel description information.

The data downloader is used for determining the type of the data downloader, initializing parameters and downloading parameters, and the FTP downloader is provided with FTP connection information and corresponding path information.

For the data loader, the data loader is configured to determine a data loader type, an initialization parameter, and an execution parameter, and the TXT file in the example is delimiter, source character string, and data loading mapping class information.

For the calculation relationship between the entity class and the data loading mapping class, the automatic copying operation can be performed through the name of the calculation expression (such as the data mapping expression) or the attribute (such as the field) of the entity class. For the data mapping expression, a configurator designates an entity class at the position, loads an entity class attribute, and designates a calculation relationship between the entity class and the data loading mapping class attribute by configuring a calculation expression, for example, an entity class overdue total field can be filled in a mapping class field and "overduePricipal + overdueInterest", that is, overdue principal + overdue interest.

The system is more flexible through interface configuration of information and free selection of a data source structure, the system obviously improves the docking flexibility of multi-source isomerism by utilizing visual configuration and module selection, and data access can be completed without development work according to channels of existing templates. Moreover, the data mapping expression enables the service logic originally distributed in multiple steps to be processed in the data mapping expression in a centralized mode, and then disposable data processing is carried out when plug-in adaptation is carried out, so that the service and the function are completely separated.

In some embodiments, the step S130 may include the following steps:

step i), calculating the attribute value of the data loading mapping class;

and j) assigning the attribute values to the entity classes by utilizing the calculation relationship so as to convert the heterogeneous data into isomorphic data with the same data structure.

As shown in fig. 3 and 5, after data is loaded, a data adaptation process is performed, and data loaded by the data loader is mapped into channel custom mapping classes, but the mapping classes do not correspond to database entity classes one to one, that is, the plug-ins are loaded without a uniform data structure. A method is needed to adapt the heterogeneous data returned by the plug-in to homogeneous data.

In this step, the configured data mapping expression can be used, the data mapping expression configures the calculation relationship between the data loading mapping class and the entity class, and the data loading mapping class attribute value is obtained by analyzing the expression, is calculated and then is assigned to the entity mapping class, so that the data is isomorphic, and all the heterogeneous data are converted into entity class objects which are in one-to-one correspondence with the database in this step.

After the heterogeneous data is converted into the homogeneous data with the same data structure, necessary processing procedures of the homogeneous data, such as data validity judgment of interest rate, character string length check and the like, can be performed. As shown in fig. 3 and fig. 5, after the data processing process, data storage is performed, that is, the data is stored in this step using the ORM framework and is directly put into storage.

The following description will take the data source as Mysql database, the data structure as structured data, and the data reading mode as streaming reading as an example, and the following table shows the data.

After the system initialization process, data load mapping class preparation is performed. According to the database table structure, establishing a mapping class for exporting data, wherein the stored data is shown as the following table:

Id	name	sex
			1	xiaoming liquor	For male
2	Small red	Woman

The entity class attributes of the data are shown in fig. 6, the attributes need to be in one-to-one correspondence with heterogeneous data, the correspondence mode depends on the implementation of a data loader, and the data in the example is reflected and loaded into the entity mapping class according to the judgment basis of whether the database column name is the same as the entity class attribute name.

Next, selection by the data downloader is made after the custom plug-in implementation. In the traditional file transmission mode, each piece of data in a file can be correctly analyzed only by downloading the whole file, and a database can be read one by one (namely, a streaming reading mode), so that the step can be omitted and an empty data downloader can be directly used.

Then, as shown in fig. 3 and 5, a selection process of the data loader is performed. If no corresponding loader exists, the implementation is needed according to the basic interface of the data loader, and the data loading mapping class object is returned. The database data in the example is natural streaming read data, so that data reading can be directly realized in a data loader, the data is read out item by using a database cursor, and the data is analyzed into the entity mapping class according to the one-to-one correspondence relationship between the database column name and the attribute name of the entity mapping class. After the data loader is selected to be completed, channel information configuration, data adaptation, data processing, and data storage processes are performed as shown in fig. 3 and 5.

Fig. 7 provides a schematic structural diagram of a heterogeneous data processing apparatus. The data structures of the heterogeneous data are not identical. As shown in fig. 7, the heterogeneous data processing apparatus 700 includes:

an establishing module 701, configured to establish a data loading mapping class used in a data loading process for heterogeneous data based on a data structure;

a determining module 702, configured to generate a plurality of entity classes according to entity content attributes of heterogeneous data, and determine a calculation relationship between each entity class and a data loading mapping class;

the conversion module 703 is configured to convert the heterogeneous data into homogeneous data with the same data structure by using a calculation relationship.

In some embodiments, the establishing module 701 is specifically configured to:

selecting a data loader plug-in based on a data format in the data structure;

and establishing a data loading mapping class used by the heterogeneous data in the data loading process through the data loader plug-in.

In some embodiments, the determining module 702 is specifically configured to:

generating a plurality of entity classes according to the upstream data interface of the heterogeneous data, wherein the entity classes correspond to the entity content attributes of the upstream data interface;

the entity class of the isomorphic data is generated according to the structure of a data table, the data table contains fields related to all services, and the data conversion process of the isomerous data is executed according to the structure of the data table.

In some embodiments, the computational relationship performs an automatic copy operation by computing an expression or an attribute name of the entity class.

In some embodiments, the conversion module 703 is specifically configured to:

calculating the attribute value of the data loading mapping class;

and assigning the attribute values to the entity classes by utilizing the calculation relationship so as to convert the heterogeneous data into isomorphic data with the same data structure.

In some embodiments, the data sources of the heterogeneous data are not the same; the device also includes:

the selection module is used for selecting a data downloader plug-in according to a data source;

and the downloading module is used for downloading the heterogeneous data by using the data downloader plug-in.

In some embodiments, the data reading modes of the heterogeneous data are different; the device further comprises a loading module, wherein the loading module is specifically used for:

determining a reading analysis mode of the heterogeneous data based on the data reading mode;

and analyzing and reading the heterogeneous data downloaded by the data downloader plug-in by utilizing the data loader plug-in according to the reading analysis mode.

The processing device for heterogeneous data provided by the embodiment of the application has the same technical characteristics as the processing method for heterogeneous data provided by the embodiment of the application, so that the same technical problems can be solved, and the same technical effects can be achieved.

As shown in fig. 8, an embodiment of the present application provides a computer device 800, including: a processor 801, a memory 802 and a bus, wherein the memory 802 stores machine-readable instructions executable by the processor 801, when a computer device runs, the processor 801 communicates with the memory 802 through the bus, and the processor 801 executes the machine-readable instructions to execute the steps of the processing method of the heterogeneous data.

Specifically, the memory 802 and the processor 801 can be general-purpose memories and processors, which are not specifically limited herein, and when the processor 801 executes a computer program stored in the memory 802, the heterogeneous data processing method can be performed.

The processor 801 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 801. The Processor 801 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 802, and the processor 801 reads the information in the memory 802, and combines the hardware to complete the steps of the method.

Corresponding to the processing method of the heterogeneous data, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores machine executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the steps of the processing method of the heterogeneous data.

The heterogeneous data processing device provided by the embodiment of the present application may be specific hardware on a device, or software or firmware installed on a device, and the like. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments where no part of the device embodiments is mentioned. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

For another example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method for processing heterogeneous data according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.

Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the scope of the embodiments of the present application. Are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.