阅读说明：本技术 数据处理方法、装置和可读存储介质 (Data processing method, device and readable storage medium ) 是由 邱明杰 尹洪伟 高海龙 于 2021-07-21 设计创作，主要内容包括：本发明提供了一种数据处理方法、装置和可读存储介质。其中,数据处理方法包括：接收数据处理请求；根据数据处理请求,获取待处理数据源的信息和多个数据库脚本的信息；根据待处理数据源的信息在数据库中获取待处理数据源；根据多个数据库脚本的信息和待处理数据源的信息,生成多个相互独立的线程池；利用线程池,根据数据库脚本的信息,对待处理数据源进行处理,生成处理报告。本发明提供的技术方案通过对每个数据库脚本创建独立的线程池,使得每个数据库脚本的执行任务享有专用的线程池,进而使得多个数据库脚本能够同时执行过程,且执行过程中线程池相互独立互不影响,有效提高了数据库脚本的执行效率。(The invention provides a data processing method, a data processing device and a readable storage medium. The data processing method comprises the following steps: receiving a data processing request; acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request; acquiring a data source to be processed in a database according to the information of the data source to be processed; generating a plurality of mutually independent thread pools according to the information of the database scripts and the information of the data source to be processed; and processing the data source to be processed by utilizing the thread pool according to the information of the database script to generate a processing report. According to the technical scheme provided by the invention, the independent thread pool is established for each database script, so that the execution task of each database script shares the special thread pool, and further, a plurality of database scripts can be executed simultaneously, the thread pools are independent from each other and do not influence each other in the execution process, and the execution efficiency of the database scripts is effectively improved.)

1. A data processing method, comprising:

receiving a data processing request;

acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request;

acquiring the data source to be processed in a database according to the information of the data source to be processed;

generating a plurality of mutually independent thread pools according to the information of the database scripts and the information of the data source to be processed;

and processing the data source to be processed according to the information of the database script by using the thread pool to generate a processing report.

2. The data processing method according to claim 1, wherein the information about the data sources to be processed includes the number of the data sources to be processed, and the step of generating a plurality of thread pools independent of each other according to the information about the plurality of database scripts and the information about the data sources to be processed specifically includes:

analyzing the information of the database scripts to acquire script statements in each database script and the execution sequence of the script statements;

and generating a plurality of thread pools according to the script statements, the execution sequence and the number of the data sources to be processed of each database script.

3. The data processing method according to claim 2, wherein the step of processing the data source to be processed according to the information of the database script by using the thread pool specifically comprises:

acquiring a thread corresponding to the data source to be processed in the thread pool according to the script statement and the execution sequence;

and processing the data source to be processed according to the script statement and the execution sequence by utilizing the thread.

4. The data processing method according to claim 3, wherein the step of processing the data source to be processed by using the thread according to the script statement and the execution sequence specifically comprises:

comparing the execution sequence with a preset operation sequence of the database;

if the execution sequence is the same as the preset operation sequence, processing the data source to be processed by using the thread according to the script statement;

and if the execution sequence is different from the preset operation sequence, stopping processing the data source to be processed.

5. The data processing method according to claim 4, wherein the step of processing the data source to be processed by using the thread pool according to the information of the database script to generate a processing report specifically comprises:

acquiring processing process information of the data source to be processed and a processing result of the data source to be processed;

and generating the processing report according to the processing process information and the processing result.

6. The data processing method of claim 5,

the process information includes at least one of: the information of the data source to be processed, the processing duration information of the data source to be processed and the processing content information of the data source to be processed.

7. The data processing method according to any one of claims 1 to 5, wherein the obtaining information of a data source to be processed and information of a plurality of database scripts according to the data processing request specifically includes:

acquiring a first database type of the data processing request and a second database type of the database;

comparing the first database type to the second database type;

and if the first database type is the same as the second database type, acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request.

8. The data processing method according to any one of claims 1 to 5, wherein after the processing the data source to be processed according to the information of the database script by using the thread pool and generating a processing report, the method further comprises:

and displaying the processing report.

9. A data processing apparatus, comprising:

the receiving module is used for receiving a data processing request;

the first acquisition module is used for acquiring the information of a data source to be processed and the information of a plurality of database scripts according to the data processing request;

the second acquisition module is used for acquiring the data source to be processed in a database according to the information of the data source to be processed;

the first generation module is used for generating a plurality of mutually independent thread pools according to the information of the database scripts and the information of the data source to be processed;

the processing module is used for processing the data source to be processed according to the information of the database script by utilizing the thread pool;

and the second generation module is used for generating a processing report.

10. A readable storage medium on which a program or instructions are stored, characterized in that the program or instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8.

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method, a data processing apparatus, and a readable storage medium.

Background

In the related art, when multiple data sources of database scripts are executed, different database scripts are executed by using the same thread pool, however, resource contention occurs when multiple database scripts are executed simultaneously by using the same thread pool, and the execution efficiency is low.

Disclosure of Invention

The present invention is directed to solving at least one of the problems of the prior art or the related art.

To this end, a first aspect of the invention provides a data processing method.

The second aspect of the invention also provides a data processing device.

A third aspect of the invention also provides a readable storage medium.

In view of this, a first aspect of the present invention provides a data processing method, including: receiving a data processing request; acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request; acquiring a data source to be processed in a database according to the information of the data source to be processed; generating a plurality of mutually independent thread pools according to the information of the database scripts and the information of the data source to be processed; and processing the data source to be processed by utilizing the thread pool according to the information of the database script to generate a processing report.

In the technical scheme, after a data processing request is received, information of a data source to be processed and a plurality of database script information contained in the data processing request are determined. The information of the data source to be processed comprises the name of the data source to be processed and the number of the data source to be processed, and corresponding multiple data sources are selected from the database to execute the database script according to the determined name of the data source to be processed and the number of the data sources to be processed. Further, a plurality of thread pools are created according to the information of the database scripts and the determined multiple data sources, specifically, each thread pool corresponds to a respective database script, and the thread pools are independent and do not affect each other. And executing a plurality of database scripts for the plurality of data sources by using the created thread pools which are independent of each other, and generating a processing report.

Through the mode, the independent thread pool is established for each database script, the execution task of each database script enjoys the special thread pool, and in the execution process of different database scripts, the executed thread pools are independent and do not influence each other, the problem that resource contention occurs when multiple database scripts are executed simultaneously due to the fact that the same thread pool is adopted by different database scripts in the prior art is avoided, and the execution efficiency of the database scripts is effectively improved.

According to the above data processing method provided by the present invention, the following additional technical features may be further provided:

in the above technical solution, further, the step of generating a plurality of mutually independent thread pools according to the information of the plurality of database scripts and the information of the data sources to be processed, where the information of the data sources to be processed includes the number of the data sources to be processed, specifically includes: analyzing the information of the plurality of database scripts to acquire script statements and execution sequences of the script statements in each database script; and generating a plurality of thread pools according to the script statements, the execution sequence and the number of the data sources to be processed of each database script.

In the technical scheme, after a data processing request is received, a plurality of database scripts included in the data processing request are determined, the plurality of database scripts are analyzed, and a script statement and an execution sequence of the script statement of each database script are determined. And generating a special thread pool of each database script according to the script statements and the execution sequence, and executing a plurality of database scripts for multiple data sources simultaneously by using the special thread pool of each database script according to the script statements and the execution sequence of the database scripts.

Furthermore, each database script needs to process a large number of data sources at the same time, obtain the number of the data sources to be processed, and determine the number of dedicated thread pools corresponding to the generated database script according to the number of the data sources to be processed, where a plurality of thread pools corresponding to each database script are the same, and then execute the database script on a large number of data sources in batches by using a plurality of the same thread pools at the same time.

Through the mode, on one hand, the multiple independent thread pools are utilized, the multiple data sources can simultaneously execute the multiple database scripts, the resource contention is avoided, and the execution efficiency of the database scripts is effectively improved; on the other hand, the database script is executed for multiple data sources in batches, so that the problem that the server is congested due to the fact that the data of the database script is excessively large when the number of the data sources to be executed is too large is avoided.

Specifically, the set number of the dedicated thread pools corresponding to each database script is determined according to the number of the actual data sources to be processed, and is not specifically limited herein.

In any of the above technical solutions, further, the step of processing the data source to be processed by using the thread pool according to the information of the database script specifically includes: acquiring a thread corresponding to a data source to be processed in a thread pool according to the script statement and the execution sequence; and processing the data source to be processed by utilizing the thread according to the script statement and the execution sequence.

In the technical scheme, in the process of executing a plurality of database scripts, threads for executing the database scripts to multiple data sources are determined in a thread pool according to script statements and execution sequences among the statements in each database script, and the database scripts are executed to the multiple data sources according to the statements and the execution sequences among the statements of the database scripts by using the determined threads.

By the method, the multiple data sources are processed by the dedicated thread corresponding to each database script, so that a user can uniformly execute different database scripts in the selected multiple data sources according to needs to adapt to service scenes such as multiple data sources or a multi-tenant mode.

In any of the above technical solutions, further, the step of processing, by using a thread, the data source to be processed according to the script statement and the execution sequence specifically includes: comparing the execution sequence with a preset operation sequence of the database; if the execution sequence is the same as the preset operation sequence, processing the data source to be processed by using the thread according to the script statement; and if the execution sequence is different from the preset operation sequence, stopping processing the data source to be processed.

In the technical scheme, in the process of executing the database script on multiple data sources, the execution sequence of each script statement in the database script is compared with the statement execution sequence (namely, the preset operation sequence) of the database, and if the execution sequence of the script statement is consistent with the statement execution sequence of the database, the script statement is executed on the multiple data sources by using threads in a thread pool; and if the execution sequence of any script statement is found to be inconsistent with the execution sequence of the database in the execution process, stopping executing the database script on the multiple data sources.

By the method, the statement execution sequence of the database scripts is compared with the statement sequence of the database, so that the consistency of the statement execution sequence of each database script in the execution process of multiple data sources and the correct execution sequence of the database is ensured, and the coordination and the accuracy of the multiple data sources for executing the database scripts are ensured.

In any of the above technical solutions, further, the step of processing the data source to be processed by using the thread pool according to the information of the database script and generating the processing report specifically includes: acquiring processing process information of a data source to be processed and a processing result of the data source to be processed; and generating a processing report according to the processing process information and the processing result.

In the technical scheme, in the execution process of the multiple data sources executing the database script, the execution process information of each data source and the execution result of each data source, which is executed successfully or failed, are recorded and stored, and the execution process and the execution result of the multiple data sources executing the database script are summarized to generate an execution report (i.e., a processing report).

By the method, the process and the result of each statement in the database script are executed by the multiple data sources are recorded and stored, and then all the database script execution processes and results are collected to generate a report after the execution is finished. Compared with the prior art that the execution result of a single data source is checked, the user can check the database script executed by multiple data sources in real time, and further, the execution report is generated by combining the execution process and the execution result, so that the convenience of checking and contrastive analysis of the user is improved.

In any of the above technical solutions, further, the processing procedure information includes at least one of: the information of the data source to be processed, the processing duration information of the data source to be processed and the processing content information of the data source to be processed.

In the technical scheme, in the process of executing the database script by multiple data sources, the execution process of each statement of each data source is recorded and stored to obtain processing process information, wherein the processing process information comprises the names of the multiple data sources, the execution consumption duration of each data source, the execution start and stop time of each data source and the execution content of each data source.

By the method, the execution process of each statement of each data source is recorded and stored in real time in the execution process according to the data source as a unit, so that the completeness of the execution record of the database script is ensured, and a user can conveniently and comprehensively know the execution condition.

In any of the above technical solutions, further, acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request specifically includes: acquiring a first database type of a data processing request and a second database type of a database; comparing the first database type to the second database type; and if the first database type is the same as the second database type, acquiring the information of the data source to be processed and the information of the plurality of database scripts according to the data processing request.

In the technical scheme, the received data processing request can simultaneously process data sources in a plurality of different types of databases, the database type (namely, the first database type) in the data processing request and the database type (namely, the second database type) of the database script to be executed are obtained, the two database types are compared, if the database type in the data processing request is the same as the type of the database to be executed, namely, the database script in the data processing request can be successfully executed, at the moment, the information of the data source to be processed and the plurality of database scripts are determined according to the data processing request, and then the database script information is executed on a plurality of data sources. Further, if the type of the database in the data processing request is different from the type of the database to be executed, that is, the database script in the data processing request cannot be executed successfully, the next action of the data processing request is stopped, that is, the database script in the data processing request is stopped being executed.

By the method, the database type in the data processing request is compared with the database type to be executed, and the next execution action is determined according to the comparison result. On one hand, the consistency of the data processing request and the database to be executed is ensured, and the correctness of the database script executed by multiple data sources is further ensured; on the other hand, if the data processing request is inconsistent with the type comparison result of the database, the database script does not need to be executed for multiple data sources in inconsistent types, so that the resource occupation is effectively reduced, and the execution efficiency of the database script is improved.

In any of the above technical solutions, further, after processing, by using the thread pool, the data source to be processed according to the information of the database script and generating the processing report, the method further includes: and displaying the processing report.

In the technical scheme, the execution process and the execution result of the database script executed by multiple data sources are written into a log, and after an execution report is generated, all the database script execution reports are provided to a browser associated with the data processing device, and the browser generates a table form for displaying.

Through the method, the processing report generated by the database script executed by multiple data sources at each time is sent to the browser, so that the browser can be displayed to a user in a form of a table. Compared with the prior art that a user usually logs in each database server to continue inspection, the execution information of the database script is executed through the collected multiple data sources and is sent to the front end, the execution information is visually displayed in a tabular mode, the user does not need to frequently log in the server to perform inspection, manual repetitive labor is avoided, and the problem that the time is consumed for respectively checking the execution results of the data sources in the database is solved.

According to a second aspect of the present invention, there is provided a data processing apparatus comprising: the receiving module is used for receiving a data processing request; the first acquisition module is used for acquiring the information of a data source to be processed and the information of a plurality of database scripts according to the data processing request; the second acquisition module is used for acquiring the data source to be processed in the database according to the information of the data source to be processed; the first generation module is used for generating a plurality of mutually independent thread pools according to the information of the database scripts and the information of the data source to be processed; the processing module is used for processing the data source to be processed according to the information of the database script by utilizing the thread pool; and the second generation module is used for generating a processing report.

In the technical scheme, the data processing device comprises a receiving module, a first obtaining module, a second obtaining module, a first generating module, a processing module and a second generating module.

Specifically, the receiving module is configured to receive a data processing request. The first acquisition module is used for determining the information of the data source to be processed and the information of the plurality of database scripts contained in the data processing request after receiving the data processing request. The second acquisition module is used for selecting corresponding multiple data sources from the database to execute the database script according to the determined name of the data source to be processed and the number of the data sources to be processed. Further, the first generating module is configured to create a plurality of thread pools according to information of the plurality of database scripts and the determined data source to be processed, specifically, each thread pool corresponds to a respective database script, and the thread pools are independent of each other and do not affect each other. The processing module is used for executing a plurality of database scripts for a plurality of data sources by utilizing the created thread pools which are independent from each other, and the second generating module is used for generating a processing report.

Through the mode, the independent thread pool is established for each database script, the execution task of each database script enjoys the special thread pool, and in the execution process of different database scripts, the executed thread pools are independent and do not influence each other, the problem that in the prior art, different database scripts adopt the same thread pool, multiple database scripts are executed simultaneously, resource contention occurs is avoided, and the execution efficiency of the database scripts is effectively improved.

According to a third aspect of the present invention, a readable storage medium is proposed, on which a program or instructions are stored, which program or instructions, when executed by a processor, perform the data processing method proposed by the first aspect. Therefore, the readable storage medium has all the advantages of the data processing method provided by the first aspect, and redundant description is omitted to avoid repetition.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 shows one of the flow diagrams of a data processing method of an embodiment of the invention;

FIG. 2 is a second flowchart of a data processing method according to an embodiment of the present invention;

FIG. 3 shows a schematic block diagram of a data processing apparatus of an embodiment of the present invention;

FIG. 4 is a process report diagram illustrating an exemplary embodiment of the present invention.

Wherein, the corresponding relationship between the reference numbers and the names of the components in fig. 3 is:

300 data processing apparatus, 302 receiving module, 304 first obtaining module, 306 second obtaining module, 308 first generating module, 310 processing module, 312 second generating module.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the description in the present invention as referring to "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.

A data processing method, a data processing apparatus, and a readable storage medium according to some embodiments of the present invention are described below with reference to fig. 1 to 4.

Example 1:

as shown in fig. 1, according to an embodiment of the present invention, there is provided a data processing method including:

step 102, receiving a data processing request;

104, acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request;

step 106, acquiring a data source to be processed in a database according to the information of the data source to be processed;

108, generating a plurality of mutually independent thread pools according to the information of the plurality of database scripts and the information of the data source to be processed;

step 110, processing the data source to be processed by using the thread pool according to the information of the database script;

at step 112, a process report is generated.

In this embodiment, after receiving the data processing request, information of the data source to be processed and the plurality of database script information included in the data processing request are determined. The information of the data source to be processed comprises the name of the data source to be processed and the number of the data source to be processed, and corresponding multiple data sources are selected from the database to execute the database script according to the determined name of the data source to be processed and the number of the data sources to be processed. Further, a plurality of thread pools are created according to the information of the database scripts and the determined multiple data sources, specifically, each thread pool corresponds to a respective database script, and the thread pools are independent and do not affect each other. And executing a plurality of database scripts for the plurality of data sources by using the created thread pools which are independent of each other, and generating a processing report.

Through the mode, the independent thread pool is established for each database script, the execution task of each database script enjoys the special thread pool, and in the execution process of different database scripts, the executed thread pools are independent and do not influence each other, the problem that in the prior art, different database scripts adopt the same thread pool, multiple database scripts are executed simultaneously, resource contention occurs is avoided, and the execution efficiency of the database scripts is effectively improved.

In a specific embodiment, according to the received data processing request, three database scripts and the execution time of the three database scripts included in the data processing request are confirmed. Before executing the first database script according to the execution time, a first thread pool corresponding to the first database script is established through the information of the database script, and the first database script is executed by utilizing the exclusive first thread pool of the database script. In the process of executing the first database script, according to the execution time, a second database script needs to be executed at the same time, at the moment, a second thread pool corresponding to the second database script is established according to the information of the second database script, then the second database script is executed according to a second thread pool exclusive for the database script, and so on, before each database script is executed, the thread pool is initialized, the thread pool corresponding to the database script is established, and a plurality of database scripts can utilize the exclusive thread pool to execute tasks at the same time.

Example 2:

as shown in fig. 2, according to an embodiment of the present invention, there is provided a data processing method including:

step 202, receiving a data processing request;

step 204, acquiring information of a data source to be processed and information of a plurality of database scripts according to the data processing request;

step 206, acquiring the data source to be processed in the database according to the information of the data source to be processed;

step 208, analyzing the information of the plurality of database scripts to obtain script statements and execution sequences of the script statements in each database script;

step 210, generating a plurality of thread pools according to the script statements, the execution sequence and the number of the data sources to be processed of each database script;

step 212, processing the data source to be processed according to the information of the database script by using the thread pool;

at step 214, a process report is generated.

In this embodiment, after receiving the data processing request, a plurality of database scripts included in the data processing request are determined, the plurality of database scripts are analyzed, and a script statement and an execution sequence of the script statement of each database script are determined. And generating a special thread pool of each database script according to the script statements and the execution sequence, and executing a plurality of database scripts for multiple data sources simultaneously by using the special thread pool of each database script according to the script statements and the execution sequence of the database scripts.

Furthermore, each database script needs to process a large number of data sources at the same time, obtain the number of the data sources to be processed, and determine the number of dedicated thread pools corresponding to the generated database script according to the number of the data sources to be processed, where a plurality of thread pools corresponding to each database script are the same, and then execute the database script on a large number of data sources in batches by using a plurality of the same thread pools at the same time.

Through the mode, on one hand, the multiple independent thread pools are utilized, the multiple data sources can simultaneously execute the multiple database scripts, the resource contention is avoided, and the execution efficiency of the database scripts is effectively improved; on the other hand, the database script is executed for multiple data sources in batches, so that the problem that the server is congested due to the fact that the data of the database script is excessively large when the number of the data sources to be executed is too large is avoided.

Specifically, the set number of the dedicated thread pools corresponding to each database script is determined according to the number of the actual data sources to be processed, and is not specifically limited herein.

In a specific embodiment, the multiple database scripts are analyzed to obtain script statements in the first database script, execution sequences of the statements, and names of corresponding 10 data sources to be processed. Generating thread pools corresponding to 5 first database scripts through the determined script statements, the execution sequence of the statements and 10 data sources to be processed, and processing the 10 data sources to be processed in two batches according to the statements and the execution sequence of the first data scripts by utilizing the exclusive thread pools of the 5 first database scripts.

Example 3:

in this embodiment, the step of processing the data source to be processed by using the thread pool according to the information of the database script includes: acquiring a thread corresponding to a data source to be processed in a thread pool according to the script statement and the execution sequence; and processing the data source to be processed by utilizing the thread according to the script statement and the execution sequence.

In the embodiment, in the process of executing a plurality of database scripts, according to script statements and execution sequence among the statements in each database script, threads for executing the database scripts to a plurality of data sources are determined in a thread pool, and the database scripts are executed to the plurality of data sources according to the statements and the execution sequence among the statements of the database scripts by using the determined threads.

By the method, the multiple data sources are processed by the dedicated thread corresponding to each database script, so that a user can uniformly execute different database scripts in the selected multiple data sources according to needs to adapt to service scenes such as multiple data sources or a multi-tenant mode.

Example 4:

in this embodiment, the step of processing the data source to be processed by using the thread according to the script statement and the execution sequence includes: comparing the execution sequence with a preset operation sequence of the database; if the execution sequence is the same as the preset operation sequence, processing the data source to be processed by using the thread according to the script statement; and if the execution sequence is different from the preset operation sequence, stopping processing the data source to be processed.

In this embodiment, in the process of executing the database script on multiple data sources, the execution sequence of each script statement in the database script is compared with the statement execution sequence of the database itself (i.e. the preset operation sequence), and if the execution sequence of the script statement is consistent with the statement execution sequence of the database itself, the script statement is executed on multiple data sources by using threads in the thread pool; and if the execution sequence of any script statement is found to be inconsistent with the execution sequence of the database in the execution process, stopping executing the database script on the multiple data sources.

By the method, the statement execution sequence of the database scripts is compared with the statement sequence of the database, so that the consistency of the statement execution sequence of each database script in the execution process of multiple data sources and the correct execution sequence of the database is ensured, and the coordination and the accuracy of the multiple data sources for executing the database scripts are ensured.

Example 5:

in this embodiment, the step of processing the data source to be processed by using the thread pool according to the information of the database script and generating the processing report includes: acquiring processing process information of a data source to be processed and a processing result of the data source to be processed; and generating a processing report according to the processing process information and the processing result.

In the technical scheme, in the execution process of the multiple data sources executing the database script, the execution process information of each data source and the execution result of each data source, which is executed successfully or failed, are recorded and stored, and the execution process and the execution result of the multiple data sources executing the database script are summarized to generate an execution report (i.e., a processing report).

By the method, the process and the result of each statement in the database script are executed by the multiple data sources are recorded and stored, and then all the database script execution processes and results are collected to generate a report after the execution is finished. Compared with the prior art that the execution result of a single data source is checked, the user can check the database script executed by multiple data sources in real time, and further, the execution report is generated by combining the execution process and the execution result, so that the convenience of checking and contrastive analysis of the user is improved.

Example 6:

in this embodiment, the processing procedure information includes at least one of: the information of the data source to be processed, the processing duration information of the data source to be processed and the processing content information of the data source to be processed.

In the technical scheme, in the process of executing the database script by multiple data sources, the execution process of each statement of each data source is recorded and stored to obtain processing process information, wherein the processing process information comprises the names of the multiple data sources, the execution consumption duration of each data source, the execution start and stop time of each data source and the execution content of each data source.

By the method, the execution process of each statement of each data source is recorded and stored in real time in the execution process according to the data source as a unit, so that the completeness of the execution record of the database script is ensured, and a user can conveniently and comprehensively know the execution condition.

Example 7:

in this embodiment, the obtaining information of the data source to be processed and information of the plurality of database scripts according to the data processing request includes: acquiring a first database type of a data processing request and a second database type of a database; comparing the first database type to the second database type; and if the first database type is the same as the second database type, acquiring the information of the data source to be processed and the information of the plurality of database scripts according to the data processing request.

In the technical scheme, the received data processing request can simultaneously process data sources in a plurality of different types of databases, the database type (namely, the first database type) in the data processing request and the database type (namely, the second database type) of the database script to be executed are obtained, the two database types are compared, if the database type in the data processing request is the same as the type of the database to be executed, namely, the database script in the data processing request can be successfully executed, at the moment, the information of the data source to be processed and the plurality of database scripts are determined according to the data processing request, and then the database script information is executed on a plurality of data sources. Further, if the type of the database in the data processing request is different from the type of the database to be executed, that is, the database script in the data processing request cannot be executed successfully, the next action of the data processing request is stopped, that is, the database script in the data processing request is stopped being executed.

By the method, the database type in the data processing request is compared with the database type to be executed, and the next execution action is determined according to the comparison result. On one hand, the consistency of the data processing request and the database to be executed is ensured, and the correctness of the database script executed by multiple data sources is further ensured; on the other hand, if the data processing request is inconsistent with the type comparison result of the database, the database script does not need to be executed for multiple data sources in inconsistent types, so that the resource occupation is effectively reduced, and the execution efficiency of the database script is improved.

Example 8:

in this embodiment, after processing the data source to be processed according to the information of the database script by using the thread pool and generating the processing report, the method further includes: and displaying the processing report.

In the technical scheme, the execution process and the execution result of the database script executed by multiple data sources are written into a log, and after an execution report is generated, all the database script execution reports are provided to a browser associated with the data processing device, and the browser generates a table form for displaying.

Through the method, the processing report generated by the database script executed by multiple data sources at each time is sent to the browser, so that the browser can be displayed to a user in a form of a table. Compared with the prior art that a user usually logs in each database server to continue inspection, the method and the system for inspecting the database script in the database send the execution information of the database script to the front end through the collected multiple data sources, and visually display the execution information in a tabular mode without frequently logging in the server to inspect by the user, avoid manual repetitive labor, and solve the problem that the execution result of the data sources in the database is time-consuming to check.

In a specific embodiment, as shown in table 1, a processing report of database scripts is executed for multiple data sources displayed in a tabular form, wherein the name of each data source, the execution result of the data source, the start and stop time of the execution, the execution consumption duration, the executor and the recording time are respectively stored in units of the data sources, and specifically, the execution consumption duration is the time required for the database scripts of each data source to execute from the beginning to the completion of the execution and is measured in milliseconds. Further, as shown in fig. 4, to execute the execution content of the database script on multiple data sources, the content of the execution statements corresponding to all data sources can be clearly seen.

TABLE 1

Example 9:

according to a specific embodiment of the invention, the data processing method is provided, when the database scripts execute multiple data sources, independent thread pools are started to execute through each database script, and the thread pools executed by different database scripts are independent and do not influence each other. By executing and controlling database transactions for each different type of database script, consistency of sequential execution and database transactions of different types of statements of the same database script is ensured.

In the prior art, with the development of internet technology, multi-tenant technology (multi-tenant technology) and Software-as-a-service (SaaS) are increasingly applied, the independence and integrity of data of each tenant (user) cannot be isolated by using database multi-tenant technology, when the number of tenants is thousands of tenants, the number of data sources corresponding to the tenant is also thousands of tenants, and the basic information management and maintenance work of a large number of data sources, such as the workload and complexity of execution of various types of database scripts and result summary viewing, are greatly increased, and the probability of time and labor waste and error is greater than that of tools and systems when the tasks are completed manually. Therefore, a new multiple data source database script execution and result summarization scheme is needed, which can support the execution and real-time viewing of results in all or a specified data source of a data Query language dql (data Query language), a data Manipulation language dml (data management language), a data Definition language ddl (data Definition language), a data control language dcl (data Command language) type data script, so as to reduce the execution workload and complexity of the multiple data source database script, improve the execution efficiency, and reduce the workload.

Compared with the prior art that the same thread pool technology with different tasks is generally adopted, the data processing method enables the execution task of each database script to share the special thread pool, solves the problem of resource contention caused by simultaneous execution of multiple scripts, and improves the execution efficiency of the database scripts.

Furthermore, different types of data sources can be customized, database scripts are uniformly executed in the customized data sources, and the execution result after the aggregation is checked. In contrast to the prior art, database scripts are executed for a single data source. The database script is executed for multiple data sources, and the execution of the database script in the databases of all tenants is supported, so that a user can uniformly execute the database script in a plurality of selected data sources according to the requirement, and the method can adapt to business scenes such as multiple data sources or a multi-tenant mode.

Furthermore, the method and the device can be used for uniformly summarizing and checking the script execution results of the multiple data source databases. Compared with the prior art, the execution result of a single data source is checked. By summarizing the execution results of the multiple data source database scripts, the database scripts executed by multiple data sources at each time can check the summarized results in real time, and the problem that the time consumption is consumed for checking the execution results of single data sources respectively is solved.

Example 10:

as shown in fig. 3, an embodiment of the second aspect of the present invention proposes a data processing apparatus 300, wherein the data processing apparatus 300 comprises:

in this embodiment, the data processing apparatus 300 includes a receiving module 302, a first obtaining module 304, a second obtaining module 306, a first generating module 308, a processing module 310, and a second generating module 312.

Specifically, the receiving module 302 is configured to receive a data processing request. The first obtaining module 304 is configured to, after receiving the data processing request, determine information of a to-be-processed data source and multiple database script information included in the data processing request. The information of the data source to be processed includes a name of the data source to be processed and a number of the data source to be processed, and the second obtaining module 306 is configured to select a corresponding multiple data sources from the database to execute the database script according to the determined name of the data source to be processed and the number of the data source to be processed. Further, the first generating module 308 is configured to create a plurality of thread pools according to information of the plurality of database scripts and the determined data source to be processed, specifically, each thread pool corresponds to a respective database script, and the thread pools are independent from each other and do not affect each other. The processing module 310 is configured to execute a plurality of database scripts on a plurality of data sources by using the created plurality of independent thread pools, and the second generating module 312 is configured to generate a processing report.

Through the mode, the independent thread pool is established for each database script, the execution task of each database script enjoys the special thread pool, and in the execution process of different database scripts, the executed thread pools are independent and do not influence each other, the problem that in the prior art, different database scripts adopt the same thread pool, multiple database scripts are executed simultaneously, resource contention occurs is avoided, and the execution efficiency of the database scripts is effectively improved.

Example 11:

in an embodiment of the third aspect of the present invention, a readable storage medium is provided, on which a program or instructions are stored, and the program or instructions, when executed by a processor, implement the steps of the data processing method according to any one of the above-mentioned technical solutions.

The readable storage medium includes a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.

The readable storage medium, the program or the instructions provided by the present invention, when executed by the processor, implement the steps of the data processing method according to any of the above technical solutions, and therefore the readable storage medium includes all the beneficial effects of the data processing method according to any of the above technical solutions, which are not described herein again. Fresh air outlet

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.