阅读说明：本技术 一种上传加载文件并解析入库方法及系统 (Method and system for uploading, analyzing and warehousing loaded files ) 是由 时敏 于 2020-04-21 设计创作，主要内容包括：本发明公开了一种上传加载文件并解析入库方法及系统,涉及数据采集分析领域。该方法包括：步骤1、获取上传后的数据文件,所述数据文件包括由分隔符定义的自定义文件；步骤2、对所述数据文件进行类别识别；步骤3、加载识别后的所述数据文件,并通过可视化界面将加载识别后的所述数据文件与本地数据进行比较；步骤4、根据比较的结果进行判断,并显示判断结果,若所述判断结果为无误,则入库存档。本发明适用于数据传输中,能够解决开发及运维工作量大安全性差的问题,达到保障数据安全,提高对上传文件的正确性的效果。(The invention discloses a method and a system for uploading, analyzing and warehousing a loaded file, and relates to the field of data acquisition and analysis. The method comprises the following steps: step 1, acquiring uploaded data files, wherein the data files comprise custom files defined by delimiters; step 2, identifying the category of the data file; step 3, loading the identified data file, and comparing the loaded identified data file with local data through a visual interface; and 4, judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free. The method and the device are suitable for data transmission, can solve the problems of large workload and poor safety of development and operation and maintenance, and achieve the effects of ensuring data safety and improving the correctness of uploaded files.)

1. A method for uploading, analyzing and warehousing a loaded file is characterized by comprising the following steps:

step 1, acquiring uploaded data files, wherein the data files comprise custom files defined by delimiters;

step 2, identifying the category of the data file;

step 3, loading the identified data file, and comparing the loaded identified data file with local data through a visual interface;

and 4, judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free.

2. The method for uploading, loading, analyzing and warehousing of the claim 1, wherein the data file is subjected to category identification, specifically:

and if the data file is newly added, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

3. The method for uploading the loaded file and analyzing the file as claimed in claim 1, wherein the step of identifying the category of the data file further comprises:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

4. The method for uploading the loaded file and analyzing the loaded file for warehousing according to claim 1, wherein the loading of the identified data file specifically comprises:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

5. The method for uploading the loaded file and analyzing the loaded file into the warehouse according to any one of claims 1 to 5, wherein if the judgment result is that the file is wrong, the file is uploaded again, wherein the uploading times are as follows: finite or infinite.

6. A system for uploading and loading files and analyzing the files in a warehouse is characterized by comprising:

the acquisition module is used for acquiring the uploaded data files, and the data files comprise custom files defined by delimiters;

the identification module is used for carrying out category identification on the data file;

the loading module is used for loading the identified data file and comparing the data file with local data through a visual interface;

and the judging module is used for judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free.

7. The system for uploading the loaded files and analyzing the files for warehousing according to claim 6, wherein the class identification of the data files is specifically as follows:

and if the data file is newly added, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

8. The system of claim 6, wherein the class recognition of the data file further comprises:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

9. The system for uploading the loading files and analyzing the warehousing according to claim 6, wherein the loading of the identified data files specifically comprises:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

10. The system according to any one of claims 6 to 9, wherein if the determination result is that there is an error, the uploaded file is re-uploaded, where the number of times of uploading is: finite or infinite.

Technical Field

The invention relates to the field of data acquisition and analysis, in particular to a method and a system for uploading, analyzing and warehousing a loaded file.

Background

In enterprise financial departments, a large number of manual report files are involved, but based on a fixed template format, the flexible, changeable or self-defined requirements of financial staff are difficult to support, if the financial staff all depend on the IT department to continuously increase new filling templates, time and labor are also very wasted, the visualization and timeliness functions are poor, and meanwhile, the correctness of uploaded files cannot be judged.

Disclosure of Invention

The invention provides a method for uploading, analyzing and warehousing a loaded file, aiming at the defects of the prior art.

The technical scheme for solving the technical problems is as follows: a method for uploading, analyzing and warehousing a loaded file comprises the following steps:

step 1, acquiring uploaded data files, wherein the data files comprise custom files defined by delimiters;

step 2, identifying the category of the data file;

step 3, loading the identified data file, and comparing the loaded identified data file with local data through a visual interface;

and 4, judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free.

The invention has the beneficial effects that: by adopting the separator to define the format and the specification of the self-defined file, the method realizes the purposes of no-code maintenance, flexibility and quickness, simultaneously reduces the development and operation and maintenance work, reduces the offline data transmission, in addition, ensures the data safety because all the data files are uploaded by the user and loaded into the relational database, can timely visually feed back the uploading result through the visual interface, increases the visualization and timeliness functions, and simultaneously improves the judgment on the correctness of the uploaded files.

On the basis of the technical scheme, the invention can be further improved as follows.

Further, performing category identification on the data file, specifically:

and if the data file is newly added, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

The beneficial effect of adopting the further scheme is that: the data can be classified through the establishment of the first database, so that the data can be conveniently searched and called in the future, and meanwhile, the uploading and loading efficiency can be greatly improved.

Further, the class identification of the data file further comprises:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

The beneficial effect of adopting the further scheme is that: through the establishment of the second database, the historical data can be effectively classified and managed, meanwhile, the historical data can be conveniently called, the checking is convenient, and the efficiency is improved.

Further, loading the identified data file specifically comprises:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

Further, if the judgment result is that the error exists, the data is uploaded again, wherein the uploading times are as follows: finite or infinite.

The beneficial effect of adopting the further scheme is that: the data uploading and loading safety can be improved by limiting the uploading times, and meanwhile, the flexibility is also improved.

Another technical solution of the present invention for solving the above technical problems is as follows: a system for uploading and analyzing a loaded file into a warehouse comprises:

the acquisition module is used for acquiring the uploaded data files, and the data files comprise custom files defined by delimiters;

the identification module is used for carrying out category identification on the data file;

the loading module is used for loading the identified data file and comparing the data file with local data through a visual interface;

and the judging module is used for judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free.

Adopt the beneficial effect of above-mentioned scheme: by adopting the separator to define the format and the specification of the self-defined file, the method realizes the purposes of no-code maintenance, flexibility and quickness, simultaneously reduces the development and operation and maintenance work, reduces the offline data transmission, in addition, ensures the data safety because all the data files are uploaded by the user and loaded into the relational database, can timely visually feed back the uploading result through the visual interface, increases the visualization and timeliness functions, and simultaneously improves the judgment on the correctness of the uploaded files.

Further, performing category identification on the data file, specifically:

and if the data file is newly added, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

The beneficial effect of adopting the further scheme is as follows: by adopting the separator to define the format and the specification of the self-defined file, the method realizes the purposes of no-code maintenance, flexibility and quickness, simultaneously reduces the development and operation and maintenance work, reduces the offline data transmission, in addition, ensures the data safety because all the data files are uploaded by the user and loaded into the relational database, can timely visually feed back the uploading result through the visual interface, increases the visualization and timeliness functions, and simultaneously improves the judgment on the correctness of the uploaded files.

Further, performing category identification on the data file, specifically:

and if the data file is newly added, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

The beneficial effect of adopting the further scheme is that: the data can be classified through the establishment of the first database, so that the data can be conveniently searched and called in the future, and meanwhile, the uploading and loading efficiency can be greatly improved.

Further, the class identification of the data file further comprises:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

The beneficial effect of adopting the further scheme is as follows: through the establishment of the second database, the historical data can be effectively classified and managed, meanwhile, the historical data can be conveniently called, the checking is convenient, and the efficiency is improved.

Further, loading the identified data file specifically comprises:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

Further, if the judgment result is that the error exists, the data is uploaded again, wherein the uploading times are as follows: finite or infinite.

The beneficial effect of adopting the further scheme is as follows: the data uploading and loading safety can be improved by limiting the uploading times, and meanwhile, the flexibility is also improved.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

FIG. 1 is a schematic flow chart illustrating a method for uploading a loaded file and parsing the file for storage according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart illustrating a method for uploading a loaded file and parsing the file for storage according to another embodiment of the present invention;

fig. 3 is a structural framework diagram provided in an embodiment of a system for uploading a loaded file and parsing a library according to the present invention.

Detailed Description

The principles and features of this invention are described below in conjunction with the following drawings, which are set forth to illustrate, but are not to be construed to limit the scope of the invention.

As shown in fig. 1, a schematic flow diagram provided by an embodiment of a method for uploading a loaded file and analyzing the loaded file for storage according to the present invention includes:

step 1, acquiring uploaded data files, wherein the data files comprise custom files defined by delimiters; step 2, carrying out category identification on the data file;

step 3, loading the identified data file, and comparing the loaded identified data file with local data through a visual interface;

and 4, judging according to the comparison result, displaying the judgment result, and warehousing and archiving if the judgment result is error-free.

By adopting the separator to define the format and the specification of the self-defined file, the method realizes the purposes of no-code maintenance, flexibility and quickness, simultaneously reduces the development and operation and maintenance work, reduces the offline data transmission, in addition, ensures the data safety because all the data files are uploaded by the user and loaded into the relational database, can timely visually feed back the uploading result through the visual interface, increases the visualization and timeliness functions, and simultaneously improves the judgment on the correctness of the uploaded files.

It should be noted that, in preparing the data file, there are two operation modes: firstly, downloading a system preset data file template, filling data and uploading; the other is to directly upload a prepared data file, which can support file formats such as excel, csv, txt, and the like, and only requires that the data file must define separators, which include common ",", "|", hexadecimal "010 x" symbols, and the like, and are generally defined according to a source file, mainly aiming at ensuring that the file is consistent with the structure of a database table, so that each file column and the field of the database are mapped one by one, which can be arbitrarily defined by a user, and in addition, the category identification is as follows: recognition of suffixes to file formats, such as: ". dat", ". cvs", ". xls", ". txt", etc., furthermore, local data comparison here, firstly, needs to make clear local files, a local file generally refers to a file stored on its local storage medium by a user, and is different from file data uploaded to a server and loaded to a visual interface, the purpose of local data comparison is to check the accuracy and reliability of data after loading, and there are generally the following methods for local files and visual data comparison: 1. the file sizes are consistent, 2, the data record numbers are consistent, the judgment is carried out according to the first two points, and the judgment result is selected as follows: on the basis of meeting the first 2 points, 5% -10% of records are sampled and checked, the attribute value of each field of each record is consistent, and if the attribute values are consistent, the judgment is correct.

Preferably, in any of the above embodiments, the class identification is performed on the data file, specifically:

if the data file is a new type, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

The data can be classified through the establishment of the first database, so that the data can be conveniently searched and called in the future, and meanwhile, the uploading and loading efficiency can be greatly improved.

It should be noted that if the new file is added, the file format is judged first, and the size and the number of lines of the file are counted at the same time; the second is a separator to identify the columns needed to determine the data storage, and then create the database table based on them.

Preferably, in any of the above embodiments, performing category identification on the data file further includes:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

Through the establishment of the second database, the historical data can be effectively classified and managed, meanwhile, the historical data can be conveniently called, the checking is convenient, and the efficiency is improved.

Preferably, in any of the above embodiments, loading the identified data file specifically includes:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

It should be noted that, according to the column names and the column numbers of the sublist definition table, loading data in the file data by a loading algorithm can support loading data in a manner such as oracle sql L oad, and feeding back information such as the data line number and size after loading.

Preferably, in any of the above embodiments, if the determination result is an error, the upload is resumed, where the number of uploads is: finite or infinite.

The data uploading and loading safety can be improved by limiting the uploading times, and meanwhile, the flexibility is also improved.

It should be noted that, as shown in fig. 2, a flow diagram provided by another embodiment of the method for uploading a loaded file and analyzing the uploaded file into a storage is provided, a user logs in a system and enters an operation interface to prepare a data file, and two operation modes are provided, namely, a system preset data file template is downloaded and data uploading is filled, a prepared data file is directly uploaded, a prepared data file can support file formats such as excel, csv, txt and the like, only a separator is required to be defined for the data file, the separator can be arbitrarily defined by the user and is judged to be a new addition or a history selection, if the history is uploaded, a template or a record which is uploaded in the history is selected, if the separator is not required to be operated, the data file is clicked and uploaded and then enters a background operation flow, the system firstly judges a new addition identifier, if the new addition is made, firstly judges the file format and counts the file size and the number of lines, secondly, the separator is used for identifying and judging columns required for data storage, then data retransmission is created, if the history is a corresponding data base table is selected according to the selected template, the column definition of the subcolumn, the column name of the loaded file, the column definition, the number of the column is compared with the number of the local loading, and the number of the supported by an algorithm L, and if the data is wrongly checked, the data retransmission is carried out, and the load is carried out.

As shown in fig. 3, a structural framework diagram provided by an embodiment of uploading a loading file and parsing a warehousing system includes:

the acquisition module 100 is configured to acquire an uploaded data file, where the data file includes a custom file defined by a delimiter;

the identification module 200 is used for performing category identification on the data file;

the loading module 300 is configured to load the identified data file, and compare the loaded identified data file with local data through a visual interface;

the judging module 400 is configured to perform judgment according to the comparison result, display the judgment result, and store in a warehouse and archive if the judgment result is error-free.

By adopting the separator to define the format and the specification of the self-defined file, the method realizes the purposes of no-code maintenance, flexibility and quickness, simultaneously reduces the development and operation and maintenance work, reduces the offline data transmission, in addition, ensures the data safety because all the data files are uploaded by the user and loaded into the relational database, can timely visually feed back the uploading result through the visual interface, increases the visualization and timeliness functions, and simultaneously improves the judgment on the correctness of the uploaded files.

Preferably, in any of the above embodiments, the class identification is performed on the data file, specifically:

if the data file is a new type, counting the size and the line number of the data file according to the format of the data file, and creating a first database table corresponding to the separator of the data file according to the separator of the data file.

The data can be classified through the establishment of the first database, so that the data can be conveniently searched and called in the future, and meanwhile, the uploading and loading efficiency can be greatly improved.

Preferably, in any of the above embodiments, performing category identification on the data file further includes:

and if the data file is of a history type, selecting a second database table corresponding to the template which is uploaded by the data file.

Through the establishment of the second database, the historical data can be effectively classified and managed, meanwhile, the historical data can be conveniently called, the checking is convenient, and the efficiency is improved.

Preferably, in any of the above embodiments, loading the identified data file specifically includes:

and carrying out loading identification on the column name or the column number of the first database table or the second database table through a loading algorithm.

Preferably, in any of the above embodiments, if the determination result is an error, the upload is resumed, where the number of uploads is: finite or infinite.

The data uploading and loading safety can be improved by limiting the uploading times, and meanwhile, the flexibility is also improved.

It is understood that some or all of the alternative embodiments described above may be included in some embodiments.

It should be noted that the above embodiments are product embodiments corresponding to the previous method embodiments, and for the description of each optional implementation in the product embodiments, reference may be made to corresponding descriptions in the above method embodiments, and details are not described here again.

The reader should understand that in the description of this specification, reference to the description of the terms "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.