阅读说明：本技术 一种高性能的数据湖系统及数据存储方法 (High-performance data lake system and data storage method ) 是由 刘昊 屠勇刚 张涛 陈志凌 郁晨熹 陈巍 刘哲峰 刘荫隆 王秋烨 王朋 于 2021-11-18 设计创作，主要内容包括：本发明提供了一种高性能的数据湖系统及数据存储方法,数据存储方法包括以下步骤：S1.将文件转成文件流；S2.将文件流转成一个数组嵌套多个子数组的形式；S3.将所述的转成RDD后存储至数据湖的存储层中。本方案提出嵌套的字段结构,为出湖过程的并行处理提供字段结构基础,有效提高出湖性能,并且根据硬件内核灵活生成嵌套的子数组数量,使数据湖具有更强的扩展性能,针对不同的用户均能够保持最佳的入湖效率,满足用户需求。(The invention provides a high-performance data lake system and a data storage method, wherein the data storage method comprises the following steps: s1, converting a file into a file stream; s2, converting the file flow into a form that an array is nested into a plurality of sub-arrays; and S3, converting the RDD into the RDD and storing the RDD into a storage layer of a data lake. The nested field structure is provided, a field structure foundation is provided for parallel processing in the lake outlet process, the lake outlet performance is effectively improved, the nested subarray number is flexibly generated according to the hardware kernel, the data lake has stronger expansion performance, the optimal lake inlet efficiency can be kept for different users, and the user requirements are met.)

1. A data storage method of a data lake system is characterized by comprising the following steps:

s1, converting a file into a file stream;

s2, converting the file flow into a form that an array is nested into a plurality of sub-arrays;

and S3, converting the array into RDD and storing the RDD in a storage layer of a data lake.

2. The data storage method of the data lake system of claim 1, wherein in step S2, the number of the array nested subarrays is 2-8.

3. The data storage method of the data lake system of claim 2, wherein in step S2, the number of the array nested subarrays is consistent with the number of CPU cores in the current hardware environment.

4. The data storage method of the data lake system of claim 3, wherein the system dynamically determines the corresponding number of subarrays by reading a configuration file to obtain the number of CPU cores in the current hardware environment.

5. The data storage method of the data lake system of claim 4, further comprising between steps S2 and S3:

detecting the size of the current array, and if the size of the current array is within a set value, storing the current array in a single-row form; if the number of the current array exceeds the set value, the current array is segmented and stored in rows, and the array size of each row is smaller than the set value.

6. The data storage method of the data lake system of claim 5, wherein when the array size is greater than the set value, the number of rows n is n = m/s +1, where m denotes the array size, s denotes the set value, and/denotes the integer division quotient.

7. The data storage method of the data lake system of claim 6, wherein the array that is greater than the set value is divided into a plurality of segments and then returned to the iterator that comprises a plurality of RDDs.

8. The data storage method of the data lake system of claim 7, further comprising by polling an automated storage method:

A1. polling the monitored files, acquiring absolute paths of all files under the specified path in each polling, and putting the absolute paths into an array as the latest offset;

A2. taking the result of the last polling, and making a difference set by the two arrays;

A3. the absolute path of the newly added file in this polling is obtained from the difference set result, and the newly added file is then stored in the data lake through steps S1-S3.

9. The data storage method of the data lake system of claim 8, wherein the system rewrites the batch read and the batch write to the data lake system by inheriting a FileFormat class so that the data lake system supports the batch read and the batch write.

10. A high performance data lake system, characterized in that data storage is performed using the storage method of any one of claims 1 to 9.

Technical Field

The invention belongs to the technical field of lake entering and exiting of data, and particularly relates to a high-performance data lake system and a data storage method.

Background

A data lake is a large warehouse for storing a wide variety of raw data of an enterprise, where the data is accessible, processed, analyzed, and transmitted. One part of the value of the data lake is to gather different kinds of data together, and the other part of the value is to perform data analysis without a predefined model.

The data lake converts the format of a file and stores the file as a queue supported by the data lake, specifically, as shown in fig. 1, the data lake converts the file needing to enter the lake into a file stream by using an api interface, then converts the file stream into a byte array, and converts the byte array into an RDD, where the RDD includes original data (byte array) and metadata (file name, size, etc.) of the file, and the RDD is stored in the queue format by using the data lake bottom code, so that the data conforms to the data lake format. Applicants have discovered during use of current data lake systems that as the documents grow in size, the rate at which documents are read from the lake grows in multiples, severely affecting the performance of the data lake.

In order to solve the technical problems, the applicant consults a large amount of data and does not inquire a scheme capable of effectively solving the performance of the lake-out of the data. Through a great deal of research, the applicant finds that in some non-relational databases, one field can contain a plurality of fields corresponding to a plurality of field types, while all fields of the current data lake only correspond to one field type when the file stream is converted into an array.

Disclosure of Invention

The invention aims to solve the problems and provide a high-performance data lake system;

another object of the present invention is to solve the above problems and provide a data storage method that can help improve the performance of a data lake system.

In order to achieve the purpose, the invention adopts the following technical scheme:

a data storage method of a data lake system comprises the following steps:

s1, converting a file into a file stream;

s2, converting the file flow into a form that an array is nested into a plurality of sub-arrays;

and S3, converting the RDD into the RDD and storing the RDD into a storage layer of a data lake.

In the data storage method of the data lake system, in step S2, the number of the array nested subarrays is 2-8.

In the data storage method of the data lake system, in step S2, the number of the array nested child arrays is consistent with the number of CPU cores in the current hardware environment.

In the data storage method of the data lake system, the system obtains the number of the CPU cores in the current hardware environment in a mode of reading the configuration file so as to dynamically determine the corresponding number of the sub-arrays.

In the data storage method of the data lake system, between steps S2 and S3, the method further includes:

detecting the size of the current array, and if the size of the current array is within a set value, storing the current array in a single-row form; if the number of the current array exceeds the set value, the current array is segmented and stored in rows, and the array size of each row is smaller than the set value.

In the data storage method of the data lake system, when the array size is larger than the set value, the number n of the lines is n = m/s +1, where m represents the array size, s represents the set value, and/represents the integer division quotient.

In the data storage method of the data lake system, the array which is larger than the set value is cut into a plurality of arrays and then returns to the iterator which comprises a plurality of RDDs.

In the data storage method of the data lake system, the method further comprises the following steps:

A1. polling the monitored files, acquiring absolute paths of all files under the specified path in each polling, and putting the absolute paths into an array as the latest offset;

A2. taking the result of the last polling, and making a difference set by the two arrays;

A3. the absolute path of the newly added file in this polling is obtained from the difference set result, and the newly added file is then stored in the data lake through steps S1-S3.

In the data storage method of the data lake system, the system rewrites batch reading and batch writing to the data lake system by inheriting a FileFormat class so that the data lake system supports batch reading and batch writing.

A high-performance data lake system stores data by adopting the storage method.

The invention has the advantages that:

a nested field structure is provided, a field structure foundation is provided for parallel processing in the lake outlet process, and the lake outlet performance is effectively improved;

the nesting and multi-line storage mode improves the storage capacity of large files and the upper limit of storage;

the newly added file is actively stored in the data lake in a polling monitoring mode, so that the stream processing of the newly added file is realized, and the storage performance of the data lake is improved;

the number of nested subarrays is flexibly generated according to the hardware kernel, so that the data lake has stronger expansion performance, the optimal lake entering efficiency can be kept for different users, and the user requirements are met;

the array is cut into a plurality of arrays according to the specified size and then returned through the iterator without using list return, so that outofmemoryreror (stack memory overflow generation) errors can be effectively avoided.

Drawings

FIG. 1 is a flow chart of document lake entry;

FIG. 2 is a graph of the results of a single txt file write-out time-consuming experiment in the existing lake access system;

FIG. 3 is a graph of the results of a single pdf file write-in and write-out time-consuming experiment of the existing lake access system;

FIG. 4 is a graph of the results of a single mp4 file write-out time-consuming experiment in a conventional lake access system;

FIG. 5 is a graph of the results of a single written-out time-consuming experiment of a single gz file in a conventional lake access system;

FIG. 6 is a graph showing the result of a time-consuming experiment performed by writing a single txt file in the lake entering and exiting system according to the present embodiment;

FIG. 7 is a graph showing the experimental results of reading out a single txt file of the lake entering and exiting system according to the present embodiment;

FIG. 8 is a graph showing the result of a single pdf file writing experiment in the lake entering and exiting system;

FIG. 9 is a graph showing the experimental results of the time spent reading a single pdf file in the lake or lake entering system according to the present embodiment;

FIG. 10 is a graph showing the result of an experiment of the time spent by writing a single mp4 file in the lake and entrance system according to the scheme;

FIG. 11 is a graph showing the experimental results of the reading time of a single mp4 file in the lake and entrance system;

FIG. 12 is a graph showing the result of a test of the time taken for writing a single gz file in the lake entering and exiting system;

fig. 13 is a graph showing the experimental results of the reading time of a single gz file of the lake entering and exiting system in the scheme.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

The embodiment discloses a data storage method of a data lake system, which comprises the following steps:

s1, converting a file into a file stream;

s2, converting the file flow into a form that an array is nested into a plurality of sub-arrays;

and S3, converting the array into RDD, and storing the RDD into a storage layer of the data lake after the RDD meets the request of the parquet format of the data lake.

Preferably, the number of the cores of the CPU in the current hardware environment is obtained by reading the configuration file, so as to dynamically determine the corresponding number of the nests, specifically, the file stream is converted into an array in which the number of the array nested sub-arrays is consistent with the number of the cores of the CPU in the current hardware environment. If the array form of a file stream converted in a non-nested manner is [1,2,3,4,5,6,7,8], then if the number of CPU cores in the current hardware environment is 2 cores, the array form converted in the nested manner in the scheme is a form of nesting two sub-arrays, namely [ [1,2,3,4], [5,6,7,8] ]; if the number of the CPU cores in the current hardware environment is 4 cores, the array form after the nested mode conversion in the scheme is a nested four-sub-word array form, namely [ [1,2], [3,4] [5,6], [7,8] ]. And so on for other core numbers. When the number of the nested arrays is consistent with that of the CPU cores, the fastest reading speed can be obtained, and the nested arrays can be matched with the CPU cores in the self-adaptive mode, so that the optimal reading speed is kept, and the lake outlet performance is effectively improved.

Preferably, the present embodiment further includes, between steps S2 and S3:

detecting the size of the current array, and if the size of the current array is within a set value, storing the current array in a single-row form; if the number of the current array exceeds the set value, the current array is segmented and stored in rows, and the array size of each row is smaller than the set value. And when the size of the array is larger than a set value, the number n of the lines is n = m/s +1, wherein m represents the size of the array, s represents the set value, and/represents the integer division quotient. If the size of the array is 50Mb and the set value is 60Mb, n =50/60+1=0+1=1;

if the array size is 100Mb and the set value is 60Mb, n =100/60+1=1+1= 2. When the data is divided into a plurality of rows, the size of each row in the previous rows can be set, the last row is the residual bytes of the current array, each row can be distributed evenly, the data can be distributed freely according to the array condition, and for example, the bytes belonging to the same sub-array can be preferably arranged in the same row.

And further, after the array which is larger than the set value is cut into a plurality of arrays, returning to the iterator comprising a plurality of RDDs. The array is cut into a plurality of arrays according to the specified size and then returned through the iterator without using list return, so that outofmemoryreror (stack memory overflow generation) errors can be effectively avoided.

Further, the method also comprises a polling automatic storage method, which is mainly used for monitoring whether the files in a certain folder are newly added or not in a code level in a polling (polling) mode, and if the files are newly added, the newly added files are stored in a data lake, and the specific process is as follows:

A1. in each polling, obtaining absolute paths of all files under the specified path, and putting the absolute paths into an array as the latest offset;

A2. taking the result of the last polling, and making a difference set by the two arrays;

A3. the absolute path of the newly added file in this polling is obtained from the difference set result, and the newly added file is then stored in the data lake through steps S1-S3.

According to the scheme, the folder is monitored in a polling mode, the newly added files can be actively added into the data lake, after the data lake is connected with the data source, a user starts a polling automatic storage function aiming at the data source, the newly added files do not need to be manually added by the user, the actual service requirements are met, and the storage performance of the data lake is further improved.

Furthermore, the method also realizes batch reading and batch writing functions by inheriting FileFormat class to rewrite build reader and prepareWrite to the data lake system.

The build reader is similar to that of the existing spark company, the RDD format types returned to the data lake after being stored in the data lake bottom layer by virtue of the partial are different, the spark returns to a single RDD, a plurality of iterators are returned in the scheme, the plurality of RDDs are included, a small part is returned every time, the computer returns to the next part for processing after processing the current part, and OutOfMemoryError can be effectively avoided.

The prepareWrite implementation method comprises the steps of obtaining a plurality of related arrays, splicing the related arrays into an array, and outputting the array through an output stream.

The embodiment also provides a high-performance data lake system for storing data by adopting the storage method.

In order to verify the superiority of the data storage method of the data lake system, the comparison of the lake entering and exiting experiments is carried out on the data lake system stored based on the method and the existing commonly stored data lake system:

the experimental environment is as follows:

hardware environment, li 710 core 8, l16G memory;

software environment, lhandoop pseudo-distributed storage, lubuntu 18.04;

JVM Environment, l-Xms2048m (initial heap size), l-Xmx4096m (maximum heap size).

Part of the test data:

single write-out test for files of different formats:

text files (txt, pdf); video files (mp 4); compressed file (gz)

The real experimental data are adopted at this time, files generated randomly by codes are not adopted, writing of each group of data is repeated for 3-5 times, and the average value is taken as a result.

Fig. 2-13 show some experimental results, and fig. 2-5 show experimental results of the lake access system of the conventional storage method of the prior art, wherein,

FIG. 2 shows the write-out time of a single txt file, which is to say that when the write-out file reaches 356, the write-out time is 143347ms, and the write-out time is 13240 ms.

Fig. 3 shows the write-out time of a single pdf file, and it can be seen that, when the write-out file reaches 200.8MB, the write-out time is 78663ms, and the write-out time is 8796 ms.

FIG. 4 shows that writing out of a single mp4 file takes 44242ms and 11003ms, when writing out of a file reaches 391 MB.

FIG. 5 shows write-out time for a single gz file, which is 309683ms and 9754ms when the write-out file reaches 325 MB.

Fig. 6-13 are graphs showing experimental results of lake inlet and outlet experiments of the lake inlet and outlet system adopting the storage mode of the scheme, wherein,

FIG. 6 shows the write time of a single txt file, and it can be seen that when the write file reaches 356, the write time of the array structure with the number of nested child arrays of 2, 4, 6 and 8 needs 10417ms, 11328ms, 10636ms and 9978ms, respectively. Comparing with fig. 2, it can be seen that the writing speed is slightly faster than the ordinary data lake system.

FIG. 7 shows the read time of a single txt file, and it can be seen that when the read file reaches 356, the read time of the array structure with the number of nested sub-arrays of 2, 4, 6 and 8 respectively needs 42439ms, 14849ms, 12103ms and 9868 ms. As can be seen by comparing fig. 2, the read-out speed for each nested number case is much higher than that of the ordinary data lake system.

Fig. 8 shows the write time of a single pdf file, and it can be seen that when the write file reaches 200.8Mb, the write time of the array structure with the number of nested subarrays of 2, 4, 6 and 8 needs 7664ms, 77107ms, 7038ms and 7307ms, respectively. Comparing with fig. 3, it can be seen that the writing speed is slightly faster than the ordinary data lake system.

FIG. 9 shows the read time of a single pdf file, and it can be seen that the read time of the array structure with the number of nested subarrays of 2, 4, 6 and 8 is 22114ms, 7724ms, 5702ms and 6549ms respectively when the read file reaches 200.8 Mb. As can be seen by comparison with fig. 3, the read-out speed for each nested number case is much improved over the normal data lake system.

FIG. 10 shows the write time of a single mp4 file, and it can be seen that when the write file reaches 391Mb, the write time of the array structure with the number of nested sub-arrays of 2, 4, 6 and 8 respectively needs 11732ms, 9702ms, 10861ms and 11008 ms. Comparing with fig. 4, it can be seen that the writing speed is close to that of the ordinary data lake system.

FIG. 11 shows that reading time of a single mp4 file is consumed, and it can be seen from the figure that reading time of array structures with nested subarray numbers of 2, 4, 6 and 8 is required to be 114045ms, 33027ms, 16389ms and 14851ms respectively when the file is read to 391 Mb. As can be seen by comparing fig. 4, the read-out speed is slower for the number of nests 2 than for the normal data lake system, but the read-out speed is significantly faster for the remaining number of nests 4, 6 and 8 than for the normal data lake system, especially for the number of nests 8.

FIG. 12 shows the write time of a single gz file, and it can be seen that the write time of the array structure with the number of nested subarrays of 2, 4, 6, and 8 needs 10002ms, 9415ms, 8633ms, and 8417ms, respectively, when the write file reaches 325 Mb. Comparing with fig. 5, it can be seen that the writing speed is close to that of the ordinary data lake system.

FIG. 13 shows the read time of a single gz file, and it can be seen that when the read file reaches 325Mb, the read time of the array structure with the number of nested sub-arrays of 2, 4, 6 and 8 is 81297ms, 25790ms, 12503ms and 9386ms respectively. Comparing with fig. 5, it can be seen that the read-out speed for each nested number case is much improved over the normal data lake system.

Compared with the data lake system in the prior art, the data lake system stored by adopting the scheme has slightly improved lake entering performance, but can greatly improve lake exiting performance, so that the scheme can effectively improve the lake exiting performance of the data lake. In addition, the experimental environment of this experiment is an 8-core kernel, so the array structure with the number of nesting 8 has the highest in-and-out performance, as is apparent from the comparison of fig. 6-13 with the numbers of nesting 2, 4 and 6, respectively.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Although the terms array, storage layer, filestream, nesting, iterator, etc. are used more herein, the possibility of using other terms is not excluded. These terms are used merely to more conveniently describe and explain the nature of the present invention; they are to be construed as being without limitation to any additional limitations that may be imposed by the spirit of the present invention.