阅读说明：本技术 数据处理方法、装置、电子设备及存储介质 (Data processing method and device, electronic equipment and storage medium ) 是由 郑宁 王晋强 蔡雪梅 于 2021-09-03 设计创作，主要内容包括：本申请提供一种数据处理方法、装置、电子设备及存储介质,该方法中,响应于用户的数据处理请求,该数据处理请求携带有目标页面的标识,并根据目标页面的标识,从列存文件的脚注部分中确定出目标页面中的当前空间,该脚注部分包括：至少一个页面的当前空间的标识和时间戳,之后对当前空间中的第一数据进行数据处理,得到第二数据,最后将第二数据存储至当前空间对应的对偶空间,该当前空间对应的对偶空间为新的当前空间,对偶空间为备用存储第一数据的空间。该技术方案中,从当前空间和对偶空间出发,利用两个存储空间数据的交互,从而实现了列式存储下数据进行增删改等处理时步骤繁琐且效率较低的问题。(The application provides a data processing method, a data processing device, an electronic device and a storage medium, wherein in the method, a data processing request of a user is responded, the data processing request carries an identifier of a target page, and a current space in the target page is determined from a footnote part of a column storage file according to the identifier of the target page, and the footnote part comprises the following steps: and finally, storing the second data to a dual space corresponding to the current space, wherein the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data. According to the technical scheme, the interaction of data in two storage spaces is utilized from the current space and the dual space, so that the problems of complicated steps and low efficiency when data are subjected to processing such as increasing, deleting, modifying and the like under column type storage are solved.)

1. A data processing method is applied to an electronic device, and the method comprises the following steps:

responding to a data processing request of a user, wherein the data processing request is used for requesting to process first data of a target page in a list storage file, and the data processing request carries an identifier of the target page;

according to the identification of the target page, determining the current space in the target page from the footnote part of the column storage file, wherein the footnote part comprises: an identification of a current space of at least one page, the current space being a space where first data is validly stored, and a timestamp being a time of a last time the footnote part was updated;

performing data processing on the first data in the current space to obtain second data, wherein the data processing comprises: deleting data in the first data and/or updating data in the first data;

and storing the second data to a dual space corresponding to the current space, wherein the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data.

2. The method of claim 1, wherein prior to said responding to the user's data processing request, the method further comprises:

acquiring data to be processed;

dividing the data to be processed according to a preset storage structure to obtain a plurality of first data, wherein the preset storage structure comprises: a plurality of page line groups, a plurality of columns corresponding to the page line groups, and the footer portion, each column in each page line group comprising: at least one page, each page comprising: the current space and a dual space corresponding to the current space;

respectively storing the plurality of first data into current spaces corresponding to the pages;

storing the identification and the time stamp of the current space of each page to the footnote part;

and determining the column memory file according to each page for storing the plurality of first data and the footnote part.

3. The method of claim 1 or 2, wherein the data processing further comprises: adding target data into the first data, wherein the target data are data which are required to be added into the file in the list storage by a user;

correspondingly, before the obtaining the second data, the method further includes:

acquiring a space required by the target data, an unstored space in the current space and an unstored space in a dual space corresponding to the current space;

judging whether the space required by the target data meets a preset condition or not according to the space required by the target data, the non-storage space in the current space and the non-storage space in the dual space corresponding to the current space, wherein the preset condition comprises any one of the following conditions: the space required by the target data is less than or equal to the non-storage space in the current space, and the space required by the target data and the first data is less than or equal to the non-storage space in the dual space corresponding to the current space;

and if the space required by the target data meets a preset condition, performing the processing operation on the first data in the current space.

4. The method according to claim 3, wherein after the determining whether the space required by the target data meets a preset condition according to the space required by the target data, the non-storage space in the current space, and the non-storage space in the dual space corresponding to the current space, the method further comprises:

if the space required by the target data does not meet the preset condition, acquiring first data of the current space of each page in the column storage file;

and determining data to be processed according to the target data and the first data of the current space of each page in the column storage file.

5. The method according to claim 1 or 2, wherein after the storing the second data to the dual space corresponding to the current space to obtain a new current space, the method further comprises:

clearing first data of the current space, and setting the current space as a new dual space corresponding to the new current space;

and updating the identifier and the timestamp of the current space corresponding to the target page in the footnote part according to the identifier and the timestamp of the new current space and the identifier of the target page corresponding to the new current space.

6. The method of claim 4, wherein the current space and the dual space are structurally the same, and wherein the current space or the dual space comprises:

a header, an effective data space and the non-storage space, wherein the effective data space is a space in which data has been stored, and the header includes: the effective data starting address of the data to be processed, the effective data length, the sizes of the effective data space and the non-storage space, and other metadata.

7. A data processing apparatus, applied to an electronic device, the apparatus comprising: the device comprises a response module, a determination module, a processing module and a storage module;

the response module is used for responding to a data processing request of a user, wherein the data processing request is used for requesting to process first data of a target page in a file in a list, and the data processing request carries an identifier of the target page;

the determining module is configured to determine, according to the identifier of the target page, a current space in the target page from a footer portion of the list storage file, where the footer portion includes: an identification of a current space of at least one page, the current space being a space where first data is validly stored, and a timestamp being a time of a last time the footnote part was updated;

the processing module is configured to perform data processing on the first data in the current space to obtain second data, where the data processing includes: deleting data in the first data and/or updating data in the first data;

the storage module is configured to store the second data to a dual space corresponding to the current space, where the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data.

8. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored by the memory to implement the data processing method of any of claims 1-6.

9. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, are configured to implement the data processing method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program realizes the data processing method of any one of claims 1-6 when executed by a processor.

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

With the popularization and development of the internet and the technology of the internet of things, the amount of data to be processed by an application program is almost exponentially increased, so how to efficiently process massive data in the application of a database becomes a problem to be solved urgently.

In the prior art, a columnar storage method is generally adopted. Compared with the storage according to rows, the data storage method can improve the compression rate of the data, reduce the occupation of storage space, accelerate the data analysis process by orders of magnitude and powerfully support the development of services.

However, in the above method using columnar storage, data is arranged closely in columns, and the data structure is almost invariable, so that the processing steps for adding and deleting data are complicated and the efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data processing method and device, electronic equipment and a storage medium, and aims to solve the problems of complicated steps and low efficiency when processing such as adding, deleting, modifying and the like is carried out on data stored in a column mode.

In a first aspect, an embodiment of the present application provides a data processing method applied to an electronic device, where the method includes:

responding to a data processing request of a user, wherein the data processing request is used for requesting to process first data of a target page in a list storage file, and the data processing request carries an identifier of the target page;

according to the identification of the target page, determining the current space in the target page from the footnote part of the column storage file, wherein the footnote part comprises: an identification of a current space of at least one page, the current space being a space where first data is validly stored, and a timestamp being a time of a last time the footnote part was updated;

performing data processing on the first data in the current space to obtain second data, wherein the data processing comprises: deleting data in the first data and/or updating data in the first data;

and storing the second data to a dual space corresponding to the current space, wherein the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data.

In a possible design of the first aspect, before the responding to the data processing request of the user, the method further includes:

acquiring data to be processed;

dividing the data to be processed according to a preset storage structure to obtain a plurality of first data, wherein the preset storage structure comprises: a plurality of page line groups, a plurality of columns corresponding to the page line groups, and the footer portion, each column in each page line group comprising: at least one page, each page comprising: the current space and a dual space corresponding to the current space;

respectively storing the plurality of first data into current spaces corresponding to the pages;

storing the identification and the time stamp of the current space of each page to the footnote part;

and determining the column memory file according to each page for storing the plurality of first data and the footnote part.

In another possible design of the first aspect, the data processing further includes: adding target data into the first data, wherein the target data are data which are required to be added into the file in the list storage by a user;

correspondingly, before the obtaining the second data, the method further includes:

acquiring a space required by the target data, an unstored space in the current space and an unstored space in a dual space corresponding to the current space;

judging whether the space required by the target data meets a preset condition or not according to the space required by the target data, the non-storage space in the current space and the non-storage space in the dual space corresponding to the current space, wherein the preset condition comprises any one of the following conditions: the space required by the target data is less than or equal to the non-storage space in the current space, and the space required by the target data and the first data is less than or equal to the non-storage space in the dual space corresponding to the current space;

and if the space required by the target data meets a preset condition, performing the processing operation on the first data in the current space.

In this possible design, after determining whether the space required by the target data meets a preset condition according to the space required by the target data, the non-storage space in the current space, and the non-storage space in the dual space corresponding to the current space, the method further includes:

if the space required by the target data does not meet the preset condition, acquiring first data of the current space of each page in the column storage file;

and determining data to be processed according to the target data and the first data of the current space of each page in the column storage file.

In yet another possible design of the first aspect, after the storing the second data to the dual space corresponding to the current space to obtain a new current space, the method further includes:

clearing first data of the current space, and setting the current space as a new dual space corresponding to the new current space;

and updating the identifier and the timestamp of the current space corresponding to the target page in the footnote part according to the identifier and the timestamp of the new current space and the identifier of the target page corresponding to the new current space.

Optionally, the structures of the current space and the dual space are the same, and the structure of the current space or the structure of the dual space includes:

a header, an effective data space and the non-storage space, wherein the effective data space is a space in which data has been stored, and the header includes: the effective data starting address of the data to be processed, the effective data length, the sizes of the effective data space and the non-storage space, and other metadata.

In a second aspect, an embodiment of the present application provides a data processing apparatus, which is applied to an electronic device, and the apparatus includes: the device comprises a response module, a determination module, a processing module and a storage module;

the response module is used for responding to a data processing request of a user, wherein the data processing request is used for requesting to process first data of a target page in a file in a list, and the data processing request carries an identifier of the target page;

the determining module is configured to determine, according to the identifier of the target page, a current space in the target page from a footer portion of the list storage file, where the footer portion includes: an identification of a current space of at least one page, the current space being a space where first data is validly stored, and a timestamp being a time of a last time the footnote part was updated;

the processing module is configured to perform data processing on the first data in the current space to obtain second data, where the data processing includes: deleting data in the first data and/or updating data in the first data;

the storage module is configured to store the second data to a dual space corresponding to the current space, where the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data.

In one possible design of the second aspect, the apparatus further includes: an acquisition module;

the acquisition module is used for acquiring data to be processed before responding to a data processing request of a user;

the processing module is further configured to divide the data to be processed according to a preset storage structure to obtain a plurality of first data, where the preset storage structure includes: a plurality of page line groups, a plurality of columns corresponding to the page line groups, and the footer portion, each column in each page line group comprising: at least one page, each page comprising: the current space and a dual space corresponding to the current space;

the storage module is further configured to: the first data are respectively stored in the current space corresponding to each page, and the identification and the time stamp of the current space of each page are stored in the footnote part;

the determining module is further configured to determine the column file according to each page storing the plurality of first data and the footer portion.

In another possible design of the second aspect, the data processing further includes: adding target data into the first data, wherein the target data are data which are required to be added into the file in the list storage by a user;

correspondingly, before the second data is obtained, the obtaining module is further configured to obtain a space required by the target data, an unstored space in the current space, and an unstored space in a dual space corresponding to the current space;

the processing module is further configured to determine whether the space required by the target data meets a preset condition according to the space required by the target data, the non-storage space in the current space, and the non-storage space in the dual space corresponding to the current space, where the preset condition includes any one of: the space required by the target data is less than or equal to the non-storage space in the current space, and the space required by the target data and the first data is less than or equal to the non-storage space in the dual space corresponding to the current space; and when the space required by the target data meets a preset condition, performing the processing operation on the first data in the current space.

In this possible design, the processing module is further configured to:

when the space required by the target data does not meet the preset condition, acquiring first data of the current space of each page in the column storage file;

and determining data to be processed according to the target data and the first data of the current space of each page in the column storage file.

In yet another possible design of the second aspect, the processing module is further configured to:

clearing first data of the current space, and setting the current space as a new dual space corresponding to the new current space;

and updating the identifier and the timestamp of the current space corresponding to the target page in the footnote part according to the identifier and the timestamp of the new current space and the identifier of the target page corresponding to the new current space.

Optionally, the structures of the current space and the dual space are the same, and the structure of the current space or the structure of the dual space includes:

a header, an effective data space and the non-storage space, wherein the effective data space is a space in which data has been stored, and the header includes: the effective data starting address of the data to be processed, the effective data length, the sizes of the effective data space and the non-storage space, and other metadata.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory;

the memory stores computer-executable instructions;

the processor executes the computer-executable instructions to cause the electronic device to perform the data processing method as described in the first aspect and various possible designs above.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the data processing method as described in the first aspect and various possible designs.

In a fifth aspect, embodiments of the present application provide a computer program product comprising a computer program, which when executed by a processor, is configured to implement the data processing method as described in the first aspect and various possible designs.

In the method, a data processing request of a user is responded, the data processing request carries an identifier of a target page, and a current space in the target page is determined from a footer part of a list storage file according to the identifier of the target page, wherein the footer part comprises: the method comprises the steps of identifying a current space of at least one page and a timestamp, wherein the current space is a space for effectively storing first data, then carrying out data processing on the first data in the current space to obtain second data, and finally storing the second data to a dual space corresponding to the current space, wherein the dual space corresponding to the current space is a new current space, and the dual space is a space for storing the first data in a standby mode. According to the technical scheme, the interaction of information between the two storage spaces is ingeniously utilized from the current space and the dual space, so that the problems of complicated steps and low efficiency when processing such as increasing, deleting, modifying and the like of data under column type storage are realized.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a schematic flowchart of an embodiment of a data processing method according to an embodiment of the present application;

fig. 2 is a schematic diagram illustrating division of data to be processed according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a page provided by an embodiment of the present application;

FIG. 4 is a schematic view of a footnote provided in accordance with an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating the transformation of the current space and the dual space provided by an embodiment of the present application;

fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

Before introducing the embodiments of the present application, the background of the present application is explained first:

with the popularization of the internet and internet of things technology, the amount of data to be processed by the application is almost exponentially increased. In database applications, a column-type storage method is generally used to efficiently process mass data. Compared with the storage according to rows, the data storage method can improve the compression rate of the data, reduce the occupation of storage space, accelerate the data analysis process by orders of magnitude and powerfully support the development of services.

In conventional columnar storage, data is closely arranged in columns, in which arrangement the data is almost immutable. Although this arrangement is beneficial to writing a scene that is read once and many times, it is difficult to support operations such as data insertion, update or deletion, which brings some additional limitations and inconveniences to real-world production.

In order to solve the technical problems, the technical conception process of the inventor is as follows: after the storage device with transparent compression in the prior art stores data according to columns, because the data are closely arranged, if the data can be divided into different blocks, when the data needs to be modified, deleted and added with new data, all information in the block where the data is located is extracted, the corresponding operation is carried out, the data is stored into another dual block, and after the storage is successful, the original block is removed, so that the problem of inconvenient operation in the prior art can be solved, and the problem of high occupied space of the data is avoided.

The technical solution of the present application will be described in detail by specific examples. It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 1 is a schematic flowchart of an embodiment of a data processing method according to an embodiment of the present application. As shown in fig. 1, the data processing method includes the steps of:

and step 11, responding to a data processing request of a user.

The data processing request is used for requesting to process first data of a target page in the list storage file, and the data processing request carries an identifier of the target page;

in the scheme, the data processing method is applied to the electronic equipment and can be a transparent compressed storage device.

In this step, the user needs to process the data stored in the transparent compressed storage device, and when the user triggers the data processing operation, from the perspective of the transparent compressed storage device, the user needs to respond to a data processing request of the user, where the request carries an identifier of a target page, that is, the request is for processing the data in the target page in the stored data.

Optionally, before this step, a memory file needs to be established, and fig. 2 is a schematic diagram of dividing the to-be-processed data provided in the embodiment of the present application, where the process may include the following steps:

step 1, acquiring data to be processed.

The data to be processed is data which needs to be stored in a local cache of the storage device.

And step 2, dividing the data to be processed according to a preset storage structure to obtain a plurality of first data, wherein the preset storage structure comprises: the page line groups are provided with a plurality of columns and a footer part, and each column in each page line group comprises: at least one page, each page comprising: the current space and the dual space corresponding to the current space.

As shown in fig. 2, the data to be processed is divided into line groups, which may be page line group 0, page line group 1, page line group 2, … … page line group m (positive integer), for example, the data may be information of 10000 students, and m is 100, so that each page line group contains information of 100 students (it should be understood that each page line group may contain more or less number, and is not necessarily evenly distributed according to the number of page line groups), and each page line group may contain at least one page. Further, when stored in columns, there may be column 1, column 2, … …, column n, for example, where n is 3, it may be name, gender, and school number, and each page row group includes 3 pages.

The plurality of page line groups form a line group, and the number of the line groups is not limited in the embodiment of the present application.

Optionally, fig. 3 is a schematic view of a page provided in the embodiment of the present application, and as shown in fig. 3, the page includes: current space and dual space, wherein, current space is the same with dual space's structure, specifically is:

the device comprises a page header, an effective data space and an unstored space, wherein the effective data space is a space in which data is stored, and the page header comprises: effective data starting address, effective data length, sizes of effective data space and non-storage space and other metadata of the data to be processed.

The non-storage space is used to ensure that both the current space and the dual space are aligned with a Physical Block Address (PBA) of the storage device, and the other metadata may be, for example, a data type, an encoding mode, or the like.

And step 3, respectively storing the plurality of first data into the current space corresponding to each page.

For example, the first data may be names of 100 students, sexes of 100 students, and school numbers of 100 students, respectively, the names of 100 students are stored to the current space in column 1 corresponding to page line group 0, the sexes of 100 students are stored to the current space in column 2 corresponding to page line group 0, and the school numbers of 100 students are stored to the current space in column 3 corresponding to page line group 0, which is only an example here.

It should be understood that in the dual space corresponding to the current space, there is no data storage, i.e., the dual space is filled with all 0's of highly compressible data.

And step 4, storing the identification and the time stamp of the current space of each page to the footnote part.

The identification of the current space indicates that the space where the first data are currently stored in the page is the current space or the dual space, and the timestamp records the latest modification time of the space where the first data are stored.

Optionally, fig. 4 is a schematic view of a footnote portion provided in the embodiment of the present application, and as shown in fig. 4, the footnote portion may also include a current space and a dual space corresponding to the current space.

In specific application, information such as the identifier and the timestamp of each current page space can be stored in the current space, when the identifier and the timestamp of each current page space in the information are changed, the identifier and the timestamp of each current page space are read, the information is stored in the dual space corresponding to the current space after the change, and then the current space is reset, so that the change of the footer part information is realized, and data processing of the dual space corresponding to the current space and the current space in the following pages can be specifically referred.

Further, the footnote portion includes: an identifier and a timestamp of a current space of the at least one page, the current space being a space in which the first data is valid to be stored, the timestamp being a time of a latest time of updating the footer portion;

optionally, the identifier of the current space of at least one page may be stored in the metadata of the row group, and the number of the row groups is not limited.

And step 5, determining a column storage file according to each page and the footnote part for storing a plurality of first data.

And generating a column storage file according to each page and the footnote part in which each first data is stored, wherein the data in the file are closely arranged in columns and the operations of inserting, updating or deleting the data are supported.

And step 12, determining the current space in the target page from the footnote part of the column storage file according to the identification of the target page.

In this step, because the footer part is recorded with the identifiers of the current spaces of all the pages, according to the identifier of the target page, the current space, i.e., the effective space, in the target page, in which the first data is currently stored, can be determined in the footer part, and then the corresponding target page and the current space of the target page are found in the list file, thereby providing conditions for the subsequent processing of the first data.

And step 13, performing data processing on the first data in the current space to obtain second data.

Wherein, the data processing comprises: deleting data in the first data and/or updating data in the first data;

in this step, the first data in the current space is read, and the first data is correspondingly processed in the memory layer.

Optionally, taking the current space in column 3 corresponding to the page row group 0 where the school numbers of 100 students are stored as an example, if one of the 100 school numbers needs to be updated, the old school number is replaced in the 100 school numbers.

Further, the subsequent scholarly number, i.e., the second data, is processed.

Optionally, the data processing further includes: and adding target data into the first data, wherein the target data is data which needs to be added into the list file by the user.

Correspondingly, before obtaining the second data, the data processing process further includes the following steps:

step 1, obtaining a space required by target data, an unstored space in a current space and an unstored space in a dual space corresponding to the current space;

optionally, it needs to be determined whether a space occupied by the target data to be added is smaller than or equal to an unstored space in the current space, so as to ensure that the added target data can be completely added, and at this time, a space required by the target data, an unstored space in the current space, and an unstored space in the dual space corresponding to the current space need to be obtained.

Step 2, judging whether the space required by the target data meets a preset condition or not according to the space required by the target data, the non-storage space in the current space and the non-storage space in the dual space corresponding to the current space, wherein the preset condition comprises any one of the following conditions: the space required by the target data is less than or equal to the non-storage space in the current space, and the space required by the target data and the first data is less than or equal to the non-storage space in the dual space corresponding to the current space;

in one possible implementation, for example, the header portion of the current space is 100B, the valid data space occupies 6KB, one PBA of the storage device is 4KB, and the total space of the current space is 8KB, so that the size of the non-storage space in the current space is 8KB-100B-6KB — 1948B.

In addition, the space required for the target data and the first data is also required to be less than or equal to the non-storage space in the dual space, in contrast to the non-storage space in the dual space.

Step 3, when the space required by the target data meets a preset condition, processing the first data in the current space;

illustratively, for example, if the space required by the target data is less than or equal to 1948B, the target data is considered to be added in its entirety, and the normal adding process is entered.

Similarly, the size of the space required by the target data and the first data is required to be smaller than or equal to the size of the non-storage space in the dual space, and it can be considered that the target data can be completely added, and a normal adding flow is entered.

And step 4, when the space required by the target data does not meet the preset condition, acquiring first data of the current space of each page in the column storage file, and then determining data to be processed according to the target data and the first data of the current space of each page in the column storage file.

Optionally, if the space required by the target data does not satisfy the preset condition, that is, the sum of the space required by the target data and the space required by the first data is larger than the size of the non-storage space in the dual space, and the target data and the first data cannot be stored in the non-storage space in the dual space, at this time, the first data and the target data in the current space of each page in the column storage file need to be extracted.

Further, the first data and the target data, i.e. the data to be processed, in the current space of each page may perform the operation of step 2 in step 11 to form a new file, i.e. merge the target data and the old file.

And 14, storing the second data to a dual space corresponding to the current space.

The dual space corresponding to the current space is a new current space, and the dual space is a space for storing the first data in a standby mode.

Optionally, fig. 5 is a schematic diagram of the conversion between the current space and the dual space provided in the embodiment of the present application:

for example, in the footnote part, a space in the target page where the second data/first data is actually stored, that is, a current space, may be represented by 1; the dual space corresponding to the current space is represented by 0.

Taking data updating as an example, after data in the current space is read, updating operation is performed to obtain second data, the second data is stored in a dual space corresponding to the current space, and the identifier of the original dual space is changed from 0 to 1, so that a new current space is obtained.

After this step, the data processing method further includes: and clearing the first data of the current space, setting the current space as a new dual space corresponding to the new current space, and updating the identifier and the timestamp of the current space corresponding to the target page in the footnote part according to the identifier and the timestamp of the new current space and the identifier of the target page corresponding to the new current space.

Optionally, compared with other storage modes, the column storage has the advantage that the space occupied by the data storage is relatively small, so that in order to avoid increasing the dual space, the occupied space of the original storage is increased, and at this time, after the above operation, the first data in the current space is cleared, that is, all 0 data is filled, so as to form a new dual space.

And further, updating the identification of the new current space, the timestamp and the identification of the target page corresponding to the new current space, and storing the updated identification and the timestamp in the footnote part.

Optionally, when the data of the page needs to be processed next time, a space corresponding to the latest timestamp in the footer portion is taken as a current space, and then corresponding page content, that is, information such as an identifier and a timestamp of the current space, is read according to metadata of each line group in the current space in the footer portion.

The application provides a data processing method, which responds to a data processing request of a user, wherein the data processing request carries an identifier of a target page, and determines a current space in the target page from a footer part of a column storage file according to the identifier of the target page, and the footer part comprises: the method comprises the steps of identifying a current space of at least one page and a timestamp, wherein the current space is a space for effectively storing first data, then carrying out data processing on the first data in the current space to obtain second data, and finally storing the second data to a dual space corresponding to the current space, wherein the dual space corresponding to the current space is a new current space, and the dual space is a space for storing the first data in a standby mode. In the technical scheme, the problems of complicated steps and low efficiency when data are processed under the column storage are solved, in the operations of inserting, updating and deleting a small part of data, the Input/Output (IO) flow of the storage device is reduced as much as possible, the influence of the IO operation on the performance of a column storage database is reduced, the atomicity and the consistency of data updating are ensured, meanwhile, the compression function of the transparent compression storage device is fully utilized, and the waste of double spaces in a page to a physical storage space is avoided by filling high-compressible data.

On the basis of the above method embodiment, fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. Applied to electronic equipment, the data processing device comprises: a response module 61, a determination module 62, a processing module 63 and a storage module 64;

the response module 61 is configured to respond to a data processing request of a user, where the data processing request is used to request to process first data of a target page in a file to be stored, and the data processing request carries an identifier of the target page;

a determining module 62, configured to determine, according to the identifier of the target page, a current space in the target page from a footer portion of the list storage file, where the footer portion includes: an identifier and a timestamp of a current space of the at least one page, the current space being a space in which the first data is valid to be stored, the timestamp being a time of a latest time of updating the footer portion;

a processing module 63, configured to perform data processing on the first data in the current space to obtain second data, where the data processing includes: deleting data in the first data and/or updating data in the first data;

and a storage module 64, configured to store the second data in a dual space corresponding to the current space, where the dual space corresponding to the current space is a new current space, and the dual space is a spare space for storing the first data.

In one possible design of the embodiment of the present application, the data processing apparatus further includes: an acquisition module 65;

the obtaining module 65 is used for obtaining the data to be processed before responding to the data processing request of the user;

the processing module 63 is further configured to divide the data to be processed according to a preset storage structure to obtain a plurality of first data, where the preset storage structure includes: the page line groups are provided with a plurality of columns and a footnote part, and each column in each page line group comprises: at least one page, each page comprising: the current space and the dual space corresponding to the current space;

a storage module 64, further configured to: the multiple first data are respectively stored in the current space corresponding to each page, and the identification and the time stamp of the current space of each page are stored in the footnote part;

the determining module 62 is further configured to determine a list file according to each page and the footer portion storing the plurality of first data.

In another possible design of the embodiment of the present application, the data processing further includes: adding target data into the first data, wherein the target data are data which are required to be added into the list storage file by a user;

correspondingly, before obtaining the second data, the obtaining module 65 is further configured to obtain a space required by the target data, an unstored space in the current space, and an unstored space in the dual space corresponding to the current space;

the processing module 63 is further configured to determine whether the space required by the target data meets a preset condition according to the space required by the target data, the non-storage space in the current space, and the non-storage space in the dual space corresponding to the current space, where the preset condition includes any one of the following conditions: the space required by the target data is less than or equal to the non-storage space in the current space, and the space required by the target data and the first data is less than or equal to the non-storage space in the dual space corresponding to the current space; and when the space required by the target data meets the preset condition, processing the first data in the current space.

In this possible design, the processing module 63 is further configured to:

when the space required by the target data does not meet the preset condition, acquiring first data of the current space of each page in the column storage file;

and determining data to be processed according to the target data and the first data of the current space of each page in the column storage file.

In yet another possible design of the embodiment of the present application, the processing module 63 is further configured to:

clearing first data of the current space, and setting the current space as a new dual space corresponding to the new current space;

and updating the identifier and the timestamp of the current space corresponding to the target page in the footnote part according to the identifier and the timestamp of the new current space and the identifier of the target page corresponding to the new current space.

Optionally, the structures of the current space and the dual space are the same, and the structure of the current space or the dual space includes:

the device comprises a page header, an effective data space and an unstored space, wherein the effective data space is a space in which data is stored, and the page header comprises: effective data starting address, effective data length, sizes of effective data space and non-storage space and other metadata of the data to be processed.

The data processing apparatus provided in the embodiment of the present application may be configured to execute the technical solutions corresponding to the data processing methods in the foregoing embodiments, and the implementation principles and technical effects thereof are similar and will not be described herein again.

It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 7, the computer may include: a processor 70, a memory 71, and computer program instructions stored on the memory 71 and operable on the processor 70.

Wherein the electronic device may be a transparent compressed storage device.

The processor 70 executes computer-executable instructions stored by the memory 71, causing the processor 70 to perform the aspects of the embodiments described above. The processor 70 may be a general-purpose processor including a central processing unit CPU, a Network Processor (NP), and the like; but also a digital signal processor DSP, an application specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components.

Optionally, the electronic device may further include: a transceiver 72.

The memory 71 and the transceiver 72 are connected to the processor 70 via a system bus and communicate with each other, and the memory 71 is used for storing computer program instructions.

The transceiver 72 is used for communication with other devices, the transceiver 72 constituting a communication interface.

Optionally, in terms of hardware implementation, the obtaining module 65 in the embodiment shown in fig. 6 corresponds to the transceiver 72 in this embodiment.

The system bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The electronic device provided in the embodiment of the present application may be configured to execute the technical solution corresponding to the data processing method in the foregoing embodiment, and the implementation principle and the technical effect are similar, which are not described herein again.

The embodiment of the application also provides a chip for running the instructions, and the chip is used for executing the technical scheme of the data processing method in the embodiment.

An embodiment of the present application further provides a computer-readable storage medium, where a computer instruction is stored in the computer-readable storage medium, and when the computer instruction runs on a computer device, the electronic device is enabled to execute the technical solution of the data processing method in the foregoing embodiment.

The embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is used for executing the technical solution of the data processing method in the foregoing embodiment when being executed by a processor.

The computer-readable storage medium described above may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. Readable storage media can be any available media that can be accessed by a general purpose or special purpose computer device.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.