阅读说明：本技术 一种数据存储生命周期管理方法及系统 (Data storage life cycle management method and system ) 是由 苏国忠 潘家铭 于 2020-12-25 设计创作，主要内容包括：本发明提供的一种数据存储生命周期管理方法及系统,该方法包括：获取生命周期操作指令；根据生命周期操作指令获取待执行数据对象,并将各待执行数据对象进行合并,生成待传输数据；根据生命周期操作指令对待传输数据进行数据传输。通过将各待执行数据对象进行数据合并处理,将合并后的待传输数据进行数据传输,优化了数据传输方式,提高了数据传输效率。(The invention provides a data storage life cycle management method and a system, wherein the method comprises the following steps: acquiring a life cycle operation instruction; acquiring data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate data to be transmitted; and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction. The data to be transmitted is transmitted by carrying out data merging processing on each data object to be executed and carrying out data transmission on the merged data to be transmitted, so that the data transmission mode is optimized, and the data transmission efficiency is improved.)

1. A data storage lifecycle management method, comprising:

acquiring a life cycle operation instruction;

acquiring data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate data to be transmitted;

and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction.

2. The data storage lifecycle management method of claim 1, wherein the obtaining lifecycle operational instructions comprises:

acquiring a life cycle request message;

and analyzing the life cycle request message to generate the life cycle operation instruction.

3. The data storage lifecycle management method of claim 1, wherein the retrieving the data object to be executed according to the lifecycle operation instruction comprises:

acquiring a currently stored data object;

and sorting according to the size of each data object, and screening out the data objects to be executed within a preset level range according to the life cycle operation instruction.

4. The data storage life cycle management method according to claim 1, wherein the data transmission of the data to be transmitted according to the life cycle operation instruction includes:

acquiring an original storage address of the data to be transmitted;

determining a target storage address of the data to be transmitted according to the life cycle operation instruction;

and carrying out data transmission on the data to be transmitted according to the original storage address and the target storage address.

5. The data storage lifecycle management method of claim 1, further comprising:

and generating life cycle management process information according to the life cycle operation instruction and the data transmission result.

6. The data storage lifecycle management method of claim 1, further comprising:

acquiring the access frequency of a current data object;

and adjusting the storage mode of the current data object according to the access frequency.

7. The data storage lifecycle management method of claim 6, wherein the adjusting the storage pattern of the current data object according to the access frequency comprises:

when the access frequency is greater than a preset access frequency, storing the current data object to a preset frequent access layer;

and when the access frequency is not more than the preset access frequency, storing the current data object to a preset infrequent access layer.

8. A data storage lifecycle management system, comprising:

the acquisition module is used for acquiring a life cycle operation instruction;

the processing module is used for acquiring data objects to be executed according to the life cycle operation instruction, merging the data objects to be executed and generating data to be transmitted;

and the transmission module is used for carrying out data transmission on the data to be transmitted according to the life cycle operation instruction.

9. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the data storage lifecycle management method of any of claims 1-7.

10. A computer device, comprising: a memory and a processor communicatively coupled to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the data storage lifecycle management method of any of claims 1-7.

Technical Field

The invention relates to the field of big data storage and blue light storage, in particular to a data storage life cycle management method and system.

Background

In the existing Amazon S3 distributed system, a special performance optimization strategy is proposed for data transmission or storage type conversion of a storage object at a high throughput, so as to improve data transmission efficiency. However, the existing performance optimization strategy can only implement data processing with storage object sizes at MB level, GB level, and even TB level, and when a large number of objects with less contents at KB level are stored in a storage space, when a user needs to transmit the objects, concurrent data transmission congestion occurs in the existing Amazon S3 distributed system, which affects data transmission efficiency.

Disclosure of Invention

Therefore, the technical problem to be solved by the present invention is to overcome the defect of low data transmission efficiency in the prior art, and to provide a method and a system for managing the life cycle of object storage.

In order to achieve the purpose, the invention provides the following technical scheme:

in a first aspect, an embodiment of the present invention provides a data storage lifecycle management method, including: acquiring a life cycle operation instruction; acquiring data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate data to be transmitted; and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction.

In one embodiment, the obtaining the lifecycle operational instructions comprises: acquiring a life cycle request message; and analyzing the life cycle request message to generate the life cycle operation instruction.

In an embodiment, the acquiring the data object to be executed according to the lifecycle operation instruction includes: acquiring a currently stored data object; and sorting according to the size of each data object, and screening out the data objects to be executed within a preset level range according to the life cycle operation instruction.

In an embodiment, the performing data transmission on the data to be transmitted according to the lifecycle operation instruction includes: acquiring an original storage address of the data to be transmitted; determining a target storage address of the data to be transmitted according to the life cycle operation instruction; and carrying out data transmission on the data to be transmitted according to the original storage address and the target storage address.

In one embodiment, the data storage lifecycle management method further comprises: and generating life cycle management process information according to the life cycle operation instruction and the data transmission result.

In one embodiment, the data storage lifecycle management method further comprises: acquiring the access frequency of a current data object; and adjusting the storage mode of the current data object according to the access frequency.

In one embodiment, the adjusting the storage mode of the current data object according to the access frequency includes: when the access frequency is greater than a preset access frequency, storing the current data object to a preset frequent access layer; and when the access frequency is not more than the preset access frequency, storing the current data object to a preset infrequent access layer.

In a second aspect, an embodiment of the present invention provides a data storage lifecycle management system, including: the acquisition module is used for acquiring a life cycle operation instruction; the processing module is used for acquiring data objects to be executed according to the life cycle operation instruction, merging the data objects to be executed and generating data to be transmitted; and the transmission module is used for carrying out data transmission on the data to be transmitted according to the life cycle operation instruction.

In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to cause a computer to execute the data storage lifecycle management method according to the first aspect of the embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention provides a computer device, including: the data storage life cycle management system comprises a memory and a processor, wherein the memory and the processor are mutually connected in a communication mode, the memory stores computer instructions, and the processor executes the computer instructions so as to execute the data storage life cycle management method of the first aspect of the embodiment of the invention.

The technical scheme of the invention has the following advantages:

the data storage life cycle management method provided by the invention comprises the following steps: acquiring a life cycle operation instruction; acquiring data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate data to be transmitted; and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction. The data to be transmitted is transmitted by carrying out data merging processing on each data object to be executed and carrying out data transmission on the merged data to be transmitted, so that the data transmission mode is optimized, and the data transmission efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flowchart of a specific example of a data storage lifecycle management method according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating an exemplary embodiment of obtaining lifecycle operations instructions according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating a specific example of obtaining a data object to be executed according to a lifecycle operation instruction in an embodiment of the present invention;

fig. 4 is a flowchart of a specific example of data transmission performed on data to be transmitted according to a lifecycle operation instruction in the embodiment of the present invention;

FIG. 5 is a flowchart of another specific example of a data storage lifecycle management method in an embodiment of the present invention;

FIG. 6 is a functional block diagram of a specific example of a data storage lifecycle management system in an embodiment of the present invention;

fig. 7 is a block diagram of a specific example of a computer device according to an embodiment of the present invention.

Detailed Description

The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

The embodiment of the invention provides a data storage life cycle management method, which is applied to an Amazon S3 distributed system and comprises the following steps as shown in FIG. 1:

step S1: and acquiring a life cycle operation instruction.

In one embodiment, as shown in fig. 2, the obtaining of the lifecycle operation instruction includes the following steps:

step S11: and acquiring a life cycle request message.

Step S12: and analyzing the life cycle request message to generate a life cycle operation instruction.

In the embodiment of the invention, after the life cycle request message sent by the user is received, the life cycle request message is analyzed, and the semantic definition analysis of the life cycle is realized. Specifically, the lifecycle manager is responsible for managing and executing lifecycle policies. The specific implementation of the lifecycle management is by defining the lifecycle policy of S3 and executing according to the determined policy. Wherein, the semantic definition of the life cycle strategy of S3 is as follows: 1) each configuration contains one or more lifecycle rules (there may be up to 1000 rules). 2) Each rule consists of (ID element, Status element, Filter element, element for describing the lifecycle operation), wherein the ID element uniquely identifies the rule, the ID element being at most 255 characters in length; the Status element value may be Enabled or Disabled, but if the lifecycle rule is in Disabled state, then this lifecycle policy does not perform any operations defined in the lifecycle rule; the Filter element is applied to all objects or a part of objects in the storage bucket, and keys prefix, object labels or a combination of the prefix and the object labels are pressed to screen the objects; elements for describing lifecycle operations, one or more of the following predefined specific operations to be performed are specified in the lifecycle rules. In the embodiment of the present invention, the specific operations to be performed include: a storage type conversion operation and a deletion operation.

Step S2: and acquiring the data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate the data to be transmitted.

In an embodiment, as shown in fig. 3, acquiring the data object to be executed according to the lifecycle operation instruction includes the following steps:

step S21: and acquiring the currently stored data object.

Step S22: and sorting according to the size of each data object, and screening out the data objects to be executed within a preset level range according to the life cycle operation instruction.

In the embodiment of the invention, after the currently stored data objects are acquired according to the lifecycle operation instruction, the data objects are sorted according to the storage space occupied by the data objects (namely the size of the data objects), and the KB-level data objects to be executed with less content are screened out. And then, by utilizing the characteristic of high IO throughput capacity in the source node, merging the data of the data objects to be executed with less content and KB level into data packets, and transmitting the merged data packets as data to be transmitted. Specifically, the size and content of the data packet may be calculated according to parameters such as a destination source of the data object to be executed, an area of the data object to be executed, the size of the data object to be executed, characteristics of the data object to be executed, and an available bandwidth between the source and the destination source at that time. Because the existing performance optimization strategies can only realize data processing with the storage object size at MB level, GB level and even TB level, an effective performance optimization strategy is lacked for data objects to be executed at KB level. Therefore, in the embodiment of the invention, the data objects to be executed at KB level with less content are merged into the data packet occupying larger storage space by using the client, and the merged data packet is used as the data to be transmitted and transmitted to the target server, so that the data transmission efficiency is improved.

Step S3: and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction.

In a specific embodiment, as shown in fig. 4, the data transmission of the data to be transmitted according to the lifecycle operation instruction includes the following steps:

step S31: and acquiring an original storage address of the data to be transmitted.

Step S32: and determining the destination storage address of the data to be transmitted according to the life cycle operation instruction.

Step S33: and carrying out data transmission on the data to be transmitted according to the original storage address and the target storage address.

In the embodiment of the invention, the original storage address can be a magnetic disk or a blue-ray memory, and the target storage address can be a magnetic disk or a blue-ray memory. When the original storage address is a disk and the target storage address is a blue-ray memory, the transmission direction of the data to be transmitted is that the data to be transmitted is written into the blue-ray memory by the disk; when the original storage address is a blue-ray memory and the target storage address is a disk, the transmission direction of the data to be transmitted is that the data to be transmitted is read out to the disk by the blue-ray memory. However, during the transition between storage types (data transfer between disk and blu-ray storage), there may be a limiting requirement between transitions. For example, the following lifecycle transitions may occur between storage classes of WS S3 lifecycle configuration: from the STANDARD storage class, conversion to any other storage class may occur. From any storage class, it can be converted into a GLACER or DEEP _ ARCHIVE storage class. From the STANDARD _ IA storage class to the INTELLIGENT _ TIERING or ONE ZONE _ IA storage class. From INTELLIGENT _ TIERING storage class to the OneZONE _ IA storage class. The class is converted from the GLACER storage class to the DEEP _ ARCHIVE storage class. The storage types not supported by the AWS 3 lifecycle configuration are converted as follows: transition from any storage class to the STANDARD storage class. From any memory class to the redundant _ redundant memory class. From INTELLIGENT _ TIERING storage class to STANDARD _ IA storage class. From the ONE _ IA storage class to the STANDARD _ IA or INTELLIGENT _ TIERING storage class.

The data storage life cycle management method provided by the invention comprises the following steps: acquiring a life cycle operation instruction; acquiring data objects to be executed according to the life cycle operation instruction, and merging the data objects to be executed to generate data to be transmitted; and carrying out data transmission on the data to be transmitted according to the life cycle operation instruction. The data to be transmitted is transmitted by carrying out data merging processing on each data object to be executed and carrying out data transmission on the merged data to be transmitted, so that the data transmission mode is optimized, and the data transmission efficiency is improved.

In one embodiment, the data storage lifecycle management method further comprises: and generating life cycle management process information according to the life cycle operation instruction and the data transmission result.

In a specific embodiment, by constructing and configuring the message middle Kafka and using the existing programming language, the life cycle management process information is acquired and sent to the client of the message middleware of the user, so that the client can conveniently inquire the relevant information of the life cycle management process.

In one embodiment, as shown in fig. 5, the data storage lifecycle management method includes the following steps:

step S4: acquiring the access frequency of a current data object;

step S5: and adjusting the storage mode of the current data object according to the access frequency.

In a specific embodiment, the access layers in the Amazon S3 distributed system are divided into frequent access layers and infrequent access layers in advance, and then the storage mode of the current data object is adjusted according to the access frequency of the current data object. In the embodiment of the invention, when the access frequency is greater than the preset access frequency, the current data object is stored to a preset frequent access layer; and when the access frequency is not more than the preset access frequency, storing the current data object to a preset infrequent access layer. By presetting the frequent access layer and the infrequent access layer, different access bandwidths can be provided for objects of different frequent access layers, and the user experience effect is optimized; more reasonable hardware configuration can be provided for objects with different frequent access levels, and the hardware operation cost is optimized; in addition, more reasonable network setting can be provided for objects with different frequent access levels, and the network condition and the network operation cost are optimized. The method has the advantages that the corresponding life cycle strategy is configured to migrate the infrequently accessed objects to the infrequently accessed data access layer, so that the bandwidth of the infrequently accessed data access layer is reduced, more bandwidth is given to the frequently accessed objects, the data access efficiency is optimized, and the storage cost can be remarkably reduced.

An embodiment of the present invention further provides a data storage lifecycle management system, as shown in fig. 6, including:

the obtaining module 1 is used for obtaining a life cycle operation instruction. For details, refer to the related description of step S1 in the above embodiment, and are not described herein again.

And the processing module 2 is used for acquiring the data objects to be executed according to the life cycle operation instruction, merging the data objects to be executed and generating data to be transmitted. For details, refer to the related description of step S2 in the above embodiment, and are not described herein again.

And the transmission module 3 is used for transmitting data to be transmitted according to the life cycle operation instruction. For details, refer to the related description of step S3 in the above embodiment, and are not described herein again.

According to the data storage life cycle management system provided by the invention, the data merging processing is carried out on each data object to be executed, and the merged data to be transmitted is subjected to data transmission, so that the data transmission mode is optimized, and the data transmission efficiency is improved.

An embodiment of the present invention further provides a computer device, as shown in fig. 7, the device may include a processor 61 and a memory 62, where the processor 61 and the memory 62 may be connected by a bus or in another manner, and fig. 7 takes the connection by the bus as an example.

The processor 61 may be a Central Processing Unit (CPU). The Processor 61 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.

The memory 62, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the corresponding program instructions/modules in embodiments of the present invention. The processor 61 executes various functional applications and data processing of the processor by running non-transitory software programs, instructions and modules stored in the memory 62, namely, implements the data storage lifecycle management method in the above method embodiments.

The memory 62 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 61, and the like. Further, the memory 62 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 62 may optionally include memory located remotely from the processor 61, and these remote memories may be connected to the processor 61 via a network. Examples of such networks include, but are not limited to, the internet, intranets, mobile communication networks, and combinations thereof.

One or more modules are stored in the memory 62, and when executed by the processor 61, perform the electric energy meter data interaction method provided by the embodiment of the present invention or perform the data storage lifecycle management method provided by the implementation of the present invention.

The details of the computer device can be understood by referring to the corresponding descriptions and effects in the embodiments shown in fig. 1-5, and are not described herein again.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program that can be stored in a computer-readable storage medium and that when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.

It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications of the invention may be made without departing from the spirit or scope of the invention.