阅读说明：本技术 一种数据读写方法及装置 (Data reading and writing method and device ) 是由 周涛 于 2021-07-14 设计创作，主要内容包括：本申请涉及数据存储技术领域,特别涉及一种数据读写方法及装置。该方法应用于分布式存储系统,该方法包括：接收客户端发送的多个读写请求；判断当前执行的读写请求为读请求/写请求；若判定当前执行的读写请求为目标读请求,且独占锁未被其它读写请求占用,则执行所述目标读请求,并将下一个读写请求作为当前执行的读写请求,否则,不执行所述目标读请求；若判定当前执行的读写请求为目标写请求,且所述独占锁未被其它读写请求占用,则占用所述独占锁,并执行所述目标写请求。(The present application relates to the field of data storage technologies, and in particular, to a data reading and writing method and apparatus. The method is applied to a distributed storage system and comprises the following steps: receiving a plurality of read-write requests sent by a client; judging whether the currently executed read-write request is a read request/write request; if the currently executed read-write request is judged to be a target read request and the exclusive lock is not occupied by other read-write requests, executing the target read request and taking the next read-write request as the currently executed read-write request, otherwise, not executing the target read request; and if the currently executed read-write request is judged to be the target write request and the exclusive lock is not occupied by other read-write requests, occupying the exclusive lock and executing the target write request.)

1. A data read-write method is applied to a distributed storage system, and the method comprises the following steps:

receiving a plurality of read-write requests sent by a client;

judging whether the currently executed read-write request is a read request/write request;

if the currently executed read-write request is judged to be a target read request and the exclusive lock is not occupied by other read-write requests, executing the target read request and taking the next read-write request as the currently executed read-write request, otherwise, not executing the target read request;

and if the currently executed read-write request is judged to be the target write request and the exclusive lock is not occupied by other read-write requests, occupying the exclusive lock and executing the target write request.

2. The method of claim 1, wherein the method further comprises:

and if the target write request is determined to be executed completely, releasing the exclusive lock.

3. The method of claim 1, wherein the step of executing the target read request comprises:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

4. The method of claim 1, wherein the step of executing the target write request comprises:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

5. The method of claim 3 or 4, wherein the cache storage is a redis cache.

6. A data reading/writing apparatus, applied to a distributed storage system, the apparatus comprising:

the receiving unit is used for receiving a plurality of read-write requests sent by the client;

the judging unit is used for judging the currently executed read-write request as a read request/write request;

the execution unit is used for executing the target read request and taking the next read-write request as the currently executed read-write request if the judgment unit judges that the currently executed read-write request is the target read request and the exclusive lock is not occupied by other read-write requests, otherwise, the execution unit does not execute the target read request;

and if the judging unit judges that the currently executed read-write request is the target write request and the exclusive lock is not occupied by other read-write requests, the executing unit is used for occupying the exclusive lock and executing the target write request.

7. The apparatus of claim 6, wherein the apparatus further comprises a determining unit and a releasing unit:

and if the determining unit determines that the target write request is completely executed, the releasing unit is used for releasing the exclusive lock.

8. The apparatus as claimed in claim 6, wherein, when executing the target read request, the execution unit is specifically configured to:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

9. The apparatus as claimed in claim 6, wherein, when executing the target write request, the execution unit is specifically configured to:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

10. The apparatus of claim 8 or 9, wherein the cache storage is a redis cache.

Technical Field

The present application relates to the field of data storage technologies, and in particular, to a data reading and writing method and apparatus.

Background

At present, many large-scale systems and applications are deployed in a distributed manner, the problem of data consistency is always an important topic, a distributed lock is a way of controlling synchronous access to shared resources among distributed systems, but too many locks often cause slow system performance and poor user experience, and typical use scenarios are as follows: different systems or different hosts of the same system share one or a group of data, and therefore when accessing the data, mutual interference needs to be prevented through a certain mutual exclusion means to ensure consistency.

Disclosure of Invention

The application provides a data reading and writing method and device, which are used for solving the problem of low processing efficiency of reading and writing requests in the prior art.

In a first aspect, the present application provides a data read-write method, which is applied to a distributed storage system, and the method includes:

receiving a plurality of read-write requests sent by a client;

judging whether the currently executed read-write request is a read request/write request;

if the currently executed read-write request is judged to be a target read request and the exclusive lock is not occupied by other read-write requests, executing the target read request and taking the next read-write request as the currently executed read-write request, otherwise, not executing the target read request;

and if the currently executed read-write request is judged to be the target write request and the exclusive lock is not occupied by other read-write requests, occupying the exclusive lock and executing the target write request.

Optionally, the method further comprises:

and if the target write request is determined to be executed completely, releasing the exclusive lock.

Optionally, the step of executing the target read request includes:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

Optionally, the step of executing the target write request includes:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

Optionally, the cache storage is a redis cache.

In a second aspect, the present application provides a data reading and writing apparatus, which is applied to a distributed storage system, and the apparatus includes:

the receiving unit is used for receiving a plurality of read-write requests sent by the client;

the judging unit is used for judging the currently executed read-write request as a read request/write request;

the execution unit is used for executing the target read request and taking the next read-write request as the currently executed read-write request if the judgment unit judges that the currently executed read-write request is the target read request and the exclusive lock is not occupied by other read-write requests, otherwise, the execution unit does not execute the target read request;

and if the judging unit judges that the currently executed read-write request is the target write request and the exclusive lock is not occupied by other read-write requests, the executing unit is used for occupying the exclusive lock and executing the target write request.

Optionally, the apparatus further comprises a determining unit and a releasing unit:

and if the determining unit determines that the target write request is completely executed, the releasing unit is used for releasing the exclusive lock.

Optionally, when the target read request is executed, the execution unit is specifically configured to:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

Optionally, when the target write request is executed, the execution unit is specifically configured to:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

Optionally, the cache storage is a redis cache.

In a third aspect, an embodiment of the present application provides a data reading and writing apparatus, which is applied to a distributed storage system, and the data reading and writing apparatus includes:

a memory for storing program instructions;

a processor for calling program instructions stored in said memory and for executing the steps of the method according to any one of the above first aspects in accordance with the obtained program instructions.

In a fourth aspect, the present application further provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps of the method according to any one of the above first aspects.

In summary, the data read-write method applied to the distributed storage system provided in the embodiment of the present application receives multiple read-write requests sent by a client; judging whether the currently executed read-write request is a read request/write request; if the currently executed read-write request is judged to be a target read request and the exclusive lock is not occupied by other read-write requests, executing the target read request and taking the next read-write request as the currently executed read-write request, otherwise, not executing the target read request; and if the currently executed read-write request is judged to be the target write request and the exclusive lock is not occupied by other read-write requests, occupying the exclusive lock and executing the target write request.

By adopting the data reading and writing method provided by the embodiment of the application, when a multi-thread task is executed, other threads are blocked only when a writing thread is executed, and when a reading thread is executed, other threads are not required to be blocked, so that the blocking function of the distributed reading and writing lock system is realized, and meanwhile, the multi-thread operation is allowed, and the system performance is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.

Fig. 1 is a detailed flowchart of a data read/write method according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a data reading/writing apparatus according to an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of a data reading and writing device according to an embodiment of the present application.

Detailed Description

The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".

Exemplarily, referring to fig. 1, a detailed flowchart of a data read/write method provided in an embodiment of the present application is shown, where the method is applied to a distributed storage system, and the method includes the following steps:

step 100: and receiving a plurality of read-write requests sent by the client.

In the embodiment of the application, the distributed storage system refers to a distributed storage system based on a redis storage cluster, and firstly, a redis cluster environment is built, and the distributed storage system starts a multithreading running state, that is, a plurality of data processing threads can be executed in parallel.

Then, when the distributed storage system receives the read-write requests sent by the multiple clients, the corresponding read-write threads can be generated for each read-write request, and in practical application, the read-write threads are arranged in the queue to be processed according to the generation time of each read-write thread. In the embodiment of the present application, the simultaneous execution of multiple threads means that when one thread is processed, if other threads still satisfy the processing condition, the other threads can still be processed, and at this time, the other threads do not need to be processed after the currently processed threads are processed.

Step 110: and judging that the currently executed read-write request is a read request/write request.

As can be seen from the above, the read-write threads corresponding to the multiple read-write requests are maintained in the queue to be processed, and the first read-write thread in the queue to be processed is used as the thread corresponding to the currently executed read-write request, and it is determined whether the thread (read request) is a read thread or a write thread (write request).

Step 120: and if the currently executed read-write request is judged to be the target read request and the exclusive lock is not occupied by other read-write requests, executing the target read request and taking the next read-write request as the currently executed read-write request, otherwise, not executing the target read request.

In the embodiment of the application, if it is determined that the currently executed read-write request is a read request, it is further determined whether a preset exclusive lock is occupied by other read-write requests, and if it is determined that the exclusive lock is not occupied by other read-write requests, the read request is executed.

In the embodiment of the application, when a write request is executed, an exclusive lock needs to be occupied for preventing the execution of other threads.

Specifically, in the embodiment of the present application, when the target read request is executed, a preferred implementation manner is:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

That is, in order to improve the efficiency of data reading, the preset data is cached in the cache storage, for example, the specified data read by the client is cached in the cache storage, and/or the written specified data is cached in the cache storage. Then, when reading a piece of data, first, it is determined whether the data is cached in the cache memory, if so, the data is directly read from the cache, and if not, the data needs to be read from the back-end memory.

Further, in the embodiment of the present application, when it is determined to execute one read request, since the read request does not occupy an exclusive lock, other read and write requests may also be executed at this time, and then, at this time, the read and write request next to the read request in the queue may be used as the currently executed read and write request.

It should be noted that, in the embodiment of the present application, when the exclusive lock is occupied by other write requests, other read and write requests in the queue are not executed until the write request is executed, and after the exclusive lock is released, the other read and write requests in the queue are executed.

Step 130: and if the currently executed read-write request is judged to be the target write request and the exclusive lock is not occupied by other read-write requests, occupying the exclusive lock and executing the target write request.

Specifically, in the embodiment of the present application, when the target write request is executed, a preferred implementation manner is:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

That is, if the target write request is a request for updating the second data, and at this time, the second data is already cached in the cache memory, the second data cached in the cache memory is updated after the second data stored in the back end is updated. If the target write request is a request for writing new data (second data), the second data is cached in the cache memory while being stored to the back-end storage.

The data reading and writing method provided by the embodiment of the present application is described in detail below with reference to specific application scenarios. The detailed flow of the data reading and writing method provided by the embodiment of the application is as follows:

step 1, building a redis cluster environment.

And step 2, starting a multithreading running state.

Step 3, judging whether to execute the writing request, if so, executing step 4; otherwise, step 7 is performed.

And 4, acquiring the exclusive lock, and setting the sync value to false so as to block other threads from executing.

It should be noted that if the sync value of the exclusive lock is true, it indicates that the exclusive lock is not occupied; if the sync value of the exclusive lock is false, it indicates that the exclusive lock is occupied.

And 5, executing the write request, re-querying the database, performing operation and generating redis cache data.

And 6, releasing the exclusive lock.

Step 7, judging whether the exclusive lock is occupied, and if not, executing a read operation; otherwise, no read operation is performed.

Specifically, if the data to be read exists in the cache memory, the data to be read is directly read from the cache memory; if the data to be read does not exist in the cache storage, the data to be read needs to be read from the back-end storage, and the data to be read is cached in the cache storage.

Exemplarily, referring to fig. 2, a schematic structural diagram of a data read/write apparatus provided in an embodiment of the present application is shown, where the apparatus is applied to a distributed storage system, and the apparatus includes:

a receiving unit 20, configured to receive multiple read-write requests sent by a client;

a judging unit 21, configured to judge that a currently executed read/write request is a read request/write request;

an executing unit 22, if the determining unit 21 determines that the currently executed read-write request is a target read request and the exclusive lock is not occupied by other read-write requests, the executing unit 22 is configured to execute the target read request and take the next read-write request as the currently executed read-write request, otherwise, the executing unit 22 does not execute the target read request;

if the determining unit 21 determines that the currently executed read/write request is the target write request and the exclusive lock is not occupied by other read/write requests, the executing unit 22 is configured to occupy the exclusive lock and execute the target write request.

Optionally, the apparatus further comprises a determining unit and a releasing unit:

and if the determining unit determines that the target write request is completely executed, the releasing unit is used for releasing the exclusive lock.

Optionally, when the target read request is executed, the execution unit 22 is specifically configured to:

determining first data to be read based on the target read request;

judging whether the first data exists in a cache storage;

if the first data exists in the cache storage, reading the first data from the cache storage, and returning the first data to a client;

and if the first data does not exist in the cache storage, reading the first data from a back-end storage, returning the first data to a client, and caching the first data in the cache storage.

Optionally, when the target write request is executed, the execution unit 22 is specifically configured to:

determining second data to be written based on the target write request;

and writing the second data into a back-end storage, and updating the second data in a cache storage.

Optionally, the cache storage is a redis cache.

The above units may be one or more integrated circuits configured to implement the above methods, for example: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above units is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

Further, in the data reading and writing device provided in the embodiment of the present application, from a hardware aspect, a schematic diagram of a hardware architecture of the data reading and writing device may be shown in fig. 3, where the data reading and writing device may include: a memory 30 and a processor 31, which,

the memory 30 is used for storing program instructions; the processor 31 calls the program instructions stored in the memory 30 and executes the above-described method embodiments in accordance with the obtained program instructions. The specific implementation and technical effects are similar, and are not described herein again.

Optionally, the present application also provides a data reading and writing device, which includes at least one processing element (or chip) for executing the above method embodiments.

Optionally, the present application also provides a program product, such as a computer-readable storage medium, having stored thereon computer-executable instructions for causing the computer to perform the above-described method embodiments.

Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and so forth. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.