阅读说明：本技术 一种数据传输方法、装置、计算机可读存储介质和计算机设备 (Data transmission method and device, computer readable storage medium and computer equipment ) 是由 陈亚川 苗辉 于 2019-04-30 设计创作，主要内容包括：本发明提供了一种数据传输方法、装置、计算机可读存储介质和计算机设备。涉及数据通信技术,解决了发送端数据缓存过程中资源消耗大、部署不便等问题。该方法包括：在内存中创建内存缓冲单元,将待发送数据写入所述内存缓冲单元；在所述内存缓冲单元写满后,将所述内存缓冲单元中存储的数据序列化到磁盘；创建新的内存缓冲单元,继续向所述内存缓冲单元中写入待发送数据。本发明提供的技术方案适用于数据跨网传输,实现了适应各种网络环境的低风险、简化的数据缓存。(The invention provides a data transmission method, a data transmission device, a computer readable storage medium and computer equipment. The method relates to a data communication technology, and solves the problems of large resource consumption, inconvenience in deployment and the like in the data caching process of a sending end. The method comprises the following steps: creating a memory buffer unit in a memory, and writing data to be sent into the memory buffer unit; after the memory buffer unit is fully written, serializing the data stored in the memory buffer unit to a disk; and creating a new memory buffer unit, and continuously writing data to be sent into the memory buffer unit. The technical scheme provided by the invention is suitable for data cross-network transmission, and realizes low-risk and simplified data caching suitable for various network environments.)

1. A method of data transmission, comprising:

creating a memory buffer unit in a memory, and writing data to be sent into the memory buffer unit;

after the memory buffer unit is fully written, serializing the data stored in the memory buffer unit to a disk;

and creating a new memory buffer unit, and continuously writing data to be sent into the memory buffer unit.

2. The data transmission method according to claim 1, wherein the step of serializing the data stored in the memory buffer unit to disk after the memory buffer unit is full comprises:

after the memory buffer unit is fully written, serializing data stored in the memory buffer unit into a disk buffer unit on a disk;

and releasing the memory occupied by the memory buffer unit.

3. The data transmission method of claim 2, further comprising:

Setting an upper limit of the buffer data volume of the magnetic disk;

and when the data volume serialized to the disk reaches the upper limit of the disk buffer data volume, stopping creating a new memory buffer unit.

4. The data transmission method according to claim 2, wherein the step of creating a new memory buffer unit and continuing to write data to be transmitted into the memory buffer unit comprises:

and the memory buffer unit in the memory and the disk buffer unit on the disk form a linked list structure according to the creation time sequence.

5. The data transmission method of claim 4, further comprising:

and when the data to be sent exists in the memory buffer unit, taking out and sending the data to be sent in the memory buffer unit.

6. The data transmission method of claim 4, further comprising:

when the memory buffer unit is empty, turning to the next memory buffer unit pointed by the memory buffer unit;

and taking out and sending the data to be sent in the next memory buffer unit.

7. The data transmission method of claim 4, further comprising:

When the memory buffer unit is empty and points to one disk buffer unit, turning to the next disk buffer unit pointed by the memory buffer unit;

releasing the memory occupied by the memory buffer unit, and deserializing the next disk buffer unit into a new memory buffer unit in the memory;

and taking out and sending the data to be sent in the new memory buffer unit.

8. A data transmission apparatus, comprising:

the memory buffer area management module is used for creating a memory buffer unit in a memory, writing data to be sent into the memory buffer unit, creating a new memory buffer unit after the memory buffer unit is written with a full sequence to a disk, and continuously writing the data to be sent into the memory buffer unit;

and the data persistence management module is used for serializing the data stored in the memory buffer unit to a disk after the memory buffer unit is fully written.

9. The data transmission apparatus of claim 8,

the memory buffer management module is further configured to, when data to be sent exists in the memory buffer unit, take out and send the data to be sent in the memory buffer unit, or,

When the memory buffer unit is empty, turning to the next memory buffer unit pointed by the memory buffer unit, releasing the memory space occupied by the memory buffer unit, and taking out and sending the data to be sent in the next memory buffer unit, or,

when the memory buffer unit is empty and points to a disk buffer unit, the next disk buffer unit pointed by the memory buffer unit is switched to, the memory occupied by the memory buffer unit is released, the next disk buffer unit is deserialized into a new memory buffer unit in the memory, and the data to be sent in the new memory buffer unit is taken out;

and the data persistence management module deserializes the next disk buffer unit pointed by the memory buffer unit into a new memory buffer unit in the memory.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed, implements the steps of the method according to any one of claims 1-7.

11. A computer arrangement comprising a processor, a memory and a computer program stored on the memory, characterized in that the steps of the method according to any of claims 1-7 are implemented when the computer program is executed by the processor.

Technical Field

The present invention relates to data communication technologies, and in particular, to a data transmission method, an apparatus, a computer-readable storage medium, and a computer device.

Background

In data cross-network transmission, due to the reasons of network quality deterioration, operator cutover, network failure and the like, the data transmission speed is slow, and even the transmission is interrupted. When a data sending end passively receives data to be sent, newly added data occupies a large amount of memory space, and even memory overflow (OOM) is triggered to cause process crash. Even if the process of the sending end is not broken down, the cache data to be sent occupy a large amount of memory and can also influence the service of other processes.

Taking open source software flash as an example, flash uses a channel component as a data buffer, and a common channel type includes a memory, a file, and the like, and managing the data buffer by using flash has the following problems:

1. the channel of the memory type only occupies memory cache data, so that the performance is high, but the data can be lost after the process is restarted, and the reliability is poor; and when the network is blocked, a large amount of memory resources are consumed, and the OOM is easy to occur.

2. All data transmission of the file type channel occupies disk I/O, and a large amount of disk resources are consumed.

3. Other types of channels rely on third party software and are inconvenient to deploy.

In conclusion, the problems of large resource consumption, inconvenience in deployment and the like exist in the data caching process of the sending end.

Disclosure of Invention

The present invention is directed to solving the problems described above.

According to an aspect of the present invention, there is provided a data transmission method, including:

creating a memory buffer unit in a memory, and writing data to be sent into the memory buffer unit;

after the memory buffer unit is fully written, serializing the data stored in the memory buffer unit to a disk;

and creating a new memory buffer unit, and continuously writing data to be sent into the memory buffer unit.

Preferably, after the memory buffer unit is full, serializing the data stored in the memory buffer unit to the disk includes:

after the memory buffer unit is fully written, serializing data stored in the memory buffer unit into a disk buffer unit on a disk;

and releasing the memory occupied by the memory buffer unit.

Preferably, the method further comprises:

setting an upper limit of the buffer data volume of the magnetic disk;

and when the data volume serialized to the disk reaches the upper limit of the disk buffer data volume, stopping creating a new memory buffer unit.

Preferably, the step of creating a new memory buffer unit and continuing to write data to be sent into the memory buffer unit includes:

and the memory buffer unit in the memory and the disk buffer unit on the disk form a linked list structure according to the creation time sequence.

Preferably, the method further comprises:

and when the data to be sent exists in the memory buffer unit, taking out and sending the data to be sent in the memory buffer unit.

Preferably, the method further comprises:

when the memory buffer unit is empty, turning to the next memory buffer unit pointed by the memory buffer unit;

And taking out and sending the data to be sent in the next memory buffer unit.

Preferably, the method further comprises:

when the memory buffer unit is empty and points to one disk buffer unit, turning to the next disk buffer unit pointed by the memory buffer unit;

releasing the memory occupied by the memory buffer unit, and deserializing the next disk buffer unit into a new memory buffer unit in the memory;

and taking out and sending the data to be sent in the new memory buffer unit.

According to another aspect of the present invention, there is also provided a data transmission apparatus, including:

the memory buffer area management module is used for creating a memory buffer unit in a memory, writing data to be sent into the memory buffer unit, creating a new memory buffer unit after the memory buffer unit is written with a full sequence to a disk, and continuously writing the data to be sent into the memory buffer unit;

and the data persistence management module is used for serializing the data stored in the memory buffer unit to a disk after the memory buffer unit is fully written.

Preferably, the memory buffer management module is further configured to, when data to be sent exists in the memory buffer unit, take out and send the data to be sent in the memory buffer unit, or,

When the memory buffer unit is empty, turning to the next memory buffer unit pointed by the memory buffer unit, releasing the memory space occupied by the memory buffer unit, and taking out and sending the data to be sent in the next memory buffer unit, or,

when the memory buffer unit is empty and points to a disk buffer unit, the next disk buffer unit pointed by the memory buffer unit is switched to, the memory occupied by the memory buffer unit is released, the next disk buffer unit is deserialized into a new memory buffer unit in the memory, and the data to be sent in the new memory buffer unit is taken out;

and the data persistence management module deserializes the next disk buffer unit pointed by the memory buffer unit into a new memory buffer unit in the memory.

According to yet another aspect of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed, performs the steps of the method as described above.

According to yet another aspect of the present invention, there is also provided a computer device comprising a processor, a memory and a computer program stored on the memory, the processor implementing the steps of the above method when executing the computer program.

The invention provides a data transmission method, a data transmission device, a computer readable storage medium and computer equipment. Creating a memory buffer unit in a memory, writing data to be sent into the memory buffer unit, serializing the data stored in the memory buffer unit to a disk after the memory buffer unit is full, creating a new memory buffer unit, and continuously writing the data to be sent into the memory buffer unit. The method adapts to the characteristic that the network transmission environment changes continuously, keeps high throughput when the network is unobstructed, does not interrupt the process when the network is blocked, and always keeps less resource occupation. The problems of large resource consumption, inconvenience in deployment and the like in the data caching process of the sending end are solved, low-risk and simplified data caching which is suitable for various network environments is achieved, and the reliability of a program is improved.

Other characteristic features and advantages of the invention will become apparent from the following description of exemplary embodiments, which is to be read in connection with the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention. In the drawings, like reference numerals are used to indicate like elements. The drawings in the following description are directed to some, but not all embodiments of the invention. For a person skilled in the art, other figures can be derived from these figures without inventive effort.

Fig. 1 schematically shows a flow of a data transmission method provided by an embodiment of the present invention;

FIG. 2 schematically shows a detailed flow of step 101 in FIG. 1;

FIG. 3 illustrates a specific conventional buffer cell and buffer implementation;

FIG. 4 schematically shows a detailed flow of step 102 in FIG. 1;

FIG. 5 illustrates a specific implementation of creating a temporary buffer unit;

FIG. 6 illustrates an implementation of serializing data in a buffer to disk;

fig. 7 exemplarily shows a flow of still another data transmission method provided by an embodiment of the present invention;

FIG. 8 illustrates one particular implementation of the sending component reading the buffer location data;

fig. 9 exemplarily shows a structure of a data transmission apparatus provided by an embodiment of the present invention;

fig. 10 exemplarily shows a structure of the buffer unit management module 901 in fig. 9;

FIG. 11 exemplarily shows a structure of the data persistence management module 902 in FIG. 9;

fig. 12 exemplarily shows a structure of a data transmission apparatus provided by an embodiment of the present invention;

fig. 13 exemplarily shows a structure of still another data transmission apparatus provided by an embodiment of the present invention;

Fig. 14 exemplarily shows a structure of a computer device provided by an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.

In data cross-network transmission, due to the reasons of network quality deterioration, operator cutover, network failure and the like, the data transmission speed is slow, and even the transmission is interrupted. When the data sending end passively receives data to be sent, newly added data can occupy a large amount of memory space, and even OOM is triggered to cause process crash. Even if the process of the sending end is not broken down, the cache data to be sent occupy a large amount of memory and can also influence the service of other processes.

In order to solve the above problem, embodiments of the present invention provide a data transmission method, an apparatus, and a computer device. Creating a memory buffer unit in a memory, writing data to be sent into the memory buffer unit, serializing the data stored in the memory buffer unit to a disk after the memory buffer unit is full, creating a new memory buffer unit, and continuously writing the data to be sent into the memory buffer unit. The method adapts to the characteristic that the network transmission environment changes continuously, keeps high throughput when the network is unobstructed, does not interrupt the process when the network is blocked, and always keeps less resource occupation.

An embodiment of the present invention provides a data transmission method, where a flow of completing data transmission using the method is shown in fig. 1, and the method includes:

step 101, creating a memory buffer unit in a memory, and writing data to be sent into the memory buffer unit.

As shown in fig. 2, the steps include:

in step 1011, a memory buffer unit is created in the memory.

In this step, an upper limit of the capacity of the memory buffer unit is defined to limit memory resources occupied by the buffered data and guarantee memory usage of other processes. Preferably, at most two memory buffer units are created and exist in the system, and normally, one memory buffer unit exists in the memory for data caching.

For example, a Box is a fixed-size memory space as a buffer unit, and a FIFO queue data structure is used inside the Box.

As shown in fig. 3, Box1 is a memory buffer unit, and is connected to the receiving element and the sending element to receive and send data to be sent.

Step 1012, writing the data to be sent to the memory buffer unit.

And 102, after the memory buffer unit is fully written, serializing the data stored in the memory buffer unit to a disk.

As shown in fig. 4, the present step includes:

step 1021, after the memory buffer unit is full, serializing the data stored in the memory buffer unit into a disk buffer unit on a disk.

Step 1022, the memory occupied by the memory buffer unit is released.

Specifically, after the memory buffer unit Box1 is full, a new memory buffer unit Box2 is created, and data to be sent is continuously written into the memory buffer unit Box 2.

When the network is open, only one Box1 generally exists in the memory, the receiving component writes data into the Box1, the sending component reads data from the Box1, the data is dequeued faster than the data is entered into the queue, and the queue is not filled up all the time. When the queue is empty, the sending component may attempt to connect to the next Box, Box2, as shown in FIG. 5. If Box2 does not exist, Box1 is still connected to wait for writing, at this time, there is a timeout decision, and after the timeout, the next Box is retried or the wait continues.

When network congestion or interruption occurs, the data dequeuing speed is lower than the data enqueuing speed, Box1 is fully written, a new Box2 is created as a temporary buffer unit at the moment, writing continues, and the Box set is a linked list structure.

When Box2 is full, and no sending component is connected to Box2 at this time, if new data is received, the full Box2 is serialized into a disk buffer unit, as shown in FIG. 6.

Step 103, creating a new memory buffer unit, and continuing to write the data to be sent into the memory buffer unit.

In the embodiment of the present invention, preferably, an upper limit of the disk buffer data may be set, and when the data amount serialized to the disk reaches the upper limit of the disk buffer data, the creation of a new memory buffer unit is stopped. Therefore, the disk space occupied by the cache data is limited, and the utilization of the disk space is reasonable.

Preferably, the memory buffer unit in the memory and the disk buffer unit on the disk form a linked list structure according to a creation time sequence. When two memory buffer units exist in the memory, the head and the tail of the linked list are both the memory buffer units, and the middle is just the disk buffer unit.

Specifically, as shown in fig. 6, after Box2 is serialized to disk, a new Box3 is created, data is continuously written, if the network is continuously blocked, Box4 and Box5 … … BoxN are continuously created, and the program exits until the set upper limit of Box number in disk or the disk space is insufficient.

An embodiment of the present invention further provides a data transmission method, where a flow of taking out and sending data to be sent by a sending component is shown in fig. 7, and the flow includes:

step 701, when data to be sent exists in the memory buffer unit, taking out and sending the data to be sent in the memory buffer unit.

Step 702, when the memory buffer unit is empty, go to the next memory buffer unit pointed by the memory buffer unit.

As shown in FIG. 8, when Box 1's data is empty, the sending component may attempt to read Box 2.

And step 703, taking out and sending the data to be sent in the next memory buffer unit. And releasing the memory buffer unit after all the data to be sent in the next memory buffer unit is taken out.

Step 704, when the memory buffer unit is empty and the memory buffer unit points to a disk buffer unit, turning to the next disk buffer unit pointed by the memory buffer unit.

Step 705, releasing the memory occupied by the memory buffer unit, and deserializing the next disk buffer unit into a new memory buffer unit in the memory.

Step 706, the data to be sent in the new memory buffer unit is taken out and sent.

Specifically, if the next Box2 is persisted in the disk, the memory of the current Box1 is released, and the Box2 is loaded from the disk into the memory; if the next Box2 is the Box in memory, the Box2 is read directly.

An embodiment of the present invention further provides a data transmission apparatus, whose structure is shown in fig. 9, including:

a buffer unit management module 901, configured to create a memory buffer unit in a memory, and write data to be sent into the memory buffer unit; and releasing the memory space occupied by the memory buffer unit and maintaining the linked list relationship of the memory buffer unit and the disk buffer unit.

A data persistence management module 902, configured to serialize all data in the memory buffer unit into a disk buffer unit after the memory buffer unit is full.

Preferably, as shown in fig. 10, the buffer unit management module 901 includes:

a buffer unit creating unit 9011, configured to create a memory buffer unit in a memory, define an upper limit of a capacity of the memory buffer unit,

after the memory buffer unit is fully written, at most one new memory buffer unit is created;

and the data writing unit 9012 is configured to write data to be sent into the memory buffer unit, and continue to write data to be sent into the newly-built memory buffer unit after the memory buffer unit is full.

A memory release unit 9013, configured to release a memory space originally occupied by the memory buffer unit;

a data reading unit 9014, configured to read data in the memory buffer unit, and is responsible for blocking data writing in the waiting memory buffer unit;

and the buffer unit linked list management unit 9015 is used for controlling data serialization and deserialization, and forming a linked list structure by the memory buffer unit and the disk buffer unit according to a created time sequence.

Preferably, the structure of the data persistence management module 902 is shown in fig. 11, and includes:

a data persistence unit 9021, configured to serialize the memory buffer unit that is fully written into a disk buffer unit after the memory buffer unit is fully written into the disk buffer unit;

and the data deserializing unit 9022 is used for deserializing the disk buffer unit into a memory buffer unit.

An embodiment of the present invention further provides a data transmission apparatus, whose structure is shown in fig. 12, including:

a memory buffer management module 1201, configured to create a memory buffer unit in a memory, write data to be sent into the memory buffer unit, create a new memory buffer unit after the memory buffer unit is written with a full sequence to a disk, and continue to write data to be sent into the memory buffer unit;

A data persistence management module 1202, configured to serialize, to a disk, data stored in the memory buffer unit after the memory buffer unit is full.

Preferably, the memory buffer management module 1201 is further configured to, when data to be sent exists in the memory buffer unit, take out the data to be sent in the memory buffer unit.

When the memory buffer unit is empty and only one memory buffer unit exists in the memory, the sending component waits until new data is written in the memory buffer unit;

when the memory buffer unit is empty, turning to the next memory buffer unit pointed by the memory buffer unit, releasing the memory space occupied by the memory buffer unit, and taking out the data to be sent in the next memory buffer unit, or,

when the memory buffer unit is empty and the memory buffer unit points to a disk buffer unit, the next disk buffer unit pointed by the memory buffer unit is switched to, the memory occupied by the memory buffer unit is released, the next disk buffer unit is deserialized into a new memory buffer unit in the memory, and the data to be sent in the new memory buffer unit is taken out.

Preferably, the structure of the data transmission device is as shown in fig. 13, and further includes:

a data sending module 1203, configured to trigger the memory buffer management module 1201 to fetch data when receiving a data obtaining request, and send the fetched data.

An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, which when executed implements the steps of the data transmission method as provided by the embodiment of the present invention.

An embodiment of the present invention further provides a computer device, whose structure is shown in fig. 14, and includes a processor, a memory, and a computer program stored in the memory, where the processor executes the computer program to implement the steps of the data transmission method provided in the embodiment of the present invention.

The embodiment of the invention provides a data transmission method, a data transmission device, a computer readable storage medium and computer equipment. Creating a memory buffer unit in a memory, writing data to be sent into the memory buffer unit, serializing the data stored in the memory buffer unit to a disk after the memory buffer unit is full, creating a new memory buffer unit, and continuously writing the data to be sent into the memory buffer unit. The method adapts to the characteristic that the network transmission environment changes continuously, keeps high throughput when the network is unobstructed, does not interrupt the process when the network is blocked, and always keeps less resource occupation. The problems of large resource consumption, inconvenience in deployment and the like in the data caching process of the sending end are solved, low-risk and simplified data caching which is suitable for various network environments is achieved, and the reliability of a program is improved.

When the sending component is blocked, the process occupies the memory space of 2 buffer units at most, the rest newly added data can be written into the disk in sequence, and the process can be stopped only when the disk is full. The risk of process exit is reduced, subsequent operations caused by network blockage or interruption are reduced, and the program reliability is improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus (device), or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied in the medium. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer, and the like. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of additional like elements in the article or device comprising the element.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, it is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.