File transmission method and device
阅读说明：本技术 一种文件传输方法及装置 (File transmission method and device ) 是由 陈青龙 陈影旺 吴小云 于 2018-07-09 设计创作，主要内容包括：本发明公开了一种文件传输方法及装置。所述方法包括：针对所述节点服务器集群中的任一节点服务器,该节点服务器可以先根据接收到的上传文件生成对应的元信息,然后将上传文件及元信息发送给云存储设备,以使云存储设备进行存储。如此,一方面,客户端集群中的多个客户端设备可以同时将上传文件发送给节点服务器集群,从而能够实现文件快速传输的目的,提高文件传输的速率；另一方面,各个节点服务器在接收到客户端设备发送的文件后,可以将文件同步存储到云存储设备中的同一存储位置,从而使得用户能够直接通过云存储设备对文件进行管理,便于用户对文件进行集中管理。(The invention discloses a file transmission method and device. The method comprises the following steps: for any node server in the node server cluster, the node server may generate corresponding meta information according to the received upload file, and then send the upload file and the meta information to the cloud storage device, so that the cloud storage device stores the meta information. Therefore, on one hand, a plurality of client devices in the client cluster can simultaneously send the uploaded files to the node server cluster, so that the aim of quickly transmitting the files can be fulfilled, and the file transmission rate is improved; on the other hand, after receiving the file sent by the client device, each node server can synchronously store the file to the same storage position in the cloud storage device, so that a user can directly manage the file through the cloud storage device, and the user can conveniently perform centralized management on the file.)
1. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
aiming at any node server in the node server cluster, the node server receives an upload file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the node server generates meta information of the uploaded file, wherein the meta information comprises a data main body to which the uploaded file belongs;
the node server sends the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information and stores the meta information and the uploaded file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.
2. The method of claim 1, wherein the node server sends the upload file and the meta information to the cloud storage device, and wherein the sending comprises:
the node server sends a synchronization request to the cloud storage device, wherein the synchronization request comprises the meta information;
after receiving a response message of the synchronization request returned by the cloud storage device, the node server sends the upload file to the cloud storage device; the response information is sent when the cloud storage device determines that available transmission bandwidth exists.
3. The method according to claim 1 or 2, wherein after the node server sends the upload file and the meta information to the cloud storage device, the method further comprises:
and the node server deletes the uploaded file and the meta information.
4. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
for any client device in the client cluster, the client device sends a download request to the cloud storage device, wherein the download request comprises a target file identifier;
the client device receives a response message of the downloading request sent by the cloud storage device, and acquires the target file from a target node server according to the position information of the target node server included in the response message;
the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored; the correspondence is determined from the meta-information.
5. The method of claim 4, further comprising:
and the client device receives a target file sent by the cloud storage device, wherein the target file is a file stored in the cloud storage device and corresponding to the target file identifier.
6. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
the cloud storage equipment receives an uploaded file sent by any node server in the node server cluster and meta information of the uploaded file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the cloud storage equipment determines a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set;
and the cloud storage equipment stores the uploaded file and the meta information to the target storage area.
7. The method according to claim 6, wherein the receiving, by the cloud storage device, the uploaded file sent by any node server in the node server cluster and the meta information of the uploaded file comprises:
the cloud storage device receives a synchronization request sent by the node server, wherein the synchronization request comprises the meta information;
after determining that the available transmission bandwidth exists, the cloud storage device returns a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the upload file to the cloud storage device;
and the cloud storage equipment receives the uploaded file sent by the node server.
8. The method of claim 6, further comprising:
the cloud storage device receives a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;
if the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;
the cloud storage device sends a response message of the downloading request to the client, the response message comprises the position information of the target node server, and the response message is used for indicating the client device to acquire the target file from the target node server according to the position information of the target node server.
9. The method of claim 8, further comprising:
and if the cloud storage equipment determines that the target file corresponding to the target file identifier is stored, sending the target file to the client.
10. A server, characterized in that the server comprises:
the receiving and sending unit is used for receiving the uploading file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the processing unit is used for generating meta information of the uploaded file, and the meta information comprises a data main body to which the uploaded file belongs;
the receiving and sending unit is configured to send the upload file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and stores the meta information and the upload file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.
11. The server according to claim 10, wherein the transceiver unit is specifically configured to:
sending a synchronization request to the cloud storage device, the synchronization request including the meta information; after receiving a response message of the synchronization request returned by the cloud storage device, sending the upload file to the cloud storage device; the response information is sent when the cloud storage device determines that available transmission bandwidth exists.
12. The server according to claim 10 or 11, wherein the processing unit is further configured to:
and deleting the uploaded file and the meta information.
13. A client device, the client device comprising:
a sending unit, configured to send a download request to the cloud storage device, where the download request includes a target file identifier;
a receiving unit, configured to receive a response message of the download request sent by the cloud storage device, and obtain the target file from a target node server according to location information of the target node server included in the response message;
the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored; the correspondence is determined from the meta-information.
14. The client device of claim 13, wherein the receiving unit is further configured to:
and receiving a target file sent by the cloud storage equipment, wherein the target file is a file stored in the cloud storage equipment and corresponding to the target file identifier.
15. A cloud storage device, the cloud storage device comprising:
the receiving and sending unit is used for receiving an uploading file sent by any node server in the node server cluster and meta information of the uploading file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the processing unit is used for determining a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set; and storing the uploaded file and the meta information to the target storage area.
16. The cloud storage device of claim 15, wherein the transceiver unit is specifically configured to:
receiving a synchronization request sent by the node server, wherein the synchronization request comprises the meta information; after determining that the available transmission bandwidth exists, returning a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the uploaded file to the cloud storage device; and receiving the uploading file sent by the node server.
17. The cloud storage device of claim 15, wherein the transceiving unit is further configured to:
receiving a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;
the processing unit is further to:
if the target file corresponding to the target file identifier is determined not to be stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;
the receiving and sending unit is further configured to send a response message of the download request to the client, where the response message includes location information of the target node server, and the response message is used to instruct the client device to acquire the target file from the target node server according to the location information of the target node server.
18. The cloud storage device of claim 17, wherein the transceiving unit is further configured to:
and if the target file corresponding to the target file identifier is determined to be stored, sending the target file to the client.
19. A computer-readable storage medium, characterized in that the storage medium stores instructions that, when executed on a computer, cause the computer to carry out performing the method of any one of claims 1 to 3, or the method of claim 4 or 5, or the method of any one of claims 6 to 9.
20. A computer device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory to execute the method of any one of the claims in accordance with the obtained program.
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a file transmission method and apparatus.
Background
With the rapid development of internet technology, the amount of network information data is also getting larger and larger, and the requirement of enterprises for storing mass data becomes more important. The traditional file storage system has the problems of limited disk space, insufficient processing capacity and the like, and cannot meet the storage requirement of a large capacity of a user.
At present, an existing solution is to upload files to be stored by a user to a plurality of node servers in a distributed manner, so as to meet the storage requirement of the user for a large capacity. For example, if a user needs to upload 1TB files, 500GB of the files may be stored in node server a and another 500GB of files may be stored in node server B by using a distributed method. However, since the files are distributed in different node servers, when a user manages the files, a part of the files stored in the node server a and a part of the files stored in the node server B need to be managed at the same time, and thus, the requirement of centralized management by the user cannot be met.
The other solution is to adopt a centralized cloud storage service to upload files to be stored by a user to a cloud storage device in a centralized manner. However, although this method can meet the requirement of the user for centralized management of the files, the uploading rate that the cloud storage device can provide is limited, and therefore, the requirement of the user for uploading the files at high concurrency and high speed cannot be met under the condition that the data size of the files is excessively large.
Based on this, there is a need for a file transmission method, which is used to solve the problem in the prior art that quick uploading and centralized management cannot be considered in the file transmission process.
Disclosure of Invention
The embodiment of the invention provides a file transmission method and device, and aims to solve the technical problem that distributed storage of data in the prior art cannot meet the requirement of centralized management of users.
The embodiment of the invention provides a file uploading method, which is applied to a distributed storage system, wherein the distributed storage system comprises a cloud storage device, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
aiming at any node server in the node server cluster, the node server receives an upload file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the node server generates meta information of the uploaded file, wherein the meta information comprises a data main body to which the uploaded file belongs;
the node server sends the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information and stores the meta information and the uploaded file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.
Therefore, for services with large data volume and transmission delay requirements, on one hand, a plurality of client devices in a client cluster can simultaneously send the uploaded file set of the same data main body to a node server cluster, and a plurality of node servers in the node server cluster all provide available bandwidth to receive the uploaded file set sent by the client cluster, so that the aim of rapidly transmitting files can be achieved, and the file transmission rate is improved; on the other hand, in order to ensure that data of the service is convenient and fast to manage in the later period, each node server can synchronously store the uploaded files of the same data main body in the cloud storage device after receiving the uploaded files sent by the client device. The analysis shows that the data uploading service which can meet the transmission delay requirement can be provided for the same data main body, and the requirement of centralized storage can be met.
In a possible implementation manner, the sending, by the node server, the upload file and the meta information to the cloud storage device includes:
the node server sends a synchronization request to the cloud storage device, wherein the synchronization request comprises the meta information;
after receiving a response message of the synchronization request returned by the cloud storage device, the node server sends the upload file to the cloud storage device; the response information is sent after the cloud storage device determines to receive the meta information after storing the meta information.
Therefore, the node server can send the meta information to the cloud storage device, and the meta information is smaller in data volume compared with the uploaded file, so that the meta information can be rapidly transmitted to the cloud storage device, and the uploaded file is transmitted under the condition of enough bandwidth, and the file transmission efficiency is improved.
In a possible implementation manner, after the node server sends the upload file and the meta information to the cloud storage device, the method further includes:
and the node server deletes the uploaded file and the meta information. Therefore, the storage burden of the node server can be reduced, and the problem that the content of the file stored in the cloud storage device and the content of the file stored in the node server are repeated is avoided.
The embodiment of the invention provides a file downloading method, which is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
for any client device in the client cluster, the client device sends a download request to the cloud storage device, wherein the download request comprises a target file identifier;
the client device receives a response message of the downloading request sent by the cloud storage device, and acquires the target file from a target node server according to the position information of the target node server included in the response message;
and the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored.
Therefore, when the client device downloads the target file, if the file is not synchronously stored in the cloud storage device, the client device can still download the target file from the corresponding node server, so that the situation that the target file cannot be downloaded by the client device in the file synchronization process or the file is not synchronized is avoided.
In one possible implementation, the method further includes:
and the client device receives a target file sent by the cloud storage device, wherein the target file is a file stored in the cloud storage device and corresponding to the target file identifier.
Therefore, when the client device downloads the target file, if the file is synchronously stored in the cloud storage device, the client device can directly acquire the target file.
The embodiment of the invention provides a file uploading method, which is applied to a distributed storage system, wherein the distributed storage system comprises a cloud storage device, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:
the cloud storage equipment receives an uploaded file sent by any node server in the node server cluster and meta information of the uploaded file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the cloud storage equipment determines a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set;
and the cloud storage equipment stores the uploaded file and the meta information to the target storage area.
Therefore, on one hand, a plurality of client devices in the client cluster can simultaneously send the uploaded files to the node server cluster, so that the aim of quickly transmitting the files can be fulfilled, and the file transmission rate is improved; on the other hand, after receiving the file sent by the client device, each node server can synchronously store the file in the cloud storage device, and compared with the mode that the files sent by a plurality of client devices are stored in a plurality of node servers in a distributed manner in the prior art, the file management method and the system in the embodiment of the invention have the advantage that the user can directly manage the file in the cloud storage device, so that the user can conveniently manage the file in a centralized manner.
In a possible implementation manner, the receiving, by the cloud storage device, an upload file sent by any node server in the node server cluster and meta information of the upload file include:
the cloud storage device receives a synchronization request sent by the node server, wherein the synchronization request comprises the meta information;
after determining that the available transmission bandwidth exists, the cloud storage device returns a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the upload file to the cloud storage device;
and the cloud storage equipment receives the uploaded file sent by the node server.
Therefore, the node server can send the meta information to the cloud storage device, and the meta information is smaller in data volume compared with the uploaded file, so that the meta information can be rapidly transmitted to the cloud storage device, and the uploaded file is transmitted under the condition of enough bandwidth, and the file transmission efficiency is improved.
In one possible implementation, the method further includes:
the cloud storage device receives a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;
if the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;
the cloud storage device sends a response message of the downloading request to the client, the response message comprises the position information of the target node server, and the response message is used for indicating the client device to acquire the target file from the target node server according to the position information of the target node server.
Therefore, when the client device downloads the target file, if the file is not synchronously stored in the cloud storage device, the client device can still download the target file from the corresponding node server, so that the situation that the target file cannot be downloaded by the client device in the file synchronization process or the file is not synchronized is avoided.
In one possible implementation, the method further includes:
and if the cloud storage equipment determines that the target file corresponding to the target file identifier is stored, sending the target file to the client.
Therefore, when the client device downloads the target file, if the file is synchronously stored in the cloud storage device, the client device can directly acquire the target file.
An embodiment of the present invention provides a server, where the server includes:
the receiving and sending unit is used for receiving the uploading file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the processing unit is used for generating meta information of the uploaded file, and the meta information comprises a data main body to which the uploaded file belongs;
the receiving and sending unit is configured to send the upload file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and stores the meta information and the upload file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.
In a possible implementation manner, the transceiver unit is specifically configured to:
sending a synchronization request to the cloud storage device, the synchronization request including the meta information; after receiving a response message of the synchronization request returned by the cloud storage device, sending the upload file to the cloud storage device; the response information is sent when the cloud storage device determines that available transmission bandwidth exists.
In one possible implementation, the processing unit is further configured to:
and deleting the uploaded file and the meta information.
An embodiment of the present invention provides a client device, where the client device includes:
a sending unit, configured to send a download request to the cloud storage device, where the download request includes a target file identifier;
a receiving unit, configured to receive a response message of the download request sent by the cloud storage device, and obtain the target file from a target node server according to location information of the target node server included in the response message;
the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored; the correspondence is determined from the meta-information.
In one possible implementation manner, the receiving unit is further configured to:
and receiving a target file sent by the cloud storage equipment, wherein the target file is a file stored in the cloud storage equipment and corresponding to the target file identifier.
An embodiment of the present invention provides a cloud storage device, where the cloud storage device includes:
the receiving and sending unit is used for receiving an uploading file sent by any node server in the node server cluster and meta information of the uploading file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;
the processing unit is used for determining a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set; and storing the uploaded file and the meta information to the target storage area.
In a possible implementation manner, the transceiver unit is specifically configured to:
receiving a synchronization request sent by the node server, wherein the synchronization request comprises the meta information; after determining that the available transmission bandwidth exists, returning a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the uploaded file to the cloud storage device; and receiving the uploading file sent by the node server.
In a possible implementation manner, the transceiver unit is further configured to:
receiving a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;
the processing unit is further to:
if the target file corresponding to the target file identifier is determined not to be stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;
the receiving and sending unit is further configured to send a response message of the download request to the client, where the response message includes location information of the target node server, and the response message is used to instruct the client device to acquire the target file from the target node server according to the location information of the target node server.
In a possible implementation manner, the transceiver unit is further configured to:
and if the target file corresponding to the target file identifier is determined to be stored, sending the target file to the client.
The embodiment of the present application further provides an apparatus, which has a function of implementing the file transmission method described above. This function may be implemented by hardware executing corresponding software, and in one possible design, the apparatus includes: a processor, a transceiver, a memory; the memory is used for storing computer execution instructions, the transceiver is used for realizing the communication between the device and other communication entities, the processor and the memory are connected through the bus, and when the device runs, the processor executes the computer execution instructions stored in the memory so as to enable the device to execute the file transmission method described above.
An embodiment of the present invention further provides a computer storage medium, where a software program is stored, and when the software program is read and executed by one or more processors, the software program implements the file transmission method described in the foregoing various possible implementation manners.
Embodiments of the present invention further provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the file transfer method described in the foregoing various possible implementations.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that are required to be used in the description of the embodiments will be briefly described below.
FIG. 1 is a diagram illustrating a system architecture suitable for use with an embodiment of the present invention;
fig. 2 is a schematic flow chart corresponding to a file transmission method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a method for uploading a file by a client device according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another system architecture according to an embodiment of the present invention;
fig. 5 is a schematic flowchart corresponding to a method for synchronizing files by a node server according to an embodiment of the present invention;
fig. 6 is a schematic flowchart illustrating another file transmission method according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of an overall process involved in a second embodiment of the present invention;
FIG. 8 is a block diagram of a server according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a client device according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a cloud storage device according to an embodiment of the present invention.
Detailed Description
The present application will be described in detail below with reference to the accompanying drawings, and the specific operation methods in the method embodiments can also be applied to the apparatus embodiments.
Fig. 1 exemplarily shows a system architecture schematic diagram applicable to an embodiment of the present invention, and as shown in fig. 1, a system architecture 100 applicable to an embodiment of the present invention includes a
It should be noted that the client device in the present application may be various types of devices, for example, a notebook computer, a smart phone, a tablet computer, a smart television, and the like, and is not limited specifically.
- 上一篇：一种医用注射器针头装配设备
- 下一篇：一种数据读写方法、终端及计算机可读存储介质