阅读说明：本技术 一种分布式文件交换方法及装置 (Distributed file exchange method and device ) 是由 王济平 黎刚 汤克云 周健雄 刘子焯 于 2020-09-28 设计创作，主要内容包括：本发明涉及一种分布式文件交换方法及装置。文件交换方法包括以下步骤：步骤S1,建立交换服务协议并定时获取相关信息；步骤S2,提供方将文件基础信息上传至交换子系统并进行校验；步骤S3,提供方将文件分块上传至交换子系统,并对块的信息校验,最终把所有文件块合并；步骤S4,接收方获取文件基础信息并与交换子系统建立交换服务,交换子系统将文件分块传输至接收方,完成传输后将所有文件块合并。本发明通过分块、分布式文件交换,避免了文件传输中出现资源瓶颈的问题,提高了传输效率、满足了大量传输的需求,实现了文件断点续传,解决了文件重传耗时长的问题；同时系统中业务表也与文件基础信息关联,提供了可靠的文件交换服务。(The invention relates to a distributed file exchange method and a distributed file exchange device. The file exchange method comprises the following steps: step S1, establishing an exchange service protocol and acquiring related information at regular time; step S2, the provider uploads the basic information of the file to the exchange subsystem and verifies the basic information; step S3, the provider uploads the file blocks to the exchange subsystem, checks the information of the blocks, and finally merges all the file blocks; and step S4, the receiver acquires the basic information of the file and establishes the exchange service with the exchange subsystem, the exchange subsystem transmits the file blocks to the receiver, and all the file blocks are merged after the transmission is finished. According to the invention, through block and distributed file exchange, the problem of resource bottleneck in file transmission is avoided, the transmission efficiency is improved, the requirements of mass transmission are met, the breakpoint continuous transmission of the file is realized, and the problem of long time consumption in file retransmission is solved; meanwhile, the service table in the system is also associated with the file basic information, and reliable file exchange service is provided.)

1. A distributed file exchange method is characterized in that a file provider, a file receiver, a switching center and a distributed exchange subsystem are involved, and the file exchange method comprises the following steps:

step S1, establishing an exchange service protocol and acquiring related information at regular time;

step S2, the provider uploads the basic information of the file to the exchange subsystem and verifies the basic information;

step S3, the provider uploads the file blocks to the exchange subsystem, checks the information of the blocks, and finally merges all the file blocks;

and step S4, the receiver acquires the basic information of the file and establishes the exchange service with the exchange subsystem, the exchange subsystem transmits the file blocks to the receiver, and all the file blocks are merged after the transmission is finished.

2. The distributed file exchange method of claim 1, wherein the step S1 includes: the provider establishes an exchange service protocol with the exchange subsystem in the exchange center, and the exchange subsystem acquires department directory information from the exchange center through timing scheduling and sets an associated field of file basic information.

3. The distributed file exchange method of claim 1, wherein the step S2 includes: a provider acquires a batch identifier and caches the batch identifier in a memory, and then uploads file basic information to a cache of an exchange subsystem; and after the batch is finished, the exchange subsystem acquires a file finishing batch instruction and stores the file basic information in the batch data in the cache.

4. The distributed file exchange method of claim 3, wherein the step S2 further comprises: after the provider finishes uploading the basic information of the file, the provider provides basic parameters of the uploaded file and establishes a file exchange service with the exchange subsystem; and the exchange subsystem compares and verifies the basic parameters of the uploaded files with the basic information of the files in the library.

5. The distributed file exchange method of claim 1, wherein the step S3 includes: the provider divides the file into blocks and uploads the blocks to a temporary directory of the exchange subsystem; and the exchange subsystem checks the information of the file blocks and caches the information of the file blocks to the distributed file storage system.

6. The distributed file exchange method of claim 5, wherein the step S3 further comprises: after all the file blocks are transmitted, carrying out final verification, storing the file blocks in a distributed file storage system and merging the file blocks; and then storing the file basic information into a front-end processor of the exchange subsystem and feeding back success information to the provider.

7. The distributed file exchange method of claim 6, wherein the step S4 includes:

the switching center switches the file basic information stored in the front-end processor of the switching subsystem to a receiving party;

the receiver takes the file basic information acquired from the switching center and the local storage directory information of the receiver as parameters and provides the parameters to the switching subsystem to establish the file switching service;

and the exchange subsystem compares and verifies the received parameters and the file information in the distributed file storage system, and if the verification is successful, the file is transmitted to a receiver in blocks.

8. The distributed file exchange method of claim 7, wherein the step S4 further comprises: the receiver stores the file blocks in a local storage directory provided by the receiver; when a completion instruction is received, the file transmission state in the modification cache is completed, the monitoring thread monitors state modification, and file blocks are merged.

9. An electronic device, comprising:

a processor; and

a memory having computer readable instructions stored thereon which, when executed by the processor, implement the method of any of claims 1 to 8.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 8.

Technical Field

The present invention relates to data exchange technologies, and in particular, to a distributed file exchange method and apparatus.

Background

At present, government departments and institutions in various regions of China more or less establish own informatization systems, including a portal site content management system, an OA office system, a work handling approval system, other business systems and the like. However, due to the influence of many factors, even government agencies in the same region cannot reasonably and effectively communicate with each other, so that the information communication is an information isolated island of one seat.

The government affair data sharing and exchanging platform is a government electronic government affair supporting platform, bridges are erected among a plurality of different government affair application systems, sharing of data resources is achieved, the problem of information isolated islands formed in the past government informatization process is effectively solved, cross-department and cross-organization transaction processing is achieved, and the platform is one-stop electronic government affair basic software.

In the traditional government affair file data sharing and exchanging process, a centralized file storage system is mainly adopted, and a storage server is adopted to store all data. The storage server becomes a bottleneck of system performance and cannot meet the requirements of large-scale storage application. When the file provides service to the outside, the file is separated from the record information of the service table, and a user cannot associate the corresponding file according to the service. File transfers are interrupted by network fluctuations and must be retransmitted. If the file is transmitted in the FTP protocol mode, when the FTP is used for file transmission, the file does not support the capabilities of automatic incremental file discovery, file breakpoint resuming due to network failure and the like, and the certificated affair data sharing and exchanging platform cannot monitor and manage the file transmission condition of the opposite end in real time, so that the real-time condition of the file exchanging and sharing task cannot be known.

In the government affair data exchange and sharing process, the data structured by database tables and the file data are exchanged as the main data source. The main modes in the file data exchange process are the FTP mode and the file uploading and downloading mode. The FTP mode is that a government affair data sharing exchange platform is adopted to set up an FTP server channel for a department front-end computer needing to carry out file exchange, and FTP files of a source front-end computer are transmitted to a target department file server front-end computer in a full amount in a timing scheduling mode through an FTP file transmission protocol. The FTP is used for file transmission, incremental files cannot be transmitted, breakpoint continuous transmission is not carried out on the files, and when a network is broken in the middle, connection service needs to be created again for retransmission. The FTP file transmission mainly carries out file transmission exchange end to end, and a government affair data exchange platform cannot carry out real-time monitoring and real-time management on the condition that each front-end computer of the FTP carries out file transmission exchange, so that reliable file exchange service cannot be provided. The file uploading and downloading service is mainly characterized in that a file providing department login platform manually uploads files through the last function of a government affair data sharing exchange platform, and a department is obtained and then logs in the platform to manually download the files. Through manual uploading and downloading, on one hand, efficient transmission of a large amount of file contents cannot be met, and the efficiency of the whole uploading and downloading process is extremely low.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provides a distributed file exchange method, which realizes the block and distributed file exchange, improves the transmission efficiency, meets the requirement of mass transmission and can also meet the requirement of breakpoint continuous transmission.

In order to achieve the purpose, the invention adopts the following technical scheme: a distributed file exchange method relates to a file provider, a file receiver, an exchange center and a distributed exchange subsystem, and comprises the following steps:

step S1, establishing an exchange service protocol and acquiring related information at regular time;

step S2, the provider uploads the basic information of the file to the exchange subsystem and verifies the basic information;

step S3, the provider uploads the file blocks to the exchange subsystem, checks the information of the blocks, and finally merges all the file blocks;

and step S4, the receiver acquires the basic information of the file and establishes the exchange service with the exchange subsystem, the exchange subsystem transmits the file blocks to the receiver, and all the file blocks are merged after the transmission is finished.

Step S1 includes: the provider establishes an exchange service protocol with the exchange subsystem in the exchange center, and the exchange subsystem acquires department directory information from the exchange center through timing scheduling and sets an associated field of file basic information.

Step S2 includes: a provider acquires a batch identifier and caches the batch identifier in a memory, and then uploads file basic information to an exchange subsystem; and after the batch is finished, the exchange subsystem acquires a file finishing batch instruction and stores the file basic information in the batch data in the cache.

Step S2 further includes: after the provider finishes uploading the basic information of the file, the provider provides basic parameters of the uploaded file and establishes a file exchange service with the exchange subsystem; and the exchange subsystem compares and verifies the basic parameters of the uploaded files with the basic information of the files in the library.

Step S3 includes: the provider divides the file into blocks and uploads the blocks to a temporary directory of the exchange subsystem; and the exchange subsystem checks the information of the file blocks and caches the information of the file blocks to the distributed file storage system.

Step S3 further includes: after all the file blocks are transmitted, carrying out final verification, storing the file blocks in a distributed file storage system and merging the file blocks; and then storing the file basic information into a front-end processor of the exchange subsystem and feeding back success information to the provider.

Step S4 includes:

the switching center switches the file basic information stored in the front-end processor of the switching subsystem to a receiving party;

the receiver takes the file basic information acquired from the switching center and the local storage directory information of the receiver as parameters and provides the parameters to the switching subsystem to establish the file switching service;

and the exchange subsystem compares and verifies the received parameters and the file information in the distributed file storage system, and if the verification is successful, the file is transmitted to a receiver in blocks.

Step S4 further includes: the receiver stores the file blocks in a local storage directory provided by the receiver; when a completion instruction is received, the file transmission state in the modification cache is completed, the monitoring thread monitors state modification, and file blocks are merged.

The invention also discloses an electronic device, comprising: the file exchange system comprises a processor and a memory, wherein computer readable instructions are stored on the memory, and when being executed by the processor, the computer readable instructions realize the file exchange method.

The invention also discloses a computer readable storage medium on which a computer program is stored, the computer program implementing the above file exchange method when executed by a processor.

Compared with the prior art, the invention has the beneficial effects that: by means of blocking and distributed file exchange, the problem of resource bottleneck in file transmission is avoided, transmission efficiency is improved, the requirement for mass transmission is met, file breakpoint continuous transmission is realized, and the problem of long time consumption for file retransmission is solved; meanwhile, the service table in the system is also associated with the file basic information, and reliable file exchange service is provided.

Drawings

Fig. 1 is a diagram of the framework connection of the file exchange method according to the present invention with respect to each terminal.

FIG. 2 is a flowchart of a file exchange method according to the present invention.

It should be noted that, the products shown in the above views are all appropriately reduced/enlarged according to the size of the drawing and the clear view, and the size of the products shown in the views is not limited.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, devices, steps, and so forth. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in the form of software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.

The embodiment is a distributed file exchange method, and relates to a file provider, a file receiver, a switching center and a distributed exchange subsystem (as shown in fig. 1). The switching center is the switching platform of the whole government affairs system data, and the distributed switching subsystem is the system for specifically executing file switching.

As shown in fig. 2, the file exchange method includes the following steps: step S1, establishing an exchange service protocol and acquiring related information at regular time; step S2, the provider uploads the basic information of the file to the exchange subsystem and verifies the basic information; step S3, the provider uploads the file blocks to the exchange subsystem, checks the information of the blocks, and finally merges all the file blocks; and step S4, the receiver acquires the basic information of the file and establishes the exchange service with the exchange subsystem, the exchange subsystem transmits the file blocks to the receiver, and all the file blocks are merged after the transmission is finished.

The file exchange method of the embodiment avoids the problem of resource bottleneck in file transmission through blocking and distributed file exchange, improves transmission efficiency, meets the requirement of mass transmission, realizes file breakpoint continuous transmission, and solves the problem of long time consumption for file retransmission. Meanwhile, the service table in the system is also associated with the file basic information, and reliable file exchange service is provided.

The steps of the distributed file exchange method are described in detail below.

Step S1 is for establishing the exchange service agreement and periodically obtaining the relevant information. Specifically, in step S1, the provider establishes an exchange service agreement with the exchange subsystem at the switching center. The data transfer and file transfer protocols of the switching subsystem are self-defined protocols based on the TCP protocol under the Netty framework. The switching subsystem acquires department directory information from the switching center through timing scheduling and sets the associated fields of the basic information of the files. The department directory information refers to file directory information under each department of government affairs. The associated field of the file basic information at least comprises a file name, a path, a size and the like.

Step S2 is used to control the provider to upload the file basic information to the exchange subsystem and verify it. Specifically, in step S2, the provider acquires the batch identifier of the file to be uploaded and caches the batch identifier in the memory, and then uploads the file basic information of the file to be uploaded to the cache of the exchange subsystem. The file basic information comprises a file unique identifier, a file name, a file MD5, a file path, a target system identifier, a department name and a user name. And after the batch is finished, the provider finishes the batch uploading, and the exchange subsystem stores the file basic information in the batch data in the cache into the database after acquiring the file finishing batch instruction. After the provider finishes uploading the basic information of the file, the provider provides basic parameters of the file to be uploaded. The basic parameters of the file include: file path, target system identification, user name, file, and file unique identification. And finally, based on the associated field table record of the basic file information set in the step S1, the switching subsystem compares and verifies the basic parameters of the uploaded file and the basic file information stored in the database. If the comparison and verification are successful, the subsequent steps can be carried out. If the comparison and verification are unsuccessful, returning verification error information and recording the error information into the exchange log so as to inquire the file exchange state.

Step S3 is used to control the provider to upload the file blocks to the switching subsystem, check the information of the blocks, and finally merge all the file blocks. Specifically, in step S3, based on the basic parameters of the file verified successfully in step S2, the file verified successfully in step S2 is provided with a chunk size and then is chunked according to the chunk size. The file block is mainly composed of basic information such as a block number, a file name, a path, data content, and a data length. The provider then uploads each file block to the temporary directory of the switching subsystem. The switching subsystem checks the information of the file block, wherein the sequence number, path and file name of the file block need to be checked. And meanwhile, the exchange subsystem caches the verified information of the file blocks to the distributed file storage system. The file blocks and the file block information are transmitted together, but the file blocks are large in data volume and the unstructured data are not suitable for being stored in the database, the data pressure of the database is large due to the fact that the data are stored in the database, the query efficiency is low, and the file blocks and the file block information are stored separately because the data are required to be processed for a long time. The file block information is used for searching a file block storage directory and verifying integrity.

When all file blocks are completely transmitted, the switching subsystem detects a completion instruction and then performs final verification on the file blocks. The final check is to check the file name, the file path and the block number with the file block information in the cache to check whether the file block is missing. And then storing all the file blocks into the distributed file storage system, and carrying out merging operation on all the file blocks in the distributed file storage system through a merging instruction to form a complete file. And after the file merging operation is completed, storing the file basic information stored in the database in the step S2 into a front-end processor of the exchange subsystem, and feeding back operation success information to a provider.

Step S4 is used to control the receiving party to obtain the basic information of the file and establish the exchange service with the exchange subsystem, then the exchange subsystem transmits the file block to the receiving party, and after the transmission is completed, all the file blocks are merged. Specifically, in step S4, the switching center first exchanges the file basic information stored in the front-end processor of the switching subsystem to the receiving side. And then the receiver takes the file basic information acquired from the switching center and the local storage directory information of the receiver as parameters and provides the parameters to the switching subsystem so as to establish the file switching service. And finally, the exchange subsystem compares and verifies the file basic information in the received parameters with the file information in the database, and if the verification is successful, the exchange subsystem transmits the file blocks to a receiver. The receiver stores the received file blocks in a local storage directory provided by the receiver. When receiving the completion instruction, the file transmission state in the modification cache is completed, the monitoring thread monitors the state modification, and finally the file blocks are combined into a complete file to complete the file exchange service process.

In addition, in the embodiment of the present invention, an electronic device capable of implementing the file exchange method is also provided.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

The electronic device is in the form of a general purpose computing device. Components of the electronic device may include, but are not limited to: the system comprises at least one processing unit, at least one storage unit, a bus for connecting different system components (comprising the storage unit and the processing unit), and a display unit.

Wherein the storage unit stores program code which is executable by the processing unit to cause the processing unit to perform steps according to various exemplary embodiments of the present invention as described in the above section "exemplary methods" of the present description. For example, the processing unit may perform steps S1 through S4 of the file exchange method of the present invention.

The memory unit may include a readable medium in the form of a volatile memory unit, such as a random access memory unit (RAM) and/or a cache memory unit, and may further include a read only memory unit (ROM).

The storage unit may also include a program/utility having a set (at least one) of program modules including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

The bus may be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device may also communicate with one or more external devices (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface. Also, the electronic device may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via a network adapter. As shown, the network adapter communicates with other modules of the electronic device over a bus. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described file exchange method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above-mentioned "exemplary methods" section of the description, when the program product is run on the terminal device.

According to the program product for realizing the method, the portable compact disc read only memory (CD-ROM) can be adopted, the program code is included, and the program product can be operated on terminal equipment, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the following claims.