阅读说明：本技术 一种数据分发过程中的失败检测及恢复方法 (Failure detection and recovery method in data distribution process ) 是由 门殿春 江何 周鑫 李忱 陈忠国 孟繁荣 姚志强 于 2021-11-05 设计创作，主要内容包括：本发明公开了一种数据分发过程中的失败检测及恢复方法,具体涉及数据分发技术领域,通过采用RSA签名的秘钥处理,使其能够在公钥的同步发送过程中,实现数据的完整性验证,同时采用序列号反馈方式方式,实现数据的接收登记,实现丢包后能够实现对一个时间段内的丢包数据进行及时修复,使其能够实现数据的传输及时检测,以及完整性的有效检验,配合RSA数字签名完整性检测同时,其能够保障缓存节点及下层数据库内暂存的缓存文件进行很好的保护,且采用下层数据库配合对传输历史文件暂存的方式,使其能够实现一段时间内丢失数据恢复的直接处理,无需对发送端口发出数据重新调用请求,降低传输压力,提升整体的运作效率。(The invention discloses a failure detection and recovery method in the data distribution process, in particular to the technical field of data distribution, which can realize the integrity verification of data in the synchronous transmission process of a public key by adopting the secret key processing of RSA signature, simultaneously realize the receiving and registering of the data by adopting a serial number feedback mode, realize the timely repair of the lost data in a time period after the lost data is lost, realize the timely detection of the transmission of the data and the effective verification of the integrity, ensure that a buffer node and a buffer file temporarily stored in a lower database can be well protected by matching with the integrity detection of the RSA digital signature, and realize the direct processing of the recovery of the lost data in a period of time by matching with the lower database and transmitting a temporary storage mode of a history file without sending a data recall request to a transmitting port, reduce transmission pressure, promote holistic operating efficiency.)

1. A failure detection and recovery method in the data distribution process is characterized in that: the method comprises the following steps:

s1, reading dds distribution data, adding a unique serial number to the distribution data, performing RSA signature processing, and synchronously acquiring a public key and a private key of an RSA digital signature;

s2, establishing a data distribution cache node and a lower-layer database, wherein the cache node compresses or directly distributes data, and the synchronous data is recorded into the lower-layer database;

s3, a sequence receiving table is established by the receiving end and the cache node, and the public key and the distributed data are synchronously distributed and transmitted in the data distribution process of the cache node;

s4, verifying the integrity of the distribution data through a public key after the distribution data are received, recording a sequence receiving table through a serial number, and retrieving the sequence position of the sequence receiving table;

s5, if the sequence number retrieval is not in accordance with the actual receiving sequence, the receiving end recovers the data after a time period;

and S6, before the data recovery operation, the failure detection module is matched with the QoS service strategy in advance to carry out the overall detection of the system.

2. The method according to claim 1, wherein the method for detecting and recovering failure in data distribution process comprises: the data recovery process comprises the steps of:

s5.1, in the process of verifying the integrity of the public key data in S1, if dds data part is missing, the public key cannot be decrypted, and the receiving port which cannot be decrypted feeds back the public key;

s5.2, in the integrity detection process, detecting the increment change of the sequence number through the sequence receiving table;

s5.3, accessing the sequence number log and the sending port thereof in the last time period to the cache node, and comparing the sequence number log and the sending port with an actual receiving port receiving sequence receiving table to obtain missing data;

s5.4, after data loss occurs, detecting the distribution system by starting a failure detection module, immediately sending a missing file sequence port to a cache node by a receiving port, matching a temporarily stored missing file with a lower-layer database of the cache node, calling again and sending the missing file, and stopping response when the missing file is verified to be correct by a public key;

and S5.5, extracting error reporting information of the log after the failure detection module fails to repair, and feeding back through a manual window.

3. The method according to claim 2, wherein the method for detecting and recovering failure in data distribution process comprises: the failure detection module includes the following repair policies:

detecting a database instance;

verifying a client connection mode;

verifying the secure connection of the ssl connection mode;

verifying a connection command;

network detection;

connection number margin verification of the instance;

self-checking hardware of a magnetic disk and a processor;

log reading error reporting information extraction;

and feeding back error reporting information.

4. The method of claim 3, wherein the failure detection and recovery method comprises: the client connection mode verification comprises client operation detection, detection of whether dds and ecs instances are located in cpc or not, and synchronization detection of whether dds instances are bound into eip or not, and the ssl connection mode safety connection verification comprises operation of verifying the ssl switch opening state based on dds and uploading a root certificate to ecs again.

5. The method of claim 3, wherein the failure detection and recovery method comprises: the connection command verification comprises connection address verification, port parameter configuration verification, user name and password verification and ssl configuration item connection command verification, and the network detection comprises intranet access verification, public network access verification and cross-network access verification.

6. The method of claim 3, wherein the failure detection and recovery method comprises: and the error reporting information feedback and the log reading error reporting information are both recorded with fault history data and remarked with corresponding processing strategies.

7. The method according to claim 1, wherein the method for detecting and recovering failure in data distribution process comprises: the sequence number is an incremental type, the sequence format is date + time period + current day transmission number sequence, and the sequence receiving table of the cache node comprises transmission corresponding port data added with the corresponding sequence number.

8. The method according to claim 1, wherein the method for detecting and recovering failure in data distribution process comprises: the lower-layer database uses a MySQL relational database based on a memory storage engine.

Technical Field

The present invention relates to the field of data distribution technologies, and in particular, to a failure detection and recovery method in a data distribution process.

Background

The core of Data Distribution Services (DDS) is a Data-Centric Publish-Subscribe DCPS (Data-Centric publishing-subscription), i.e. the publisher can efficiently deliver the correct information to the appropriate subscribers, the DDS middleware is a software layer that abstracts the application from the details of the operating system, network transport, and underlying Data format. The DDS provides APIs in a variety of programming languages, allowing applications to exchange information across operating systems, across languages, and across processor architectures. The data format, the discovery matching mechanism, the connection mode, the reliability, the network protocol, the transmission mode selection, the QoS, the security and other bottom-layer details are managed by the middleware, and the DDS conceptually creates a local data storage space of a global data space. To the application, the global data space appears to be local memory accessed through the API: when the sending end application program writes data, the data is written to a local storage space; instead, a message is sent by the DDS to update the stored content on the remote node; the receiving end application program side can obtain the data by reading the local storage.

For the key dds data distribution with large capacity and small task amount, in the data distribution process, except for the distribution failure caused by common fault reasons, the packet loss phenomenon and the data partial loss caused by the network in the data transmission process are data transmission failures to a certain extent, however, the integrity cannot be effectively verified easily due to the data transmission loss, so that the integrity cannot be effectively guaranteed, and meanwhile, in the data packet loss process, the processing task amount can be increased in the recovery process, so that the normal operation efficiency is influenced.

Disclosure of Invention

In order to overcome the above defects in the prior art, the present invention provides a failure detection and recovery method in the data distribution process, and the technical problem to be solved by the present invention is: the missing problem caused by the data transmission process is very easy to ensure the integrity of the data transmission process under the condition that the integrity cannot be effectively verified, and meanwhile, the problem that the normal operation efficiency of a sending end is influenced by the increase of the processing task amount in the recovery process in the data packet loss process is solved.

In order to achieve the purpose, the invention provides the following technical scheme: a failure detection method in a data distribution process comprises the following steps:

s1, reading dds distribution data, adding a unique serial number to the distribution data, performing RSA signature processing, and synchronously obtaining a public key and a private key of the RSA digital signature.

S2, establishing a data distribution cache node and a lower-layer database, wherein the cache node compresses or directly distributes data, and synchronizes the data to be recorded in the lower-layer database.

And S3, the receiving end and the cache node establish a sequence receiving table, and the cache node synchronously distributes and transmits the public key and the distributed data in the data distribution process.

And S4, verifying the integrity of the distribution data through a public key after receiving the distribution data, recording the distribution data into a sequence receiving table through a serial number, and retrieving the sequence position of the distribution data.

S5, if the sequence number search does not match the actual receiving sequence, the receiving end performs data recovery after a period of time.

And S6, before the data recovery operation, the distribution system is detected in advance through the cooperation of a failure detection module and a QoS service strategy.

As a further scheme of the invention: the data recovery comprises the following steps:

and S5.1, in the process of verifying the integrity of the public key data in the S1, if the public key cannot be decrypted if the dds data part is missing, the public key is fed back by the receiving port which cannot be decrypted.

And S5.2, detecting the incremental change of the sequence number through the sequence receiving table in the integrity detection process.

And S5.3, accessing the sequence number log and the sending port thereof in the last time period to the cache node, and comparing the sequence number log with an actual receiving port receiving sequence receiving table to obtain missing data.

And S5.4, after data loss occurs, detecting the distribution system by starting the failure detection module, immediately sending the missing file sequence port to the cache node by the receiving port, matching the missing file sequence port with a lower-layer database of the cache node to obtain a temporarily stored missing file, calling again and sending the temporarily stored missing file, and stopping responding after the missing file is verified to be correct by the public key.

And S5.5, extracting error reporting information of the log after the failure detection module fails to repair, and feeding back through a manual window.

As a further scheme of the invention: the failure detection module includes the following repair policies:

a. and detecting the database instance.

b. And verifying the connection mode of the client.

c. And (4) performing safe connection verification by using the ssl connection mode.

d. And verifying the connection command.

e. And (5) detecting the network.

f. Connection number margin verification for the example.

g. And self-checking hardware of the magnetic disk and the processor.

h. And log reading error reporting information extraction.

i. And feeding back error reporting information.

As a further scheme of the invention: the client connection mode verification specifically comprises client operation detection, detection of whether dds and ecs instances are located in a cpc, and synchronization detection of whether dds instances are bound to eip, and the ssl connection mode secure connection verification specifically comprises operations of verifying the ssl switch opening state based on dds and uploading a root certificate to ecs again.

As a further scheme of the invention: the connection command verification comprises connection address verification, port parameter configuration verification, user name and password verification and ssl configuration item connection command verification, and the network detection comprises intranet access verification, public network access verification and cross-network access verification.

As a further scheme of the invention: the sequence number is an incremental type, the sequence format is date + time period + current day transmission number sequence, and the sequence receiving table of the cache node comprises transmission corresponding port data added with the corresponding sequence number.

As a further scheme of the invention: the lower-layer database uses a MySQL relational database based on a memory storage engine.

As a further scheme of the invention: and the error reporting information feedback and the log reading error reporting information are both recorded with fault history data and remarked with corresponding processing strategies.

The invention has the beneficial effects that:

the invention realizes the integrity verification of data in the synchronous sending process of the public key by adopting the secret key processing of RSA signature, simultaneously adopts a serial number feedback mode to realize the receiving and registering of the data, can realize the timely repair of the lost data in a time period after the lost packet is realized, provides the starting condition of a failure detection module, increases the self-checking repair strategy of the system, can realize the coordination with a QoS service strategy, improves the integrity of the whole, can realize the timely detection of the transmission of the data and the effective inspection of the integrity, is convenient for the recovery of data distribution, can ensure that the buffer node and the buffer files temporarily stored in a lower database are well protected by matching with the integrity detection of the RSA digital signature, plays a certain data encryption effect, reduces the risk that the temporarily stored files are excessively attacked and leaked, and adopts the lower database to coordinate with the mode of temporarily storing the transmission history files, the method and the device can realize direct processing of lost data recovery within a period of time, do not need to send a data recall request to the sending port, reduce transmission pressure and improve the overall operation efficiency.

Drawings

FIG. 1 is a general schematic flow diagram of the present invention;

FIG. 2 is a block diagram of a data recovery process according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1:

a failure detection method in a data distribution process comprises the following steps:

s1, reading dds distribution data, adding a unique serial number to the distribution data, performing RSA signature processing, and synchronously obtaining a public key and a private key of the RSA digital signature.

S2, establishing a data distribution cache node and a lower-layer database, wherein the cache node compresses or directly distributes data, and the synchronous data is recorded into the lower-layer database.

S3, the receiving end and the cache node establish the sequence receiving list, and the public key and the distributed data are synchronously distributed and transmitted in the data distribution process of the cache node.

And S4, verifying the integrity of the distributed data through a public key after receiving the distributed data, recording the distributed data into a sequence receiving table through a serial number, and searching the sequence position of the distributed data.

S5, if the sequence number search does not match the actual receiving sequence, the receiving end recovers the data after a period of time.

And S6, before the data recovery operation, the failure detection module is matched with the QoS service strategy to detect the whole distribution system in advance.

The data recovery comprises the following steps:

in the process of verifying the integrity of the public key data in S5.1 and S1, if the public key cannot be decrypted if dds part of the data is missing, the public key is fed back to the receiving port that cannot be decrypted.

And S5.2, detecting the incremental change of the sequence number through the sequence receiving table in the integrity detection process.

And S5.3, accessing the sequence number log and the sending port thereof in the last time period to the cache node, and comparing the sequence number log with an actual receiving port receiving sequence receiving table to obtain missing data.

And S5.4, after data loss occurs, detecting the distribution system by starting the failure detection module, immediately sending the missing file sequence port to the cache node by the receiving port, matching the missing file sequence port with a lower-layer database of the cache node to obtain a temporarily stored missing file, calling again and sending the temporarily stored missing file, and stopping responding after the missing file is verified to be correct by the public key.

And S5.5, extracting error reporting information of the log after the failure detection module fails to repair, and feeding back through a manual window.

The failure detection module includes the following repair strategies:

a. and detecting the database instance.

b. And verifying the connection mode of the client.

c. And (4) performing safe connection verification by using the ssl connection mode.

d. And verifying the connection command.

e. And (5) detecting the network.

f. Connection number margin verification for the example.

g. And self-checking hardware of the magnetic disk and the processor.

h. And log reading error reporting information extraction.

i. And feeding back error reporting information.

In other embodiments, the client connection mode verification specifically includes client operation detection, detection of whether dds and ecs instances are located in cpc, and synchronization detection of whether dds instances are bound to eip, and the ssl connection mode secure connection verification specifically includes ssl switch open state verification based on dds and root certificate re-upload to ecs operation. The connection command verification comprises connection address verification, port parameter configuration verification, user name and password verification and ssl configuration item connection command verification, and the network detection comprises intranet access verification, public network access verification and cross-network access verification.

By adopting the mode that the console detects the state of the database instance, the DDS system can judge and eliminate the problems of DDS system faults, instance state abnormity and instance or table locking, so that the DDS system can feed back the state of the instance or table in time for processing.

In other embodiments, the sequence number is incremental, and the sequence format is, date + time period + transmit number sequence on the same day, and the sequence receiving table of the cache node includes transmit corresponding port data to which the corresponding sequence number is added. The lower database uses a MySQL relational database based on a memory storage engine. And inputting fault history data and remarking a corresponding processing strategy by both error reporting information feedback and log error reporting information.

Through setting up the serial number, the serial number adopts the date + time quantum + the form that the array was sent on the same day, and its date data can realize the repeated differentiation of the sequence of the same day, and its time quantum is twenty four hours system simultaneously, adopts the mode of hour or moment differentiation time quantum, controls the detection cycle of data recovery, keeps coordinating the fine interim persistence of lower floor's database to the data in the same moment, carries out direct call recovery processing.

Example 2:

a failure detection method in a data distribution process comprises the following steps:

s1, reading dds distribution data, adding a unique serial number to the distribution data, and performing ECDSA signature processing.

S2, establishing a data distribution cache node and a lower-layer database, wherein the cache node compresses or directly distributes data, and the synchronous data is recorded into the lower-layer database.

S3, the receiving end and the buffer node establish the sequence receiving list, and the buffer node distributes and transmits the key and the distributed data synchronously in the data distribution process.

And S4, verifying the integrity of the distributed data through a key after receiving the distributed data, recording the distributed data into a sequence receiving table through a serial number, and searching the sequence position of the distributed data.

S5, if the sequence number search does not match the actual receiving sequence, the receiving end recovers the data after a period of time.

And S6, before the data recovery operation, the failure detection module is matched with the QoS service strategy to detect the whole distribution system in advance.

The data recovery comprises the following steps:

in the process of verifying the integrity of the key data in S5.1 and S1, if dds data is missing, the key cannot be decrypted, and the receiving port that cannot be decrypted feeds back the key.

And S5.2, detecting the incremental change of the sequence number through the sequence receiving table in the integrity detection process.

And S5.3, accessing the sequence number log and the sending port thereof in the last time period to the cache node, and comparing the sequence number log with an actual receiving port receiving sequence receiving table to obtain missing data.

And S5.4, after data loss occurs, detecting the distribution system by starting the failure detection module, immediately sending the missing file sequence port to the cache node by the receiving port, matching the missing file sequence port with a lower-layer database of the cache node to obtain a temporarily stored missing file, calling again and sending the temporarily stored missing file, and stopping responding after the missing file is verified to be correct by a secret key.

And S5.5, extracting error reporting information of the log after the failure detection module fails to repair, and feeding back through a manual window.

The failure detection module includes the following repair strategies:

b. and detecting the database instance.

b. And verifying the connection mode of the client.

c. And (4) performing safe connection verification by using the ssl connection mode.

d. And verifying the connection command.

e. And (5) detecting the network.

f. Connection number margin verification for the example.

g. And self-checking hardware of the magnetic disk and the processor.

h. And log reading error reporting information extraction.

i. And feeding back error reporting information.

The client connection mode verification specifically comprises client operation detection, detection of whether dds and ecs instances are located in a cpc, and synchronization detection of whether dds instances are bound to eip, and the ssl connection mode secure connection verification specifically comprises operation of verifying the ssl switch opening state based on dds and uploading a root certificate to ecs again. The connection command verification comprises connection address verification, port parameter configuration verification, user name and password verification and ssl configuration item connection command verification, and the network detection comprises intranet access verification, public network access verification and cross-network access verification.

The sequence number is an incremental type, the sequence format is a date + time period + current day transmission number sequence, and the sequence receiving table of the cache node comprises transmission corresponding port data added with the corresponding sequence number. The lower database uses a MySQL relational database based on a memory storage engine. And inputting fault history data and remarking a corresponding processing strategy by both error reporting information feedback and log error reporting information.

Example 3:

a failure detection method in a data distribution process comprises the following steps:

s1, reading dds distribution data, adding a unique serial number to the distribution data, performing DSA signature processing, and synchronously acquiring a secret key and a private key of the DSA digital signature.

S2, establishing a data distribution cache node and a lower-layer database, wherein the cache node compresses or directly distributes data, and the synchronous data is recorded into the lower-layer database.

S3, the receiving end and the buffer node establish the sequence receiving list, and the buffer node distributes and transmits the key and the distributed data synchronously in the data distribution process.

And S4, verifying the integrity of the distributed data through a key after receiving the distributed data, recording the distributed data into a sequence receiving table through a serial number, and searching the sequence position of the distributed data.

S5, if the sequence number search does not match the actual receiving sequence, the receiving end recovers the data after a period of time.

And S6, before the data recovery operation, the failure detection module is matched with the QoS service strategy to detect the whole distribution system in advance.

The data recovery comprises the following steps:

in the process of verifying the integrity of the key data in S5.1 and S1, if dds data is missing, the key cannot be decrypted, and the receiving port that cannot be decrypted feeds back the key.

And S5.2, detecting the incremental change of the sequence number through the sequence receiving table in the integrity detection process.

And S5.3, accessing the sequence number log and the sending port thereof in the last time period to the cache node, and comparing the sequence number log with an actual receiving port receiving sequence receiving table to obtain missing data.

And S5.4, after data loss occurs, detecting the distribution system by starting the failure detection module, immediately sending the missing file sequence port to the cache node by the receiving port, matching the missing file sequence port with a lower-layer database of the cache node to obtain a temporarily stored missing file, calling again and sending the temporarily stored missing file, and stopping responding after the missing file is verified to be correct by a secret key.

And S5.5, extracting error reporting information of the log after the failure detection module fails to repair, and feeding back through a manual window.

The failure detection module includes the following repair strategies:

c. and detecting the database instance.

b. And verifying the connection mode of the client.

c. And (4) performing safe connection verification by using the ssl connection mode.

d. And verifying the connection command.

e. And (5) detecting the network.

f. Connection number margin verification for the example.

g. And self-checking hardware of the magnetic disk and the processor.

h. And log reading error reporting information extraction.

i. And feeding back error reporting information.

The client connection mode verification specifically comprises client operation detection, detection of whether dds and ecs instances are located in a cpc, and synchronization detection of whether dds instances are bound to eip, and the ssl connection mode secure connection verification specifically comprises operation of verifying the ssl switch opening state based on dds and uploading a root certificate to ecs again. The connection command verification comprises connection address verification, port parameter configuration verification, user name and password verification and ssl configuration item connection command verification, and the network detection comprises intranet access verification, public network access verification and cross-network access verification.

The sequence number is an incremental type, the sequence format is a date + time period + current day transmission number sequence, and the sequence receiving table of the cache node comprises transmission corresponding port data added with the corresponding sequence number. The lower database uses a MySQL relational database based on a memory storage engine. And inputting fault history data and remarking a corresponding processing strategy by both error reporting information feedback and log error reporting information.

In conclusion, the present invention: the matching of serial number packet loss verification and electronic signature integrity verification modes is adopted, the electronic signature mode comprises the three embodiments but is not limited to the three embodiments, the sequence verification and the various electronic signature modes can ensure stable and efficient verification of data, and meanwhile, the mode that a failure detection module is matched with a QoS (quality of service) strategy is adopted, so that the operation stability of the system is ensured to a greater extent.

The points to be finally explained are: although the present invention has been described in detail with reference to the general description and the specific embodiments, on the basis of the present invention, the above embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.