阅读说明：本技术 基于去重规则的物联网单数据存储方法及装置 (Internet of things single data storage method and device based on deduplication rule ) 是由 温文坤 林英喜 马凤鸣 陈名峰 李玮棠 刘毅 于 2021-01-20 设计创作，主要内容包括：本发明实施例公开了一种基于去重规则的物联网单数据存储方法及装置,该方法包括：第一节点设备接收第二节点设备发送的物联网数据集合,确定所述物联网数据集合中数据内容的关联度；根据所述数据内容的关联度以及数据类型确定去重处理规则；确定所述去重处理规则对应的处理集群中的处理服务器,将所述第二节点设备发送的物联网数据集合发送至所述处理服务器进行去重处理后,进行数据保存。本方案,提高了数据存储效率,便于数据的传输和维护,同时保证了数据稳定性、安全性。(The embodiment of the invention discloses a method and a device for storing single data of an Internet of things based on a deduplication rule, wherein the method comprises the following steps: the method comprises the steps that a first node device receives an internet of things data set sent by a second node device, and the relevance of data content in the internet of things data set is determined; determining a duplicate removal processing rule according to the relevance of the data content and the data type; and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data. According to the scheme, the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are guaranteed.)

1. The Internet of things single data storage method based on the deduplication rule is characterized by comprising the following steps:

the method comprises the steps that a first node device receives an internet of things data set sent by a second node device, and the relevance of data content in the internet of things data set is determined;

determining a duplicate removal processing rule according to the relevance of the data content and the data type;

and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data.

2. The Internet of things single-data storage method based on the deduplication rule as claimed in claim 1, wherein the determining the relevancy of the data content in the Internet of things data set comprises:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

3. The Internet of things single-data storage method based on the deduplication rule as recited in claim 1, wherein the determining the deduplication processing rule according to the relevancy of the data content and the data type comprises:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

4. The Internet of things single data storage method based on the deduplication rules according to any one of claims 1 to 3, wherein the processing server performs deduplication processing, and comprises:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

5. Thing networking list data storage device based on deduplication rule, its characterized in that includes:

the data receiving and processing module is used for receiving an internet of things data set sent by second node equipment and determining the relevance of data contents in the internet of things data set;

the rule determining module is used for determining a deduplication processing rule according to the relevance of the data content and the data type;

and the data sending module is used for determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data.

6. The Internet of things single-data storage device based on the deduplication rule of claim 5, wherein the data receiving and processing module is specifically configured to:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

7. The Internet of things single-data storage device based on the deduplication rules of claim 5, wherein the rule determination module is specifically configured to:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

8. The Internet of things single-data storage device based on the deduplication rules according to any one of claims 5-7, wherein the data sending module is specifically configured to:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

9. An internet of things single data storage device based on deduplication rules, the device comprising: one or more processors; a storage device to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the deduplication rule based single data storage method of the internet of things of any of claims 1-4.

10. A storage medium containing computer-executable instructions for performing the deduplication rule-based internet of things single data storage method of any one of claims 1-4 when executed by a computer processor.

Technical Field

The embodiment of the application relates to the technical field of Internet of things, in particular to a method and a device for storing single data of the Internet of things based on a deduplication rule.

Background

With the popularization of the internet of things technology and the powerful functions of intelligent equipment, the internet of things equipment plays an increasingly important role in daily life of people. Various kinds of intelligent physical network terminal devices are applied to various fields. Generally, the internet of things connects articles with the internet through various information sensing devices, so that information exchange can be realized for all common physical objects which can be independently addressed, and the purposes of intelligent identification, positioning, tracking, monitoring and management are finally achieved.

The data of the internet of things are from different sensing devices, such as a recognizer, a video device, a temperature sensor, a humidity sensor and the like, and generally, the data format and the semantic structure have certain difference. Meanwhile, with the popularization and the large-scale application of the internet of things equipment, the data volume of the internet of things is increased sharply, and if the internet of things data with various types and various requirements are stored reasonably and efficiently, the problem which needs to be solved at present is urgently solved.

Disclosure of Invention

The embodiment of the invention provides a duplicate removal rule-based method and device for storing single data of the Internet of things, which improve the data storage efficiency, facilitate the transmission and maintenance of data, and simultaneously ensure the stability and the safety of the data.

In a first aspect, an embodiment of the present invention provides a method for storing single data of an internet of things based on a deduplication rule, where the method includes:

the method comprises the steps that a first node device receives an internet of things data set sent by a second node device, and the relevance of data content in the internet of things data set is determined;

determining a duplicate removal processing rule according to the relevance of the data content and the data type;

and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data.

Optionally, the determining the association degree of the data content in the data set of the internet of things includes:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

Optionally, the determining the deduplication processing rule according to the association degree of the data content and the data type includes:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

Optionally, the processing server performs deduplication processing, including:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

In a second aspect, an embodiment of the present invention further provides an internet of things single data storage device based on a deduplication rule, including:

the data receiving and processing module is used for receiving an internet of things data set sent by second node equipment and determining the relevance of data contents in the internet of things data set;

the rule determining module is used for determining a deduplication processing rule according to the relevance of the data content and the data type;

and the data sending module is used for determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data.

Optionally, the data receiving and processing module is specifically configured to:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

Optionally, the rule determining module is specifically configured to:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

Optionally, the data sending module is specifically configured to:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

In a third aspect, an embodiment of the present invention further provides an internet of things single data storage device based on a deduplication rule, where the device includes:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the method for storing single data of the internet of things based on the deduplication rule according to the embodiment of the invention.

In a fourth aspect, the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform the deduplication rule based single data storage method for the internet of things according to the present invention.

In the embodiment of the invention, a first node device receives an Internet of things data set sent by a second node device, and determines the association degree of data contents in the Internet of things data set; determining a duplicate removal processing rule according to the relevance of the data content and the data type; and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data. According to the scheme, the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are guaranteed.

Drawings

Fig. 1 is a flowchart of a method for storing single data of an internet of things based on a deduplication rule according to an embodiment of the present invention;

fig. 2 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention;

fig. 3 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention;

fig. 4 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention;

fig. 5 is a block diagram of a single data storage device of the internet of things based on a deduplication rule according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.

Detailed Description

The embodiments of the present invention will be described in further detail with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad invention. It should be further noted that, for convenience of description, only some structures, not all structures, relating to the embodiments of the present invention are shown in the drawings.

Fig. 1 is a flowchart of a method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention, which is suitable for storing device data of the internet of things. The scheme of one embodiment of the application specifically comprises the following steps:

step S101, receiving an Internet of things data set sent by second node equipment by first node equipment, and determining the relevance of data contents in the Internet of things data set.

The first node device may be an internet of things gateway device, and the second node device may be an internet of things terminal device covered by the internet of things gateway device. The second node equipment uploads data to the first node equipment, and the data are sent to the server through the first node equipment to be stored. Optionally, the first node device may also be a transit device selected from a plurality of second node devices.

In one embodiment, the first node device processes the received data set of the internet of things of the second node device, specifically, determines the relevance of data content in the data set of the internet of things. Illustratively, taking character data as an example, the data a uploaded by the second node device 1 includes 8 fields, where the fields 1 to 6 are general fields, and the fields 7 and 8 are fields characterizing characteristics of current data, and then the fields 1 to 6 are associated fields in each data; taking image data as an example, the second node device 2 uploads an image a at a first time and uploads an image b at a second time, and the image association between the image a and the image b can be determined, if the contents of background images are consistent and no special target object exists, the association between the image a and the image b is 100%, if the difference between the target objects exists, the association is determined by exemplarily considering the area size of the whole image occupied by the target object, and if the difference between the target objects exists, the association is determined to be 90%; taking the text content as an example, the second node device 3 uploads the text a and the text b, and if the contents of the two are determined to be consistent at the beginning or the end by similarity comparison, the association degree is 100% of the middle content part, there is an obvious text difference, and the similarity is 0.

And S102, determining a deduplication processing rule according to the relevance of the data content and the data type.

In one embodiment, taking the data content including a character type, an audio type and a video image type as an example, for each type of data, after determining the similarity, there is a difference in the corresponding deduplication processing mechanisms.

Specifically, a relevance threshold value can be set for each data type, and when the relevance of the data in the determined internet of things data set is greater than the relevance threshold value, the corresponding deduplication processing rule is matched according to the corresponding data type. Examples are shown in the following table:

wherein, the deduplication rule 1 may exemplarily be: extracting characters with the same content as public data, and taking the rest of the characters as private data, wherein the public data is stored separately and the private data is stored separately; the deduplication rule 2 may exemplarily be: deleting an audio portion that does not contain a sound of a specific object (e.g., a person); the deduplication rule 3 may exemplarily be: and deleting the background image in the video image, or extracting the image of the target object (such as a vehicle) for independent storage.

Step S103, determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node device to the processing server for duplicate removal processing, and then storing the data.

In one embodiment, different deduplication rules correspond to different processing servers, processing servers in a processing cluster corresponding to the deduplication processing rules are determined, and the internet of things data set sent by the second node device is sent to the processing servers for deduplication processing and then data storage. The server which corresponds to each duplication elimination rule is arranged for batch processing, and duplication elimination storage efficiency is improved.

Therefore, the first node device receives the data set of the internet of things sent by the second node device, and determines the association degree of the data content in the data set of the internet of things; determining a duplicate removal processing rule according to the relevance of the data content and the data type; and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for data storage after the duplicate removal processing, so that the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are ensured.

Fig. 2 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention. On the basis of the technical scheme, the determining the relevance of the data content in the data set of the internet of things comprises:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things. The method specifically comprises the following steps:

step S201, a first node device receives an Internet of things data set sent by a second node device, and determines the relevance of data content according to the content similarity of adjacent data in the Internet of things set.

In one embodiment, the method for determining the association degree is further defined, specifically, the association degree of the data content is determined according to the content similarity of the adjacent data in the internet of things set. Such as two images uploaded back and forth, or collected sensor data uploaded back and forth, and so on.

Step S202, determining a deduplication processing rule according to the relevance of the data content and the data type.

Step S203, determining the processing server in the processing cluster corresponding to the deduplication processing rule, and after sending the internet of things data set sent by the second node device to the processing server for deduplication processing, storing the data.

According to the data storage method and the data storage device, the first node device receives the Internet of things data set sent by the second node device, and the relevance of the data content is determined according to the content similarity of the adjacent data in the Internet of things set, so that the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are ensured.

Fig. 3 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention. On the basis of the above technical solution, the determining a deduplication processing rule according to the association degree of the data content and the data type includes:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule. The method specifically comprises the following steps:

step S301, the first node device receives an Internet of things data set sent by the second node device, and determines the association degree of data content according to the content similarity of adjacent data in the Internet of things set.

Step S302, if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining that the corresponding deduplication processing rule is a merge processing rule, and if the association degree of the data content is greater than a first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

In one embodiment, the first preset value may be 90%, the first preset type may be a video image type, and the second preset type may be an audio type. The method comprises the following steps of reserving one part of the same (namely associated) data content for merging processing, and separately reserving processing rules for non-associated parts; the deletion processing rule may be to delete an audio data segment not containing a vocal part.

Step S303, determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the internet of things data set sent by the second node device to the processing server for duplicate removal processing, and then storing the data.

According to the scheme, if the association degree of the data content is greater than a first preset value and the data type is a first preset type, the corresponding deduplication processing rule is determined to be a merging processing rule, and if the association degree of the data content is greater than a first preset threshold value and the data type is a second preset type, the corresponding deduplication processing rule is determined to be a deletion processing rule, so that the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are guaranteed.

Fig. 4 is a flowchart of another method for storing single data of the internet of things based on a deduplication rule according to an embodiment of the present invention. On the basis of the technical scheme, the processing server performs deduplication processing, and the deduplication processing comprises the following steps:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory. The method specifically comprises the following steps:

step S401, the first node device receives an Internet of things data set sent by the second node device, and determines the relevance of data content according to the content similarity of adjacent data in the Internet of things set.

Step S402, if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining that the corresponding deduplication processing rule is a merging processing rule, and if the association degree of the data content is greater than a first preset threshold value and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

Step S403, determining a processing server in the processing cluster corresponding to the deduplication processing rule, sending the internet of things data set sent by the second node device to the processing server, if the internet of things data set is a server corresponding to the deletion processing rule, determining data deletion content according to the data type, merging the deletion content, and storing the merged deletion content in a memory.

In one embodiment, there are two different data processing types of server devices. For example, if the server corresponding to the deletion processing rule determines data deletion content according to the data type, and merges and stores the deletion content to the memory, specifically, taking the audio data type as an example, if the corresponding deletion content is an audio data segment not containing human voice, the audio data segment not containing human voice is deleted, merged and stored to the memory.

According to the data storage method, the processing server in the processing cluster corresponding to the duplicate removal processing rule is determined, the data set of the internet of things sent by the second node device is sent to the processing server, if the processing server is a server corresponding to the delete processing rule, data delete contents are determined according to the data type, the delete contents are merged and then stored in the storage, the data storage efficiency is improved, data transmission and maintenance are facilitated, and meanwhile data stability and safety are guaranteed.

Fig. 5 is a structural block diagram of an internet of things single data storage device based on a deduplication rule according to an embodiment of the present invention, where the device is configured to execute the internet of things single data storage method based on the deduplication rule according to the data receiving end embodiment, and has functional modules and beneficial effects corresponding to the execution method. As shown in fig. 5, the apparatus specifically includes: a data reception processing module 101, a rule determination module 102, and a data transmission module 103, wherein,

the data receiving and processing module 101 is configured to receive an internet of things data set sent by a second node device, and determine a relevance of data content in the internet of things data set;

a rule determining module 102, configured to determine a deduplication processing rule according to the association degree of the data content and the data type;

and the data sending module 103 is configured to determine a processing server in the processing cluster corresponding to the deduplication processing rule, send the internet of things data set sent by the second node device to the processing server for deduplication processing, and store the data.

According to the scheme, the first node equipment receives the Internet of things data set sent by the second node equipment, and the association degree of the data content in the Internet of things data set is determined; determining a duplicate removal processing rule according to the relevance of the data content and the data type; and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data. According to the scheme, the data storage efficiency is improved, the data transmission and maintenance are facilitated, and the data stability and safety are guaranteed.

In a possible embodiment, the data receiving and processing module is specifically configured to:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

In a possible embodiment, the rule determining module is specifically configured to:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

In a possible embodiment, the data sending module is specifically configured to:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

Fig. 6 is a schematic structural diagram of an internet-of-things single data storage device based on a deduplication rule according to an embodiment of the present invention, as shown in fig. 6, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of the processors 201 in the device may be one or more, and one processor 201 is taken as an example in fig. 6; the processor 201, the memory 202, the input device 203 and the output device 204 in the apparatus may be connected by a bus or other means, for example in fig. 6. The memory 202 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the internet-of-things single-data storage method based on the deduplication rule in the embodiment of the present invention. The processor 201 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 202, that is, the above-mentioned single data storage method of the internet of things based on the deduplication rule is realized. The input device 203 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function controls of the apparatus. The output device 204 may include a display device such as a display screen.

Embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a deduplication rule-based single data storage method for an internet of things, the method including:

the method comprises the steps that a first node device receives an internet of things data set sent by a second node device, and the relevance of data content in the internet of things data set is determined;

determining a duplicate removal processing rule according to the relevance of the data content and the data type;

and determining a processing server in the processing cluster corresponding to the duplicate removal processing rule, sending the Internet of things data set sent by the second node equipment to the processing server for duplicate removal processing, and then storing the data.

In one possible embodiment, the determining the relevancy of the data content in the data set of the internet of things includes:

and determining the association degree of the data content according to the content similarity of the adjacent data in the set of the Internet of things.

In a possible embodiment, the determining the deduplication processing rule according to the relevancy of the data content and the data type includes:

if the association degree of the data content is greater than a first preset value and the data type is a first preset type, determining the corresponding deduplication processing rule as a merging processing rule;

and if the association degree of the data content is greater than the first preset threshold and the data type is a second preset type, determining that the corresponding deduplication processing rule is a deletion processing rule.

In one possible embodiment, the processing server performs deduplication processing, including:

and if the server is the server corresponding to the deletion processing rule, determining data deletion contents according to the data type, merging the deletion contents and storing the merged deletion contents in a memory.

From the above description of the embodiments, it is obvious for those skilled in the art that the embodiments of the present invention can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but the former is a better implementation in many cases. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions to make a computer device (which may be a personal computer, a service, or a network device) perform the methods described in the embodiments of the present invention.

It should be noted that, in the embodiment of the single data storage device of the internet of things based on the deduplication rule, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.

It should be noted that the foregoing is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will appreciate that the embodiments of the present invention are not limited to the specific embodiments described herein, and that various obvious changes, adaptations, and substitutions are possible, without departing from the scope of the embodiments of the present invention. Therefore, although the embodiments of the present invention have been described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and many other equivalent embodiments may be included without departing from the concept of the embodiments of the present invention, and the scope of the embodiments of the present invention is determined by the scope of the appended claims.