阅读说明：本技术 数据迁移方法、装置、系统和计算机可读存储介质 (Data migration method, device, system and computer readable storage medium ) 是由 金均生 熊汉韬 齐浩 王昆垚 董志平 于 2021-01-29 设计创作，主要内容包括：本公开涉及一种数据迁移方法、装置、系统和计算机可读存储介质,涉及计算机技术领域。本公开的方法包括：确定新增的图引擎服务的标签在哈希环中的位置；根据标签在哈希环中的位置和已分配的各个图分片数据在哈希环中的位置的距离,确定属于新增的图引擎服务的待迁移的图分片数据；向新增的图引擎服务和待迁移的图分片数据的原图引擎服务发送指示信息,将待迁移的图分片数据由原图引擎服务迁移到新增的图引擎服务。(The disclosure relates to a data migration method, a data migration device, a data migration system and a computer-readable storage medium, and relates to the technical field of computers. The method of the present disclosure comprises: determining the position of a label of the newly added graph engine service in the hash ring; determining graph fragment data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each distributed graph fragment data in the hash ring; and sending indication information to the newly added graph engine service and the original graph engine service of the graph fragment data to be migrated, and migrating the graph fragment data to be migrated from the original graph engine service to the newly added graph engine service.)

1. A method of data migration, comprising:

determining the position of a label of the newly added graph engine service in the hash ring;

determining the graph fragment data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each distributed graph fragment data in the hash ring;

and sending indication information to the newly added graph engine service and the original graph engine service of the graph fragment data to be migrated, and migrating the graph fragment data to be migrated to the newly added graph engine service from the original graph engine service.

2. The data migration method of claim 1, wherein the number of the labels of the newly added graph engine service is multiple, and the determining the position of the label of the newly added graph engine service in the hash ring comprises:

acquiring a plurality of labels of the newly added graph engine service;

determining the hash value of each label according to a preset hash function;

and determining the position of each label in the hash ring according to the hash value of each label.

3. The data migration method according to claim 1, wherein the determining, according to the distance between the position of each label in the hash ring and the position of each allocated graph fragment data in the hash ring, the graph fragment data to be migrated belonging to the newly added graph engine service includes:

sequentially determining labels which are closest to the positions of the image fragment data in the hash ring according to a preset sequence in the hash ring, and taking the labels as labels corresponding to the image fragment data;

and regenerating a mapping table of the graph fragment data and the graph engine service according to the label corresponding to each graph fragment data and the graph engine service to which each label belongs, wherein the graph fragment data to be migrated corresponds to the newly added graph engine service in the mapping table.

4. The data migration method according to claim 3, wherein the sending of the indication information to the original graph engine service of the newly added graph engine service and the graph fragment data to be migrated, and the migrating the graph fragment data to be migrated from the original graph engine service to the newly added graph engine service includes:

sending indication information to each graph engine service, wherein the indication information comprises the mapping table;

receiving a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes loading of the graph fragment data to be migrated according to the mapping table;

sending a deletion instruction to the original image engine service;

and receiving a deletion completion message sent by the original image engine service, wherein the deletion completion message indicates that the original image engine service completes the deletion of the image fragment data to be migrated according to the mapping table.

5. The data migration method of claim 1, further comprising:

and sending mapping tables of the graph fragment data and the graph engine service to a graph data query service under the condition that the graph fragment data to be migrated is migrated from the original graph engine service to the newly added graph engine service, wherein the mapping tables correspond to the newly added graph engine service and are the graph fragment data to be migrated.

6. The data migration method of claim 1, further comprising:

determining hash values of identifications of various entities in the graph, wherein the entities comprise data of nodes or data of edges of the graph;

determining the identifier of the graph fragment data to which each entity belongs according to the hash value of the identifier of each entity;

determining the hash value of the identifier of each graph fragment data;

and determining the position of each graph fragment data in the hash ring according to the identified hash value of each graph fragment data, wherein the position of each graph fragment data in the hash ring is used as the position of each distributed graph fragment data in the hash ring.

7. The data migration method of claim 1, wherein the number of tags per graph engine service is determined according to the processing power of the graph engine service, wherein the stronger the graph engine service processing power, the greater the number of tags.

8. The data migration method of claim 5, further comprising:

the graph data query service receives a data query request, wherein the data query request comprises: an entity to be queried, the entity comprising data of a node or data of an edge of a graph;

the graph data query service determines a hash value of the identity of the entity to be queried;

the graph data query service determines the identifier of the graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried;

the graph data query service searches a corresponding graph engine service in the mapping table according to the identifier of the graph fragment data to which the entity to be queried belongs, and the graph engine service is used as a target graph engine service;

the graph data query service queries the target graph engine service for the entity to be queried.

9. A data migration apparatus, comprising:

the label position determining module is used for determining the position of a label of the newly added graph engine service in the hash ring;

the migration information determining module is used for determining the graph fragment data to be migrated which belongs to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each distributed graph fragment data in the hash ring;

and the migration indicating module is used for sending indicating information to the newly added graph engine service and the original graph engine service of the to-be-migrated graph fragment data, and migrating the to-be-migrated graph fragment data from the original graph engine service to the newly added graph engine service.

10. A data migration apparatus, comprising:

a processor; and

a memory coupled to the processor for storing instructions that, when executed by the processor, cause the processor to perform the data migration method of any of claims 1-7.

11. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the steps of the method of any one of claims 1-7.

12. A data migration system, comprising: the data migration apparatus of claim 9 or 10, and a plurality of graph engine services;

and the graph engine service is used for receiving the indication information sent by the data migration device to realize the migration of the graph fragment data to be migrated under the condition that the graph engine service is a newly-added graph service engine or an original graph engine service of the graph fragment data to be migrated.

13. The data migration system of claim 12, wherein the indication information includes the mapping table; and the graph engine service is used for completing the loading of the graph fragment data to be migrated according to the mapping table and sending a loading completion message to the data migration device under the condition that the graph engine service is a newly added graph service engine, and is used for receiving a deletion instruction sent by the data migration device, completing the deletion of the graph fragment data to be migrated according to the mapping table and sending a deletion completion message to the data migration device under the condition that the graph engine service is the original graph service engine.

14. The data migration system of claim 12, further comprising:

and the map data query service is used for receiving the mapping table sent by the migration device and providing query service according to the mapping table.

15. The data migration system of claim 14,

the graph data query service is used for receiving a data query request, determining a hash value of an identifier of the entity to be queried, determining an identifier of graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried, searching a corresponding graph engine service according to the identifier of the graph fragment data to which the entity to be queried belongs in the mapping table, taking the graph engine service as a target graph engine service, and querying the entity to be queried from the target graph engine service; wherein the data query request comprises: an entity to be queried, the entity comprising data of a node or data of an edge of a graph.

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data migration method, apparatus, system, and computer-readable storage medium.

Background

In recent years, the popularization of the mobile internet has promoted rapid development in the fields of electronic Commerce (E-Commerce), Social networking (Social networking), and the like, and has generated massive user and commodity data. How to efficiently utilize this data to optimize the user experience and continue to create value becomes a significant challenge for this class of application providers. The data structure of the graph has strong expression capability and is very suitable for modeling the relationship between users in a social network and the relationship between the users and commodities in an e-commerce system. Therefore, the data structure is introduced into the e-commerce system and the social network to depict the relationship between the users and the commodities and between the users, so that the evolution process of the user interest can be captured better, more guide information can be provided for personalized recommendation, online advertisement delivery, friend recommendation in the social network and the like in the e-commerce system, and the user experience is optimized and the value is created.

In a practical application of graph modeling, an online graph engine service is usually constructed to load graph data. The service provides a query service of graph data for other services. In real industrial-level application, the quantity of graph data is very large, a single machine cannot load the graph data, the graph data needs to be split to obtain graph fragment data when the graph data is actually online, a distributed graph engine is constructed to load the graph data, and graph data query service is provided for the outside.

As active users in e-commerce systems and social networks continue to grow, user behavior is becoming more and more abundant. Graph data from graph modeling of e-commerce systems and social networks is growing. And because the number N of the online graph fragment data is generally kept constant, the size of a single fragment file is continuously increased, so that the memory loading of a graph engine service machine is not performed. Therefore, the number M of the graph engine services needs to be increased to re-distribute the graph data fragment files to the graph engine services, so that the graph data fragment files can be smoothly loaded.

Disclosure of Invention

The inventor finds that: when the graph data of the online distributed graph engine continuously increases and the number of the graph data engine services needs to be expanded, the existing method can cause the problem that the migration amount of the graph fragment data is too large, and even partial query fails in a large migration process.

One technical problem to be solved by the present disclosure is: how to reduce the migration volume of the graph fragment data under the condition of expanding the graph engine.

According to some embodiments of the present disclosure, there is provided a data migration method, including: determining the position of a label of the newly added graph engine service in the hash ring; determining graph fragment data to be migrated belonging to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each distributed graph fragment data in the hash ring; and sending indication information to the newly added graph engine service and the original graph engine service of the graph fragment data to be migrated, and migrating the graph fragment data to be migrated from the original graph engine service to the newly added graph engine service.

In some embodiments, the number of the labels of the newly added graph engine service is multiple, and determining the position of the label of the newly added graph engine service in the hash ring includes: acquiring a plurality of labels of the newly added graph engine service; determining the hash value of each label according to a preset hash function; and determining the position of each label in the hash ring according to the hash value of each label.

In some embodiments, determining the graph fragment data to be migrated belonging to the newly added graph engine service according to the distance between the position of each label in the hash ring and the position of each allocated graph fragment data in the hash ring includes: sequentially determining labels which are closest to the positions of the image fragment data in the hash ring according to a preset sequence in the hash ring, and taking the labels as labels corresponding to the image fragment data; and regenerating a mapping table of the graph fragment data and the graph engine service according to the label corresponding to each graph fragment data and the graph engine service to which each label belongs, wherein the graph fragment data to be migrated corresponds to the newly added graph engine service in the mapping table.

In some embodiments, sending the indication information to the original graph engine service of the newly added graph engine service and the graph fragment data to be migrated, and migrating the graph fragment data to be migrated from the original graph engine service to the newly added graph engine service includes: sending indication information to each graph engine service, wherein the indication information comprises a mapping table; receiving a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes the loading of the graph fragment data to be migrated according to the mapping table; sending a deletion instruction to an original image engine service; and receiving a deletion completion message sent by the original image engine service, wherein the deletion completion message indicates that the original image engine service completes the deletion of the image fragment data to be migrated according to the mapping table.

In some embodiments, the method further comprises: and sending the mapping table of the graph fragment data and the graph engine service to the graph data query service under the condition that the graph fragment data to be migrated is migrated from the original graph engine service to the newly added graph engine service, wherein the mapping table corresponds to the newly added graph engine service and is the graph fragment data to be migrated.

In some embodiments, the method further comprises: determining hash values of the identifications of the various entities in the graph, wherein the entities comprise data of nodes or data of edges of the graph; determining the identifier of the graph fragment data to which each entity belongs according to the hash value of the identifier of each entity; determining the hash value of the identifier of each graph fragment data; and determining the position of each graph fragment data in the hash ring according to the identified hash value of each graph fragment data, wherein the position of each graph fragment data in the hash ring is used as the position of each distributed graph fragment data in the hash ring.

In some embodiments, the number of tags per graph engine service is determined according to the processing power of the graph engine service, wherein the stronger the graph engine service processing power, the greater the number of tags.

In some embodiments, the method further comprises: the graph data query service receives a data query request, wherein the data query request comprises: an entity to be queried, the entity comprising data of a node or data of an edge of a graph; the graph data query service determines a hash value of an identifier of an entity to be queried; the graph data query service determines the identifier of the graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried; in a mapping table, the graph data query service searches a corresponding graph engine service according to the identifier of the graph fragment data to which the entity to be queried belongs, and the graph engine service is used as a target graph engine service; the graph data query service queries the target graph engine service for entities to be queried.

According to further embodiments of the present disclosure, there is provided a data migration apparatus including: the label position determining module is used for determining the position of a label of the newly added graph engine service in the hash ring; the migration information determining module is used for determining the graph fragment data to be migrated which belongs to the newly added graph engine service according to the distance between the position of the label in the hash ring and the position of each distributed graph fragment data in the hash ring; and the migration indicating module is used for sending indicating information to the newly added graph engine service and the original graph engine service of the graph fragment data to be migrated, and migrating the graph fragment data to be migrated from the original graph engine service to the newly added graph engine service.

According to still other embodiments of the present disclosure, there is provided a data migration apparatus including: a processor; and a memory coupled to the processor for storing instructions that, when executed by the processor, cause the processor to perform a data migration method as in any of the preceding embodiments.

According to still further embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the data migration method of any of the foregoing embodiments.

According to still other embodiments of the present disclosure, there is provided a data migration system including: the data migration apparatus of any of the preceding embodiments, and a plurality of graph engine services; the graph engine service is used for receiving indication information sent by the data migration device and realizing migration of the graph fragment data to be migrated when the graph engine service is a newly added graph service engine or an original graph engine service of the graph fragment data to be migrated.

In some embodiments, the indication information comprises a mapping table; and the graph engine service is used for finishing the loading of the graph fragment data to be migrated according to the mapping table and sending a loading finishing message to the data migration device under the condition that the graph engine service is a newly added graph service engine, and is used for receiving a deletion instruction sent by the data migration device, finishing the deletion of the graph fragment data to be migrated according to the mapping table and sending a deletion finishing message to the data migration device under the condition that the graph engine service is an original graph service engine.

In some embodiments, the system further comprises: and the map data query service is used for receiving the mapping table sent by the migration device and providing query service according to the mapping table.

In some embodiments, the graph data query service is configured to receive a data query request, determine a hash value of an identifier of an entity to be queried, determine an identifier of graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried, search, in a mapping table, for a corresponding graph engine service according to the identifier of the graph fragment data to which the entity to be queried belongs, serve as a target graph engine service, and query the entity to be queried from the target graph engine service; wherein, the data query request includes: and the entity to be queried comprises data of the nodes or edges of the graph.

The method and the device adopt a Hash ring mode to determine the relationship between the graph fragment data and the graph engine service, and further determine the graph fragment data to be migrated which belongs to the newly added graph engine service, so that the graph fragment data to be migrated is migrated to the newly added graph engine service from the original graph engine service. When the number of the graph engine services is expanded, data migration between the original graph engine services does not exist, and only data migration from the original graph engine services to the new graph engine services exists, so that the migration quantity of the graph fragment data is reduced, the migration efficiency of the graph fragment data is improved, the real-time property and the query effect of graph data query can be improved, the user experience is optimized, and the graph modeling benefit is improved.

Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic diagram illustrating a mapping method of some existing graph fragment data to a graph engine service.

FIG. 2 illustrates a flow diagram of a data migration method of some embodiments of the present disclosure.

Fig. 3 illustrates a schematic diagram of a mapping method of graph fragment data to a graph engine service of some embodiments of the present disclosure.

FIG. 4 shows a flow diagram of a data migration method of further embodiments of the present disclosure.

Fig. 5 shows a flow diagram of a data migration method of further embodiments of the present disclosure.

FIG. 6 illustrates a schematic structural diagram of a data migration apparatus of some embodiments of the present disclosure.

FIG. 7 shows a schematic structural diagram of a data migration apparatus according to further embodiments of the present disclosure.

Fig. 8 is a schematic structural diagram of a data migration apparatus according to still other embodiments of the present disclosure.

FIG. 9 illustrates a structural schematic diagram of a data migration system of some embodiments of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

Fig. 1 shows a method for migrating graph-sliced data known to the inventors. Since the current graph fragments data to the graph engine clothesThe traffic is mapped primarily by modulo arithmetic as a routing rule. Thus, increasing the number of graph engine services is equivalent to re-customizing the routing rules. For example, as shown in fig. 1, it is assumed that the current map data sharded file number N is 12, that is, there is a map data sharded file set { P {₀,P₁,P₂,P₃，P₄,P₅,P₆,P₇,P₈,P₉,P₁₀,P₁₁}. The number of graph engine services M is 3, i.e. there is a set of graph engine services S₀,S₁,S₂}. When M is increased by 1, a graph engine service is added, and the graph engine set is S₀,S₁,S₂,S₃}. Distribution of graph fragment data to graph Engine service As shown in FIG. 1, before adding the graph Engine service S3, S₀The corresponding graph fragment data is P₀,P₃，P₆,P₉，S₁The corresponding graph fragment data is P₁,P₄,P₇,P₁₀，S₂The corresponding graph fragment data is P₂,P₅,P₈,P₁₁. After adding the graph Engine service S3, S₀The corresponding graph fragment data is P₀,P₄,P₈，S₁The corresponding graph fragment data is P₁,P₅,P₉，S₂The corresponding graph fragment data is P₂,P₆,P₁₀，S₃The corresponding graph fragment data is P₃，P₇，P₁₁. It can be seen that there is a large amount of data migration between graph engine services.

In view of the above problems, the present disclosure proposes a data migration method, which is described below with reference to fig. 2 to 5.

FIG. 2 is a flow chart of some embodiments of the disclosed data migration method. As shown in fig. 2, the method of this embodiment includes: steps S202 to S206.

In step S202, the position of the label of the newly added graph engine service in the hash ring is determined.

In some embodiments, the tags of the newly added graph engine service are one or more. The label of the graph engine service refers to a specific identification of the graph engine service. May be constructed in the form of a domain name + index (index) or ip + index served by the graph engine. Assuming that one graph engine services domain name is graph _ service _1 and there are three labels, each label can be represented as graph _ service _1_0, graph _ service _1_1, and graph _ service _1_ 2.

The labels of the graph engine service can be understood as virtual nodes of the graph engine service. In the consistent hash algorithm, the number of labels of each graph engine service may be set to 1, that is, each graph engine service only has one unique (physical) node in the hash ring, and the graph fragment data mapped to the node is loaded by the graph engine service. However, when the flag of the graph engine service is set to 1, the image fragment data in different graph engine services is not uniform during capacity expansion, and even a service avalanche phenomenon is caused. Therefore, the labels of the graph engine service are arranged to be multiple, so that the distribution of the graph fragment data can be more uniform, and the occurrence of the avalanche phenomenon can be reduced.

In some embodiments, the number of tags per graph engine service is determined according to the processing power of the graph engine service, wherein the stronger the graph engine service processing power, the greater the number of tags. The greater the number of tags of a graph engine service, the more graph fragment data can be distributed to the graph engine service. The graph engine service with stronger processing capacity distributes more graph fragment data, so that the resources of the graph engine service can be fully utilized, the load of the graph engine service with poor capacity is reduced, and the processing efficiency of the graph fragment data is improved.

In some embodiments, a plurality of tags of the newly added graph engine service are obtained; determining the hash value of each label according to a preset hash function; and determining the position of each label in the hash ring according to the hash value of each label. The preset hash function is, for example, an existing hash function such as murmurhash3 or md 5. And constructing a hash ring, wherein the maximum value of the hash function corresponds to the radian value 2 pi, and the minimum value corresponds to the radian value 0. Different hash function values correspond to different positions in the hash ring. The position of each label in the hash ring may be determined in a preset order (clockwise or counterclockwise). The method for calculating the position in the hash ring may refer to the prior art, and is not described in detail.

In some embodiments, in response to the increase of the graph fragment data reaching a preset amount, the number of the newly added graph engine services is determined according to the increase of the graph fragment data. And determining the number of corresponding labels and each label according to the processing capacity of the graph engine service aiming at each newly added graph engine service. The method of and after step S102 is then performed.

In step S204, the graph fragment data to be migrated belonging to the newly added graph engine service is determined according to the distance between the position of the label in the hash ring and the position of each assigned graph fragment data in the hash ring.

The position of each graph fragment data in the hash ring is determined before the graph engine service is added. In some embodiments, sequentially determining, in the hash ring, a label closest to a position of each image fragment data in the hash ring according to a preset order (clockwise or counterclockwise), as a label corresponding to each image fragment data; and regenerating a mapping table of the graph fragment data and the graph engine service according to the label corresponding to each graph fragment data and the graph engine service to which each label belongs. The mapping table includes a corresponding relationship between the identifier of each graph fragment data and the identifier of each graph engine service. The identity of the graph engine service is a unique identity of the graph engine service, unlike the tags of the graph engine service. And the mapping table corresponds to the newly added graph engine service and is graph fragment data to be migrated.

As shown in fig. 3, for example, the newly added engine service S3 has three tags S3.0, S3.1, and S3.2. By calculating the position of each label in the hash ring, three labels S3.0, S3.1 and S3.2 are additionally added in the consistent hash ring. Determining picture slice data P in a counter-clockwise direction₆Corresponding to the tag S3.1, P₉Corresponding to the tag S3.2, P₁Corresponding to tag S3.0.

In step S206, instruction information is sent to the original graph engine service of the newly added graph engine service and the graph fragment data to be migrated, and the graph fragment data to be migrated is migrated from the original graph engine service to the newly added graph engine service.

The identifier of the image fragment data to be migrated may be directly sent to the original image engine service and the newly added image engine service, and the newly added image engine service loads the image fragment data to be migrated according to the identifier of the image fragment data to be migrated (for example, loads the image fragment data to be migrated from a network disk). After the loading is completed, the newly added graph engine service sends a loading completion message to the data migration apparatus (an apparatus that performs the method of the present disclosure). The data migration device sends a deletion instruction to the original image engine service, and the original image engine service deletes the image fragment data to be migrated. The data migration device can inform the newly added graph engine service of the online providing service.

In some embodiments, indication information is sent to each graph engine service, wherein the indication information comprises a mapping table; receiving a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes the loading of the graph fragment data to be migrated according to the mapping table; sending a deletion instruction to an original image engine service; and receiving a deletion completion message sent by the original image engine service, wherein the deletion completion message indicates that the original image engine service completes the deletion of the image fragment data to be migrated according to the mapping table. And sending the mapping table to each graph engine service, wherein each graph engine service can judge whether certain graph fragment data needs to be loaded or deleted according to the mapping table and the stored graph fragment data.

For example, the deletion indication may be sent to the artwork engine service via a heartbeat mechanism. Whether each graph engine service works normally or not can be determined through a heartbeat mechanism, and the deleting indication is sent to the original graph engine service through the message of the heartbeat mechanism, so that the additional sending of the deleting indication can be reduced.

In the method of the above embodiment, the relationship between the graph fragment data and the graph engine service is determined in a hash ring manner, and then the graph fragment data to be migrated belonging to the newly added graph engine service is determined, so that the graph fragment data to be migrated is migrated from the original graph engine service to the newly added graph engine service. When the number of the graph engine services is expanded in the method of the embodiment, data migration between the original graph engine services does not exist, and only data migration from the original graph engine services to the new graph engine services exists, so that the migration quantity of the graph fragment data is reduced, the migration efficiency of the graph fragment data is improved, the real-time performance and the query effect of graph data query can be improved, the user experience is optimized, and the graph modeling benefit is improved.

Further embodiments of the disclosed data migration method are described below in conjunction with FIG. 4.

FIG. 4 is a flow chart of further embodiments of the data migration method of the present disclosure. As shown in fig. 4, the method of this embodiment includes: steps S402 to S408.

In step S402, hash values of the identifications of the respective entities in the graph are determined.

The entities comprise, for example, data of nodes or data of edges of the graph. And hashing the entities in the full-scale graph data.

In step S404, the identifier of the graph fragment data to which each entity belongs is determined according to the hash value of the identifier of each entity.

For example, the hash value is modulo the number N of pieces of image data, and the pieces of image data to which each entity belongs are determined.

In step S406, hash values of the identifications of the respective pieces of map-slice data are determined.

In step S408, the position of each graph slice data in the hash ring is determined according to the identified hash value of each graph slice data, as the position of each assigned graph slice data in the hash ring.

The method of the above embodiment may slice the full amount of graph data and determine the position of each graph slice data in the hash ring.

The data migration apparatus maintains a state machine responsible for communicating with the graph engine service through a heartbeat mechanism. The data migration device can provide the registration function of the graph engine service and can acquire the graph fragment data information loaded by each graph engine service. The data migration device comprises a consistent hashing module which is responsible for mapping the graph data fragments into different graph engine services and maintaining a mapping table from the graph data fragments to the graph engine. When the number M of the graph engine services is increased, the consistent hash algorithm re-determines the mapping relation from the graph data fragments to the graph engine services, generates a new mapping table and sends the new mapping table to other modules of the graph engine service hall. And after the graph engine service acquires the graph data fragments from the data migration device to the mapping table, loading the corresponding graph data fragment data and getting online after the loading is finished.

Some embodiments of querying graph-sliced data of the present disclosure are described below in conjunction with FIG. 5.

FIG. 5 is a flow chart of yet other embodiments of the disclosed data migration method. As shown in fig. 5, the method of this embodiment includes: steps S502 to S510.

In step S502, when the image fragment data to be migrated is migrated from the original image engine service to the newly added image engine service, the data migration apparatus sends the image fragment data and the mapping table of the image engine service to the image data query service.

In step S504, the graph data query service receives a data query request.

The data query request includes, for example: the entities to be queried, e.g., the entities comprise data of nodes or data of edges of the graph.

In step S506, the graph data querying service determines the identifier of the graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried.

For example, the hash value is modulo the number N of the graph fragment data, and the graph fragment data to which the entity to be queried belongs is determined.

In step S508, the graph data query service searches, in the mapping table, a corresponding graph engine service according to the identifier of the graph fragment data to which the entity to be queried belongs, as a target graph engine service.

In step S510, the graph data query service queries the target graph engine service for entities to be queried.

According to the method, the mapping table is sent to the data query service, and the data query service provides query service according to the new mapping table, so that the probability of query errors is reduced.

The present disclosure also provides a data migration apparatus, described below in conjunction with fig. 6.

FIG. 6 is a block diagram of some embodiments of a data migration apparatus of the present disclosure. As shown in fig. 6, the apparatus 60 of this embodiment includes: a tag location determination module 610, a migration information determination module 620, and a migration indication module 630.

The label position determination module 610 is configured to determine the position of the label of the newly added graph engine service in the hash ring.

In some embodiments, the number of the tags of the newly added graph engine service is multiple, and the tag position determining module 610 is configured to obtain the multiple tags of the newly added graph engine service; determining the hash value of each label according to a preset hash function; and determining the position of each label in the hash ring according to the hash value of each label.

The migration information determining module 620 is configured to determine, according to the distance between the position of the label in the hash ring and the position of each allocated graph fragment data in the hash ring, the graph fragment data to be migrated that belongs to the newly added graph engine service.

In some embodiments, the migration information determining module 620 is configured to sequentially determine, in the hash ring according to a preset order, a label closest to a position of each image fragment data in the hash ring, as a label corresponding to each image fragment data; and regenerating a mapping table of the graph fragment data and the graph engine service according to the label corresponding to each graph fragment data and the graph engine service to which each label belongs, wherein the graph fragment data to be migrated corresponds to the newly added graph engine service in the mapping table.

The migration indication module 630 is configured to send indication information to the original graph engine service of the newly added graph engine service and the to-be-migrated graph fragment data, and migrate the to-be-migrated graph fragment data from the original graph engine service to the newly added graph engine service.

In some embodiments, the migration indication module 630 is configured to send indication information to each graph engine service, where the indication information includes a mapping table; receiving a loading completion message sent by the newly added graph engine service, wherein the loading completion message indicates that the newly added graph engine service completes the loading of the graph fragment data to be migrated according to the mapping table; sending a deletion instruction to an original image engine service; and receiving a deletion completion message sent by the original image engine service, wherein the deletion completion message indicates that the original image engine service completes the deletion of the image fragment data to be migrated according to the mapping table.

In some embodiments, the migration indication module 630 is further configured to, when the graph fragment data to be migrated is migrated from the original graph engine service to the newly added graph engine service, send the graph fragment data and a mapping table of the graph engine service to the graph data query service, where the mapping table corresponds to the newly added graph engine service and is the graph fragment data to be migrated.

In some embodiments, the apparatus 60 further comprises: a data location determining module 640, configured to determine hash values of identifiers of respective entities in the graph, where an entity includes data of a node or data of an edge of the graph; determining the identifier of the graph fragment data to which each entity belongs according to the hash value of the identifier of each entity; determining the hash value of the identifier of each graph fragment data; and determining the position of each graph fragment data in the hash ring according to the identified hash value of each graph fragment data, wherein the position of each graph fragment data in the hash ring is used as the position of each distributed graph fragment data in the hash ring.

In some embodiments, the number of tags per graph engine service is determined according to the processing power of the graph engine service, wherein the stronger the graph engine service processing power, the greater the number of tags.

The data migration apparatus in the embodiments of the present disclosure may each be implemented by various computing devices or computer systems, which are described below in conjunction with fig. 7 and 8.

FIG. 7 is a block diagram of some embodiments of a data migration apparatus of the present disclosure. As shown in fig. 7, the apparatus 70 of this embodiment includes: a memory 710 and a processor 720 coupled to the memory 710, the processor 720 configured to perform a data migration method in any of the embodiments of the present disclosure based on instructions stored in the memory 710.

Memory 710 may include, for example, system memory, fixed non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), a database, and other programs.

FIG. 8 is a block diagram of further embodiments of a data migration apparatus of the present disclosure. As shown in fig. 8, the apparatus 80 of this embodiment includes: memory 810 and processor 820 are similar to memory 710 and processor 720, respectively. An input output interface 830, a network interface 840, a storage interface 850, and the like may also be included. These interfaces 830, 840, 850 and the memory 810 and the processor 820 may be connected, for example, by a bus 860. The input/output interface 830 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 840 provides a connection interface for various networked devices, such as a database server or a cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as an SD card and a usb disk.

The present disclosure also provides a data migration system, described below in conjunction with fig. 9.

FIG. 9 is a block diagram of some embodiments of the data migration system of the present disclosure. As shown in fig. 9, the system 9 of this embodiment includes: the data migration device 60/70/80 of any of the preceding embodiments, and a plurality of graph engine services 92.

The graph engine service 92 is configured to receive instruction information sent by the data migration apparatus when the graph engine service 92 is a newly added graph service engine or an original graph engine service of the graph fragment data to be migrated, and implement migration of the graph fragment data to be migrated. The graph engine service 92 may be a server, a PC, or the like.

In some embodiments, the indication information comprises a mapping table; the graph engine service 92 is configured to, in a case where the graph engine service 92 is a newly added graph service engine, complete loading of the graph fragment data to be migrated according to the mapping table, and send a loading completion message to the data migration apparatus 60/70/80, and in a case where the graph engine service 92 is an original graph service engine, receive a deletion instruction sent by the data migration apparatus 60/70/80, complete deletion of the graph fragment data to be migrated according to the mapping table, and send a deletion completion message to the data migration apparatus 60/70/80.

In some embodiments, the system 9 further comprises: the map data query service 94 is configured to receive the mapping table sent by the migration device 60/70/80, and provide a query service according to the mapping table.

In some embodiments, the graph data query service 94 is configured to receive the data query request, determine a hash value of an identifier of the entity to be queried, determine an identifier of graph fragment data to which the entity to be queried belongs according to the hash value of the identifier of the entity to be queried, find, in the mapping table, a corresponding graph engine service according to the identifier of the graph fragment data to which the entity to be queried belongs, serve as a target graph engine service, and query the entity to be queried from the target graph engine service; wherein, the data query request includes: and the entity to be queried comprises data of the nodes or edges of the graph.

As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.