阅读说明：本技术 图划分方法、装置、计算机设备和存储介质 (Graph partitioning method and device, computer equipment and storage medium ) 是由 李云波 张琛 万化 李健 陶学慎 朱仁欢 李洁 赵贇 于 2021-07-30 设计创作，主要内容包括：本申请涉及一种图划分方法、装置、计算机设备和存储介质,涉及图计算技术领域,其中,该图划分方法应用于图划分系统的目标计算节点中,所述图划分系统包括多个计算节点,所述目标计算节点为所述多个计算节点中的任意一个,目标计算节点在对边划分时,可以获取到各个计算节点的存储状态信息,存储状态信息用于指示计算节点已经存储的各个边,然后根据各个计算节点的存储状态信息以及边包括的两个顶点将与该边的关联度较高的计算节点确定为该边对应的候选计算节点,以供候选计算节点存储该边。这样在划分读取到的每个边时,是基于计算节点已经存储的各个边来确定读取到的边应该划分到哪个计算节点,因此,提高了划分结果的合理性。(The application relates to a graph partitioning method, a graph partitioning device, computer equipment and a storage medium, and relates to the technical field of graph computation, wherein the graph partitioning method is applied to a target computing node of a graph partitioning system, the graph partitioning system comprises a plurality of computing nodes, the target computing node is any one of the computing nodes, when a target computing node partitions a side, the target computing node can acquire storage state information of each computing node, the storage state information is used for indicating each side which is stored by the computing node, and then the computing node with higher association degree with the side is determined as a candidate computing node corresponding to the side according to the storage state information of each computing node and two vertex points included by the side, so that the candidate computing node can store the side. In this way, when each read edge is divided, the calculation node to which the read edge should be divided is determined based on each edge already stored by the calculation node, so that the rationality of the division result is improved.)

1. A graph partitioning method applied to a target computing node of a graph partitioning system, wherein the graph partitioning system comprises a plurality of computing nodes, and the target computing node is any one of the plurality of computing nodes, the method comprising:

acquiring a target edge set, wherein the target edge set is any one of a plurality of edge sets obtained by initially dividing an original image data set, and the target edge set comprises a plurality of edges;

sequentially reading each edge in the target edge set, and determining a candidate computing node corresponding to each edge from each computing node according to a vertex included by the edge and storage state information of each computing node for each read edge so as to trigger the candidate computing node to store the edge and update the storage state information of the candidate computing node;

wherein the storage status information is used for indicating each edge which the computing node has stored.

2. The method according to claim 1, wherein determining, from each of the computing nodes, a candidate computing node corresponding to the edge according to the vertex included in the edge and the storage state information of each of the computing nodes comprises:

determining vertex information corresponding to the vertex included by the edge according to the vertex included by the edge and storage state information of each computing node, wherein the vertex information is used for indicating the distribution state of the vertex included by the edge at each computing node;

and determining candidate computing nodes corresponding to the edges from each computing node according to vertex information corresponding to the vertexes included by the edges.

3. The method according to claim 2, wherein determining candidate compute nodes corresponding to the edge from each of the compute nodes according to vertex information corresponding to vertices included in the edge comprises:

if the vertex information corresponding to the vertexes included by the edge indicates that the two vertexes included by the edge are respectively present in different computing nodes, caching the edge into a local cache queue of the target computing node;

and determining candidate computing nodes corresponding to the edges stored in the local cache queue of the target computing node according to the edges stored in the local cache queue of each computing node.

4. The method according to claim 3, wherein the determining, according to the edges stored in the local cache queue of each computing node, the candidate computing node corresponding to each edge stored in the local cache queue of the target computing node comprises:

reading a target edge from the local cache queue, wherein the target edge is any one of edges cached in the local cache queue;

acquiring information of an edge which is in common vertex with the target edge from local cache queues of other computing nodes except the target computing node;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the vertexes included by the target edges, the storage state information of the computing nodes and the information of the edges sharing the vertexes with the target edges.

5. The method of claim 4, wherein the obtaining information of edges that are co-vertex with the target edge from local cache queues of other computing nodes except the target computing node comprises:

generating a vertex request according to two vertexes included by the target edge, and sending the vertex request to the other computing nodes, wherein the vertex request is used for indicating the other computing nodes to send information of edges, which share a vertex with the target edge, in local cache queues of the other computing nodes to the target computing nodes;

and receiving the information of the edge which is in the local cache queue of the other computing node and has the vertex which is the same as the target edge and is sent by the other computing node.

6. The method according to claim 4, wherein the determining, from the vertices included in the target edge, the storage state information of each of the computing nodes, and the information of the edge that is co-vertex with the target edge, a candidate computing node corresponding to the target edge from each of the computing nodes comprises:

counting the number of edges which are stored in each computing node and share a vertex with a first vertex of the target edge according to the vertex included by the target edge, the storage state information of each computing node and the information of the edge which shares the vertex with the target edge, so as to obtain a first number; counting the number of edges which are stored in each computing node and have the same vertex with the second vertex of the target edge to obtain a second number;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the first quantity and the second quantity.

7. The method of claim 6, wherein determining the candidate compute node corresponding to the target edge from the respective compute nodes according to the first number and the second number comprises:

determining the computing nodes corresponding to the maximum number according to the first number and the second number;

and determining candidate computing nodes corresponding to the target edge from the computing nodes corresponding to the maximum number based on a preset strategy.

8. The method of claim 1, wherein after sequentially reading each edge in the target edge set, the method further comprises:

detecting whether any vertex in two vertexes included by the edge is locked by other computing nodes except the target computing node;

if the edge is locked by other computing nodes, continuously detecting whether any vertex in the two vertices included by the edge is locked by other computing nodes except the target computing node until the two vertices included by the edge are not locked by other computing nodes;

if the edge is not locked by other computing nodes, marking that two vertexes included by the edge are in a locked state in a shared list maintained by the graph partitioning system, and determining candidate computing nodes corresponding to the edge from each computing node according to the vertexes included by the edge and the storage state information of each computing node.

9. The method according to claim 2, wherein determining candidate compute nodes corresponding to the edge from each of the compute nodes according to vertex information corresponding to vertices included in the edge comprises:

and if the vertex information corresponding to the vertex included by the edge indicates that the two vertices included by the edge appear in the same computing node at the same time, determining a candidate computing node corresponding to the edge based on a preset strategy from the computing nodes in which the two vertices included by the edge appear at the same time.

10. The method according to claim 2, wherein determining candidate compute nodes corresponding to the edge from each of the compute nodes according to vertex information corresponding to vertices included in the edge comprises:

if the vertex information corresponding to the vertex included in the edge indicates that only one vertex appears in the computation node, determining a candidate computation node corresponding to the edge based on a preset strategy from the computation nodes where one vertex appears in the two vertices included in the edge.

11. The method according to claim 2, wherein determining candidate compute nodes corresponding to the edge from each of the compute nodes according to vertex information corresponding to vertices included in the edge comprises:

and if the vertex information corresponding to the vertex included by the edge indicates that two vertices included by the edge do not appear in each computing node, determining a candidate computing node corresponding to the edge from each computing node based on a preset strategy.

12. The method according to any one of claims 7 and 9 to 11, wherein the predetermined policy is to select the computing node with the smallest load.

13. A graph partitioning apparatus applied to a target computing node of a graph partitioning system, the graph partitioning system including a plurality of computing nodes, the target computing node being any one of the plurality of computing nodes, the apparatus comprising:

an obtaining module, configured to obtain a target edge set, where the target edge set is any one of multiple edge sets obtained by initially dividing an original graph data set, and the target edge set includes multiple edges;

the dividing module is used for sequentially reading each edge in the target edge set, and for each read edge, determining a candidate computing node corresponding to the edge from each computing node according to a vertex included by the edge and storage state information of each computing node so as to trigger the candidate computing node to store the edge and update the storage state information of the candidate computing node; wherein the storage status information is used for indicating each edge which the computing node has stored.

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 12.

15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 12.

Technical Field

The present application relates to the field of graph computation technologies, and in particular, to a graph partitioning method and apparatus, a computer device, and a storage medium.

Background

The graph is an abstract data structure for representing the association relationship between objects, and is described by using vertexes and edges, wherein the vertexes are used for representing the objects, and the edges are used for representing the association relationship between the objects. The graph calculation is a calculation method for expressing and solving a problem by using a graph as a data model. In the process of graph calculation, the current main-flow graph calculation system usually needs to load the whole graph into a memory. However, with the development of the times, the size of the graph is larger and larger, and the occupied memory is larger and larger. However, the memory of a single computer device is limited, so that the single computer device cannot process the data of the super-large scale map, and the data of the super-large scale map needs to be divided into a plurality of small maps, and then different computer devices respectively process each small map.

In the prior art, computer devices generally read each edge in the data of the super-large-scale map one by one, then randomly divide each read edge into one computer device, and divide the edges on the basis of the principle that the number of the edges stored in each computer device is equal or approximately equal.

However, in the graph partitioning method, multiple adjacent edges are easily placed into different computer devices in the partitioning process, so that the traffic across partitions is increased, and finally, the computational efficiency of the graph computation stage is reduced. Therefore, the division result is not reasonable.

Disclosure of Invention

In view of the above, it is necessary to provide a graph partitioning method, apparatus, computer device, and storage medium capable of improving the rationality of the partitioning result in view of the above technical problems.

A graph partitioning method is applied to a target computing node of a graph partitioning system, the graph partitioning system comprises a plurality of computing nodes, and the target computing node is any one of the computing nodes, and the method comprises the following steps:

acquiring a target edge set, wherein the target edge set is any one of a plurality of edge sets obtained by initially dividing an original image data set, and the target edge set comprises a plurality of edges;

sequentially reading each edge in the target edge set, and determining candidate computing nodes corresponding to the edge from each computing node according to the vertex included by the edge and the storage state information of each computing node for each read edge so as to trigger the storage edge of the candidate computing node and update the storage state information of the candidate computing node;

wherein the storage status information is used to indicate the edges that the compute node has stored.

In one embodiment, determining candidate computing nodes corresponding to edges from computing nodes according to vertices included by the edges and storage state information of the computing nodes includes:

determining vertex information corresponding to the vertexes included by the edges according to the vertexes included by the edges and the storage state information of each computing node, wherein the vertex information is used for indicating the distribution states of the vertexes included by the edges at each computing node;

and determining candidate computing nodes corresponding to the edges from the computing nodes according to the vertex information corresponding to the vertexes included by the edges.

In one embodiment, determining candidate computing nodes corresponding to edges from computing nodes according to vertex information corresponding to vertices included in the edges includes:

if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges are respectively present in different computing nodes, caching the edges into a local cache queue of a target computing node;

and determining candidate computing nodes corresponding to the edges stored in the local cache queue of the target computing node according to the edges stored in the local cache queue of the computing nodes.

In one embodiment, determining candidate computing nodes corresponding to edges stored in a local cache queue of a target computing node according to the edges stored in the local cache queue of each computing node includes:

reading a target edge from the local cache queue, wherein the target edge is any one of edges cached in the local cache queue;

acquiring information of edges which are in common vertex with the target edges from local cache queues of other computing nodes except the target computing node;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the vertexes included by the target edges, the storage state information of the computing nodes and the information of the edges sharing the vertexes with the target edges.

In one embodiment, obtaining information of an edge that is co-vertex with a target edge from a local cache queue of a computing node other than the target computing node comprises:

generating a vertex request according to two vertexes included by the target edge, and sending the vertex request to other computing nodes, wherein the vertex request is used for indicating the other computing nodes to send information of edges which are in the same vertex with the target edge in local cache queues of the other computing nodes to the target computing node;

and receiving the information of the edges which are in the local cache queues of the other computing nodes and are in the vertex sharing with the target edge, which is sent by the other computing nodes.

In one embodiment, determining candidate computing nodes corresponding to a target edge from each computing node according to a vertex included by the target edge, storage state information of each computing node, and information of an edge sharing the vertex with the target edge includes:

counting the number of edges which are stored in each computing node and share a vertex with a first vertex of the target edge according to the vertex included by the target edge, the storage state information of each computing node and the information of the edge which shares the vertex with the target edge to obtain a first number; counting the number of edges which are stored by each computing node and have the same vertex with the second vertex of the target edge to obtain a second number;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the first quantity and the second quantity.

In one embodiment, determining candidate compute nodes corresponding to the target edge from the compute nodes according to the first number and the second number includes:

determining the computing nodes corresponding to the maximum number according to the first number and the second number;

and determining candidate computing nodes corresponding to the target edge from the computing nodes corresponding to the maximum number based on a preset strategy.

In one embodiment, after reading each edge in the target edge set in sequence, the method further includes:

detecting whether any vertex in two vertexes included by the edge is locked by other computing nodes except the target computing node;

if the two vertexes are locked by other computing nodes, continuously detecting whether any one vertex in the two vertexes included by the edge is locked by other computing nodes except the target computing node or not until the two vertexes included by the edge are not locked by other computing nodes;

if the edge is not locked by other computing nodes, marking two vertexes included by the edge in a shared list maintained by the graph partitioning system to be in a locked state, and determining candidate computing nodes corresponding to the edge from the computing nodes according to the vertexes included by the edge and the storage state information of the computing nodes.

In one embodiment, determining candidate computing nodes corresponding to edges from computing nodes according to vertex information corresponding to vertices included in the edges includes:

and if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges simultaneously appear in the same computing node, determining candidate computing nodes corresponding to the edges from the computing nodes of the two vertexes included by the simultaneously appearing edges based on a preset strategy.

In one embodiment, determining candidate computing nodes corresponding to edges from computing nodes according to vertex information corresponding to vertices included in the edges includes:

and if the vertex information corresponding to the vertex included by the edge indicates that only one vertex appears in the computing node in the two vertexes included by the edge, determining a candidate computing node corresponding to the edge based on a preset strategy from the computing nodes of the vertex in the two vertexes included by the edge.

In one embodiment, determining candidate computing nodes corresponding to edges from computing nodes according to vertex information corresponding to vertices included in the edges includes:

and if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges do not appear in each computing node, determining candidate computing nodes corresponding to the edges from each computing node based on a preset strategy.

In one embodiment, the predetermined policy is to select the compute node with the least load.

A graph partitioning apparatus applied to a target compute node of a graph partitioning system, the graph partitioning system including a plurality of compute nodes, the target compute node being any one of the plurality of compute nodes, the apparatus comprising:

the acquisition module is used for acquiring a target edge set, wherein the target edge set is any one of a plurality of edge sets obtained by initially dividing an original image data set, and comprises a plurality of edges;

the dividing module is used for sequentially reading each edge in the target edge set, and for each read edge, determining a candidate computing node corresponding to the edge from each computing node according to a vertex included by the edge and the storage state information of each computing node so as to trigger the storage edge of the candidate computing node and update the storage state information of the candidate computing node; wherein the storage status information is used to indicate the edges that the compute node has stored.

A computer device comprising a memory storing a computer program and a processor implementing the steps of the method of any of the above embodiments when the processor executes the computer program.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any of the preceding embodiments.

The graph dividing method, the graph dividing device, the computer equipment and the storage medium can improve the rationality of the dividing result. According to the graph partitioning method, an original graph data set is initially partitioned to obtain a plurality of edge sets, and then each computing node concurrently partitions edges in different edge sets, so that the partitioning efficiency is improved compared with a single computing node partitioning mode. When the target computing node divides the edge, the storage state information of each computing node can be acquired, the storage state information is used for indicating each edge which is stored by the computing node, then the candidate computing node corresponding to the edge is determined according to the storage state information of each computing node and two vertexes included by the edge, therefore, when each read edge is divided, the computing node to which the read edge is divided is determined based on each edge which is stored by the computing node, namely, the read edge is divided based on the relation between the read edge and other edges, and therefore the rationality of the dividing result is improved.

Drawings

FIG. 1 is a schematic diagram of a graph partitioning system in one embodiment;

FIG. 2 is a flow diagram illustrating a graph partitioning method according to one embodiment;

FIG. 3 is a diagram illustrating an embodiment of an original graph data set before it is initially partitioned;

FIG. 4 is a diagram illustrating an embodiment in which each compute node obtains an edge set;

FIG. 5 is a diagram that illustrates the sharing of storage state information among compute nodes, according to one embodiment;

FIG. 6 is a flowchart illustrating a method for determining candidate compute nodes corresponding to read edges in one embodiment;

FIG. 7 is a flowchart illustrating a method for a target compute node to partition edges stored in a local cache queue according to an embodiment;

FIG. 8 is a flowchart illustrating a method for a target compute node to obtain information for an edge that shares a vertex with a target edge in one embodiment;

FIG. 9 is a flowchart illustrating a method for a target compute node to obtain information for edges in a local cache queue of other compute nodes that share a vertex with a target edge in one embodiment;

FIG. 10 is a flowchart illustrating a graph partitioning method according to another embodiment;

a comparison of plot time using the three methods is shown in fig. 11;

FIG. 12 is a diagram illustrating the effect of cache window size on the copy factor;

FIG. 13 is a schematic diagram showing the effect of another cache window size on the copy factor;

FIG. 14 is a diagram illustrating the effect of the number of compute nodes on the replica factor;

FIG. 15 is a schematic diagram showing the effect of another number of compute nodes on the replica factor;

FIG. 16 is a block diagram showing the structure of a graph dividing apparatus according to an embodiment;

FIG. 17 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. The technical features mentioned in the different embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

Current mainstream graph computing systems typically require the entire graph to be loaded into memory prior to graph computation. However, with the development of the times, the size of the graph is larger and larger, and the occupied memory is larger and larger. However, the memory of a single computer device is limited, so that the single computer device cannot process the data of the super-large-scale map, and the data of the super-large-scale map needs to be divided into a plurality of small maps, and then different computer devices respectively process each small map, namely distributed map calculation.

In the process of graph computation in a distributed environment, communication between different computer devices (i.e., different partitions), i.e., across partitions, is required. Because of the super nodes in the graph, the ratio of the number of the edges owned by the super nodes to the total number of the edges of the whole graph is obviously not. If the vertex shared with the super node is divided into computer devices (partitions) different from the super node, a large amount of communication resources are consumed in the graph calculation process, so that the communication cost is greatly increased, and finally, the calculation time of the whole graph is prolonged.

In order to reduce the communication overhead in the graph calculation process, when the graph is divided, the division is performed only on the principle that the numbers of the edges stored in the computer devices (partitions) are equal or approximately equal, so that the division result is unreasonable, and the subsequent graph calculation time is prolonged and the efficiency is low.

Further, because the whole graph cannot be placed in the memory in the single-machine memory, the current graph partitioning method is generally a stream graph partitioning method, which is a method in which graph data sequentially enters the computer device in a side stream form for partitioning, and the method can only read information on one side each time, so that the amount of information according to the edge partitioning is small, and the local and global information is not fully utilized, and thus, the partitioning result is unreasonable.

Based on this, the graph partitioning method provided by the embodiment of the application provides that multiple computing nodes are partitioned in parallel, so that the partitioning efficiency is improved, and meanwhile, the storage state information of the multiple computing nodes is shared, so that when each computing node partitions the read edges, the maximum local and global information can be obtained, so that the rationality of the partitioning result is improved, the communication overhead is reduced for the subsequent graph computing process, and the processing efficiency is improved.

In the following, a brief description will be given of an implementation environment related to the graph partitioning method provided in the embodiment of the present application.

As shown in FIG. 1, the implementation environment is a graph partitioning system, wherein the graph partitioning system includes a plurality of compute nodes, 4 compute nodes are exemplarily shown in FIG. 1, P1, P2, P3 and P4, respectively, and each compute node performs the same task. The connecting lines among the P1, the P2, the P3 and the P4 indicate that the computing nodes can communicate with each other through a network. Each computing node can also share the storage state information of the computing node to other computing nodes.

Optionally, in this embodiment of the present application, the computing node may be a terminal or a server, where the server may be implemented by an independent server or a server cluster formed by multiple servers. The terminal is, for example, a desktop computer, a notebook computer, or the like.

As shown in fig. 2, a graph partitioning method is provided, which may be applied to a plurality of computing nodes included in the graph partitioning system shown in fig. 1, and for convenience of description, the method is described below by taking as an example that the method is applied to a target computing node of the graph partitioning system, where the target computing node is any one of the plurality of computing nodes included in the graph partitioning system, and the method includes the following steps:

in step 201, a target computing node obtains a target edge set.

The target edge set is any one of a plurality of edge sets obtained by initially dividing the original graph data set, and the target edge set comprises a plurality of edges.

In this embodiment, one of the plurality of computing nodes included in the graph partitioning system is randomly selected as a master computing node, where the randomly selected master computing node may be a target computing node in this embodiment.

The main computing node is mainly used for carrying out initial division on the original graph data set. The master computing node may peer the entire data set into k blocks of data, k representing the number of computing nodes included in the graph partitioning system. After equally dividing the original graph data set into k blocks by the main computing node, k edge sets are obtained, and the edge sets are transmitted to each computing node (including the main computing node) included in the graph dividing system through a communication network.

Thus, each compute node is assigned to a set of edges. For example, the total number of edges included in the original graph data set is E, then the number of edges included in each edge set may be EAnd (3) strips. When the division cannot be performed, redundant edges are randomly distributed to each computing node.

As shown in fig. 3, fig. 3 is a schematic diagram of the original graph data set before the original graph data set is initially divided, and the graph dividing system shown in fig. 3 includes four computing nodes, P1, P2, P3 and P4, respectively, where P1 is the main computing node, and the original graph data set is initially divided by P1. Fig. 4 is a schematic diagram of each computing node acquiring an edge set.

Step 202, the target computing node sequentially reads each edge in the target edge set, and for each read edge, determines a candidate computing node corresponding to the edge from each computing node according to a vertex included by the edge and the storage state information of each computing node, so as to trigger the storage edge of the candidate computing node and update the storage state information of the candidate computing node.

The target computing node sequentially reads each edge in the target edge set, that is, the edges included in the target edge set sequentially enter the target computing node in an edge stream form.

The target computing node processes each read edge according to the graph partitioning method provided by the embodiment of the application, and when each computing node included in the graph partitioning system completes processing of an edge in the edge set allocated to the computing node, the whole graph partitioning process is completed.

The following describes a process in which the target computing node processes each read edge:

first, after reading an edge, a target computing node may obtain two vertices included in the edge, where the two vertices have vertex identifiers, and the vertex identifiers are used to distinguish the vertices.

Secondly, the target computing node may obtain storage state information of each computing node (including the target computing node) included in the graph partitioning system. The storage state information is used to indicate each edge that the compute node has stored, and an edge is composed of a vertex, so it can also be understood that the storage state information is vertex storage state information.

Optionally, in this embodiment of the present application, the target computing node may share its own storage state information with other computing nodes. Correspondingly, other computing nodes can also share the storage state information of the other computing nodes to the target computing node, and the target computing node acquires the storage state information of the other computing nodes based on the sharing of the other computing nodes.

Optionally, in this embodiment of the application, the graph partitioning system maintains a shared list, and each computing node may update data in the shared list, and may also read data from the shared list. As shown in fig. 5, four computing nodes P1, P2, P3, and P4 are shown in fig. 5, where the computing nodes P1, P2, P3, and P4 may write their own storage state information a (P1), a (P2), a (P3), and a (P4) into a shared list, thereby forming a union a (P1) @ a (P2) @ (P3) @ ua (P4) of the storage state information of a plurality of computing nodes in the shared list. Each computing node may also read the storage state information of other computing nodes from the shared list. Based on this, the target computing node may obtain storage status information of each computing node included in the graph partitioning system from the shared list.

And finally, the target computing node determines candidate computing nodes corresponding to the edges from the computing nodes according to the vertexes included by the edges and the storage state information of the computing nodes.

The target computing node may determine, according to a vertex included in the edge and storage state information of each computing node (including storage state information of the target computing node itself), a computing node with a higher association degree with the edge, and determine the computing node with the higher association degree as a candidate computing node corresponding to the edge.

The association degree of each computing node with the edge is represented by the number of edges which are stored in each computing node and share the vertex with the edge, and the greater the number, the higher the association degree is, and the smaller the number is, the lower the association degree is.

The method for determining the association degree between each computing node and the edge may be: the target computing node may count the number of edges that have been stored in each computing node and that share a vertex with the edge, and then the target computing node may determine the association degree between each computing node and the edge according to the number of edges that have been stored in each computing node and that share a vertex with the edge.

Optionally, in the partitioning algorithm provided in the embodiment of the present application, when determining the candidate computing node corresponding to the edge, the main basis is vertex information of each current computing node, and each computing node is evaluated and scored based on the vertex information, so as to select a more appropriate partition (i.e., a candidate computing node). When the division algorithm is used for dividing, vertex information of the edges which are divided and stored in each computing node needs to be counted, so that the more the edges which are divided and stored in each computing node are, the larger the gain of the division algorithm is.

In practical applications, the candidate computing node corresponding to the edge may be the target computing node itself, or may be another computing node other than the target computing node.

And if the candidate computing node corresponding to the edge is the target computing node, the target computing node stores the edge. Since the storage state information of the target computing node changes after the edge is stored, the target computing node needs to update its own storage state information.

If the candidate computing node corresponding to the edge is other computing nodes except the target computing node, the target computing node may send the edge to the candidate computing node, and the candidate computing node receives the edge, stores the edge, and updates the storage state information of the candidate computing node.

Optionally, the target computing node and the candidate computing node update their respective storage state information in a shared list maintained by the graph partitioning system.

According to the graph partitioning method provided by the embodiment of the application, the original graph data set is initially partitioned to obtain a plurality of edge sets, and then each computing node concurrently partitions edges in different edge sets, so that the partitioning efficiency is improved compared with a single computing node partitioning mode. When the target computing node divides the edge, the storage state information of each computing node can be acquired, the storage state information is used for indicating each edge which is already stored by the computing node, and then the computing node with higher association degree with the edge is determined as a candidate computing node corresponding to the edge according to the storage state information of each computing node and two top points included by the edge, so that the candidate computing node can store the edge. In this way, when each read edge is divided, which computing node the read edge should be divided into is determined based on each edge already stored by the computing node, that is, the read edge is divided based on the relationship between the read edge and other edges, so that the rationality of the division result is improved.

In an embodiment of the present application, as shown in fig. 6, a process of determining, by a target computing node, a candidate computing node corresponding to a read edge from each computing node includes the following:

step 601, the target computing node determines vertex information corresponding to the vertices included by the edges according to the vertices included by the edges and the storage state information of each computing node.

The vertex information is used for indicating the distribution state of the vertices included by the edge at each computing node.

Optionally, the vertex information corresponding to the vertex included in the edge may include the following: whether the vertex included by the edge exists in each compute node and the number of vertices included by the edge in each compute node.

For example, the graph partitioning system includes 4 compute nodes, P1, P2, P3, and P4, respectively, where the target compute node is any one of the four compute nodes, e.g., the target compute node is P2. The target compute node P2 reads an edge as an A-edge, which includes two vertices A1 and A2. Then the vertex information corresponding to the vertex included in the a-edge may be: the number of edges including the vertex a1 and the number of edges including the vertex a2 among the edges already stored in the computation node P1, the number of edges including the vertex a1 and the number of edges including the vertex a2 among the edges already stored in the computation node P2, the number of edges including the vertex a1 and the number of edges including the vertex a2 among the edges already stored in the computation node P3, the number of edges including the vertex a1 among the edges already stored in the computation node P4, and the number of edges including the vertex a 2.

Where the number may be 0, the number of 0 representing the vertices in the compute node where the edge is not included.

Step 602, the target computing node determines candidate computing nodes corresponding to the edges from each computing node according to vertex information corresponding to vertices included by the edges.

In the embodiment of the present application, the process of determining, by a target computing node, a candidate computing node corresponding to an edge according to vertex information corresponding to a vertex included in the edge is as follows:

s1, if the vertex information corresponding to the vertices included in the edge indicates that the two vertices included in the edge appear in the same computing node at the same time, the target computing node determines, based on a preset policy, a candidate computing node corresponding to the edge from the computing nodes of the two vertices included in the edge that appear at the same time.

For example, the graph partitioning system includes 4 compute nodes, P1, P2, P3, and P4, respectively, where the target compute node is any one of the four compute nodes, e.g., the target compute node is P2. The target compute node P2 reads an edge as an A-edge, which includes two vertices A1 and A2.

In the case that a1 and a2 coexist in the computing node P1, a1 and a2 coexist in the target computing node P2, and a1 and a2 coexist in the computing node P3, and in this case, the computing nodes P1, P2, and P3 are computing nodes where two vertices included in an edge appear simultaneously, candidate computing nodes corresponding to the edge are determined from the computing nodes P1, P2, and P3 based on a preset policy.

Optionally, the preset policy is to select the compute node with the smallest load, that is, select the compute node with the smallest load from the compute nodes P1, P2, and P3 as the candidate compute node corresponding to the edge.

The load of each computing node is determined according to the number of edges already stored in each computing node, so that the target computing node can determine the load of each computing node through the storage state information of each computing node.

Optionally, the preset policy is to count the number of edges sharing a vertex with the edge in each computing node, and select the computing node corresponding to the largest number as the candidate computing node corresponding to the edge. And if the maximum number of corresponding computing nodes is multiple, selecting the computing node with the minimum load or randomly selecting one computing node as a candidate computing node corresponding to the edge.

S2, if the vertex information corresponding to the vertex included in the edge indicates that only one vertex appears in the computation node, the target computation node determines a candidate computation node corresponding to the edge based on a preset policy from the computation nodes where one vertex appears in the two vertices included in the edge.

Taking the above example as an example, the target computing node is P2, the edge read by the target computing node P2 is an a-edge, the a-edge includes two vertices a1 and a2, the vertex a1 appears in the target computing node P2 and the computing nodes P3 and P4, and the vertex a2 does not appear in any computing node, in which case, the target computing node P2 and the computing nodes P3 and P4 are computing nodes where one vertex of the two vertices included by the edge appears. The target compute node determines candidate compute nodes corresponding to the edges from the compute nodes P2, P3, and P4 based on a preset policy.

Optionally, the preset policy is: the computing node with the least load is selected.

Optionally, the preset policy is: and counting the number of edges which share a vertex with the edge in each computing node, and selecting the computing node corresponding to the maximum number as a candidate computing node corresponding to the edge.

And S3, if the vertex information corresponding to the vertex included in the edge indicates that the two vertices included in the edge do not appear in each computing node, determining a candidate computing node corresponding to the edge from each computing node based on a preset strategy.

Bearing the above example, the target computing node is P2, the edge read by the target computing node P2 is an a-edge, the a-edge includes two vertices a1 and a2, neither the vertex a1 nor the vertex a2 is stored in the target computing node P2 and the computing nodes P1, P3, and P4, and then the target computing node P2 determines a candidate computing node corresponding to the edge from P1, P2, P3, and P4 based on a preset policy.

Optionally, the preset policy is: the computing node with the least load is selected.

Optionally, the preset policy is: and counting the number of edges which share a vertex with the edge in each computing node, and selecting the computing node corresponding to the maximum number as a candidate computing node corresponding to the edge.

In addition, in the embodiments of the present application, the schemes disclosed in S1, S2, and S3 may be combined to divide the edges.

And S4, if the vertex information corresponding to the vertex included in the edge indicates that the two vertexes included in the edge are respectively present in different computing nodes, caching the edge into a local cache queue of the target computing node. And determining candidate computing nodes corresponding to the edges stored in the local cache queue of the target computing node according to the edges stored in the local cache queue of the computing nodes.

Bearing in the above example, the target computing node is P2, the edge read by the target computing node P2 is an a-edge, and the a-edge includes two vertices a1 and a2, where the vertex a1 appears in the target computing node P2 and the computing node P3, and the vertex a2 appears in the computing node P1 and the computing node P4, that is, the two vertices appear in different computing nodes, respectively. In this case, the target computing node P2 cannot determine into which computing node the a-edge is divided, that is, cannot determine the candidate computing node corresponding to the a-edge. In this case, the target computing node P2 may cache the a-edge into the target computing node's local cache queue.

The local cache queue of the target compute node is explained below.

In this embodiment of the present application, the target computing node may set a cache window based on its own memory capacity. The buffer window stores data in the form of buffer queues. After the target computing node initializes the cache queue, the cache queue may be used to store edges for which the target computing node is temporarily unable to determine candidate computing nodes.

Next, a process of caching the edge in the local cache queue of the target computing node by the target computing node is described.

And the target computing node determines whether the local cache queue is full, and if the local cache queue is not full, the target computing node caches the edge in the local storage queue.

If the processing result is full, the target computing node stops processing the edges in the target edge set, reads the edges arranged at the top from the local cache queue, determines the candidate computing nodes corresponding to the edges arranged at the top according to the edges stored in the local cache queue of each computing node, stores the edges which cannot determine the candidate computing nodes into the cache queue after dividing the edges arranged at the top, and then the target computing node can continue processing the edges in the target edge set.

After the edges in the target edge set are divided, the target computing node may determine, according to the edges stored in the local cache queue of each computing node, candidate computing nodes corresponding to each edge stored in the local cache queue of the target computing node, for details, see the following description.

According to the embodiment of the application, vertex information corresponding to the vertex included by the edge is determined according to the vertex included by the edge and the storage state information of each computing node, and then candidate computing nodes corresponding to the edge are determined from each computing node based on the vertex information corresponding to the vertex included by the edge, namely, the edge is divided into the candidate computing nodes for storage. The read edges are divided based on the distribution state of the vertexes included by the edges at each computing node, the storage state information of each computing node is obtained under the distributed condition, and the candidate computing nodes corresponding to the edges are determined based on the storage state information of each computing node, so that the performance of the graph division result is optimized, and the graph division time is greatly reduced.

It should be noted that each computing node is provided with a local cache window, and the size of the local cache window of each computing node is set based on the memory of each computing node itself. Because each computing node can obtain the data of the edges stored in the cache queues in the cache windows of other computing nodes, for the whole graph dividing system, the edges are cached by the sum of the cache windows of a plurality of computing nodes, and the larger the capacity of the cache window is, the more information can be obtained when the edges in the cache window are processed, so that each computing node can conveniently obtain more global information, and the rationality of edge division is improved. If the capacity of the cache window is small, the number of edges stored in the cache window is small, that is, the amount of information in the cache window is limited, and the edge division performance is poor.

Optionally, when determining the candidate computing nodes corresponding to the edges, the partitioning algorithm provided in the embodiment of the present application not only depends on the vertex information of each current computing node, but also depends on the vertex information of the edges cached in the cache window. Specifically, each computing node is scored according to the vertex information of the edge which is divided and stored by each computing node and the vertex information of the edge in the cache window, so that a more appropriate candidate computing node is selected. In each computing node, the more the cached edges in the cache window are, the more the cached vertex information is, and the larger the gain on the partitioning algorithm is.

Based on this, the capacity of the cache window may also have an impact on determining candidate computing nodes. The method is mainly characterized in that as the capacity of the cache window is increased, when each edge in the cache window is divided, the division algorithm can take more vertex information of other edges from the cache window, and meanwhile, based on the vertex and edge states of each existing computing node, compared with a scheme without the cache window or with a small capacity of the cache window, the distributed cache window scheme based on the larger capacity can avoid a situation of local optimization when the division algorithm is divided, so that the division quality is improved.

Next, as shown in fig. 7, a process of dividing edges stored in the local cache queue by the target computing node will be described.

In step 701, a target computing node reads a target edge from a local cache queue.

In the embodiment of the application, after the edge division in the target edge set is completed, the target computing node divides the edges stored in the local cache queue no matter whether the local cache queue is full or not.

And the target edge is any one of edges cached in the local cache queue. Alternatively, the target edge may be the edge that is arranged at the top in the local buffer queue.

In step 702, the target computing node obtains information of an edge which is vertex-shared with the target edge from a local cache queue of other computing nodes except the target computing node.

In an optional implementation manner, the target computing node may share the edge cached in the local cache queue to other computing nodes, and correspondingly, the target computing node may also obtain information of the edge in the local cache queue of the other computing node shared by the other computing nodes.

The target computing node can obtain the information of the edge which is vertex-shared with the target edge from the local cache queues of other computing nodes. Wherein the information of the edge that is co-vertex with the target edge includes two vertices that the edge that is co-vertex with the target edge includes.

In an alternative implementation, as shown in fig. 8, the process of the target computing node acquiring information of an edge that is co-vertex with the target edge includes the following steps:

step 801, a target computing node generates a vertex request according to two vertices included by a target edge, and sends the vertex request to other computing nodes.

The vertex request is used for instructing other computing nodes to send information of edges which are in the local cache queues of the other computing nodes and share the vertex with the target edge to the target computing node.

In step 802, the target computing node receives information of an edge which is in a local cache queue of the other computing node and has a vertex which is the same as the target edge, and the information is sent by the other computing node.

In this embodiment of the application, a vertex request carries two vertices included by a target edge, for example, the target edge is e (u, v), the target computing node asks for information related to the vertices u, v from other computing nodes by sending the vertex request, and after receiving the vertex request, the other computing nodes may extract edges that share vertices with the vertices u, v, that is, edges adjacent to the vertex u and edges adjacent to the vertex v, from respective local cache queues, and send the edges that share vertices with the target edge to the target computing node. In this way, the target computing node may receive information of edges of other computing nodes that are co-vertex with the target edge in local cache queues of other computing nodes sent by other computing nodes.

As shown in fig. 9, the target computing node is P1, and the target edge is e (u, v), where the target computing node P1 sends vertex requests to computing node P2, computing node P3, and computing node P4, respectively, and the vertex requests are represented by arrows 1, 2, and 3, respectively, and are used to instruct computing node P2, computing node P3, and computing node P4 to extract an edge that is vertex-shared with vertex u or vertex v from respective local cache queues, respectively, and computing node P2, computing node P3, and computing node P4 send an edge that is vertex-shared with vertex u or vertex v to target computing node P1, where 1 ', 2 ', and 3 ' represent feedback information of the partitioned computing node P2, computing node P3, and computing node P4 to target computing node P1, respectively. If no edge co-vertex with vertex u or vertex v exists in the local cache queues of compute node P2, compute node P3, and compute node P4, then an empty is returned.

In step 703, the target computing node determines a candidate computing node corresponding to the target edge from each computing node according to the vertex included by the target edge, the storage state information of each computing node, and the information of the edge sharing the vertex with the target edge.

In an optional implementation manner, a target computing node counts the number of edges, which share a vertex, of a first vertex of a target edge, in edges stored by each computing node according to the vertex included by the target edge, storage state information of each computing node, and information of the edges which share the vertex with the target edge, so as to obtain a first number; counting the number of edges which are stored by each computing node and have the same vertex with the second vertex of the target edge to obtain a second number; and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the first quantity and the second quantity.

The target computing node counts the number of edges which are stored by each computing node and have the same vertex with the first vertex of the target edge to obtain a first number; counting the number of edges, which are stored in each compute node and have vertices common to the second vertex of the target edge, and obtaining the second number may include:

for example, the target edge is e (u, v), the target compute node may traverse the storage state information of each compute node and the information of the edge sharing the vertex with the target edge in the local cache queue of each compute node, find an adjacent node of the vertex v, and count one adjacent node as 1 score, to obtain the first number. And finding adjacent nodes of the vertex u, and recording one adjacent node as 1 score to obtain a second number.

The process that the target computing node determines the candidate computing node corresponding to the target edge from the computing nodes according to the first quantity and the second quantity comprises the following steps: determining the computing nodes corresponding to the maximum number according to the first number and the second number; and determining candidate computing nodes corresponding to the target edge from the computing nodes corresponding to the maximum number based on a preset strategy.

Optionally, the preset policy is: the computing node with the least load is selected. Namely, after each computing node is traversed, the computing node with the highest score is searched. And if the number of the computing nodes with the highest scores is larger than 1, selecting the computing node with the lowest load from the computing nodes with the highest scores as the candidate computing node corresponding to the target edge. And if the number of the selected computing nodes is still larger than 1, randomly selecting one computing node from the computing nodes with the highest score and the lowest load as a candidate computing node corresponding to the target edge e (u, v).

In another alternative implementation manner, the target computing node may determine vertex information of vertices included by the target edge according to vertices included by the target edge, storage state information of each computing node, and information of edges that share vertices with the target edge, where the vertex information of the vertices included by the target edge represents a distribution state of the vertices included by the target edge at each computing node. Then, the target computing node may determine, from the computing nodes, a candidate computing node corresponding to the target edge according to vertex information of a vertex included in the target edge.

If the vertex information corresponding to the vertex included by the target edge indicates that two vertices included by the target edge appear in the same computing node at the same time, the target computing node determines a candidate computing node corresponding to the target edge based on a preset strategy from the computing nodes where the two vertices included by the target edge appear at the same time.

If the vertex information corresponding to the vertex included by the target edge indicates that only one vertex appears in the computing node in the two vertices included by the target edge, the target computing node determines a candidate computing node corresponding to the target edge based on a preset strategy from the computing nodes where one vertex appears in the two vertices included by the target edge.

And if the vertex information corresponding to the vertex included by the target edge indicates that the two vertexes included by the target edge do not appear in each computing node, determining candidate computing nodes corresponding to the target edge from each computing node based on a preset strategy.

And if the vertex information corresponding to the vertex included by the target edge indicates that two vertexes included by the edge are respectively present in different computing nodes, counting the number of edges sharing the vertex with the target edge in each computing node, and selecting the computing node with the minimum load from the computing nodes with the maximum number as the candidate computing node corresponding to the target edge.

It should be noted that, although the process of determining the candidate computing node corresponding to the target edge is similar to the process in S1, S2, and S3 in the foregoing embodiment, the technical idea in the embodiment of the present application is that when it is temporarily not determined to which computing node the edge is divided, the edge is cached in the local cache queue, and as the number of edges stored by dividing each computing node increases, the number of vertices that each computing node can acquire increases, and on this basis, the global information according to when dividing the edge in the local cache queue increases, which is more beneficial for the dividing algorithm to make a better division decision.

In the embodiment of the application, the target computing node may determine, according to the storage state information of each computing node and the information of the edge sharing the vertex with the target edge, which is already divided and is not already divided, in each computing node, that is, global distribution information of two vertices included in the target edge is obtained, and candidate computing nodes corresponding to the target edge are determined from each computing node based on the global distribution information, so that the result of dividing the target edge is more reasonable.

In practical application, because a plurality of computing nodes included in the graph partitioning system are concurrently partitioned, edges read by more than two computing nodes at the same time may be edges sharing a vertex, and the plurality of computing nodes partition the edges sharing the vertex independently, which may cause each computing node to ignore information of the edges sharing the vertex with the edges read by itself, which are currently processed by other computing nodes, so that when the edges are partitioned, the obtained global information is reduced, and an unreasonable partitioning result may be caused. In order to avoid that multiple computing nodes divide edges sharing a vertex at the same time, as shown in fig. 10, an embodiment of the present application provides another graph dividing method, where the graph dividing method includes:

step 1001, a target computing node obtains a target edge set.

The target edge set is any one of a plurality of edge sets obtained by initially dividing the original graph data set, and the target edge set comprises a plurality of edges.

In step 1002, the target computing node sequentially reads each edge in the target edge set, and for each read edge, the target computing node detects whether any vertex of two vertices included in the edge is locked by other computing nodes except the target computing node.

In the embodiment of the application, when a plurality of computing nodes perform edge division in parallel, two vertices included in an edge currently being divided may be locked, and a locking state of the two vertices included in the edge currently being divided may be shared to each computing node included in the graph division system. In this way, when the target computing node divides the read edge, it may be determined whether two vertices included in the edge read by the target computing node are locked by other computing nodes by reading the vertices already locked by other computing nodes.

Optionally, in this embodiment of the present application, a shared list maintained by the graph partitioning system includes a locking table, and the locking table is used for temporarily storing, by each computing node, two vertices included in an edge currently being partitioned. It will be appreciated that the compute node writes the two vertices that the edge currently being partitioned includes to the locked table, i.e., indicating that the two vertices are in a locked state. After the computing node completes the edge division or stores the edge in the local cache queue, the computing node may remove the two vertices included in the edge from the locked table, i.e., unlock the two vertices included in the edge.

The target computing node may traverse the vertices written in the locked table to determine whether two vertices included in the currently read edge are on the locked table, and if so, it indicates that the two vertices included in the currently read edge of the target computing node are locked by other computing nodes except the target computing node. If not, the fact that two vertexes included in the edge currently read by the target computing node are not locked by other computing nodes is indicated.

Step 1003, if the edge is locked by other computing nodes, the target computing node continuously detects whether any vertex of the two vertices included by the edge is locked by other computing nodes except the target computing node until the two vertices included by the edge are not locked by other computing nodes.

For example, two vertexes u and v included in the edge e (u, v) have been written in the locking table, and if the edge currently read by the target computing node is e (u, h), since the vertex u is locked, the two vertexes included in the edge read by the target computing node are locked by other computing nodes, in this case, the target computing node cannot immediately divide the edge e (u, h), and it is necessary to continuously detect whether the two vertexes u and h included in the currently read edge e (u, h) are locked by other computing nodes until the two vertexes included in the edge e (u, h) are not locked by other computing nodes.

In the embodiment of the present application, if two vertices included in an edge currently read by a target computing node are locked by other computing nodes, it indicates that the target computing node cannot partition the edge e (u, h) temporarily, but needs to wait for the vertex u and the vertex h to be in an unlocked state before partitioning the edge e (u, h).

Step 1004, if the edge is not locked by other computing nodes, the target computing node marks that two vertexes included by the edge are in a locked state in a shared list maintained by the graph partitioning system, and determines candidate computing nodes corresponding to the edge from each computing node according to the vertexes included by the edge and the storage state information of each computing node.

For example, two vertices u and v included in an edge e (u, v) have been written in the locked table, and if the edge currently read by the target computing node is e (m, n), vertex m and vertex n are not in the locked table, so that neither vertex of the two vertices included in the edge currently read by the target computing node is locked by the other computing node, in this case, the target computing node may write vertex m and vertex n into the locked table to mark that the two vertices are in a locked state. Then, the target computing node may determine, from the vertices included by the edge and the storage state information of each computing node, a candidate computing node corresponding to the edge e (m, n) from each computing node.

By way of example, when vertex u changes from the locked state to the unlocked state, in this case, the target compute node may determine that two vertices u and h included in edge e (u, h) are not locked by other compute nodes, and then write vertex u and vertex h into the lock table for locking to mark that two vertices included in edge e (u, h) are in the locked state in the shared list maintained by the graph partitioning system. Then, the edge e (u, h) is divided.

In this embodiment, after the read edge is divided or stored in the local cache queue, the target computing node may remove the edge from the locking table, which means that two vertices included in the edge are no longer locked.

In the embodiment of the application, by establishing a locking mechanism, a target computing node locks two vertexes included in a currently read edge, at this time, if an edge read by other computing nodes includes any one of the two vertexes, the other computing nodes are in suspension waiting, when the target computing node completes the division of the currently read edge or stores the currently read edge into a local storage queue, the two vertexes included in the currently read edge are released, and when the other computing nodes find that the two vertexes are in an unlocked state, the previous unfinished process can be continuously processed. By the method, when the edges are divided, the information of the edges as much as possible can be acquired, so that the edges can be divided more accurately and reasonably.

The superiority of the graph partitioning method provided by the application is verified through comparison, the graph partitioning method is mainly evaluated from three indexes, namely graph partitioning time, graph partitioning performance and load balancing, and comparison and explanation are respectively performed from the three aspects.

As shown in Table 1, Table 1 illustrates a number of sources of raw graph data sets and the number of vertices and edges they contain in the raw graph data sets.

TABLE 1

Raw graph dataset source	Number of vertices	Number of edges
			DBLP	317,080	1,049,866
com-Orkut	3,072,441	117,185,083
			com-Livejournal	3,997,962	34,681,189
web-Google	875,713	5,105,039

Wherein DBLP is an integrated DataBase system (DBLP for short, in English, DataBase systems and Logic Programming), com-Orkut is a social service network launched by Google, com-live journal is a comprehensive friend-making website, and web-Google is a Google web network. Wherein the vertices represent users and the edges represent associations between users.

The graph partitioning method provided by the embodiment of the present application is evaluated from graph partitioning time.

The graph partitioning method and two existing flow graph partitioning methods provided by the embodiment of the present application are respectively adopted to partition four kinds of original graph data sets shown in table 1, where the existing flow graph partitioning methods are obivious and HDRF (english: High Degree replication First, chinese: High replication priority), and the obtained results are shown in fig. 11. A comparison diagram of graph partitioning time using the three methods is shown in fig. 11 (for simplicity, the graph partitioning method provided in the embodiment of the present application is represented by "present scheme" in fig. 11). As can be seen from fig. 11, the graph partitioning method provided by the embodiment of the present application can be greatly reduced in time.

The graph partitioning method provided by the embodiment of the present application is evaluated in terms of load balancing.

The load balancing index calculation formula can be expressed as follows:

wherein MAX (P)_i) Representing a computing node P_iThe number of stored edges, E represents the total number of edges, and P represents the number of computing nodes included in the graph partitioning system, as shown in table 2, which shows obivious, HDRF, and load balancing indexes of the graph partitioning method provided by the embodiment of the present application.

TABLE 2

	DBLP	web-google	com-livejournal	com-orkut
					Oblivious	≤1‰	≤1‰	≤1‰	≤1‰
HDRF	≤1‰	≤1‰	≤1‰	≤1‰
					This scheme	≤1‰	≤1‰	≤1‰	≤1％

The graph partitioning method provided by the embodiment of the application is evaluated from the aspect of graph partitioning performance.

Graph partitioning performance is particularly represented by the size of the copy factor, and smaller copy factors represent higher graph partitioning quality, and vice versa. Graph partitioning performance can be measured in two dimensions, from different partition numbers and different window sizes.

First, partition performance is compared to window size.

In the embodiment of the present application, each computing node may set a cache window based on its own memory capacity. SlowThe memory window stores data in the form of a buffer queue. The sizes of different cache windows have different influences on the copy factor, wherein the calculation expression of the copy factor is as follows:

fig. 12 shows the influence of the sizes of different cache windows of each compute node on the copy factor when the graph partitioning method provided by the embodiment of the present application and two existing flow graph partitioning methods are used to partition the original graph data set, i.e., the DBLP in table 1. Fig. 13 shows the influence of the sizes of different cache windows of each computing node on the copy factor when the graph partitioning method provided by the embodiment of the present application and two existing flow graph partitioning methods are used to partition the com-Livejournal original graph data set in table 1.

As shown in fig. 12 and 13, the copy factor shows a continuously decreasing trend as the window size increases. On the original graph data set DBLP, when the window size is equal to 15% of the total number of edges, compared with an Oblivious method, the method can reduce copy generation by 16.7%; compared with the HDRF method, the scheme can reduce the copy generation by 12.5 percent. Compared with the Oblivious method, the scheme can reduce the generation of the copies by about 20% on the com-Livejournal original graph data set; compared with the HDRF method, the scheme can reduce the copy generation by 15.7%. Reducing the generation of duplicates is intended to be equivalent to reducing the communication overhead of the graph computation phase. As can be seen from fig. 12 and fig. 13, compared with the two existing flow graph partitioning methods, the method and the apparatus can significantly reduce the copy factor, thereby reducing the communication overhead introduced by data synchronization in the graph computation stage, and accelerating the graph computation iteration speed.

Second, the partitioning performance is compared to the number of compute nodes included in the graph partitioning system.

Fig. 14 shows an effect of the number of computing nodes included in the graph partitioning system on the copy factor when the graph partitioning method provided by the embodiment of the present application and two existing graph partitioning methods are used to partition the original graph data set, i.e., the DBLP in table 1. Fig. 15 shows the influence of the number of computing nodes included in the graph partitioning system on the copy factor when the graph partitioning method provided by the embodiment of the present application and two existing flow graph partitioning methods are used to partition an original graph data set com-Livejournal in table 1.

According to fig. 14 and fig. 15, in a distributed scenario, adding a computing node included in a graph partitioning system increases the complexity of graph partitioning, and affects the performance of a graph partitioning algorithm. Therefore, the running results of all algorithms, namely the copy factors, are in an ascending trend. On an original graph data set DBLP, copy factor indexes are greatly increased by an Oblivious and HDRF flow graph dividing method along with the increase of the number of computing nodes, and compared with the Oblivious and HDRF methods, the copy factor indexes are greatly reduced by 19.8% and 15.2% respectively. Compared with the Oblivious and HDRF methods, the scheme still greatly reduces the copy factor index on the com-Livejournal original graph data set by 17% and 10.5% respectively.

In summary, in the scheme, on the two key indexes of the copy factor and the partition time for measuring the flow graph partition performance, compared with a competition scheme, the performance is greatly improved, and the time and communication overhead are reduced.

It should be understood that although the various steps in the flow charts of fig. 2-10 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-10 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.

In one embodiment, as shown in fig. 16, there is provided a graph partitioning apparatus 1600 applied in a target computing node of a graph partitioning system, where the graph partitioning system includes a plurality of computing nodes, and the target computing node is any one of the plurality of computing nodes, including: an obtaining module 1601 and a dividing module 1602, wherein:

an obtaining module 1601, configured to obtain a target edge set, where the target edge set is any one of a plurality of edge sets obtained by initially dividing an original graph data set, and the target edge set includes a plurality of edges;

a dividing module 1602, configured to sequentially read each edge in the target edge set, determine, for each read edge, a candidate computing node corresponding to the edge from each computing node according to a vertex included in the edge and storage state information of each computing node, so as to trigger a candidate computing node storage edge, and update the storage state information of the candidate computing node; wherein the storage status information is used to indicate the edges that the compute node has stored.

In one embodiment, the dividing module 1602 is specifically configured to:

determining vertex information corresponding to the vertexes included by the edges according to the vertexes included by the edges and the storage state information of each computing node, wherein the vertex information is used for indicating the distribution states of the vertexes included by the edges at each computing node;

and determining candidate computing nodes corresponding to the edges from the computing nodes according to the vertex information corresponding to the vertexes included by the edges.

In one embodiment, the dividing module 1602 is specifically configured to:

if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges are respectively present in different computing nodes, caching the edges into a local cache queue of a target computing node;

and determining candidate computing nodes corresponding to the edges stored in the local cache queue of the target computing node according to the edges stored in the local cache queue of the computing nodes.

In one embodiment, the dividing module 1602 is specifically configured to: reading a target edge from the local cache queue, wherein the target edge is any one of edges cached in the local cache queue;

acquiring information of edges which are in common vertex with the target edges from local cache queues of other computing nodes except the target computing node;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the vertexes included by the target edges, the storage state information of the computing nodes and the information of the edges sharing the vertexes with the target edges.

In one embodiment, the dividing module 1602 is specifically configured to: generating a vertex request according to two vertexes included by the target edge, and sending the vertex request to other computing nodes, wherein the vertex request is used for indicating the other computing nodes to send information of edges which are in the same vertex with the target edge in local cache queues of the other computing nodes to the target computing node;

and receiving the information of the edges which are in the local cache queues of the other computing nodes and are in the vertex sharing with the target edge, which is sent by the other computing nodes.

In one embodiment, the dividing module 1602 is specifically configured to: counting the number of edges which are stored in each computing node and share a vertex with a first vertex of the target edge according to the vertex included by the target edge, the storage state information of each computing node and the information of the edge which shares the vertex with the target edge to obtain a first number; counting the number of edges which are stored by each computing node and have the same vertex with the second vertex of the target edge to obtain a second number;

and determining candidate computing nodes corresponding to the target edges from the computing nodes according to the first quantity and the second quantity.

In one embodiment, the dividing module 1602 is specifically configured to: determining the computing nodes corresponding to the maximum number according to the first number and the second number;

and determining candidate computing nodes corresponding to the target edge from the computing nodes corresponding to the maximum number based on a preset strategy.

In one embodiment, the dividing module 1602 is specifically configured to: detecting whether any vertex in two vertexes included by the edge is locked by other computing nodes except the target computing node;

if the two vertexes are locked by other computing nodes, continuously detecting whether any one vertex in the two vertexes included by the edge is locked by other computing nodes except the target computing node or not until the two vertexes included by the edge are not locked by other computing nodes;

if the edge is not locked by other computing nodes, marking two vertexes included by the edge in a shared list maintained by the graph partitioning system to be in a locked state, and determining candidate computing nodes corresponding to the edge from the computing nodes according to the vertexes included by the edge and the storage state information of the computing nodes.

In one embodiment, the dividing module 1602 is specifically configured to: and if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges simultaneously appear in the same computing node, determining candidate computing nodes corresponding to the edges from the computing nodes of the two vertexes included by the simultaneously appearing edges based on a preset strategy.

In one embodiment, the dividing module 1602 is specifically configured to: and if the vertex information corresponding to the vertex included by the edge indicates that only one vertex appears in the computing node in the two vertexes included by the edge, determining a candidate computing node corresponding to the edge based on a preset strategy from the computing nodes of the vertex in the two vertexes included by the edge.

In one embodiment, the dividing module 1602 is specifically configured to: and if the vertex information corresponding to the vertexes included by the edges indicates that the two vertexes included by the edges do not appear in each computing node, determining candidate computing nodes corresponding to the edges from each computing node based on a preset strategy.

In one embodiment, the predetermined policy is to select the compute node with the least load.

For the specific definition of the graph dividing apparatus, reference may be made to the above definition of the graph dividing method, which is not described herein again. The various blocks of the partitioning apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, the internal structure of which may be as shown in fig. 17. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store edges. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a graph partitioning method.

Those skilled in the art will appreciate that the architecture shown in fig. 17 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.