graph processing memory access optimization method and system based on activity level layout
阅读说明：本技术 一种基于活跃度布局的图处理访存优化方法及系统 (graph processing memory access optimization method and system based on activity level layout ) 是由 施展 冯丹 王芳 朱涛 单玉祥 龚慧媛 黄世财 吴雨桐 何雨迪 于 2019-09-20 设计创作，主要内容包括：本发明公开了一种基于活跃度布局的图处理访存优化方法及系统,属于计算机数据处理技术领域,包括：获得待处理的目标图的数据后,按照中介中心性降序顺序对顶点排序,并根据活跃百分比将排序结果划分为活跃顶点集和不活跃顶点集；将活跃顶点集聚集组织并存储；将不活跃顶点集聚集组织并存储后,将其等分为多个大小为C的顶点块,并将对应的出边切分为多个边块,每个边块包括一个或多个编号连续的顶点的出边,且每个边块不小于C,同时维护状态表TActvB和TActeB,以筛除不活跃顶点集中的无用数据；TActvB用于记录各顶点块的活跃性,TActeB用于记录各边块的活跃性及下一边块的偏移量。本发明能够减少图应用执行过程中的访存等待时间,提高图处理效率。(The invention discloses graph processing access and memory optimization methods and systems based on activity layout, and belongs to the technical field of computer data processing, the method comprises the steps of obtaining data of a target graph to be processed, sequencing vertexes according to a medium centrality descending order, dividing a sequencing result into an active vertex set and an inactive vertex set according to an activity percentage, organizing and storing the active vertex set in an aggregation mode, organizing and storing the inactive vertex set in an aggregation mode, equally dividing the inactive vertex set into a plurality of vertex blocks with the size of C, dividing corresponding outgoing edges into a plurality of edge blocks, enabling each edge block to comprise or a plurality of outgoing edges of vertexes with continuous numbers, enabling each edge block to be not smaller than C, simultaneously maintaining a state table TACtvB and a TACteB to screen out useless data in the inactive vertex set, enabling the TACteB to record the activity of each vertex block, enabling the TACteB to record the activity of each edge block and the offset of a lower active edge block, and being capable of reducing access and memory processing waiting time in an application execution process of the graph.)
1, graph processing memory access optimization method based on activity layout, which is characterized by comprising:
after data of a target graph to be processed are obtained, sequencing vertexes in the target graph according to a medium centrality descending order, and dividing the sequenced vertexes into an active vertex set and an inactive vertex set according to a preset active percentage; the intermediary centrality of vertices in the active set of vertices is higher than the intermediary centrality of vertices in the inactive set of vertices;
organizing and storing the vertex aggregates in the active vertex set;
after the vertexes in the inactive vertex set are organized in a gathering mode and stored, equally dividing the inactive vertex set into a plurality of vertex blocks with the size of C, dividing outgoing edges corresponding to the inactive vertex set into a plurality of edge blocks, wherein each edge block comprises outgoing edges of vertexes with continuous numbers, the size of each edge block is not smaller than C, and simultaneously maintaining a state table TACtvB and a state table TACteB to screen out useless data in the inactive vertex set;
wherein C is the size of Cacheline.
2. The method for optimizing graph processing access and memory based on activity level layout according to claim 1, further comprising:
when the inactive vertex set is accessed, sequentially acquiring every vertex blocks, acquiring the activity of the vertex blocks according to the state table TACtvB, and not operating the inactive vertex blocks;
for each active vertex blocks verteesBlock, traversing the vertexes therein, judging the activity of the traversed vertexes, if the traversed vertexes are active vertexes, obtaining the update value of the vertexes from the update table to calculate the new value of the vertexes, and updating the vertex table by using the calculated new value;
when accessing the outgoing edge corresponding to the inactive vertex, sequentially acquiring every edge blocks, acquiring the activity of the edge blocks according to a state table TACteB, and not operating the inactive edge blocks;
for each active edge blocks edgesBlock, traversing the edges in the edge blocks, judging whether a new update value needs to be calculated for a destination node of the traversed edge according to the data of the source node of the traversed edge, if so, calculating the update value of the destination node by using the data of the source node, and then updating the update table;
the vertex table is used for recording data of each vertex in the target graph, and the update table is used for recording an update value of each vertex in the target graph.
3. The method for graph processing access optimization based on liveness layout according to claim 2, further comprising:
if there are two consecutive edge blocks edgesBlock that are both inactive1And edgesBlock2Then the two edge blocks are merged into new edge blocks edgesBlock3And updates the state table TActeB.
4. The method for optimizing map processing access and memory based on activity layout as claimed in claim 2, further comprising performing AND calculation after obtaining the activity of each vertex block for all vertex blocks stored in partitions to judge the activity of the whole partition, and when accessing the inactive vertex set, not performing operation for all vertex blocks stored in an inactive partition.
5. The graph processing access and memory optimization method based on activity level layout as claimed in claim 1 or 2, wherein the vertices in the inactive vertex set are organized in an aggregation manner, specifically by a BFS layout method with a limited step size.
6. The method for optimizing map processing access and memory based on activeness layout as claimed in claim 1 or 2, wherein the activity percentage is：PA＝10％。
A system of , comprising a processor and a computer-readable storage medium, wherein the computer-readable storage medium is configured to store an executable program;
the processor is used for reading an executable program in the computer readable storage medium and executing the graph processing access and memory optimization method based on the activity level layout as claimed in any of claims 1-6.
Technical Field
The invention belongs to the technical field of computer data processing, and particularly relates to graph processing access optimization methods and systems based on liveness layout.
Background
The graph can clearly describe the relationship among various things as important data structures in computer science, so that a method for analyzing complex data by using a graph model is more and more popular, the graph is applied to a plurality of fields of social life by and helps solve complex problems generated by the fields, such as optimal routes in the traffic field, protein molecular structure decomposition in the biomedical field, community discovery in the social network field, and knowledge mapping, data mining, semantic networking, link prediction and the like of current fire and heat, along with the arrival of a big data age, the scale for generating graph data along with the fields is rapidly increasing, and the number of active users reaches 23.2 all the year round, and occupies the population number of in the world, for example, according to the declaration of social network Facebook in 2018.
With the rapid growth of graph data size, a plurality of dedicated efficient graph processing systems are available, wherein the graph processing system with external memory mode can use cheap and low-speed external memory to store data, and can process graph data with hundreds of millions of scales in a single-machine environment by increasing the access speed through data layout or system optimization work.
The map data locality is key factors influencing CPU access data, and brings a non-negligible influence on the map processing efficiency, so that the map data organization and the processor access behavior have a close relationship, in recent years, the access optimization method aiming at the map processing mainly utilizes the locality of active data to improve the access speed, and an external memory system GraphChi sorts edge sets in partitions according to source vertexes, namely representative examples.
Disclosure of Invention
Aiming at the defects and improvement requirements of the prior art, the invention provides graph processing memory access optimization methods and systems based on activity layout, and aims to reduce the memory access waiting time in the graph application execution process so as to improve the graph processing efficiency.
To achieve the above object, according to of the present invention, there is provided a graph processing access and memory optimization method based on activity level layout, including:
after data of a target graph to be processed are obtained, the vertexes in the target graph are sorted according to the descending order of the centrality of the medium, and the sorted vertexes are divided into an active vertex set and an inactive vertex set according to a preset active percentage; the intermediary centrality of the vertices in the active vertex set is higher than the intermediary centrality of the vertices in the inactive vertex set;
the vertexes in the active vertex set are gathered, organized and stored;
after the vertexes in the inactive vertex set are organized in a gathering mode and stored, equally dividing the inactive vertex set into a plurality of vertex blocks with the size of C, dividing outgoing edges corresponding to the inactive vertex set into a plurality of edge blocks, wherein each edge block comprises outgoing edges of vertexes with continuous numbers, the size of each edge block is not smaller than C, and simultaneously maintaining a state table TACtvB and a state table TACteB to screen out useless data in the inactive vertex set;
wherein C is the size of Cacheline.
The mesocentrality is a ratio of the number of shortest paths through a certain vertex to the number of shortest paths in the graph, the higher the mesocentrality, the more the shortest paths through the vertex, so that a vertex connection with high mesocentrality is required among a plurality of communities in graphs, and is irrelevant to connectivity, and it is measured that vertices have the capability of being intermediaries in two or more clustered groups, and generally need to combine global information, which is important in network analysis, especially in two large groups with small connections.
The invention measures the activity of the vertex by using the centrality of the medium, and can more accurately complete the division of the active vertex and the inactive vertex on the whole; the active vertex set obtained by division is subjected to aggregation organization and storage, so that the locality of active data can be increased, and the memory access speed is increased; the grouped inactive vertex sets obtained by division are also subjected to aggregation organization and storage, useless data in the inactive vertex sets are screened out through the blocks and the state table, a large amount of data which does not need to be processed can be directly skipped, and the transmission quantity of inactive data can be reduced. Therefore, the invention can reduce the access waiting time in the graph application execution process and improve the graph processing efficiency.
Further , the method for optimizing map processing access based on activity level layout according to the aspect of the present invention further includes:
when an inactive vertex set is accessed, sequentially acquiring every vertex blocks, acquiring the activity of the vertex blocks according to a state table TACtvB, and not operating the inactive vertex blocks;
for each active vertex blocks, traversing vertexes therein, judging the activity of the traversed vertex, if the traversed vertex is an active vertex, obtaining an update value of the vertex from an update table to calculate a new value of the vertex, and updating the vertex table by using the calculated new value;
when an outgoing edge corresponding to an inactive vertex is accessed, sequentially acquiring every edge blocks, acquiring the activity of the edge blocks according to a state table TACteB, and not operating the inactive edge blocks;
for each active edge blocks edgesBlock, traversing the edges, judging whether a new update value needs to be calculated for the destination node of the traversed edges according to the data of the source node of the traversed edges, if so, calculating the update value of the destination node by using the data of the source node, and then updating an update table;
the vertex table is used for recording data of each vertex in the target graph, and the updating table is used for recording an updating value of each vertex in the target graph.
Through the operation, the activity of the data block (the vertex block or the edge block) can be detected in a coarse granularity mode, and then the active data block is detected in a fine granularity mode, so that the blocked inactive data can be effectively filtered, a large amount of inactive data are prevented from being accessed and processed, the transmission quantity of the inactive data is reduced, the access and storage waiting time in the graph application execution process is effectively reduced, and the graph processing efficiency is improved.
Further , the method for optimizing map processing access based on activity level layout according to the aspect of the present invention further includes:
if there are two consecutive edge blocks edgesBlock that are both inactive1And edgesBlock2Then the two edge blocks are merged into new edge blocks edgesBlock3And updates the state table TActeB.
The invention is beneficial to effectively filtering out inactive edge data and improving the graph processing efficiency by combining continuous inactive edge blocks.
, the method for optimizing graph processing access and memory based on activity level layout according to the aspect of the present invention further includes, for all vertex blocks stored in partitions, performing and calculation after obtaining the activity of each vertex block to determine the activity of the entire partition, and when accessing an inactive vertex set, not performing an operation on all vertex blocks stored in an inactive partition.
And , aggregating the vertexes in the inactive vertex set, and specifically completing the aggregation through a BFS ( degree first search) layout method with a limited step size.
According to the invention, vertices in a set of inactive vertices are gathered by a step-limited BFS layout method, a re-layout can be carried out according to a direct adjacency relation, vertices with short distribution distances are gathered to , the outgoing edges of high-degree nodes are distributed in a set manner, and therefore, when the nodes do not need to be processed, corresponding edge blocks can be screened in a whole block, and therefore, the screening efficiency of useless data can be improved.
step, the active percentage is
According to a second aspect of the present invention, there is provided systems comprising a processor and a computer readable storage medium for storing an executable program;
the processor is used for reading an executable program in a computer-readable storage medium and executing the graph processing access optimization method based on the liveness layout according to the aspect of the present invention.
Generally, by the above technical solution conceived by the present invention, the following beneficial effects can be obtained:
(1) the graph processing access optimization method and system based on the activity degree layout, provided by the invention, have the advantages that the activity degree of the vertex is measured by using the centrality of the medium, and the division of the active vertex and the inactive vertex can be integrally and accurately finished; the active vertex set obtained by division is subjected to aggregation organization and storage, so that the locality of active data can be increased, and the memory access speed is increased; the grouped inactive vertex sets obtained by division are also subjected to aggregation organization and storage, useless data in the inactive vertex sets are screened out through the blocks and the state table, a large amount of data which does not need to be processed can be directly skipped, and the transmission quantity of inactive data can be reduced. Therefore, the invention can reduce the access waiting time in the graph application execution process and improve the graph processing efficiency.
(2) According to the graph processing memory access optimization method and system based on the activity degree layout, the maintained state table is used, firstly, the activity of the data blocks (the vertex blocks or the edge blocks) is detected in a coarse-grained mode, then, the active data blocks are detected in a fine-grained mode, the blocking inactive data can be effectively filtered, a large amount of inactive data are prevented from being accessed and processed, the transmission quantity of the inactive data is reduced, the memory access waiting time in the graph application execution process is effectively reduced, and the graph processing efficiency is improved.
(3) According to the graph processing access optimization method and system based on activity layout, provided by the invention, vertices in an inactive vertex set are gathered and organized by a step-length-limited BFS layout method, re-layout can be carried out according to a direct adjacency relation, and vertices with short distribution distances are gathered to , so that edges of nodes with higher degrees are intensively distributed, and therefore, when the nodes do not need to be processed, corresponding edge blocks can be screened in a whole block, and therefore, the screening efficiency of useless data can be improved.
In general, the present invention divides vertices into active vertex sets and inactive vertex sets based on the liveness characteristics of the centrally delineated vertices by an intermediary. And (4) performing aggregation arrangement on the active vertex sets in the graph data to improve the access speed of the active data. Meanwhile, according to the arrangement of degree centrality and direct adjacency relation leading inactive data, the data which are not frequently accessed are gathered and stored, the locality of the inactive data is improved, then the transmission quantity of the inactive data between each layer of storage equipment is effectively reduced by combining the proposed data screening strategy, the use efficiency of cache is improved, and therefore the graph processing efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of a graph processing access optimization method based on liveness layout according to an embodiment of the present invention;
FIG. 2 is an aggregate distribution plot of a set of active vertices provided by an embodiment of the present invention;
FIG. 3 is a diagram illustrating original graph data and partition access thereof according to an embodiment of the present invention; wherein, (a) is a diagram of original graph data, and (b) is a diagram of corresponding partition access condition;
FIG. 4 is a diagram illustrating graph data after aggregate storage and partition access thereof according to an embodiment of the present invention; wherein, (a) is a diagram of the image data after the rearrangement, and (b) is a diagram of the corresponding partition access condition;
fig. 5 is a schematic diagram of a processing procedure of an edge block according to an embodiment of the present invention.
Detailed Description
For purposes of making the objects, aspects and advantages of the present invention more apparent, the present invention will be described in detail below with reference to the accompanying drawings and examples, it is to be understood that the specific examples described herein are for the purpose of illustrating the invention only and are not to be construed as limiting the invention.
In order to reduce the access waiting time in the graph application execution process and improve the graph processing efficiency, the invention provides graph processing access optimization methods and systems based on activity layout, and the overall thought is that the method comprises the steps of rearranging according to the activity characteristics of graph data, simultaneously considering the distribution of active data and inactive data, improving the data layout condition of graph partitions, and further optimizing the access behavior of graph processing, specifically, preprocessing the data, dividing into an active vertex set and an inactive vertex set according to the activity of the vertex, and performing vertex rearrangement work to enable the data to be respectively gathered, wherein the locality of the active data is improved in aspect, the access speed of the active data is improved, the locality of the inactive data is improved in aspect, and the data which does not participate in processing is skipped by combining a data screening strategy in the graph processing process, so that the redundant data transmission quantity is reduced.
The graph processing access optimization method based on activity level layout, as shown in fig. 1, includes:
after data of a target graph to be processed are obtained, the vertexes in the target graph are sorted according to the descending order of the centrality of the medium, and the sorted vertexes are divided into an active vertex set and an inactive vertex set according to a preset active percentage; the intermediary centrality of the vertices in the active vertex set is higher than the intermediary centrality of the vertices in the inactive vertex set;
the vertexes in the active vertex set are gathered, organized and stored;
in optional embodiments, the vertices in the inactive vertex set are organized in an aggregation manner, which can be specifically completed by a BFS ( degree first search) layout method with a limited step size;
equally dividing the inactive vertex set into a plurality of vertex blocks with the size of C, wherein C is the size of Cacheline; assuming that the number of vertices in the inactive set of vertices is V, the number of vertex blocks is
The outgoing edge is divided according to the outgoing degree of the vertex, the space occupied by each vertex is assumed to be U, the outgoing degree of the vertex is assumed to be D, if U multiplied by D is larger than or equal to C, the outgoing edge neighbor vertex set of the vertex is divided into sets, if the inequality is not satisfied, the neighbor vertex sets with continuous vertex numbers are merged, so that the vertex of the merged set is not smaller than C, each Cacheline can be fully placed, and other unrelated data are avoided;
the method comprises the steps of dividing an inactive vertex set and an outgoing edge, and simultaneously maintaining a state table TACtvB and a state table TACteB to assist in screening out useless data in the inactive vertex set, wherein the state table TACtvB is used for recording the activity of each vertex block, the state table TACteB is used for recording the activity of each edge block, and the state table TACtvB is also used for recording the offset of edge blocks because the size of the edge blocks is not fixed and corresponding indication information is needed when the edge blocks which do not need to be processed need to be skipped;
wherein C is the size of Cacheline.
As preferred embodiments, in the graph processing access and memory optimization method based on the activity degree layout, the activity percentage is PA＝10％；
In practical application, less than about 10% of the vertexes in the graph can participate in most of the iteration process of the graph processing, the rest 90% of the vertexes do not participate in calculation after a small number of iterations, and the method sets the active percentage PAWhen the vertex with the highest center-to-center relation among all the vertices is 10%, the vertex with the highest center-to-center relation among all the vertices is used as an active vertex, and other vertices are used as inactive vertices, so that the active vertex and the inactive vertices can be divided more accurately as a whole;
the division of the active vertex set and the inactive vertex set is shown in fig. 2, wherein the sequence Arr1 is an original vertex sequence in the graph data, and active vertices and inactive vertices in the graph data are distributed in a crossed manner; after the vertices are sorted according to the medium centrality, the vertex sequence is shown as a
The method comprises the following steps of performing data layout on logical relations among the inactive vertices, increasing the locality of inactive data and fully playing the performance of a screening strategy, wherein in the map processing access optimization method based on the liveness layout, a limited-step BFS layout method can be particularly adopted for aggregating and organizing vertices in an inactive vertex set, a limited-step BFS ( degree first search) layout method can be used for aggregating vertices with a close distribution distance and acquiring adjacency relations by adopting a single-step BFS, the BFS starts to visit from a certain vertex v in a map, visits neighboring vertices which are not visited first in the vertex v are accessed, a set of neighboring vertices are acquired, and then visit every 34 vertices in 39v in sequence, the set of neighboring vertices in the map is accessed first, the set of the neighboring vertices is acquired, the set of the neighboring vertices is accessed first, the set of the neighboring vertices is processed according to a specific rule that the neighboring vertices are accessed, and the neighboring vertices are processed before the set of the neighboring vertices are accessed, so that when the set of the neighboring vertices are accessed, the set of the neighboring vertices are processed by adopting a high-step BFS 4656, the method can be performed on the basis that the following specific example:
assuming that graphs contain 12 vertices in total and conform to a power law distribution;
original graph data is as shown in fig. 3(a),
After the graph data is rearranged by adopting a BFS layout method with a limited step length, the graph data is as shown in FIG. 4(a), wherein the height vertex becomes 7, 8; then, partition is performed, and the partition result is as shown in fig. 4(b), taking
The graph processing access optimization method based on activity level layout may further include:
when an inactive vertex set is accessed, sequentially acquiring every vertex blocks, acquiring the activity of the vertex blocks according to a state table TACtvB, and not operating the inactive vertex blocks;
for each active vertex blocks, traversing vertexes therein, judging the activity of the traversed vertex, if the traversed vertex is an active vertex, obtaining an update value of the vertex from an update table to calculate a new value of the vertex, and updating the vertex table by using the calculated new value;
when an outgoing edge corresponding to an inactive vertex is accessed, sequentially acquiring every edge blocks, acquiring the activity of the edge blocks according to a state table TACteB, and not operating the inactive edge blocks;
for each active edge blocks edgesBlock, traversing the edges, judging whether a new update value needs to be calculated for the destination node of the traversed edges according to the data of the source node of the traversed edges, if so, calculating the update value of the destination node by using the data of the source node, and then updating an update table;
the vertex table is used for recording data of each vertex in the target graph, and the updating table is used for recording an updating value of each vertex in the target graph;
through the operation, the activity of the data block (a vertex block or an edge block) can be detected in a coarse granularity mode, and then the active data block is detected in a fine granularity mode, so that blocked inactive data can be effectively filtered, a large amount of inactive data are prevented from being accessed and processed, the transmission quantity of the inactive data is reduced, the access and storage waiting time in the graph application execution process is effectively reduced, and the graph processing efficiency is improved;
in order to further , to improve the efficiency of filtering the useless data, in the above method for optimizing map processing access based on activity level layout, the method may further include:
if there are two consecutive edge blocks edgesBlock that are both inactive1And edgesBlock2Then the two edge blocks are merged into new edge blocks edgesBlock3And updates the state table TActeB, as shown in fig. 5,
by combining continuous inactive edge blocks, the method is beneficial to more effectively filtering inactive edge data and improving the graph processing efficiency.
As optional implementation manners, in the graph processing access and memory optimization method based on the activity layout, the method further comprises the steps of performing AND calculation after obtaining the activity of each vertex block for all vertex blocks stored in partitions so as to judge the activity of the whole partition;
the state of the entire partition can be acquired by the and calculation, thereby determining whether the entire partition needs to be processed.
The invention also provides systems comprising a processor and a computer readable storage medium for storing an executable program;
the processor is used for reading an executable program in a computer readable storage medium and executing the graph processing access optimization method based on the activity layout.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
- 上一篇：一种医用注射器针头装配设备
- 下一篇：图模型数据的处理方法、装置和终端设备