阅读说明：本技术 集群数据处理方法、装置、计算机集群及可读存储介质 (Cluster data processing method and device, computer cluster and readable storage medium ) 是由 陈武 于 2019-09-16 设计创作，主要内容包括：本申请实施例提供一种集群数据处理方法、装置、计算机集群及可读存储介质,通过获取每个集群作业在各单位时间内的作业统计数据,以确定每个集群作业在各单位时间内的计算节点时长,并根据每个计算节点在单位时间内的占用成本信息以及每个集群作业在各单位时间内的计算节点时长,能够有效确定每个集群作业在参与每次计算过程中所产生的作业成本信息,从而能够将集群计算资源的成本分摊到每个工作单元,进而为企业的成本核算提供数据支持,相较于现有技术中统计占用资源比例的方式更加直观,同时在对企业外部用户开放计算资源时,也便于制定计费标准。(The embodiment of the application provides a cluster data processing method, a device, a computer cluster and a readable storage medium, which can effectively determine the operation cost information generated by each cluster operation in each calculation process according to the occupation cost information of each computing node in unit time and the calculation node time length of each cluster operation in unit time by acquiring operation statistical data of each cluster operation in each unit time, thereby being capable of distributing the cost of cluster calculation resources to each working unit and providing data support for cost calculation of enterprises.)

1. A cluster data processing method is applied to a computer cluster, and comprises the following steps:

acquiring operation statistical data of each cluster operation in each unit time, wherein the operation statistical data comprises resource duration corresponding to each computing resource occupied by the cluster operation and each level of corresponding working unit;

determining the calculation node time length of each cluster job in each unit time according to the job statistical data of each cluster job in the unit time;

determining the operation cost information of each cluster operation in different statistical time periods according to the occupation cost information of each computing node in unit time and the computing node duration of each cluster operation in each unit time;

and calculating the operation cost information of each level of working unit in each statistical time period according to the operation cost information of each cluster operation in the statistical time period and each level of working unit corresponding to each cluster operation.

2. The method according to claim 1, wherein the step of obtaining job statistics per cluster job per unit time comprises:

for each cluster job, after the cluster job starts to be calculated at the first moment, respectively acquiring the resource duration corresponding to each cluster resource occupied by the cluster job in each unit time period;

and after the cluster job is calculated at the second moment, acquiring the total resource duration of the cluster job between the first moment and the second moment.

3. The method according to claim 2, wherein the step of obtaining the resource duration corresponding to each type of cluster resource occupied by the cluster job in each unit time period comprises:

acquiring the job submission time of the cluster job and the task time and occupied resources of each cluster task in the cluster job;

acquiring resource duration corresponding to each cluster resource occupied by each cluster task in each unit time period according to the operation information of the cluster operation;

and taking the sum of the resource duration corresponding to each kind of cluster resource occupied by each cluster task in each unit time period as the resource duration corresponding to each kind of cluster resource occupied by the cluster job in each unit time period.

4. The cluster data processing method according to claim 3, wherein the task time includes a task start time and a task end time, and the step of obtaining, according to the job information of the cluster job, a resource duration corresponding to each type of cluster resource occupied by each cluster task in the cluster job in each unit time period comprises:

aiming at each cluster task, acquiring the occupation duration of the cluster task in each unit time period according to the task time of the cluster task and the job submission time of the cluster job;

aiming at each unit time period, judging whether the occupied time length of the cluster task in the unit time period is greater than 0;

if the occupation time of the cluster task in the unit time period is greater than 0, calculating the resource time corresponding to each type of cluster resource occupied by the cluster task in the unit time period according to the occupied resources of the cluster task and the occupation time in the unit time period;

and if the occupying time of the cluster task in the unit time period is equal to 0, determining that the resource time corresponding to each cluster resource occupied by the cluster task in the unit time period is 0.

5. The method according to claim 4, wherein the duration of occupancy includes a first duration of occupancy and a second duration of occupancy, the first duration of occupancy being a duration of occupancy of the clustered task within each unit time period between the first time and a task start time of the clustered task, the second duration of occupancy being a duration of occupancy within each unit time period between the task start time of the clustered task and the second time, wherein the first duration of occupancy is 0.

6. The method according to claim 4, wherein the occupied resources include resources occupied by a CPU and resources occupied by a memory, and the step of calculating the resource duration corresponding to each type of cluster resource occupied by the cluster task in the unit time slot according to the occupied resources of the cluster task and the occupied duration in the unit time slot includes:

calculating the product of the CPU occupied resource and the occupied time length as the CPU resource time length, and calculating the product of the memory occupied resource and the occupied time length as the memory resource time length;

the resource duration comprises the CPU resource duration and the memory resource duration.

7. The cluster data processing method according to claim 1, wherein the step of determining a computing node duration of each cluster job in each unit time according to the job statistics of each cluster job in unit time comprises:

for each kind of computing resource, computing the computing node time length corresponding to each cluster job occupying the computing resource in each unit time, wherein the computing node time length is the ratio of the resource time length corresponding to the computing resource occupied by the cluster job to the configuration quantity of the computing resource in each computing node;

and respectively selecting the maximum calculation node time length from the calculation node time lengths of each cluster job in each unit time as the calculation node time length of the cluster job in each unit time.

8. The cluster data processing method according to any one of claims 1 to 7, wherein the step of determining the job cost information of each cluster job in different statistical time periods according to the occupation cost information of each computing node in unit time and the computing node time length of each cluster job in each unit time comprises:

aiming at each cluster job, obtaining the time length of a computing node occupied by the cluster job in each statistical time period according to the time length of the computing node of the cluster job in each unit time;

and calculating the product of the time length of the calculation node occupied by the cluster job in the statistical time period and the occupation cost information of each calculation node in unit time as the job cost information of the cluster job in the statistical time period.

9. The method of cluster data processing according to claim 1, wherein said method further comprises:

acquiring cost information of the computing node, wherein the cost information comprises purchase cost, warranty duration and operation cost in each preset time period;

and acquiring the occupation cost information of the computing node in unit time according to the cost information of the computing node.

10. A cluster data processing apparatus, for use in a cluster of computers, the apparatus comprising:

the acquisition module is used for acquiring job statistical data of each cluster job in each unit time, wherein the job statistical data comprises resource duration corresponding to each computing resource occupied by the cluster job and each level of corresponding working unit;

the first determining module is used for determining the computing node time length of each cluster job in each unit time according to the job statistical data of each cluster job in the unit time;

the second determining module is used for determining the operation cost information of each cluster operation in different statistical time periods according to the occupation cost information of each computing node in unit time and the computing node duration of each cluster operation in each unit time;

and the calculation module is used for calculating the operation cost information of each level of working unit in each statistical time period according to the operation cost information of each cluster operation in the statistical time period and each level of working unit corresponding to each cluster operation.

11. A computer cluster comprising a plurality of compute nodes, a machine-readable storage medium having stored thereon machine-executable instructions, and a processor, which when executed causes the computer cluster to implement the cluster data processing method of any one of claims 1-9.

12. A readable storage medium having stored therein machine executable instructions which when executed perform the method of cluster data processing of any one of claims 1 to 9.

Technical Field

The present application relates to the field of cluster cost accounting, and in particular, to a cluster data processing method, an apparatus, a computer cluster, and a readable storage medium.

Background

Currently, a computer cluster may use a resource management framework such as Yarn to manage, allocate, and schedule computing resources of multiple computing nodes. For example, when a cluster job including one or more cluster tasks capable of running in parallel is submitted to the resource management framework, the resource management framework allocates corresponding cluster resources to each cluster task uniformly according to a corresponding resource management policy, and schedules the cluster jobs to run in a computer cluster.

In the cluster job calculation process, the resource management framework calculates the overall resource allocation condition and the resource condition to be allocated in the current operation process. Cost accounting is often essential for an enterprise. How to distribute the cost of cluster computing resources to each unit of work is a big problem. In the traditional scheme, only the occupied resource proportion and the resource occupation development trend of the clusters are counted generally, and the operation cost generated in each calculation process of each cluster operation is not considered, so that the cost accounting of an enterprise is lack of data support, and meanwhile, when the calculation resources are opened for users outside the enterprise, the charging standard is difficult to be made objectively.

Disclosure of Invention

In view of the above, an object of the present application is to provide a cluster data processing method, an apparatus, a computer cluster and a readable storage medium, which can determine job cost information generated by each cluster job in each calculation process, so as to allocate the cost of cluster calculation resources to each work unit, and further provide data support for cost accounting of an enterprise.

According to an aspect of the present application, there is provided a cluster data processing method applied to a computer cluster, the method including:

acquiring job information of each cluster job in the computer cluster, wherein the job information comprises job submission time of the cluster job and task time and occupied resources of each cluster task in the cluster job;

according to the operation information of each cluster operation, acquiring the resource occupation duration and the resource waiting distribution of each cluster task in each unit time period in each cluster operation;

and acquiring occupied resource information and resource waiting distribution information in each unit time period according to the resource occupation time and the resource waiting distribution time of each cluster task in each cluster operation in each unit time period and the total resource occupation time of each computing node of the computer cluster in each unit time period.

According to another aspect of the present application, there is provided a cluster data processing apparatus applied to a computer cluster, the apparatus including:

the acquisition module is used for acquiring the job information of each cluster job in the computer cluster, wherein the job information comprises the job submission time of the cluster job and the task time and occupied resources of each cluster task in the cluster job;

the first calculation module is used for obtaining the resource occupation duration and the resource waiting distribution of each cluster task in each unit time period in each cluster job according to the job information of each cluster job;

and the second computing module is used for obtaining the occupied resource information and the resource waiting distribution information in each unit time period according to the resource occupied time and the resource waiting distribution time of each cluster task in each cluster operation in each unit time period and the total resource occupied time of each computing node of the computer cluster in each unit time period.

According to another aspect of the present application, there is provided a computer cluster comprising a machine-readable storage medium storing machine-executable instructions and a processor, which when executed causes the computer cluster to implement the aforementioned cluster data processing method.

According to another aspect of the present application, there is provided a readable storage medium having stored therein machine executable instructions which, when executed, implement the aforementioned cluster data processing method.

Based on any aspect, the method and the device for calculating the cluster job cost effectively determine the job cost information generated in each calculation process of each cluster job by acquiring the job statistical data of each cluster job in each unit time to determine the calculation node time length of each cluster job in each unit time and according to the occupation cost information of each calculation node in each unit time and the calculation node time length of each cluster job in each unit time, so that the cost of the cluster calculation resources can be distributed to each working unit, data support is provided for cost calculation of enterprises, compared with the prior art, the method for calculating the occupation resource proportion is more intuitive, and meanwhile, when the calculation resources are opened for external users of the enterprises, the charging standard can be conveniently established.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

FIG. 1 is a block diagram illustrating a schematic structure of a computer cluster provided by an embodiment of the present application;

fig. 2 shows one of the flow diagrams of the cluster data processing method provided in the embodiment of the present application;

FIG. 3 shows a flow diagram of the sub-steps of step S110 shown in FIG. 2;

FIG. 4 is a diagram illustrating resource durations provided by an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating an occupancy duration provided by an embodiment of the present application;

fig. 6 shows a second flowchart of a cluster data processing method provided in the embodiment of the present application;

fig. 7 shows a functional module schematic diagram of a cluster data processing apparatus provided in an embodiment of the present application.

Detailed Description

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for illustrative and descriptive purposes only and are not used to limit the scope of protection of the present application. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this application illustrate operations implemented according to some of the embodiments of the present application. It should be understood that the operations of the flow diagrams may be performed out of order, and steps without logical context may be performed in reverse order or simultaneously. One skilled in the art, under the guidance of this application, may add one or more other operations to, or remove one or more operations from, the flowchart.

Fig. 1 shows a schematic block diagram of a structure of a computer cluster 100 provided in an embodiment of the present application, where the computer cluster 100 includes a plurality of computing nodes 110, and these computing nodes 110 may be servers, computer devices, and other machines with computing Processing capability, and each computing node 110 may provide a cluster resource for computing a cluster task, for example, may provide a Central Processing Unit (CPU) computing resource and a Memory (Memory) computing resource.

The computer cluster 100 may manage, allocate, and schedule the computing resources of the plurality of computing nodes 110 it includes using a resource management framework. Taking the Resource management framework as yann as an example, the Resource system framework of yann includes a Resource Manager (RM) and an Application Manager (AM), the Resource Manager may be responsible for Resource scheduling management of the cluster task, and the Application Manager may be responsible for forming a one-to-one mode with the Application task of the user to apply for the Resource and monitor the operating condition of the cluster task.

The computer cluster 100 may include a machine-readable storage medium 120 and a processor 130.

The machine-readable storage medium 120 is a computer-readable storage medium, and can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the cluster data processing method in the embodiment of the present application (for example, the cluster data processing apparatus 200 shown in fig. 7). The processor 130 executes various functional applications and data processing of the computer cluster 100 by executing software programs, instructions and modules stored in the machine-readable storage medium 120, that is, implements a cluster data processing method described below.

The machine-readable storage medium 120 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the machine-readable storage medium 120 may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a Read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double data rate Synchronous Dynamic random access memory (DDR SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchronous link SDRAM (SLDRAM), and direct memory bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory. In some examples, the machine-readable storage medium 120 may further include memory located remotely from the processor 130, which may be connected to the terminal device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor 130 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 130. The processor 130 may be a general-purpose processor, a digital signal processor (digital signal processor dsp), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor.

In order to solve the foregoing problems in the background art, the cluster data processing method provided by the embodiment of the present application shown in fig. 2 is described in detail below with reference to fig. 1, and the cluster data processing method may be executed by the computer cluster 100 shown in fig. 1. It should be understood that, in other embodiments, the order of some steps in the cluster data processing method of this embodiment may be interchanged according to actual needs, or some steps may be omitted or deleted. The detailed steps of the cluster data processing method are described as follows.

Step S110, acquiring job statistics data of each cluster job in each unit time.

In this embodiment, the job statistical data may include resource durations corresponding to each type of computing resource occupied by the cluster job and corresponding working units of each level.

In detail, each compute node 110 may be considered a computing resource of the computer cluster 100. For example, the computing resources may be, but are not limited to, the number of CPU cores and the memory size of each compute node 110. Assume that the computer cluster 100 includes 1000 compute nodes 110, each compute node 110 having 28 CPU cores and 122880MB of memory. If a cluster job needs to be submitted to the computer cluster 100 for operation, the cluster job includes 3 cluster tasks, and each cluster task needs 2 CPU cores and 4000MB of memory, and the operation time consumption of the 3 cluster tasks in the calculation process is 10s, 12s, and 15s, respectively, then the CPU resource duration of the CPU resource occupied by the cluster job is 2 × 10+2 × 12+2 × 15 ═ 74, and then the memory resource duration of the memory resource occupied by the cluster job is 4000 × 10+4000 × 12+4000 × 15 ═ 148000. The CPU resource duration and the memory resource duration may reflect a situation that the cluster job occupies the computing resources of the computer cluster 100.

In this embodiment, the unit time period may be set according to actual requirements. For example, if each unit minute is set as one unit time period, the present embodiment may obtain job statistics of each cluster job in each minute.

Step S120, determining the time length of the calculation node of each cluster job in each unit time according to the job statistical data of each cluster job in the unit time.

Step S130, determining job cost information of each cluster job in different statistical time periods according to the occupation cost information of each computing node 110 in unit time and the computing node duration of each cluster job in each unit time.

In this embodiment, the statistical time period may be set according to actual requirements. For example, each minute may be set as one statistical time period, each hour may be set as another statistical time period, and each day may be set as another statistical time period, so that the present embodiment may obtain the job cost information of each cluster job in each minute, each hour, and each day.

Step S140, for each statistical time period, calculating the job cost information of each stage of work unit in the statistical time period according to the job cost information of each cluster job in the statistical time period and each stage of work unit corresponding to each cluster job.

In this embodiment, hierarchical association relationships may exist between the working units of each level. For example, if the job submitting user of the cluster job is user a, and the department in which user a is located is the B department, the C research and development department, and the D group, the work units at each level are the B department, the C research and development department, the D group, and user a, respectively.

In the cluster data processing method provided by this embodiment, the job statistical data of each cluster job in each unit time is obtained to determine the computing node duration of each cluster job in each unit time, and according to the occupation cost information of each computing node 110 in each unit time and the computing node duration of each cluster job in each unit time, the job cost information generated in each computing process of each cluster job can be effectively determined, so that the cost of cluster computing resources can be distributed to each working unit, and data support is provided for cost accounting of an enterprise.

In a possible implementation manner, regarding step S110, in order to facilitate statistics of complete statistical data of each cluster job, referring to fig. 3, step S110 may be implemented by the following sub-step S111 and sub-step S112, which are described in detail below.

And a substep S111, for each cluster job, after the cluster job starts to calculate at the first time, respectively acquiring the resource duration corresponding to each type of cluster resource occupied by the cluster job in each unit time period.

And a substep S112, after the cluster job is calculated at the second time, acquiring the total resource duration of the cluster job between the first time and the second time.

In this embodiment, the cluster job starts to be calculated at the first time, which may refer to the time when the cluster job starts to be scheduled by the computer cluster 100 to a certain computing node 110 to start to use the computing resource of the computing node 110, that is, the first time is the time when the cluster job starts to be scheduled by the computer cluster 100 to a certain computing node 110 to start to use the computing resource of the computing node 110. The second time when the cluster job is finished computing may refer to the time when the cluster job is removed from the computing process of the computing node 110 by the computer cluster 100 and thus the computing resource of the computing node 110 is no longer used, that is, the second time when the cluster job is removed from the computing process of the computing node 110 by the computer cluster 100.

For example, referring to fig. 4, for a cluster job a, assuming that the cluster job a is scheduled by the computer cluster 100 to the computing node 110A to use the computing resource of the computing node 110A at time t1, and is removed from the computing process of the computing node 110A by the computer cluster 100 at time tn, time t1 is the first time, and time tn is the second time.

After the cluster job a starts to calculate at time t1, resource time lengths x1, x2, and the total resource time length corresponding to each cluster resource occupied by the cluster job a between t1 and tn is x1, x2, and the total resource time length corresponding to each cluster resource occupied by the cluster job a between t1 and tn may be acquired after the cluster job a starts to calculate at time t1 and then each unit time period t1-t2, t2-t3, and each cluster resource time period t-1-tn corresponds to each cluster resource.

For example, the job statistics for each cluster job per unit time may be as shown in table 1 below:

current unit time period

Cluster job identification

Working units of each stage

Resource duration

TABLE 1

For example, if each unit time period is every minute and the cluster job takes 30 minutes, then table 1 may include 30 rows of data.

In one possible implementation, for sub-step S111, the job submission time of the cluster job and the task time and occupied resources of each cluster task in the cluster job may be obtained first. And then, according to the operation information of the cluster operation, acquiring the resource duration corresponding to each cluster resource occupied by each cluster task in each unit time period. On the basis, the sum of the resource duration corresponding to each kind of cluster resource occupied by each cluster task in each unit time period is used as the resource duration corresponding to each kind of cluster resource occupied by the cluster task in each unit time period.

Optionally, the task time of each cluster task may specifically include a task start time and a task end time. For example, assume that a certain cluster task A1 is at 12: 00 starts the computation and ends at 16:00, then the cluster task a1 has a task start time of 12: 00. the task end time is 16: 00.

It should be noted that, when the job submission time, the task start time, and the task end time are not integer minutes, the integer minutes that is the latest of the job submission time, the task start time, and the task end time is taken as the corresponding job submission time, task start time, and task end time. For example, if the task start time is actually 12: 01: 22, then the task start time is determined to be 12: 01. as another example, if the task end time is actually 17: 21: 42, then the task start time is determined to be 17: 22.

in a possible implementation manner, for each cluster task, the occupied time of the cluster task in each unit time period may be obtained according to the task time of the cluster task and the job submission time of the cluster job. Then, for each unit time period, judging whether the occupied time length of the cluster task in the unit time period is greater than 0, if so, calculating the resource time length corresponding to each type of cluster resource occupied by the cluster task in the unit time period according to the occupied resource of the cluster task and the occupied time length in the unit time period. Or if the occupation time of the cluster task in the unit time period is equal to 0, determining that the resource time corresponding to the occupation of each cluster resource by the cluster task in the unit time period is 0.

Alternatively, the occupied resources may include CPU occupied resources and memory occupied resources, and then the resource duration may include CPU resource duration and memory resource duration when calculating the resource duration. In detail, the product of the CPU resource occupied by the CPU and the occupied duration may be calculated as the CPU resource duration, and the product of the memory resource occupied by the memory and the occupied duration may be calculated as the memory resource duration.

Optionally, referring to fig. 5 in combination, the occupancy duration may include a first occupancy duration and a second occupancy duration, the first occupancy duration may refer to an occupancy duration of the trunking task in each unit time period between the first time and the task start time of the trunking task, and the second occupancy duration may refer to an occupancy duration (X seconds) of the trunking task in each unit time period between the task start time of the trunking task and the second time, where the first occupancy duration is 0.

For example, still taking the aforementioned occupied resources including CPU occupied resources and memory occupied resources as an example, if the CPU occupied resources of the cluster task a1 are 3 CPU cores, and the memory occupied resources are 4000MB of memory, in 12: 01-12:02 for 30 seconds >0, then the cluster task A1 is at 12: the CPU resource occupation time length of 01-12:02 is 30 × 3-90, and the memory resource occupation time length is 30 × 4000-120000. Additionally, if the cluster task A1 is at 12: 01-12:02 equals 0, then cluster task A1 is at 12: the resource occupation time in the range of 01-12:02 is 0.

Therefore, through the steps, the resource duration corresponding to each type of computing resource occupied by each cluster job can be obtained, wherein the resource duration corresponding to each type of cluster resource occupied by each cluster job in each unit time period and the total resource duration between the first time and the second time are included, and therefore the computing node duration of each cluster job in each unit time can be conveniently determined subsequently.

For example, on the basis of the above, for step S120, in order to accurately determine the computing node time length, for each computing resource, the computing node time length corresponding to the computing resource occupied by each cluster job in each unit time may be calculated, where the computing node time length is a ratio of the resource time length corresponding to the computing resource occupied by the cluster job to the configured number of the computing resource in each computing node 110. Then, from the time lengths of the computing nodes of each cluster job in each unit time, the maximum time length of the computing node is respectively selected as the time length of the computing node of the cluster job in each unit time.

For example, still taking the aforementioned occupied resources including CPU occupied resources and memory occupied resources as an example, assuming that each compute node 110 in the computer cluster 100 includes m CPU cores and memory resources of nMB, if the duration of time that the cluster job a occupies the CPU resources corresponding to the CPU cores in the unit time t1-t2 is x, and the duration of time that the cluster job a occupies the memory resources corresponding to the memory resources in the unit time t1-t2 is y, the duration of time that the cluster job a occupies the compute nodes corresponding to the CPU cores in the unit time t1-t2 is x/m, and the duration of time that occupies the memory resources in the unit time t1-t2 is y/m. Then, the largest computing node duration is selected from x/m and y/m as the computing node duration of the cluster job A within the unit time t1-t 2.

Next, in one possible implementation, for step S130, a plurality of statistical time periods may be determined according to actual needs, so as to perform aggregation according to the statistical time periods. Specifically, for each cluster job, the calculation node duration occupied by the cluster job in each statistical time period is obtained according to the calculation node duration of the cluster job in each unit time.

For example, still taking fig. 4 as an example, the computing node durations of cluster job a in unit times t1-t2, t2-t3,... said.. and tn-1-tn may be obtained, assuming that the unit time is one minute, the statistical time period includes each minute and each day, and the duration of t1-tn is less than one day, then the computing node durations M1, M2, M3,. said.. and Mn-1-tn occupied by the cluster job a in each minute (t1-t2, t2-t3,. said.. and tn-1-tn) and the computing node duration D1 occupied in one day (t1-tn) may be obtained.

Then, for each statistical time period, a product of a computing node duration occupied by the cluster job in the statistical time period and cost information occupied by each computing node 110 in a unit time may be calculated as the job cost information of the cluster job in the statistical time period.

In this embodiment, the occupation cost information of each computing node 110 in unit time may be preset according to actual requirements, and it is assumed that the cost of each computing node 110 in each minute is cost_minThen, taking the above example as an example, the operation cost information of cluster operation a in each minute (t1-t2, t2-t3,..... multidot.tn-1-tn) is M1 × cost_min、M2*cost_min、M3*cost_min、......、Mn-1*cost_minThe operation cost information in one day (t1-tn) is D1 × cost_min。

Thus, for step S140, in order to facilitate the cost of the cluster computing resources to be distributed to each working unit and further provide data support for cost accounting of the enterprise, on the basis of determining the job cost information of each cluster job in different statistical time periods, each cluster job corresponds to each working unit at each level, so that the job cost information of each working unit at each level in different statistical time periods can be obtained according to the job cost information of each cluster job in different statistical time periods and each working unit at each level corresponding to each cluster job.

For example, taking the example that each level of work unit includes a user and a department where the user is located, and the statistical time period includes each minute and each day, the operation cost information of each user in each minute is shown in table 2:

minute (min)

User' s

Cost of work information

TABLE 2

The operation cost information for each user on each day is shown in table 3:

sky

User' s

Cost of work information

TABLE 3

The operating cost information per minute for each department is shown in table 4:

minute (min)

Department of department

Cost of work information

TABLE 4

The operating cost information for each department on each day is shown in table 5:

sky

Department of department

Cost of work information

TABLE 5

Therefore, the operation cost information of each user in each minute and each day and the operation cost information of each department in each minute and each day can be obtained, so that the cost of cluster computing resources can be distributed to each user and each department, and data support is provided for cost accounting of each user and each department.

In order to improve the accuracy of the job cost information, the present embodiment needs to consider whether the occupation cost information of each computing node 110 in unit time is reasonable and objective. Based on this, the following provides an exemplary implementation of determining the occupation cost information of each computing node 110 in a unit time, please further refer to fig. 6, the cluster data processing method provided in this embodiment may further include step S101 and step S102, which are described in detail as follows.

In step S101, cost information of the computing node 110 is acquired.

Step S102, obtaining the occupation cost information of the computing node 110 in unit time according to the cost information of the computing node 110.

In this embodiment, the cost information of the computing node 110 may include a purchase cost, a warranty duration, and an operation cost for each preset time period. For example, suppose that the purchase cost of the computing node 110 is a yuan, the operating cost (such as equipment management fee, electricity fee, etc.) of each month is about b yuan on average, and the warranty duration is c years. Taking the unit time as each minute, the cost per minute occupied by the computing node 110_min(a + b 12 months c)/c/365 days/24 hours/60 minutes.

Fig. 7 is a schematic diagram illustrating functional modules of a cluster data processing apparatus 200 according to an embodiment of the present application, where the embodiment may perform functional module division on the cluster data processing apparatus 200 according to the foregoing method embodiment. For example, the functional blocks may be divided for the respective functions, or two or more functions may be integrated into one processing block. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, the division of the modules in the present application is schematic, and is only a logical function division, and there may be another division manner in actual implementation. For example, in the case of dividing each function module according to each function, the cluster data processing apparatus shown in fig. 7 is only a schematic apparatus diagram. The cluster data processing apparatus 200 may include an obtaining module 210, a first determining module 220, a second determining module 230, and a calculating module 240, and the functions of the functional modules of the cluster data processing apparatus 200 are described in detail below.

The obtaining module 210 is configured to obtain job statistics data of each cluster job in each unit time, where the job statistics data includes a resource duration corresponding to each computing resource occupied by the cluster job and each level of corresponding work unit. It is understood that the obtaining module 210 can be configured to perform the step S110, and for a detailed implementation of the obtaining module 210, reference may be made to the content related to the step S110.

The first determining module 220 is configured to determine, according to job statistics data of each cluster job in unit time, a computing node duration of each cluster job in each unit time. It is understood that the first determining module 220 can be used to perform the step S120, and for the detailed implementation of the first determining module 220, reference can be made to the above description regarding the step S120.

The second determining module 230 is configured to determine, according to the occupation cost information of each computing node 110 in the unit time and the computing node duration of each cluster job in each unit time, job cost information of each cluster job in different statistical time periods. It is understood that the second determining module 230 can be used to perform the step S130, and for the detailed implementation of the second determining module 230, reference can be made to the contents related to the step S130.

And a calculating module 240, configured to calculate, for each statistical time period, job cost information of each level of work unit in the statistical time period according to the job cost information of each cluster job in the statistical time period and each level of work unit corresponding to each cluster job. It is understood that the calculating module 240 may be used to execute the step S140, and for the detailed implementation of the calculating module 240, reference may be made to the contents related to the step S140.

In a possible implementation manner, the obtaining module 210 may specifically obtain the job statistics data of each cluster job in each unit time by:

for each cluster job, after the cluster job starts to be calculated at the first moment, respectively acquiring the resource duration corresponding to each cluster resource occupied by the cluster job in each unit time period;

and after the cluster job is calculated at the second moment, acquiring the total resource duration of the cluster job between the first moment and the second moment.

In a possible implementation manner, the first determining module 220 may specifically determine the computing node duration of each cluster job in each unit time by:

for each kind of computing resource, computing a computing node duration corresponding to each cluster job occupying the computing resource in each unit time, wherein the computing node duration is a ratio of the resource duration corresponding to the computing resource occupied by the cluster job to the configuration number of the computing resource in each computing node 110;

and respectively selecting the maximum calculation node time length from the calculation node time lengths of each cluster job in each unit time as the calculation node time length of the cluster job in each unit time.

In a possible implementation manner, the second determining module 230 may specifically determine the job cost information of each cluster job in different statistical time periods by:

aiming at each cluster job, obtaining the time length of a computing node occupied by the cluster job in each statistical time period according to the time length of the computing node of the cluster job in each unit time;

and calculating the product of the time length of the calculation node occupied by the cluster job in the statistical time period and the occupation cost information of each calculation node 110 in unit time as the job cost information of the cluster job in the statistical time period.

Embodiments of the present application further provide a readable storage medium containing computer-executable instructions, which when executed, may be used to perform the cluster data processing method provided in the foregoing method embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only for various embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and all such changes or substitutions are included in the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.