阅读说明：本技术 一种卫星影像处理算法混合调度系统及方法 (Satellite image processing algorithm hybrid scheduling system and method ) 是由 胡风华 周健 赵文波 张建学 臧远慧 于 2020-03-19 设计创作，主要内容包括：本申请提供了一种卫星影像处理算法混合调度系统及方法。所述调度系调度系统包括：管理节点、管理网络、Linux服务器、Windows工作站、数据网络和网络共享存储设备；所述管理节点包括工作流调度设备,用于创建、管理、调度和更新工作流；所述管理网络,用于在工作流调度设备与所述Linux服务器和与所述Windows工作站之间进行工作流作业派发和状态收集；所述Linux服务器用于执行Linux算法；所述Windows工作站用于执行Windows算法；所述数据网络,用于所述Linux服务器和所述Windows工作站与网络共享存储设备之间的文件传输；所述网络共享存储设备,用于存储所述Linux服务器和所述Windows工作站执行算法所需的文件。本发明在获得高性能优点的同时,实现影像处理流程的自动化。(The application provides a satellite image processing algorithm hybrid scheduling system and a satellite image processing algorithm hybrid scheduling method. The scheduling system comprises: the system comprises a management node, a management network, a Linux server, a Windows workstation, a data network and a network sharing storage device; the management node comprises workflow scheduling equipment used for creating, managing, scheduling and updating workflows; the management network is used for dispatching the workflow operation and collecting the state between the workflow dispatching equipment and the Linux server and between the workflow dispatching equipment and the Windows workstation; the Linux server is used for executing a Linux algorithm; the Windows workstation is used for executing Windows algorithm; the data network is used for file transmission between the Linux server and the Windows workstation as well as the network sharing storage equipment; and the network sharing storage equipment is used for storing files required by the Linux server and the Windows workstation to execute the algorithm. The invention realizes the automation of the image processing flow while obtaining the advantage of high performance.)

1. A satellite image processing algorithm hybrid scheduling system, comprising: the system comprises a management node, a management network, a Linux server, a Windows workstation, a data network and a network sharing storage device;

the management node comprises workflow scheduling equipment used for creating, managing, scheduling and updating workflows;

the management network is used for dispatching the workflow operation and collecting the state between the workflow dispatching equipment and the Linux server and between the workflow dispatching equipment and the Windows workstation;

the Linux server is used for executing a Linux algorithm;

the Windows workstation is used for executing Windows algorithm;

the data network is used for file transmission between the Linux server and the Windows workstation as well as the network sharing storage equipment;

and the network sharing storage equipment is used for storing files required by the Linux server and the Windows workstation to execute the algorithm.

2. The scheduler system according to claim 1, wherein the workflow scheduler supports serial and parallel workflows, both of which comprise a hybrid mode of Windows and Linux algorithms.

3. The dispatching system of claim 2, wherein the serial workflow comprises four modes of Linux algorithm + Linux algorithm, Linux algorithm + Windows algorithm, Window algorithm + Windows algorithm, and Windows algorithm + Linux algorithm.

4. A scheduling system according to any one of claims 1 to 3 wherein the workflow scheduling apparatus comprises: the system comprises a workflow manager, a workflow analyzer, a resource scheduler and a task distributor;

the workflow manager is used for managing the relevant information and state of the workflow submitted by the user;

the workflow analyzer is used for analyzing each job in a workflow to be operated, reading related parameters and transmitting the parameters to the task distributor;

the resource scheduler is used for managing computing resources, recording relevant configuration information and allocation conditions of the computing resources, and allocating proper resources for each job in the workflow;

the computing resources include: the Linux server and the Windows workstation;

the task distributor is used for selecting a Linux server or a Windows workstation according to the parameters of each job and the resources distributed by the resource scheduler to distribute each job to the Linux server or the Windows workstation for processing, and receiving the execution result and the state of the Linux server or the Windows workstation.

5. A satellite image processing algorithm hybrid scheduling method is characterized by comprising the following steps:

receiving related information and state of a workflow submitted by a user, and creating the workflow;

analyzing each job in the workflow, and reading related parameters;

recording relevant configuration information and distribution conditions of computing resources, and distributing proper resources for each job in the workflow, wherein the computing resources comprise the Linux server and the Windows workstation;

selecting a Linux server or a Windows workstation according to the parameters and the allocated resources of each job, and distributing each job to the selected Linux server or Windows workstation for processing;

the Linux server executes a Linux algorithm, and the Windows workstation executes a Windows algorithm.

6. The scheduling method of claim 5 wherein the workflow comprises a serial workflow and a parallel workflow;

the serial workflow comprises four modes of a Linux algorithm + a Linux algorithm, a Linux algorithm + a Windows algorithm, a Window algorithm + a Windows algorithm and a Windows algorithm + a Linux algorithm;

the parallel workflow comprises a Linux algorithm and/or a Windows algorithm which are run in parallel.

Technical Field

The invention relates to the technical field of image processing, in particular to a system and a method for hybrid scheduling of a satellite image processing algorithm.

Background

At present, most of the traditional satellite image processing algorithm running platforms are PCs or workstations of a Windows operating system. With the rapid increase of the number of satellites and the rapid increase of the image resolution, the image data shows exponential increase. The performance of a conventional PC or workstation-based platform is completely unable to cope with such a scenario, and a cluster-based High Performance Computing (HPC) system can connect multiple machines together through a high-speed network, aggregate computing power, have performance several times, several tens of times or even several thousands of times higher than that of the conventional PC or workstation, and can perform nearly linear expansion according to business growth, so that more and more image processing platforms are shifted to cluster-based systems. Unlike PCs and workstations, high performance computer clusters typically make widespread use of the Linux operating system, and therefore algorithms also need to be written and compiled based on the Linux platform into binary code that the Linux platform can run. Another convenience is that a large number of algorithms exist on the Windows platform, and it is very costly to migrate the algorithms to the Linux platform, some algorithms need to be modified (for example, some algorithms based on C/C + +), some algorithms need to be rewritten (for example, algorithms based on C #), and some algorithms cannot be migrated due to being too large.

In order to utilize the algorithm under the Windows platform in the Linux cluster, the common method is to manually copy the data to the Windows platform when necessary, call the algorithm for processing, copy the data back to the Linux system after the processing is finished, the whole process mainly depends on manual intervention, and the automatic operation of the flow cannot be realized.

Disclosure of Invention

The invention aims to provide a scheduling system and a scheduling method for mixing a satellite image processing algorithm of a multi-operating system platform, which enable a user to realize automation of a process while obtaining the high-performance advantage of an HPC platform by processing a satellite image.

To this end, according to a first aspect of the present invention, there is provided a satellite image processing algorithm hybrid scheduling system, comprising: the system comprises a management node, a management network, a Linux server, a Windows workstation, a data network and a network sharing storage device;

the management node comprises workflow scheduling equipment used for creating, managing, scheduling and updating workflows;

the management network is used for dispatching the workflow operation and collecting the state between the workflow dispatching equipment and the Linux server and between the workflow dispatching equipment and the Windows workstation;

the Linux server is used for executing a Linux algorithm;

the Windows workstation is used for executing Windows algorithm;

the data network is used for file transmission between the Linux server and the Windows workstation as well as the network sharing storage equipment;

and the network sharing storage equipment is used for storing files required by the Linux server and the Windows workstation to execute the algorithm.

Further, the workflow scheduling device supports serial workflows and parallel workflows, and both workflows comprise a mixed mode of Windows and Linux algorithms.

Further, the serial workflow comprises four modes of a Linux algorithm + a Linux algorithm, a Linux algorithm + a Windows algorithm, a Window algorithm + a Windows algorithm and a Windows algorithm + a Linux algorithm.

Further, the workflow scheduling apparatus includes: the system comprises a workflow manager, a workflow analyzer, a resource scheduler and a task distributor;

the workflow manager is used for managing the relevant information and state of the workflow submitted by the user;

the workflow analyzer is used for analyzing each job in a workflow to be operated, reading related parameters and transmitting the parameters to the task distributor;

the resource scheduler is used for managing computing resources, recording relevant configuration information and allocation conditions of the computing resources, and allocating proper resources for each job in the workflow;

the computing resources include: the Linux server and the Windows workstation;

the task distributor is used for selecting a Linux server or a Windows workstation according to the parameters of each job and the resources distributed by the resource scheduler to distribute each job to the Linux server or the Windows workstation for processing, and receiving the execution result and the state of the Linux server or the Windows workstation.

According to a second aspect of the present invention, there is provided a hybrid scheduling method for satellite image processing algorithms, comprising:

receiving related information and state of a workflow submitted by a user, and creating the workflow;

analyzing each job in the workflow, and reading related parameters;

recording relevant configuration information and distribution conditions of computing resources, and distributing proper resources for each job in the workflow, wherein the computing resources comprise the Linux server and the Windows workstation;

selecting a Linux server or a Windows workstation according to the parameters and the allocated resources of each job, and distributing each job to the selected Linux server or Windows workstation for processing;

the Linux server executes a Linux algorithm, and the Windows workstation executes a Windows algorithm.

Further, the workflow comprises two types of serial workflow and parallel workflow;

the serial workflow comprises four modes of a Linux algorithm + a Linux algorithm, a Linux algorithm + a Windows algorithm, a Window algorithm + a Windows algorithm and a Windows algorithm + a Linux algorithm;

the parallel workflow comprises a Linux algorithm and/or a Windows algorithm which are run in parallel.

The invention has the beneficial effects that: through the hybrid scheduling processing of the satellite image processing algorithm, the workflow is automatically distributed to a proper Linux server or a proper Windows workstation, so that a user does not need to transplant or rewrite the Windows algorithm, and the automation of the process is realized while the high-performance advantage of the HPC platform is obtained.

Drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a block diagram of a hybrid satellite image processing algorithm scheduling system according to an embodiment of the present invention;

FIG. 2 is a block diagram of a management node in a hybrid scheduling system for satellite image processing algorithms according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a serial workflow according to an embodiment of the invention;

FIG. 4 is a schematic diagram of a parallel workflow according to an embodiment of the invention;

FIG. 5 is a flowchart illustrating a hybrid scheduling method for satellite image processing algorithms according to an embodiment of the present invention.

Detailed Description

In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.

As shown in fig. 1, according to an embodiment of the present invention, a satellite image processing algorithm hybrid scheduling system is provided, including a management node 1, a management network 2, a Linux server 3, a Windows workstation 4, a data network 5, and a network shared storage device 6, where the management node 1 is connected to the Linux server 3 and the Windows workstation 4 through the management network 2, and the Linux server 3 and the Windows workstation 4 are connected to the network shared storage device 6 through the data network 5;

the management node 1 comprises a workflow scheduling device 11 for creating, managing, scheduling and updating workflows.

The management network 2 is used for dispatching the workflow jobs and collecting the states between the workflow dispatching equipment 11 and the Linux server 3 and between the workflow dispatching equipment and the Windows workstation 4, the jobs distributed by the management node 1 reach the corresponding Linux server 3 or Windows workstation 4 through the management network 2, and the states of the Linux server 3 and the Windows workstation 4 and the job processing results are fed back to the management node 1 through the management network 2.

The Linux server 3 is configured to execute a Linux algorithm. The Linux algorithm is an algorithm which is compiled and compiled into Linux platform-executable binary codes based on a Linux platform. The Windows workstation 4 is used to execute Windows algorithms. The Windows algorithm refers to an algorithm which is compiled and compiled into binary codes which can be run by a Windows platform based on the Windows platform.

The data network 5 is used for file transmission between the Linux server 3 and the Windows workstation 4 and the network shared storage device 6.

The network shared storage device 6 is used for storing files required by the Linux server 3 and the Windows workstation 4 to execute the algorithm, including input files and output files of the algorithm.

As shown in fig. 2, the management node 1 includes a workflow scheduling apparatus 11 and a workflow database 12, and the workflow database 12 is used for storing information about workflows submitted by users in a workflow manager 111.

The workflow scheduling apparatus 11 specifically includes:

the workflow manager 111 is used for managing the relevant information and state of the workflow submitted by the user, including receiving the relevant information and state of the workflow submitted by the user and creating the workflow;

a workflow parser 112, configured to parse each job in a workflow to be executed, read related parameters, and transmit the parameters to the task distributor 114;

a resource scheduler 113, configured to manage the Linux server 3 and the Windows workstation 4, record relevant configuration information and allocation status of the computing resources, and allocate appropriate resources to each job in the workflow, where the computing resources include the Linux server 3 and the Windows workstation 4;

and the task distributor 114 is configured to select the Linux server 3 or the Windows workstation 4 according to the parameter of each job and the resource allocated by the resource scheduler, distribute each job to the Linux server 3 or the Windows workstation 4 for processing, and receive an execution result and a status of the Linux server 3 or the Windows workstation 4.

Optionally, the Linux server 3 comprises a Linux dispatching agent, the Windows workstation 4 comprises a Windows dispatching agent, and the dispatching agent comprises a communication agent for communicating with the workflow dispatcher and an executor for calling the algorithm of the platform to execute the job.

The dispatching system supports serial workflow and parallel workflow, and both the two workflows comprise a mixed mode of Windows and Linux algorithms;

the serial workflow is shown in fig. 3, and includes four types:

the Linux algorithm and the Linux algorithm, namely continuous operation is executed by the Linux algorithm;

linux algorithm + Windows algorithm, i.e. continuous operation is executed by Linux algorithm first and then by Windows algorithm;

window algorithm + Windows algorithm, i.e. continuous operation is executed by Windows algorithm;

windows algorithm + Linux algorithm, that is, continuous operation is executed by the Windows algorithm first and then by the Linux algorithm.

The parallel workflow is as shown in fig. 4, the Linux algorithm and/or the Windows algorithm run in parallel, that is, the continuous jobs are executed by the Linux algorithm and/or the Windows algorithm at the same time.

Through the hybrid scheduling processing of the satellite image processing algorithm, the workflow is automatically distributed to a proper Linux server 3 or a proper Windows workstation 4, so that a user does not need to transplant or rewrite the Windows algorithm, and the automation of the process is realized while the high-performance advantage of the HPC platform is obtained.

The embodiment of the invention provides a scheduling method for satellite image processing of multiple operating system platforms, as shown in fig. 5, comprising the following steps:

s51, receiving related information of the workflow submitted by the user, and creating the workflow;

the workflow manager 111 receives information about workflows submitted by users, creates workflows, and stores the workflows in the workflow database 12. At run-time, the workflow manager 111 selects an appropriate workflow from the workflow database 12 by a predetermined policy.

S52, analyzing each job in the workflow, and reading related parameters;

after a workflow to be run is selected, each job in the workflow to be run is parsed by workflow parser 112, the relevant parameters are read, and the parameters are transmitted to the task dispatcher 114.

S53, recording the relevant configuration information and allocation status of computing resources, and allocating appropriate resources for each job in the workflow;

the computing resources comprise available Linux servers and Windows workstations, and the resource scheduler 113 records relevant configuration information and allocation conditions of the computing resources and allocates appropriate resources for each job in the workflow according to the information. Therefore, the processing capacity can be distributed evenly, and the image processing efficiency is improved.

S54, selecting a Linux server or a Windows workstation according to the parameters and the allocated resources of each job, and distributing each job to the selected Linux server or Windows workstation for processing;

the task distributor 114 selects a Linux server 3 or a Windows workstation 4 according to the parameter of each job and the resource allocated by the resource scheduler 113, distributes each job to the corresponding Linux server 3 or Windows workstation 4 for processing, and receives the execution result and the state of the Linux server 3 or Windows workstation 4, wherein the Linux server 3 executes a Linux algorithm and the Windows workstation 4 executes a Windows algorithm.

The satellite image processing hybrid scheduling method supports a serial workflow and a parallel workflow, wherein the serial workflow is shown in fig. 3, and the parallel workflow is shown in fig. 4.

It will be understood by those skilled in the art that all or part of the steps in the methods of the embodiments described above may be implemented by hardware instructions of a program, and the program may be stored in a computer-readable storage medium, where the storage medium includes Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM), or other Memory, such as a magnetic disk, or a combination thereof, A tape memory, or any other medium readable by a computer that can be used to carry or store data.

The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.