阅读说明：本技术 信息处理系统、信息处理方法以及建筑机械 (Information processing system, information processing method, and construction machine ) 是由 西田裕平 志垣富雄 大前谦 荒木望 芳住幸平 于 2020-08-21 设计创作，主要内容包括：本发明是鉴于现有的作业机械的课题而完成的,其目的之一在于提供一种能够用简单的结构来估计作业部的姿势的作业机械的信息处理系统、信息处理方法以及建筑机械。信息处理系统(10)具备存储部(30)和图像信息获取部(12)。存储部(30)将作业机械(100)的作业部(40)的参照图像的数据(Gs)和作业机械(100)的作业部(40)的姿势的数据(Ks)相对应地存储。图像信息获取部(12)获取作业部(40)的图像的数据(Gj)。(The present invention has been made in view of the problems of the conventional work machine, and an object thereof is to provide an information processing system and an information processing method for a work machine, and a construction machine, which are capable of estimating the posture of a working unit with a simple configuration. The information processing system (10) is provided with a storage unit (30) and an image information acquisition unit (12). A storage unit (30) stores data (Gs) of a reference image of a working unit (40) of a working machine (100) and data (Ks) of the posture of the working unit (40) of the working machine (100) in association with each other. An image information acquisition unit (12) acquires data (Gj) of an image of a work unit (40).)

1. An information processing system is provided with:

a storage unit that stores data of a reference image of a working unit of a working machine and data of a posture of the working unit of the working machine in association with each other; and

and an acquisition unit that acquires data of an image of the working unit of the working machine and compares the data with data of the reference image.

2. The information processing system according to claim 1,

the acquisition unit is configured to rotate integrally with a working unit of the working machine.

3. The information processing system according to claim 1 or 2,

the image processing apparatus further includes an estimation unit that estimates a posture of the working unit of the working machine based on the data of the image and the storage information of the storage unit.

4. The information processing system according to any one of claims 1 to 3,

the storage unit stores a posture estimation model generated by machine learning from data of the reference image and data of the posture.

5. The information processing system according to any one of claims 1 to 3,

the image processing apparatus further includes a correction unit that corrects the image data based on at least one of information about an ambient environment when the image data is acquired, a number of years elapsed for the work machine, and an individual difference of the work machine.

6. The information processing system according to claim 3,

the estimation unit estimates the operation state based on at least one of information about a surrounding environment when the image data is acquired, a number of years elapsed for the work machine, and an individual difference of the work machine.

7. The information processing system according to any one of claims 1 to 6,

the image processing apparatus further includes a color information removal unit for compressing or removing color information of the data of the reference image.

8. The information processing system according to any one of claims 1 to 7,

the image processing apparatus further includes a background information removal unit configured to remove a background image of the data of the reference image.

9. An information processing system is provided with:

a storage unit that stores data of a reference image of a working unit of a working machine and data of a posture of the working unit of the working machine in association with each other;

an acquisition unit configured to rotate integrally with the working unit of the working machine, and acquire data of an image of the working unit of the working machine to compare with the data of the reference image; and

an estimating unit that estimates a posture of the working unit of the working machine based on the data of the image and the storage information of the storage unit.

10. An information processing method comprising the steps of:

storing data of a reference image of a working part of a working machine and data of a posture of the working part of the working machine in a storage part in association with each other;

acquiring data of an image of the working part to compare with data of the reference image; and

estimating the posture of the working unit of the working machine based on the data of the image and the storage information of the storage unit.

11. The information processing method according to claim 10,

the estimating step includes a step of referring to a posture estimation model generated by machine learning from data of the reference image and data of the posture.

12. A computer program for causing a computer to realize functions of:

storing data of a reference image of a working part of a working machine and data of a posture of the working part of the working machine in a storage part in association with each other;

acquiring data of an image of a working part of the working machine for comparison with data of the reference image; and

estimating the posture of the working unit of the working machine based on the data of the image and the storage information of the storage unit.

13. A construction machine is provided with:

an operation unit of an operation machine;

a storage unit that stores data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in association with each other;

an acquisition unit that acquires data of an image of a working unit of the working machine and compares the data with data of the reference image; and

an estimating unit that estimates a posture of the working unit of the working machine based on the data of the image and the storage information of the storage unit.

Technical Field

The present invention relates to an information processing system, an information processing method, and a construction machine.

Background

There is known a construction machine including a camera for photographing a working device to determine a posture of the working device. For example, patent document 1 describes a construction machine including: a camera which is provided on the revolving structure and shoots the working device; an angle detection unit for detecting a relative angle; and a gesture determination section for determining a gesture. The construction machine detects a relative angle between links from edges of the links extracted based on an image of the camera, and determines a posture of the work implement with respect to the revolving structure based on the relative angle.

Documents of the prior art

Patent document

Patent document 1: japanese patent laid-open publication No. 2017-053627

Disclosure of Invention

Problems to be solved by the invention

The present inventors have made the following findings on a construction machine including a boom, an arm, and an attachment (attachment) driven by power such as hydraulic pressure.

A certain construction machine drives an arm mechanism including a boom, an arm, and the like by power to operate an attachment member such as a bucket attached to the arm mechanism to perform a predetermined work. In order to avoid a collision between the arm mechanism and the attachment (hereinafter referred to as "working unit") or to control the operation of the working unit by feeding back the position of the working unit, it is considered to determine and use information on the three-dimensional position of each part of the working unit (hereinafter referred to as "attitude of the working unit" or simply "attitude") such as a link angle between each part of the working unit such as a boom, an arm, and a bucket, or a stroke length of each cylinder.

In order to determine the posture of the working unit, it is conceivable to provide a stroke sensor for each cylinder for driving each unit, and calculate and obtain the posture of the working unit from the detection result of the stroke sensor. In this case, since the stroke sensor is provided for each part and the wiring is provided between the stroke sensor and the control device, the structure becomes complicated, which is disadvantageous in terms of cost.

From the viewpoint of estimating the posture of the working unit with a simple configuration, it cannot be said that the construction machine described in patent document 1 sufficiently copes with. Such a problem occurs not only in construction machines but also in other types of working machines.

The present invention has been made in view of the above problems, and an object thereof is to provide an information processing system for a working machine capable of estimating the posture of a working unit with a simple configuration.

Means for solving the problems

In order to solve the above problem, an information processing system according to an aspect of the present invention includes: a storage unit that stores data of a reference image of a working unit of a working machine and data of a posture of the working unit of the working machine in association with each other; and an acquisition unit that acquires data of an image of the working unit of the working machine and compares the data with data of the reference image.

In addition, any combination of the above, and a mode in which the constituent elements or expressions of the present invention are replaced with each other among a method, an apparatus, a program, a transient or non-transient storage medium in which a program is recorded, a system, and the like are also effective as modes of the present invention.

ADVANTAGEOUS EFFECTS OF INVENTION

According to the present invention, it is possible to provide an information processing system for a working machine capable of estimating the posture of a working unit with a simple configuration.

Drawings

Fig. 1 is a side view schematically showing a working machine including an information processing system according to a first embodiment.

Fig. 2 is a block diagram schematically showing the information processing system of fig. 1.

Fig. 3 is an explanatory diagram for explaining the posture of the working unit of the working machine of fig. 1.

Fig. 4 is a diagram showing learning data of the information processing system of fig. 1.

Fig. 5 is an explanatory diagram for explaining the posture estimation processing of the information processing system of fig. 1.

Fig. 6 is a flowchart showing the operation of the information processing system of fig. 1.

Fig. 7 is a flowchart showing an operation of the information processing system of fig. 1.

Description of the reference numerals

10: an information processing system; 20 d: a model generation unit; 20 e: a posture estimation unit; 20k is as follows: a background information removal unit; 20 n: a color information removal unit; 30: a storage unit; 32: a model storage unit; 40: an operation section; 42: a movable arm; 44: a bucket rod; 46: a bucket; 100: a working machine; 1000: a construction machine.

Detailed Description

The present inventors have studied a method of estimating the posture of a working section from image information of a camera, and have found the following. In the construction machine, the following structure is considered: edges of each portion are extracted from image information of a camera for photographing the work portion, relative angles of each portion are detected from the edges, and a posture is determined from the relative angles. However, in this configuration, since the multi-stage information processing is performed successively, it is necessary to perform a large amount of information processing using an expensive high-performance CPU. If a CPU with low processing power is used, the processing cannot keep up with the CPU, the estimation accuracy is degraded, and collision avoidance and operation control do not function properly. In addition, the information processing also has the following problems: since it is necessary to develop a corresponding information processing program for each of various shapes of construction machines, a large number of development steps are required.

In light of such background, the present inventors developed a technique for estimating the posture of a working unit using image information of a camera, reference image data of the working unit, and a database of posture data. According to this technique, the posture can be estimated with a simple configuration while suppressing a decrease in estimation accuracy. An example of this technique will be described below based on the embodiments.

The present invention will be described below based on preferred embodiments with reference to the drawings. In the embodiment and the modifications, the same or equivalent constituent elements and members are denoted by the same reference numerals, and overlapping descriptions are appropriately omitted. In addition, the dimensions of the components in the drawings are shown enlarged and reduced as appropriate for ease of understanding. In the drawings, some components that are not important in describing the embodiments are not shown.

The terms including the ordinal numbers such as the first and second are used for describing a plurality of kinds of the constituent elements, and the terms are used only for the purpose of distinguishing one constituent element from another constituent element, and the constituent elements are not limited by the terms.

[ first embodiment ]

The configuration of the information processing system 10 of a work machine according to the first embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a side view schematically showing a working machine 100 including an information processing system 10 according to a first embodiment. Fig. 2 is a block diagram schematically showing the information processing system 10.

The information processing system 10 includes an image information acquisition unit 12, an environment information acquisition unit 14, a posture information acquisition unit 16, a control unit 20, and a storage unit 30. The information processing system 10 has a normal operation time (hereinafter referred to as a "non-learning operation time") at the machine learning time and the non-machine learning time. The information processing system 10 can generate a posture estimation model based on the learning image information and the learning posture information of each part of the working unit 40 of the working machine 100 at the time of machine learning. The information processing system 10 can estimate the posture of the working unit 40 based on the real-time image information and the posture estimation model during the non-learning operation. The work machine 100 can control the operation of the working unit 40 based on the posture of the working unit 40 estimated by the information processing system 10.

The image information acquiring unit 12 acquires data of an image of the working unit 40. The environment information acquisition unit 14 acquires information on the environment around the work machine 100. The posture information acquiring unit 16 acquires data relating to the posture of the working unit 40 (hereinafter referred to as "posture data"). The control section 20 executes various data processes related to the generation of the posture estimation model and the posture estimation of the working section 40. The storage unit 30 stores data referred to or updated by the control unit 20. First, the configuration of the work machine 100 will be described, and the other configurations will be described later.

The work machine 100 of the present embodiment is a construction machine that performs work by moving the bucket 46, and functions as a so-called Power shovel (Power shovel). The work machine 100 includes a lower traveling unit 36, an upper body unit 34, an arm mechanism 48, and a bucket 46. In the present embodiment, the arm mechanism 48 and the bucket 46 constitute the working unit 40. The lower traveling unit 36 is configured to be capable of traveling in a predetermined direction on an endless track or the like. The upper hull portion 34 is mounted on the lower traveling portion 36. The upper body portion 34 and the working portion 40 are configured to be rotatable about the rotation axis La with respect to the lower traveling portion 36 by the rotation driving portion 60. The swing drive unit 60 can be constituted by, for example, a swing motor (not shown) and a swing gear (not shown). The upper body portion 34 is provided with a cabin 38.

An operation unit 54 for operating the working unit 40 is provided in the operation cabin 38. When an operation is input from the operation unit 54, the plurality of hydraulic valves 58 open and close in accordance with the operation. The hydraulic fluid supplied from a hydraulic pump (not shown) is delivered to the plurality of hydraulic cylinders 56 in response to opening and closing of the hydraulic valve 58. The hydraulic cylinder 56 includes hydraulic cylinders 56a, 56b, and 56c arranged in this order from the base end side to the tip end side of the arm mechanism 48. The hydraulic cylinders 56a, 56b, and 56c expand and contract according to the amount of hydraulic oil delivered.

As an example, a base end portion of the arm mechanism 48 is provided on the right side of the cab 38 in the upper body portion 34. The arm mechanism 48 includes, for example, a boom 42 and an arm 44 extending forward from the upper body portion 34. A bucket 46 is attached to the front end side of the arm mechanism 48. In this way, work machine 100 can drive bucket 46 to perform a desired work by changing the posture of work unit 40 in accordance with the manipulation by the operator. Further, work machine 100 can rotate upper body portion 34 and work portion 40, thereby moving bucket 46 three-dimensionally.

Fig. 3 is an explanatory diagram for explaining the posture of the working unit 40. The hydraulic cylinders 56a, 56b, 56c can change their telescopic lengths L1, L2, L3 in accordance with the hydraulic pressure. The boom 42 is configured to rotate the tip end portion vertically around the base end portion on the upper body portion 34 side by expansion and contraction of the hydraulic cylinder 56 a. Arm 44 is configured to rotate the tip portion back and forth about the base end portion on the boom 42 side by expansion and contraction of hydraulic cylinder 56 b. Bucket 46 is configured such that the tip end portion thereof is rotated back and forth or up and down about the base end portion on the arm 44 side by extending and contracting hydraulic cylinder 56 c.

In work unit 40, by changing the extension and contraction lengths L1, L2, and L3 of hydraulic cylinders 56a, 56b, and 56c, the bending angles θ 1, θ 2, and θ 3 of the joint portions connecting boom 42, arm 44, and bucket 46 can be changed. For example, angle θ 1 is an angle of boom 42 with respect to a horizontal plane, angle θ 2 is a bending angle of a joint portion connecting boom 42 and arm 44, and angle θ 3 is a bending angle of a joint portion connecting arm 44 and bucket 46.

The posture of work unit 40 can be defined by the positions and relative angles of boom 42, arm 44, and bucket 46. The shapes of boom 42, arm 44, and bucket 46 are fixed, and the posture of work unit 40 can be determined by geometric calculation based on the sizes of the respective parts of boom 42, arm 44, and bucket 46, and the extension and contraction lengths L1, L2, and L3 or the bending angles θ 1, θ 2, and θ 3 of hydraulic cylinders 56a, 56b, and 56 c.

Returning to fig. 2. Each block shown in fig. 2 can be realized by a component represented by a processor, a CPU, and a memory of a computer, an electronic circuit, or a mechanical device in hardware, and realized by a computer program or the like in software, but functional blocks realized by cooperation of these are depicted here. Thus, those skilled in the art will appreciate that these functional blocks can be implemented in various forms by a combination of hardware and software.

The image information acquiring unit 12 will be described. The image information acquiring unit 12 of the present embodiment includes an image sensor for capturing an image of the working unit 40. In machine learning described later, the image information acquisition unit 12 supplies the imaging result obtained by the imaging operation unit 40 to the control unit 20 as image data. Hereinafter, the data of the image acquired at the time of machine learning is referred to as "data Gs of the reference image". In the non-learning operation, the image information acquiring unit 12 also supplies the imaging result obtained by the imaging operation unit 40 to the control unit 20 as image data. Hereinafter, the image data acquired in the non-learning operation will be simply referred to as "image data Gj". The data Gj of the image may be data of a real-time image.

The image information acquiring unit 12 is configured to rotate integrally with the working unit 40. Specifically, the image information acquiring unit 12 is disposed on the ceiling of the control cabin 38 so as to be able to take an image of the working unit 40. When the working unit 40 is rotated, the image information acquiring unit 12 moves integrally with the working unit 40, and therefore, even if the rotation is performed, the relative positional relationship with the working unit 40 does not change.

The posture information acquiring unit 16 will be described. The hydraulic cylinders 56a, 56b, and 56c of the present embodiment include stroke sensors 16a, 16b, and 16c that acquire the extension/contraction lengths L1, L2, and L3. The posture information acquiring unit 16 is attached at the time of machine learning and detached at the time of non-learning operation. At the time of machine learning, the posture information acquiring unit 16 supplies data of the expansion/contraction lengths L1, L2, and L3 to the control unit 20.

The environmental information acquisition unit 14 will be described. When data of an image is acquired, if the surrounding environment such as weather is different, the brightness and color temperature of the image are also different, which becomes a factor of increasing an error of pose estimation. Therefore, in the present embodiment, the environment information acquiring unit 14 acquires information on the surrounding environment, and corrects the image data based on the acquired result. The environmental information acquisition unit 14 of the present embodiment includes an illuminance sensor for acquiring ambient brightness and a color temperature sensor for acquiring a color temperature. The environmental information acquisition unit 14 supplies the acquisition result to the control unit 20 as the environmental information Mp. In this example, the environmental information acquisition unit 14 is disposed on the ceiling of the control cabin 38.

The storage unit 30 will be explained. The storage section 30 includes a model storage section 32. The model storage unit 32 stores a posture estimation model for estimating the posture of the working unit 40, the posture estimation model being a model generated by known machine learning based on the data Gs of the reference image and the data of the posture. The pose estimation model may also be said to be a function in which the data format of the inputs and outputs is predetermined. The data Gj of the image is input to the posture estimation model of the embodiment. In addition, the posture estimation model of the embodiment outputs information on the estimated posture corresponding to the image data. The method of generating the posture estimation model will be described later.

The control unit 20 will be explained. The control unit 20 includes an image information receiving unit 20a, a posture information receiving unit 20b, a model generating unit 20d, a posture estimating unit 20e, an estimated posture presenting unit 20f, an individual information holding unit 20h, an environmental information receiving unit 20g, an image information correcting unit 20j, a background information removing unit 20k, and a color information removing unit 20 n. An application program in which a plurality of modules corresponding to these plurality of functional blocks are installed may be installed in the storage device (e.g., the storage section 30) of the information processing system 10. The processor (e.g., CPU) of the information processing system 10 can exert the functions of the respective functional blocks by reading out the application program into the main memory and executing the application program.

The image information receiving unit 20a receives an input of the imaging result of the working unit 40 from the image information acquiring unit 12. In particular, at the time of machine learning, the image information receiving unit 20a receives the reference image data Gs for learning from the image information acquiring unit 12. In the non-learning operation, the image information receiving unit 20a receives the image data Gj from the image information acquiring unit 12.

The posture information receiving unit 20b receives input of posture information of the working unit 40 from the posture information acquiring unit 16. Specifically, the posture information receiving unit 20b receives data of the expansion/contraction lengths L1, L2, and L3 of the stroke sensors 16a, 16b, and 16c during machine learning. When the data of the expansion lengths L1, L2, and L3 are collectively referred to as gesture data Ks.

Fig. 4 is a diagram showing learning data of the information processing system 10. In this figure, for easy understanding, the data Gs of the reference image from the image information acquiring unit 12 is replaced with the data Ks of the posture by a plan view, and the description is given. At the time of machine learning, the control unit 20 stores the received data Gs of the reference image and the data Ks of the posture in the storage unit 30 in association with each other. During machine learning, the control unit 20 changes the posture of the working unit 40 greatly within the movable range, and stores the data Gs of the reference image and the data Ks of the posture in the storage unit 30 in association with each other every time the posture of the working unit changes. The data Gs of the reference image and the data Ks of the posture stored in the storage unit 30 are referred to as learning data Sd. The learning data Sd is desired to cover the posture that the working unit 40 can take. Thus, as shown in fig. 4, the learning data Sd contains a large number of mutually corresponding data Gs of the reference image and data Ks of the posture.

The model generation unit 20d generates a posture estimation model using the data Gs of the reference image and the data Ks of the posture corresponding to each other in the learning data Sd as teacher data. The model generation unit 20d of the present embodiment generates a posture estimation model by machine learning (supervised learning) using the data Gs of the reference image and the data Ks of the posture as teacher data. The model generation unit 20d may generate the pose estimation model using a known machine learning method such as a support vector machine, a neural network (including deep learning), and a random forest. The model generation unit 20d stores the generated posture estimation model in the model storage unit 32.

In the non-learning operation, the posture estimation unit 20e estimates the posture of the working unit 40 based on the image data Gj and the storage information of the storage unit 30. For example, the posture estimation unit 20e may compare the image data Gj with the reference image data Gs of the learning data Sd, and use the posture data Ks corresponding to the reference image data Gs having the highest degree of similarity as the estimation result. In this case, since a large amount of the data Gs of the reference image is referred to, it may take time to obtain the result. Therefore, the posture estimation unit 20e of the present embodiment derives the estimated posture from the image data Gj using the posture estimation model generated by the model generation unit 20 d.

Fig. 5 is an explanatory diagram for explaining the posture estimation processing of the posture estimation model generated by the model generation unit 20 d. When data Gj of an image is input to the posture estimation model, estimated posture information Ke corresponding to the image data is output. The estimated attitude information Ke of the present embodiment includes bending angles θ 1, θ 2, and θ 3 of the joints of the boom 42, the arm 44, and the bucket 46 of the working unit 40.

The estimated posture presenting part 20f transmits the estimated posture information Ke estimated by the posture estimating part 20e to the outside of the information processing system 10. In the present embodiment, the estimated orientation presenting unit 20f transmits the estimated orientation information Ke to the work machine control unit 62. The work machine control unit 62 controls the operation of the work machine 100. For example, work machine control unit 62 controls opening and closing of hydraulic valve 58 to extend and contract hydraulic cylinders 56a, 56b, and 56c, thereby controlling operations of boom 42, arm 44, and bucket 46.

The image information correction unit 20j according to the present embodiment corrects the image data Gj based on information about the surrounding environment when the image data Gj is acquired, the elapsed years of the work machine, or individual differences of the work machine. The environmental information receiving unit 20g and the individual information holding unit 20h supply the image information correcting unit 20j with the correction information.

The environmental information receiving unit 20g receives the input of the acquisition result from the environmental information acquiring unit 14. In particular, the environmental information receiving unit 20g receives the environmental information Mp from the environmental information acquiring unit 14. The image information correcting section 20j corrects the brightness and color temperature of the image data Gj based on the environmental information Mp. The image information correcting unit 20j corrects the brightness of the image data Gj so as to be the same as the brightness of the reference image data Gs. The image information correcting section 20j corrects the color temperature of the image data Gj so as to be the same as the color temperature of the reference image data Gs. With this configuration, estimation errors due to the surrounding environment can be reduced.

The appearance of the working unit 40 varies from one working machine 100 to another. This individual difference may cause an error in the posture estimation. Therefore, the individual information holding unit 20h according to the present embodiment holds the individual information Me of each individual. The individual information Me includes information related to the number of years elapsed in the work machine 100, damage to the working unit 40, attachment, and individual differences in appearance due to deformation or the like. The image information correcting unit 20j corrects the image data Gj based on the individual information Me. According to this configuration, estimation errors due to individual differences can be reduced.

The image data Gj includes a background image that differs for each site where the work machine 100 is operating. Therefore, the background image included in the image data Gj may cause an error in the posture estimation. Therefore, the background information removal unit 20k according to the present embodiment removes information related to the background image from the image data Gj. With this configuration, the estimation error due to the background image can be reduced.

When the reference image data Gs and the image data Gj are stored as full-color image data and processed, the amount of data to be processed increases, which is disadvantageous in terms of processing speed and storage capacity. Therefore, the color information removing unit 20n according to the present embodiment removes the color information from the data Gs of the reference image and the data Gj of the image to obtain image data of a gray scale (gray scale). According to this configuration, the amount of data to be processed becomes small, which is advantageous in terms of processing speed and storage capacity.

The operation of the information processing system 10 configured as described above will be described. Fig. 6 is a flowchart showing the operation of the information processing system 10. This figure shows an action S70 of generating a pose estimation model by machine learning at the time of machine learning. The posture information acquiring unit 16 is attached to the work machine 100 in advance, and operation S70 is started at the timing when the administrator inputs the instruction for model creation.

If the model generation timing is reached (yes at step S71), the control section 20 of the information processing system 10 receives the data Gs of the reference image and the data Ks of the orientation from the image information acquisition section 12 and the orientation information acquisition section 16 (step S72). In this step, the control unit 20 greatly changes the posture of the working unit 40 within the movable range, and receives and stores the data Gs of the reference image and the data Ks of the posture in the storage unit 30 every time the posture of the working unit 40 changes.

The background information removal unit 20k removes information on the background image from the data Gs of the reference image (step S73). The color information removing section 20n removes the color information from the data Gs of the reference image (step S74). The removal of the background image and the removal of the color information may be performed on the received data Gs of the reference image each time, or the removal of the background image and the removal of the color information may be performed on the data Gs of the reference image stored in the storage unit 30.

The model generation unit 20d generates a pose estimation model by machine learning from the data Gs of the reference image from which the background image and the color information are removed and the data Ks of the pose, and stores the pose estimation model in the model storage unit 32 (step S75). If the pose estimation model is saved, act S70 ends. After completion of act S70, posture information acquiring unit 16 may be detached from work machine 100.

If the model generation timing is not reached (NO at step S71), S72 to S75 are skipped. This operation S70 is merely an example, and the order of steps may be changed, or some of the steps may be added, deleted, or changed.

Fig. 7 is also a flowchart showing the operation of the information processing system 10. This figure shows an operation S80 of estimating the posture of the working unit 40 from the image data Gj using the posture estimation model. In the non-learning operation, operation S80 is started at a timing when the administrator inputs an instruction for posture estimation.

If the timing of the posture estimation is reached (yes at step S81), the control section 20 of the information processing system 10 receives the data Gj of the image from the image information acquiring section 12 (step S82). In this step, the received image data Gj is stored in the storage unit 30.

The background information removal unit 20k removes information on the background image from the image data Gj (step S83). The color information removal unit 20n removes the color information from the image data Gj (step S84).

The image information correcting unit 20j corrects the image data Gj based on the individual information Me held in the individual information holding unit 20h (step S85). The removal of the background image, the removal of the color information, and the correction of the image are performed on the data Gj of the image stored in the storage unit 30.

The posture estimation unit 20e estimates the posture of the working unit 40 based on the posture estimation model from the image data Gj after the background image removal, the color information removal, and the image correction (step S86). In this step, estimated posture information Ke is output from the posture estimation model.

The estimated orientation presenting part 20f transmits the estimated orientation information Ke output from the orientation estimation model to the outside of the information processing system 10 (step S87). For example, the estimated posture presenting part 20f transmits the estimated posture information Ke to the work machine control part 62. If the estimated posture information Ke is transmitted, the action S80 ends. Act S80 is repeatedly performed until no indication of a pose estimate is given.

If the timing of posture estimation is not reached (NO at step S81), S82 to S87 are skipped. This operation S80 is merely an example, and the order of steps may be changed, or some of the steps may be added, deleted, or changed.

The features of the information processing system 10 of the present embodiment configured as described above will be described. The information processing system 10 includes: a storage unit 30 that stores data Gs of a reference image of the working unit 40 of the work machine 100 and data Ks of the posture of the working unit 40 of the work machine 100 in association with each other; and an image information acquiring unit 12 that acquires data Gj of an image of the working unit 40 of the working machine 100 to compare with data Gs of a reference image. With this configuration, the posture of the working unit 40 can be estimated from the image data Gj acquired by the image information acquisition unit 12, the reference image data Gs, and the posture data Ks.

The image information acquiring unit 12 may be configured to rotate integrally with the working unit 40 of the working machine 100. In this case, the operation of the working unit 40 of the working machine 100 can be easily determined.

The information processing system 10 may include an attitude estimation unit 20e, and the attitude estimation unit 20e may estimate the attitude of the working unit 40 based on the image data Gj and the storage information of the storage unit 30. In this case, the posture of the working unit 40 can be estimated from the image data Gj.

The storage unit 30 may store a posture estimation model generated by machine learning from the data Gs of the reference image and the data Ks of the posture. In this case, the posture estimation model can be generated by machine learning.

The information processing system 10 may further include an image information correction unit 20j, and the image information correction unit 20j may correct the image data Gj based on at least one of information about the surrounding environment when the image data Gj is acquired, the number of years that the work machine 100 has elapsed, and individual differences of the work machine 100. In this case, the data Gj of the image can be corrected to improve the estimation accuracy of the posture.

The posture estimation unit 20e may estimate the posture of the working unit 40 based on at least one of information about the surrounding environment when the image data Gj is acquired, the elapsed years of the working machine, and the individual differences of the working machine. In this case, the data Gj of the image can be corrected to improve the estimation accuracy of the posture.

The information processing system 10 may further include a color information removal unit 20n, and the color information removal unit 20n may compress or remove the color information of the data Gs of the reference image. In this case, it is advantageous in terms of the storage capacity and processing speed of the storage unit 30.

The information processing system 10 may further include a background information removal unit 20k for removing a background image of the data Gs of the reference image. In this case, a decrease in estimation accuracy due to the background image can be suppressed.

Next, second to fourth embodiments of the present invention will be explained. In the drawings and the description of the second to fourth embodiments, the same or equivalent components and members as those of the first embodiment are denoted by the same reference numerals. The description overlapping with the first embodiment will be omitted as appropriate, and the description will focus on the structure different from the first embodiment.

[ second embodiment ]

A second embodiment of the present invention is an information processing method for a working machine. The information processing method includes the steps of: storing data Gs of a reference image of the working unit 40 of the working machine 100 and data Ks of the posture of the working unit 40 of the working machine 100 in the storage unit 30 in association with each other (S72 to S75); acquiring data Gj of an image of the working unit 40 and comparing the data Gj with data Gs of a reference image (S82); and estimating the posture of the working unit 40 of the working machine 100 based on the image data Gj and the storage information of the storage unit 30 (S86).

The estimating step (S86) may include a step of referring to a posture estimation model generated by machine learning from the data Gs of the reference image and the data Ks of the posture. The configuration of the second embodiment provides the same effects as those of the first embodiment.

[ third embodiment ]

A third embodiment of the present invention is a construction machine 1000. The construction machine 1000 includes: a working unit (40); a storage unit 30 that stores data Gs of a reference image of the working unit 40 and data Ks of the posture of the working unit 40 of the working machine 100 in association with each other; an image information acquiring unit 12 that acquires data Gj of the image of the working unit 40 to compare with data Gs of the reference image; and a posture estimation unit 20e that estimates the posture of the working unit 40 of the work machine 100 based on the image data Gj and the storage information of the storage unit 30.

The construction machine 1000 may be a machine that performs construction work by moving the bucket 46 attached to the arm mechanism 48, for example. Instead of the bucket, various attachment members such as a fork, a hammer, and a crusher may be attached to the arm mechanism 48 of the construction machine 1000. The configuration of the third embodiment provides the same effects as those of the first embodiment.

[ fourth embodiment ]

A fourth embodiment of the present invention is a computer program P100. The computer program P100 is for causing a computer to realize the following functions: storing data Gs of a reference image of the working unit 40 of the working machine 100 and data Ks of the posture of the working unit 40 of the working machine 100 in the storage unit 30 in association with each other; acquiring data Gj of an image of the working unit 40 of the working machine 100 to compare with data Gs of a reference image; and estimating the posture of the working unit 40 of the working machine 100 based on the data Gj of the image and the storage information of the storage unit.

The computer program P100 may be an application program in which a plurality of modules corresponding to the functional blocks of the control unit 20 are installed, and may be installed in a storage device (for example, the storage unit 30) of the information processing system 10. The computer program P100 may be read out to a main memory of a processor (e.g., CPU) of the information processing system 10 and executed. The configuration of the fourth embodiment provides the same operational advantages as those of the first embodiment.

The embodiments of the present invention have been described in detail. The above embodiments are merely specific examples for carrying out the present invention. The contents of the embodiments are not intended to limit the technical scope of the present invention, and various design changes such as changes, additions, deletions, and the like of the components can be made without departing from the scope of the inventive concept defined in the claims. In the above-described embodiments, the description will be given by giving a notation such as "in the embodiments" or "in the embodiments" regarding the content that can be subjected to such a design change, but it is not permissible to subject the content that is not subjected to such a notation to a design change.

[ modified examples ]

Next, a modified example will be explained. In the drawings and the description of the modified examples, the same or equivalent constituent elements and members as those of the embodiment are denoted by the same reference numerals. The description overlapping with the embodiment is appropriately omitted, and the description will focus on the configuration different from the first embodiment.

In the first embodiment, the following example is shown: the posture estimation model is generated by each work machine 100 and stored in the model storage unit 32 of the work machine 100, but the present invention is not limited to this. The posture estimation model may be generated by a reference work machine and stored in advance in the model storage unit 32 of each work machine 100. In addition, the posture estimation model may be updated at an appropriate timing.

In the first embodiment, the following example is shown: the color information removing unit 20n completely removes the color information from the data Gs of the reference image and the data Gj of the image, but the present invention is not limited to this. The color information removal unit 20n may compress the color information of the reference image data Gs and the image data Gj by color reduction or the like.

In the description of the first embodiment, the image information acquiring unit 12 is configured by one image sensor, but the present invention is not limited to this. The image information acquiring unit 12 may be configured by a plurality of image sensors. For example, the image information acquiring unit 12 may include a so-called stereo camera.

In the description of the first embodiment, the image information acquiring unit 12 is provided on the ceiling of the control cabin 38, but the present invention is not limited thereto. For example, the image information acquiring unit 12 may be disposed on a side surface of the cabin 38 or a cover of the upper body portion 34. Further, the image information acquiring unit 12 may be disposed in the working unit 40.

In the description of the first embodiment, the working machine 100 is an example of a construction machine that performs construction work by moving the bucket 46, but the present invention is not limited to this, and can be applied to working machines other than construction machines.

In the description of the first embodiment, the arm mechanism 48 is provided on the right side of the cage 38, but the present invention is not limited to this. For example, the arm mechanism may be provided on the left side of the cage or in front of the cage.

In the description of the first embodiment, the example in which the operator operates the work machine 100 from the control cabin 38 is shown, but the present invention is not limited to this. For example, the work machine may be a machine that is automatically operated or remotely operated.

The modification described above has the same operation and effect as those of the first embodiment.

Any combination of the above-described embodiment and the modification is also useful as an embodiment of the present invention. The new embodiment resulting from the combination has both the effects of the combined embodiment and the modified example.