阅读说明：本技术 远程操作系统以及远程操作服务器 (Remote operation system and remote operation server ) 是由 大谷真辉 佐佐木均 佐伯诚司 山崎洋一郎 于 2019-12-02 设计创作，主要内容包括：本发明从使一位操作员掌握其他的操作员进行的作业机械的远程操作方式的观点出发,提供一种可以向该一位操作员提供适宜的信息的系统。在第1远程操作装置(10)中,利用视线检测器(112)检测第1操作员的视线,发送与该第1操作员的视线对应的视线检测数据。在第2远程操作装置(20)中,使指定图像区域以不同于其周围的图像区域的方式显示于第2图像输出装置(221),所述指定图像区域是在与由作业机械(40)的拍摄装置(401)获取的拍摄图像数据对应的环境图像中以第1操作员的视线为基准而扩展的图像区域。(The present invention provides a system that can provide appropriate information to one operator from the viewpoint of enabling the one operator to grasp a remote operation mode of a work machine by another operator. In a 1 st remote operation device (10), a line-of-sight detector (112) detects the line of sight of a 1 st operator, and line-of-sight detection data corresponding to the line of sight of the 1 st operator is transmitted. In the 2 nd remote operation device (20), a specified image area, which is an image area that extends with the 1 st operator's line of sight as a reference in an environment image corresponding to captured image data acquired by a capturing device (401) of a work machine (40), is displayed on the 2 nd image output device (221) so as to be different from the image area around the specified image area.)

1. A remote operation system is provided with: a working machine having a working mechanism and an imaging device for imaging an environment including at least a part of the working mechanism; a 1 st remote operation device and a 2 nd remote operation device having a wireless communicator, an image output device that displays an environment image corresponding to captured image data acquired by the imaging device of the work machine, and an operation mechanism for remotely operating the work machine,

it is characterized in that the preparation method is characterized in that,

the 1 st remote operation device includes: a sight line detector that detects a sight line of an operator; a 1 st client control device that transmits line-of-sight detection data corresponding to the line of sight of the operator detected by the line-of-sight detector to the wireless communicator,

the 2 nd remote operation device includes a 2 nd client control device configured to display a specified image area, which is an image area that extends with reference to the line of sight of the operator corresponding to the line of sight detection data received by the wireless communicator in the environment image, on the image output device so as to be different from an image area around the specified image area.

2. A remote operation server having a function of communicating with a work machine having a work mechanism and an imaging device for imaging an environment including at least a part of the work mechanism, a 1 st remote operation device and a 2 nd remote operation device, respectively, the 1 st remote operation device and the 2 nd remote operation device having an image output device for displaying an environment image corresponding to captured image data acquired by the imaging device of the work machine using a wireless communication function and an operation mechanism for remotely operating the work machine,

it is characterized in that the preparation method is characterized in that,

a 1 st server arithmetic processing element configured to receive, from the 1 st remote operation device, gaze detection data corresponding to a gaze of an operator detected in the 1 st remote operation device;

and a 2 nd server arithmetic processing element configured to transmit the line-of-sight detection data to the 2 nd remote operation device, thereby causing a designated image area to be displayed on the image output device of the 2 nd remote operation device so as to be different from an image area around the designated image area, the designated image area being an image area that extends in the environment image with reference to the line of sight of the operator corresponding to the line-of-sight detection data.

Technical Field

The present invention relates to a system for remotely operating a work machine or the like.

Background

A technique for remotely operating a work machine has been proposed (for example, see patent document 1).

Documents of the prior art

Patent document

Patent document 1 Japanese patent laid-open publication No. 2016-076801

Disclosure of Invention

Technical problem to be solved by the invention

However, from the viewpoint of improving the skill of one operator who is poor in the skill of remote operation of the working machine, it is preferable to make the operator grasp the remote operation mode of the working machine by another operator who is excellent in the skill.

Therefore, an object of the present invention is to provide a system that can provide appropriate information to one operator from the viewpoint of making the one operator grasp a remote operation mode of a work machine by another operator.

Solution for solving the above technical problem

The present invention relates to a remote operation system, comprising: a working machine having a working mechanism and an imaging device for imaging an environment including at least a part of the working mechanism; the 1 st remote control device and the 2 nd remote control device each include a wireless communicator, an image output device that displays an environment image corresponding to captured image data acquired by the imaging device of the work machine, and an operation mechanism for remotely operating the work machine.

In the remote operation system according to the present invention, the 1 st remote operation device includes: a sight line detector that detects a sight line of an operator; and a 1 st client control device that transmits line-of-sight detection data corresponding to the line of sight of the operator detected by the line-of-sight detector to the wireless communicator, wherein the 2 nd remote operation device includes a 2 nd client control device that causes a designated image area to be displayed on the image output device so as to be different from an image area around the designated image area, the designated image area being an image area that extends in the environment image with reference to the line of sight of the operator corresponding to the line-of-sight detection data received by the wireless communicator.

The present invention relates to a remote operation server having a function of communicating with a work machine having an operating mechanism and an imaging device that images an environment including at least a part of the operating mechanism, a 1 st remote operation device and a 2 nd remote operation device, respectively, wherein the 1 st remote operation device and the 2 nd remote operation device have an image output device that displays an environment image corresponding to captured image data acquired by the imaging device of the work machine using a wireless communication function, and an operating mechanism for remotely operating the work machine.

The remote operation server according to the present invention is characterized by comprising a 1 st server arithmetic processing element configured to receive, from the 1 st remote operation device, line-of-sight detection data corresponding to a line of sight of an operator detected in the 1 st remote operation device, and a 2 nd server arithmetic processing element configured to transmit the line-of-sight detection data to the 2 nd remote operation device, thereby causing a designated image area to be displayed on the image output device of the 2 nd remote device so as to be different from an image area around the designated image area, the designated image area being an image area that extends in the environment image with reference to the line of sight of the operator corresponding to the line-of-sight detection data.

Drawings

Fig. 1 is an explanatory diagram of a configuration of a remote operation system according to an embodiment of the present invention.

Fig. 2 is an explanatory diagram of the structure of the working machine.

Fig. 3 is an explanatory diagram of the configuration of the 1 st remote operation device.

Fig. 4 is an explanatory diagram of functions of a remote operation system according to an embodiment of the present invention.

Fig. 5 is an explanatory diagram of an image output method in the 1 st image output apparatus.

Fig. 6A is an explanatory diagram of the 1 st embodiment of image output in the 2 nd image output apparatus.

Fig. 6B is an explanatory diagram of the 2 nd mode for image output in the 2 nd image output apparatus.

Fig. 6C is an explanatory diagram of the 3 rd embodiment of the image output in the 2 nd image output apparatus.

Fig. 6D is an explanatory diagram of the 4 th mode of image output in the 2 nd image output apparatus.

Fig. 7 is an explanatory diagram of a remote operation system according to another embodiment of the present invention.

Detailed Description

(constitution)

The remote operation system shown in fig. 1 as an embodiment of the present invention includes a 1 st remote operation device 10, a 2 nd remote operation device 20, and a work machine 40. The common remote operation body of the work machine 40 can be switched between the 1 st remote operation device 10 and the 2 nd remote operation device 20.

(construction of work machine)

The work machine 40 includes a work machine control device 400, an imaging device 401, a wireless communication device 402, and an operating mechanism 440. The work machine control device 400 is configured by an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing in accordance with the software while targeting the data.

The working machine 40 is, for example, a crawler excavator (construction machine), and as shown in fig. 2, includes a crawler-type lower traveling structure 410 and an upper revolving structure 420 that is rotatably mounted on the lower traveling structure 410 via a revolving mechanism 430. A cab (driver's cab) 422 is provided in a front left portion of the upper revolving structure 420. A work attachment 440 is provided in a front center portion of the upper revolving structure 220.

The work attachment 440 as the work mechanism includes: a boom 441 attached to the upper revolving structure 420 so as to be able to rise and fall; an arm 443 rotatably coupled to the distal end of the boom 441; bucket 445 is rotatably coupled to the tip of arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are telescopic hydraulic cylinders, are attached to the work attachment 440.

Boom cylinder 442 is interposed between boom 441 and upper revolving unit 420, and receives a supply of hydraulic oil to extend and contract, thereby rotating boom 441 in the raising and lowering direction. Arm cylinder 444 is interposed between arm 443 and boom 441 so as to receive the supply of hydraulic oil and expand and contract, thereby rotating arm 443 about a horizontal axis with respect to boom 441. A bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and contract by receiving a supply of hydraulic oil, and the bucket 445 is rotated about a horizontal axis with respect to the arm 443.

The imaging device 401 is provided inside the cab 422, for example, and images an environment including at least a part of the operating mechanism 440 through a front window of the cab 422.

The cab 422 includes: an actual machine side operation lever corresponding to an operation lever (described later) constituting the 1 st remote operation device 10; the drive mechanism or the robot receives a signal corresponding to the operation mode of each operation lever from the remote control room, and operates the real machine operation lever based on the received signal.

(constitution of the 1 st remote control device)

The 1 st remote device 10 includes a 1 st client control device 100, a 1 st input interface 110, and a 1 st output interface 120. The 1 st client control device 100 is configured by an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing for the data in accordance with the software. The 1 st input interface 110 includes a 1 st operating mechanism 111 and a line-of-sight detector 112. The 1 st output interface 120 includes a 1 st image output device 121 and a 1 st wireless communication device 122.

The 1 st operating mechanism 111 includes a travel operating device, a swing operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that receives a rotational operation. An operation lever (travel lever) of the travel operation device is operated to operate the lower traveling structure 410. The travel bar may also double as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. An operation lever (turning lever) of the turning operation device is operated to operate a hydraulic turning motor constituting the turning mechanism 430. An operation lever (boom lever) of the boom operation device is operated to operate the boom cylinder 442. The operation lever (arm lever) of the arm operation device is operated to operate the arm cylinder 444. An operation lever (bucket lever) of the bucket operation device is operated to operate the bucket cylinder 446.

For example, as shown in fig. 3, the respective operation levers constituting the 1 st operation mechanism 111 are arranged around a seat 1100 on which an operator sits. The seat 1100 may be a high back seat with an armrest, a low back seat without a headrest, or a chair without a back, which is any type that an operator can sit on.

A pair of left and right travel levers 110 corresponding to the left and right crawler belts are arranged in parallel in the left and right in front of the seat 1100. One operation lever can double as a plurality of operation levers. For example, the right side operation lever 1111 provided in front of the right side frame of the seat 1100 shown in fig. 3 may function as a boom lever when operated in the front-rear direction and may function as a bucket lever when operated in the left-right direction. Similarly, left operation lever 1112 provided in front of the left frame of seat 1100 shown in fig. 3 may function as an arm lever when operated in the front-rear direction and as a pivot lever when operated in the left-right direction. The lever mode can be arbitrarily changed in accordance with an operation instruction of the operator.

For example, as shown in fig. 3, the 1 st image output device 121 is composed of a right oblique front image output device 1211, a front image output device 1212, and a left oblique front image output device 1213 that are respectively disposed in the right oblique front, the front, and the left oblique front of the seat 1100. The image output devices 1211 to 1213 may further include speakers (voice output devices).

The line-of-sight detector 112 detects the line of sight of the operator seated in the seat 1100 based on the position of the moving point (moving part) with respect to the reference point (non-moving part) of the operator's eye. In the case where the position of the "eye corner" is determined as a reference point and the position of the "iris" is determined as a moving point, the visual line detector 112 is constituted by one or more visual light cameras. When the position of the "corneal reflection" is determined as the reference point and the position of the "pupil" is determined as the moving point, the line-of-sight detector 112 is configured by one or more sets of an infrared LED and an infrared camera. In addition to detecting the movement of the eyes of the operator, the movement of the head of the operator may be detected, and the position and posture of the line-of-sight detector 112 may be changed to the position and posture optimal for capturing the line of sight of the operator. In this case, the operator may switch to one of the plurality of sight-line detectors 112 that is most suitable for capturing the sight line of the operator.

In the captured image acquired by the line-of-sight detector 112, pixel regions corresponding to the reference point and the moving point of the eye of the 1 st operator are determined. By determining the actual spatial position corresponding to the pixel region, a vector (a start point position and an end point position) representing the line of sight of the operator in the actual space is determined. In order to determine the actual spatial position of the eye, a distance measuring sensor can additionally be used. In a plane representing the display image coordinate system of each of the image output devices 1211 to 1213 in the real space, the intersection with the vector is determined as the center of the region of interest in the display image toward which the line of sight of the operator is directed.

(constitution of No. 2 remote control device)

The 2 nd remote operation device 20 includes a 2 nd client control device 200, a 2 nd input interface 210, and a 2 nd output interface 220. The 2 nd client control device 200 is configured by an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing in accordance with the software with the data as an object. The 2 nd input interface 210 includes a 2 nd operation mechanism 211. The 2 nd output interface 220 includes a 2 nd image output device 221 and a 2 nd wireless communication device 222.

The detailed configuration of the 2 nd remote operation device 20 is substantially the same as that of the 1 st remote operation device 10 except that the line-of-sight detector 112 is omitted and the functions of the 2 nd client control device 200 described later are omitted, and therefore, the description thereof is omitted (see fig. 3).

(function)

In the 1 st remote operation device 10, the 1 st operator performs a predetermined operation (fig. 4/step 102). The predetermined operation is, for example, an operation of a button or an operation lever constituting the 1 st input interface 110 or the 1 st operation mechanism 111. Accordingly, the 1 st client control device 100 transmits an operation start instruction to the work machine 40 from the 1 st remote operation device 10 via the 1 st wireless communication device 122 (fig. 4/step 104).

Similarly, in the 2 nd remote operation device 20, the 2 nd operator performs a predetermined operation (fig. 4/step 202). The predetermined operation is, for example, an operation of a button or an operation lever constituting the 2 nd input interface 210 or the 2 nd operation mechanism 211. Accordingly, the 2 nd client control device 200 transmits an operation start instruction to the work machine 40 from the 2 nd remote operation device 20 via the 2 nd wireless communication device 222 (fig. 4/step 204).

In the work machine 40, the work machine control device 400 receives an operation start command via the wireless communication device 402 (fig. 4/step 402). Accordingly, the work machine control device 400 outputs an instruction to the imaging device 401, and the imaging device 401 acquires the captured image according to the instruction (fig. 4/step 404). The work machine control device 400 transmits captured image data representing the captured image to the 1 st remote operation device 10 and the 2 nd remote operation device 20 via the wireless communication device 402 (fig. 4/step 406).

In the 1 st remote operation device 10, the 1 st client control device 100 receives the captured image data via the 1 st wireless communication device 122 (fig. 4/step 106). The 1 st client control apparatus 100 displays an environment image (all or a part of the captured image itself, or a simulated environment image generated based on the captured image) corresponding to the captured image data on the 1 st image output apparatus 121 (fig. 4/step 108). Similarly, in the 2 nd remote operation device 20, the captured image data is received by the 2 nd client control device 200 via the 2 nd wireless communication device 122 (fig. 4/step 206). The 2 nd client control device 200 displays the environment image corresponding to the captured image data on the 2 nd image output device 211 (fig. 4/step 208). Thus, for example, as shown in fig. 5, an environment image including a boom 441, an arm 443, a bucket 445, and an arm cylinder 444, which are part of the work attachment 440 as a working mechanism, is displayed on the 1 st image output device 121 and the 2 nd image output device 221, respectively.

In the 1 st remote operation device 10, the 1 st operator operates the 1 st operation mechanism 111 (fig. 4/step 110), and accordingly, the 1 st client control device 100 transmits an operation command corresponding to the operation mode to the work machine 40 via the 1 st wireless communication device 122 (fig. 4/step 112).

In work machine 40, an operation command is received by work machine control device 400 via wireless communication device 402 (fig. 4/step 408). Accordingly, the work attachment 440 and the like are controlled in operation by the work machine control device 400 (fig. 4/step 410). For example, the following operations are performed: the earth in front of the work machine 40 is scooped up by the bucket 445, and the upper revolving structure 410 is revolved to drop the earth from the bucket 445.

In the 1 st remote operation device 10, the 1 st operator's line of sight is detected by the line of sight detector 112 (fig. 4/step 114). Thereby, the region of interest of the 1 st operator in the environment image displayed on the 1 st image output device 111 is specified. The 1 st client control device 100 transmits the line-of-sight detection data corresponding to the line-of-sight detection result to the 2 nd remote operation device 20 via the 1 st wireless communication device 122 (fig. 4/step 116).

In the 2 nd remote operation device 20, the 2 nd client control device 200 receives the line-of-sight detection data by the 2 nd wireless communication device 222 (fig. 4/step 210). The 2 nd client control device 200 causes the image area specified by the line-of-sight detection data to be displayed more prominently than the other image areas on the 2 nd image output device 211 (fig. 4/step 212). For example, when the attention area of the 1 st operator's line of sight or the environment image corresponds to the image area of the bucket 445, the image area is highlighted as the "designated image area" to be more conspicuous than the surrounding image area.

As shown in fig. 6A, a graphic such as a pointer (pointer) can be superimposed on the designated image region S including the image region corresponding to the bucket 445. As shown in fig. 6B, a figure such as a circle surrounding the designated image area S corresponding to the bucket 445 may be displayed. As shown in fig. 6C, the image quality of the rectangular designated image area S including the image area corresponding to the bucket 445 may be higher than the image quality of the other image areas. This makes it possible to display the designated image region S in a color image and display other image regions in a grayscale image. Further, the designated image area S may be displayed with a higher resolution than the other image areas. As shown in fig. 6D, the luminance of the rectangular designated image area S including the image area corresponding to the bucket 445 may be made higher than the luminance of the other image areas.

(Effect)

According to the remote operation system having this configuration, from the viewpoint of allowing the 2 nd operator who operates the work machine 40 by the 2 nd remote operation device 20 to grasp the object focused by the 1 st operator, the 2 nd operator can be provided with a designated image area highlighted from the surrounding image area in the environment image displayed on the 2 nd image output device 221 as appropriate information, as a remote operation method of the work machine 40 by the 1 st operator who operates the work machine 40 by the 1 st remote operation device 10 (see fig. 4/step 212 and fig. 6A to 6D). The 2 nd operator can recognize the operation method or operation mode of the operation lever by the 1 st operator by visually recognizing the highlighted designated image area in the environment image displayed on the 2 nd image output device 221.

(Another embodiment of the present invention)

In the above embodiment, the 1 st remote operation device 10, the 2 nd remote operation device 20, and the work machine 40 directly communicate with each other by wireless communication, but as another embodiment, the 1 st remote operation device 10, the 2 nd remote operation device 20, and the work machine 40 may indirectly communicate with each other via the remote operation server 30 shown in fig. 7.

The remote operation server 30 includes a 1 st server arithmetic processing element 31 and a 2 nd server arithmetic processing element 32. The 1 st server arithmetic processing element 31 receives line-of-sight detection data corresponding to the line of sight of the operator detected in the 1 st remote operation device 10 from the 1 st remote operation device 10. The 2 nd server arithmetic processing element 32 causes the 2 nd image output device 221 to display the specified image area so as to be different from the image area around the specified image area by transmitting the line-of-sight detection data to the 2 nd remote operation device 20 (see fig. 6A to 6D).

Description of the reference numerals

10 st remote operation device

20 nd 2 remote operation device

30 remote operation server

31 st server arithmetic processing element

32 nd 2 nd server arithmetic processing element

40 working machine

100 st client control device

110 the 1 st input interface

111 st operating mechanism

112 line-of-sight detector

120 st output interface

121 st image output device

122 st wireless communication device

200 nd 2 client control device

210 nd 2 input interface

211 nd 2 nd operating mechanism

220 nd 2 output interface

221 nd image output device

222 nd radio communication machine

401 camera

402 radio communication machine

440 work attachment (work mechanism).