阅读说明：本技术 优化机器的有效载荷载体的装载 (Optimizing payload carrier loading of a machine ) 是由 C·哈 S·N·马修 刘阳 Q·王 于 2020-01-02 设计创作，主要内容包括：一种装载机器的有效载荷载体的方法,包括当材料被装载到有效载荷载体中时,从机器上的相机接收有效载荷载体内部的二维图像。该方法还包括过滤图像以识别装载的材料的轮廓并确定该轮廓的区域。该方法还包括控制显示设备指示所确定的区域。(A method of loading a payload carrier of a machine includes receiving a two-dimensional image of an interior of the payload carrier from a camera on the machine as material is loaded into the payload carrier. The method also includes filtering the image to identify a contour of the loaded material and determining an area of the contour. The method also includes controlling a display device to indicate the determined region.)

1. A method for loading of a payload carrier of a machine, the method comprising:

receiving a two-dimensional image of an interior of the payload carrier from a camera on the machine as material is loaded into the payload carrier;

filtering the image to identify contours of the loaded material;

determining a region of the contour; and

controlling a display device to indicate the determined area.

2. The method of claim 1, further comprising:

determining that the region is equal to or greater than a threshold; and

based at least in part on determining that the area is equal to or greater than the threshold, sending a signal to a controller associated with the machine, the signal indicating that the payload carrier has been loaded.

3. The method of claim 2, wherein the threshold is a percentage of a total area of the payload carrier.

4. The method of claim 2, further comprising performing, by the controller in response to the signal, one or more of:

activating an indicator light in an operator station of the machine to indicate a stop loading the payload carrier;

providing a visual indication on a display device of the machine to indicate that loading of the payload carrier is stopped;

raising the payload carrier from the ground by the machine;

causing the machine to move a flap from an open position to a closed position to prevent further loading of the payload carrier;

slowing or stopping the machine; and

causing the machine to change a current operating mode of the machine to an autonomous haulage mode.

5. The method of claim 1, wherein determining a region comprises counting a number of pixels within the contour.

6. The method of claim 1, wherein filtering the image comprises:

identifying pixels corresponding to features in the image;

calculating a motion vector for the identified pixel; and

pixels corresponding to motion vectors having magnitudes greater than a threshold are removed.

7. The method of claim 1, further comprising removing distortion caused by a wide angle lens from the image prior to determining the region.

8. A camera for assisting loading of a payload carrier of a machine, the camera comprising:

a memory to store instructions; and

a processor configured to execute the instructions to:

receiving a two-dimensional image of an interior of the payload carrier when material is loaded into the payload carrier;

filtering the image to identify contours of the loaded material;

determining a region of the contour; and

providing a signal to control a display device to indicate the determined area.

9. The camera of claim 8, wherein the processor is further configured to execute the instructions to:

determining that the region is equal to or greater than a threshold; and

based at least in part on determining that the area is equal to or greater than the threshold, providing a signal to a controller associated with the machine, the signal indicating that the payload carrier has been loaded.

10. The camera of claim 9, wherein the threshold is a percentage of a total area of the payload carrier.

11. The camera of claim 8, wherein to determine the region of the contour, the one or more processors are further configured to execute the instructions to count a number of pixels within the contour.

12. The camera of claim 8, wherein to filter the image, the one or more processors are further configured to execute the instructions to:

identifying pixels corresponding to features in the image;

calculating a motion vector for the identified pixel; and

pixels corresponding to motion vectors having magnitudes greater than a threshold are removed.

13. The camera of claim 8, wherein the one or more processors are further configured to execute the instructions to remove distortion caused by a wide angle lens from the image prior to determining the region.

14. A machine, comprising:

a display device;

a camera configured to capture a two-dimensional image of an interior of a payload carrier of the machine when material is loaded into the payload carrier; and

a controller configured to:

receiving the image from the camera;

filtering the image to identify contours of the loaded material;

determining a region of the contour; and

providing a signal to control the display device to indicate the determined region.

15. The machine of claim 14, wherein the controller is further configured to:

determining that the region is equal to or greater than a threshold; and

based at least in part on determining that the region is equal to or greater than the threshold, providing a signal indicating that the payload carrier has been loaded.

16. The machine of claim 15, wherein the threshold is a percentage of a total area of the payload carrier.

17. The machine of claim 15, further comprising an indicator light in an operator station of the machine, the indicator light configured to activate in response to receiving the signal.

18. The machine of claim 15, wherein the display device is configured to display an indication to stop loading the payload carrier in response to receiving the signal.

19. The machine of claim 15, further comprising an actuator controller configured to control one or more actuators to raise the payload carrier from the ground in response to receiving the signal.

20. The machine of claim 15, wherein the machine is a wheeled tractor-type scraper.

Technical Field

The present invention relates to machine production optimization, and more particularly, to production optimization for operation of an excavator, such as a wheeled tractor scraper.

Background

Earth moving machines may be used to move earth, rock and other materials from an excavation site. Often, it may be desirable to move excavated material from an excavation site to another location remote from the excavation site. For example, the material may be loaded onto an off-road hauling unit that may transport the material to a dumping site. As another example, material may be dug by pulling on a tractor-trailer tray behind the tractor and then hauled to a dump site via the tractor tray. As another example, a wheeled tractor scraper may be used to dig, haul, and dump excavated material.

One such machine, a wheeled tractor scraper, may be used in an operating cycle to cut material from one location during a loading phase, transport the cut material to another location during a hauling phase, unload the cut material during a dumping phase, and return to an excavation site during a return phase to repeat the operating cycle. In contrast to some other excavators or systems, the decision to use a wheeled tractor scraper may be based on factors such as the operating cost and productivity of the machine or system.

The productivity and cost of operating a machine or fleet of machines may be adversely affected by certain factors. For example, an operator of a wheeled tractor scraper may spend too much time in a loading cycle relative to the time required to complete a hauling cycle, thereby reducing efficiency. Further, utilizing an exceptionally long loading cycle to fully load or possibly overload a machine may be effective in terms of the actual productivity and cost of certain haul cycles, but for other haul cycles, productivity may be reduced and costs increased by, for example, increasing tire slip (increasing tire wear), burning more fuel, increasing wear on ground engaging tools, and increasing wear on machine structures and powertrain components.

To improve the efficiency of earth-moving machines, including during the loading phase, systems have been devised. For example, U.S. patent application publication No. 2016/0289927 to Wang et al ("the' 927 publication") describes a bowl monitoring system having a sensory sensor that provides a signal to a controller indicative of a view of a bowl of a machine. Based on the signal, the controller determines a material level in the bowl and provides an indication of a current loading status of the bowl to an operator of the machine.

While the system described in the' 927 publication helps to improve loading efficiency, the system employs three-dimensional sensing sensors such as L iDAR (light detection and ranging) or stereo cameras to monitor the bowl.

The present invention is directed to one or more improvements in the prior art.

Disclosure of Invention

One aspect of the invention relates to a method for loading of a payload carrier of a machine. The method may include receiving a two-dimensional image of an interior of the payload carrier from a camera on the machine while the material is loaded into the payload carrier. The method may include filtering the image to identify a contour of the loaded material and determining an area of the contour. The method may include controlling a display device to indicate the determined region.

Another aspect of the invention relates to a camera for assisting loading of a payload carrier of a machine. The camera may include a memory storing instructions and a processor. The processor may be configured to execute the instructions to receive a two-dimensional image of an interior of a payload carrier when material is loaded into the payload carrier, and to filter the image to identify a contour of the loaded material. The processor may be further configured to execute the instructions to determine an area of the outline and provide a signal to control a display device to indicate the determined area.

Another aspect of the invention relates to a machine. The machine may have a display device and a camera configured to capture a two-dimensional image of an interior of a payload carrier of the machine when material is loaded into the payload carrier. The machine may also have a controller configured to receive the image from the camera and filter the image to identify the contour of the loaded material. The controller may be further configured to determine an area of the contour and provide a signal to control the display device to indicate the determined area.

Drawings

FIG. 1 is a diagrammatic illustration of a machine according to an exemplary disclosed embodiment;

FIG. 2 is a graph of a load growth curve of the machine of FIG. 1;

FIG. 3 is a schematic illustration of an exemplary control system of the machine of FIG. 1.

FIG. 4 is a schematic diagram of an exemplary camera associated with the control system of FIG. 3.

Fig. 5 and 6 are representations of images captured by the camera of fig. 4.

FIG. 7 is a representation of an exemplary interface displayed on a display device of a machine.

FIG. 8 is a flow diagram of an exemplary method for optimizing loading of a payload carrier of a machine.

FIG. 9 is a flow chart of exemplary filtering steps of the method of FIG. 8.

Detailed Description

Fig. 1 schematically illustrates a machine 100, which may be, for example, a wheeled tractor-type scraper. The machine 100 may be any machine for performing a work in the field using a ground engaging tool. The machine 100 may include various components or sub-machines, such as a wheeled tractor scraper, a tractor tray, and the like.

The machine 100 may include one or more traction devices 102, such as front and rear wheels, to enable the machine 100 to function as a mobile unit. A suitable power source 104, such as a diesel engine, may be located at a front portion 106 of the machine 100. An additional power source 108, which may also be a diesel engine, may be included at the rear 110 of the machine 100.

The payload carrier 112 between the front 106 and rear 110 of the machine 100 may enable the machine 100 to transport a quantity of material, such as soil (soil, rock, etc.). On a wheeled tractor scraper, payload carrier 112 may be a container that holds and holds material for transport, and may sometimes be referred to as a scoop or bowl.

The machine 100 may also include an operator station 114. The operator station 114 may include an enclosed or partially enclosed operator compartment and may include an operator seat 116, suitable operator control devices 118, a display device 120, and/or other components for operating the machine 100.

The machine 100 may also include a suitable control system including a controller 122, various detectors or sensors, and various actuators for operating components of the machine 100. For example, machine 100 may include one or more actuators 124, such as hydraulic cylinders, for raising and lowering payload carrier 112. The actuator 124 may lower the payload carrier 112 such that the ground engaging tools 126, which are generally located at the lower front edge of the payload carrier 112, may penetrate the material to be loaded during the loading stage of the machine 100. The actuator 124 may also raise the payload carrier 112 for transporting the payload during a hauling phase of the machine 100. Additional actuators may include one or more actuators 128 and one or more actuators 132, with the actuator 128 being used to move the ejector 130 during the dumping phase and the actuator 132 being used to control the flap 134.

The actuator 132 can move the flap 134 from engagement with the front of the payload carrier 112 to an open position during a loading and dumping phase. The actuator 132, by reversing movement, may also hold the flap 134 in a closed position engaging the front of the payload carrier 112 during the hauling phase.

The flap 134 may operate in synchronization with the ejector 130 during the dumping phase, with the actuator 132 moving the flap 134 to the open position, and the actuator 128 moving the ejector 130 within the payload carrier 112 to assist in dumping the payload.

Steering of the machine 100 may be facilitated by a steering unit that includes one or more actuators 136 located, for example, at a location between the payload carrier 112 and the front 106 of the machine 100.

As shown in fig. 1, in some embodiments, a load assist unit 138 may optionally be associated with payload carrier 112. The exemplary loading assist unit 138 shown in fig. 1 represents various types of loading assist units that may be employed, including, for example, an auger unit or an elevator unit. In fig. 1, the loading aid unit 138 is shown as an auger 140. It should be understood that loading assist unit 138 may include a plurality of augers, elevator units, or other means that may assist in loading material into payload carrier 112. The load assist unit 138 may be driven by a suitable machine actuator, such as a rotary hydraulic actuator 152.

The machine 100 may include other components to assist an operator in loading and dumping the payload carrier 112 and/or autonomously control the machine 100 to do so. In the disclosed embodiment, camera 154 may be positioned to view the interior of payload carrier 112 to enable determination of the amount of material accumulated in payload carrier 112. For example, the camera 154 may be mounted on a portion of the payload carrier 112, such as on a mast or pole, to generate a view of the material entering the payload carrier and accumulating therein. In one embodiment, the camera 154 may be a two-dimensional camera, such as a class B (bridge digital) camera.

The machine 100 (e.g., a wheeled tractor scraper) to which the present invention is applied may operate in a number of cycles that may include loading, hauling, dumping and return phases. In a given earth or material handling operation, such as that performed by a wheeled tractor scraper, the machine cycle of operation may be affected by various parameters and/or factors, which may be referred to as cycle characteristics. By optimizing the machine payload, consideration of cycle characteristics during machine operation may enable enhancement, optimization and/or maximization of machine productivity, as well as control of operating costs.

Cycle characteristics may include, for example, the length of the haul phase of the cycle, the grade over which the machine is to travel, the characteristics of the ground over which the machine must travel, the characteristics of the machine (i.e., machine size and loading pattern), the type of material loaded, and the speed of the machine relative to the payload.

Another cyclic characteristic that may be considered in connection with a wheeled tractor scraper is the load growth curve. The load growth curve is a graphical representation of the increase in payload volume during machine loading. For a wheeled tractor scraper, the load growth curve may generally indicate that a majority of the payload volume is loaded early in the loading phase of the operating cycle, with the increase in payload gradually decreasing late in the loading phase.

Fig. 2 illustrates an exemplary load growth curve 200 for a machine 100 (e.g., a wheeled tractor scraper). The payload is represented on the y-axis and can typically be measured in units of real cubic code (BCY). The loading time may be measured on the x-axis, for example, time in minutes and/or fractions thereof.

The load growth curve 200 may exhibit a fairly steep portion 202 during an initial phase of loading and may flatten out as the loading phase progresses, exhibiting a less steep portion 204. Thus, a majority of the payload volume may accumulate in the portion corresponding to the loading phase of steep portion 202, followed by a gradual decrease in the increase in payload in the portion corresponding to less steep portion 204. This characteristic shape of the load growth curve can be attributed to the fact that: as payload carrier 112 receives more and more material, it may be necessary for later loaded material to lift it or force it through previously loaded material.

As shown in fig. 2, the load increase curve 200 reflects an actual stop time 206 and an optimal stop time 208. The actual stop time 206 (in this example, about 1.2 minutes) may correspond to a time at which an operator actually typically actually stops loading the payload carrier 112 with material.

The optimal stop time 208 (in this case 0.6 minutes) may correspond to the optimal time that loading should stop to maintain efficient and effective machine operation. The optimal stop time 208 may correspond to the point on the load increase curve 200 where the steep portion 202 transitions more steeply to the less steep portion 204. In the example of fig. 2, the optimal stop time 208 is approximately 0.6 minutes. Thus, in this example, the operator may typically continue loading for approximately 0.6 minutes after the optimal stop time 208 — half of the total load time. Although payload carrier 112 accumulates approximately 18BCY in the first 0.6 minutes of loading, it only accumulates additional 3BCY during an additional loading time 210 of 0.6 minutes from the optimal stop time 208 to the actual stop time 206. This makes the additional load time 210 inefficiently use resources, including fuel and time.

It should be appreciated that machines like wheel tractor scrapers may have different load growth curves depending on, for example, the size of the machine, whether the machine is self-loading, whether the machine is push-pull, whether the machine has the convenience of increasing loading (e.g., an elevator or auger), the type of material being loaded (e.g., clay, sand, gravel, a mixture of rock and earth, etc.), and/or the size and shape of the payload carrier 112. The load growth curve for a given machine operating in a given set of circumstances may be determined empirically prior to actual production operation of the given machine. This can be achieved by, for example, testing operations and prior field experience.

The controller 122 may include a central processing unit, appropriate memory components, various input/output peripherals, and other components typically associated with a machine controller. The controller 122 may include programs, algorithms, data maps, etc., associated with the operation of the machine 100. Controller 122 may be configured to receive information from a number of sources, such as actuators 124, 128, 132, and 136, camera 154, various sensors or detectors (e.g., for machine direction of travel, ground speed, engine operation, etc.), and one or more of inputs from a machine operator via, for example, control device 118. Controller 122 may be suitably positioned to send and receive appropriate signals to and from various sensors, actuators, etc. associated with machine 100. In one embodiment, as shown in FIG. 1, the controller 122 can be conveniently located within the operator station 114 or adjacent to the operator station 114. For example, the controller 122 may comprise a laptop or mobile computer of an operator. Alternatively, the controller 122 may include a dedicated Electronic Control Module (ECM) of the machine 100 or other type of on-board computer. In some embodiments, aspects of the controller 122 may be incorporated into the camera 154 such that the camera 154 is a "smart camera" configured to perform the disclosed operations of the controller 122. In this case, the controller 122 or some aspect thereof may be eliminated.

Fig. 3 schematically illustrates an exemplary control system 300 associated with the controller 122. The controller 122 may suitably communicate with the various machine components, for example via wires. The operator control device 118 and the display device 120 may enable an operator to manually provide signals to the controller 122. Display device 120 may, for example, provide various information to an operator to enhance the operator's awareness of various machine systems to help maintain efficient and effective machine operation. The controller 122 may receive data input 302 from a variety of sources, including a keyboard, a touch screen display (which may be associated with the display device 120, for example), a computer storage device, an internet repository, a wireless network, or other data input sources known to those skilled in the art.

The controller 122 may also communicate with various machine actuators 304 via a machine actuator module 306. The machine actuator module 306 may be configured to operate, for example, the lift actuator 124, the visor actuator 132, the ejector actuator 128, the bail actuator, the steering actuator 136, the load assist actuator 152, or any other actuator associated with the machine 100.

The controller 122 may be further configured to communicate with the speed control module 308 to control the travel speed of the machine 100. The speed control module 306 may include, for example, engine speed control means, throttle control, transmission shift control, and the like.

The controller 122 may be further configured to communicate with the autonomous control module 310. The autonomous control module 310 may control the machine 100 to perform various tasks without any operator input, or with only a certain amount of operator input. For example, autonomous control module 310 may be configured to operate machine 100 in: a loading mode for performing a loading phase at a certain loading position; a hauling mode for performing a hauling phase of hauling loaded material from a loading location to a specific dumping location; a dumping mode for performing a dumping phase of dumping material at a dumping location; and/or a return mode for returning the machine 100 to the loading position. In response to signals from controller 122, autonomous control module 310 may control machine 100 to execute cycles of loading, hauling, dumping, and return phases.

The controller 122 may receive input data relating to cycle characteristics, for example, on an ongoing basis. This may enable relatively continuous updating of the calculated optimal payload of the machine 100. For example, consistent with the disclosed embodiments, the controller 122 may receive data from the camera 154. In some embodiments, the controller 122 may employ other components (not shown), such as an odometer, an inclinometer, a wheel slide sensor, another payload sensor (e.g., a scale), and/or various other sensors, detectors, diagnostic devices, and so forth. The controller 122 may use the data received from these components to collect data related to cycle characteristics and control the operation of the machine 100.

Consistent with the disclosed embodiments, controller 122 may be configured to receive data from camera 154 indicating whether payload carrier 112 has been optimally filled with material according to load growth curve 200. In response to the data, the controller 122 may be configured to provide signals to one or more components of the machine 100, such as the operator control device 118, the display device 120, the machine actuator module 306, the speed control module 308, or the autonomous control module 310.

For example, in response to receiving a signal from camera 154 indicating that payload carrier 112 is optimally filled, controller 122 may provide a signal to operator control device 118. The controller 122 can provide a signal to change the load indicator light in the operator station 114 from green (continue loading) to red (stop loading) to indicate to the operator that the payload carrier 112 is optimally filled.

Alternatively or additionally, the controller 122 may be configured to provide a signal to the display device 120 indicating that the payload carrier 112 is optimally filled. The display device 120 may in turn provide a visual indication on the display to let the operator know that the payload carrier 112 is optimally filled.

Alternatively or additionally, the controller 122 may be configured to provide a signal to the machine actuator module 306 indicating that the payload carrier 112 is optimally filled. The machine actuator module 306, in turn, may provide signals to actuate one or more actuators. For example, machine 306 may provide one or more signals to: (1) lift actuator 124 to raise payload carrier 112; (2) a flap actuator 132 to move a flap 134 from an open position to a closed position engaging the front of the payload carrier 112; (3) an ejector actuator 128 to move an ejector 130 within the payload carrier 112 in order to dump the payload or to stow the ejector 130 for a hauling phase; (4) a bail actuator 312 to manipulate the bail at the front 106 of the machine 100; (5) steering the actuator 136 to change the angle between the front 106 and rear 110 sections of the machine 100; or (6) load assist actuator 152 to stow the load assist unit for the haul phase.

Alternatively or additionally, the controller 122 may be configured to provide a signal to the speed control module 308 indicating that the payload carrier 112 is optimally filled. In response to the signal, the speed control module 308 may be configured to reduce the speed of the machine 100 or stop the machine 100, reduce the speed of the throttle or power source 104, 108, and so forth.

Alternatively or additionally, the controller 122 may be configured to provide a signal to the autonomous control module 310 indicating that the payload carrier 112 is optimally filled. In response to the signal, the autonomous control module 310 may, for example, change the current operating mode of the machine 100 from a loading mode to a hauling mode, or perform other functions to complete the loading phase.

Fig. 4 shows an exemplary schematic of the camera 154. The camera 154 may have a computing component for a digital camera. For example, the camera 154 may have a memory 400, a data store 402, a communication unit 404, a lens unit 406, a processor 408 configured to execute a payload optimization algorithm 410.

Memory 400 may include temporary data storage such as RAM. The data storage 402 may include permanent storage such as ROM, flash, solid state, or other types of data storage known in the art.

The communication unit 404 may be configured to communicate with external components (e.g., the controller 122). The communication unit 404 may include, for example, USB, firewire, Bluetooth, Wi-Fi, CAN bus, Ethernet, or other electronic communication interfaces known in the art for interconnecting computing devices. Under the command of the processor 408, the communication unit 404 may intermittently or continuously send data signals to the controller, including signals indicating whether the payload carrier 112 has been determined to be optimally loaded. In some embodiments, the communication unit 404 may also stream the live video data to the controller 122 for display or processing.

For example, lens unit 406 may embody a digital single lens reflex (DS L R) system that includes a lens (e.g., a 35mm lens) and a two-dimensional image sensor (e.g., a CCD or CMOS image sensor). it should be appreciated that lens unit 406 may be the same type of lens unit used in a conventional digital camera or smart phone.

Lens unit 406 may output an image to processor 408 in the form of a continuous or intermittent data stream that includes color values for each pixel of a two-dimensional image sensor (e.g., a color filter array). Thus, the data stream may comprise two-dimensional image information. The camera 154 may be positioned such that the lens unit 406 views the interior of the payload carrier 112 without obstruction and the output data stream thus comprises two-dimensional image information of the interior of the payload carrier 112.

Processor 408 may be an image processor known in the art for use with digital cameras. For example, the processor 408 may be a Digital Signal Processor (DSP) configured to perform various types of imaging processing, such as Bayer transformation, demosaicing, noise reduction, imaging sharpening, edge detection, or coordinate system transformation.

The payload optimization algorithm 410 may include computer program instructions installed on the processor 408 and/or stored in the memory 400 or storage 402 for execution by the processor 408. A payload optimization algorithm 410 executed by processor 408 may be configured to process the two-dimensional digital image data received from lens unit 406 to determine when payload carrier 112 is optimally loaded. To this end, algorithm 410 may transform the two-dimensional data received from lens unit 406 from a first coordinate system associated with camera 154 to a second reference coordinate system for determining the amount of material in payload carrier 112.

FIG. 5 shows an image sequence 500-504 that illustrates how the algorithm 410 may transform the two-dimensional image data from the lens unit 406. It should be appreciated that the camera 154 may need to be placed so as not to interfere with the operation of the machine 100, but still have a view of the interior of the payload carrier 112. For example, the payload carrier 112 may have a generally rectangular shape, and the camera 154 may be located on a mast at the right rear of the payload carrier 112. Thus, camera 154 may have a view looking diagonally across payload carrier 112 and above payload carrier 112. Thus, the camera 154 may produce an image 500 of the payload carrier 112 similar to that shown in fig. 5.

Additionally, in some embodiments, the camera 154 may have a fisheye lens or other wide angle lens to capture the entire payload carrier 112 from a closed position. Thus, in the image 500 generated by the camera 154, the top edge 506 of the payload carrier 112 may define an axis relative to a real distorted coordinate system 508(X ", Y"). For example, coordinate system 508 may bulge outward from the "barrel" shaped distortion caused by the wide angle lens. It should be appreciated that the pixel location (X ", Y") of the image 500 within the coordinate system 508 may therefore be visually distorted relative to reality.

Using image distortion correction techniques and specifications of camera 154, algorithm 410 may be configured to correct image 500 to remove distortion, resulting in an undistorted image 502. For example, prior to mounting the camera 154 to the machine 100, the camera 154 may be calibrated using an open source "checkerboard" technique that outputs calibration parameters required to produce a rectangular checkerboard of predetermined dimensions from the distorted image 500. Algorithm 410 may be configured to apply calibration parameters to image 500 to produce image 502.

In undistorted image 502, top edge 506 of payload carrier 112 may define undistorted coordinate system 510(X ', Y'), where the X 'and Y' axes extend along straight lines. Algorithm 410 may be configured to translate a pixel location (X ", Y") in coordinate system 508 to its corresponding location (X ', Y') in coordinate system 510.

As shown in fig. 5, in image 510, payload carrier 112 appears to be rotated approximately 45 degrees from horizontal. In addition, because the view from image 502 spans across payload carrier 112 from a diagonal and remains above payload carrier 112, the X 'and Y' axes intersect each other at an obtuse angle (i.e., greater than 90 degrees).

To generate the image 504 from which the amount of material in the payload carrier 112 can be determined, the algorithm 410 can be configured to rotate and translate the pixel locations (X ', Y') in the coordinate system 510 to corresponding locations (X, Y) in the working coordinate system 512. In coordinate system 512, the X-axis and Y-axis intersect each other at right angles (i.e., 90 degrees), as in a standard Cartesian coordinate system. To accomplish this, the algorithm 410 may use a rotation and translation transform technique. For example, the algorithm 410 may be configured to apply a perspective transformation (e.g., an OpenCV algorithm) such that corners of the payload carrier 402 form a rectangle, transforming the image 502 into a top-down image 504. Image 504 may correspond to a top view of payload carrier 112 from vertically above it, i.e., looking directly down at payload carrier 112 from above.

When payload carrier 112 may be filled with material during loading, sequence image 600-604 of payload carrier 112 is shown in fig. 6 from top-down perspective image 504. In image 600, loading of payload carrier 112 has just begun, and thus payload carrier 112 may include only a small amount of material 606. Material 606 may define a contour 608 surrounding an area 610 around its edge or perimeter. It should be appreciated that, due to the top-down perspective of image 504, region 610 may be a vertical cross-sectional area of material 606 (i.e., an area of the "footprint" of material 606) when material 606 is positioned in payload carrier 112.

As payload carrier 112 is further filled with material 606, contours 608 may expand, thereby enclosing larger and larger areas 610. Image 602 shows payload carrier 112 when partially filled (e.g., 50%) with material 606, and image 604 shows payload carrier 112 when optimally filled (e.g., 85%) with material 606. In this description, "optimally" filled may refer to a payload volume corresponding to a desired optimal stop time 208 on the load growth curve 200, where the steep portion 202 transitions to the less steep portion 204.

The optimal volume and corresponding desired optimal stop time 208 may be determined in different ways. For example, they may be determined empirically through field testing of the machine 100. Alternatively, an operator or other personnel having knowledge of the performance characteristics of the machine 100 may select the optimal volume/stop time 208 based on experience. The optimal volume/stop time 208 may also be determined mathematically by selecting the point on the load increase curve 200 at which the slope of the load increase curve 200 reaches some desired threshold (e.g., 30%).

The region 610 of material 606 may generally correspond to the actual volume of material 606 in the payload carrier 112. For example, certain materials are known to be stationary at a certain angle of repose. Thus, if material 606 has a particular region 610, material 606 may have a particular corresponding height. Likewise, when region 610 expands by a known amount, it may be assumed that the height of material 606 also grows by a corresponding known amount. This may allow the relative volume of material to be calculated based on the cross-sectional area 610. Additionally, the actual value of the volume of material may be calculated from the relative volume based on the known dimensions of payload carrier 112. Thus, in the disclosed embodiment, the region 610 of material 608 in the payload carrier 112 may be used as a substitute for or in place of a volume.

In the example shown in fig. 6, it may be assumed that region 610 of material 606 shown in image 504 corresponds to an optimal loading volume (e.g., 85%) of payload carrier 112 based on load growth curve 200. Thus, based on the above discussion, it may be desirable to stop loading payload carrier 112 when region 610 reaches the threshold region shown in image 504.

The algorithm 410 may be configured to use image processing techniques to calculate a region 610 enclosed by the outline 608. In one embodiment, the algorithm 410 may be configured to apply a feature detection algorithm (e.g., an OpenCV algorithm) that may output pixel values (X, Y) of features in the image 504. For example, an acute angle (e.g., >60 degrees) may be identified as a feature, such as a corner of payload carrier 112 or a corner of material 606.

For example, the algorithm 410 may apply the L ucas-Kanade optical flow technique implemented in OpenCV, using the pixel values (X, Y) of the detected features, the current image 504, and the previous image 504 as inputs, the algorithm 410 may be configured to provide motion vectors corresponding to the pixel values (X, Y) included by each detected feature, as an output, the motion vectors may include a motion magnitude and a motion direction of the feature between the two images 504.

The algorithm 410 may be configured to remove pixel values (X, Y) corresponding to features having motion vectors with magnitudes above a threshold. For example, the algorithm 410 may be configured to determine an average magnitude of the motion vectors and a standard deviation of the motion vectors. The algorithm 410 may then remove features having pixel values (X, Y) with motion vectors having magnitudes greater than some threshold (e.g., two standard deviations) average. As explained below, after this process, only the coordinates (X, Y) corresponding to the material 606 may be retained.

Algorithm 410 may be configured to determine a region 610 of outline 608. For example, algorithm 410 may be configured to count the number of pixels remaining after filtering, which is an area 610 measured in square pixels. Algorithm 410 may also be configured to determine a percentage fill factor for payload carrier 112. The area of the payload carrier 112 may be the total number of pixels in the four corners of the payload carrier 112, which may be predetermined and/or fixed. To determine the percentage fill factor, algorithm 410 may be configured to divide the number of pixels within counted outline 608 by the total number of pixels within the four corners of payload carrier 112.

Algorithm 410 may be configured to determine region 610 of outline 608 using other techniques, if desired. For example, the algorithm 410 may apply edge detection to identify the pixel value (X, Y) corresponding to the contour 608. In one embodiment, algorithm 410 may be configured to identify pixels in image 504 where the color value transitions from the color of material 606 (e.g., brown, black, or dark gray in a grayscale image) to the color of payload carrier 200 (e.g., yellow or light gray in a grayscale image). The algorithm 410 may treat these identified pixels as contours 608. Next, the algorithm 410 may be configured to identify all pixels in the image 504 that fall within the outline 608. The algorithm 410 may be configured to count the total number of pixels that make up the outline 608 and are within the area enclosed by the outline 608. Algorithm 410 may be determined to determine the percentage fill factor by dividing the total number of pixels of outline 608 by the total number of pixels in the four corners of payload carrier 112.

As shown in fig. 7, the algorithm 410 may be configured to provide a signal to the controller 122 to control the display device 120 to display the payload bearer status interface 700. The interface 700 may have one or more user interface elements that allow an operator to provide or control information regarding the status of the payload carrier 112. For example, the interface 700 may include a video feed window 702 that displays a live video feed from the camera 154 corresponding to the image 500. The interface 700 may also include a payload growth curve 704 showing the current loading state of the payload carrier 112.

Interface 700 may further have a fill factor indicator 706 that indicates the percent fill factor of payload carrier 112, i.e., the percentage of payload carrier 112 that is filled with material 606. The interface 700 may also be configured to provide notification that the payload carrier 112 is best populated upon receiving a corresponding signal from the camera 154. For example, the interface 700 may be configured to display a "stop loading" message upon receiving such a signal so that the operator knows to stop the current loading stage.

Interface 700 may also include an option 708 to set a top corner 710 and a bottom corner 712 of payload carrier 112. For example, upon selecting option 708, the operator or technician may set the corners 710, 712 using a mouse, a touch screen of display device 120, or other user input device. Once set, the algorithm 410 may be configured to convert the image 500 to the image 502 using the angles 710, 712, as described above.

Fig. 8 illustrates an exemplary method 800 for optimizing loading of a payload carrier 112 during machine operation. The method 800 may be performed by the algorithm 410 when executed by the processor 408. The steps of method 800 need not be performed in the order shown in fig. 8, and may be performed in a different order consistent with the disclosed embodiments.

In step 802, the algorithm 410 may receive a live video feed corresponding to the image 500 from the lens unit 406. The video feed may be displayed in a window 702 of the interface 700.

In step 803, algorithm 410 may remove distortion from image 500 in the feed as described above. For example, in embodiments using wide-angle lenses, algorithm 410 may remove barrel distortion. Thus, for example, step 803 may convert pixel values (X ", Y") in coordinate system 508 to corresponding pixel values (X ', Y') in coordinate system 510.

In step 804, algorithm 410 may transform image 500 from coordinate system 510 to coordinate system 512 of image 504, as discussed above with reference to fig. 5. For example, the previous operator may use option 708 to select top corner 710 and bottom corner 712 of payload carrier 112. The algorithm 410 may use the pixel values (X ', Y') of the corners 710 and/or 712 of the payload carrier 112 to convert all the pixel values (X ', Y') in the coordinate system 510 into their corresponding values (X, Y) in the coordinate system 512, as explained.

In step 806, the algorithm 410 may filter the transformed image 504 to remove pixel values corresponding to features other than the material 606, as described above. Fig. 9 shows a method for exemplary step 806.

In step 900, the algorithm 410 may detect features in the image 504. For example, as described above, the algorithm 410 may apply a feature detection process (e.g., an OpenCV process). The feature detection process may output pixel values (X, Y) of any identified features in image 504, such as corners of payload carrier 112 and edges or corners of material 606 in payload carrier 112.

In step 902, the algorithm 410 may calculate motion vectors for the features identified in step 900. for example, as described above, the algorithm 410 may apply motion detection techniques such as the L ucas-Kanade optical flow technique Using the pixel values (X, Y) of the features detected in step 900, the inputs of the previous image 504 and the current image 504 as inputs, the algorithm 410 may calculate a motion vector for each detected feature.

In step 904, the algorithm 410 may calculate an average magnitude (e.g., in pixels) of the motion vector calculated in step 902. Algorithm 410 may additionally calculate a standard deviation (e.g., in pixels) of the magnitude.

In step 906, the algorithm 410 may determine whether the magnitude of each vector calculated in step 902 is greater than a threshold (e.g., 12 pixels). In one embodiment, the threshold may be the average magnitude calculated in step 904 plus a certain number X of standard deviations. This is because X-2 standard deviation may provide a threshold to distinguish the feature corresponding to material 606 from other moving features. For example, when payload carrier 112 is filled with material 606 and outline 608 expands, the pixel values (X, Y) corresponding to material 606 may be shifted by a particular amount from one image 504 to the next image 504. Accordingly, the threshold may be selected such that various magnitudes of material 606 movement generally between images 504 fall within the threshold. In the example of X-2 above, it can be known that a feature having a motion vector whose magnitude is less than or equal to the average magnitude plus two standard deviations corresponds to a moving or stationary material 606. On the other hand, a feature whose motion vector is known to have a magnitude that is more than two standard deviations greater than the mean magnitude moves too fast to be material 606. For example, they may be ambient surroundings, shadows, transient obstructions to the view of camera 154, or other features other than material 606.

If the result of step 906 is negative, then it has been determined that the pixel values corresponding to the feature correspond to material 606. Thus, in step 908, the pixel values may be retained for region calculation. For example, the algorithm 410 may store the pixel values in the memory 400 of the camera 154 for further processing.

If the result of step 906 is yes, then it has been determined that the pixel values corresponding to the feature do not correspond to material 606. Thus, in step 910, the pixel values may be discarded and not used for region calculation.

Returning to FIG. 8, in step 808, the algorithm 410 may determine an area 610 of the outline 608. For example, the algorithm 410 may count the number of pixels remaining in step 908 and thus remaining after the filtering of fig. 9. The number of pixels may correspond to the area 610 in a square pixel. Additionally, algorithm 410 may determine the percentage fill factor by dividing the counted number of pixels by the total number of pixels in the four corners of payload carrier 112 in image 504, which may be predetermined and/or fixed. The algorithm 410 may provide a signal to the controller 122 such that the controller 122 controls the display device 120 to indicate the calculated fill factor on the indicator 706 within the interface 700.

In step 810, the algorithm 410 may determine whether the region 610 determined in step 808 is equal to or greater than a threshold. For example, the algorithm 410 may determine whether the fill factor determined in step 808 is greater than or equal to a threshold percentage (e.g., 85%). As described above, the threshold percentage may be predetermined to correspond to a desired optimal loading volume (e.g., 85%) of payload carriers 112 relative to the total loading capacity.

If the result of step 810 is "no," the algorithm 410 may return to step 802 and the algorithm may repeat step 802 and 812 until the region 610 reaches a threshold, meaning that the payload carrier 112 has been loaded to the optimal volume.

If the result of step 810 is "yes," then in step 812, the algorithm 410 may notify the controller 122 that the payload carrier 122 has been determined to be optimally loaded. For example, as described above, the processor 408 may send a signal to the controller 122 via the communication unit 404 indicating that the payload carrier 122 has been optimally loaded.

As described above, the controller 122 may take one or more actions based on the notification, including, for example, any combination of:

provide a signal to change the load indicator light in the operator station 114 from green (continue loading) to red (stop loading) so that the operator knows that the payload carrier 112 is optimally filled. The operator may then manually control the machine 100 to stop loading the payload carrier.

Provide a signal to the display device 120 indicating that the payload carrier 112 is optimally filled. The display device 120 may in turn provide a visual indication (e.g., a "stop loading" message) on the display to let the operator know that the payload carrier 112 is optimally filled. The operator may then manually control the machine 100 to stop loading the payload carrier.

Provide a signal to the machine actuator module 306 indicating that the payload carrier 112 is optimally filled. The machine actuator module 306 may, in turn, provide signals to actuate the actuators to complete the loading phase. For example, machine 306 may provide one or more signals to: (1) lift actuator 124 to raise payload carrier 112; (2) a flap actuator 132 to move a flap 134 from an open position to a closed position engaging the front of the payload carrier 112; (3) an ejector actuator 128 to move an ejector 130 within the payload carrier 112 in order to dump the payload or to stow the ejector 130 for a hauling phase; (4) a bail actuator 312 to manipulate the bail at the front 106 of the machine 100; (5) steering the actuator 136 to change the angle between the front 106 and rear 110 sections of the machine 100; or (6) load assist actuator 152 to stow the load assist unit for the haul phase.

Provide a signal to the speed control module 308 indicating that the payload carrier 112 is optimally filled. In response to the signal, the speed control module 308 may be configured to end the loading phase by reducing the speed of the machine 100, stopping the machine 100, reducing the speed of the throttle or power source 104, 108, and so forth.

Provide a signal to the autonomous control module 310 indicating that the payload carrier 112 is optimally filled. In response to the signal, the autonomous control module 310 may, for example, change the current operating mode of the machine 100 from a loading mode to a hauling mode, or perform other functions to complete the loading phase.

Industrial applicability

The disclosed embodiments may be applied to work machines, such as wheeled tractor scrapers, which may operate in a cycle that includes loading, hauling, dumping and return phases. It is beneficial to complete these cycles as efficiently as possible by eliminating wasted time and resources (e.g., man-hours and fuel). Efficiency can be improved by taking into account cycle characteristics-one of which is the load growth curve of the machine discussed herein.

In particular, time and resources may be saved by considering the load growth curve 200 of the machine 100. By ensuring that the operator does not continue to load more than the optimal volume of payload carrier 112 for each loading cycle, more cycles can be completed in less time using less fuel.

The disclosed embodiments may provide a relatively inexpensive yet effective technique to notify an operator to stop loading or autonomously control the machine 100 to do so once an optimal fill volume is reached, although three-dimensional imaging systems such as L iDAR or stereo cameras may be utilized to determine payload volume, such systems are expensive.

Instead, the disclosed embodiments may instead employ an inexpensive two-dimensional camera 154, such as the type used on conventional mass-produced smart phones or digital cameras. The volume may not be calculated directly from the two-dimensional image, but the disclosed embodiments may use the region 610 of the payload material 606 instead to indirectly determine when the optimal payload volume has been reached. This advantageously allows the use of a relatively inexpensive two-dimensional camera rather than a more expensive three-dimensional camera. In addition, two-dimensional image processing typically requires less computational resources than three-dimensional image processing. Accordingly, the disclosed embodiments may reduce costs by requiring less computing resources.

Additionally, the disclosed embodiments may enable an operator to better focus on safely driving the machine. By providing a notification (e.g., an indicator light or an indication on a display) when the payload carrier is optimally filled, the operator may not need to turn to view the payload carrier to determine whether it is full as in the case of a conventional wheeled tractor scraper.

It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed payload overload control system without departing from the scope of the invention. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. For example, the present invention may describe embodiments in which the camera 154 is "smart" and configured to execute the algorithm 410. This may allow the camera 154 to be provided as a stand-alone unit (e.g., equipment) for retrofitting older machines that otherwise do not have the disclosed payload optimization functionality. However, the machine may also be equipped with such functionality as a standard or optional feature. For example, the present invention also includes the use of a generic camera in place of a "smart" camera. In such embodiments, one or more functions of the camera 154, including one or more functions of the algorithm 410, may be embedded in the controller 122 rather than the camera, and the camera need only capture images of the payload carrier and provide them to the controller. It is intended, therefore, that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.