阅读说明：本技术 包括自动再循环处理工位的用于处理物体的系统和方法 (System and method for processing objects including an automated recycling processing station ) 是由 T·瓦格纳 K·埃亨 B·科恩 M·道森-哈格蒂 C·盖尔 V·欣奇 T·科勒舒卡 K 于 2018-10-30 设计创作，主要内容包括：公开了一种使用可编程运动设备处理物体的方法。该方法包括以下步骤：提供输入输送系统,通过该输入输送系统,可以将物体的输入箱提供给处理工位,该处理工位包括具有端部执行器的可编程运动设备；在处理工位处感测表示在输入输送系统的输入区域处的多个物体的身份的识别标记；使用端部执行器抓持获取的物体；使用可编程运动设备将获取的物体朝向识别的处理容器移动,该识别的处理容器与识别标记相关联,并且所述识别的处理容器在处理工位处被设置为多个处理容器中的一个；并提供与处理工位连通的输出输送系统,通过该输出输送系统可以提供容纳经处理的物体的处理容器。(A method of processing an object using a programmable motion device is disclosed. The method comprises the following steps: providing an input conveyor system by which input bins of objects can be provided to a processing station comprising a programmable motion device having an end effector; sensing, at a processing station, identifying indicia representative of identities of a plurality of objects at an input area of an input conveyor system; grasping the captured object using the end effector; moving the acquired object towards an identified processing receptacle using a programmable motion device, the identified processing receptacle being associated with an identification mark and the identified processing receptacle being disposed as one of a plurality of processing receptacles at a processing station; and an output conveyor system in communication with the processing stations, by which output conveyor system processing containers containing processed objects can be provided.)

1. A method of processing an object using a programmable motion device, the method comprising the steps of:

providing an input conveyor system by which input bins of objects can be provided to a processing station comprising a programmable motion device having an end effector;

sensing, at the processing station, identifying indicia representative of identities of a plurality of objects at an input area of the input conveyor system;

grasping the captured object using the end effector;

moving the acquired object towards an identified processing receptacle using the programmable motion device, the identified processing receptacle being associated with the identification indicia and the identified processing receptacle being set as one of a plurality of processing receptacles at the processing station; and

an output conveyor system is provided in communication with the processing stations, by which processing containers containing processed objects can be provided.

2. A method according to claim 1, characterized in that the processing stations comprise at least processing conveyors, one on either side of the programmable movement device, and at least three processing containers in each processing conveyor.

3. The method of claim 1, wherein the processing stations include an inner process conveyor and an outer process conveyor.

4. A method according to claim 3, further comprising the step of moving a process vessel from the inner process conveyor to the outer process conveyor.

5. A method according to claim 3, further comprising the step of moving a process vessel from the outer process conveyor to the inner process conveyor.

6. A method according to claim 1, characterised in that the processing stations comprise two groups of two processing conveyors each, each group of processing conveyors comprising an inner processing conveyor and an outer processing conveyor.

7. The method of claim 6 further comprising the step of moving a process vessel from one of the inner process conveyors to one of the outer process conveyors.

8. The method of claim 7 wherein the step of moving a processing container from one of the inner processing conveyors to one of the outer processing conveyors is accomplished by the programmable motion device.

9. The method of claim 6 further comprising the step of moving a process vessel from one of the outer process conveyors to one of the inner process conveyors.

10. The method of claim 9 wherein the step of moving a processing container from one of the outer processing conveyors to one of the inner processing conveyors is accomplished using a box kicker system.

11. A method of processing an object using a plurality of programmable motion devices, the method comprising the steps of:

providing an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for gripping objects;

sensing, at each processing station, identifying indicia representative of identities of a plurality of objects at an input area of the input conveyor system;

at each processing station, moving the acquired object towards an identified processing receptacle using the programmable motion device, the identified processing receptacle being associated with an identifying indicia and the identified processing receptacle being provided as one of a plurality of processing receptacles; and

an output conveyor system is provided in communication with each processing station, by which processing containers containing processed objects can be provided.

12. The method of claim 11, wherein the step of sensing the identity of the plurality of objects involves sensing a mark on one of the plurality of objects.

13. The method of claim 11, wherein the step of sensing the identity of the plurality of objects involves sensing indicia on an input bin containing the plurality of objects.

14. The method of claim 11, wherein the step of identifying the plurality of objects occurs while the plurality of objects are on the input conveyor system.

15. The method of claim 11, wherein the input area comprises a process station input in communication with the input conveyor system, and wherein the step of identifying the plurality of objects occurs while the plurality of objects are at the process station input.

16. The method of claim 11, wherein the plurality of objects are provided in one of a plurality of bins, and wherein different bins each include a plurality of different objects.

17. The method of claim 11, further comprising the steps of: determining a gripping position of an object to be acquired from the plurality of objects at the input region.

18. The method of claim 11, wherein the processing station is positioned between the input conveyor system and the output conveyor system.

19. The method of claim 11, wherein the input conveyor system is provided as a loop, and the processing station is located outside of the loop.

20. The method of claim 19, wherein the output conveyor system comprises at least two output conveyors positioned outside of the processing station.

21. The method of claim 11, wherein the output conveyor system is provided as a loop, and the processing station is located outside of the loop.

22. The method of claim 21, wherein the input conveyor system comprises at least two input conveyors positioned outside of the processing station.

23. The method of claim 11, further comprising the steps of: an identification dispatch is dynamically assigned for each processing container according to the inventory.

24. A method of processing an object using a plurality of programmable motion devices, the method comprising the steps of:

providing an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for gripping objects; and

providing an output conveyor system in communication with each processing station, through which processing containers containing processed objects can be provided, at least one of the input conveyor system and the output conveyor system being provided as a loop;

wherein at each processing station, the method further comprises the steps of:

sensing identifying indicia representative of identities of a plurality of objects at an input area of the input conveyor system; and

the acquired object is moved towards the identified processing receptacle using a programmable motion device.

25. The method of claim 24, wherein the step of sensing the identity of the plurality of objects involves sensing a mark on one of the plurality of objects.

26. The method of claim 24, wherein the step of sensing the identity of the plurality of objects involves sensing indicia on an input bin containing the plurality of objects.

27. The method of claim 24, wherein the step of identifying the plurality of objects occurs while the plurality of objects are on the input conveyor system.

28. The method of claim 24, wherein the input area comprises a process station input in communication with the input conveyor system, and wherein the step of identifying the plurality of objects occurs while the plurality of objects are at the process station input.

29. The method of claim 24, wherein the plurality of objects are provided in one of a plurality of bins, and wherein different bins each include a plurality of different objects.

30. The method of claim 24, further comprising the steps of: determining a gripping position of an object to be acquired from the plurality of objects at the input region.

31. The method of claim 24, wherein the processing station is positioned between the input conveyor system and the output conveyor system.

32. The method of claim 24, wherein the input conveyor system is provided as a loop, and the processing station is located outside of the loop.

33. The method of claim 32, wherein the output conveyor system comprises at least two output conveyors positioned outside of the processing station.

34. The method of claim 24, wherein the output conveyor system is provided as a loop, and the processing station is located outside of the loop.

35. The method of claim 34, wherein the input conveyor system comprises at least two input conveyors positioned outside of the processing station.

36. The method of claim 34, further comprising the steps of: an identification dispatch is dynamically assigned for each processing container according to the inventory.

37. A processing system for processing an object using a programmable motion device, the processing system comprising:

an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for gripping objects;

a sensing unit at each processing station for sensing identifying indicia representative of the identity of a plurality of objects at an input area of the input conveyor system;

a routing system at each processing station for causing an identified processing container to be directed to the programmable movement apparatus of the processing station such that an object can be placed in the identified processing container by an end effector of the programmable movement apparatus, the identified processing container being associated with the identifying indicia and the identified processing container being provided as one of a plurality of processing containers; and

an output conveyor system in communication with each processing station, by which processing containers containing processed objects can be provided.

38. The processing system of claim 37, wherein the input area comprises a process station input in communication with the input conveyor system, and wherein the sensing unit is positioned above the process station input.

39. A handling system according to claim 38, wherein the plurality of objects are provided in one of a plurality of bins, and wherein different bins each comprise a plurality of different objects.

40. The processing system of claim 37, wherein said processing station is positioned between said input conveyor system and said output conveyor system.

41. A processing system according to claim 37, wherein the input conveyor system is provided as a loop, the processing stations being located outside the loop.

42. The processing system of claim 41, wherein said output conveyor system comprises at least two output conveyors positioned outside said processing stations.

43. A processing system according to claim 37, wherein the output conveyor system is provided as a loop, the processing stations being located outside the loop.

44. The processing system of claim 43, wherein the input conveyor system comprises at least two input conveyors positioned outside the processing station.

45. A handling system according to claim 37, wherein the identified handling container is provided as a cassette tray assembly.

Background

The present invention relates generally to automated programmable motion control systems, such as robotic, sorting, and other handling systems, and in particular to programmable motion control systems intended for use in environments where various objects (e.g., articles, packages, parcels, etc.) need to be handled and moved to a number of processing destinations.

For example, many parcel dispensing systems receive parcels in irregular streams or bulk shipments, which may be provided as individual parcels, or in groups of aggregated parcels (e.g., in bags) that arrive on any of a number of different delivery vehicles, typically conveyors, trucks, pallets, garlords or boxes, and the like. Each package must then be assigned to the correct destination location (e.g., container) as determined by the identification information associated with the package, typically as determined by a label printed on the package. The destination location may take a variety of forms such as a bag, shelf, container or box.

For example, FIG. 1 shows an object dispensing system 10 in which objects arriving, for example, in a truck, as shown at 12, are separated and stored in packages, each package including a particular combination of objects as shown at 14, and then the packages are shipped to different retail stores as shown at 16, provided that each retail store receives the particular combination of objects in each package. Each package received from the conveyor 16 at a retail store is unpacked at the store, and such packages are commonly referred to as unpackages. In particular, the incoming truck 12 contains vendor boxes 18 of the same type of group object. For example, each vendor box may be provided by the manufacturer of each object. Items from the vendor box 18 are moved into a dump box 20 and then brought to the processing area 14 which includes unpacking storage packaging 22. At the processing area 14, the unpackaged storage packaging 22 is filled by a staff member who selects items from the dumped vendor bins to fill the unpackaged storage packaging according to a manifest. For example, a first set of unpackaged storage packages may go to a first store (as shown at 24), while a second set of unpackaged storage packages may go to a second store (as shown at 26). In this way, the system can receive a large number of products from a manufacturer and then repackage these objects as unpackages for supply to retail stores where a wide variety of objects are provided in a particular controlled distribution.

However, such systems have inherent inefficiencies and flexibility as the desired goal is to match incoming objects with the distributed collection bin. Such systems may in part require a large number of collection bins (and therefore a large amount of physical space, investment costs and operating costs), as sorting all objects to all destinations at once is not always most efficient. In addition, such unpacking systems must also monitor the volume of each similar object in the case, requiring personnel to continuously count the items in the case.

Furthermore, the current state-of-the-art sorting systems also rely to some extent on manual labor. Most solutions rely on the workers performing the sorting by scanning each object from an induction area (chute, table, etc.) and then placing each object at a buffer, conveyor or collection bin. When the box is full, another worker empties the box into a bag, box or other container, and then sends the container to the next processing step. Such systems are limited in throughput (i.e., how fast the worker can sort to or empty the bins in this manner) and the number of diverts (i.e., only so many bins can be arranged to be within effective reach of the worker for a given bin size).

Unfortunately, these systems do not address the limitation of the total number of boxes in the system. The system simply shunts an equal share of the total object to each parallel manual unit. Therefore, each parallel sorting unit must have all the same collection bin names; otherwise, the object may be delivered to a cell of the bin to which it is not mapped. Accordingly, there remains a need for a more efficient and cost effective object handling system that handles objects of various sizes and weights into appropriate collection bins or trays of fixed size while efficiently handling objects of various sizes and weights.

Disclosure of Invention

According to one embodiment, the present invention provides a method of processing an object using a programmable motion device. The method comprises the following steps: providing an input conveyor system by which input bins of objects can be provided to a processing station comprising a programmable motion device having an end effector; sensing, at a processing station, identifying indicia representative of identities of a plurality of objects at an input area of an input conveyor system; grasping the captured object using the end effector; moving the acquired object towards an identified processing receptacle using a programmable motion device, the identified processing receptacle being associated with an identification mark and the identified processing receptacle being disposed as one of a plurality of processing receptacles at a processing station; and an output conveyor system in communication with the processing stations, by which output conveyor system processing containers containing processed objects can be provided.

According to another embodiment, the invention provides a method of processing an object using a plurality of programmable motion devices. The method comprises the following steps: providing an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for grasping an object; sensing, at each processing station, identifying indicia representative of identities of a plurality of objects at an input area of an input conveyor system; at each processing station, the acquired object is moved using a programmable motion device towards an identified processing receptacle, which is associated with an identifying mark and which is arranged as one of a plurality of processing receptacles, and an output conveyor system is provided in communication with each processing station, by means of which output conveyor system the processing receptacles containing the processed objects can be provided.

According to yet another embodiment, the invention provides a method of processing an object using a plurality of programmable motion devices. The method comprises the following steps: providing an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for grasping an object; and providing an output conveyor system in communication with each processing station through which processing containers containing processed objects can be provided, at least one of the input conveyor system and the output conveyor system being provided as a loop. At each processing station, the method further comprises the steps of: sensing identification indicia representing identities of a plurality of objects at an input area of an input conveyor system; and moving the acquired object towards the identified processing receptacle using the programmable motion device.

According to another embodiment, the invention provides a processing system for processing an object using a programmable motion device. The processing system includes an input conveyor system by which input bins of objects can be provided to a plurality of processing stations, each processing station including a programmable motion device including an end effector for grasping an object; a sensing unit at each processing station for sensing identifying indicia representative of the identity of the plurality of objects at the input area of the input conveyor system; a routing system at each processing station for causing the identified processing container to be directed to the programmable movement apparatus of the processing station such that the object can be placed in the identified processing container by the end effector of the programmable movement apparatus, the identified processing container being associated with an identifying indicia and the identified processing container being provided as one of a plurality of processing containers; and an output conveyor system in communication with each processing station, by which processing containers containing processed objects can be provided.

Drawings

The following description may be further understood with reference to the accompanying drawings, in which:

FIG. 1 shows an illustrative schematic diagram of an object handling system according to the prior art;

FIG. 2 shows an illustrative schematic of an object handling system in accordance with an embodiment of the invention;

3A-3D show illustrative schematic diagrams of an input conveyor system in the system of FIG. 2;

FIG. 4 shows an illustrative schematic bottom side view of the system of FIG. 2;

FIG. 5 shows an illustrative schematic diagram of the sensing system of FIGS. 2-4;

FIG. 6 shows an illustrative schematic view of the sensing system from FIGS. 2-4 showing a view of an object within an object bin to be processed;

FIGS. 7A and 7B show illustrative diagrams of a grip selection process in an object handling system of an embodiment of the invention;

FIGS. 8A and 8B show illustrative diagrams of a gripping planning process in an object handling system of an embodiment of the invention;

FIGS. 9A and 9B show illustrative diagrams of a gripping execution process in the object handling system of the embodiment of the invention;

FIGS. 10A-10D show illustrative schematic views of the case handling system of FIGS. 2-4 with the cases moved to an external handling conveyor;

FIGS. 11A-11E show illustrative schematic views of another case handling system of FIGS. 2-4 in which cases are moved to an input handling conveyor;

FIG. 12 shows an illustrative schematic diagram of a processing system including a plurality of the object processing systems of FIG. 2 in accordance with an embodiment of the invention;

FIG. 13 shows an illustrative schematic exploded view of a cassette tray assembly used in accordance with an embodiment of the present invention;

FIG. 14 shows an illustrative schematic view of the cassette tray assembly of FIG. 13;

FIG. 15 shows an illustrative schematic of an object handling system in accordance with another embodiment of the invention;

FIG. 16 shows an illustrative schematic top view of the object handling system of FIG. 15;

17A and 17B show illustrative schematics of an input conveyor system in the system of FIG. 15;

FIG. 18 shows an illustrative schematic diagram of a processing system including a plurality of the object processing systems of FIG. 15 in accordance with an embodiment of the invention;

FIG. 19 shows an illustrative schematic of an object handling system in accordance with another embodiment of the invention;

FIG. 20 shows an illustrative schematic top view of the object handling system of FIG. 19;

FIG. 21 shows an illustrative schematic diagram of a processing system including a plurality of the object processing systems of FIG. 19 in accordance with an embodiment of the invention;

FIG. 22 shows an illustrative schematic of object assignment relationships in a conventional sorting system;

FIG. 23 shows an illustrative diagram of object assignment relationships, in accordance with certain embodiments of the invention;

FIG. 24 shows an illustrative schematic view of the object dispensing system of FIG. 23;

25A-25I show illustrative diagrams of object assignment steps in systems according to certain embodiments of the invention;

FIG. 26 shows an illustrative flow chart of a process according to an embodiment of the invention; and

FIG. 27 shows an illustrative flow diagram of an overall method of providing dynamic processing of an object.

The drawings are shown for illustrative purposes only.

Detailed Description

According to one embodiment, the invention provides a method of processing an object using a programmable motion device. The method comprises the following steps: sensing an identification mark representing an identity of the plurality of objects and directing the plurality of objects from the at least one input conveyor system towards the input area, acquiring objects from the plurality of objects at the input area using an end effector of the programmable motion device and moving the acquired objects towards an identified processing position using the programmable motion device, the identified processing position being associated with the identification mark and the identified processing position being provided as one of the plurality of processing positions.

According to various embodiments, the system of the present invention benefits from uniformity; the input boxes on the loop are homogeneous or the input boxes at the stations are homogeneous. In this way, the system does not have to wait for the correct box to arrive at the station. In the former case, the system knows exactly which outgoing boxes are needed for the workstation, since all objects in the boxes at the workstation are identical and need to be assigned to a subset of the outgoing boxes. Alternatively, the boxes at the station are outgoing and it is known exactly which boxes are to be pulled out of the loop, since they contain the objects required by the box. Thus, in various embodiments, the input bins may be cycled, or the processing receptacles may be cycled.

For example, fig. 2 illustrates a processing system 30 according to an embodiment of the invention, the processing system 30 including an incoming feed conveyor 34 (such as may be coupled to an Automated Storage and Retrieval System (ASRS)), on which incoming feed conveyor 34 source containers 32 may be provided to a processing station 30, as shown in fig. 3A and 4. The incoming feed conveyor 34 is disposed below the processed container conveyor 60, and as further shown in fig. 3B, the incoming feed conveyor 34 may include a diverter 37 that selectively diverts containers 32 onto an input area 35 of the incoming feed conveyor 34, as shown in fig. 3C. As shown in 3D, the tracks 36 on the conveyors of the input area 35 may also help to retain the source containers 32 on the conveyors of the input area and to redirect the containers 32 back onto the incoming feed conveyor 34.

As further shown in fig. 4, the input region 35 into the feed conveyor 34 positions the source container 32 in a position such that it is within the sensing region of the sensing unit 50 (mounted on the frame 38) and such that it is accessible by the programmable motion device 40. Each of the conveyors at the entry feed conveyor 34 and the input area 35 may be independently actuatable, allowing the source container 32 to remain in the input area 35 as long as it is needed for object processing, while the other storage containers 31, 33 continue continuously or intermittently along the entry feed conveyor 34.

Referring again to fig. 2, the processing station 30 includes an empty container conveyor 66 on which empty containers 65 can be selectively provided to the processing station via bidirectional rollers 67, 68 to one of the two inner conveyors 47, 57 adjacent the programmable movement apparatus 40. Outside of the inner conveyor (remote from the programmable motion device 40) are outer conveyors 48, 58 which communicate with a processed container conveyor 60 via bidirectional conveyors 62, 64.

The internal conveyors 47, 57 may receive empty containers (e.g., 65) from an empty container conveyor 66, and each processing station 30 is configured with a programmable motion device 40 to retrieve individual objects from the source containers 32 and provide the objects to one of a small number of active processing containers 41, 42, 43, 44, 45, 46, 51, 52, 53, 54, 55, 56. One set of processing vessels 41-43 is disposed on the inner conveyor 47, while the other set of processing vessels 51-53 is disposed on the inner conveyor 57. As shown in FIG. 2, another set of processing receptacles 44-46 are disposed on the outer conveyor 48, and another set of processing receptacles 54-56 are disposed on the outer conveyor 58.

During use, a first set of processing receptacles (41, 42, 43) and a second set of processing receptacles (51, 52, 53) are disposed on the inner conveyors 47, 57 and are accessible by the programmable movement device 40 (e.g., a robot). Objects from the source container 32 are dispensed to the processing containers (41, 42, 43, 51, 52, 53) as may be required according to the manifest. Two further sets of processing receptacles (44, 45, 46) and (54, 55, 56) are provided on the external conveyor and can be selectively moved to the internal conveyor by using receptacle kickers 49, 59 and one of a single or pair of rollers 69 as discussed further below. Thus, the system is arranged such that six processing containers are immediately accessible and six more can be easily brought to the robot 40. Although the processing station 30 can accept a wide range of source containers via the infeed conveyor 34, a small number of processing containers are processed in the vicinity of the robot, with an otherwise small number of processing containers being readily accessible, providing significant economies. The containers may be moved between the inner and outer conveyors until completed, and then moved to a processed container conveyor 60, for example, using bi-directional conveyors 62, 64. In certain embodiments, each individual roller on the inner and outer conveyors 47, 48, 57, 58 may be individually actuatable, thereby allowing one container to be moved on the conveyors without moving all containers on the conveyors.

Fig. 4 shows the bottom side of the processing station 30, where the incoming feed conveyor 34 (below the processed container conveyor 60) can be seen delivering the source containers 32 to an input area 35 accessible to the robot 40. In various embodiments, the source containers may be provided to each processing station in a variety of ways.

The container may be provided as a handbag, box tray assembly or any other type of device that can receive and hold items. In other embodiments, the boxes may be provided in a unified tray (to provide spacing and process uniformity), and may further include an open lid that may hold the boxes in an open position, and may also provide uniformity in processing by any of spacing, alignment, or marking.

It is assumed that the boxes of objects are marked in one or more places on their exterior with visually unique indicia, such as bar codes (e.g., providing UPC codes) or Radio Frequency Identification (RFID) tags or mailing labels, so that they can be sufficiently identified for processing with a scanner. The type of flag depends on the type of scanning system used, but may include 1D or 2D code symbols. A variety of symbols or marking methods may be employed. It is assumed that the type of scanner used is compatible with the marking method. The indicia encodes an identifying indicia (e.g., a string of symbols), such as by a bar code, RFID tag, mailing label, or other means, which is typically a string of letters and/or numbers. The symbol string uniquely associates the vendor box with a particular group of homogeneous objects.

The operation of the system described above is coordinated with a central control system 70 as shown in fig. 2-4, the central control system 70 being in communication (e.g., wirelessly) with the programmable motion device 40, the sensing unit 50, the conveyors 34, 47, 48, 57, 58, 60, 66, 69, all diverters (e.g., 37), the bi-directional conveyors 62, 64, 67, 68, and the vessel kickers 49, 59. The system determines a UPC associated with the vendor's lot and an outbound (outbend) destination for each object based on the symbolic string. The central control system 70 includes one or more workstations or Central Processing Units (CPUs). For example, the correspondence between UPCs or mailing labels and outbound destinations is maintained by a central control system in a database called a manifest. The central control system maintains inventory by communicating with a Warehouse Management System (WMS). The inventory provides the outbound destination for each warehousing object.

As discussed above with reference to fig. 2-4, the system of an embodiment includes a sensing system (e.g., 50) mounted above the bin of objects to be processed near the programmable motion device 40, looking down into the bin 32. For example, and as shown in fig. 5, the system 50 may include (on its bottom side) a camera 72, a depth sensor 74, and a light 76. A combination of 2D and 3D (depth) data is acquired. The depth sensor 74 may provide depth information that may be used with the camera image data to determine depth information related to various objects in the field of view. The lights 76 may be used to remove shadows and aid in identifying the edges of objects, and may be illuminated throughout use, or may be illuminated according to a desired sequence to aid in identifying objects. The system uses the images and various algorithms to generate a set of candidate gripping positions for the objects in the box, as discussed in more detail below.

Fig. 6 shows an image view from the sensing unit 50. The image view shows the box 32 in the input area (conveyor) and the box 32 contains objects 78, 80, 82, 84, and 86. In the present embodiment, the objects are homogeneous and intended to be distributed into different unpacking packages. Superimposed on the objects 78, 80, 82, 84, 86 (for illustrative purposes) are candidate gripping locations 79, 81, 83 and 85 of the objects. Note that while the candidate grip positions 79, 83, and 85 are shown as good grip positions, the grip position 81 is not shown as good because its associated object is at least partially under another object. The system may not even have attempted to identify a gripping location for the object 84 because the object 84 is too obstructed by other objects. Candidate grasping positions may be indicated using a 3D model of the robotic end effector placed in a position where the actual end effector will serve as a grasping position, as shown in fig. 11. The gripping locations may be considered good, for example, if they are near the center of mass of the object to provide greater stability during gripping and transport, and/or if they avoid places on the object where good vacuum tightness may not be obtained, such as closures, seams, etc.

If an object is not fully sensed by the detection system, the sensing system treats the object as two different objects and may suggest more than one candidate grip for the two different objects. If the system performs a grip at any of these poor gripping locations, it will either fail to acquire the object because the gripping point is poor without a vacuum seal occurring (e.g., on the right side), or will acquire the object at a gripping location that is far from the center of mass of the object (e.g., on the left side), thereby causing great instability during any attempted transport. Each of these results is undesirable.

If a poor grip position is experienced, the system may remember the position of the associated object. By identifying good and bad grip positions, a correlation is established between features in the 2D/3D image and the concept of good or bad grip positions. Using this data and these correlations as inputs to the machine learning algorithm, the system can ultimately learn where to best grip the object and where to avoid gripping the object for each image presented to it.

As shown in fig. 7A and 7B, the sensing system can also identify the flattest portion of the object in the generation of good grip position information. In particular, if the object includes a tubular end and a flat end, such as object 87, the system will identify the flatter end, as shown at 88 in FIG. 7B. In addition, the system may select the region of the object where the UPC code occurs because such codes are typically printed on a relatively flat portion of the object to facilitate scanning of the bar code.

Fig. 8A and 8B show that for each object 90, 92, the grip selection system can determine a direction normal to a selected flat portion of the object 90, 92. As shown in fig. 9A and 9B, the robotic system will then direct the end effector 94 to approach each object 90, 92 from a direction normal to the surface to better facilitate producing a good grip on each object. By approaching each object from a direction substantially normal to the surface of the object, the robotic system significantly increases the likelihood of obtaining a good grip of the object, especially when employing a vacuum end effector.

Thus, the present invention provides in some embodiments that the grip optimization may be based on the determination of the surface normal, i.e., moving the end effector to be normal to the sensing surface of the object (as opposed to vertical or gantry pickup), and that such a grip point may be selected as a grip point using a reference feature, such as pickup on a bar code if a given bar code is almost always applied to a flat point on the object.

Thus, according to various embodiments, the present invention further provides a processing system that can learn an object holding position from experience (and optionally manual guidance). The system is designed to work in the same environment as the various objects, poses, etc. that the worker will face. This diversity almost ensures that the robotic system will encounter some object or object configurations that it cannot optimally manipulate; at such times, it is desirable to enable the operator to assist the system and learn from the non-optimal grip.

The system optimizes the grip point based on a wide variety of features that can be extracted offline or online, tailored to the features of the gripper. The nature of the suction cup affects its adaptability to the lower surface and therefore it is more likely that an optimal grip is achieved when picking on the estimated surface normal of the object rather than performing a vertical gantry type pick as is common in current industrial applications.

In addition to geometric information, the system also uses appearance-based features, as depth sensors may not always be accurate enough to provide sufficient information about grippability. For example, the system may learn the location of fiducials such as bar codes on the object, which may be used as indicators of a flat and airtight surface patch, and thus be suitable for suction cups. One such example is the use of bar codes on consumer products. Another example is shipping boxes and bags which tend to have shipping labels at the centroid of the object and provide an air impermeable surface compared to the original bag material, which, by contrast, may be slightly porous and therefore not exhibit a good grip.

Establishing a correlation between features in the 2D/3D image and the concept of good or bad grasp points by identifying bad or good grasp points on the image; using this data and these correlations as inputs to the machine learning algorithm, the system is ultimately able to learn where to grasp and where to avoid for each image presented to it.

This information is added to the collected experience-based data for each successful or failed pick-up attempt by the system. Over time, the robot learns to avoid features that cause grip failure, either specific to the object type or specific to the surface/material type. For example, regardless of the object to which the robot is applied, the robot may wish to avoid picking on shrink wrap, and only wish to place the gripper near a datum of certain object types, such as a transport bag.

This learning can be accelerated by generating the human correction image offline. For example, one might take thousands of images from previous system operations and manually annotate the good and bad points of grip for each operation. This will generate a large amount of data that can also be input into the machine learning algorithm to increase the speed and efficiency of system learning.

In addition to training data based on experience or based on human experts, a large set of tagged training data can be generated based on detailed object models in physical simulations, using known gripper and object features. This process allows for rapid and intensive generation of grippability data on large groups of objects, since the process is not limited by the speed of the physical robotic system or manual input.