阅读说明：本技术 目标检测模型的更新方法、装置及系统 (Target detection model updating method, device and system ) 是由 刘伟峰 程云建 刘旭 于 2021-08-04 设计创作，主要内容包括：本申请公开了一种目标检测模型的更新方法、装置及系统。方法的一具体实施方式包括：在根据目标检测模型对待检测图像中的目标物的检测结果,控制拣选机器人执行拣选任务的过程中,确定拣选机器人的实际位姿信息和期望位姿信息；根据实际位姿信息和期望位姿信息,确定目标物在待检测图像中的更正结果；以待检测图像为图像样本,以更正结果为标签,得到训练样本,并将训练样本添加至训练样本集；响应于确定目标检测模型的检测精度低于预设阈值,通过训练样本集训练目标检测模型,得到更新后的目标检测模型。本申请提供了一种自动生成训练数据、自动更新目标检测模型的方法,提高了目标检测模型更新的便捷性和检测结果的准确度。(The application discloses a method, a device and a system for updating a target detection model. One embodiment of the method comprises: determining actual pose information and expected pose information of a picking robot in the process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected in a target detection model; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model. The application provides a method for automatically generating training data and updating a target detection model, and improves the convenience of updating the target detection model and the accuracy of a detection result.)

1. An updating method of an object detection model comprises the following steps:

determining actual pose information and expected pose information of the picking robot in the process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected by the target detection model, wherein the target detection model is used for representing the corresponding relation between the image to be detected and the detection result;

determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information;

taking the image to be detected as an image sample, taking the correction result as a label to obtain a training sample, and adding the training sample to a training sample set;

and in response to the fact that the detection precision of the target detection model is lower than a preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

2. The method of claim 1, wherein the determining actual and expected pose information for the picking robot comprises:

in response to determining that the picking robot picks the target object and moves to a preset position, acquiring actual pose information of the picking robot and point cloud data of the target object;

and determining the expected pose information according to the actual pose information and the point cloud data.

3. The method of claim 2, wherein the determining the expected pose information from the actual pose information and the point cloud data comprises:

determining pose information corresponding to a central point of a target surface of the target object according to the actual pose information and the point cloud data, wherein the target surface is a surface of the target object, which is in contact with the picking robot;

determining the pose information as the expected pose information.

4. The method according to claim 1, wherein the determining a correction result of the object in the image to be detected according to the actual pose information and the expected pose information comprises:

determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information;

and obtaining the correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

5. The method of claim 1, wherein the training the target detection model through the training sample set in response to determining that the detection accuracy of the target detection model is below a preset threshold, resulting in an updated target detection model, comprises:

determining the similarity of the detection result and the correction result;

counting a plurality of current similarities, and determining a similarity mean value;

and in response to determining that the similarity mean value is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

6. The method of any of claims 1-5, further comprising:

and controlling the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

7. An apparatus for updating an object detection model, comprising:

a first determination unit configured to determine actual pose information and expected pose information of a picking robot in a process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected by the target detection model, wherein the target detection model is used for representing a corresponding relation between the image to be detected and the detection result;

a second determination unit configured to determine a correction result of the target object in the image to be detected, based on the actual pose information and the expected pose information;

the obtaining unit is configured to obtain a training sample by taking the image to be detected as an image sample and the correction result as a label, and add the training sample to a training sample set;

an updating unit configured to train the target detection model through the training sample set in response to determining that the detection precision of the target detection model is lower than a preset threshold, resulting in an updated target detection model.

8. The apparatus of claim 7, wherein the first determining unit is further configured to:

in response to determining that the picking robot picks the target object and moves to a preset position, acquiring actual pose information of the picking robot and point cloud data of the target object; and determining the expected pose information according to the actual pose information and the point cloud data.

9. The apparatus of claim 8, wherein the first determining unit is further configured to:

determining pose information corresponding to a central point of a target surface of the target object according to the actual pose information and the point cloud data, wherein the target surface is a surface of the target object, which is in contact with the picking robot; determining the pose information as the expected pose information.

10. The apparatus of claim 7, wherein the second determining unit is further configured to:

determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information; and obtaining the correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

11. The apparatus of claim 7, wherein the update unit is further configured to:

determining the similarity of the detection result and the correction result; counting a plurality of current similarities, and determining a similarity mean value; and in response to determining that the similarity mean value is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

12. The apparatus of any of claims 7-11, further comprising:

and the subsequent execution unit is configured to control the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

13. An update system for an object detection model, comprising: the device comprises an image acquisition device, a control device and a picking robot; wherein the content of the first and second substances,

the image acquisition device is used for acquiring an image to be detected comprising a target object;

the control device is used for obtaining a detection result of a target object in the image to be detected through the target detection model and controlling the picking robot to execute a picking task according to the detection result; determining actual pose information and expected pose information of the picking robot in the process of executing the picking task by the picking robot; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking the image to be detected as an image sample, taking the correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than a preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

14. The system of claim 13, further comprising: the point cloud acquisition device is arranged at a preset position; wherein the content of the first and second substances,

the point cloud acquisition device is used for responding to the fact that the picking robot picks the target object and moves to the preset position, and acquiring point cloud data of the target object;

the control device is further used for acquiring the actual pose information of the picking robot at the preset position; and determining the expected pose information according to the actual pose information and the point cloud data.

15. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-6.

16. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a method, a device and a system for updating a target detection model.

Background

In the field of intelligent warehousing automation, the mode of executing the picking task by the picking robot has better application prospect. The picking robot carries out a picking task, namely the picking robot picks the target object to a specified position on the basis of visual guidance. High-precision target detection is the core technical basis of the application scenario. The target detection technology based on deep learning is a relatively practical technology at present. The deep learning technology aiming at target detection belongs to the category of supervised learning, and the basic process is as follows: (1) acquiring scene data in batch in advance; (2) carrying out manual marking on the data; (3) training a target detection model; (4) and deploying the trained target detection model to an online application. It will be appreciated that object detection models generally suffer from timeliness issues. For example, in e-commerce scenarios, where commodity updates are frequent, the accuracy of a trained target detection model based on early data may degrade over time. If the above target detection model obtaining process is executed again, although the model can be restored to the accuracy again, the process is time-consuming and labor-consuming.

Disclosure of Invention

The embodiment of the application provides a method, a device and a system for updating a target detection model.

In a first aspect, an embodiment of the present application provides an update method of a target detection model, including: determining actual pose information and expected pose information of a picking robot in the process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected by a target detection model, wherein the target detection model is used for representing the corresponding relation between the image to be detected and the detection result; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model.

In some embodiments, the determining actual pose information and desired pose information for the picking robot comprises: in response to the fact that the picking robot picks the target object to move to the preset position, acquiring actual pose information of the picking robot and point cloud data of the target object; and determining expected pose information according to the actual pose information and the point cloud data.

In some embodiments, the determining the expected pose information according to the actual pose information and the point cloud data includes: determining pose information corresponding to a central point of a target surface of the target object according to the actual pose information and the point cloud data, wherein the target surface is a surface of the target object, which is in contact with the picking robot; the pose information is determined to be expected pose information.

In some embodiments, the determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information includes: determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information; and obtaining a correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

In some embodiments, the training the target detection model through the training sample set in response to determining that the detection accuracy of the target detection model is lower than the preset threshold to obtain an updated target detection model includes: determining the similarity of the detection result and the correction result; counting a plurality of current similarities, and determining a similarity mean value; and in response to the fact that the similarity mean value is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

In some embodiments, the above method further comprises: and controlling the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

In a second aspect, an embodiment of the present application provides an apparatus for updating a target detection model, including: a first determination unit configured to determine actual pose information and expected pose information of the picking robot in a process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected in a target detection model, wherein the target detection model is used for representing a corresponding relation between the image to be detected and the detection result; a second determination unit configured to determine a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; the obtaining unit is configured to obtain a training sample by taking the image to be detected as an image sample and a correction result as a label, and add the training sample to a training sample set; an updating unit configured to train the target detection model through the training sample set to obtain an updated target detection model in response to determining that the detection accuracy of the target detection model is lower than a preset threshold.

In some embodiments, the first determining unit is further configured to: in response to the fact that the picking robot picks the target object to move to the preset position, acquiring actual pose information of the picking robot and point cloud data of the target object; and determining expected pose information according to the actual pose information and the point cloud data.

In some embodiments, the first determining unit is further configured to: determining pose information corresponding to a central point of a target surface of the target object according to the actual pose information and the point cloud data, wherein the target surface is a surface of the target object, which is in contact with the picking robot; the pose information is determined to be expected pose information.

In some embodiments, the second determining unit is further configured to: determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information; and obtaining a correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

In some embodiments, the update unit is further configured to: determining the similarity of the detection result and the correction result; counting a plurality of current similarities, and determining a similarity mean value; and in response to the fact that the similarity mean value is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

In some embodiments, the above apparatus further comprises: and the subsequent execution unit is configured to control the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

In a third aspect, an embodiment of the present application provides an update system of a target detection model, including: the device comprises an image acquisition device, a control device and a picking robot; the image acquisition device is used for acquiring an image to be detected comprising a target object; the control device is used for obtaining a detection result of a target object in an image to be detected through the target detection model and controlling the picking robot to execute a picking task according to the detection result; determining actual pose information and expected pose information of the picking robot in the process of executing the picking task by the picking robot; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model.

In some embodiments, the system further comprises: the point cloud acquisition device is arranged at a preset position; the device comprises a point cloud acquisition device, a picking robot and a control device, wherein the point cloud acquisition device is used for responding to the condition that the picking robot picks the target object and moves to a preset position and acquiring point cloud data of the target object; the control device is further used for acquiring the actual pose information of the picking robot at a preset position; and determining expected pose information according to the actual pose information and the point cloud data.

In a fourth aspect, the present application provides a computer-readable medium, on which a computer program is stored, where the program, when executed by a processor, implements the method as described in any implementation manner of the first aspect.

In a fifth aspect, an embodiment of the present application provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method as described in any implementation of the first aspect.

According to the method, the device and the system for updating the target detection model, the actual pose information and the expected pose information of the picking robot are determined in the process of controlling the picking robot to execute the picking task according to the detection result of the target object in the image to be detected in the target detection model; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, the target detection model is trained through the training sample set, and the updated target detection model is obtained, so that the method for automatically generating training data and updating the target detection model is provided, and the convenience of updating the target detection model and the accuracy of the detection result are improved.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied;

FIG. 2 is an exemplary system architecture diagram in which yet another embodiment of the present application may be applied;

FIG. 3 is a flow diagram of one embodiment of a method for updating an object detection model according to the present application;

fig. 4 is a schematic diagram of an application scenario of the update method of the object detection model according to the present embodiment;

FIG. 5 is a flow diagram of yet another embodiment of an update method of an object detection model according to the present application;

FIG. 6 is a block diagram of one embodiment of an apparatus for updating an object detection model according to the present application;

FIG. 7 is a block diagram of a computer system suitable for use in implementing embodiments of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Fig. 1 illustrates an exemplary architecture 100 to which the object detection model update methods and apparatus of the present application may be applied.

As shown in fig. 1, the system architecture 100 may include image acquisition devices 101, 102, networks 103, 104, a control device 105, and a picking robot 106. The image capturing devices 101, 102 are communicatively connected to the control device 105 and the picking robot 106 to form a topological network, and the networks 103, 104 are used to provide a medium for communication links between the image capturing devices 101, 102 and the control device 105 and the picking robot 106. The networks 103, 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The image capturing apparatuses 101, 102 may be hardware devices or software having an image capturing function and an information transmission function. When the image capturing devices 101 and 102 are hardware, they may be various electronic devices supporting network connection, image capturing, interaction, display, processing, and the like, including but not limited to 2D cameras, 3D cameras, smart phones, tablet computers, desktop computers, and the like. When the image capturing devices 101 and 102 are software, they can be installed in the electronic devices listed above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.

The control device 105 may be a server that provides various services, such as a server that automatically generates training data and automatically updates a target detection model in a process of controlling a picking robot to perform a picking task based on a detection result of a target object in an image to be detected. As an example, the control device 105 may be a cloud server. Specifically, the control device 105 obtains a detection result of the target object in the image to be detected through the target detection model, and controls the picking robot to execute the picking task according to the detection result; determining actual pose information and expected pose information of the picking robot in the process of executing the picking task by the picking robot; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model.

The control device may be hardware or software. When the control device is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the control means is software, it may be implemented as a plurality of software or software modules (for example, software or software modules for providing distributed services), or as a single software or software module. And is not particularly limited herein.

The picking robot may be various robots having a picking function, such as a multi-degree-of-freedom robot arm.

As shown in fig. 2, yet another exemplary architecture 200 of an update system for an object detection model is shown, comprising an image acquisition device 201, a control device 202, a picking robot 203, and a point cloud acquisition device 204.

The point cloud collecting device 204 is disposed at a predetermined position. The preset position can be any position which is passed by the picking robot in the moving process of the picking target object. For example, the preset position is disposed near a picking station where an object to be picked is placed. The point cloud acquisition device 204 may be, for example, a 3D camera, and is configured to acquire point cloud data of a target object in response to determining that a picking robot picks the target object to a preset position in the process that the picking robot picks the target object from the picking station to the palletizing station; the control device 202 is further used for acquiring the actual pose information of the picking robot at a preset position; and determining expected pose information according to the actual pose information and the point cloud data.

The method for updating the object detection model provided by the embodiment of the present application may be executed by the control device, and accordingly, each part (e.g., each unit) included in the apparatus for updating the object detection model may be entirely provided in the control device.

It should be understood that the number of image acquisition devices, networks, control devices, and picking robots in fig. 1 is merely illustrative. There may be any number of image acquisition devices, networks, control devices, and picking robots, as desired for implementation. When the electronic device on which the update method of the object detection model is executed does not need to perform data transmission with other electronic devices, the system architecture may include only the electronic device (e.g., the control apparatus) on which the update method of the object detection model is executed.

With continued reference to FIG. 3, a flow 300 of one embodiment of a method for updating a target detection model is shown, comprising the steps of:

step 301, determining actual pose information and expected pose information of the picking robot in the process of controlling the picking robot to execute a picking task according to the detection result of the target object in the image to be detected in the target detection model.

In this embodiment, an execution subject (for example, the control device in fig. 1) of the method for updating the target detection model may obtain the image to be detected from a remote location or a local location by using a wired network connection manner or a wireless network connection manner, and determine the actual pose information and the expected pose information of the picking robot during the process of controlling the picking robot to execute the picking task according to the detection result of the target object in the image to be detected by the target detection model.

The target detection model is used for representing the corresponding relation between the image to be detected and the detection result, and can be obtained based on the neural network model with the target object detection function through training, and the target detection model comprises but is not limited to a convolutional neural network, a residual error neural network and a cyclic neural network.

The image to be detected is image data including an object, including but not limited to a 2D image and a 3D image. The 2D image may be, for example, an RGB (red, green, blue, red, green, blue) image, and the 3D image may be, for example, point cloud data characterizing the target object. As an example, above a picking station where an object to be picked is placed, a 2D camera and a 3D camera are provided to acquire a 2D image and a 3D image including the object.

The image to be detected is input into the target detection model, a detection frame indicating a target object in the image to be detected can be determined, and then the position information of the target object on the picking station in the real environment is determined according to the calibration information of the image acquisition device for acquiring the image to be detected. Further, the picking robot may be controlled to pick the target object at the determined position information.

In this embodiment, the expected pose information represents pose information of the picking robot in the case of accurately picking the target object. In the actual use process of the target detection model, due to the reduction of the detection precision of the target detection model, a detection frame indicated by the detection result of the target object in the image to be detected and an actual detection frame of the target object in the image to be detected may have a deviation, so that the actual pose information of the picking robot is not the expected pose information in the process of executing the picking task. As an example, the picking robot takes the pose information corresponding to the center point of the upper surface of the target object (the surface of the target object in contact with the picking robot) as the expected pose information, and during the actual picking process, the picking robot deviates from the center point of the upper surface, grasps one end of the target object, resulting in a deviation between the actual pose information and the expected pose information thereof.

In this implementation, after the picking robot picks the target object, the execution main body may further control the image acquisition device to acquire an image of the picking robot in a state of picking the target object, and determine size information of the target object and a central point of the upper surface according to point cloud information corresponding to the target object, so as to determine expected pose information of the picking robot. Actual pose information for the picking robot may be provided by the picking robot.

Wherein the pose information can be (x)_c,y_c,z_cα, β, γ). Wherein (x)_c,y_c,z_c) The (α, β, γ) is a coordinate system rotation relationship expressed in euler angles, that is, a rotation relationship of an xyz axis of the world coordinate system to an xyz axis of the picking robot tool coordinate system. The transformation relation between the coordinate system of the end tool of the picking robot and the world coordinate system of the robot can be calculated according to the pose of the picking robot, in other words, the transformation matrix of the world coordinate system of the robot to the coordinate system of the end tool of the robot is recorded asWherein R is₃₃Calculated by (alpha, beta, gamma), representing the rotation of the coordinate system, and is a 3 multiplied by 3 matrix; t is t₃₁I.e. (x)_c,y_c,z_c) Indicating translation, as a 3 × 1 matrix.

In some alternative implementations of this embodiment, the executing body may determine the actual pose information and the expected pose information of the picking robot by:

firstly, in response to the fact that the picking robot is determined to move to the preset position, actual pose information of the picking robot and point cloud data of the target object are obtained.

In this implementation, a point cloud collection device for acquiring point cloud data of a target object is provided at a preset position. The point cloud acquisition device may be, for example, a 3D camera. The preset position may be any position through which the picking robot moves to pick the target object. For example, the preset position is disposed near a picking station where an object to be picked is placed.

The picking robot picks the target object and moves to a preset position, the picking robot pauses above the point cloud acquisition device, a camera lens of the point cloud acquisition device is arranged upwards, and the point cloud data of the lower surface of the target object are acquired. When the picking robot grabs the target object and moves to the preset position, only one target object is arranged at the tail end of the mechanical arm, and therefore point cloud data can be accurately segmented, and point cloud data of the target object are obtained.

And secondly, determining expected pose information according to the actual pose information and the point cloud data.

As an example, the execution subject may determine size information of the target object according to point cloud data corresponding to the target object in the point cloud data, and then determine expected pose information of the picking robot

In some optional implementations of this embodiment, the executing body may execute the second step by: firstly, according to the actual pose information and the point cloud data, the pose information corresponding to the central point of the target surface of the target object is determined. The target surface is a surface of the target object, which is in contact with the picking robot, namely the top surface of the target object. Then, the pose information is determined as the expected pose information.

Specifically, the execution main body determines the coordinates, the length and the width of the bottom surface center point of the target object according to the point cloud data of the target object at the preset position; determining the height of a target object according to the actual pose information of the picking robot and the bottom surface center point coordinate, and determining the top surface center point coordinate of the target object according to the height of the target object and the bottom surface center point coordinate; and determining a length direction vector and a width direction vector of the article according to the length and the width of the target object, and further determining expected pose information according to the length direction vector, the width direction vector and the top surface center point coordinate. Wherein the bottom surface and the top surface of the target object at the preset position are parallel to the xy plane of the world coordinate system.

And 302, determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information.

In this embodiment, the execution subject may determine a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information.

As an example, the execution subject may determine the actual position information of the target object by using a deviation between the actual pose information and the expected pose information, and the position information of the target object determined by the execution subject according to the detection result of the target object in the image to be detected. Further, the correct target frame of the target object in the image to be detected is reversely deduced according to the actual position information of the target object (correction result). The reverse process may be regarded as a reverse process of determining the position information of the target object according to the detection result of the target object.

In some optional implementations of this embodiment, the executing main body may execute the step 302 by:

first, position information of an area occupied by a target object is determined according to deviation information between actual pose information and expected pose information.

And secondly, obtaining a correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

And according to deviation information between the actual pose information and the expected pose information, the accurate three-dimensional bounding rectangle of the target object in the picking station can be reversely deduced, and then the accurate three-dimensional bounding rectangle is mapped onto the 2D image by utilizing the calibration relation between the 3D camera and the 2D camera, so that the detection frame (correction result) of the target object on the 2D image is obtained.

Step 303, using the image to be detected as an image sample, using the corrected result as a label to obtain a training sample, and adding the training sample to a training sample set.

In this embodiment, the execution subject may use the image to be detected as an image sample, use the correction result as a label, obtain a training sample, and add the training sample to the training sample set.

The executing agent may take a large number of training samples as the picking task is executed.

And 304, in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

In this embodiment, the executing entity may, in response to determining that the detection accuracy of the target detection model is lower than the preset threshold, train the target detection model through the training sample set to obtain an updated target detection model.

The preset threshold value can be specifically set according to actual conditions. As an example, when the detection accuracy required by the picking task is higher, a higher preset threshold value may be set; when the detection accuracy required by the picking task is not high, a lower preset threshold value may be set.

When the detection accuracy of the target detection model is lower than a preset threshold, a training sample can be selected from the training sample set, a sample image in the selected training sample is used as input, a label corresponding to the input sample image is used as expected output, and the updated target detection model is obtained through training.

In some optional implementations of this embodiment, the executing main body may execute the step 304 by:

first, the similarity between the detection result and the correction result is determined.

As an example, the execution body may use IoU (Intersection-over-Union) of the detection frame of the target object in the detection result and the detection frame of the target object in the correction result as the similarity between the detection result and the correction result.

Secondly, counting a plurality of current similarities, and determining a similarity mean value.

In order to avoid the contingency of the detection result error, in the embodiment, the currently obtained multiple similarities are counted and cut off, so that a similarity mean value is obtained.

Thirdly, in response to determining that the similarity mean is lower than the preset threshold, training the target detection model through the training sample set to obtain an updated target detection model

With continued reference to fig. 4, fig. 4 is a schematic diagram 400 of an application scenario of the method for updating the object detection model according to the present embodiment. In the application scenario of fig. 4, the 2D camera 401 and the 3D camera 402 collect images to be detected including the target object at the picking station and transmit the images to be detected to the server 403. The server 403 generates a detection result of the target object in the image to be detected through the target detection model, and controls the picking robot 404 to execute a picking task according to the detection result. Determining actual and expected pose information for the picking robot 404 during performance of the picking task; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model.

According to the method provided by the embodiment of the application, the actual pose information and the expected pose information of the picking robot are determined in the process of controlling the picking robot to execute the picking task according to the detection result of the target object in the image to be detected in the target detection model; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, the target detection model is trained through the training sample set, and the updated target detection model is obtained, so that the method for automatically generating training data and updating the target detection model is provided, and the convenience of updating the target detection model and the accuracy of the detection result are improved.

In some optional implementation manners of this embodiment, after the updated target detection model is obtained, the executing body may control the picking robot to execute the subsequent picking task according to a detection result of the updated target detection model on the subsequent target object in the image to be detected.

The updated target detection model has high detection precision aiming at the target object, so that the picking robot can be controlled to more accurately execute the picking task.

With continuing reference to FIG. 5, an exemplary flow 500 of one embodiment of a method for updating an object detection model according to the present application is shown, including the steps of:

step 501, in the process of controlling the picking robot to execute the picking task according to the detection result of the target object in the image to be detected in the target detection model, in response to the fact that the picking robot moves to the preset position, acquiring actual pose information of the picking robot and point cloud data of the target object.

And 502, determining pose information corresponding to the central point of the target surface of the target object according to the actual pose information and the point cloud data.

Wherein the target surface is a surface of the target object that is in contact with the picking robot.

And step 503, determining the pose information as expected pose information.

And step 504, determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information.

And 505, obtaining a correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

Step 506, the image to be detected is taken as an image sample, the corrected result is taken as a label, a training sample is obtained, and the training sample is added to the training sample set.

Step 507, determining the similarity between the detection result and the correction result.

And step 508, counting a plurality of current similarities, and determining a similarity mean value.

Step 509, in response to determining that the similarity mean is lower than the preset threshold, training the target detection model through the training sample set to obtain an updated target detection model.

And step 510, controlling the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

As can be seen from this embodiment, compared with the embodiment corresponding to fig. 3, the flow 500 of the method for updating the target detection model in this embodiment specifically illustrates a process of determining the correction result, and an update process of the target detection model, wherein the point cloud data of the target object at the preset position is skillfully used to determine the correction result, so that the convenience of updating the target detection model and the accuracy of the detection result are further improved.

With continuing reference to fig. 6, as an implementation of the method shown in the above-mentioned figures, the present application provides an embodiment of an apparatus for updating a target detection model, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 3, and the apparatus may be applied to various electronic devices.

As shown in fig. 6, the apparatus for updating the object detection model includes: a first determining unit 601 configured to determine actual pose information and expected pose information of the picking robot in a process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected by a target detection model, wherein the target detection model is used for representing a corresponding relation between the image to be detected and the detection result; a second determining unit 602 configured to determine a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; an obtaining unit 603 configured to obtain a training sample by using the image to be detected as an image sample and using the correction result as a label, and add the training sample to the training sample set; an updating unit 604 configured to train the target detection model through the training sample set to obtain an updated target detection model in response to determining that the detection accuracy of the target detection model is lower than a preset threshold.

In some embodiments, the first determining unit 601 is further configured to: in response to the fact that the picking robot picks the target object to move to the preset position, acquiring actual pose information of the picking robot and point cloud data of the target object; and determining expected pose information according to the actual pose information and the point cloud data.

In some embodiments, the first determining unit 601 is further configured to: determining pose information corresponding to a central point of a target surface of the target object according to the actual pose information and the point cloud data, wherein the target surface is a surface of the target object, which is in contact with the picking robot; the pose information is determined to be expected pose information.

In some embodiments, the second determining unit 602 is further configured to: determining the position information of the area occupied by the target object according to the deviation information between the actual pose information and the expected pose information; and obtaining a correction result according to the calibration information and the position information of the camera device for obtaining the image to be detected.

In some embodiments, the updating unit 604 is further configured to: determining the similarity of the detection result and the correction result; counting a plurality of current similarities, and determining a similarity mean value; and in response to the fact that the similarity mean value is lower than the preset threshold value, training the target detection model through the training sample set to obtain an updated target detection model.

In some embodiments, the above apparatus further comprises: and a subsequent execution unit (not shown in the figure) configured to control the picking robot to execute a subsequent picking task according to the detection result of the updated target detection model on the target object in the subsequent image to be detected.

In this embodiment, a first determining unit in the apparatus for updating the target detection model determines actual pose information and expected pose information of the picking robot in a process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected in the target detection model, where the target detection model is used to represent a corresponding relationship between the image to be detected and the detection result; the second determining unit determines a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; the obtaining unit takes an image to be detected as an image sample, takes a correction result as a label, obtains a training sample, and adds the training sample to a training sample set; the updating unit responds to the fact that the detection precision of the target detection model is lower than the preset threshold value, the target detection model is trained through the training sample set, and the updated target detection model is obtained, so that the device for automatically generating training data and updating the target detection model is provided, and the convenience of updating the target detection model and the accuracy of the detection result are improved.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use in implementing devices of embodiments of the present application (e.g., devices 101, 102, 105, 106 shown in FIG. 1). The apparatus shown in fig. 7 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present application.

As shown in fig. 7, the computer system 700 includes a processor (e.g., CPU, central processing unit) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the system 700 are also stored. The processor 701, the ROM702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by the processor 701, performs the above-described functions defined in the method of the present application.

It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the client computer, partly on the client computer, as a stand-alone software package, partly on the client computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a first determining unit, a second determining unit, an obtaining unit, and an updating unit. Where the names of the units do not in some cases constitute a limitation on the units themselves, for example, the updating unit may also be described as "a unit that trains the target detection model by a training sample set to obtain an updated target detection model in response to determining that the detection accuracy of the target detection model is lower than a preset threshold".

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the computer device to: determining actual pose information and expected pose information of a picking robot in the process of controlling the picking robot to execute a picking task according to a detection result of a target object in an image to be detected by a target detection model, wherein the target detection model is used for representing the corresponding relation between the image to be detected and the detection result; determining a correction result of the target object in the image to be detected according to the actual pose information and the expected pose information; taking an image to be detected as an image sample, taking a correction result as a label to obtain a training sample, and adding the training sample to a training sample set; and in response to the fact that the detection precision of the target detection model is lower than the preset threshold value, training the target detection model through a training sample set to obtain an updated target detection model.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.