阅读说明：本技术 一种适用于串型水果的采摘方法 (Picking method suitable for string-shaped fruits ) 是由 姚昱岑 曹川川 王维勋 郑讯佳 于 2021-01-21 设计创作，主要内容包括：本发明提供一种适用于串型水果的采摘方法,该方法首先通过单目相机与双目相机的配合对母枝实现预定位,得到母枝的预定位图像及预定位几何中心点；然后通过双目视觉相机获取采摘过程中母枝的实际定位图像及实际定位几何中心点,再利用单目视觉缩放法,比较母枝的实际定位图像与预定位图像,确定母枝受干扰类型；然后利用空气动力进行主动抗扰；最终确定采摘机器人的进给终点与进给路径,实现采摘。该方法能快速、精确的实现对串型水果母枝的剪切,实现完全智能化采摘,从而有效提高采摘的工作效率及串型水果的收率,降低采摘作业的能耗、减少采摘劳动力及成本。(The invention provides a picking method suitable for string-shaped fruits, which comprises the steps of firstly, realizing pre-positioning on a mother branch through the matching of a monocular camera and a binocular camera to obtain a pre-positioning image and a pre-positioning geometric central point of the mother branch; then acquiring an actual positioning image and an actual positioning geometric central point of the mother branch in the picking process through a binocular vision camera, comparing the actual positioning image and the pre-positioning image of the mother branch by using a monocular vision zooming method, and determining the type of the interference on the mother branch; then, active disturbance rejection is carried out by utilizing aerodynamic force; finally, the feeding terminal point and the feeding path of the picking robot are determined, and picking is achieved. The method can rapidly and accurately realize shearing of the string-shaped fruit mother branches and complete intelligent picking, so that the picking work efficiency and the string-shaped fruit yield are effectively improved, the energy consumption of picking operation is reduced, and the picking labor force and the picking cost are reduced.)

1. A picking method suitable for string type fruits is characterized in that:

s100, pre-positioning a mother branch is realized by matching a monocular camera and a binocular camera, and the method comprises the following specific steps:

s101: randomly acquiring a plurality of color images of the fruits, the leaves and the branches of the fruits in the string form by adopting a monocular CCD camera; selecting and dividing a plurality of fruit targets and non-fruit targets in the color image, and respectively extracting texture characteristic values and color characteristic values of the fruit targets and the non-fruit targets as positive and negative samples;

s102, training positive and negative samples by adopting a Support Vector Machine (SVM) to generate a plurality of weak classifiers; then, an AdaBoost algorithm is adopted to construct a strong classifier, the color image acquired by the monocular CCD camera is segmented by the strong classifier, and the fruit target identified from the image is separately stored into a color image I₁；

S103, repeating the steps S101 to S102 to respectively obtain strong classifiers for identifying branches and leaves; then respectively removing the image I₁The branches and leaves are extracted from the original color image and stored into a color image I respectively₂And a color image I₃；

S104, dividing the fruit clusters into single fruit clusters, double fruit clusters and multi-fruit clusters according to the distribution positions of the fruits, and further determining a color image I₁The fruit cluster type of the randomly distributed fruits;

s105, determining mother branches according to the fruit string type determined in the step S104:

for a single fruit cluster, the branch connected with the fruit is the mother branch;

for the double fruit cluster and the multi-fruit cluster, firstly, in the color image I₁The middle making fruit string is combined with the color image I through the perpendicular bisector of the upper and lower bottoms of the outline circumscribed rectangle₁With color images I₂Performing fusion analysis by color image I₁Perpendicular bisector and color image I₂Determining the parent branch of the fruit bunch according to the branch tangency condition;

s106, acquiring a binocular stereo image of the fruit cluster mother branches by using a binocular CCD camera, taking a geometric central point of a mother branch circumscribed rectangle in a left image of the binocular CCD camera as a feature matching point, searching a point which is closest to a gray value of the feature matching point and enables a normalized cross-correlation function to reach a maximum value in a right image, realizing feature matching, and obtaining the geometric central point of the mother branches; finally, calculating the spatial coordinates of the geometric center point of the mother branch to realize the pre-positioning of the mother branch, and obtaining a pre-positioning image and a pre-positioning geometric center point of the mother branch;

s200, in the picking process, acquiring an actual positioning image and an actual positioning geometric center point of a mother branch in the picking process by using the method in the step S100 and a binocular vision camera; then, comparing the actual positioning image and the pre-positioning image of the mother branch by using a monocular vision zooming method, and determining the type of the interference on the mother branch;

s300, performing active anti-interference by using aerodynamic force according to the type of the parent branch interfered determined in the step S200;

s400, determining a feeding terminal point of the mechanical arm of the picking robot based on the active disturbance rejection method in the step S300; and determining a feeding path of a mechanical arm of the picking robot according to the feeding end point, and picking the string-shaped fruits.

2. A picking process for string fruit according to claim 1, characterised in that: in the step S104, a fruit cluster classification principle is adopted to classify the single fruit cluster, the double fruit cluster and the multi-fruit cluster;

single fruit bunch: if the Euclidean distance between the geometric center of one fruit and the geometric center of any other fruit is larger than the average diameter of a single fruit, the fruit is a single fruit cluster; double fruit bunch: if the Euclidean distance between the geometric centers of two adjacent fruits is smaller than the sum of the diameters of the two fruits, the two fruits are a double-fruit cluster; multi-fruit clusters: if the Euclidean distance between the geometric centers of any two fruits in more than two fruits is smaller than the sum of the diameters of the two fruits, the group of fruits is a multi-fruit cluster.

3. A picking process for strung fruit according to any one of claims 1 or 2, characterised in that: the step of searching the point which is closest to the gray value of the feature matching point and enables the normalized cross-correlation function to reach the maximum value in the right image in the step S106 is specifically as follows:

first, a left image P is obtained by a binocular CCD camera₁The feature matching point (x, y) of (a) constructs a matching window while the right image P is taken₂The point (x + m, y + m) to be matched also constructs a matching window, and measures the correlation degree by normalizing the correlation function, wherein the specific formula is as follows:

wherein F (m, n) represents a normalized correlation function; (m, n) represents a position vector of the right image relative to the left image; w_pRepresenting a matching window taking the coordinates of the point to be matched as a center; p₂(x + m, y + n) represents the gray value of the point to be matched of the right image;representing a gray level mean of the right image matching window; p₁(x, y) represents the gray value of the left image feature matching point;mean value of gray levels representing a matching window of a left image；

Wherein, the value range of F (m, n) is [ -1,1 ];

when F (m, n) is-1, the feature matching point of the left image is completely unrelated to the point to be matched of the right image;

when F (m, n) is 1, the feature matching point of the left image is completely matched with the point to be matched of the right image, that is, the point to be matched is the point at which the gray value of the feature matching point is closest and the normalized cross-correlation function of the point to be matched reaches the maximum value.

4. A picking process for strung fruit according to any one of claims 1 to 3, characterised in that: the step S200 is specifically:

s201, firstly, judging whether the parent branches of the fruit clusters in the actual positioning image are blocked or not through whether a binocular vision camera identifies the parent branches of the fruit clusters in the actual positioning image or not;

if the fruit cluster mother branches are not identified in the actual positioning image, the pre-positioning image is superposed on the actual positioning image, and whether eight neighborhood pixels of the pre-positioning image and the actual positioning image belong to the same thing is judged through a monocular vision zooming method;

if the actual positioning image belongs to the tree branch positioning image, judging that the parent branch in the actual positioning image is shielded by the tree branch;

if not, further locating eight neighborhood pixels of the image and the color image I₁Color image I₃Comparing the eight neighborhood pixels, and judging whether the eight neighborhood pixels are shielded by fruits or leaves;

s202, if the fruit string mother branch is identified in the actual positioning image, the mother branch in the actual positioning image is not shielded, and then the pre-positioning geometric center point O in the pre-positioning image is determined₁And an actual positioning geometric center point O in the actual positioning image₂Judging whether the parent branch deviates or not according to the relation between the deviation relative value d and a preset threshold value;

if the deviation relative value d is not greater than the preset threshold value, the interference is not generated;

if the deviation relative value d is larger than the preset threshold value, and O₂Point-through-monocular visual zoomDetermining O if the scaled eight-neighborhood pixels belong to branches judged by the multi-dimensional SVM classifier₂The parent branch represented by the point is relative to O under the interference of dynamic factors₁The position of the point is shifted; if the relative deviation value d is greater than the preset threshold value, and O₂And if the point is judged not to belong to a branch by the multi-dimensional SVM classifier through the eight-neighborhood pixels zoomed by the monocular vision zooming method, outputting an error signal.

5. A picking process for string fruit according to any one of claims 1 or 4, characterised in that: the preset threshold is obtained by pre-positioning an external rectangle of the image parent branch, and specifically comprises the following steps:

firstly, constructing an external rectangular frame of a parent branch in a pre-positioned image; then obtaining the number p of bottom edge pixels of the external rectangular frame and the total number a of horizontal lines of pixels of the pre-positioned image; finally obtaining a preset threshold d₀(ii) a The concrete formula is as follows:

6. a picking process for string fruit according to claim 4, characterised in that: the step S300 specifically includes:

if the mother branch is shielded, firstly separating the shielding object from the target mother branch by using aerodynamic force; then, returning the geometric center point of the parent branch to the position of the pre-positioned geometric center point by using aerodynamic force, recording the motion trail of the geometric center point of the parent branch and the farthest and closest points of the plane of the camera to the feeding starting point by using a single-binocular vision camera and a binocular vision camera to obtain the motion period of the geometric center point of the parent branch, simultaneously obtaining the spatial point cloud of the shielding object by using the binocular vision camera, constructing a spatial enclosure completely tangent to the shielding object, and recording the motion trail of the spatial enclosure and the farthest and closest points of the plane of the camera to the feeding starting point to obtain the motion period of the shielding object enclosed by the spatial enclosure;

if the mother branch is not shielded and only deviates, returning the geometric center point of the mother branch to the position of the geometric center point of the preset position by using aerodynamic force, and recording the motion trail of the geometric center point of the mother branch and the points which are farthest and closest to the feeding starting point on the camera plane by using single and double-eye vision cameras to obtain the motion period of the geometric center point of the mother branch.

7. A picking process for string fruit according to claim 6, characterised in that: the step S400 specifically includes:

s401, determination of a feed end point:

if the mother branch is not shielded and only deviates, the space coordinate of the movement locus of the geometric central point of the mother branch at the point closest to the feeding starting point in the camera plane is the feeding end point of the mechanical arm of the picking robot;

if the mother branch is shielded, determining the maximum space redundancy between the mother branch and the shielding object at the farthest position and the nearest position of the camera plane from the feeding starting point respectively through the geometric center point of the mother branch and the space surrounding body of the shielding object, wherein the center of the maximum space redundancy is the feeding end point of the mechanical arm of the picking robot; the space redundancy is a circle formed by connecting the geometric center point of the parent branch and the space surrounding body of the shielding object at the farthest position and the nearest position of the camera plane from the feeding starting point respectively, and the circle center is the feeding end point;

s402, determining a feeding path:

firstly, constructing a space enclosure of an obstacle in a potential path from a feeding starting point to a feeding destination by adopting a binocular vision camera and a space geometric principle; then respectively defining the space enclosing body as a first obstacle, a second obstacle, a third obstacle, … … and an nth obstacle from the position close to the feeding starting point to the position far away from the feeding starting point, making a tangent line of the first obstacle through the feeding starting point, taking a tangent point as a first path point, making a vertical line segment of a connecting line of the second obstacle and the geometric center of the third obstacle through the first path point, taking the vertical point as a second path point, making a tangent line of the third obstacle through the second path point, taking the tangent point as a third path point, and sequentially circulating until the final path point is directly connected with the feeding terminal point … …; the connecting line of the feeding starting point, each path point and the feeding end point is the feeding path.

Technical Field

The invention relates to the technical field of intelligent fruit picking, in particular to a picking method suitable for string-shaped fruits.

Background

The fruit strings such as litchi, longan, grapes and the like are widely distributed in the southwest, the south and the southeast of China, and the two fruit strings of the litchi and the longan are called four fruits in the south China together with the bananas and the pineapples, and are important economic pillars of the fruit industry in China. For picking of string-shaped fruits, at present, manual picking is mainly relied on, but the manual picking is high in picking labor intensity and picking cost, and a large amount of manpower and material resources are wasted; secondly, the manual picking efficiency is low, the time consumption is long, and the cost is high.

With the progress of science and technology, mechanical automation and intellectualization are widely applied to various industries; to the field of intelligent fruit picking, the picking robot is more and more researched and applied, so that the labor force is effectively liberated, the labor cost is reduced, the picking efficiency is improved, and the picking time is saved. In the process of picking the cluster-shaped fruits, firstly, determining the mother branches of the fruit clusters manually, and then picking and cutting off the mother branches by adopting a picking robot to realize picking; the cutting of the fruit clusters from the mother branches ensures the integrity of the fruit clusters and prevents the fruit clusters from being scattered into a plurality of small clusters or particles, thereby being beneficial to storage and transportation, ensuring the aesthetic property of the fruit clusters and ensuring the economic value of the fruit clusters. However, due to the growth characteristics of the string-type fruits, the whole fruit string is easy to randomly distribute and grow, so that the problems that the fruit string is difficult to identify, the mother branch is difficult to identify and position, and the whole string-type fruit mother branch cannot be accurately found are caused; meanwhile, due to the fact that external environments such as wind, illumination and dew are accompanied in the picking process, the position of the mother branch can be changed or shielded in the picking process, and then the picking robot fails to pick or mistakenly shears or leaks to shear and damage fruits, picking efficiency is low, picking time periods are lengthened, and picking cost is improved.

Therefore, if the picking of the string-shaped fruits is to be realized, a visual perception system, an interference recognition and active anti-interference system and a manipulator motion track analysis system are required to be cooperatively modeled, so that the mother branches of the string-shaped fruits are accurately cut, wrong cutting and missing cutting are avoided, the picking efficiency is improved, the picking cost is reduced, and the picking intelligence of the string-shaped fruits is realized.

Disclosure of Invention

In view of the above problems in the prior art, the present invention is directed to a picking method for stringed fruits, which solves the above problems in the prior art.

The purpose of the invention is realized by the following technical scheme:

a picking method suitable for string type fruits is characterized in that:

s100, pre-positioning a mother branch is realized by matching a monocular camera and a binocular camera, and the method comprises the following specific steps:

s101: randomly acquiring a plurality of color images of the fruits, the leaves and the branches of the fruits in the string form by adopting a monocular CCD camera; selecting and dividing a plurality of fruit targets and non-fruit targets in the color image, and respectively extracting texture characteristic values and color characteristic values of the fruit targets and the non-fruit targets as positive and negative samples;

s102, training positive and negative samples by adopting a Support Vector Machine (SVM) to generate a plurality of weak classifiers; then, an AdaBoost algorithm is adopted to construct a strong classifier, the color image acquired by the monocular CCD camera is segmented by the strong classifier, and the fruit target identified from the image is separately stored into a color image I₁；

S103, repeating the steps S101 to S102 to respectively obtain strong classifiers for identifying branches and leaves; then respectively removing the image I₁The branches and leaves are extracted from the original color image and stored into a color image I respectively₂And a color image I₃；

S104, dividing the fruit clusters into single fruit clusters, double fruit clusters and multi-fruit clusters according to the distribution positions of the fruits, and further determining a color image I₁The fruit cluster type of the randomly distributed fruits;

s105, determining mother branches according to the fruit string type determined in the step S104:

for a single fruit cluster, the branch connected with the fruit is the mother branch;

for the double fruit cluster and the multi-fruit cluster, firstly, in the color image I₁The middle making fruit string is combined with the color image I through the perpendicular bisector of the upper and lower bottoms of the outline circumscribed rectangle₁With color images I₂Performing fusion analysis by color image I₁Perpendicular bisector and color image I₂Determining the parent branch of the fruit bunch according to the branch tangency condition;

s106, acquiring a binocular stereo image of the fruit cluster mother branches by using a binocular CCD camera, taking a geometric central point of a mother branch circumscribed rectangle in a left image of the binocular CCD camera as a feature matching point, searching a point which is closest to a gray value of the feature matching point and enables a normalized cross-correlation function to reach a maximum value in a right image, realizing feature matching, and obtaining the geometric central point of the mother branches; finally, calculating the spatial coordinates of the geometric center point of the mother branch to realize the pre-positioning of the mother branch, and obtaining a pre-positioning image and a pre-positioning geometric center point of the mother branch;

s200, in the picking process, acquiring an actual positioning image and an actual positioning geometric center point of a mother branch in the picking process by using binocular vision by using the method in the step S100; then, comparing the actual positioning image and the pre-positioning image of the mother branch by using a monocular vision zooming method, and determining the type of the interference on the mother branch;

s300, performing active anti-interference by using aerodynamic force according to the type of the parent branch interfered determined in the step S200;

s400, determining a feeding terminal point of the mechanical arm of the picking robot based on the active disturbance rejection method in the step S300; and determining a feeding path of a mechanical arm of the picking robot according to the feeding end point, and picking the string-shaped fruits.

The fruit cluster type classification and the mother branch determination by using the perpendicular bisector of the through-string contour line circumscribed rectangle are adopted, so that firstly, the interference of branches connected with each fruit can be effectively eliminated, and the recognition precision is improved; secondly, the mother branches of the fruit clusters are quickly identified, so that the identification efficiency is improved, and the identification time is shortened.

For further optimization, in step S104, a fruit cluster classification principle is adopted to classify single fruit clusters, double fruit clusters and multi-fruit clusters;

the method specifically comprises the following steps: single fruit bunch: if the Euclidean distance between the geometric center of one fruit and the geometric center of any other fruit is larger than the average diameter of a single fruit, the fruit is a single fruit cluster; double fruit bunch: if the Euclidean distance between the geometric centers of two adjacent fruits is smaller than the sum of the diameters of the two fruits, the two fruits are a double-fruit cluster; multi-fruit clusters: if the Euclidean distance between the geometric centers of any two fruits in more than two fruits is smaller than the sum of the diameters of the two fruits, the group of fruits is a multi-fruit cluster.

For further optimization, the color image I is passed through in the step S105₁Perpendicular bisector and color image I₂The method is characterized in that the method comprises the following steps of determining the mother branch of a fruit cluster according to the branch tangency condition: and rotating the perpendicular bisector of the upper and lower bottoms of the outline circumscribed rectangle of the fruit cluster around a point below the perpendicular bisector to one side close to the branches, wherein the branch which is firstly tangent to the perpendicular bisector is the parent branch of the fruit cluster.

Further optimization, the step of searching the point which is closest to the gray value of the feature matching point and enables the normalized cross-correlation function to reach the maximum value in the right image in the step S106 specifically includes:

first, a left image P is obtained by a binocular CCD camera₁The feature matching point (x, y) of (a) constructs a matching window while the right image P is taken₂The point (x + m, y + m) to be matched also constructs a matching window, and measures the correlation degree by normalizing the correlation function, wherein the specific formula is as follows:

wherein F (m, n) represents a normalized correlation function; (m, n) represents a position vector of the right image relative to the left image; w_pRepresenting a matching window taking the coordinates of the point to be matched as a center; p₂(x + m, y + n) represents the gray value of the point to be matched of the right image;representing a gray level mean of the right image matching window; p₁(x, y) represents the gray value of the left image feature matching point;representing a gray level mean of the left image matching window;

wherein, the value range of F (m, n) is [ -1,1 ];

when F (m, n) is-1, the feature matching point of the left image is completely unrelated to the point to be matched of the right image;

when F (m, n) is 1, the feature matching point of the left image is completely matched with the point to be matched of the right image, that is, the point to be matched is the point at which the gray value of the feature matching point is closest and the normalized cross-correlation function of the point to be matched reaches the maximum value.

Preferably, the matching window is typically a 3 x 3 matching window.

Further preferably, the left image and the right image are subjected to epipolar line correction before feature matching is carried out, so that the epipolar lines of the left image and the right image are in the horizontal direction, even if the optical centers of the left image and the right image are in the same horizontal line.

Preferably, the epipolar correction is performed using the Bouguet epipolar correction algorithm.

For further optimization, in the step S106, a triangulation principle is adopted to calculate the spatial coordinates of the geometric center point of the parent branch.

For further optimization, the step S200 specifically includes:

s201, firstly, judging whether the parent branches of the fruit clusters in the actual positioning image are blocked or not through whether a binocular vision camera identifies the parent branches of the fruit clusters in the actual positioning image or not;

if the fruit cluster mother branches are not identified in the actual positioning image, the pre-positioning image is superposed on the actual positioning image, and whether eight neighborhood pixels of the pre-positioning image and the actual positioning image belong to the same thing is judged through a monocular vision zooming method;

if the actual positioning image belongs to the tree branch positioning image, judging that the parent branch in the actual positioning image is shielded by the tree branch;

if not, further positioning the imageEight neighborhood pixels and color image I₁Color image I₃Comparing the eight neighborhood pixels, and judging whether the eight neighborhood pixels are shielded by fruits or leaves;

s202, if the fruit string mother branch is identified in the actual positioning image, the mother branch in the actual positioning image is not shielded, and then the pre-positioning geometric center point O in the pre-positioning image is determined₁And an actual positioning geometric center point O in the actual positioning image₂Relative value d (i.e. O) of deviation therebetween₁And O₂The linear distance between the two points) and judging whether the mother branch deviates or not according to the relation between the deviation relative value d and a preset threshold value;

if the deviation relative value d is not greater than the preset threshold value, the interference is not generated;

if the deviation relative value d is larger than the preset threshold value, and O₂The point is zoomed by a monocular vision zooming method, the point belongs to a branch through judgment of a multi-dimensional SVM classifier, and then O is determined₂The parent branch represented by the point is relative to O under the interference of dynamic factors₁The position of the point is shifted; if the relative deviation value d is greater than the preset threshold value, and O₂And if the point is judged not to belong to a branch by the multi-dimensional SVM classifier through the eight-neighborhood pixels zoomed by the monocular vision zooming method, outputting an error signal.

Preferably, the power factor is wind power, gravity and the like.

Further optimization is carried out, the preset threshold is obtained by prepositioning a circumscribed rectangle of the image parent branch, and the method specifically comprises the following steps:

firstly, constructing an external rectangular frame of a parent branch in a pre-positioned image; then obtaining the number p of bottom edge pixels of the external rectangular frame and the total number a of horizontal lines of pixels of the pre-positioned image; finally obtaining a preset threshold d₀(ii) a The concrete formula is as follows:

for further optimization, the step S300 specifically includes:

if the mother branch is shielded, firstly separating the shielding object from the target mother branch by using aerodynamic force; then, returning the geometric center point of the parent branch to the position of the pre-positioned geometric center point by using aerodynamic force (setting the direction, strength and the like of aerodynamic force), recording the motion trail of the geometric center point of the parent branch and the points which are farthest and closest to the feeding starting point in a camera plane by using a single-binocular vision camera and a binocular vision camera to obtain the motion period of the geometric center point of the parent branch, simultaneously obtaining the spatial point cloud of the shielding object by using the binocular vision camera, constructing a spatial enclosure which is completely tangent to the shielding object, and recording the motion trail of the spatial enclosure and the points which are farthest and closest to the feeding starting point in the camera plane by using the single-binocular vision camera and the binocular vision camera to obtain the motion period of the shielding object surrounded by the spatial enclosure;

if the mother branch is not shielded and only deviates, returning the geometric center point of the mother branch to the position of the geometric center point of the preset position by using aerodynamic force (setting the direction, the strength and the like of aerodynamic force), and recording the motion trail of the geometric center point of the mother branch and the points farthest from and closest to the feeding starting point in the plane of the camera by using single and double-eye vision cameras to obtain the motion period of the geometric center point of the mother branch.

For further optimization, the step S400 specifically includes:

s401, determination of a feed end point:

if the parent branch is not shielded and only deviates, the space coordinate of the movement locus of the geometric central point of the parent branch to the nearest point of the plane of the camera to the feeding starting point is the feeding terminal point of the mechanical arm of the picking robot;

if the mother branch is shielded, determining the maximum space redundancy between the mother branch and the shielding object at the farthest position and the nearest position of the camera plane from the feeding starting point respectively through the geometric center point of the mother branch and the space surrounding body of the shielding object, wherein the center of the maximum space redundancy is the feeding end point of the mechanical arm of the picking robot; the space redundancy is a circle formed by connecting a geometric center point of the parent branch and a space surrounding body of the shielding object at the farthest position (point) and the nearest position (point) of the camera plane from the feeding starting point respectively by diameters, and the center of the circle is the feeding end point;

s402, determining a feeding path:

firstly, constructing a space enclosure of an obstacle in a potential path from a feeding starting point to a feeding destination by adopting a binocular vision camera and a space geometric principle; then respectively defining the space enclosing body as a first obstacle, a second obstacle, a third obstacle, … … and an nth obstacle from the position close to the feeding starting point to the position far away from the feeding starting point, making a tangent line of the first obstacle through the feeding starting point, taking a tangent point as a first path point, making a vertical line segment of a connecting line of the second obstacle and the geometric center of the third obstacle through the first path point, taking the vertical point as a second path point, making a tangent line of the third obstacle through the second path point, taking the tangent point as a third path point, and sequentially circulating until the final path point is directly connected with the feeding terminal point … …; the connecting line of the feeding starting point, each path point and the feeding end point is the feeding path.

Preferably, the feeding path needs to be fused with a coordinate system and a camera coordinate system of the picking robot, and the spatial position and the posture of the picking robot mechanical arm clipping mother branches are determined based on variables of each joint of the picking robot.

The invention has the following technical effects:

the method can quickly, effectively and accurately identify the mother branches of the string-shaped fruits by combining the monocular camera and the binocular camera, can accurately judge the type of interference borne by the mother branches by a view zooming mode, and can effectively solve the interference of the mother branches due to power factors through aerodynamics; meanwhile, the invention can accurately and effectively obtain the optimal feeding path by determining the parent branches, the interference types and the anti-interference action, thereby realizing the minimum energy, the shortest path and the highest efficiency in the picking process and ensuring the lowest overall cost.

The method disclosed by the invention has the advantages that the mother branches are effectively determined, and the problem that the serial fruits are scattered or the fruits are easy to fall off due to the fact that the mother branches cannot be accurately cut during picking is effectively avoided, so that the subsequent storage and transportation are influenced; by means of the air power, interference resistance can be achieved, the situations of wrong shearing, missed shearing and the like in the picking process can be effectively avoided, and picking failure or damage to fruits is avoided; through the analog calculation of the feeding path, the minimum overall energy consumption is ensured, the picking efficiency is improved, and the picking time is saved.

The invention realizes the complete intelligent picking in the picking process, automatically performs photographing and analysis on fruit trees, confirms the spatial coordinate point of the mother branch, identifies the interference type of the mother branch in the picking process, establishes the feeding end point and calculates the feeding path, thereby improving the picking work efficiency and the yield of the string-shaped fruits and reducing the picking labor force.

Drawings

FIG. 1 is a flow chart of the method for identifying the fruit main branch and determining the interference type of the main branch according to the embodiment of the present invention.

FIG. 2 is a flow chart of determining a feed end point and a feed path according to an embodiment of the present invention.

Fig. 3 is a schematic diagram of the principle of identifying the string-type fruit mother branch in the embodiment of the invention.

Fig. 4 is a schematic diagram illustrating a principle of determining a parent branch interference type according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of a picking robot path planning in an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example (b):

as shown in fig. 1 to 5, a litchi picking method suitable for stringed fruits is characterized in that:

s100, pre-positioning the mother branch by matching a monocular camera and a binocular camera, and the method comprises the following specific steps:

s101: randomly acquiring a plurality of color images of the fruits, the leaves and the branches of the fruits in the string form by adopting a monocular CCD camera; selecting and dividing a plurality of fruit targets and non-fruit targets in the color image, and respectively extracting texture characteristic values and color characteristic values of the fruit targets and the non-fruit targets as positive and negative samples;

s102, training positive and negative samples by adopting a Support Vector Machine (SVM) to generate a plurality of weak classifiers; then, an AdaBoost algorithm is adopted to construct a strong classifier, the color image acquired by the monocular CCD camera is segmented by the strong classifier, and the fruit target identified from the image is separately stored into a color image I₁；

S103, repeating the steps S101 to S102 to respectively obtain strong classifiers for identifying branches and leaves; then respectively removing the image I₁The branches and leaves are extracted from the original color image and stored into a color image I respectively₂And a color image I₃；

S104, dividing the fruit clusters into single fruit clusters, double fruit clusters and multi-fruit clusters according to the distribution positions of the fruits and by adopting a fruit cluster classification principle, and further determining a color image I₁The fruit cluster type of the randomly distributed fruits;

the method specifically comprises the following steps: single fruit bunch: if the Euclidean distance between the geometric center of one fruit and the geometric center of any other fruit is larger than the average diameter of a single fruit, the fruit is a single fruit cluster; double fruit bunch: if the Euclidean distance between the geometric centers of two adjacent fruits is smaller than the sum of the diameters of the two fruits, the two fruits are a double-fruit cluster; multi-fruit clusters: if the Euclidean distance between the geometric centers of any two fruits in more than two fruits is smaller than the sum of the diameters of the two fruits, the group of fruits is a multi-fruit cluster.

S105, determining mother branches according to the fruit string type determined in the step S104:

for a single fruit cluster, the branch connected with the fruit is the mother branch;

for the double fruit cluster and the multi-fruit cluster, firstly, in the color image I₁The middle making fruit string is combined with the color image I through the perpendicular bisector of the upper and lower bottoms of the outline circumscribed rectangle₁With color images I₂Performing fusion analysis by color image I₁Perpendicular bisector and color image I₂Determining the parent branch of the fruit bunch according to the branch tangency condition; the method specifically comprises the following steps: winding the perpendicular bisector of the upper and lower bottoms of the outline circumscribed rectangle of the fruit cluster around the point below the perpendicular bisector,Rotating to one side close to the branches, wherein the branch which is firstly tangent to the perpendicular bisector is the mother branch of the fruit bunch.

S106, acquiring binocular stereo images of fruit cluster mother branches by using a binocular CCD camera, taking a mother branch circumscribed rectangle geometric central point in a left image of the binocular CCD camera as a feature matching point, and searching a point which is closest to the gray value of the feature matching point and enables a normalized cross-correlation function to reach the maximum value in a right image, wherein the method specifically comprises the following steps:

first, a left image P is obtained by a binocular CCD camera₁The feature matching point (x, y) of (a) constructs a matching window while the right image P is taken₂The point (x + m, y + m) to be matched also constructs a matching window, and measures the correlation degree by normalizing the correlation function, wherein the specific formula is as follows:

wherein F (m, n) represents a normalized correlation function; (m, n) represents a position vector of the right image relative to the left image; w_pRepresenting a matching window taking the coordinates of the point to be matched as a center; p₂(x + m, y + n) represents the gray value of the point to be matched of the right image;representing a gray level mean of the right image matching window; p₁(x, y) represents the gray value of the left image feature matching point;representing a gray level mean of the left image matching window;

wherein, the value range of F (m, n) is [ -1,1 ];

when F (m, n) is-1, the feature matching point of the left image is completely unrelated to the point to be matched of the right image;

when F (m, n) is 1, the feature matching point of the left image is completely matched with the point to be matched of the right image, that is, the point to be matched is the point at which the gray value of the feature matching point is closest and the normalized cross-correlation function of the point to be matched reaches the maximum value. Realizing the characteristic matching of the left image and the right image to obtain the geometric central point of the mother branch;

the matching window is typically a 3 x 3 matching window.

Epipolar line correction is needed to be carried out on the left image and the right image before feature matching is carried out on the left image and the right image, so that the epipolar lines of the left image and the right image are in the horizontal direction, even if the optical centers of the left image and the right image are in the same horizontal line; epipolar rectification was performed using the Bouguet epipolar rectification algorithm.

And finally, calculating the space coordinate of the geometric center point of the mother branch, realizing pre-positioning of the mother branch, and obtaining a pre-positioning image and the pre-positioning geometric center point of the mother branch.

S200, in the picking process, acquiring an actual positioning image and an actual positioning geometric center point of a mother branch in the picking process by using the method in the step S100 and a binocular vision camera; then, comparing the actual positioning image and the pre-positioning image of the mother branch by using a monocular vision zooming method, and determining the type of the interference on the mother branch;

the method specifically comprises the following steps:

s201, firstly, judging whether the parent branches of the fruit clusters in the actual positioning image are blocked or not through whether a binocular vision camera identifies the parent branches of the fruit clusters in the actual positioning image or not;

if the fruit cluster mother branches are not identified in the actual positioning image, superposing the pre-positioning image on the actual positioning image, and judging whether eight neighborhood pixels of the pre-positioning image and the actual positioning image belong to the same thing or not by a monocular vision zooming method;

if the actual positioning image belongs to the tree branch positioning image, judging that the parent branch in the actual positioning image is shielded by the tree branch;

if not, further locating eight neighborhood pixels of the image and the color image I₁Color image I₃Comparing the eight neighborhood pixels, and judging whether the eight neighborhood pixels are shielded by fruits or leaves;

as shown in FIG. 3, O₁The' point is the pre-positioning geometric central point of the mother branch in the pre-positioning image, but is O-shaped in the picking process due to external dynamic factors₁The point is the shielding of the mother branch of the prepositioned geometric central point, so that in the actual positioning image, O is used₂' Point is the mother branch quilt of the geometric centre point of actual locationOcclusion, judging that the same object of eight adjacent pixels is covered by O₂The point is the branch shelter of the mother branch at the prepositioned geometric central point.

S202, if the fruit string mother branch is identified in the actual positioning image, the mother branch in the actual positioning image is not shielded, and then the pre-positioning geometric center point O in the pre-positioning image is determined₁And an actual positioning geometric center point O in the actual positioning image₂Relative value d (i.e. O) of deviation therebetween₁And O₂The linear distance between the two points) and judging whether the mother branch deviates or not according to the relation between the deviation relative value d and a preset threshold value;

if the deviation relative value d is not larger than the preset threshold value, the operation is considered to be non-interference (namely the interference without power factors is the deviation of the parent branch);

if the deviation relative value d is larger than the preset threshold value, and O₂The point is zoomed by a monocular vision zooming method, the point belongs to a branch through judgment of a multi-dimensional SVM classifier, and then O is determined₂The parent branch represented by the point is relative to O under the interference of dynamic factors₁The position of the point is shifted; if the relative deviation value d is greater than the preset threshold value, and O₂And (4) outputting an error signal if the point is judged not to belong to a branch by a multi-dimensional SVM classifier through the eight-neighborhood pixels zoomed by the monocular vision zooming method (after the error signal alarms, the error signal is analyzed and judged manually).

The power factors are wind power, gravity and the like.

The preset threshold is obtained by the external rectangle of the pre-positioned image parent branch, and specifically comprises the following steps:

firstly, constructing an external rectangular frame of a parent branch in a pre-positioned image; then obtaining the number p of the bottom edge pixels of the circumscribed rectangle frame and the total number a of the horizontal rows (namely each row of pixels) of the pixels of the pre-positioned image; finally obtaining a preset threshold d₀(ii) a The concrete formula is as follows:

wherein d is₀Is calculated in relation to the image pixel, i.e. the value of a influences d₀The calculation result of (2); meanwhile, the preset threshold value is related to the type of the fruit in the cluster, namely the thickness of the mother branch influences the final p value.

Taking litchi as an example, if the image pixel is 640x480 and the bottom edge of the circumscribed rectangle of the parent branch of litchi is 22, the preset threshold value of litchi can be calculated to be 0.03.

S300, performing active anti-interference by using aerodynamic force according to the parent branch interfered type determined in the step S200, specifically:

if the mother branch is shielded, firstly separating the shielding object from the target mother branch by using aerodynamic force; then, enabling the geometric center point of the parent branch to return to the position of the pre-positioned geometric center point by utilizing aerodynamic force (direction, strength and the like of the aerodynamic force), recording the motion track of the geometric center point of the parent branch and the points which are farthest and closest to the feeding starting point in a camera plane by utilizing a single-binocular vision camera and a binocular vision camera to obtain the motion cycle of the geometric center point of the parent branch, simultaneously obtaining the spatial point cloud (namely a shielding object space three-dimensional coordinate set) of a shielding object by utilizing the binocular vision camera, constructing a spatial enclosure which is completely tangent to the shielding object, recording the motion track of the spatial enclosure and the points which are farthest and closest to the feeding starting point in the camera plane by utilizing the single-binocular vision camera and the binocular vision camera, and obtaining the motion cycle of the shielding object;

if the mother branch is not shielded and only deviates, returning the geometric center point of the mother branch to the position of the geometric center point of the preset position by using aerodynamic force (setting the direction, the strength and the like of aerodynamic force), and recording the motion trail of the geometric center point of the mother branch and the points farthest from and closest to the feeding starting point in the plane of the camera by using single and double-eye vision cameras to obtain the motion period of the geometric center point of the mother branch.

S400, determining a feeding terminal point of the mechanical arm of the picking robot based on the active disturbance rejection method in the step S300; according to the terminal point of feeding, confirm the feed path of picking robot robotic arm, carry out the picking of string type fruit, specifically do:

s401, determination of a feed end point:

if the parent branch is not shielded and only deviates, the space coordinate of the movement locus of the geometric central point of the parent branch to the nearest point of the plane of the camera to the feeding starting point is the feeding terminal point of the mechanical arm of the picking robot;

if the mother branch is shielded, determining the maximum space redundancy between the mother branch and the shielding object at the farthest position and the nearest position of the camera plane from the feeding starting point respectively through the geometric center point of the mother branch and the space surrounding body of the shielding object, wherein the center of the maximum space redundancy is the feeding terminal point of the mechanical arm of the picking robot; the space redundancy is that the connecting lines of the geometric center point of the parent branch and the space surrounding body of the shielding object at the farthest position (point) and the nearest position (point) of the camera plane from the feeding starting point are circles formed by diameters, and the circle center is the feeding end point;

s402, determining a feeding path:

firstly, constructing a space enclosure of an obstacle in a potential path from a feeding starting point to a feeding destination by adopting a binocular vision camera and a space geometric principle; then respectively defining the space enclosing body as a first obstacle, a second obstacle, a third obstacle, … … and an nth obstacle from the position close to the feeding starting point to the position far away from the feeding starting point, making a tangent line of the first obstacle through the feeding starting point, taking a tangent point as a first path point, making a vertical line segment of a connecting line of the second obstacle and the geometric center of the third obstacle through the first path point, taking the vertical point as a second path point, making a tangent line of the third obstacle through the second path point, taking the tangent point as a third path point, and … … sequentially circulating until the final path point is directly connected with the feeding terminal point; the connecting line of the feeding starting point, each path point and the feeding end point is the feeding path.

The feeding path needs to be fused with a coordinate system and a camera coordinate system of the picking robot, and the spatial position and the posture of the picking robot mechanical arm for shearing and clamping the mother branches are determined based on variables of all joints of the picking robot.

The fruit cluster type classification and the mother branch determination by using the perpendicular bisector of the through-string contour line circumscribed rectangle are adopted, so that firstly, the interference of branches connected with each fruit can be effectively eliminated, and the recognition precision is improved; secondly, the mother branches of the fruit clusters are quickly identified, so that the identification efficiency is improved, and the identification time is shortened.

The method effectively improves the intelligent level in the picking process of the string-shaped fruits, automatically takes pictures and analyzes the fruit trees, confirms the spatial coordinate point of the mother branch and identifies the interference type of the mother branch in the picking process, improves the picking work efficiency and the yield of the string-shaped fruits, and avoids the actions of damaging the string-shaped fruits and cutting by mistake in the picking process.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.