Distance measuring method and apparatus

文档序号：816980 发布日期：2021-03-26 浏览：46次中文

阅读说明：本技术 距离测量方法以及设备 (Distance measuring method and apparatus ) 是由周游刘洁严嘉祺于 2018-08-21 设计创作，主要内容包括：一种使用无人飞行器(UAV)(102)测量距离的方法,包括：识别待测量的目标对象(106)(S502)；当UAV(102)正在移动并且相机(1022)正在跟踪目标对象(106)时,接收由UAV(102)的相机(1022)捕获的多个图像(S504)；采集与多个图像的捕获时刻相对应的UAV(106)的移动信息(S506)；以及根据移动信息和多个图像,计算目标对象(106)与UAV(102)之间的距离(S508)。(A method of measuring distance using an Unmanned Aerial Vehicle (UAV) (102), comprising: identifying a target object (106) to be measured (S502); receiving a plurality of images (S504) captured by a camera (1022) of the UAV (102) while the UAV (102) is moving and the camera (1022) is tracking the target object (106); acquiring movement information (S506) of the UAV (106) corresponding to a time of capture of the plurality of images; and calculating a distance (S508) between the target object (106) and the UAV (102) from the movement information and the plurality of images.)

1. A method for measuring distance using an unmanned aerial vehicle UAV, comprising:

identifying a target object to be measured;

receiving a plurality of images captured by the camera of the UAV while the UAV is moving and a camera is tracking the target object;

acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and

calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

2. The method of claim 1, wherein identifying the target object comprises:

receiving an initial image captured by the camera of the UAV containing the target object; and

the target object is identified in the initial image.

3. The method of claim 2, wherein identifying the target object further comprises:

displaying the initial image on a graphical user interface;

obtaining a user selection of a target region in the initial image; and

obtaining the target object based on the target area.

4. The method of claim 3, wherein displaying the initial image comprises:

displaying the initial image on the graphical user interface on a screen of a remote control of the UAV.

5. The method of claim 3, wherein the user selection comprises: clicking on the center of the target area, double clicking on the center of the target area, or a drag operation with a start point and an end point, wherein the start point and the end point define a bounding box of the target area.

6. The method of claim 3, wherein identifying the target object comprises:

obtaining superpixels of the initial image by clustering pixels of the initial image based on image features of the pixels;

obtaining one or more super-pixels located in the target region; and

identifying an image region formed by the one or more superpixels as a region representing the target object.

7. The method of claim 6, wherein obtaining the one or more superpixels located in the target region comprises:

obtaining superpixels partially located in the target region;

determining a percentage by dividing a number of pixels in the superpixel that are located within the target region by a total number of pixels in the superpixel; and

in response to the percentage being greater than a preset threshold, determining that the superpixel is located in the target region.

8. The method of claim 6, wherein the image characteristic of the pixel comprises at least one of a texture, a color, or a brightness of the pixel.

9. The method of claim 1, further comprising:

after identifying the target object, determining whether the target object is a moving object using a Convolutional Neural Network (CNN);

wherein, in response to the target object being determined as a moving object, a warning message is given indicating impaired measurement accuracy.

10. The method of claim 1, further comprising:

after the target object is identified, extracting target feature points corresponding to the target object; and

determining whether the number of the target feature points is smaller than a preset number threshold;

wherein, in response to the number of target feature points being less than the preset number threshold, a warning message is given indicating impaired measurement accuracy.

11. The method of claim 1, further comprising:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

12. The method of claim 11, wherein the curved path corresponds to a circle centered at or near the target object.

13. The method of claim 11, wherein determining the initial radius comprises:

estimating a distance to the target object based on image data acquired by a stereo camera of the UAV.

14. The method of claim 11, wherein determining the initial radius comprises:

using a preset value as the initial radius, the preset value being a maximum distance that the UAV can measure.

15. The method of claim 11, further comprising:

determining a position of the target object in one of the captured plurality of images; and

adjusting at least one of an attitude of a pan-tilt carrying the camera or a speed of the UAV based on the position of the target object.

16. The method of claim 1, further comprising:

obtaining readings from a gyroscope of the UAV while the UAV is moving and the camera is tracking the target object;

determining whether the UAV is in a steady state of movement based on readings of the gyroscope and accelerometer; and

calculating a distance between the target object and the UAV using the plurality of images captured while the UAV is in the steady state of motion.

17. The method of claim 1, further comprising:

obtaining a plurality of estimated camera poses based on the movement information corresponding to the capturing moments of the plurality of images, each of the plurality of images corresponding to one of the estimated camera poses.

18. The method of claim 17, wherein acquiring movement information of the UAV comprises:

acquiring, by an Inertial Measurement Unit (IMU) of the UAV, pose information of the UAV, the pose information including an orientation and a position of the UAV.

19. The method of claim 18, further comprising:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

20. The method of claim 19, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame;

the method further comprises the following steps:

in response to the first camera pose relationship not satisfying the preset condition, obtaining a second camera pose relationship between the key frame and a second image frame captured after the first image frame;

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

21. The method of claim 19, further comprising:

after selecting the image frame as one of the plurality of images, using the image frame as the key frame, and determining whether to select another image frame captured after the image frame as one of the plurality of images based on whether a camera pose relationship between the image frame and the another image frame satisfies the preset condition.

22. The method of claim 19, wherein:

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

the camera pose relationship comprises at least one of a rotational change from a time the keyframe is captured to a time the image frame is captured or a positional change of the camera from a time the keyframe is captured to a time the image frame is captured; and

determining whether the camera pose relationship satisfies the preset condition comprises at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

Determining that the camera pose relationship satisfies the preset condition in response to the rotation change being less than the rotation threshold and the position change being greater than the displacement threshold.

23. The method of claim 19, wherein obtaining a plurality of estimated camera poses comprises:

obtaining a current estimated camera pose corresponding to a current image frame based on a previous estimated camera pose corresponding to the previous image frame and the movement information corresponding to the current image frame, the current image frame and the previous image frame captured while the UAV is moving.

24. The method of claim 23, wherein the previous image frame and the current image frame are continuously captured by the camera while the UAV is moving.

25. The method of claim 24, wherein acquiring the camera pose relationship between the keyframe and the image frame comprises:

accumulating camera pose relationships between each pair of successive image frames captured from the keyframe to the image frame.

26. The method of claim 17, further comprising:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

obtaining three-dimensional (3D) positions and refined camera pose information of the plurality of feature points based on the 2D positions of the plurality of feature points in the plurality of images, the plurality of estimated camera poses corresponding to the capturing moments of the plurality of images, and state information of a pan-tilt head of the camera carrying the UAV corresponding to the capturing moments of the plurality of images; and

calculating a distance between the target object and the UAV from a 3D position of one or more of the feature points associated with the target object and a 3D position of the camera indicated by refined camera pose information.

27. The method of claim 26, wherein tracking the 2D locations of the plurality of feature points comprises:

tracking a displacement of the plurality of feature points between each two consecutive images of the plurality of images; and

obtaining optical flow vectors of the plurality of feature points from the tracked displacement.

28. The method of claim 27, wherein:

the plurality of feature points includes a center point of the target object;

tracking the 2D locations of the plurality of feature points in the plurality of images comprises: tracking a 2D position of a center point of the target object in the plurality of images based on optical flow vectors of a plurality of target feature points identified from the plurality of feature points, the target feature points being within a region of the target object;

obtaining the 3D positions of the plurality of feature points comprises: obtaining a 3D position of a central point in the plurality of images based on a 2D position of the central point and a plurality of estimated camera poses corresponding to capturing moments of the plurality of images; and

wherein calculating the distance between the target object and the UAV comprises: calculating the distance from the 3D position of the center point and the 3D position of the camera indicated by the refined camera pose information.

29. The method of claim 26, wherein tracking the 2D location of the center point comprises:

determining a positional relationship between the center point and the target feature point in the plurality of images;

assigning a weight corresponding to the optical flow vector of the target feature point according to the positional relationship; and

and fitting the optical flow vector of the central point based on the optical flow vector of the target feature point and the corresponding weight.

30. The method of claim 24, wherein obtaining the 3D positions and refined camera pose information for the plurality of feature points comprises:

refining the 3D positions of the plurality of feature points and the plurality of estimated camera poses simultaneously by solving an optimization problem based on a beam adjustment algorithm that minimizes gross projection errors; and

obtaining the 3D positions of the plurality of feature points and refined camera pose information from the optimal solution of the optimization problem.

31. The method of claim 1, further comprising:

after the distance is calculated, the distance is displayed on a graphical user interface.

32. The method of claim 31, further comprising:

displaying the plurality of images in real-time on the graphical user interface; and

marking the distance on an image currently displayed on the graphical user interface.

33. The method of claim 32, further comprising:

updating, in real-time, a distance between the target object and the UAV based on a second image captured by the camera and movement information corresponding to a time of capture of the second image.

34. The method of claim 1, further comprising:

calculating a size of the target object based on a distance between the target object and the UAV.

35. The method of claim 34, wherein the size of the target object comprises at least one of a length of the target object or a width of the target object.

36. The method of any of claims 1-35, wherein the plurality of images are captured while the target object is stationary.

37. The method of any of claims 1-35, wherein the plurality of images are captured while the target object is moving and a background object of the target object is stationary.

38. The method of any of claims 1-35, wherein the movement information of the UAV includes data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

39. A system for measuring distance using an unmanned aerial vehicle UAV, comprising:

a camera of the UAV;

at least one memory; and

at least one processor, wherein:

the at least one processor is configured to identify a target object to be measured;

the camera is configured to capture a plurality of images while the UAV is moving and the camera is tracking the target object;

the at least one processor is further configured to: acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

40. The system of claim 39, wherein:

the camera of the UAV is further configured to capture an initial image containing the target object; and

the at least one processor is further configured to identify the target object in the initial image.

41. The system of claim 40, wherein the at least one processor is further configured to:

displaying the initial image on a graphical user interface;

obtaining a user selection of a target region in the initial image; and

obtaining the target object based on the target area.

42. The system of claim 41, wherein the at least one processor is further configured to display the initial image by:

displaying the initial image on the graphical user interface on a screen of a remote control of the UAV.

43. The system of claim 41, wherein the user selection comprises: clicking the center of the target area, double clicking the center of the target area, or a drag operation with a start point and an end point, which define a bounding box of the target area.

44. The system of claim 41, wherein the at least one processor is configured to identify the target object by:

obtaining superpixels of the initial image by clustering pixels of the initial image based on image features of the pixels;

obtaining one or more super-pixels located in the target region; and

identifying a region formed by the one or more superpixels as a region representing the target object.

45. The system of claim 44, wherein the at least one processor is further configured to obtain the one or more superpixels located in the target region by:

obtaining superpixels partially located in the target region;

determining a percentage by dividing a number of pixels in the superpixel that are located within the target region by a total number of pixels in the superpixel; and

in response to the percentage being greater than a preset threshold, determining that the superpixel is located in the target region.

46. The system of claim 44, wherein the image characteristic of the pixel comprises at least one of a texture, a color, or a brightness of the pixel.

47. The system of claim 39, wherein the at least one processor is further configured to:

after identifying the target object, determining whether the target object is a moving object using a Convolutional Neural Network (CNN);

wherein, in response to the target object being determined as a moving object, a warning message is given indicating impaired measurement accuracy.

48. The system of claim 39, wherein the at least one processor is further configured to:

after the target object is identified, extracting target feature points corresponding to the target object; and

determining whether the number of the target feature points is smaller than a preset number threshold;

wherein, in response to the number of target feature points being less than the preset number threshold, a warning message is given indicating impaired measurement accuracy.

49. The system of claim 39, wherein the at least one processor is further configured to:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

50. The system of claim 49, wherein the curved path corresponds to a circle centered at or near the target object.

51. The system of claim 49, wherein the at least one processor is configured to determine the initial radius by:

estimating a distance to the target object based on image data acquired by a stereo camera of the UAV.

52. The system of claim 49, wherein the at least one processor is further configured to determine the initial radius by:

using a preset value as the initial radius, the preset value being a maximum distance that the UAV can measure.

53. The system of claim 49, wherein the at least one processor is further configured to:

determining a position of the target object in one of the captured plurality of images; and

adjusting at least one of an attitude of a pan-tilt carrying the camera or a speed of the UAV based on the position of the target object.

54. The system of claim 39, wherein the at least one processor is further configured to:

obtaining readings from a gyroscope of the UAV while the UAV is moving and the camera is tracking the target object;

determining whether the UAV is in a steady state of movement based on readings of the gyroscope and accelerometer; and

calculating a distance between the target object and the UAV using the plurality of images captured while the UAV is in the steady state of motion.

55. The system of claim 39, wherein the at least one processor is further configured to:

56. The system of claim 55, wherein the at least one processor is configured to acquire movement information of the UAV by:

acquiring, by an Inertial Measurement Unit (IMU) of the UAV, pose information of the UAV, the pose information including an orientation and a position of the UAV.

57. The system of claim 56, wherein the at least one processor is further configured to:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

58. The system of claim 57, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame;

wherein the at least one processor is further configured to:

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

59. The system of claim 57, wherein the at least one processor is further configured to:

60. The system of claim 57, wherein:

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

the at least one processor is further configured to determine whether the camera pose relationship satisfies the preset condition by at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

61. The system of claim 57, wherein the at least one processor is configured to obtain a plurality of estimated camera poses by:

62. The system of claim 61, wherein the previous image frame and the current image frame are continuously captured by the camera while the UAV is moving.

63. The system of claim 62, wherein the at least one processor is further configured to obtain the camera pose relationship between the keyframe and the image frame by:

accumulating camera pose relationships between each pair of successive image frames captured from the keyframe to the image frame.

64. The system of claim 55, wherein the at least one processor is further configured to:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

65. The system of claim 64, wherein the at least one processor is further configured to track 2D locations of the plurality of feature points by:

tracking a displacement of the plurality of feature points between each two consecutive images of the plurality of images; and

obtaining optical flow vectors of the plurality of feature points from the tracked displacement.

66. The system of claim 65, wherein:

the plurality of feature points includes a center point of the target object;

67. The system of claim 64, wherein the at least one processor is further configured to track the 2D position of the center point by:

determining a positional relationship between the center point and the target feature point in the plurality of images;

assigning a weight corresponding to the optical flow vector of the target feature point according to the positional relationship; and

and fitting the optical flow vector of the central point based on the optical flow vector of the target feature point and the corresponding weight.

68. The system of claim 62, wherein the at least one processor is further configured to obtain the 3D positions and refined camera pose information for the plurality of feature points by:

obtaining the 3D positions of the plurality of feature points and refined camera pose information from the optimal solution of the optimization problem.

69. The system of claim 39, wherein the at least one processor is further configured to:

after the distance is calculated, displaying the distance on the graphical user interface.

70. The system of claim 69, wherein the at least one processor is further configured to:

displaying the plurality of images in real-time on the graphical user interface; and

marking the distance on an image currently displayed on the graphical user interface.

71. The system of claim 70, wherein the at least one processor is further configured to:

72. The system of claim 39, wherein the at least one processor is further configured to:

calculating a size of the target object based on a distance between the target object and the UAV.

73. The system of claim 72, wherein the size of the target object comprises at least one of a length of the target object or a width of the target object.

74. The system of any one of claims 39-73, wherein the plurality of images are captured while the target object is stationary.

75. The system of any one of claims 39-73, wherein the plurality of images are captured while the target object is moving and a background object of the target object is stationary.

76. The system of any of claims 39-73, wherein the movement information of the UAV comprises data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

77. An Unmanned Aerial Vehicle (UAV), comprising:

a camera on the UAV; and

a processor configured to:

identifying a target object to be measured;

receiving a plurality of images captured by a camera while the UAV is moving and the camera is tracking the target object;

acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and

calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

78. The UAV of claim 77, wherein,

the camera is further configured to capture an initial image containing the target object; and

the processor is configured to identify the target object in the initial image.

79. The UAV of claim 78, wherein the processor is configured to identify the target object by:

obtaining a user selection of a target region in the initial image; and

obtaining the target object based on the target area.

80. The UAV of claim 77, wherein the processor is further configured to:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

81. The UAV of claim 77, wherein the processor is further configured to:

82. The UAV of claim 81, wherein the processor is further configured to:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

83. The UAV of claim 82, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame; and

the processor is further configured to:

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

84. The UAV of claim 82, wherein,

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

determining whether the camera pose relationship satisfies the preset condition comprises at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

85. The UAV of claim 81, wherein the processor is further configured to:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

86. The UAV of claim 77, wherein the UAV movement information includes data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

87. The UAV of claim 77, wherein the processor is further configured to:

calculating a size of the target object based on a distance between the target object and the UAV.

88. A non-transitory storage medium storing computer-readable instructions that, when executed by at least one processor, cause the at least one processor to:

identifying a target object to be measured;

receiving a plurality of images captured by a camera of an Unmanned Aerial Vehicle (UAV) while the UAV is moving and the camera is tracking the target object;

acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and

calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

89. The storage medium of claim 88, wherein identifying the target object comprises:

capturing an initial image including the target object using the camera of the UAV; and

the target object is identified in the initial image.

90. The storage medium of claim 89, wherein identifying the target object further comprises:

obtaining a user selection of a target region in the initial image; and

obtaining the target object based on the target area.

91. A storage medium as defined in claim 88, wherein the computer readable instructions further cause the at least one processor to:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

92. A storage medium as defined in claim 88, wherein the computer readable instructions further cause the at least one processor to:

93. A storage medium as defined in claim 92, wherein the computer readable instructions further cause the at least one processor to:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

94. The storage medium of claim 93, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame; and

the computer-readable instructions further cause the at least one processor to perform:

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

95. The storage medium of claim 93, wherein:

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

determining whether the camera pose relationship satisfies the preset condition comprises at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

96. A storage medium as defined in claim 92, wherein the computer readable instructions further cause the at least one processor to:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

97. The storage medium of claim 88, wherein the UAV movement information includes data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

98. A storage medium as defined in claim 88, wherein the computer readable instructions further cause the at least one processor to:

calculating a size of the target object based on a distance between the target object and the UAV.

99. A method for measuring distance using an unmanned aerial vehicle UAV, comprising:

identifying a target object;

receiving a plurality of images captured by a camera of an Unmanned Aerial Vehicle (UAV) while the UAV is moving and the camera is tracking the target object;

acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and

calculating a distance between an object to be measured contained in the plurality of images and the UAV based on the movement information and the plurality of images.

100. The method according to claim 99, further comprising identifying an object to be measured contained in the plurality of images by:

obtaining a user selection of a region in one of the plurality of images displayed on a graphical user interface; and

obtaining the object to be measured based on the selected region.

101. The method according to claim 99, further comprising identifying an object to be measured contained in the plurality of images by:

automatically identifying at least one object other than the target object contained in one of the plurality of images;

receiving a user instruction specifying the object to be measured; and

obtaining the object to be measured from the identified at least one object based on the user instruction.

102. The method of claim 99, further comprising:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

103. The method of claim 99, further comprising:

104. The method of claim 103, further comprising:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

105. The method of claim 104, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame; and

the method further comprises the following steps:

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

106. The method of claim 105, wherein:

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

determining whether the camera pose relationship satisfies the preset condition comprises at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

107. The method of claim 103, further comprising:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

calculating a distance between the object to be measured and the UAV from the 3D position of one or more of the feature points associated with the object to be measured and the 3D position of the camera indicated by the refined camera pose information.

108. The method of claim 99, where the UAV movement information includes data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

109. The method of claim 99, further comprising:

calculating a size of the object to be measured based on a distance between the object to be measured and the UAV.

110. An Unmanned Aerial Vehicle (UAV), comprising:

a camera on the UAV; and

a processor configured to:

identifying a target object to be measured;

receiving a plurality of images captured by a camera while the UAV is moving and the camera is tracking the target object;

acquiring movement information of the UAV corresponding to a time of capture of the plurality of images; and

calculating a distance between an object to be measured contained in the plurality of images and the UAV based on the movement information and the plurality of images.

111. The UAV of claim 110, wherein the processor is configured to identify the object to be measured by:

obtaining a user selection of a region in one of the plurality of images displayed on a graphical user interface; and

obtaining the object to be measured based on the selected region.

112. The UAV of claim 110, wherein the processor is configured to identify the object to be measured by:

automatically identifying at least one object other than the target object contained in one of the plurality of images;

receiving a user instruction specifying the object to be measured; and

obtaining the object to be measured from the identified at least one object based on the user instruction.

113. The UAV of claim 110, wherein the processor is further configured to:

determining an initial radius, the initial radius being an estimated distance between the target object and the UAV;

determining an initial velocity based on the initial radius; and

moving the UAV around the target object along a curved path having the initial radius at the initial velocity.

114. The UAV of claim 110, wherein the processor is further configured to:

115. The UAV of claim 114, wherein the processor is further configured to:

obtaining a camera pose relationship between a key frame and an image frame captured after the key frame, the key frame being one of the plurality of images;

determining whether the camera attitude relationship meets a preset condition; and

in response to the camera pose relationship satisfying the preset condition, selecting the image frame as one of the plurality of images.

116. The UAV of claim 115, wherein the camera pose relationship is a first camera pose relationship and the image frame is a first image frame; and

the processor is further configured to:

determining whether the second camera attitude relationship meets the preset condition; and

in response to the second camera pose relationship satisfying the preset condition, selecting the second image frame as one of the plurality of images.

117. The UAV of claim 116, wherein,

the preset condition comprises at least one of a rotation threshold or a displacement threshold;

determining whether the camera pose relationship satisfies the preset condition comprises at least one of:

in response to the rotation change being less than the rotation threshold, determining that the camera pose relationship satisfies the preset condition; or

118. The UAV of claim 114, wherein the processor is further configured to:

extracting a plurality of feature points from each of the plurality of images;

tracking two-dimensional (2D) locations of the plurality of feature points in the plurality of images;

119. The UAV of claim 110, wherein the UAV movement information comprises data acquired by at least one of an accelerometer, a gyroscope, or a pan-tilt of the UAV.

120. The UAV of claim 110, wherein the processor is further configured to:

calculating a size of the object to be measured based on a distance between the object to be measured and the UAV.

Technical Field

The present disclosure relates to a distance measuring technology, and more particularly, to a distance measuring method and apparatus using an unmanned aerial vehicle.

Background

In many industrial activities, it is often desirable to measure the distance to a particular building or sign. Conventional laser ranging methods are cumbersome and require specialized equipment. For locations that are difficult to access, the measurement method is even more limited.

With the development of today's technology, aircraft, such as Unmanned Aerial Vehicles (UAVs), have been used in a variety of application scenarios. Existing distance measurement techniques using UAVs include: this may be complicated or inefficient with the UAV's Global Positioning System (GPS) location or installing a dedicated laser ranging device on the UAV. There is a need to develop autonomous operation in UAVs for distance measurement.

Disclosure of Invention

In accordance with the present disclosure, a method of measuring distance using an Unmanned Aerial Vehicle (UAV) is provided. The method comprises the following steps: identifying a target object to be measured; and receiving a plurality of images captured by a camera of the UAV while the UAV is moving and the camera is tracking the target object; acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

Also in accordance with the present disclosure, a system for measuring distance using an Unmanned Aerial Vehicle (UAV) is provided. The system includes a camera of a UAV, at least one memory, and at least one processor coupled to the memory. The at least one processor is configured to identify a target object to be measured. The camera is configured to capture a plurality of images while the UAV is moving and the camera is tracking the target object. The at least one processor is further configured to: acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; and calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

Also in accordance with the present disclosure, an Unmanned Aerial Vehicle (UAV) is provided. The UAV includes a camera and a processor on the UAV. The processor is configured to: identifying a target object to be measured; and receiving a plurality of images captured by the camera while the UAV is moving and the camera is tracking the target object; acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; and calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

Also in accordance with the present disclosure, a non-transitory storage medium storing computer readable instructions is provided. The computer readable instructions, when executed by the at least one processor, may cause the at least one processor to perform: identifying a target object to be measured; and receiving a plurality of images captured by a camera of the UAV while the UAV is moving and the camera is tracking the target object; acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; calculating a distance between the target object and the UAV based on the movement information and the plurality of images.

Also in accordance with the present disclosure, a method for measuring distance using an Unmanned Aerial Vehicle (UAV) is provided. The method comprises the following steps: identifying a target object; and receiving a plurality of images captured by a camera of the UAV while the UAV is moving and the camera is tracking the target object; acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; calculating a distance between the object to be measured contained in the plurality of images and the UAV based on the movement information and the plurality of images.

Also in accordance with the present disclosure, an Unmanned Aerial Vehicle (UAV) is provided. The UAV includes a camera and a processor on the UAV. The processor is configured to: identifying a target object; and receiving a plurality of images captured by the camera while the UAV is moving and the camera is tracking the target object; acquiring movement information of the UAV corresponding to capturing moments of the plurality of images; and calculating a distance between the object to be measured contained in the plurality of images and the UAV based on the movement information and the plurality of images.

Drawings

FIG. 1 is a schematic diagram illustrating an operating environment in accordance with an exemplary embodiment of the present disclosure;

FIG. 2 is a schematic block diagram of a movable object according to an exemplary embodiment of the present disclosure;

fig. 3 shows an image sensor of a UAV according to an exemplary embodiment of the present disclosure;

FIG. 4 is a schematic block diagram illustrating a computing device in accordance with an illustrative embodiment of the present disclosure;

FIG. 5 is a flow chart of a distance measurement process according to an exemplary embodiment of the present disclosure;

FIG. 6 is a graphical user interface associated with identifying a target object according to an example embodiment of the present disclosure;

FIG. 7A is a superpixel segmentation result image according to an exemplary embodiment of the present disclosure;

FIG. 7B is an enlarged portion of the image shown in FIG. 7A;

FIG. 8 illustrates a distance calculation process according to an exemplary embodiment of the present disclosure; and

fig. 9 illustrates a key frame extraction process according to an exemplary embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments according to the present disclosure will be described with reference to the accompanying drawings, which are merely examples for illustrative purposes and are not intended to limit the scope of the present disclosure. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

The present disclosure provides a method for measuring distance using an Unmanned Aerial Vehicle (UAV). Unlike traditional ranging methods, the disclosed methods can provide distance measurements of user-selected objects in real-time by implementing machine vision techniques and integrating inertial navigation data from the UAV's own Inertial Measurement Unit (IMU). The disclosed method is intuitive and convenient and can provide reliable measurement results at fast calculation speeds.

FIG. 1 is a schematic block diagram illustrating an operating environment in accordance with an illustrative embodiment of the present disclosure. As shown in fig. 1, the movable object 102 may communicate wirelessly with a remote control 104. The movable object 102 may be, for example, an Unmanned Aerial Vehicle (UAV), an unmanned vehicle, a mobile robot, an unmanned ship, a submarine, a spacecraft, a satellite, or the like. The remote controller 104 may be a remote controller or a terminal device having an application (app) that can control the movable object 102. The terminal device may be, for example, a smartphone, a tablet, a gaming device, etc. The movable object 102 may carry a camera 1022. Images or video (e.g., successive image frames) captured by the camera 1022 of the movable object 102 may be sent to the remote control 104 and displayed on a screen coupled to the remote control 104. As used herein, a screen coupled to the remote control 104 may refer to a screen embedded in the remote control 104 and/or a screen of a display device operatively connected to the remote control. The display device may be, for example, a smartphone or a tablet computer. The camera 1022 may be a load of the movable object 102 supported by a carrier 1024 (e.g., a pan-tilt head) of the movable object 102. The camera 1022 may track the target object 106, and the image captured by the camera 1022 may include the target object 106. As used herein, tracking an object by a camera may refer to using a camera to capture one or more images containing the object. For example, the camera 1022 may capture multiple images of the target object 106 as the movable object 102 moves in certain patterns. Since the relative position between the target object 106 and the camera 1022 may change due to movement of the movable object 102, the target object 106 may appear in different positions in the plurality of images. It will be appreciated that the captured plurality of images may also contain one or more background objects other than the target object, and that the background objects may also appear in different locations in the plurality of images. The movable object 102 may move in any suitable pattern, such as along a straight line, a polyline, an arc, a curved path, and so forth. The movement pattern may be predetermined or adjusted in real time based on feedback from sensors of the movable object 102. One or more processors on and/or outside of the movable object 102 (e.g., a processor on the UAV and/or a processor in the remote control 104) are configured to calculate a distance between the movable object 102 (e.g., the camera 1022 of the movable object) and the target object 106, for example, by analyzing images captured by the camera 1022 and/or other sensor data acquired by the movable object 102.

FIG. 2 is a schematic block diagram of a movable object according to an exemplary embodiment of the present disclosure. As shown in fig. 2, a movable object 200 (e.g., movable object 102) (e.g., a UAV) may include a sensing system 202, a propulsion system 204, communication circuitry 206, and an onboard controller 208.

The propulsion system 204 may be configured to enable the movable object 200 to perform a desired movement (e.g., in response to control signals from the onboard controller 208 and/or the remote control 104), such as take-off from or landing on a surface, hover at a particular location and/or orientation, move along a particular path, move at a particular speed in a particular direction, and so forth. The propulsion system 204 may include one or more of any suitable propellers, blades, rotors, motors, engines, etc., that enable the movable object 200 to move. The communication circuit 206 may be configured to establish wireless communication with the remote control 104 and perform data transmission. The transmitted data may include sensing data and/or control data. The onboard controller 208 may be configured to control operation of one or more components on the movable object 200 (e.g., based on analysis of sensed data from the sensing system 202) or operation of an external device in communication with the movable object 200.

The sensing system 202 may include one or more sensors that may sense the spatial arrangement, velocity, and/or acceleration of the movable object 200 (e.g., the pose of the movable object 200 with respect to up to three degrees of translation and up to three degrees of rotation). Examples of sensors may include, but are not limited to, location sensors (e.g., Global Positioning System (GPS) sensors, mobile device sensors that implement location triangulation), visual sensors (e.g., imaging devices capable of detecting visible, infrared, or ultraviolet light, such as cameras), proximity sensors (e.g., ultrasonic sensors, lidar, time-of-flight cameras), inertial sensors (e.g., accelerometers, gyroscopes, Inertial Measurement Units (IMUs)), altitude sensors, pressure sensors (e.g., barometers), audio sensors (e.g., microphones), or field sensors (e.g., magnetometers, electromagnetic sensors). Any suitable number and/or combination of sensors may be included in sensing system 202. The sensing data acquired and/or analyzed by the sensing system 202 may be used to control the spatial arrangement, speed, and/or orientation of the movable object 200 (e.g., using a suitable processing unit such as the onboard controller 206 and/or the remote control 104). Further, the sensing system 202 may be used to provide data about the environment surrounding the movable object 200, such as proximity to potential obstacles, location of geographical features, location of man-made structures, and the like.

In some embodiments, the movable object 200 may also include a carrier for supporting a load carried by the movable object 200. The carrier may include a pan-tilt head that carries and controls movement and/or orientation of the load (e.g., in response to control signals from the onboard controller 208) such that the load may move in one, two, or three degrees of freedom relative to the center/body of the movable object 200. The load may be a camera (e.g., camera 1022). In some embodiments, the load may be fixedly coupled to the movable object 200.

In some embodiments, the sensing system 202 includes at least an accelerometer, a gyroscope, an IMU, and an image sensor. The accelerometer, gyroscope, and IMU may be located at the center/body of the movable object 200. The image sensor may be a camera located in the center/body of the movable object 200, or may be a load of the movable object 200. When the load of the movable object 200 comprises a camera carried by a pan-tilt head, the sensing system 202 may also comprise other components to acquire and/or measure attitude information of the load camera, such as a photoelectric encoder, a hall effect sensor and/or a second set of accelerometers, a gyroscope and/or an IMU located at or embedded in the pan-tilt head.

In some embodiments, the sensing system 202 may also include a plurality of image sensors. Fig. 3 illustrates an image sensor of a UAV according to an exemplary embodiment of the present disclosure. As shown in fig. 3, the UAV comprises as a payload a camera 2022 carried by a pan-tilt, a forward looking system 2024 comprising two lenses (which together constitute a stereoscopic camera), and a downward looking system 2026 comprising a stereoscopic camera. The images/video acquired by any image sensor may be sent to and displayed on the UAV's remote control 104. In some embodiments, the camera 2022 may be referred to as a primary camera. The distance to the target object 106 may be measured by tracking the camera pose of the primary camera at the time the plurality of images are captured and analyzing the captured plurality of images containing the target object 106. In some embodiments, the camera 2022 carried by the pan-tilt can be a monocular camera that captures color images.

In some embodiments, a camera as used hereinIn the model, a camera matrix is used to describe a projection mapping from three-dimensional (3D) world coordinates to two-dimensional (2D) pixel coordinates. Let [ u, v, 1)]^TRepresenting the 2D point location in homogeneous/projected coordinates (e.g., the 2D coordinates of the point in the image), let [ x_w，y_w，z_w]^TRepresenting a 3D point location in world coordinates (e.g., a 3D location in the real world), where z_cDenotes the z-axis from the optical center of the camera, K denotes the camera calibration matrix, R denotes the rotation matrix, and T denotes the transformation matrix. The mapping from world coordinates to pixel coordinates can be described as:

the camera calibration matrix K describes the intrinsic parameters of the camera. For a limited projection camera, its eigenmatrix K includes five eigenparameters:

where f is the focal length (in distance) of the camera. Parameter alpha_x＝fm_x，α_y＝fm_yDenotes the focal length in pixels, where m_xAnd m_yIs a scaling factor in the x-axis and y-axis directions (e.g., the x-axis and y-axis directions of a pixel coordinate system) that associates a pixel with a unit distance (i.e., the number of pixels corresponding to a unit distance (e.g., one inch)). γ represents the skew factor between the x-axis and the y-axis because the pixels are not square in a CCD (charge coupled device) camera. Mu.s₀，v₀Coordinates representing a principal point, which in some embodiments is located at the center of the image.

The rotation matrix R and the translation matrix T are extrinsic parameters of the camera, which represent a coordinate system transformation from 3D world coordinates to 3D camera coordinates.

The forward looking system 2024 and/or the downward looking system 2026 may include stereoscopic cameras that capture grayscale stereoscopic image pairs. The sensing range of the camera 2022 may be larger than that of a stereo camera. The visual range (VO) circuitry of the UAV may be configured to analyze image data acquired by the stereo cameras of the forward looking system 2024 and/or the downward looking system 2026. The VO circuit of the UAV may implement any suitable visual odometry algorithm to track the position and movement of the UAV based on the acquired grayscale stereo image data. The visual ranging algorithm may include: changes in the positions of a plurality of feature points (i.e., optical flows of the feature points) in a series of captured images are tracked, and camera motion is obtained based on the optical flows of the feature points. In some embodiments, the forward looking system 2024 and/or the downward looking system 2026 are fixedly coupled to the UAV, so the camera motions/poses obtained by the VO circuit may represent the motions/poses of the UAV. By analyzing the change in position of the feature points from one image at a first capture time to another image at a second capture time, the VO circuit may obtain the camera/UAV pose relationship between the two capture times. As used herein, the camera pose relationship or UAV pose relationship between any two time instants (i.e., points in time) may be described by: a rotational change of the camera or the UAV from a first time to a second time, and a spatial displacement of the camera or the UAV from the first time to the second time. As used herein, the capture moment refers to the point in time at which an image/frame is captured by a camera onboard the movable object. The VO circuit may also integrate inertial navigation data to obtain the pose of the camera/UAV with enhanced accuracy (e.g., by implementing a visual inertial ranging algorithm).

Fig. 4 is a schematic block diagram illustrating a computing device 400 in accordance with an exemplary embodiment of the present disclosure. The computing device 400 may be implemented in the movable object 102 and/or the remote control 104 and may be configured to perform a distance measurement method according to the present disclosure. As shown in fig. 4, the computing device 400 includes at least one processor 404, at least one storage medium 402, and at least one transceiver 406. In accordance with the present disclosure, the at least one processor 404, the at least one storage medium 402, and the at least one transceiver 406 may be separate devices, or any two or more of them may be integrated in one device. In some embodiments, computing device 400 may also include a display 408.

The at least one storage medium 402 may include a non-transitory computer-readable storage medium, such as Random Access Memory (RAM), read only memory, flash memory, volatile memory, hard disk memory, or an optical medium. At least one storage medium 402 coupled to at least one processor 404 may be configured to store instructions and/or data. For example, the at least one storage medium 402 may be configured to store data acquired by the IMU, images captured by a camera, computer-executable instructions for implementing a distance measurement process, and the like.

The at least one processor 404 may include any suitable hardware processor, such as a microprocessor, microcontroller, Central Processing Unit (CPU), Network Processor (NP), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The at least one storage medium 402 stores computer program code that, when executed by the at least one processor 404, controls the at least one processor 404 and/or the at least one transceiver 406 to perform a distance measurement method according to the present disclosure, such as one of the exemplary methods described below. In some embodiments, the computer program code also controls the at least one processor 404 to perform some or all of the functions that may be performed by the movable object and/or the remote control as described above, each of which may be examples of the computing device 400.

The at least one transceiver 406 is controlled by the at least one processor 404 to transmit data to and/or receive data from another device. The at least one transceiver 406 may include any number of transmitters and/or receivers suitable for wired and/or wireless communication. Transceiver 406 may include one or more antennas for wireless communication over any supported frequency channel. The display 408 may include one or more screens for displaying content in the computing device 400 or sent from another device, e.g., displaying images/video captured by a camera of a movable object, displaying a graphical user interface requesting user input to determine a target object, displaying a graphical user interface indicating a measured distance to a target object, etc. In some embodiments, the display 408 may be a touch screen display configured to receive touch inputs/gestures by a user. In some embodiments, computing device 400 may include other I/O (input/output) devices, such as a joystick, a control panel, speakers, and the like. In operation, the computing device 400 may implement the distance measurement methods disclosed herein.

The present disclosure provides a distance measurement method. Fig. 5 is a flowchart of a distance measurement process according to an exemplary embodiment of the present disclosure. The disclosed distance measurement process may be performed by the movable object 102 and/or the remote control 104. The disclosed distance measurement process may be implemented by a system including a processor, a storage medium, and a camera onboard a movable object. The storage medium may store computer readable instructions executable by the processor, and the computer readable instructions may cause the processor to perform the disclosed distance measurement method. Hereinafter, in describing the disclosed method, a UAV is used as an example of the movable object 102. However, it should be understood that the disclosed methods may be implemented by any suitable movable object.

As shown in fig. 5, the disclosed method may include identifying a target object (S502). A target object is identified from the image based on a user input. Images may be captured by the camera 1022 and may be displayed on the remote control 104.

In some embodiments, a human-machine-interaction terminal (e.g., remote control 104), such as a smart phone, smart tablet, smart glasses, or the like, may receive a user selection of a target object to be measured. Fig. 6 is a graphical user interface related to identifying a target object according to an example embodiment of the present disclosure. As shown in fig. 6, the graphical user interface may display an initial image 602. The initial image 602 may be displayed on a screen of a remote control in communication with the UAV. The initial image 602 may be a real-time image captured by and transmitted from the UAV. The remote control may allow the user to identify a target region 604 in the initial image 602. The target area 604 may be identified based on user selection, such as a single click at the center of the target area, a double click at any location in the target area, a single/double click at a first corner and a single/double click at a second corner of a bounding box defining the target area, a free drawing of a shape surrounding the target area, or a drag operation with a start point and an end point of a bounding box defining the target area. When the user input identifies only one point in the image as corresponding to the target object, an image segmentation process may be performed to obtain a plurality of segmented image portions, and the target region may be determined as a segmented portion including the identified point. In some embodiments, the user input may be an object name or an object type. A pattern recognition or image classification algorithm may be implemented to recognize one or more objects in the initial image based on name/type and determine an object matching the name or type entered by the user as the target object.

In some embodiments, when the camera of the UAV is tracking a target object (i.e., capturing an image containing the target object), the user may request to measure the distance to another object also contained in the captured image, for example, by selecting an area corresponding to the object to be measured in the image shown on the graphical user interface, or entering a name or type of the object to be measured. The object to be measured may be a background object of the target object. In other words, both the target object and the background object are contained in multiple images captured by the camera of the UAV.

In some embodiments, identifying the object to be measured may comprise: obtaining a user selection of a region in one of a plurality of images displayed on a graphical user interface; and acquiring the object to be measured based on the selected region. For example, as shown in fig. 6, the user may select an area 606 as an area corresponding to an object to be measured. In some other embodiments, identifying the object to be measured may include: automatically identifying at least one object other than the target object contained in one of the plurality of images; receiving a user instruction for specifying an object to be measured; acquiring an object to be measured from the at least one identified object based on a user instruction. Pattern recognition or image classification algorithms may be implemented to automatically identify one or more objects in a captured image based on name, type, or other object characteristics. For example, the identified objects may be: umbrellas, orange cars, buildings with flat roofs. Further, an object matching the name or type input by the user is determined as an object to be measured. Object recognition may be performed after receiving user input regarding a particular name or type. Alternatively, a plurality of identified objects may be presented on a graphical user interface (e.g. by listing the names/properties of the objects, or by displaying a bounding box corresponding to the objects in the image) and a user selection of one object (e.g. a selection on one name or one bounding box) is received to determine the objects to be measured.

In some embodiments, identifying the object in the image may include identifying a region in the image representing the object. For example, identifying the target object may include identifying a region in the initial image representing the target object based on user input. It will be appreciated that the disclosed process of identifying a target object in an initial image may be applied to identify any suitable object in any suitable image. In some embodiments, the target area is considered to be an area representing a target object. In some embodiments, user selection of the target region may not be an accurate operation, and the initially identified target region may indicate an approximate location and size of the target object. The region representing the target object may be obtained by refining the target region from the initial image, for example by implementing a superpixel segmentation method.

A super-pixel may comprise a group of connected pixels having similar texture, color and/or brightness levels. A super-pixel may be an irregularly shaped block of pixels with some visual saliency. Superpixel segmentation includes dividing an image into a plurality of non-overlapping superpixels. In one embodiment, the superpixels of the initial image may be obtained by clustering (cluster) the pixels of the initial image based on the image features of the pixels. Any suitable superpixel segmentation algorithm may be used, such as a Simple Linear Iterative Clustering (SLIC) algorithm, a graph-based segmentation algorithm, an N-cut segmentation algorithm, a Turbo pixel segmentation algorithm, a fast-shift segmentation algorithm, a graph cut a segmentation algorithm, a graph cut b segmentation algorithm, and so forth. It will be appreciated that the superpixel segmentation algorithm may be used in both color and grayscale images.

Further, one or more super pixels located in the target region may be obtained, and a region formed by the one or more super pixels may be identified as a region representing the target object. Excluding superpixels that lie outside the target region. For superpixels that are partially located in the target region, the percentage may be determined by dividing the number of pixels located within the target region in the superpixels by the total number of pixels in the superpixels. If the percentage is greater than a preset threshold (e.g., 50%), the superpixel may be deemed to be located in the target region. The preset threshold may be adjusted based on the actual application.

Fig. 7A illustrates a superpixel segmentation result image according to an exemplary embodiment of the present disclosure. Fig. 7B shows an enlarged portion of the image shown in fig. 7A. As shown in FIG. 7B, a plurality of superpixels, all or in part, are located within a user selected target region 702, including superpixels 704, 706, and 708. The superpixel 704 is fully enclosed in the target region 702 and is considered to be included in the region representing the target object. In some embodiments, the preset percentage threshold may be 50%. Thus, in these embodiments, since less than 50% of the superpixels 706 are located within the target region 702, the superpixels 706 are excluded from the region representing the target object. On the other hand, because more than 50% of the superpixels 708 are located within the target region 702, the superpixels 708 are included in the region representing the target object.

In some embodiments, the disclosed method may include presenting a warning message indicating impaired measurement accuracy after identifying the target object. In some cases, the target object may have certain characteristics that affect the accuracy of the measurement, for example, when the target object may be moving quickly, or when the target object does not include sufficient detail to be tracked. If the target object is determined to have one or more of certain characteristics, the remote control may present a warning message and a reason that may compromise the accuracy of the measurement. In some embodiments, the warning message may also include an option to abort or continue the measurement, and the measuring step may continue after receiving a confirmation selection based on the user input.

In some embodiments, the disclosed method may include determining whether the target object is a moving object. In some embodiments, the disclosed method may further comprise: if it is determined that the target object is a moving object, a warning message is presented indicating impaired measurement accuracy. For example, a Convolutional Neural Network (CNN) may be implemented on the target object to identify the type of the target object. The type of the target object may be one of the following: for example, a high mobility type, which indicates that the target object has a high probability of moving, such as a human, an animal, an automobile, an airplane, or a ship; a low mobility type indicating that the target object has a low likelihood of moving, such as a door or a chair; and non-mobile types such as buildings, trees, or road signs. A warning message may be presented accordingly. In some embodiments, the disclosed method may include determining whether a movement speed of the target object is below a preset threshold. That is, the disclosed method may provide an accurate measurement of the distance to the target object if the target object moves below a certain threshold speed. In some embodiments, the disclosed method may further comprise: if the moving speed of the target object is not less than the preset threshold, a warning message indicating impaired measurement accuracy is presented.

In some embodiments, the disclosed methods may include: extracting a target feature point corresponding to the target object (for example, a region representing the target object in the initial image); and determining whether the number of the target characteristic points is less than a preset number threshold. In some embodiments, the disclosed method may further comprise: and presenting a warning message in response to the number of the target feature points being less than the preset number threshold, the warning message indicating impaired measurement accuracy. Whether the target object may be tracked over a series of image frames may be determined based on whether the target object includes sufficient texture detail or a sufficient number of feature points. The feature points may be extracted by any suitable feature extraction method, such as a harris angle detector, a HOG (histogram of oriented gradients) feature descriptor, or the like.

In some embodiments, when determining the target area of the target object, the graphical user interface on the remote control may display, for example, a boundary line or bounding box overlaying the target area on the initial image, a warning message in response to determining a potentially compromised measurement accuracy, and/or an option to confirm a continuous distance measurement and/or further edit the target area.

Referring again to fig. 5, as the UAV moves, a camera of the UAV may track the target object and capture a series of images, and a processor may receive the captured images (S504). In other words, as the UAV moves, a camera on the UAV may capture a series of images that contain the target object. In some embodiments, the image capture may be a conventional operation of the UAV (e.g., at a fixed frequency), and the remote control may receive a real-time transmission of the captured image from the UAV and display on the screen. Conventional operation of a UAV refers to the operation of a UAV that may typically be performed during flight of the UAV. In addition to image capture, conventional operations may include: hovering steadily when no movement control is received, automatically avoiding obstacles, responding to control commands from a remote control (e.g., adjusting altitude, speed, and/or direction of flight based on user input to the remote control, flying toward a user selected location on the remote control), and/or providing feedback to the remote control (e.g., reporting location and flight status, sending real-time images). Determining the direction and/or speed of movement of the UAV may be an operation that facilitates distance measurements. At the beginning of the distance measurement process, the UAV may move at an initial velocity along an arc or curved path having an initial radius around the target object. The target object may be located at or near the center of the arc or curved path. The initial radius may be an estimated distance between the target object and the UAV. In some embodiments, the initial velocity may be determined based on the initial radius. For example, the initial velocity may have a positive correlation with the initial radius.

In some embodiments, the estimated distance between the target object and the UAV may be determined based on data obtained from a stereo camera of the UAV (e.g., forward looking system 2024). For example, after identifying a target object in an initial image captured by a main camera (e.g., camera 2022) of the UAV, images captured by stereo cameras at substantially the same time instance may be analyzed to obtain a depth map. That is, the depth map may also include an object corresponding to the target object. The depth of the corresponding object may be used as an estimated distance between the target object and the UAV. It will be appreciated that the estimated distance between the target object and the UAV may be determined based on data obtained from any suitable depth sensor (e.g., laser sensor, infrared sensor, radar, etc.) on the UAV.

In some embodiments, the estimated distance between the target object and the UAV may be determined based on a preset value. The preset value may be the farthest distance that the UAV may measure (e.g., based on the resolution of the UAV's main camera). For example, when it is difficult to identify an object corresponding to a target object in the depth map, the initial radius may be directly determined as a preset value.

In some embodiments, as the UAV moves, sensed data of the UAV (e.g., images captured by a camera) may be used as feedback data, and at least one of a speed of the UAV, a direction of movement of the UAV, a degree of rotation of the UAV, or a degree of rotation of a pan/tilt head carrying the camera may be adjusted based on the feedback data. In this way, closed loop control may be achieved. The feedback data may include pixel coordinates corresponding to a target object in the captured image. In some embodiments, the degree of rotation of the camera-bearing pan/tilt head may be adjusted to ensure that the target object is included in the captured image. In other words, the camera tracks the target object. In some cases, the target object is tracked at some predetermined location (e.g., the center of the image) or at some predetermined size (e.g., in pixels). That is, when it is determined that a part of the target object is not in the captured image based on the feedback data, the degree of rotation of the pan/tilt head may be adjusted. For example, if the remaining pixels corresponding to the target object are located at the upper edge of the captured image, the pan-tilt may rotate the camera upward to some extent to ensure that the next captured image includes the entire target object. In some embodiments, the speed of the UAV may be adjusted based on a difference in position of the target object (e.g., 2D coordinates of matching superpixels) in the current image and the previously captured image. The current image and the previously captured image may be two consecutively captured frames, or frames captured at predetermined intervals. For example, if the position difference is less than a first threshold, the speed of the UAV may be increased; and if the position difference is greater than a second threshold, the speed of the UAV may be reduced. In other words, the difference in the position of the target object in the two images is less than the first threshold, indicating that redundant information is being acquired and analyzed, so the speed of the UAV may be increased to create enough displacement between frames to save computational power/resources and speed up the measurement process. On the other hand, a large difference in the position of the target object in the two images may result in difficulty in tracking the same feature point between multiple captured images and result inaccuracy, so the speed of the UAV may be reduced to ensure measurement accuracy and stability. In some embodiments, if the user requests to measure the distance to a background object other than the target object, the movement of the UAV and/or pan/tilt head may be adjusted based on the difference in the position of the background object in the current image and in previously captured images.

In some embodiments, movement of the UAV may be manually controlled based on user input. When it is determined that the speed of the UAV or the degree of rotation of the pan/tilt head should be adjusted based on the feedback data, the remote control may prompt the user to request automatic correction or provide a suggestion for manual operation (e.g., display a prompt message or play an audio message, such as "slow down speed to measure distance"). In some embodiments, when there is no manual input, the UAV may fly automatically based on a pre-emptive process for distance measurement (e.g., selecting an initial speed and radius, adjusting speed and degree of rotation based on feedback data, as described above).

When the UAV is moving and captures an image, movement information of the UAV corresponding to a capturing timing of the image is also acquired (S506). The movement information may include various sensor data recorded by the UAV, such as accelerometer and gyroscope readings as the UAV moves. In some embodiments, the movement information may include attitude information of a pan/tilt head carrying the main camera, such as a degree of rotation of the pan/tilt head. In some embodiments, the movement information may also include other sensor data periodically generated for routing operations of the UAV, such as UAV pose relationships obtained from the IMU and VO circuits as the UAV moves, pose information (e.g., orientation and position) of the UAV in a world coordinate system obtained by integrating the IMU data, VO data, and GPS data. It is to be appreciated that capturing images of the target object (S504) and acquiring movement information of the UAV (S506) may be performed while the UAV is moving. Further, the images captured and the acquired movement information in S504 and S506 may include data regularly generated for normal operations, and may be directly obtained and used for distance measurement.

A distance between the object contained in the plurality of captured images and the UAV may be calculated based on the plurality of captured images and the movement information corresponding to the capturing times of the plurality of images (S508). The object to be measured may be a target object or a background object also contained in the plurality of images. By analyzing the data from the IMU and VO circuits and the images captured by the main camera, the 3D position of the image point and the camera pose information corresponding to the moment of capture of the plurality of images can be determined. Furthermore, distances to objects contained in the plurality of images may be determined based on the 3D positions of the image points. The distance calculation may be performed on the UAV and/or a remote control.

Fig. 8 illustrates a distance calculation process according to an exemplary embodiment of the present disclosure. As shown in fig. 8, in an exemplary embodiment, a plurality of key frames may be selected from consecutive image frames captured by the main camera (S5081). The selected key frames may form a sequence of key frames. In some embodiments, a sequence of raw image frames is captured at a fixed frequency, and some raw image frames may not be selected as key frames if they do not satisfy certain conditions. In some embodiments, the keyframes include image frames captured when the UAV is in steady motion (e.g., small rotational changes). In some embodiments, if the change in position from the most recent key frame to the current image frame is greater than a preset threshold (e.g., significant displacement), the current image frame is selected as the new key frame. In some embodiments, the first keyframe may be an initial image, or an image captured within a particular period of time of the initial image when the UAV is in a steady state (e.g., to avoid motion blur). Image frames captured after the first key frame may be determined and selected as key frames based on the pose relationship between the time of capture of the image frame and the nearest key frame. In other words, by evaluating the pose relationship of the master camera for two capture moments (e.g., rotational changes and displacements of the master camera from the moment the most recent key frame was captured to the moment the current image frame was captured), it can be determined whether the current image frame can be selected as a key frame.

In some embodiments, as the UAV moves and the camera captures image frames, new key frames may be determined and added to the sequence of key frames. Each keyframe may have a corresponding estimated camera pose of the primary camera. The estimated camera pose may be obtained by integrating IMU data of the UAV, VO data of the UAV, and position/rotation data of a pan-tilt head carrying the main camera. When the key frames in the sequence of key frames reach a certain number m (e.g. 10 key frames), they can be used to calculate the distance to the object to be measured.

When the key frames are determined, feature extraction may be performed for each key frame (S5082). In some embodiments, once a key frame is determined/selected, feature extraction may be performed. That is, feature extraction of a key frame may be performed while identifying the next key frame. In some other embodiments, feature extraction may be performed when a certain number of key frames are determined, for example when all key frames in a sequence of key frames are determined. Any suitable feature extraction method may be implemented herein. For example, sparse feature extraction may be used to reduce the amount of computation. A corner detection algorithm may be performed to obtain corners as feature points, such as FAST (accelerated segmentation detection feature), SUSAN (minimum equal segmentation absorption kernel) corner operators, harris corner operators, and the like. Taking the harris corner detection algorithm as an example, given an image point I, consider and shift (u, v) an image block on a region (u, v), the structure tensor a is defined as follows:

wherein, I_xAnd I_yIs the partial derivative of point I. Gradient information M in x-and y-directions corresponding to image points_cThe following can be defined:

M_c＝λ₁λ₂-κ(λ₁+λ₂)²＝det(A)-κtrace²(A)

where det (A) is the determinant of matrix A, trace (A) is the trace of matrix A, and κ is the adjustable sensitivity parameter. A threshold M may be set_th. When M is_c＞M_thThe image points are regarded as feature points.

Feature points in one key frame may appear in one or more other key frames. In other words, two consecutive key frames may include matching feature points that describe the same environment/object. The 2D positions of these feature points in the keyframe may be tracked to obtain the optical flow of the feature points (S5083). Any suitable feature extraction/tracking and/or image registration method may be implemented herein. Taking the Kanade-Lucas-tomasi (klt) feature tracker as an example, assuming that h represents the displacement between two images F (x) and g (x), and g (x) ═ F (x + h), the displacement of feature points in the keyframe can be obtained based on an iteration of the following equation:

wherein the content of the first and second substances,f (x) capture earlier than g (x), w (x) is a weighting function, and x is a vector representing position. Furthermore, after obtaining the displacement h of the current image relative to the previous image, an inverse calculation may be performed to obtain the displacement h' of the previous image relative to the current image. Theoretically, h ═ h'. If the actual calculation satisfies the theoretical condition, i.e., h ═ h', it can be determined that the feature points are correctly tracked, i.e., the feature points in one image match the feature points in the other image. In some embodiments, the tracked feature points may be identified in some or all of the keyframes, and each tracked feature point may be identified in at least two consecutive frames.

Based on the 2D positions of the feature points tracked in the keyframe, three-dimensional (3D) positions of the feature points and accurate camera pose information may be obtained by solving an optimization problem with respect to a 3D structure of scene geometry and viewing parameters related to camera pose (S5084). In an exemplary embodiment, a beam adjustment (RA) algorithm for minimizing the reprojection error between the image positions of the observed image point and the predicted image point may be used in this step. Given a set of images depicting a plurality of 3D points (i.e., feature points from a keyframe) from different viewpoints, beam adjustment can be defined as a problem that refines the 3D coordinates describing the scene geometry, the relative motion parameters (e.g., camera pose changes when capturing the keyframe), and the optical characteristics of the camera used to acquire the images, according to an optimization criterion involving the corresponding image projections of all the points. The mathematical representation of the BA algorithm is:

where i denotes the ith tracked 3D point (e.g., tracked feature point from S5083), n is the number of tracked points, and b_iRepresenting the 3D position of the ith point. j denotes the jth image (e.g., key frame from S5081), m is the number of images, and a_jThe camera pose information representing the jth image includes rotation information R, transformation information T, and/or intrinsic parameters K. v. of_ijIndicating whether the ith point has a projection in the jth image; if the jth image contains the ith point, v_ij1, otherwise v_ij＝0。Q(a_j，b_i) Is based on camera pose information a_jThe predicted projection of the ith point in the jth image. x is the number of_ijIs a vector describing the actual projection of the ith point in the jth image (e.g., the 2D coordinates of that point in the image). ad (x1, x2) represents the euclidean distance between image points represented by vectors x1 and x 2.

In some embodiments, the beam adjustment is equal to the set of jointly refined initial camera and structure parameter estimates to find the set of parameters that most accurately predict the position of the observed point in the set of available images. Initial camera and structural parameter estimation, i.e. a_jIs estimated camera pose information obtained based on normal operating data from the IMU of the UAV and the VO circuit of the UAV. That is, in maintaining normal operation of the UAV, the IMU and VO circuits may be splitThe sensor data is analyzed to identify the pose information of the UAV itself. The initial value of the estimated camera pose of the camera capturing the keyframe may be obtained by combining the pose information of the UAV at the matched capture time and the pose information of the pan-tilt carrying the camera at the matched capture time. In one embodiment, the initial value of the estimated camera pose may further integrate GPS data of the UAV.

The distance between the object to be measured and the UAV may be obtained from the 3D positions of the one or more feature points associated with the object to be measured (S5085). Hereinafter, in describing an embodiment of distance calculation and size determination, a target object is used as an example of an object to be measured. It will be appreciated that the disclosed procedure relating to a target object may be applied to any suitable object to be measured contained in a key frame. In some embodiments, the distance to the target object is considered to be the distance to the center point of the target object. The center point of the target object may be, for example, a geometric center of the target object, a centroid of the target object, or a center of a bounding box of the target object. The center point may or may not be included in the feature points extracted from S5082. When the center point is included in the extracted feature points, the distance to the center point may be directly determined based on the 3D position of the center point obtained from the beam adjustment result.

In one embodiment, when the center point is not included in the feature points extracted from S5082, tracking the 2D position of the feature point in the key frame (S5083) may further include adding the center point to the feature point and tracking the 2D position of the center point of the target object in the key frame according to an optical flow vector of the center point obtained based on the optical flow vector of the target feature point. In some embodiments, the target feature point may be a feature point extracted from S5082 and located within the region of the target object. That is, by adding the center point as a tracking point for BA algorithm calculation, the 3D position of the center point can be directly obtained from the BA algorithm results. Mathematically, assume x_iAn optical flow vector representing the ith target feature point, and if n feature points exist in the region corresponding to the target object, the center point x₀The optical flow vector of (a) can be obtained by:

wherein, w_iIs based on the weight that the distance between the center point and the ith target feature point corresponds to the ith target feature point. In one embodiment, w may be obtained based on a Gaussian distribution as follows_i：

Where σ can be adjusted based on experience, d_iRepresenting the distance between the central point on the image and the ith target feature point, i.e.Wherein (u)_i，v_i) Is the 2D image position of the ith target feature point, (u)₀，v₀) Is the 2D image position of the center point. In some embodiments, some target feature points used to obtain the optical flow vector for the center point may not necessarily be within the area of the target object. For example, feature points whose 2D positions are within a certain range of the center point may be used as the target feature points. The range may be larger than the area of the target object, for example, to include more feature points when computing the optical flow vector for the center point. It will be appreciated that a similar method of obtaining and adding optical flow vectors for points to the RA computation may be used to obtain 3D positions for points other than the center point based on the 2D positional relationship between the point to be added and the extracted feature points. For example, corner points of the target object may be tracked and added to the BA calculation, and the size of the target object may be obtained based on the 3D positions of the corner points of the target object.

In another embodiment, when the center point is not included in the feature points extracted from S5082, calculating a distance to the target object from the 3D positions of one or more feature points associated with the target object (S5085) may further include determining the 3D position of the center point based on the 3D positions of the plurality of target feature points. Feature points located within a range of a center point in the 2D image may be identified, and depth information of the identified feature points may be obtained based on their 3D positions. In one example, most of the identified feature points may have the same depth information or similar depth information within a preset variance range and may be considered to be located in the same image plane as the target object. That is, the majority depth of the identified feature points may be considered the depth of the target object, i.e., the distance between the target object and the UAV. In another example, a weighted average of the depths of the identified feature points may be determined as the depth of the target object. The weight may be determined based on a distance between the center point and the identified feature point.

In some embodiments, the size of the target object may be obtained based on the distance between the target object and the UAV. The dimensions of the target object may include, for example, the length, width, height, and/or volume of the target object. In one embodiment, assuming that the target object is a parallelepiped such as a rectangular parallelepiped, the size of the target object may be obtained by evaluating the 3D coordinates of two points/vertices of a body diagonal of the target object. In one embodiment, the length or height of the target object in the 2D image may be obtained in units of pixels (e.g., 2800 pixels), and based on the ratio of the depth of the target object to the focal length of the camera (e.g., 9000mm/60mm) and the camera sensor resolution (200 pixels/mm), the length or height of the target object in standard length units (e.g., 2.1m) may be obtained.

Referring again to fig. 5, the disclosed method further includes presenting the calculated distance to the user (S510). For example, the distance may be displayed on a graphical user interface and/or broadcast in an audio message. In some embodiments, the remote control may display the captured image on a graphical user interface and mark the distance on the image currently displayed on the graphical user interface. Further, the image currently displayed on the graphical user interface may be an initial image with the identified object to be measured or a real-time fed image containing the object to be measured.

In some embodiments, the distance between the object (e.g., the target object or the background object) and the UAV may be updated in real-time based on additional second images captured by the camera and movement information corresponding to the time of capture of the second images. After obtaining the 3D positions of the objects corresponding to the key frames (e.g., from S5084 and S5085), when a new image (e.g., a second image) is captured at any time after the 3D positions of the objects are determined, the positions of the objects corresponding to the second image may be obtained by combining the camera pose relationship between the 3D position of the object corresponding to the last key frame and the capturing time of the last key frame and the second image. In some embodiments, the distance may be updated at certain time intervals (e.g., every second) or each time a new key frame is selected, without repeatedly performing S5082-S5085. In one example, since a 3D location of the object is available, the updated distance between the object and the UAV may be conveniently determined by integrating (e.g., calculating the euclidean distance between the 3D locations) the current 3D location of the UAV with the 3D location of the object. In another example, since the positional relationship between the object and the UAV at a particular time is known (e.g., the positional relationship at the time of capture of the last key frame may be described by a first displacement vector), the updated distance between the object and the UAV may be conveniently determined by integrating the known positional relationship with the change in position of the UAV between the current time and the point in time corresponding to the known positional relationship (e.g., calculating the vector absolute value obtained by adding the first displacement vector to a second displacement vector describing the change in position of the UAV itself since the last key frame). In some other embodiments, when accumulating a certain number of new keyframes to form a new sequence of keyframes, the system may again perform S5082-S5085 to calculate the updated distance to the object.

In some embodiments, the key frames are captured when the target object is stationary. In some embodiments, key frames are captured when the target object is moving while the background object of the target object is stationary. The disclosed method can be used to obtain the 3D position of the background object. Further, based on the relative position between the background object and the target object, the distance to the target object may be obtained based on the motion of the tracked target object and the 3D position of the background object. For example, the background object is a building, while the target object is a car moving toward/away from the building as the UAV moves and captures images that include both the building and the car. By implementing the disclosed processes (e.g., S5081-S5085), the 3D location of the building and the positional relationship between the building and the UAV may be obtained. Further, a 3D positional relationship between the car and the building may be obtained from the relative 2D positional change between the building and the car as suggested by the captured image and combined with the relative depth change between the building and the car as suggested by an onboard depth sensor (e.g., stereo camera, radar, etc.). By integrating the 3D positional relationship between the building and the UAV and the 3D positional relationship between the vehicle and the building, the 3D positional relationship between the vehicle and the UAV and the distance between the vehicle and the UAV may be obtained.

In some embodiments, calculating the distance between the object to be measured and the UAV (S508) may further include accessing data generated in normal operation of maintaining the UAV, and using the data of normal operation for calculating the distance between the object to be measured and the UAV. As the UAV operates, various sensor data is re-encoded and analyzed in real-time to maintain regular operation of the UAV. The normal operations may include: capturing an image using an onboard camera and sending the captured image to a remote control for display; hovering steadily when no movement control is received, automatically avoiding obstacles, responding to control commands from a remote control (e.g., adjusting altitude, speed, and/or direction of flight based on user input to the remote control, flying toward a user selected location on the remote control), and/or providing feedback to the remote control (e.g., reporting location and flight status, sending real-time images). The recorded sensor data may include: gyroscope data, accelerometer data, rotation of a pan/tilt head carrying the main camera, GPS data, color image data acquired by the main camera, and grayscale image data acquired by the stereoscopic camera system. The inertial navigation system of the UAV may be used to obtain the current position/location of the UAV for normal operation. The inertial navigation system may be implemented by an Inertial Measurement Unit (IMU) of the UAV based on gyroscope data and accelerometer data and/or GPS data. The current location/position of the UAV may also be obtained through a VO circuit that implements a visual odometry mechanism based on grayscale image data acquired by a stereo camera of the UAV. Data from the IMU and VO circuits may be integrated and analyzed to obtain UAV pose information including the position of the UAV in a world coordinate system with greater accuracy. In some embodiments, the disclosed distance measurement system may determine whether the data needed to calculate the distance is readily available from data acquired for normal operation of the UAV. If a particular type of data is not available, the system may communicate with a corresponding sensor or other component of the UAV to enable data acquisition and acquire the missing type of data. In some embodiments, the disclosed distance measurement process does not require the acquisition of any other data than that acquired for normal operation of the UAV. In addition, the disclosed distance measurement process may utilize data that has been processed and generated in maintaining normal operation, such as data generated by the IMU and VO circuits.

In some embodiments, data generated by the IMU and VO circuits for normal operation of the UAV may be used directly in the distance measurement process. The data generated for normal operation may be used to select a keyframe in the distance measurement process (e.g., at S5081) and/or determine an initial value for beam adjustment (e.g., at S5084).

In some embodiments, the data generated for maintaining regular operation of the UAV that may be used to select key frames includes: the pose of the UAV at the time of capture of the previous image frame; and IMU data acquired since the time of capture of the previous image frame. In some embodiments, such data may be used to determine an estimated camera pose corresponding to a current image frame and accordingly determine whether the current image frame is a keyframe. For example, conventional operations include continuously calculating the pose of the UAV based on the IMU data and the VO/GPS data (e.g., by applying a visual inertial ranging algorithm). Thus, the pose of the UAV at the time of capture using the previous image frame is ready for use. At the time of determining whether the current image frame is a key frame, the pose of the UAV corresponding to the current image frame may not be immediately resolved or prepared. Accordingly, an estimated camera pose of the main camera corresponding to the current image frame may be obtained from the pose of the UAV at the time of capture of the previous image frame and IMU data corresponding to the time of capture of the current image frame (e.g., IMU data acquired between the time of capture of the previous image frame and the time of capture of the current image frame).

In some embodiments, IMU pre-integration may be implemented to estimate the UAV movement/position change between the time of capture of a series of image frames based on previous UAV positions and current IMU data. For example, the position of the UAV when capturing the current image frame may be estimated based on the position of the UAV when capturing the previous image frame and IMU pre-integration of data from the inertial navigation system. IMU pre-integration is a process that uses the position of the UAV at time point a and the accumulation of inertial measurements obtained between time points a and B to estimate the position of the UAV at time point B.

The mathematical description of discrete form IMU pre-integration is as follows:

v_k+1＝v_k+(R_wi(a_m-b_a)+g)Δt

Δq＝q{(ω-b_ω)Δt}

(b_a)_k+1＝(b_a)_k

(b_ω)_k+1＝(b_ω)_k

wherein p is_k+1Is the estimated 3D position, p, of the UAV at the time of capturing the current image frame_kIs based on data from conventional operations (e.g., calculated based on the IMU, VO circuitry, and/or GPS sensors) the 3D position of the UAV at the time the previous image frame was captured. v. of_k+1Is the speed, v, of the UAV at the time the current image frame was captured_kIs the speed of the UAV at the time the previous image frame was captured. q. q.s_k+1Is to capture the currentQuaternion, q, of UAV at image frame time_kIs the quaternion of the UAV at the time the previous image frame was captured. (b)_a)_k+1And (b)_a)_kIs the respective accelerometer bias at the time the current image frame and the previous image frame were captured. (b)_ω)_k+1And (b)_ω)_kAre the respective gyroscope biases at the time the current image frame and the previous image frame were captured. Δ t is the time difference between the time at which the current image frame k +1 was captured and the time at which the previous image frame k was captured. a is_mRepresenting the current reading of the accelerometer, g is the acceleration of gravity and ω represents the current reading of the gyroscope. Δ q is an estimate of the rotation between the current image frame and the previous image frame, q { } denotes the conversion from the euler angle representation to the quaternion representation. R_wiRepresenting the rotational relationship between the UAV coordinate system and the world coordinate system, can be obtained from the quaternion q.

In some embodiments, the current image frame and the previous image frame may be two consecutively captured imaging frames. In the IMU pre-integration process, the parameters obtained directly from the sensor include accelerometer readings a_mAnd gyroscope readings ω. The remaining parameters may be obtained based on the above mathematical description or any other suitable calculation. Thus, the pose of the UAV corresponding to the current image frame may be estimated by an IMU pre-integration of the pose of the UAV corresponding to the previous image frame (e.g., previously solved in normal operation of the UAV using visual inertial odometry) and IMU data corresponding to the current image frame.

In some embodiments, the frequency at which successive image frames are captured (e.g., 20-30Hz) is lower than the frequency at which accelerometer readings and gyroscope readings are recorded (e.g., 200-400 Hz). That is, multiple accelerometer readings and gyroscope readings may be obtained between the time of capture of two consecutive image frames. In one embodiment, IMU pre-integration may be performed based on the recorded frequencies of accelerometer and gyroscope readings. For example, Δ t 'represents the time difference between two consecutive accelerometer and gyroscope readings, and Δ t ═ n Δ t', n is an integer greater than 1. From Δ t', IMU pre-integration may be performed at the same frequency as the recording frequency of accelerometer and gyroscope readings. The estimated 3D position of the UAV at the time the current image frame was captured may be obtained by outputting every nth pre-integration result at the matching instant between the image capture and the accelerometer/gyroscope data record. In one embodiment, multiple accelerometer/gyroscope readings obtained between the capture instants of two consecutive image frames are filtered to obtain a noise reduction result for IMU pre-integration.

In some embodiments, using data generated for normal operation of the UAV in the distance measurement process (e.g., in key frame selection) may include: the gyroscope readings are used to determine if the UAV is in steady motion. If the UAV is not in steady motion, the captured images may not be suitable for distance measurement. For example, when the angular velocity is less than a preset threshold, i.e., when | | ω -b_ω||₂＜ω_thTime, omega_thIs a threshold angular velocity, it may be determined that the UAV is in steady motion, and images captured in steady motion may be used for distance measurements. Further, an image that is not captured in a steady moving state may not be selected as a key frame.

In some embodiments, the camera pose relationship between the capture instants of two consecutive frames (e.g., a previous image frame and a current image frame) may be estimated from the results from the IMU pre-integration. In some embodiments, when the VO algorithm is used for stereoscopic images of the UAV, the stereoscopic camera movements obtained from the VO algorithm may indicate the position and motion of the UAV. Further, the camera pose of the stereo camera or the pose of the UAV, IMU pre-integration data, and/or GPS data obtained from the VO algorithm may provide a rough estimate of the camera pose of the main camera. In some embodiments, the estimated camera pose of the main camera is obtained by combining the pose of the UAV and the pose of the pan/tilt head relative to the UAV (e.g., the degree of rotation of the pan/tilt head and/or the relative pose between the UAV and the pan/tilt head). For example, the estimated camera pose of the main camera corresponding to the previous image frame may be a combination of the pose of the UAV corresponding to the previous image frame (e.g., from normal operation) and the degree of rotation of the pan-tilt corresponding to the previous image frame. The estimated camera pose of the main camera corresponding to the current image frame may be a combination of an estimated pose of the UAV corresponding to the current image frame (e.g., from IMU pre-integration) and a degree of rotation of the pan/tilt head corresponding to the current image frame.

In some embodiments, using data generated for normal operation of the UAV in the distance measurement process (e.g., in key frame selection) may include: the camera pose relationship between the key frame and the image frame captured after the key frame is obtained using the camera pose relationship between two consecutive frames. In the case where the current key frame is determined, extracting the next key frame may include: determining whether a camera pose relationship between the key frame and an image frame captured after the key frame satisfies a preset condition; and in response to the camera attitude relationship meeting a preset condition, selecting the image frame as a next key frame.

Fig. 9 illustrates a key frame extraction process according to an exemplary embodiment of the present disclosure. As shown in fig. 9, the original image sequence includes a plurality of image frames captured at a fixed frequency (e.g., 30 Hz). VO calculations and/or IMU pre-integrations are performed for every two consecutive frames to obtain a camera pose relationship between two consecutive image capture instants. The camera pose relationship between a key frame and any image frame captured after the key frame may be obtained by repeatedly accumulating the camera pose relationship between two consecutive image capture instants, i.e. starting from the camera pose relationship of the pair of key frame and the frame next to the key frame earliest, until the camera pose relationship of the pair of image frame to be analyzed and its nearest previous frame. For example, as shown in FIG. 9, the current key frame is captured at time T0. The camera pose relationship between times T0 and T1 may be obtained from VO calculations and/or IMU pre-integration and analyzed to determine whether preset conditions are met. When the preset condition is not satisfied for the camera pose relationship between times T0 and T1, the key frame selection process continues to determine whether the camera pose relationship between times T0 and T2 satisfies the preset condition. The camera pose relationship between the times T0 and T2 can be obtained by combining the camera pose relationship between the times T0 and T1 with the camera pose relationship between the times T1 and T2. When the camera pose relationship between the times T0 and T3 satisfies the preset condition, the key frame selection process determines the image frame captured at the time T3 as the next key frame.

In some embodiments, the preset condition corresponding to the camera pose relationship comprises at least one of a rotation threshold or a displacement threshold. In one embodiment, the image frame is determined to be the next keyframe when the displacement between the image frame and the current keyframe is sufficiently large and/or the rotation between the image frame and the current keyframe is sufficiently small. In other words, the camera pose relationship includes at least one of a rotational change from a time at which the key frame is captured to a time at which the image frame is captured or a positional change of the camera from a time at which the key frame is captured to a time at which the image frame is captured. Determining whether the camera pose relationship satisfies a preset condition includes at least one of: determining that the camera attitude relationship meets a preset condition in response to the rotation change being smaller than a rotation threshold; and determining that the camera pose relationship meets a preset condition in response to the rotation change being less than the rotation threshold and the position change being greater than the displacement threshold. In some embodiments, when the change in position is less than or equal to the displacement threshold (e.g., indicating that the change in position is not significant enough to be processed), the image frame may be disqualified as a key frame and the process continues to analyze the next image frame. In some embodiments, when the change in rotation is greater than or equal to a rotation threshold (e.g., indicating that the image was not captured in a stable environment and may compromise the accuracy of the results), then the image frame may be discarded and the process continues to analyze the next image frame.

Mathematically, the rotational variation R can be described by the euler angle:the preset condition may include that the following inequality is satisfied:wherein alpha is_thIs the rotation threshold. The position/translation change t may be represented by t ═ t_x，t_y，t_z]^TA description is given. The preset condition may include that the following inequality is satisfied:wherein d is_thIs the displacement threshold.

In some embodiments, using the data for normal operation of the UAV during the distance measurement (e.g., in assigning initial values for the beam-balancing algorithm) may include: integrating data from the IMU, VO circuit, and GPS sensor to obtain pose information of the UAV corresponding to the time of capture of the keyframe. The estimated camera pose information of the main camera may be obtained by, for example, linear superposition of the camera poses of the stereo cameras (i.e., pose information of the UAV) and the positional relationship between the main camera and the UAV (i.e., the position/rotation of the pan-tilt relative to the UAV). Since the BA algorithm is an optimization problem, assigning random initial values may result in local optima rather than global optima. In S5084, using the estimated camera pose information from the IMU and VO data as an initial value of the BA algorithm, the number of iterations may be reduced, the convergence time of the algorithm may be accelerated, and the error probability may be reduced. Furthermore, in some embodiments, GPS data may also be used as initial values and constraints in the BA algorithm to obtain accurate results.

In some embodiments, data of the normal operation of the UAV used in the distance measurement procedure is acquired and generated by the UAV (e.g., at S504, S506, S5081, and when the initial values are obtained at S5084), and sent to the remote controller, and object recognition and distance calculation and presentation are performed on the remote controller (e.g., at S502, S5082-S5085, S510). In some embodiments, acquiring user input for identifying the object and presenting the calculated distance is performed only on a remote control, and the remaining steps are performed by UAVs.

It is to be understood that the mathematical process described herein for calculating camera pose information is not the only process. Other suitable processes/algorithms may be substituted for certain disclosed steps.

The present disclosure provides a method and system for measuring distance using an Unmanned Aerial Vehicle (UAV) and a UAV capable of measuring distance. Unlike traditional ranging methods, the disclosed methods provide a graphical user interface that allows a user to select an object of interest in an image captured by a camera of the UAV and provide a measured distance in near real-time (e.g., less than 500 milliseconds). Furthermore, the disclosed method can make distance measurements directly using inertial navigation data from the UAV's own IMU and data from the VO circuit generated for normal operation, which further saves computational resources and processing time. The disclosed method is intuitive and convenient and can provide reliable measurement results at fast calculation speeds.

The processes illustrated in the figures associated with method embodiments may be performed or carried out in any suitable order or sequence, and are not limited to the order or sequence illustrated in the figures and described above. For example, two sequential processes may be performed substantially concurrently or in parallel to reduce latency and processing time, or in an order reverse to that shown, where appropriate, depending on the functionality involved.

Further, components in the figures associated with the device embodiments may be coupled in different ways than shown in the figures as desired. Some components may be omitted and additional components may be added.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

38页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：三维激光扫描装置

Distance measuring method and apparatus

相关技术

网友询问留言