阅读说明：本技术 一种视觉定位方法、装置、计算机设备和存储介质 (Visual positioning method and device, computer equipment and storage medium ) 是由 王溯恺 黄淮扬 王恒立 蔡培德 薛博桓 于洋 王鲁佳 刘明 于 2020-04-01 设计创作，主要内容包括：本申请涉及一种视觉定位方法、装置、计算机设备和存储介质。所述方法包括：获取运动物体视区范围内路标的第一位置坐标,所述第一位置坐标为世界坐标系的坐标；获取由取像器采集的所述视区范围内路标的图像；计算所述图像中路标特征点的像素坐标；根据所述像素坐标和所述第一位置坐标确定所述取像器的位姿；根据所述位姿以及所述取像器与所述运动物体之间的位姿变换关系计算所述运动物体在所述世界坐标系下的第二位置坐标。采用本方法能够提高视觉定位的准确性。(The application relates to a visual positioning method, a visual positioning device, computer equipment and a storage medium. The method comprises the following steps: acquiring a first position coordinate of a road sign in a visual area range of a moving object, wherein the first position coordinate is a coordinate of a world coordinate system; acquiring an image of the road sign in the visual area range acquired by an image acquirer; calculating pixel coordinates of the road sign feature points in the image; determining the pose of the image capture device according to the pixel coordinates and the first position coordinates; and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object. The method can improve the accuracy of visual positioning.)

1. A visual positioning method, characterized in that the method comprises:

acquiring a first position coordinate of a road sign in a visual area range of a moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

acquiring an image of the road sign in the visual area range acquired by an image acquirer;

calculating pixel coordinates of the road sign feature points in the image;

determining the pose of the image capture device according to the pixel coordinates and the first position coordinates;

and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

2. The method of claim 1, wherein prior to obtaining the first position coordinates of the landmark in view of the mobile object, further comprising:

detecting the angular speed and the acceleration of the moving object through a sensor;

calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object;

the acquiring of the first position coordinates of the landmark in the visual area range of the moving object comprises the following steps:

determining a road mark number in the visual area range of the estimated position of the moving object;

and acquiring the first position coordinate of the numbered road sign.

3. The method of claim 2, wherein after the calculating the second position coordinate of the moving object in the world coordinate system, further comprising:

and updating the estimated position by using the second position coordinate.

4. The method of claim 2, wherein the calculating pixel coordinates of the landmark feature points in the image comprises:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

5. The method of claim 4, wherein the determining whether the region of interest image contains a complete landmark comprises:

carrying out binarization processing on the region-of-interest image to obtain a binarized image;

extracting road sign image features from the binarized image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

6. The method of claim 4, wherein the adjusting the region of interest and re-extracting the region of interest image containing the complete landmark comprises:

determining the gravity center of a road sign graph in the binary image;

and translating the region of interest in the binarized image, and extracting a region of interest image containing a complete landmark from the binarized image according to the translated region of interest.

7. The method according to any one of claims 1 to 6, wherein the imager is a binocular camera; the determining the pose of the image capture device according to the pixel coordinates and the first position coordinates comprises:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

8. A visual positioning device, the device comprising:

the acquisition module is used for acquiring a first position coordinate of the landmark in the visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

the acquisition module is also used for acquiring the road sign image in the visual area range acquired by the image acquirer;

the calculation module is used for calculating the pixel coordinates of the road sign feature points in the image;

the determining module is used for determining the pose of the image capture device according to the pixel coordinate and the first position coordinate;

the calculation module is further configured to calculate a second position coordinate of the moving object in the world coordinate system according to the pose and a pose transformation relationship between the imager and the moving object.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Technical Field

The present application relates to the field of computer vision technologies, and in particular, to a road sign-based visual positioning method and apparatus, a computer device, and a storage medium.

Background

With the development of computers and digital image processing technology, visual positioning technology has emerged. The visual positioning technology is widely applied to the fields of robots, unmanned driving, unmanned planes and the like. According to the currently common visual odometry method based on continuous frames, the positioning error gradually increases along with the increase of the system running time, so that the problem of inaccurate positioning is caused.

Disclosure of Invention

In view of the above, there is a need to provide a visual positioning method, apparatus, computer device and storage medium capable of accurately positioning in an application environment lacking stable and unique visual feature points.

A visual positioning method, characterized in that the method comprises:

acquiring a first position coordinate of a road sign in a visual area range of a moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

acquiring an image of the road sign in the visual area range acquired by an image acquirer;

calculating pixel coordinates of the road sign feature points in the image;

determining the pose of the image capture device according to the pixel coordinates and the first position coordinates;

and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, before obtaining the first position coordinates of the landmark in the visual area of the moving object, the method further comprises:

detecting the angular speed and the acceleration of the moving object through a sensor;

calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object;

the acquiring of the first position coordinates of the landmark in the visual area range of the moving object comprises the following steps:

determining a road mark number in the visual area range of the estimated position of the moving object;

and acquiring the first position coordinate of the numbered road sign.

In one embodiment, after the calculating the second position coordinate of the moving object in the world coordinate system, further comprises:

and updating the estimated position by using the second position coordinate.

The calculating the pixel coordinates of the road sign feature points in the image comprises:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the determining whether the region of interest image includes a complete landmark includes:

carrying out binarization processing on the region-of-interest image to obtain a binarized image;

extracting road sign image features from the binarized image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the adjusting the region of interest and re-extracting the region of interest image containing the complete landmark comprises:

determining the gravity center of a road sign graph in the binary image;

and translating the region of interest in the binarized image, and extracting a region of interest image containing a complete landmark from the binarized image according to the translated region of interest.

In one embodiment, the imager is a binocular camera; the determining the pose of the image capture device according to the pixel coordinates and the first position coordinates comprises:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

A visual positioning device, the device comprising:

the acquisition module is used for acquiring a first position coordinate of the landmark in the visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

the acquisition module is also used for acquiring the road sign image in the visual area range acquired by the image acquirer;

the calculation module is used for calculating the pixel coordinates of the road sign feature points in the image;

the determining module is used for determining the pose of the image capture device according to the pixel coordinate and the first position coordinate;

the calculation module is further configured to calculate a second position coordinate of the moving object in the world coordinate system according to the pose and a pose transformation relationship between the imager and the moving object.

In one embodiment, the apparatus further comprises:

the detection module is used for detecting the angular speed and the acceleration of the moving object through a sensor;

the calculation module is also used for calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object;

the obtaining module is also used for determining the road mark number in the visual area range of the estimated position of the moving object; and acquiring the first position coordinate of the numbered road sign.

In one embodiment, the apparatus further comprises:

and the updating module is used for updating the estimated position by utilizing the second position coordinate.

In one embodiment, the calculation module is further configured to:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the calculation module is further configured to:

carrying out binarization processing on the region-of-interest image to obtain a binarized image;

extracting road sign image features from the binarized image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the calculation module is further configured to:

determining the gravity center of a road sign graph in the binary image;

and translating the region of interest in the binarized image, and extracting a region of interest image containing a complete landmark from the binarized image according to the translated region of interest.

In one embodiment, the imager is a binocular camera; the calculation module is further to:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program implements a visual localization method comprising:

acquiring a first position coordinate of a road sign in a visual area range of a moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

acquiring an image of the road sign in the visual area range acquired by an image acquirer;

calculating pixel coordinates of the road sign feature points in the image;

determining the pose of the image capture device according to the pixel coordinates and the first position coordinates;

and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, before obtaining the first position coordinates of the landmark in the visual area of the moving object, the method further comprises:

detecting the angular speed and the acceleration of the moving object through a sensor;

calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object;

the acquiring of the first position coordinates of the landmark in the visual area range of the moving object comprises the following steps:

determining a road mark number in the visual area range of the estimated position of the moving object;

and acquiring the first position coordinate of the numbered road sign.

In one embodiment, after the calculating the second position coordinate of the moving object in the world coordinate system, further comprises:

and updating the estimated position by using the second position coordinate.

The calculating the pixel coordinates of the road sign feature points in the image comprises:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the determining whether the region of interest image includes a complete landmark includes:

carrying out binarization processing on the region-of-interest image to obtain a binarized image;

extracting road sign image features from the binarized image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the adjusting the region of interest and re-extracting the region of interest image containing the complete landmark comprises:

determining the gravity center of a road sign graph in the binary image;

and translating the region of interest in the binarized image, and extracting a region of interest image containing a complete landmark from the binarized image according to the translated region of interest.

In one embodiment, the imager is a binocular camera; the determining the pose of the image capture device according to the pixel coordinates and the first position coordinates comprises:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a visual localization method as follows:

acquiring a first position coordinate of a road sign in a visual area range of a moving object, wherein the first position coordinate is a coordinate of a world coordinate system;

acquiring an image of the road sign in the visual area range acquired by an image acquirer;

calculating pixel coordinates of the road sign feature points in the image;

determining the pose of the image capture device according to the pixel coordinates and the first position coordinates;

and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, before obtaining the first position coordinates of the landmark in the visual area of the moving object, the method further comprises:

detecting the angular speed and the acceleration of the moving object through a sensor;

calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object;

the acquiring of the first position coordinates of the landmark in the visual area range of the moving object comprises the following steps:

determining a road mark number in the visual area range of the estimated position of the moving object;

and acquiring the first position coordinate of the numbered road sign.

In one embodiment, after the calculating the second position coordinate of the moving object in the world coordinate system, further comprises:

and updating the estimated position by using the second position coordinate.

The calculating the pixel coordinates of the road sign feature points in the image comprises:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the determining whether the region of interest image includes a complete landmark includes:

carrying out binarization processing on the region-of-interest image to obtain a binarized image;

extracting road sign image features from the binarized image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the adjusting the region of interest and re-extracting the region of interest image containing the complete landmark comprises:

determining the gravity center of a road sign graph in the binary image;

and translating the region of interest in the binarized image, and extracting a region of interest image containing a complete landmark from the binarized image according to the translated region of interest.

In one embodiment, the imager is a binocular camera; the determining the pose of the image capture device according to the pixel coordinates and the first position coordinates comprises:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

According to the visual positioning method, the visual positioning device, the computer equipment and the storage medium, the image of the road sign in the visual area range is collected through the image collector, and the pixel coordinates of the road sign in the image are extracted. The position and attitude of the image capturing device can be obtained by the coordinate of the landmark pixel and the coordinate of the world coordinate system which is determined in advance through operation, and then the position and attitude of the moving object under the world coordinate system are obtained through calculation according to the position and attitude of the image capturing device and the position and attitude transformation relation between the image capturing device and the moving object. The calculated position coordinates are calculated based on the landmarks in the range of the visual area of the moving object, and the landmarks in the range of the visual area are updated in real time during the movement of the object, so the calculated position coordinates of the moving object are not accumulated to generate error accumulation. In addition, a distance is usually fixed at intervals between different road signs, and the road signs at the fixed intervals are used for visual positioning, so that even if other characteristic points in the visual range of the moving object are few, the position coordinates of the moving object can be calculated according to the road signs at the fixed intervals, the system is suitable for the environment without stable and unique visual characteristic points, the system runs stably, and the positioning accuracy is high.

Drawings

FIG. 1 is a diagram of an exemplary embodiment of a visual positioning method;

FIG. 2 is a flow diagram of a visual location method in accordance with one embodiment;

FIG. 3 is a schematic view of a visual positioning system in one embodiment;

FIG. 4 is a schematic flow chart illustrating the process of determining whether an image includes a complete landmark in one embodiment;

FIG. 5 is a flow diagram illustrating a method for visual location in one embodiment;

FIG. 6 is a flowchart of a visual positioning method according to another embodiment;

FIG. 7 is a block diagram of a visual positioning apparatus in one embodiment;

FIG. 8 is a block diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The visual positioning method provided by the application can be applied to the application environment shown in fig. 1. The application environment includes a moving object 102 and a road sign 104. The method comprises the steps that a moving object 102 obtains a first position coordinate of a landmark 104 in a visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system; acquiring an image of the landmark 104 within the visual area acquired by the imager; calculating pixel coordinates of the landmark characteristic points in the image; determining the pose of the image capture device according to the pixel coordinates and the first position coordinates; and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

The moving object 102 may be an unmanned vehicle, an unmanned aerial vehicle, a robot, or an unmanned ship. The signpost 104 is a sign that may have an indicating effect. The road sign can be various patterns drawn on the ground (such as a diamond road sign in fig. 1), or a sign board standing on the ground, or float on the water surface in the form of a buoy, or be suspended in other ways.

In one embodiment, as shown in FIG. 2, a method for visually locating a moving object includes the steps of:

s202, obtaining a first position coordinate of the landmark in the visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system.

And the first position coordinate of the road sign in the visual area range of the moving object is a coordinate in a world coordinate system. The coordinates of the world coordinate system are absolute coordinates, and the coordinates of all points on the screen are determined by the origin of the world coordinate system before the user coordinate system is established.

The visual area range is the range which can be shot by a camera on the moving object or the range which can be scanned by a radar on the moving object.

In one embodiment, the landmark may be a diamond landmark. Correspondingly, the moving object obtains the first position coordinates of the rhombic road signs in the visual area range of the moving object.

In one embodiment, the moving object is located by a built-in GPS locating system or a mobile terminal arranged in the moving object to obtain the position information of the moving object, the number of the landmark in the visual area range of the moving object can be determined according to the position information of the moving object, and the first position coordinate of the landmark is obtained according to the number.

In another embodiment, the road signs are marked with numbers, and each number corresponds to a first position coordinate measured in advance. The moving object collects images of the road signs in the visual area range through the image capturing device, the number of the images is identified through a digital image processing technology, the number of the road signs in the visual area range is obtained, and then the first position coordinates of the road signs in the visual area range are obtained according to the number.

In one embodiment, before S202, the method further includes the following steps: the moving object obtains the angular velocity and the acceleration of the moving object through a sensor; obtaining the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object; determining a road mark number in the visual area range of the estimated position of the moving object; and acquiring the first position coordinates of the numbered signposts.

The sensor is a motion data detection device and can detect the angular velocity and the acceleration of a detected moving object. The displacement of the moving object relative to the initial position can be calculated through the angular velocity and the acceleration measured by the sensor, and then the position of the moving object (namely the estimated position) is obtained. The angular velocity may be the rotational velocity of the wheels of the moving object.

In one embodiment, the moving object converts the angular velocity into a linear velocity, and the estimated position of the moving object is obtained according to the linear velocity, the acceleration and the initial position of the moving object.

In one embodiment, the sensor includes a gyroscope and an accelerometer. The gyroscope measures angular velocities of three axes of X, Y and Z according to the principle of angular momentum conservation. The accelerometer measures acceleration in the three axes X, Y, Z.

In one embodiment, the moving object obtains the number of the road sign closest to the moving object in front of the estimated position according to the estimated position.

In one embodiment, when the number of the landmark and the corresponding position coordinate are stored in the moving object, the moving object acquires the corresponding first position coordinate from the storage area according to the number of the landmark after acquiring the number of the landmark.

In one embodiment, when the number of the landmark and the corresponding position coordinate are stored in the server, the moving object generates a position coordinate acquisition request carrying the number, and sends the position coordinate acquisition request to the server, so that the server acquires the corresponding first position coordinate according to the number in the position coordinate acquisition request and sends the first position coordinate to the moving object.

And S204, acquiring the road sign image in the visual area range acquired by the image acquirer.

The image pick-up device belongs to a part of a visual positioning system of a moving object and is a device for obtaining a road sign image. The visual positioning system may further include a sensor 304 and a processor 306 in addition to the image capture device 302, as shown in fig. 3.

In one embodiment, a moving object acquires an image of a landmark through the imager 302. The sensor 306 collects motion data. The image grabber 302 and the sensor 304 transmit the acquired data to the processor 306 for processing.

In one embodiment, the image capture device may be a camera. The camera converts the optical signal into an electrical signal through the photosensitive element.

In one embodiment, the image capture device may be a lidar. A moving object emits a beam of laser through a laser transmitter in the laser radar, and then the laser is projected to the moving object to be reflected. The moving object receives the reflected laser through a laser receiver in the laser radar, senses the external environment through the received reflected laser, and reads point cloud data of the surrounding environment. And processing the point cloud data by the moving object to obtain a gray image of the moving object.

In one embodiment, the image capture device may be a binocular camera, the binocular camera may be a camera having two cameras, images captured by the two cameras are similar to images seen by the left and right eyes of a person, the two images have parallax, and the farther the object distance is, the smaller the parallax is; conversely, the greater the parallax. The same point has different pixel locations in the left and right eye images of the binocular camera. The distance between the object and the image capturing device can be calculated by the moving object according to the pixel positions of the object in the left eye image and the right eye image.

The image pick-up device has a certain visual area range, and the area which can be collected by the image pick-up device is limited. In order to be able to see clearly the road sign closest to the moving object for as much time as possible, in one embodiment the angle at which the image pick-up is mounted is slightly tilted downwards.

And S206, calculating the pixel coordinates of the landmark characteristic points in the image.

The pixel coordinates of the landmark characteristic points are relative coordinates of the landmark characteristic points in the image collected by the image collector.

In one embodiment, the moving object first performs edge detection on the image to extract the contour of the road sign. And extracting the landmark characteristic points from the contour by the moving object, and calculating the pixel coordinates of the landmark characteristic points in the image.

In another embodiment, after the image in the visual area range is acquired by the image acquirer, the moving object sharpens the acquired image to enhance the edge and the gray jump part of the image so as to enable the image contour to become clear. And extracting the contour of the road sign from the sharpened image by the moving object by adopting an edge detection algorithm. And extracting the landmark characteristic points from the contour by the moving object, and calculating the pixel coordinates of the landmark characteristic points in the image. The edge detection algorithm may be a sobel operator, a Roberts operator, a Canny operator, a Prewitt operator, or a Laplacian operator.

In one embodiment, the fast (features From estimated Segment test) algorithm is used to extract landmark feature points in the image. The FAST algorithm judges whether the candidate feature points can serve as the feature points or not based on the difference value of the gray values of the candidate feature points and the surrounding pixel points.

The image acquired by the image capturing device not only contains the road sign, but also contains other irrelevant backgrounds. In order to reduce the interference of other unrelated backgrounds to the algorithm, in one embodiment, a moving object first predicts a region in an image where a landmark is located, and the predicted region in the image where the landmark is located is used as a region of interest. And the moving object performs edge detection and feature extraction on the region of interest by using a digital image processing technology to obtain landmark feature points, and then calculates pixel coordinates of the landmark feature points.

In one embodiment, a moving object determines a region of interest in the image based on the first position coordinates of the landmark.

In one embodiment, the moving object determines the region of interest based on the estimated position of the moving object. After the moving object obtains the estimated position, the first position coordinate of the road sign closest to the moving object in the visual area range can be judged. The moving object then determines a region of interest in the image based on the first position coordinates.

For a captured image, it may occur that a part of the landmark is located inside the region of interest and another part is located outside the region of interest. And if the region of interest selected according to the estimated position does not contain a complete landmark, the moving object adjusts the region of interest, so that the landmark is positioned in the image of the region of interest, and the landmark detection is convenient for the next step.

In one embodiment, the moving object extracts a region-of-interest image from the image acquired by the image capture device according to the region of interest. Then, S402 in fig. 4 is executed, that is, it is determined whether the region of interest image contains a complete landmark. If not, executing S404, namely executing adjusting the region of interest, and re-extracting the region of interest image containing the complete landmark. If the complete landmark has been included, S406 is performed, i.e. calculating the visual feature point pixel coordinates of the landmark is performed.

In one embodiment, the moving object extracts all feature points of the landmarks in the image of the region of interest according to the extraction rule, and if the number of the extracted landmark feature points is less than the preset number, it is determined that the image of the region of interest does not contain complete landmarks.

In another embodiment, the step of determining whether the image of the region of interest includes a complete landmark includes: and carrying out binarization processing on the image of the region of interest by the moving object to obtain a binarized image. And extracting the road sign image characteristics from the binary image. And comparing the extracted road sign image characteristic with a preset road sign characteristic. And determining whether the interested area image contains a complete landmark or not according to the comparison result. If the extracted landmark features are completely the same as the preset features, the image acquired by the image capture device comprises complete landmarks, and if the extracted landmark features are not completely the same as the preset features, the image acquired by the image capture device does not comprise complete landmarks.

The binarization algorithm determines whether a pixel belongs to the background or the foreground by a certain threshold, if the pixel belongs to the background, the pixel is assigned to be 0, otherwise, the pixel is assigned to be 1 (or 255). Because the color and the brightness of the road sign are greatly different from those of the background area, the image of the region of interest is subjected to binarization processing to obtain a binarized image, and the road sign can be separated from the background. In one embodiment, the moving object performs binarization processing on the image of the region of interest by using an adaptive threshold algorithm. The adaptive threshold algorithm is that the average threshold of the image region is obtained through judgment and calculation to carry out iteration, different thresholds are adopted for different parts, and the adaptive threshold algorithm has high robustness.

And S208, determining the pose of the image capture device according to the pixel coordinates and the first position coordinates.

The pixel coordinates of the landmark are the coordinates of the landmark in the image within the field of view captured by the imager, and the first position coordinates of the landmark are the absolute coordinates of the landmark in a world coordinate system.

In one embodiment, the camera is a binocular camera, and the moving object can calculate the distance from the camera according to the parallax of the left and right images of the road sign shot by the binocular camera. Since the first position coordinates of the landmark are known, the pose of the camera can be calculated from the landmark and the position of the camera.

In one embodiment, a rhombic road sign is adopted, and the rhombic road sign has the characteristics of small calculation complexity and good real-time performance; the method for calculating the pose of the image capture device comprises the following steps:

the coordinate of four corner points of the known signpost diamond in the world coordinate system is p₁,p₂,p₃,p₄. The corresponding pixel coordinates in the left and right eye images collected by the binocular camera areComputing pose of camera in world coordinate system by minimizing reprojection error of three-dimensional point to imageThe left and right eye reprojection errors are respectively expressed as follows:

wherein, K_lAnd K_rRespectively, the internal reference matrixes of the left camera and the right camera,and converting the left eye to the right eye calibrated in advance. Solving the nonlinear least square problem through ceres-solution to obtain a pose, wherein the pose has the following expression:

wherein the content of the first and second substances,the estimated value and the estimated value of the pose of the camera are obtained by solving in an optimization modeApproximately equal to true value

And S210, calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, a pose of the imager and a center position of the moving object have a pose transformation relationship, and after the pose of the imager is obtained, the pose of the moving object can be obtained according to the pose transformation relationship.

In the above embodiment, the image of the road sign in the visual area range is collected by the image capture device, and the pixel coordinates of the road sign in the image are extracted. The position and attitude of the image capturing device can be obtained by the coordinate of the landmark pixel and the coordinate of the world coordinate system which is determined in advance through operation, and then the position and attitude of the moving object under the world coordinate system are obtained through calculation according to the position and attitude of the image capturing device and the position and attitude transformation relation between the image capturing device and the moving object. The calculated position coordinates are calculated based on the landmarks in the range of the visual area of the moving object, and the landmarks in the range of the visual area are updated in real time during the movement of the object, so the calculated position coordinates of the moving object are not accumulated to generate error accumulation. In addition, a distance is usually fixed at intervals between different road signs, and the road signs at the fixed intervals are used for visual positioning, so that even if other characteristic points in the visual range of the moving object are few, the position coordinates of the moving object can be calculated according to the road signs at the fixed intervals, the system is suitable for the environment without stable and unique visual characteristic points, the system runs stably, and the positioning accuracy is high.

In one embodiment, as shown in FIG. 5, a method for visually locating a moving object includes the steps of:

and S502, detecting the angular speed and the acceleration of the moving object through the sensor.

S504, the estimated position of the moving object is calculated according to the angular velocity, the acceleration and the initial position of the moving object.

S506, determining the road mark number in the visual area range of the estimated position of the moving object.

And S508, acquiring the first position coordinates of the numbered road signs.

And S510, acquiring the road sign image in the visual area range acquired by the image acquirer.

And S512, calculating the pixel coordinates of the landmark characteristic points in the image.

And S514, determining the pose of the image capture device according to the pixel coordinates and the first position coordinates.

And S516, calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

S502 to S516, the method for calculating the second position coordinate of the moving object is as described above.

And S518, calculating the reliability of the landmark feature points.

Because the estimated position of the moving object calculated by the moving object according to the angular velocity and the acceleration of the moving object acquired by the sensor has a certain error, and the error is accumulated continuously along with the time, the estimated position needs to be updated to improve the positioning accuracy. In the moving process of the moving object, due to the fact that the visual area range of the image capturing device is limited, the situation that a complete road sign is collected at any moment cannot be guaranteed. And the landmark profile obtained through the image processing technique may not be the same as the actual profile, so the extracted feature points of the landmark may not be reliable. So that the pose of the image capture device calculated by the moving object according to the pixel coordinates of the landmark feature points in the image of the region of interest and the first position coordinates of the landmark feature points may not be accurate. Therefore, the reliability of the landmark feature points is first calculated.

In one embodiment, the shape of the road sign is a diamond. The measure of the reliability of the landmark feature points is the pixel difference between the detected landmark profile and the ideal landmark profile obtained by the vertex. Specifically, the algorithm may output a road sign contour, but the four sides of the contour are not completely straight lines, and a graph with four straight lines can be constructed by four vertices of the contour, and empirically, the closer the two contours are, the higher the reliability of the detected diamond is. The average of the pixel differences of the two contours is therefore used as a measure.

S520, judging whether the reliability of the landmark feature points is greater than a preset threshold value.

In one embodiment, the threshold value is selected to be 5 pixels, i.e. if the average value of the pixel difference between the two contours is less than 5 pixels, the diamond detected at this time is considered reliable and can be used to calculate the position of the moving object.

If the estimated position is larger than the preset threshold value, the reliability of the landmark is considered to be high, S522 is executed, and the estimated position is updated by using the second position coordinate; and if the reliability of the landmark is lower than the preset threshold, the reliability of the landmark is considered to be low, and the moving object is positioned by the estimated position.

And when the reliability of the road signs is greater than a threshold value, the moving object calculates a second position coordinate, and when the reliability of the road signs between two selected road signs is greater than a preset threshold value, the second position coordinate of the moving object is updated. That is, when the reliability of the road sign is low, the motion information collected by the sensor is used for estimating the position of the moving object. But as this method of estimation increases over time, the error increases. Therefore, the estimated position of the moving object is updated after a period of time, the accumulated error of the estimated position coordinates can be reduced, and the accuracy of positioning the moving object is improved.

In one embodiment, as shown in fig. 6, the methods of S602-S612 are as described above. If the reliability of the landmark is greater than the preset threshold, S616 to S620 are performed to calculate the second position coordinates of the moving object, and the methods of S616 to S620 are as described above. Then, S622 is executed to determine whether the moving object passes through the new landmark. And each time a landmark passes, selecting the positioning result with the highest reliability from the previous landmark to pass the landmark, and updating the second position coordinate of the moving object.

And updating the estimated position by adopting the second position coordinate of the moving object when the reliability of the road sign between the two road signs is highest, so that the positioning accuracy can be improved to the maximum extent.

It should be understood that although the various steps in the flow charts of fig. 2-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-4 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.

In one embodiment, a visual positioning apparatus is provided, specifically comprising: an acquisition module 702, a determination module 704, and a calculation module 706; wherein:

an obtaining module 702, configured to obtain a first position coordinate of a landmark in a visual area range of a moving object, where the first position coordinate is a coordinate of a world coordinate system;

the acquiring module 702 is further configured to acquire an image of the landmark within the visual area acquired by the image capturing device;

a calculating module 706, configured to calculate pixel coordinates of landmark feature points in the image;

a determining module 704, configured to determine a pose of the imager according to the pixel coordinates and the first position coordinates;

the calculating module 706 is further configured to calculate a second position coordinate of the moving object in the world coordinate system according to the pose and the pose transformation relationship between the imager and the moving object.

In one embodiment, as shown in fig. 7, the apparatus further comprises:

a detection module 708, configured to detect an angular velocity and an acceleration of the moving object through the sensor;

the calculating module 706 is further configured to calculate an estimated position of the moving object according to the angular velocity, the acceleration, and the initial position of the moving object;

the obtaining module 702 is further configured to determine a landmark number within a view area range of the estimated position of the moving object; and acquiring the first position coordinates of the numbered signposts.

In one embodiment, as shown in fig. 7, the apparatus further comprises:

and an updating module 710 for updating the estimated position by using the second position coordinate.

In one embodiment, the calculation module 706 is further configured to:

determining an interested area in the image according to the estimated position;

extracting a region-of-interest image from the image according to the region-of-interest;

judging whether the image of the region of interest contains a complete landmark or not;

if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign;

and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the calculation module 706 is further configured to:

carrying out binarization processing on the image of the region of interest to obtain a binarized image;

extracting road sign image features from the binary image;

comparing the extracted road sign image characteristic with a preset road sign characteristic;

and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the calculation module 706 is further configured to:

determining the gravity center of a road sign graph in a binary image;

and translating the region of interest in the binary image, and extracting a region of interest image containing the complete road sign from the binary image according to the translated region of interest.

In one embodiment, the imager is a binocular camera; the calculation module is further to:

calculating left and right eye re-projection errors of the binocular camera;

and calculating the pose of the image capture device according to the reprojection error.

For specific definition of the visual positioning device, reference may be made to the definition of the visual positioning method above, and details are not repeated here. The various modules in the visual positioning apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

As shown in fig. 8, in an embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the above-described visual positioning method.

In one embodiment, a computer device is provided, which may be a moving object, and the internal structure of which may be as shown in fig. 8. The computer equipment comprises a processor, a memory, a display screen, an input device and a communication interface which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a visual positioning method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring a first position coordinate of the road sign in the visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system; acquiring an image of the road sign within the visual area range acquired by an image acquirer; calculating pixel coordinates of the landmark characteristic points in the image; determining the pose of the image capture device according to the pixel coordinates and the first position coordinates; and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, the processor, when executing the computer program, further performs the steps of: detecting the angular speed and the acceleration of a moving object through a sensor; calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object; determining a road mark number in the visual area range of the estimated position of the moving object; and acquiring the first position coordinates of the numbered signposts.

In one embodiment, the processor, when executing the computer program, further performs the steps of: and updating the estimated position by using the second position coordinate.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining an interested area in the image according to the estimated position; extracting a region-of-interest image from the image according to the region-of-interest; judging whether the image of the region of interest contains a complete landmark or not; if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign; and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the processor, when executing the computer program, further performs the steps of: carrying out binarization processing on the image of the region of interest to obtain a binarized image; extracting road sign image features from the binary image; comparing the extracted road sign image characteristic with a preset road sign characteristic; and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining the gravity center of a road sign graph in a binary image; and translating the region of interest in the binary image, and extracting a region of interest image containing the complete road sign from the binary image according to the translated region of interest.

In one embodiment, the processor, when executing the computer program, further performs the steps of: calculating left and right eye re-projection errors of the binocular camera; and calculating the pose of the image capture device according to the reprojection error.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

acquiring a first position coordinate of the road sign in the visual area range of the moving object, wherein the first position coordinate is a coordinate of a world coordinate system; acquiring an image of the road sign within the visual area range acquired by an image acquirer; calculating pixel coordinates of the landmark characteristic points in the image; determining the pose of the image capture device according to the pixel coordinates and the first position coordinates; and calculating a second position coordinate of the moving object under the world coordinate system according to the pose and the pose transformation relation between the image capture device and the moving object.

In one embodiment, the computer program when executed by the processor further performs the steps of: detecting the angular speed and the acceleration of a moving object through a sensor; calculating the estimated position of the moving object according to the angular velocity, the acceleration and the initial position of the moving object; the method for acquiring the first position coordinates of the road sign in the visual area range of the moving object comprises the following steps: determining a road mark number in the visual area range of the estimated position of the moving object; and acquiring the first position coordinates of the numbered signposts.

In one embodiment, the computer program when executed by the processor further performs the steps of: and updating the estimated position by using the second position coordinate.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining an interested area in the image according to the estimated position; extracting a region-of-interest image from the image according to the region-of-interest; judging whether the image of the region of interest contains a complete landmark or not; if not, adjusting the region of interest, and re-extracting the region of interest image containing the complete road sign; and calculating the pixel coordinates of the landmark feature points according to the re-extracted region-of-interest image.

In one embodiment, the computer program when executed by the processor further performs the steps of: carrying out binarization processing on the image of the region of interest to obtain a binarized image; extracting road sign image features from the binary image; comparing the extracted road sign image characteristic with a preset road sign characteristic; and determining whether the interested area image contains a complete landmark or not according to the comparison result.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining the gravity center of a road sign graph in a binary image; and translating the region of interest in the binary image, and extracting a region of interest image containing the complete road sign from the binary image according to the translated region of interest.

In one embodiment, the computer program when executed by the processor further performs the steps of: calculating left and right eye re-projection errors of the binocular camera; and calculating the pose of the image capture device according to the reprojection error.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile memory may include Read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.