阅读说明：本技术 基于无人机倾斜影像的真正射影像生成方法和装置 (True orthoimage generation method and device based on unmanned aerial vehicle oblique image ) 是由 刘建明 王海强 吴丽沙 张谷生 杨斌 王慧静 于 2021-06-15 设计创作，主要内容包括：本公开提供了基于无人机倾斜影像的真正射影像生成方法,包括：获取无人机倾斜影像集中的无人机倾斜影像对应的位置和姿态信息,根据位置和姿态信息确定无人机倾斜影像间的相对位置关系；根据相对位置关系对无人机倾斜影像进行特征点匹配,确定相邻影像上的像点间的对应关系；根据位置和姿态信息,以及像点间的对应关系计算无人机倾斜影像的定向参数；利用深度图方法获取密集点云,根据获取的密集点云利用泊松曲面重建算法进行曲面重构,获取无人机倾斜影像对应的场景的三维网格,在三维网格上进行纹理映射；对纹理映射后的三维网格进行正射投影,生成真正射影像。以此方式,能够有效利用无序的数据集进行快速准确的平差解算实现大场景的三维重建。(The utility model provides a real projective image generation method based on unmanned aerial vehicle slope image, include: acquiring position and attitude information corresponding to the unmanned aerial vehicle oblique images in the unmanned aerial vehicle oblique image set, and determining the relative position relationship between the unmanned aerial vehicle oblique images according to the position and attitude information; performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship, and determining the corresponding relationship between image points on adjacent images; calculating orientation parameters of the unmanned aerial vehicle oblique image according to the position and attitude information and the corresponding relation between the image points; acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to an unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid; and performing orthographic projection on the three-dimensional grid after texture mapping to generate a true orthographic image. In this way, the three-dimensional reconstruction of a large scene can be realized by effectively utilizing the unordered data set to carry out quick and accurate adjustment calculation.)

1. A true orthoimage generation method based on unmanned aerial vehicle oblique images is characterized by comprising the following steps:

acquiring position and attitude information corresponding to unmanned aerial vehicle oblique images in an unmanned aerial vehicle oblique image set, and determining a relative position relation between the unmanned aerial vehicle oblique images according to the position and attitude information;

performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship, and determining the corresponding relationship between image points on adjacent images;

calculating orientation parameters of the unmanned aerial vehicle oblique image according to the position and attitude information and the corresponding relation between the image points;

acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to an unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid;

and performing orthographic projection on the three-dimensional grid after texture mapping to generate a true orthographic image.

2. The method for generating the true orthoimage based on the oblique image of the unmanned aerial vehicle as claimed in claim 1, wherein the obtaining of the position and the attitude information corresponding to the oblique image of the unmanned aerial vehicle in the oblique image set of the unmanned aerial vehicle and the determining of the relative position relationship between the oblique images of the unmanned aerial vehicle according to the position and the attitude information comprises:

the method comprises the steps of obtaining the position and attitude information of an unmanned aerial vehicle when the unmanned aerial vehicle shoots the unmanned aerial vehicle oblique images concentrated in the unmanned aerial vehicle oblique images, determining the course or the lateral approximate overlapping area of the unmanned aerial vehicle oblique images, and determining the relative position relation between the unmanned aerial vehicle oblique images according to the approximate overlapping area.

3. The method according to claim 2, wherein the determining the relative position relationship between the oblique images of the unmanned aerial vehicle according to the approximate overlapping area comprises:

for two unmanned aerial vehicle inclined images with overlapped areas in the course or the lateral direction, one of the two unmanned aerial vehicle inclined images is used as a reference image, the other one of the two unmanned aerial vehicle inclined images is used as a search image, and the affine transformation relation between the reference image and the search image is determined according to feature points corresponding to image points with the same name in the overlapped areas of the reference image and the search image.

4. The method for generating true ortho images based on unmanned aerial vehicle oblique images according to claim 3, wherein the calculating orientation parameters of the oblique images according to the position and posture information and the corresponding relationship between the image points comprises:

determining an imaging model of the unmanned aerial vehicle oblique image according to the position and posture information of the unmanned aerial vehicle oblique image;

linearizing the imaging model according to a Taylor series to generate a linearized model;

taking the linearized model as a basic model of adjustment, and establishing an error equation set by using coordinates of feature points on the unmanned aerial vehicle oblique image;

establishing a normal equation set according to the error equation set, determining the correction number of the external orientation element corresponding to the position and attitude information by solving the normal equation set, correcting the external orientation element corresponding to the position and attitude information according to the correction number, and further determining the ground coordinate corresponding to the feature point on the unmanned aerial vehicle oblique image.

5. The method for generating true ortho images based on unmanned aerial vehicle oblique images according to claim 4, wherein the method further comprises:

and calculating the ground coordinates of other points outside the overlapping area in the unmanned aerial vehicle oblique image by utilizing multi-sheet front intersection.

6. The method for generating a true ortho image based on unmanned aerial vehicle oblique image according to claim 5, wherein the obtaining dense point cloud by using a depth map method comprises:

selecting an inclined image pair according to a preset condition;

determining corresponding ground points according to the image points with the same name in the oblique image pair;

deleting the wrong depth map points by using the neighborhood depth map to obtain an accurate depth map corresponding to each unmanned aerial vehicle oblique image;

and fusing the accurate depth map to generate dense point cloud.

7. The method for generating a true ortho-image based on an oblique image of an unmanned aerial vehicle according to claim 6, wherein the obtaining of the dense point cloud by using the depth map method, the performing of the curved surface reconstruction by using the poisson curved surface reconstruction algorithm according to the obtained dense point cloud, the obtaining of the three-dimensional mesh of the scene corresponding to the oblique image of the unmanned aerial vehicle, and the performing of the texture mapping on the three-dimensional mesh comprise:

the method comprises the steps of converting curved surface reconstruction of dense point cloud into solving Poisson equation, calculating a gradient field and a vector field by constructing the Poisson equation, selecting equivalent values meeting preset conditions to obtain a reconstructed curved surface which is best approximate to original point cloud data, using the square of volume change as an edge folding grid simplification algorithm of error measurement, and adding a triangle normal constraint factor into the error measurement to simplify the reconstructed curved surface.

8. The utility model provides a really penetrate image generation device based on unmanned aerial vehicle slope image which characterized in that includes:

the relative position relation determining module is used for acquiring position and attitude information corresponding to the unmanned aerial vehicle oblique images in the unmanned aerial vehicle oblique image set, and determining the relative position relation between the unmanned aerial vehicle oblique images according to the position and attitude information;

the feature point matching module is used for performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship and determining the corresponding relationship between image points on adjacent images;

the orientation parameter determining module is used for calculating the orientation parameters of the unmanned aerial vehicle oblique images according to the position and attitude information and the corresponding relation between the image points;

the curved surface reconstruction module is used for acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to the unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid;

and the true orthoimage generation module is used for performing orthoprojection on the three-dimensional grid after the texture mapping to generate a true orthoimage.

9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the processor, when executing the program, implements the method of any of claims 1-7.

10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the method according to any one of claims 1 to 7.

Technical Field

Embodiments of the present disclosure relate generally to the field of image processing technology, and more particularly, to a method and an apparatus for generating a true orthophoto image based on an unmanned aerial vehicle oblique image.

Background

The oblique image (oblique image) is an image obtained by an aerial camera with a certain inclination angle, and may be referred to as a multi-view oblique image. The oblique photography technology is a high and new technology developed in the field of international surveying and mapping remote sensing in recent years, and obtains more complete and accurate information of ground objects by carrying a plurality of sensors on the same flight platform and acquiring images from different angles such as vertical angle, inclination angle and the like.

With the continuous development of image acquisition means and higher requirements of various demand departments on images, more and more large-scale orthoimages enter the production field. But large scales present many new challenges for both aerial photography and agricultural production. Particularly in urban areas, splicing of orthographic images and transition of land features in border areas are difficult to realize, and high buildings in areas with dense buildings shield land surface information, so that a small amount of land feature information is lacked when the traditional orthographic images are used.

Disclosure of Invention

According to the embodiment of the disclosure, a true ortho image generation scheme based on the unmanned aerial vehicle oblique image is provided, wherein the true ortho image generation scheme can still acquire the ortho image of the ground feature information under the condition that the ground surface information is blocked by a tall building.

In a first aspect of the present disclosure, a real projective image generating method based on oblique images of an unmanned aerial vehicle is provided, including:

acquiring position and attitude information corresponding to unmanned aerial vehicle oblique images in an unmanned aerial vehicle oblique image set, and determining a relative position relation between the unmanned aerial vehicle oblique images according to the position and attitude information;

performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship, and determining the corresponding relationship between image points on adjacent images;

calculating orientation parameters of the unmanned aerial vehicle oblique image according to the position and attitude information and the corresponding relation between the image points;

acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to an unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid;

and performing orthographic projection on the three-dimensional grid after texture mapping to generate a true orthographic image.

In some embodiments, the obtaining position and posture information corresponding to the unmanned aerial vehicle oblique images in the unmanned aerial vehicle oblique image set, and determining a relative position relationship between the unmanned aerial vehicle oblique images according to the position and posture information includes:

the method comprises the steps of obtaining the position and attitude information of an unmanned aerial vehicle when the unmanned aerial vehicle shoots the unmanned aerial vehicle oblique images concentrated in the unmanned aerial vehicle oblique images, determining the course or the lateral approximate overlapping area of the unmanned aerial vehicle oblique images, and determining the relative position relation between the unmanned aerial vehicle oblique images according to the approximate overlapping area.

In some embodiments, the determining the relative position relationship between the oblique images of the drones according to the approximate overlap region includes:

for two unmanned aerial vehicle inclined images with overlapped areas in the course or the lateral direction, one of the two unmanned aerial vehicle inclined images is used as a reference image, the other one of the two unmanned aerial vehicle inclined images is used as a search image, and the affine transformation relation between the reference image and the search image is determined according to feature points corresponding to image points with the same name in the overlapped areas of the reference image and the search image.

In some embodiments, the calculating orientation parameters of the oblique image according to the position and posture information and the corresponding relationship between the image points includes:

determining an imaging model of the unmanned aerial vehicle oblique image according to the position and posture information of the unmanned aerial vehicle oblique image;

linearizing the imaging model according to a Taylor series to generate a linearized model;

taking the linearized model as a basic model of adjustment, and establishing an error equation set by using coordinates of feature points on the unmanned aerial vehicle oblique image;

establishing a normal equation set according to the error equation set, determining the correction number of the external orientation element corresponding to the position and attitude information by solving the normal equation set, correcting the external orientation element corresponding to the position and attitude information according to the correction number, and further determining the ground coordinate corresponding to the feature point on the unmanned aerial vehicle oblique image.

In some embodiments, the method further comprises:

and calculating the ground coordinates of other points outside the overlapping area in the unmanned aerial vehicle oblique image by utilizing multi-sheet front intersection.

In some embodiments, the obtaining a dense point cloud using a depth map method comprises:

selecting an inclined image pair according to a preset condition;

determining corresponding ground points according to the image points with the same name in the oblique image pair;

deleting the wrong depth map points by using the neighborhood depth map to obtain an accurate depth map corresponding to each unmanned aerial vehicle oblique image;

and fusing the accurate depth map to generate dense point cloud.

In some embodiments, the obtaining of the dense point cloud by using the depth map method, performing surface reconstruction by using a poisson surface reconstruction algorithm according to the obtained dense point cloud, obtaining a three-dimensional mesh of a scene corresponding to an oblique image of the unmanned aerial vehicle, and performing texture mapping on the three-dimensional mesh includes:

the method comprises the steps of converting curved surface reconstruction of dense point cloud into solving Poisson equation, calculating a gradient field and a vector field by constructing the Poisson equation, selecting equivalent values meeting preset conditions to obtain a reconstructed curved surface which is best approximate to original point cloud data, using the square of volume change as an edge folding grid simplification algorithm of error measurement, and adding a triangle normal constraint factor into the error measurement to simplify the reconstructed curved surface.

In a second aspect of the present disclosure, a real projective image generating apparatus based on oblique images of an unmanned aerial vehicle is provided, including:

the relative position relation determining module is used for acquiring position and attitude information corresponding to the unmanned aerial vehicle oblique images in the unmanned aerial vehicle oblique image set, and determining the relative position relation between the unmanned aerial vehicle oblique images according to the position and attitude information;

the feature point matching module is used for performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship and determining the corresponding relationship between image points on adjacent images;

the orientation parameter determining module is used for calculating the orientation parameters of the unmanned aerial vehicle oblique images according to the position and attitude information and the corresponding relation between the image points;

the curved surface reconstruction module is used for acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to the unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid;

and the true orthoimage generation module is used for performing orthoprojection on the three-dimensional grid after the texture mapping to generate a true orthoimage.

In a third aspect of the present disclosure, an electronic device is provided, comprising a memory having stored thereon a computer program and a processor implementing the method as described above when executing the program.

In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method as set forth above.

It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Through this disclosed real projective image generation method based on unmanned aerial vehicle slope image, can obtain the real projective image of this ground feature information according to the unmanned aerial vehicle slope image that is sheltered from ground feature information to there has been overcome and has been sheltered from the technical problem that there is the disappearance in the orthophoto image of ground feature information, thereby can generate the orthophoto of more complete ground feature information.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:

fig. 1 shows a flowchart of a method for generating a true ortho image based on an oblique image of an unmanned aerial vehicle according to a first embodiment of the present disclosure;

fig. 2 shows a flowchart of texture mapping of a true orthoscopic image generation method based on oblique images of the unmanned aerial vehicle according to a second embodiment of the disclosure;

fig. 3 is a functional structure diagram of a real projective image generating apparatus based on oblique imagery of an unmanned aerial vehicle according to a third embodiment of the present disclosure;

fig. 4 shows a schematic structural diagram of a real projective image generating apparatus based on oblique imagery of an unmanned aerial vehicle according to a fourth embodiment of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The real orthophoto generation method based on unmanned aerial vehicle inclined image of this disclosed embodiment can acquire the orthophoto of ground feature information when having the shelter from the thing, can realize the concatenation of orthophoto and the transition of the regional ground feature of bordering for the orthophoto of ground feature information is more complete.

Specifically, as shown in fig. 1, a flowchart of a method for generating a true ortho image based on an oblique image of an unmanned aerial vehicle according to a first embodiment of the present disclosure is shown. As shown in fig. 1, the method of this embodiment may include the following steps:

s101: the method comprises the steps of obtaining the position and the attitude information corresponding to the unmanned aerial vehicle oblique images in the unmanned aerial vehicle oblique image set, and determining the relative position relation between the unmanned aerial vehicle oblique images according to the position and the attitude information.

The method of the embodiment can be applied to the production of a large-scale real projection image of a surface feature target in an urban area, and specifically, can be used for producing an orthoscopic image of a surface feature target with a shelter, such as splicing buildings in the urban area and producing an orthoscopic image of a surface feature target at a bordering area position. In the embodiment of the disclosure, the oblique image of the target area is acquired by the unmanned aerial vehicle, and then the corresponding real projective image is made according to the acquired oblique influence.

When the method of this embodiment is used to produce the true orthographic image of the target area, first, the oblique image of the drone in the target area needs to be acquired. In general, an oblique image of a target area can be acquired through the unmanned aerial vehicle, a route of the unmanned aerial vehicle can be predetermined, and then an image acquisition interval is set according to the flight speed of the unmanned aerial vehicle, so that the acquired image has an overlapping area with an adjacent image in a course or a lateral direction, namely the acquired image can cover the target area.

When the unmanned aerial vehicle collects each inclined image, the POS information of the unmanned aerial vehicle corresponds to, namely the position and posture information when the unmanned aerial vehicle shoots the image, and the approximate overlapping area of the course and the lateral direction of the image can be calculated by utilizing the values; after the overlapping relationship of the images is obtained, an approximate relationship between all the images can be established according to the length and the width of the images, namely the approximate position of the point on the first image on the point of the same name on the second image.

S102: and performing feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship, and determining the corresponding relationship between image points on adjacent images.

In this embodiment, after establishing the approximate relationship between all the images, feature point matching is performed on two oblique images adjacent to each other (where there is an overlapping area) in the course or the lateral direction, so as to determine the corresponding relationship between the image points on the oblique images of the adjacent unmanned aerial vehicles, for example, which point on the first image corresponds to which point on the second image corresponds. The correspondence between the pixels on the oblique images of the adjacent drones determined here is the correspondence between the pixels in the overlapping area on the oblique images of the adjacent drones.

Specifically, for two oblique images of the unmanned aerial vehicle having an overlapping area in the course or the lateral direction, taking one of the two oblique images as a reference image and the other as a search image, and determining an affine transformation relationship between the reference image and the search image according to feature points corresponding to image points of the same name in the overlapping area of the reference image and the search image, specifically, the method includes:

establishing an image pyramid, extracting feature points from a reference image, primarily matching SIFT features on the top layer of the image pyramid, establishing an initial affine transformation relation between the two, transforming a feature point window to a search image by taking the initial affine transformation relation as a geometric constraint condition, sampling and correcting the search image window to an image space coordinate system of the reference image, matching correlation coefficients, then eliminating gross errors by adopting polynomial iteration in a local range, and resolving and updating the affine transformation relation between the two by using the obtained image points with the same name again; and then, carrying out lower-layer image matching until the lower-layer image is obtained, finally, converting the matching result into an image space coordinate system of the search image, carrying out least square matching, and determining the affine transformation relation between the reference image and the search image. The method comprises the steps of firstly determining an essential matrix between a reference image and a search image, then carrying out singular value decomposition on the essential matrix, determining a motion parameter rotation matrix and a translation vector of a camera, and determining three-dimensional coordinates of feature points corresponding to image points with the same name according to the motion parameter rotation matrix and the translation vector of the camera.

Specifically, an image pyramid is established for each unmanned aerial vehicle image in advance, the topmost layer of the pyramid is not less than 512 x 512, then for the image pyramid of each unmanned aerial vehicle image, feature points are extracted at the bottommost layer of the pyramid by using a Forstner operator, the distribution of the feature points is as uniform as possible, logic partitioning is performed, it is ensured that each block has approximately the same number of feature points, and the feature points and corresponding logic partitions are mapped to other layers of the image pyramid.

And performing image matching by using a SURF operator at the topmost layer of the image pyramid, and determining two images with an overlapping area, namely two images with corresponding feature points, wherein one image is used as a reference image, and the other image is used as a search image, and the two images can be unmanned aerial vehicle images with the overlapping area in the unmanned aerial vehicle course direction or unmanned aerial vehicle images with the overlapping area in the unmanned aerial vehicle lateral direction.

And calculating initial affine transformation parameters between the reference image and the search image according to the corresponding feature points in the overlapping area of the reference image and the search image. Dividing the image corresponding to the overlapping area in the reference image into a plurality of reference windows, calculating a matching window corresponding to the reference window on the searched image by using a formula (1) according to the initial affine transformation parameter at the topmost layer of the image pyramid, and performing image resampling on the matching window by using the formula (1).

Wherein, a₀、a₁、a₂、b₀、b₁、b₂For affine transformation parameters, X, Y is the coordinates of the feature points on the image, and x and y are the coordinates of the pixel points obtained after resampling the image.

At the topmost layer of the image pyramid, extracting and mapping the obtained feature points (for example, m in total) according to a Fonstner operator, performing image matching calculation on conjugate image points of the feature points in a resampled search window by using a correlation coefficient method, reserving conjugate points (for example, k in total, k < m) with the maximum correlation coefficients and larger than a threshold, eliminating mismatching points by using a Ranpac thought, and reserving the conjugate points (for example, l in total, l < k). And then updating affine transformation parameters of the top-level reference window and the matching window by using the matching result (l conjugate points), calculating quasi-conjugate points of other feature points (m-l) of the top-level reference window by using the set of affine transformation parameters, and keeping the quasi-conjugate points as well as the l conjugate points successfully matched.

And (3) conducting the matching result (m conjugate points) to the next layer of the image pyramid of the search image, calculating a matching window corresponding to the reference window on the layer by using a formula (1) according to the updated affine transformation parameters, and performing image resampling on the matching window by using the formula (1). According to the characteristic points (such as nm in total) extracted and mapped by the Fonstner operator, calculating conjugate image points of the characteristic points in a search window after resampling by using a correlation coefficient method, reserving conjugate points (such as nk points, nk < m) with the maximum correlation coefficient and larger than a threshold value, eliminating mismatching points by using a Randac idea, and reserving the conjugate points (such as nl in total, nl < nk). And then updating affine transformation parameters of the top-layer reference window and the matching window by using the matching result (nl conjugate points), calculating quasi-conjugate points of other feature points (nm-nl) of the top-layer reference window by using the group of affine transformation parameters, and keeping the quasi-conjugate points as the nl conjugate points successfully matched. Wherein n is the magnification of the layer of image relative to the previous layer of the image pyramid.

The above process is repeated until the bottom layer of the image pyramid is conducted. And obtaining a reliable conjugate point of the reference window at the bottom layer of the image pyramid, and performing least square matching on the basis to improve the accuracy of image matching. And finally, calculating corresponding image points with the same name on the searched image by using a formula (1).

In this embodiment, a plurality of reference windows may be synchronously matched to determine corresponding image points on the search image corresponding to the reference image.

S103: and calculating the orientation parameters of the oblique images of the unmanned aerial vehicle according to the position and attitude information and the corresponding relation between the image points.

In this embodiment, the three-dimensional coordinates of the feature points corresponding to the image points of the same name can also be corrected by using the block adjustment technique, so as to generate more accurate three-dimensional coordinates. Determining an imaging model of the unmanned aerial vehicle oblique image according to the position and posture information of the unmanned aerial vehicle oblique image; linearizing the imaging model according to a Taylor series to generate a linearized model; taking the linearized model as a basic model of adjustment, and establishing an error equation set by using coordinates of feature points on the unmanned aerial vehicle oblique image; establishing a normal equation set according to the error equation set, determining the correction number of the external orientation element corresponding to the position and attitude information by solving the normal equation set, correcting the external orientation element corresponding to the position and attitude information according to the correction number, and further determining the ground coordinate corresponding to the feature point on the unmanned aerial vehicle oblique image.

Specifically, POS parameters acquired by an unmanned aerial vehicle are used as initial values of external orientation elements of the image, the initial values of the external orientation elements are used for calculating to obtain rotation matrixes corresponding to all image points, transformation coordinates and image point coordinates of the image points are calculated, an error equation set is established according to a block error model, a normal equation set is established according to the error equation set, the normal equation set is solved to obtain correction numbers of the external orientation elements, approximate values of the external orientation elements in the previous time are corrected according to different external orientation element models, coordinate correction numbers of all encrypted points are calculated by using a multi-piece forward intersection formula, and the approximate values of the coordinates of the encrypted points after correction are calculated. And repeating the process until the correction numbers of the external orientation elements of all the images and the correction numbers of the coordinates of all the image points are smaller than the given limit difference. And after the ground coordinates of the overlapped characteristic points are obtained, calculating the ground coordinates of the points in the non-overlapped area in the unmanned aerial vehicle oblique image by utilizing the intersection of the plurality of pieces of front parts.

The above process is described below as a specific example, where the orientation parameters refer to the position and attitude parameters of the camera and the distortion parameters.

Suppose the exterior orientation element of the ith photograph is X_Si、Y_Si、Z_Si、ω_i、κ_iThen the imaging model is:

the above formula is linearized in accordance with a taylor series to obtain:

V＝CΔ+BX-L

X＝(dX dY dZ)^T

wherein the content of the first and second substances,

l_x、l_yis a constant term:

obtaining a linearization model:

and taking the linearized model as a basic model of adjustment, and establishing an error equation by utilizing coordinates of various observation value points.

Calculating the coordinate correction numbers of all the characteristic points by using a multi-sheet forward intersection formula, and calculating the approximate values of the coordinates of the corrected characteristic points, wherein the formula is as follows:

s104: and acquiring dense point cloud by using a depth map method, performing curved surface reconstruction by using a Poisson curved surface reconstruction algorithm according to the acquired dense point cloud, acquiring a three-dimensional grid of a scene corresponding to the unmanned aerial vehicle oblique image, and performing texture mapping on the three-dimensional grid.

For a specific implementation process of obtaining the dense point cloud by using the depth map method in this step, reference is made to embodiment two, and details are not described in this embodiment again. After dense point cloud is obtained, the curved surface reconstruction of the dense point cloud is converted into solving a Poisson equation, a gradient field and a vector field are calculated by constructing the Poisson equation, an equivalent value meeting a preset condition is selected to obtain a reconstructed curved surface which best approximates original point cloud data, the square of volume change is used as an edge folding grid simplification algorithm of error measurement, and a triangle normal constraint factor is added into the error measurement to simplify the reconstructed curved surface.

S105: and performing orthographic projection on the three-dimensional grid after texture mapping to generate a true orthographic image.

And after the three-dimensional grid after texture mapping is generated, performing orthographic projection on the three-dimensional grid after texture mapping to generate a true orthographic image.

The real projective image generating method based on the unmanned aerial vehicle inclined image can acquire the real projective image of the ground feature information according to the unmanned aerial vehicle inclined image of the sheltered ground feature information, thereby overcoming the technical problem that the orthographic image of the ground feature information lacks when the sheltered object exists, and generating the more complete orthographic image of the ground feature information.

Fig. 2 is a flowchart of texture mapping of a method for generating a true orthoimage based on an oblique image of an unmanned aerial vehicle according to a second embodiment of the present disclosure. The embodiment describes a specific implementation process for acquiring a dense point cloud by using a depth map method, and specifically includes the following steps:

s201: and selecting an inclined image pair according to a preset condition.

For each unmanned aerial vehicle oblique image, a reference image needs to be selected for stereo calculation, and the selection of a stereo pair not only influences the stereo matching precision, but also is very important for the final reconstruction result. A good reference image not only has a similar viewing angle direction as the target image, but also has a proper baseline.

Supposing that n images are provided, for the ith target image, calculating the included angle theta between the ith target image and the main optical axis of the jth image_ij(j 1.. n), then < θ for 5 ° < θ_ijCalculating the distance d between the image with angle less than 60 DEG and the target image i_ijBy using d_ijCalculating the average distanceRemoval distanceOrThe image of (a). If the number of the neighborhood images meeting the requirement is less than k1(k1 is 10), the images are all used as reference images of the ith image, otherwise d is calculated_ij*θ_ijThe first k1 images with the largest value are selected as the reference images of the ith image to form a stereo pair.

S202: and determining corresponding ground points according to the image points with the same name in the oblique image pair.

Initialization is performed by using a random algorithm: for each pixel in the input image, it is desirable to find a plane f that minimizes the matching error between the pixel and its corresponding pixel in the reference image.

Essentially, the plane f is a section of the surface of the three-dimensional scene, and can be represented by three-dimensional coordinates X in the camera coordinate system_iAnd its normal n_iRepresenting that for each pixel point p on the input image, the corresponding three-dimensional plane, C, is estimated_iAnd C_jThe camera centers of the target image and the corresponding reference image, respectively. Assuming that a plane is used in the camera C_iA three-dimensional coordinate X in the coordinate system_iAnd its normal n_iIs represented by C_iIs i input images, C_i-xyz is the camera coordinate system. For the ith input image I in the data set_iAnd its reference image I_jThe camera parameters are respectively { K_i,C_i,R_i}，{K_j,C_j,R_j}. Firstly, randomly giving an image I_iThe upper pixel point P is a three-dimensional plane f. Three-dimensional coordinate X_iOne projection depth lambda (lambda) must be randomly selected on the projection ray of P_min＜λ＜λ_max)，X_iAt C_iThe coordinates in the coordinate system are:

X_i＝λK_iP

p is the homogeneous coordinate of the pixel point. Then the normal n of a random given plane f_iThe calculation formula is as follows:

wherein θ isA random angle within the range of one angle,is composed ofAn angle within the range, the setting of which is based on a simple assumption that when a patch is normal to the patch and the camera C_iWhen the included angle of the Z axis in the coordinate system is smaller than a threshold (assuming that the threshold is 60), it is in the image I_iIs visible.

The above random initialization procedure is easily achieved by threeAt least one good assumed plane of each plane in the dimensional scene, because as the resolution of the image is improved, more and more pixel points are contained in each scene plane, more and more pixel points can be utilized. Image I obtained by the above method_iDepth map of (I)_iThe depth of each pixel point can be mapped to its reference image I_jAs above, as the initialized depth of the corresponding pixel on the reference image, the initial value is still given by using a random method for the pixel without the corresponding point. I can be given better by this method_jEach mapped pixel is better assumed to be planar because of the introduction of the stereopair I_iAnd I_jPlane consistency constraints.

By using the homography matrix between the estimated plane f of each pixel and the image, the ground point corresponding to each image point and the homonymous matching point on the reference image can be calculated. In a window of 7 x 7 with the pixel point P as the center, homonymous image points of each pixel in the window on a reference image are calculated by utilizing a homography matrix, and then the matching cost is calculated by utilizing a normalized cross-correlation algorithm, namely the normalized cross-correlation matching algorithm.

S203: and deleting the wrong depth map points by using the neighborhood depth map to obtain an accurate depth map corresponding to each unmanned aerial vehicle oblique image.

After initialization, image I_iEach pixel in (a) is associated with a three-dimensional plane. Then, we process I one by one_iAnd (5) iterating for 2 times to optimize the plane. In an iteration, we start to propagate one by one from the top left corner of the image to the bottom right corner and then from the bottom right corner to the top left corner. There are two operations per pixel in each iteration, called spatial propagation and random assignment. And the space propagation is used for comparing and propagating adjacent pixels to the three-dimensional plane of the current pixel, if fp is the plane of the neighborhood of the current pixel and f is the plane of the current pixel, matching cost is respectively calculated by using fp and f, and if the fp matching cost is better than f, the plane corresponding to the current pixel is updated to fp. This spatial propagation process relies on neighboring pixels being likely to have similar three-dimensional spatial planes, especially for high resolution images. Theoretically, even one is goodThe guess of (c) is sufficient to propagate this plane to other pixels of the matching window after the first and second spatial propagation. After spatial propagation, we further reduce the matching cost by using random distribution, and randomly selecting the projection depth λ and the normal angle θ of each pixel in a given rangeAnd calculating matching cost, and performing random distribution if the matching cost is superior to the last iteration result, so as to gradually reduce the random range of the depth and the normal line and finally obtain the optimal depth and the normal line.

After spatial propagation and random assignment, we remove unreliable points in the depth map, i.e., matching points whose matching cost is above some threshold.

Due to the presence of depth errors, the depth maps computed by the above process may not be exactly consistent in common areas, for which the image I is referred to_iIs back projected into three-dimensional space using the camera parameters and depth λ, the formula is as follows:

X＝λR^TK^-1P+C_i

wherein P is a homogeneous coordinate, and X is a three-dimensional coordinate under a world coordinate system. Projecting X onto its neighborhood image, assuming N (I) is selected I in the stereopair selection step_iThe neighborhood image of (2). Suppose N_kIs the K-th image in N (i), d (X, N)_k) Is a camera N_kAt the depth of X point, λ (X, N)_k) Is to calculate X in N_kOf the projected pixel of (2) at N_kThe depth of this pixel is obtained on the depth map of (a). If λ (X, N)_k) And d (X, N)_k) Close enough, consider X to be in I_iAnd N_kThe above is consistent. If X is consistent at least in K neighborhood images, X is considered to be a reliable scene point, its corresponding pixel in the depth map is preserved, otherwise the point is removed.

Most error points are removed through the optimization, and each image obtains a relatively clean and correct depth map.

S204: and fusing the accurate depth map to generate dense point cloud.

According to the method, error points in the unmanned aerial vehicle oblique images can be effectively removed, a relatively clean and correct depth map of each unmanned aerial vehicle oblique image is obtained, and therefore more complete orthographic images of ground feature information can be generated.

It is noted that while for simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present disclosure is not limited by the order of acts, as some steps may, in accordance with the present disclosure, occur in other orders and concurrently. Further, those skilled in the art should also appreciate that the embodiments described in the specification are exemplary embodiments and that acts and modules referred to are not necessarily required by the disclosure.

The above is a description of embodiments of the method, and the embodiments of the apparatus are further described below.

As shown in fig. 3, a functional structure diagram of a real radiographic image generating device based on oblique images of an unmanned aerial vehicle according to a third embodiment of the present disclosure is shown. The real projective image generating device based on unmanned aerial vehicle oblique image of this embodiment includes:

the relative position relation determining module 301 is configured to obtain position and posture information corresponding to the oblique images of the unmanned aerial vehicle, and determine a relative position relation between the oblique images of the unmanned aerial vehicle according to the position and posture information.

And the feature point matching module 302 is configured to perform feature point matching on the unmanned aerial vehicle oblique images according to the relative position relationship, and determine a corresponding relationship between image points on adjacent unmanned aerial vehicle oblique images.

And an orientation parameter determining module 303, configured to calculate an orientation parameter of the oblique image according to the position and posture information and the correspondence between the image points.

The curved surface reconstruction module 304 is configured to acquire dense point clouds by using a depth map method, perform curved surface reconstruction by using a poisson curved surface reconstruction algorithm according to the acquired dense point clouds, acquire a three-dimensional grid of a scene corresponding to an oblique image of the unmanned aerial vehicle, and perform texture mapping on the three-dimensional grid.

The true ortho image generating module 305 is configured to perform ortho projection on the three-dimensional mesh after texture mapping to generate a true ortho image.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the described module may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

FIG. 4 shows a schematic block diagram of an electronic device 400 that may be used to implement embodiments of the present disclosure. As shown, device 400 includes a Central Processing Unit (CPU)401 that may perform various appropriate actions and processes in accordance with computer program instructions stored in a Read Only Memory (ROM)402 or loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

A number of components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, or the like; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408 such as a magnetic disk, optical disk, or the like; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Processing unit 401 performs the various methods and processes described above, and is tangibly embodied in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When the computer program is loaded into the RAM 703 and executed by the CPU 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the CPU 401 may be configured to perform the above-described method in any other suitable manner (e.g., by way of firmware).

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a load programmable logic device (CPLD), and the like.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Further, while operations are depicted in a particular order, this should be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.