阅读说明：本技术 一种基于计算机视觉技术的前后端架构地图定位方法 (A kind of front and back end framework Orientation on map method based on computer vision technique ) 是由 施荣圳 廖勇 吴金添 舒凌洲 于 2019-08-28 设计创作，主要内容包括：一种基于计算机视觉技术的前后端架构地图定位方法,包括定位、传输和建图三个部分,所述定位部分运行于基于嵌入式设备的前端,所述传输和建图部分运行于PC后端,本发明通过使用后端提前对后续需要进行定位的地图区域进行离线的全局地图的构建,再对提前构建好的全局地图进行信息读取和预处理,然后使用前端对待定位目标所处位置进行拍照,将拍照所得图形进行语义分割、特征点提取及描述子计算、视觉单词归类处理,在对每个特征点进行视觉单词归类后,利用视觉单词类别,将前端当前图像的特征点与后端传送来的子地图的特征点进行全局地图匹配,从而估计出当前相机在全局地图中的姿态信息,从而实现对无人机或无人车的地图实时定位。(A kind of front and back end framework Orientation on map method based on computer vision technique, including positioning, transmit and build three parts of figure, the position portion runs on the front end based on embedded device, it the transmission and builds figure part and runs on the rear end PC, by the present invention in that carrying out the building of offline global map to the subsequent map area positioned in advance with rear end, information reading and pretreatment are carried out to the global map built in advance again, then it is taken pictures using front end to target present position to be positioned, the gained figure that will take pictures carries out semantic segmentation, feature point extraction and description calculate, vision word classification processing, after carrying out vision word classification to each characteristic point, utilize vision word classification, the characteristic point for the sub- map that the characteristic point of front end present image and rear end transmission are come is subjected to global map matching, to estimate Posture information of the Current camera in global map is counted out, the map of unmanned plane or unmanned vehicle is positioned in real time to realize.)

1. a kind of front and back end framework based on computer vision technique, which is characterized in that including being based on embedded device and monocular The front end of camera and rear end based on PC and binocular camera, the front end run map location module, and the rear end is operably Figure transmission module and map structuring module；

The map structuring module, for constructing global map offline and being sent to map transmission module；

The map transmission module, total track for reading global map, will run global map by target to be positioned It is divided into sub- map corresponding with sub-trajectory one by one and sub- map is sent to front end；

The map location module, for obtaining the target to be positioned monocular image of real-time local environment, globally at runtime Figure matches and realizes the tuning on-line of target to be positioned by received sub- map.

2. a kind of Orientation on map method based on front and back end framework, which comprises the following steps:

Step S1. starts map structuring module in rear end, carries out globally to the subsequent map area positioned in advance The offline building of figure determines the global information comprising three-dimensional feature point cloud and target real trace information；The three-dimensional feature point Cloud is made of a large amount of three-dimensional feature points；

It is online to start map transmission module after the completion of the building of step S2. global map, read existing global map and will be entire Global map is cut into the sub- map that one piece of block is convenient for parallel transmission；Before map location module operation, map transmission module is pre- The former pieces of sub- maps relative to positioning end are first first transmitted to map location module, to start to be positioned；

Step S3. obtains list in frontal startup monocular camera and map location module, monocular camera frame by frame in a manner of video flowing Mesh image is simultaneously input in map location module, successively carries out semantic segmentation, feature by image of the map location module to input Point extracts and description calculates, vision word is sorted out, Feature Points Matching, estimates posture of the Current camera in global map and believes Breath obtains the location information of the embedded device currently to run locator routine, to complete the tuning on-line of map.

3. a kind of Orientation on map method based on front and back end framework according to claim 2, which is characterized in that the step Semantic segmentation in S3 specifically refers to: first distinguishing the characteristic of dynamic object and stationary body, then extracts dynamic object in image Region, and the region is rejected.

4. a kind of Orientation on map method based on front and back end framework according to claim 2, which is characterized in that the step Specific step is as follows for vision word classification in S3:

Step S3.1.1. trains a visual dictionary by the characteristic point in incoherent image using FLANN；

Step S3.1.2. is based on visual dictionary by three in the two dimensional character point and back-end image in the monocular image of front end Dimensional feature point carries out vision word classification；

When the vision word number of step S3.1.3. two dimensional character point described in the step S3.1.2 and three-dimensional feature point is identical, Assert that the two characteristic points have similitude, a vision word matching pair can be become；

Step S3.1.4. when obtained in primary positioning a large amount of vision word matching to after, carry out subsequent Attitude estimation.

5. a kind of Orientation on map method based on front and back end framework according to claim 4, which is characterized in that the step Feature Points Matching specifically refers in S3, the characteristic point for the sub- map that the characteristic point of the current monocular image in front end and rear end transmission are come Carry out global map matching；Specifically includes the following steps:

Step S3.2.1. matches to sub- matching operation, the i.e. two dimension when a front end is further described vision word When the vision word number of the three-dimensional feature of characteristic point and rear end point is identical, then to description of the two characteristic points carry out away from From calculating；

If the distance of description of two characteristic points is really assert in threshold range in step S3.2.2. step S3.2.1 The two characteristic points are mutually matched；

Step S3.2.3. repeats step S3.2.2 and obtains the characteristic point that multiple groups are mutually matched, and is then mutually matched again by multiple groups Feature Points Matching to come solve the problems, such as perspective multiple spot, thus estimate camera posture information；

Step S3.2.4. using calculate between the camera posture currently estimated and sub- map the last one camera posture in XZ Distance on axis, the i.e. distance of the movement of camera are switched to down a piece of new sub- map if being less than threshold value.

6. a kind of Orientation on map method based on front and back end framework according to claim 2, which is characterized in that the three-dimensional The information that characteristic point includes has D coordinates value, vision word identifier, camera identifier and feature point description sub-information, wherein The camera identifier indicates that information is character pair point is that camera is captured under which frame.

7. a kind of Orientation on map method based on front and back end framework according to claim 2, which is characterized in that the step S2 neutron map can be positioned by the map location module that wireless transmission is sent to front end parallel and continuously；Institute State step S2 specifically includes the following steps:

Step S2.1. is read out the information of global map；

Entire global map is carried out map segmentation by step S2.2., is cut into Pork-pieces sub- map；

The map segmentation refers to: the track that positioning end will be run first is determined, then by the entirety track in global map It is divided into multistage sub-trajectory；It includes partial 3-D feature that one piece of block will be divided into comprising the global map of whole three-dimensional feature points The sub- map of point is realized；

Before step S2.3. is positioned in rear end, in advance by former pieces of sub- map transmissions to map location module, so as to start into Row positioning.

8. a kind of front and back end framework Orientation on map method based on computer vision technique according to claim 2, special Sign is that the three-dimensional feature point cloud is compressed by the method for Octree and reduces redundancy；

The method of the Octree refers to: the side length value of a square is set first, by the way that square three-dimensional space is continuous Eight pieces of small squares are divided into, until each piece small square side length of division is all the side length value for the square being previously set Position, each small square space can include several characteristic points at this time, and the point for selecting one of them most feature represents Other three-dimensional feature points in the block three-dimensional space, other three-dimensional feature represented points directly abandon.

9. a kind of front and back end framework Orientation on map method based on computer vision technique according to claim 2, special Sign is, the step S1 comprising the following specific steps

Step S1.1. photography: video flowing is carried out to the subsequent map area positioned using high-precision binocular camera The shooting of formula, obtains binocular image；

Step S1.2. pretreatment: after binocular image and some incidental informations are input to map location module, by pre-processing just Depth map, semantic segmentation figure and camera track can be obtained；

Step S1.3. extracts characteristic point: after pretreatment, the characteristic point in image being extracted and described the meter of son It calculates, and carries out the classification of vision word to each characteristic point；

Step S1.4. generates global map: generating three-dimensional feature point in conjunction with the camera track obtained and camera inside and outside parameter Cloud, i.e., final global map.

10. a kind of front and back end framework Orientation on map method based on computer vision technique as claimed in claim 2, feature It is, the front end needs to complete by the global map of rear end during being positioned, and rear end is then wireless by TCP Sub- map after global map is cut in communication is successively transferred to front end for positioning use；Positioning will be generated in the position fixing process of front end Posture information, the positioning posture information, that is, Current camera posture information, these positioning posture information will also pass through TCP channel radio Letter is transmitted to rear end and uses for monitoring；The resident state for waiting connection is set by back-end network, front end is set as needing to connect Acceptor map and the state that ability rear end TCP server initiates the connection when needing to transmit positioning posture information, to complete data Wireless transmission.

Technical field

The invention belongs to technical field of computer vision, it particularly relates to it is a kind of based on computer vision technique before Rear end framework Orientation on map method.

Background technique

In computer vision technical field, mainly positioned immediately using SLAM(and map structuring) technology is come real Existing autonomous positioning.SLAM technology can be realized the local positioning of high accuracy and real-time on power PC, mainstream Steps are as follows for major technique:

1) image obtains: image is mainly obtained by camera apparatus, and camera apparatus can be divided into monocular camera, binocular camera and depth Spend camera.If monocular camera, then the image got is cromogram, and the image that binocular camera and depth camera are got is then For cromogram and depth map.

2) feature point extraction and description calculate: there is the characteristic point for largely having substantive characteristics, the steps in image These characteristic points can be extracted from image, and calculate its description to describe its characteristic information in a manner of data.

3) Attitude estimation: the characteristic point and its description sub-information in the two field pictures of front and back are utilized, front and back two can be estimated The movement relation of frame image, to obtain the camera posture information of two frame of front and back.When camera motion is after stretch diameter, Obtain the location information relative to camera motion starting point.

4) map is established: while positioning, three-dimensional point cloud map can also be set up with local positioning information.

5) pose refinement: the accuracy in order to improve positioning, it generally can be using information such as winding come to acquired in step 3 Posture optimize and update.

Since the SLAM technology of mainstream mainly carries out Attitude estimation using the relationship between two field pictures before and after camera, because This can have following technical problem:

1) positioning track obtained after Attitude estimation is usually the local positioning for being directed to initial position, can not be realized well Positioning in global sense；

2) since there are errors for the Attitude estimation that is obtained by before and after frames image, with the propulsion of time, the accumulation of positioning track Error will be increasing, causes final location information accuracy not high enough；

3) though majority SLAM technical solution can reach good effect on power PC, cannot be simultaneous on embedded device Care for real-time and accuracy；

4) even if map self-capacity is also excessively huge to be unfavorable in embedded device in the case where being provided with map auxiliary It stores, and usually will receive the interference of dynamic object in real scene.

Summary of the invention

The present invention is directed to and carries out positioning existing defect to unmanned plane or unmanned vehicle using SLAM technology in the prior art, mentions A kind of front and back end framework Orientation on map method based on computer vision technique, base while taking into account real-time and accuracy are gone out The positioning of the global full track mark of unmanned plane or unmanned vehicle is realized in embedded device, and reduces before and after frames image posture mistake Difference.

The present invention is achieved through the following technical solutions:

A kind of front and back end framework Orientation on map method based on computer vision technique, including position and build two parts of figure, institute It states position portion and runs on the front end based on embedded device, front end is mounted on target to be positioned, such as unmanned plane or unmanned vehicle On, the figure part of building runs on the rear end based on PC, and the present invention is based on the front and rear ends to follow the steps below:

Step S1. carries out the building of offline global map using rear end to the subsequent map area positioned in advance, The global map includes by the real trace information of a large amount of three-dimensional feature point the three-dimensional feature points formed and target to be positioned；

Step S2. carries out information reading and pretreatment to the global map that builds in advance using rear end, the pretreatment be by Global map is cut into one piece of block map；

Step S3. takes pictures to target present position to be positioned using front end, and the gained figure that will take pictures carries out semantic segmentation, spy Sign point extracts and description calculates, vision word classification processing utilizes view after carrying out vision word classification to each characteristic point Feel token-category, the characteristic point for the sub- map that the characteristic point of front end present image and rear end transmission are come is subjected to global map Match, so that posture information of the Current camera in global map is estimated, to complete Orientation on map.

In order to which the present invention is better achieved, further, the semantic segmentation in the step S3 can distinguish dynamic object and The characteristic of stationary body extracts the region of dynamic object in image, and the region is rejected, so that front end is after matching When holding map, interference brought by dynamic object in scene can be removed.

In order to which the present invention is better achieved, further, specific step is as follows for the vision word classification in the step S3:

Step S3.1.1. trains a visual dictionary by the characteristic point in incoherent image using FLANN；

Step S3.1.2. is based on visual dictionary by the three-dimensional feature in the two dimensional character point and back-end image in front-end image Point carries out vision word classification；

Step S3.1.3. assert that the two characteristic points have similitude when the vision word number of two characteristic points is identical, can be at For a matching pair；

Step S3.1.4. when obtain largely matching in primary positioning to after, carry out subsequent Attitude estimation.

In order to which the present invention is better achieved, further specific step is as follows for the global map matching in the step S3:

Step S3.2.1. matches to sub- matching operation is further described vision word, i.e., when a front end two dimension is special When sign point is identical with the vision word number of a rear end three-dimensional feature point, then distance is carried out to description of the two characteristic points It calculates；

If step S3.2.2. distance in threshold range, really assert that the two characteristic points are mutually matched；

Step S3.2.3. repeats step S3.2.2 and obtains the characteristic point that multiple groups are mutually matched, and then passes through multiple groups characteristic point again It matches to solve the problems, such as perspective multiple spot, thus estimates camera posture information；

Step S3.2.4. using calculate between the camera posture currently estimated and sub- map the last one camera posture in XZ Distance on axis, the i.e. distance of the movement of camera are switched to down a piece of new sub- map if being less than threshold value.

In order to which the present invention is better achieved, further, the step S2 is related to that specific step is as follows:

Step S2.1. is read out the information of global map；

Entire global map is carried out map segmentation by step S2.2., is cut into Pork-pieces sub- map, and the map segmentation passes through It determines the track that positioning end will be run, the entirety track is divided into multistage sub-trajectory in global map, i.e., it will be comprising complete It includes the sub- map of partial 3-D characteristic point to realize that the global map of portion's three-dimensional feature point, which is divided into one piece of block,；

Step S2.3 is before map location module operation, in advance by former pieces of sub- map transmissions to map location module, to open Beginning is positioned.

In order to which the present invention is better achieved, further, the three-dimensional feature point cloud is mainly by a large amount of three-dimensional feature points institute It constitutes, the information that the three-dimensional feature point includes has D coordinates value, vision word identifier, camera identifier and characteristic point to retouch Sub-information is stated, wherein the camera identifier indicates that information is character pair point is that camera is captured under which frame.

In order to which the present invention is better achieved, further, the three-dimensional feature point cloud remains all three-dimensional feature points Capacity is more huge, is compressed by the method for Octree to three-dimensional point cloud and reduces redundancy.

In order to which the present invention is better achieved, further, the step S1 is related to that specific step is as follows:

Step S1.1. photography: video flowing is carried out to the subsequent map area positioned using high-precision binocular camera The shooting of formula, obtains binocular image；

Step S1.2. pretreatment: after binocular image and some incidental informations are input to map location module, by pre-processing just Depth map, semantic segmentation figure and camera track can be obtained；

Step S1.3. extracts characteristic point: after pretreatment, the characteristic point in image being extracted and described the meter of son It calculates, and carries out the classification of vision word to each characteristic point；

Step S1.4. generates global map: generating three-dimensional feature point in conjunction with the camera track obtained and camera inside and outside parameter Cloud, i.e., final global map.

In order to which the present invention is better achieved, further, the feature point extraction and description calculate and have used ORB algorithm Sub-information is described to extract to the characteristic point in image and calculate it.Characteristic point is partially to have certain feature in image Point, such as edge or edge intersection, and characteristic a little can then be showed in the form of data by describing son.When obtaining spy After sign point, the distance that son is described between characteristic point can be calculated, to realize matching process.

In order to which the present invention is better achieved, further, the current transmission module is mainly used for the number of front-end and back-end According to wireless communication.Specifically, need to complete by the global map of rear end in the position fixing process of front end, and rear end then by TCP wireless communication, the sub- map after global map is cut successively are transferred to front end and use for positioning；In addition to this, front end is fixed Positioning posture information, i.e. Current camera posture information will be generated during position, these positioning posture information also will be wireless by TCP Communication is transmitted to rear end and uses for monitoring.This module definition rear end is TCP server, and front end is TCP Client, i.e. back-end network Network belongs to the resident state for waiting connection, and front end is when needing to receive sub- map and needing to transmit positioning posture information, then to Rear end TCP server initiates the connection, to complete data wireless transmission.

Compared with prior art, the present invention have the following advantages that and the utility model has the advantages that

1) after cutting to global map, sub- map can be supplied to front end by wireless transmission parallel and continuously It is positioned.The sub- map of the low capacity formed after this cutting, the memory that can not only save front end embedded device are used, are allowed Embedded device is without global map information huge under disposable storage, moreover it is possible to reduce front end in two dimension is matched with three-dimensional Three-dimensional feature point matching amount, this had both improved matching speed and has also improved matching accuracy.

2) after carrying out semantic segmentation to image, the dynamic object and stationary body in scene be can recognize that.As Foundation rejects the characteristic point of the dynamic object in scene, can be effectively reduced in map match position fixing process, dynamic The interference of object bring.

3) since the global map needed for positioning is built offline, and global map can be carried out in standard in advance Artificial optimization in true property.Therefore, front end can be positioned according to more accurate map provided by rear end, fundamentally It obviates and moves forward and backward relationship using camera to carry out the scheme of Attitude estimation, to solve error accumulation in script scheme Problem.

4) traditional SLAM technology has been split into and has built figure and tuning on-line two parts offline, made it possible to build figure in early period During, the global information of map is decided in advance.It, can be according to the overall situation built when running front end tuning on-line Map is positioned, to know the specific location of embedded device currently to run locator routine, and then solves tradition In SLAM technical solution, problem that the position of embedded device can only be determined relative to positioning initial position.

5) realize that the high map structuring of accuracy needs more computing resource, the present invention gives this part to high-performance PC carries out processed offline, so that the calculation amount sharp fall of the embedded device in front end, to realize preferably real-time Property.And because rear end is capable of providing the global map of high accuracy, front end according to the global map when being positioned, and energy Enough reach better positioning accuracy.

Detailed description of the invention

Fig. 1 is overall system architecture figure；

Fig. 2 is to divide sub- map cut zone flow chart；

Fig. 3 is semantic segmentation flow chart；

Fig. 4 is that vision word sorts out flow chart；

Fig. 5 is the flow chart of sub- map of the building comprising three-dimensional point；

Fig. 6 is the flow chart of server in TCP communication；

Fig. 7 is client flow chart in TCP communication；

Fig. 8 is the flow chart of TCP communication neutron map parallel transmission.

Specific embodiment

In order to illustrate the technical solution of the embodiments of the present invention more clearly, below in conjunction with attached in the embodiment of the present invention Figure, technical scheme in the embodiment of the invention is clearly and completely described, it should be understood that described embodiment is only It is a part of the embodiments of the present invention, instead of all the embodiments, therefore is not construed as the restriction to protection scope.Base Embodiment in the present invention, ordinary skill staff institute obtained without making creative work There are other embodiments, shall fall within the protection scope of the present invention.