阅读说明：本技术 6DoF定位追踪装置、方法及电子设备 (6DoF positioning and tracking device and method and electronic equipment ) 是由 吴涛 于 2021-06-02 设计创作，主要内容包括：本发明提供一种6DoF定位追踪装置、方法及电子设备,其中的装置包括至少三个追踪相机、惯性导航单元以及计算处理单元；其中,追踪相机之间呈预设角度分布,追踪相机用于获取当前环境下的图像数据,并传送至计算处理单元；惯性导航单元用于获取当前环境下6DoF定位追踪装置的状态信息,并传送至计算处理单元；计算处理单元用于根据图像数据和状态信息,确定6DoF定位追踪装置相对当前环境的6DoF数据。利用上述发明能够降低追踪装置对物理环境的使用限制,提高与仿真物体的适配度。(The invention provides a 6DoF positioning and tracking device, a method and electronic equipment, wherein the device comprises at least three tracking cameras, an inertial navigation unit and a calculation processing unit; the tracking cameras are distributed in a preset angle mode and used for acquiring image data in the current environment and transmitting the image data to the computing and processing unit; the inertial navigation unit is used for acquiring the state information of the 6DoF positioning and tracking device in the current environment and transmitting the state information to the calculation processing unit; and the calculation processing unit is used for determining 6DoF data of the 6DoF positioning and tracking device relative to the current environment according to the image data and the state information. The invention can reduce the use limit of the tracking device to the physical environment and improve the adaptation degree with the simulation object.)

1. A6 DoF positioning and tracking device is characterized by comprising at least three tracking cameras, an inertial navigation unit and a calculation processing unit; wherein the content of the first and second substances,

the tracking cameras are distributed in a preset angle and used for acquiring image data in the current environment and transmitting the image data to the computing processing unit;

the inertial navigation unit is used for acquiring the state information of the 6DoF positioning and tracking device in the current environment and transmitting the state information to the calculation processing unit;

the calculation processing unit is used for determining 6DoF data of the 6DoF positioning and tracking device relative to the current environment according to the image data and the state information.

2. The 6DoF localization tracking device of claim 1,

the field angle of the tracking camera is not less than 160 degrees, the resolution is not less than 640 x 480, and the frame rate is 30Hz to 60 Hz; and the number of the first and second electrodes,

the optical lens of the tracking camera transmits visible light wave bands.

3. The 6DoF localization tracking device of claim 1,

the state information includes angular velocity and acceleration information;

the calculation processing unit determines the attitude information of the 6DoF positioning and tracking device according to the angular velocity and the acceleration information, and determines the 6DoF data of the 6DoF positioning and tracking device according to the attitude information and the image data.

4. The 6DoF localization tracking device of claim 1, wherein said tracking camera is provided in three; and the number of the first and second electrodes,

within a 360 ° physical area of the current environment, the three tracking cameras are evenly distributed.

5. The 6DoF localization tracking device according to claim 1, further comprising a battery connected to said tracking camera, said inertial navigation unit, and said computational processing unit; wherein the content of the first and second substances,

the battery is used for supplying electric energy required by operation to the 6DoF positioning and tracking device.

6. The 6DoF localization tracking device of claim 1,

the inertial navigation unit comprises a 9-axis inertial sensor;

the output frequency of the 9-axis inertial sensor is not less than 1000 Hz.

7. A6 DoF localization tracking method is characterized by comprising the following steps:

acquiring image data acquired by a tracking camera, and extracting image features of the image data;

matching the image characteristics of different tracking cameras pairwise to obtain three-dimensional characteristics corresponding to the image data;

based on the three-dimensional characteristics and the state information acquired by the inertial navigation unit, carrying out attitude estimation and characteristic tracking on the 6DoF positioning and tracking device in the current environment;

determining 6DoF data for the 6DoF localization tracking device based on the results of the pose estimation and feature tracking.

8. The 6DoF localization tracking method of claim 7, further comprising:

and judging whether the current environment is changed or not based on the feature tracking result, and when the current environment is changed, re-performing attitude estimation and feature tracking on the 6DoF positioning and tracking device in the new environment.

9. The 6DoF localization tracking method of claim 7, further comprising:

and performing dense semantic map construction, light beam method plane processing and repeated frame removal processing on the basis of the 6DoF data of the 6DoF positioning and tracking device.

10. An electronic device comprising the 6DoF localization tracking device of any of claims 1 to 6; wherein the content of the first and second substances,

the electronic device includes a VR device and a MR device.

Technical Field

The invention relates to the technical field of VR/AR, in particular to a 6DoF positioning and tracking device, a method and electronic equipment.

Background

At present, in a user scene in the AR/VR field, especially in a commercial scene, in order to enable a user to have more exciting and more immersive experience in VR/MR, some contents with more exciting and stronger experience, such as game contents of multi-player gunshots and the like, are often developed, a real gun apparatus in a virtual scene is generally simulated for a toy gun held by a hand of the user, in addition, some training scenes, such as a fire drill trained in a VR/MR mode, are often realized to simulate a real fire extinguisher object, and the fire drill is performed by operating one simulated fire extinguisher object in the real objects and a VR/MR head-mounted all-in-one machine.

In each of the above scenarios, a user operates a real simulation object to achieve the purpose of virtual and real combined experience, and in order to achieve the purpose, most of the current solutions fix a handle controller on some simulation objects, so that the simulation objects have a 6DoF tracking function. However, due to some limitations of the handle controller, the actual user experience is often not good enough. For example, the volume of the handle is often difficult to be matched with the simulated object properly, and is limited by some sensors of the handle, for example, the handle controller of the electromagnetic scheme is relatively easy to be affected by the limitation of some environmental objects such as magnets, and the like, and is often not matched with the simulated object properly in some scenes.

It can be known that the existing 6DoF tracking scheme has strict limitations on environmental objects, which affects the adaptation to the simulation object and results in poor user experience.

Disclosure of Invention

In view of the foregoing problems, an object of the present invention is to provide a 6DoF localization tracking apparatus, a method and an electronic device, so as to solve the problems of poor user experience and the like caused by strict restriction on environmental objects, affecting the adaptation to the simulated object, and the like in the existing 6DoF tracking scheme.

The invention provides a 6DoF positioning and tracking device, which comprises at least three tracking cameras, an inertial navigation unit and a calculation processing unit; the tracking cameras are distributed in a preset angle mode and used for acquiring image data in the current environment and transmitting the image data to the computing and processing unit; the inertial navigation unit is used for acquiring the state information of the 6DoF positioning and tracking device in the current environment and transmitting the state information to the calculation processing unit; and the calculation processing unit is used for determining 6DoF data of the 6DoF positioning and tracking device relative to the current environment according to the image data and the state information.

In addition, the optional technical scheme is that the field angle of the tracking camera is not less than 160 degrees, the resolution is not less than 640 x 480, and the frame rate is 30Hz to 60 Hz; and, the optical lens of the tracking camera transmits the visible light band.

In addition, an optional technical solution is that the state information includes angular velocity and acceleration information; the calculation processing unit determines attitude information of the 6DoF localization tracking device according to the angular velocity and the acceleration information, and determines 6DoF data of the 6DoF localization tracking device according to the attitude information and the image data.

In addition, the selectable technical scheme is that three tracking cameras are arranged; and, within the 360 physical region of the current environment, the three tracking cameras are evenly distributed.

In addition, the optional technical scheme is that the system further comprises a battery connected with the tracking camera, the inertial navigation unit and the calculation processing unit; wherein, the battery is used for providing the electric energy required by the operation for the 6DoF positioning and tracking device.

In addition, the optional technical solution is that the inertial navigation unit comprises a 9-axis inertial sensor; the output frequency of the 9-axis inertial sensor is not less than 1000 Hz.

In another aspect, the present invention further provides a 6DoF localization tracking method, including: acquiring image data acquired by a tracking camera, and extracting image characteristics of the image data; matching the image characteristics of different tracking cameras pairwise to obtain three-dimensional characteristics corresponding to the image data; based on the three-dimensional characteristics and the state information acquired by the inertial navigation unit, carrying out attitude estimation and characteristic tracking on the 6DoF positioning and tracking device in the current environment; based on the results of the pose estimation and feature tracking, 6DoF data for the 6DoF localization tracking device is determined.

In addition, an optional technical solution is that: and judging whether the current environment is changed or not based on the feature tracking result, and when the current environment is changed, re-performing attitude estimation and feature tracking on the 6DoF positioning and tracking device in the new environment.

In addition, an optional technical solution is that: based on 6DoF data of the 6DoF positioning and tracking device, dense semantic map construction, beam method plane processing and repeated frame removal processing are carried out.

According to another aspect of the present invention, an electronic device is provided, which includes the above 6DoF localization tracking apparatus; wherein the electronic device comprises a VR device and a MR device.

By using the 6DoF positioning and tracking device, the method and the electronic equipment, image data under the current environment are acquired by the tracking camera and are transmitted to the calculation processing unit; the inertial navigation unit is used for acquiring the state information of the 6DoF positioning and tracking device in the current environment and transmitting the state information to the calculation processing unit; the calculation processing unit determines 6DoF data of the 6DoF positioning and tracking device relative to the current environment according to the image data and the state information, a special sensor is not needed, the use limit of the physical environment of the sensor is not too much, the overall size is small, the device can be more easily adapted to simulation objects, the positioning and tracking effect is good, and the user experience is strong.

To the accomplishment of the foregoing and related ends, one or more aspects of the invention comprise the features hereinafter fully described. The following description and the annexed drawings set forth in detail certain illustrative aspects of the invention. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed. Further, the present invention is intended to include all such aspects and their equivalents.

Drawings

Other objects and results of the present invention will become more apparent and more readily appreciated as the same becomes better understood by reference to the following description taken in conjunction with the accompanying drawings. In the drawings:

FIG. 1 is a schematic diagram of a 6DoF localization tracking device according to an embodiment of the present invention;

fig. 2 is a flowchart of a 6DoF localization tracking method according to an embodiment of the present invention.

The same reference numbers in all figures indicate similar or corresponding features or functions.

Detailed Description

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It may be evident, however, that such embodiment(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more embodiments.

In the description of the present invention, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the invention and to simplify the description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be considered limiting of the invention.

To describe the 6DoF localization tracking device and method of the present invention in detail, the following describes an embodiment of the present invention in detail with reference to the accompanying drawings.

Fig. 1 shows a schematic principle of a 6DoF localization tracking device according to an embodiment of the present invention.

As shown in fig. 1, the 6DoF positioning and tracking device of the embodiment of the present invention includes at least three tracking cameras, an inertial navigation unit and a calculation processing unit; the tracking cameras are distributed in a preset angle mode and used for acquiring image data in the current environment and transmitting the image data to the computing and processing unit; the inertial navigation unit is used for acquiring the state information of the 6DoF positioning and tracking device in the current environment and transmitting the state information to the calculation processing unit; finally, the calculation processing unit is used for determining 6DoF data of the 6DoF positioning and tracking device relative to the current environment according to the acquired image data and the state information.

In the 6DoF localization and tracking device of the present invention, the parameters of the tracking camera may be configured as: the field angle of the tracking camera is not less than 160 degrees, the resolution is not less than 640 x 480, and the frame rate range is 30Hz to 60 Hz; in addition, the optical lens of the tracking camera can transmit the visible light wave band, and in the specific application process, the parameters can be flexibly adjusted according to the requirements or requirements of the user, and are not limited to the specific values.

In order to ensure that the tracking camera can completely acquire the image data of the current environment and reduce the number of the tracking cameras as much as possible, in one embodiment of the present invention, three tracking cameras are provided, and the three tracking cameras are uniformly distributed in a 360 ° physical area of the current environment, and are distributed like three vertexes of a triangle.

In addition, a high-precision inertial navigation unit can be built in the 6DoF positioning and tracking device, the inertial navigation unit can select a 9-axis inertial sensor, the output frequency of the 9-axis inertial sensor is not less than 1000Hz, the angular velocity and acceleration information of the 6DoF positioning and tracking device in the current environment can be obtained in real time through the inertial navigation unit, and the calculation processing unit can obtain the attitude information of the 6DoF positioning and tracking device in the current environment based on the angular velocity and acceleration information.

It can be known that the state information includes an angular velocity and an acceleration of the 6DoF localization tracking device, and the calculation processing unit may obtain the attitude information of the 6DoF localization tracking device in the current environment based on the angular velocity and the acceleration information, and further determine the 6DoF data of the 6DoF localization tracking device according to the attitude information and the image data, wherein inertial navigation (inertial navigation) mainly refers to a technique of obtaining instantaneous velocity and instantaneous position data of the device by measuring the acceleration of the corresponding device and automatically performing an integral operation.

In order to simplify the use of the 6DoF positioning and tracking device, a battery with certain capacity can be arranged in the 6DoF positioning and tracking device, the battery is connected with the tracking camera, the inertial navigation unit and the calculation processing unit, the battery provides electric energy required by operation for the 6DoF positioning and tracking device, and the built-in battery is arranged, so that the circuit connection between the 6DoF positioning and tracking device and a simulation object can be omitted, and the use operation of the device is simplified.

In addition, a small fan with certain heat dissipation capacity can be arranged in the 6DoF positioning and tracking device, so that the temperature stability of the device in the long-term operation process is ensured, and the adverse effect of high temperature on the performance of each structural part is prevented.

In one embodiment of the present invention, an example is shown in which three tracking cameras are provided, wherein a first tracking camera acquires first image data, a second tracking camera acquires second image data, and a third tracking camera acquires third image data, and then feature detection is performed on the first image data, the second image data, and the third image data, respectively, to acquire corresponding image features; and then, carrying out stereo matching processing on three different image characteristics, matching two characteristics in a stereo matching process, further acquiring corresponding three-dimensional characteristics, and sending the three-dimensional characteristics to a tracking module, wherein the tracking module carries out attitude estimation and characteristic tracking on the 6DoF positioning and tracking device according to the three-dimensional characteristics and data acquired by an inertial navigation unit, the attitude estimation is used for acquiring attitude information of the 6DoF positioning and tracking device, and the characteristic tracking is used for tracking between adjacent frames of image data.

The system also comprises a mapping module and a repositioning module which are connected with the tracking module, wherein the repositioning module is mainly used for judging whether the current environment changes or not based on the result of feature tracking, and when the current environment changes due to the movement of the 6DoF positioning and tracking device, the repositioning module is used for performing feature extraction (feature detection) and stereo matching on the 6DoF positioning and tracking device again, and determining feature description and corresponding 3D point information in the new environment. In addition, the mapping module further performs operations such as dense semantic map construction (map densification), planar processing by a light beam method, repeated frame removal processing and the like mainly according to results of feature tracking and attitude estimation, which are not listed one by one, and can implement related functions according to specific application environments and requirements.

Corresponding to the above 6DoF localization and tracking device, the present invention further provides a 6DoF localization and tracking method, and fig. 2 shows a flow of the 6DoF localization and tracking method according to an embodiment of the present invention.

As shown in fig. 2, the 6DoF localization tracking method according to the embodiment of the present invention includes:

s110: acquiring image data acquired by a tracking camera, and extracting image characteristics of the image data;

s120: matching the image characteristics of different tracking cameras pairwise to obtain three-dimensional characteristics corresponding to the image data;

s130: based on the three-dimensional characteristics and the state information acquired by the inertial navigation unit, carrying out attitude estimation and characteristic tracking on the 6DoF positioning and tracking device in the current environment;

s140: 6DoF data of the 6DoF positioning and tracking device in the current environment are determined based on the results of the attitude estimation and the feature tracking.

In addition, the 6DoF localization tracking method of the present invention further includes: and judging whether the current environment is changed or not based on the feature tracking result, and when the current environment is changed, re-performing attitude estimation and feature tracking on the 6DoF positioning and tracking device in the new environment.

In another embodiment of the present invention, based on the 6DoF data of the 6DoF localization and tracking device, dense semantic map construction, beam normal plane processing, repeated frame removal processing, etc. may be performed, which are not listed here, and the related functions may be implemented according to specific application environments and requirements.

According to another aspect of the present invention, there is also provided an electronic device, including the above 6DoF localization tracking apparatus; wherein the electronic device comprises a VR device and a MR device.

It should be noted that, the embodiments of the 6DoF localization tracking method and the electronic device can refer to the description of the embodiments of the 6DoF localization tracking device, which is not listed here.

According to the 6DoF positioning and tracking device, the method and the electronic equipment, the image data in the current environment are obtained through the tracking camera, the state information of the 6DoF positioning and tracking device in the current environment is obtained through the inertial navigation unit, the 6DoF data of the 6DoF positioning and tracking device relative to the current environment are determined through the calculation processing unit according to the image data and the state information, a special sensor is not needed, the overall size is small, the use limitation of a sensor on the physical environment is avoided, the overall size is small, the 6DoF positioning and tracking device can be easily adapted to a simulation object, a connecting circuit is not needed, power is directly supplied through a battery, the positioning and tracking effect is good, and the user experience is strong.

The 6DoF localization tracking apparatus, method and electronic device according to the present invention are described above by way of example with reference to the accompanying drawings. However, it should be understood by those skilled in the art that various modifications can be made to the 6DoF localization and tracking apparatus, method and electronic device provided by the present invention without departing from the scope of the present invention. Therefore, the scope of the present invention should be determined by the contents of the appended claims.