阅读说明：本技术 拍摄系统标定方法、装置、计算机设备和存储介质 (Shooting system calibration method and device, computer equipment and storage medium ) 是由 赖东东 谢亮 谭明朗 付伟 于 2021-06-16 设计创作，主要内容包括：本申请涉及一种拍摄系统标定方法、装置、计算机设备和存储介质。方法包括：根据视觉系统与惯性测量单元之间的旋转误差,确定时延的参考值；根据时延的参考值,对视觉系统与惯性测量单元之间的时延及旋转外参进行标定,视觉系统与惯性测量单元耦合在同一拍摄系统中,旋转误差是基于标定视频所确定的,标定视频是拍摄系统在旋转下拍摄得到的,旋转误差用于表示基于视觉系统所确定的旋转状态与基于惯性测量单元所确定的旋转状态之间的不一致程度。由于在对视觉系统与惯性测量单元之间的时延及旋转外参进行标定之前,先确定了时延的参考值,从而可以避免时延不会偏差太大,进而避免了因时延偏差太大而导致标定结果不太精准,甚至标定失败的情形。(The application relates to a shooting system calibration method, a shooting system calibration device, computer equipment and a storage medium. The method comprises the following steps: determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit; calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit. Because the reference value of the time delay is determined before the time delay and the rotating external parameter between the visual system and the inertial measurement unit are calibrated, the condition that the time delay is not too large in deviation can be avoided, and the condition that the calibration result is not accurate and even the calibration fails due to the fact that the time delay is too large in deviation is avoided.)

1. A shooting system calibration method is characterized by comprising the following steps:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating a time delay and a rotation external parameter between a visual system and an inertial measurement unit according to a reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between a rotation state determined based on the visual system and a rotation state determined based on the inertial measurement unit.

2. The method of claim 1, wherein determining a reference value for the time delay based on a rotational error between the vision system and the inertial measurement unit comprises:

updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the first updated value of the time delay, repeating the updating and determining processes to determine the rotation error meeting a first preset condition, and taking the first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

3. The method of claim 2, wherein updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the vision system and the inertial measurement unit based on the first updated value of the time delay, and repeating the updating and determining to determine a rotation error satisfying a first predetermined condition comprises:

updating the first initial value according to a preset step length until a first updated value obtained by updating is not located in a first time delay interval, wherein the first initial value is located in the first time delay interval;

determining a rotation error corresponding to a first updating value obtained by each updating;

and determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

4. The method of claim 3, wherein determining the rotation error corresponding to the first updated value obtained from each update comprises:

for a first updating value obtained by each updating, adjusting a time point corresponding to each frame of image in the calibration video according to the first updating value;

acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images;

and determining a rotation error corresponding to the first updating value according to a first rotation matrix between every two adjacent frames of images and a second rotation matrix corresponding to every two adjacent frames of images.

5. The method according to claim 4, wherein determining the rotation error corresponding to the first updated value according to a first rotation matrix between each two adjacent frames of images and a second rotation matrix corresponding to each two adjacent frames of images comprises:

calculating the corresponding shaft angle difference value of each two adjacent frames of images according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images;

and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

6. The method according to claim 1, wherein calibrating the time delay and the rotation parameter between the vision system and the inertial measurement unit according to the time delay reference value comprises:

determining a second time delay interval according to the reference value of the time delay, and determining a second initial value of the time delay in the second time delay interval;

updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertia measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertia measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system.

7. The method of claim 6, wherein obtaining the reference value of the rotational parameter between the vision system and the inertial measurement unit according to the second updated value of the time delay comprises:

determining a hand-eye calibration equation set according to a second updating value of the time delay, wherein a rotation external parameter in the hand-eye calibration equation set is an unknown quantity;

and solving the hand-eye calibration equation set to obtain a reference value of the rotation external parameter.

8. A shooting system calibration device is characterized by comprising:

the determining module is used for determining a reference value of time delay according to a rotation error between the vision system and the inertial measurement unit;

the calibration module is used for calibrating time delay and rotation external parameters between a visual system and an inertial measurement unit according to a reference value of the time delay, the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Technical Field

The present disclosure relates to the field of shooting calibration technologies, and in particular, to a method and an apparatus for calibrating a shooting system, a computer device, and a storage medium.

Background

An IMU (Inertial Measurement Unit) and a vision system are generally present in a photographing system. For example, a device for realizing positioning navigation has a camera system mounted thereon. The surrounding environment needs to be photographed through a visual system in the photographing system, and the equipment posture needs to be solved through an IMU in the photographing system. That is, implementing other functions such as positioning and navigation requires establishing that the relevant parameters between the vision system and the IMU can be accurately determined. Therefore, the camera system needs to be calibrated to determine the relevant parameters between the vision system and the IMU.

In the prior art, when a shooting system is calibrated, generally only the time delay between a vision system and an IMU is calibrated, or the time delay and the rotation external parameter are calibrated at the same time on the premise of adopting a calibration board video, or the time delay is roughly calibrated in a b-spline alignment manner, or the rotation external parameter and the time delay are calibrated simultaneously in an iterative optimization manner by constructing a residual equation. In the actual implementation process, if the time stamp interval of the calibration video frame is large or a non-calibration board video is adopted, the calibration method is prone to failure, and the calibration result obtained by the calibration method is not accurate enough and prone to errors.

Disclosure of Invention

In view of the above, it is necessary to provide a method and an apparatus for calibrating a camera system, a computer device and a storage medium, which can solve the problem of inaccurate calibration result.

A shooting system calibration method comprises the following steps:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

In one embodiment, determining a reference value for the time delay based on a rotational error between the vision system and the inertial measurement unit comprises:

updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the first updated value of the time delay, repeating the updating and determining processes to determine the rotation error meeting a first preset condition, and taking the first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

In one embodiment, the updating a first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the vision system and the inertial measurement unit according to the first updated value of the time delay, and repeating the updating and determining to determine the rotation error satisfying a first preset condition, includes:

updating the first initial value according to a preset step length until the updated first updated value is not located in a first time delay interval, wherein the first initial value is located in the first time delay interval;

determining a rotation error corresponding to a first updating value obtained by each updating;

and determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

In one embodiment, determining the rotation error corresponding to the first updated value obtained by each update includes:

adjusting and calibrating a time point corresponding to each frame of image in the video according to a first updating value obtained by updating each time;

acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images;

and determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images.

In one embodiment, determining a rotation error corresponding to the first update value according to a first rotation matrix between each two adjacent frames of images and a second rotation matrix corresponding to each two adjacent frames of images includes:

calculating the corresponding shaft angle difference value of each two adjacent frames of images according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images;

and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

In one embodiment, calibrating the time delay and the rotation external parameter between the vision system and the inertial measurement unit according to the reference value of the time delay comprises:

determining a second time delay interval according to the reference value of the time delay, and determining a second initial value of the time delay in the second time delay interval;

updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system.

In one embodiment, obtaining a reference value of the rotational parameter between the vision system and the inertial measurement unit according to the second updated value of the time delay comprises:

determining a hand-eye calibration equation set according to the second updating value of the time delay, wherein the external rotation parameter in the hand-eye calibration equation set is an unknown quantity;

and solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter.

A shooting system calibration device, the device includes:

the determining module is used for determining a reference value of time delay according to a rotation error between the vision system and the inertial measurement unit;

the calibration module is used for calibrating time delay and rotation external parameters between the visual system and the inertial measurement unit according to a reference value of the time delay, the visual system and the inertial measurement unit are coupled in the same shooting system, a rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between a rotation state determined based on the visual system and a rotation state determined based on the inertial measurement unit.

A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

The calibration method, the calibration device, the computer equipment and the storage medium of the shooting system determine the reference value of the time delay according to the rotation error between the vision system and the inertia measurement unit. And calibrating the time delay and the rotation external parameter between the visual system and the inertia measurement unit according to the reference value of the time delay. Because the reference value of the time delay is determined before the time delay and the rotating external parameter between the visual system and the inertial measurement unit are calibrated, the condition that the time delay is not too large in deviation can be avoided, and the condition that the calibration result is not accurate and even the calibration fails due to the fact that the time delay is too large in deviation is avoided. In addition, the time delay is roughly calibrated, and then the time delay and the rotating external parameter are finely calibrated. Because the optimization range of the time delay is narrowed, the calculation amount can be reduced, and the calibration efficiency is improved.

Drawings

FIG. 1 is a schematic flow chart illustrating a calibration method of a photographing system according to an embodiment;

FIG. 2 is a schematic flow chart illustrating a calibration method of a photographing system according to another embodiment;

FIG. 3 is a diagram illustrating delay adjustment in one embodiment;

FIG. 4 is a schematic diagram of a hand-eye calibration equation in one embodiment;

FIG. 5 is a graphical illustration of a time delay versus average error in one embodiment;

FIG. 6 is a block diagram showing the configuration of a photographing system calibration apparatus according to an embodiment;

FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various terms, but these terms are not limited by these terms unless otherwise specified. These terms are only used to distinguish one term from another. For example, the third preset threshold and the fourth preset threshold may be the same or different without departing from the scope of the present application.

An IMU (Inertial Measurement Unit) and a vision system are generally present in a photographing system. For example, a device for realizing positioning navigation has a camera system mounted thereon. The surrounding environment needs to be photographed through a visual system in the photographing system, and the equipment posture needs to be solved through an IMU in the photographing system. That is, implementing other functions such as positioning and navigation requires establishing that the relevant parameters between the vision system and the IMU can be accurately determined. Therefore, the camera system needs to be calibrated to determine the relevant parameters between the vision system and the IMU.

In the related art, when a shooting system is calibrated, generally only the time delay between a vision system and an IMU is calibrated, or the time delay and the rotation external parameter are calibrated at the same time on the premise of adopting a calibration board video, or the time delay coarse calibration is carried out in a b-spline alignment manner, or the rotation external parameter and the time delay are calibrated simultaneously in an iterative optimization manner by constructing a residual error equation. In the actual implementation process, if the time stamp interval of the calibration video frame is large or a non-calibration board video is adopted, the calibration method is prone to failure, and the calibration result obtained by the calibration method is not accurate enough and prone to errors.

In view of the above problems in the related art, an embodiment of the present invention provides a method for calibrating a shooting system, where the method may be applied to processing devices such as a server, and the processing devices may also be, but are not limited to, various personal computers and notebook computers. It should be noted that, the numbers of "a plurality" and the like mentioned in the embodiments of the present application each refer to a number of "at least two", for example, "a plurality" refers to "at least two".

Before describing the embodiments of the present application, a main application scenario of the present application will be described. The shooting system calibration method is mainly applied to scenes of shooting system calibration, and the shooting system can be formed by a vision system and an inertia measurement unit. The shooting system may be a shooting device such as a camera, and this is not particularly limited in the embodiment of the present invention. Taking the photographing system as a camera as an example, the vision system is an imaging module in the camera, and the inertial measurement unit may be disposed in the camera as well. In combination with the content of the above embodiments, in one embodiment, referring to fig. 1, a method for calibrating a photographing system is provided. The method is applied to a terminal, and an execution subject is taken as an example for explanation, and the method comprises the following steps:

101. determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

102. calibrating the rotation external parameter between the vision system and the inertial measurement unit according to the reference value of the time delay, wherein the vision system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the vision system and the rotation state determined based on the inertial measurement unit.

Before the step 101 is executed, the internal parameters and distortion parameters of the shooting system can be calibrated in advance, and a calibrated video is shot by the shooting system in a rotating state. It should be noted that when the calibration video is obtained by shooting, the shooting system can rotate in yaw, pitch and rolling directions as much as possible, so that the calibration video can cover various factors influencing calibration, and further calibration accuracy can be higher when calibration is subsequently performed based on the calibration video. It should be noted that the photographing system may rotate at a low speed during the rotation, and further, may rotate at a variable speed. The reason why the rotation is performed at a low speed is that if the rotation is too fast, the situation of shooting blur of the calibration video occurs, and for two adjacent frames of images, respective feature points in the two frames of images need to be extracted in the subsequent process, and a first rotation matrix is obtained, so that the feature points are not extracted accurately enough if the shooting blur of the calibration video occurs, and the extraction and tracking accuracy is affected. The variable speed rotation is because the constant speed rotation is not a normal state of motion change of the photographing system during photographing, and the variable speed rotation is a normal state of motion change. Therefore, the photographing system is set to rotate in a variable speed mode, so that the calibration video can cover the variable speed rotation factor which influences the calibration, and the calibration precision is higher when calibration is carried out on the basis of the calibration video subsequently.

In addition, according to the calibration video, a series of rotation states determined based on the vision system can be calculated, and the rotation states can be embodied as rotation state changes between every two adjacent frames of images in the calibration video. The series of rotation states may be represented by a time sequence composed of different time periods, each time period may be a time period corresponding to each two adjacent frames of images in the calibration video, and each time period corresponds to one rotation state. And the rotation state change between every two adjacent frames of images can be used for describing how the shooting system undergoes rotation in the time period corresponding to every two adjacent frames of images. That is, the calculated series of rotation states determined based on the vision system, in effect, describes how the camera system has undergone rotation, standing at the angle of the vision system.

Similarly, a series of rotation states determined based on the inertial measurement unit may also be calculated, and the series of rotation states may be represented by a time series of different time periods, one rotation state for each time period. The different time periods of the time sequence may be time periods corresponding to every two adjacent frames of images in the calibration video. The rotation state corresponding to each time period and determined based on the inertial measurement unit describes what rotation the shooting system undergoes for the angle standing on the inertial measurement unit.

Since both describe how the camera system has undergone rotation in different time periods of the time series, if the time series corresponding to both are also the same, the rotation states determined by both are identical in an ideal state. And due to the existence of time delay, deviation exists between the determined rotation states of the two, namely rotation error. Wherein, the rotation error between the rotation states determined by the two will be different due to different values of the time delay. Based on the above principle, the time delay can be determined from the rotation error between the vision system and the inertial measurement unit. Correspondingly, when step 101 is executed, q discrete delay values may be determined, and a rotation error under each delay value is calculated, so that a delay value with a smaller rotation error may be selected as a reference value of the delay. In summary, the process corresponding to step 101 can be understood as a process of roughly calibrating time delay. Specifically, step 101 may adopt a b-spline alignment mode to perform coarse calibration on the time delay, but the method is prone to a calibration error under the condition of a larger time stamp interval; or the time delay can be roughly calibrated by adopting a linear solving mode.

In step 102, the time delay and the rotation external parameter may be calibrated according to the reference value of the time delay. The calibration mode may be simultaneous calibration by constructing a hand-eye calibration equation to solve, which is not specifically limited in the embodiments of the present invention, that is, in addition to constructing the hand-eye calibration equation, calibration may be performed by constructing other equations to solve, for example, calibrating by constructing a time-delay calibration equation and a rotation external parameter calibration equation separately, or calibrating by constructing a residual error equation to perform optimization iteration. The process corresponding to step 102 may be understood as a process of performing fine calibration on the time delay and the rotation parameter.

According to the method provided by the embodiment of the invention, the reference value of the time delay is determined according to the rotation error between the vision system and the inertial measurement unit. And calibrating the time delay and the rotation external parameter between the visual system and the inertia measurement unit according to the reference value of the time delay. Because the reference value of the time delay is determined before the time delay and the rotating external parameter between the visual system and the inertial measurement unit are calibrated, the condition that the time delay is not too large in deviation can be avoided, and the condition that the calibration result is not accurate and even the calibration fails due to the fact that the time delay is too large in deviation is avoided. In addition, the time delay is roughly calibrated, and then the time delay and the rotating external parameter are finely calibrated. Because the optimization range of the time delay is narrowed, the calculation amount can be reduced, and the calibration efficiency is improved.

In combination with the above embodiments, in one embodiment, the embodiment of the present invention does not specifically limit the way of determining the reference value of the time delay according to the rotation error between the vision system and the inertial measurement unit, including but not limited to: updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the first updated value of the time delay, repeating the updating and determining processes to determine the rotation error meeting a first preset condition, and taking the first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

In the above process, the first preset condition is mainly used to measure whether the rotation error determined by the first update value based on the time delay is small enough, and the first preset condition may be set according to a requirement, which is not specifically limited in the embodiment of the present invention. The first preset condition may be that the rotation error falls within a first preset range, or that the rotation error gradually converges to a certain value, which is not specifically limited in the embodiment of the present invention. In addition, the first initial value of the time delay may be selected within a certain range, such as [ -maxT, maxT ], and when the first initial value of the time delay is updated, the first initial value of the time delay may also be updated within the above range. The updating mode may be random selection, which is not specifically limited in this embodiment of the present invention.

In the method provided by the embodiment of the invention, the first initial value of the time delay is updated to obtain the first updated value of the time delay, the rotation error between the visual system and the inertial measurement unit is determined according to the first updated value of the time delay, the updating and determining processes are repeated to determine the rotation error meeting the first preset condition, and the first updated value corresponding to the rotation error meeting the first preset condition is used as the reference value of the time delay. The first updating values with different time delays are screened based on the rotation error determined by the first updating values with different time delays, the first updating values which can meet the first preset condition are determined to be used as the reference values of the time delays, and the first preset condition can be used for measuring whether the rotation error determined based on the first updating values of the time delays is small enough, so that the determined reference values can be more accurate.

With reference to the foregoing embodiments, in an embodiment, regarding updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the vision system and the inertial measurement unit according to the first updated value of the time delay, and repeating the updating and determining processes to determine a rotation error meeting a first preset condition, which is not specifically limited in the embodiment of the present invention. Referring to fig. 2, the method process includes:

201. updating the first initial value according to a preset step length until the updated first updated value is not located in a first time delay interval, wherein the first initial value is located in the first time delay interval;

202. determining a rotation error corresponding to a first updating value obtained by each updating;

203. and determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

In the foregoing process, when the first initial value is updated, the first initial value may be accumulated according to a preset step length, and the accumulated value accumulated each time is used as a first updated value of the time delay, or after the first initial value is accumulated according to the preset step length, a numerical value around the accumulated value is selected as the first updated value of the time delay. Taking the first time delay interval [ -maxT, maxT ] as an example, the first initial value may be within the interval, and the above-mentioned accumulation process may be performed until the accumulated value exceeds the upper limit of the interval.

In addition, the preset step size may be set according to the calibrated resolution res, for example, the preset step size may be an integral multiple of res, that is, m × res is used as the preset step size. It should be understood that the smaller the preset step size is, the smaller the change of the first update value obtained when the time delay is updated each time is, and thus the calibration accuracy is higher. Accordingly, the number of times of calibration is increased, and the calculation amount is increased. That is, the smaller the preset step length is, the higher the calibration accuracy is.

According to the method provided by the embodiment of the invention, the first initial value is updated according to the preset step length until the updated first updated value is not in the first time delay interval. And determining the rotation error corresponding to the first updated value obtained by each updating. And determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition. The first updating values with different time delays are screened based on the rotation errors determined by the first updating values with different time delays, the first updating values which can meet the first preset condition are determined to be used as reference values of the time delays, the first preset condition can be used for judging whether the rotation errors determined by the first updating values based on the time delays are small enough, the first updating values of the time delays are repeatedly updated in a first time delay interval according to a preset step length until the first updating values which can meet the first preset condition are determined to be used as the reference values of the time delays, and therefore the situation that calibration errors occur under the condition that the time stamp interval between a vision system and an inertial measurement unit is too large is avoided, and the determined reference values can be more accurate.

With reference to the content of the foregoing embodiment, in an embodiment, the embodiment of the present invention does not specifically limit a manner of determining a rotation error corresponding to the first updated value obtained by each update, which includes but is not limited to: adjusting and calibrating a time point corresponding to each frame of image in the video according to a first updating value obtained by updating each time; acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images; and determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images.

As shown in fig. 3, the black solid arrow in the upper half of fig. 3 is a time point corresponding to the image, and may be represented by a sampling time point of the image, and the gray dashed arrow smaller than the black solid arrow is a sampling time of the inertial measurement unit. If the time delay between the visual system and the inertial measurement unit is dt, whether the visual system is slower than the inertial measurement unit or the inertial measurement unit is slower than the visual system by dt, reference may be made to the grey solid arrow in the lower half of fig. 3. Taking the first black solid arrow from left to right in the top half of fig. 3 as an example of the sampling time point of the first frame image, and the second black solid arrow from left to right as an example of the sampling time point of the second frame image, the first gray solid arrow from left to right in the bottom half of fig. 3 also represents the sampling time point of the first frame image, and the second gray solid arrow from left to right likewise represents the sampling time point of the second frame image, except that the sampling time point between the two is delayed by dt duration.

When the first rotation matrix between every two adjacent frames of images in the calibration video is obtained, the corresponding relation of pixel points between the two frames of images can be obtained through a direct method or a characteristic point method, and then the first rotation matrix between the two frames of images is obtained through calculation in a mode of decomposing an essential matrix or a basic matrix by utilizing epipolar geometric constraint. It should be noted that, if the exposure mode adopted by the vision system is rolling shutter exposure, since the feature points at each position in the image may not be obtained by exposure at the same time, in order to make the first rotation matrix between the calculated images closer to the middle time of image exposure, before the feature points are used to calculate the first rotation matrix, when extracting the feature points, the extracted feature points may be distributed uniformly on the image as much as possible.

By the aboveThe process can obtain the relative rotation between the time sequence and the image in the calibration videoTo carry out the presentation of the contents,in practice, different time segments and the corresponding first rotation matrix for each time segment are represented. Wherein, time represents the time sequence composed of the time periods corresponding to every two adjacent frames of images in the calibration video, andand the relative rotation between every two adjacent frame images is represented, namely a first rotation matrix.V in (a) indicates that the relative rotation is obtained by the vision system, c indicates the coordinate system of the vision system, and i indicates the number of frames, if i starts from the 0 th frame, the maximum value of i is the total number of frames in the calibration video minus 2.

As can be seen from fig. 3, although the time point corresponding to each frame of image in the calibration video is adjusted, the first rotation matrix between each two adjacent frames of images is determined according to the feature points in the images, so that the first rotation matrix is not affected by the time point corresponding to the image. And for the second rotation matrix corresponding to each two adjacent frames of images, the second rotation matrix is influenced by the time point corresponding to the image.

Referring to fig. 3, taking the example that the sampling time point of the first frame image in the upper half of fig. 3 is 1 second, and the sampling time point of the second frame image is 1.5 seconds, and the visual system is slower than the inertial measurement unit by 0.25 seconds, the measurement value of the inertial measurement unit between 1 second and 1.5 seconds should be used when acquiring the second rotation matrix corresponding to the first frame image and the second frame image. After the time point corresponding to each frame of image in the calibration video is adjusted, when a second rotation matrix corresponding to the first frame of image and the second frame of image is obtained, the used time is changed into the measurement value of the inertial measurement unit between 1.25 seconds and 1.75 seconds.

When the second rotation matrix corresponding to each two adjacent frames of images in the adjusted calibration video is obtained, pre-integration may be performed on the measurement values of the inertial measurement unit obtained in the time period corresponding to the two adjacent frames of images to obtain the second rotation matrix corresponding to the two adjacent frames of imagesWhere b denotes the inertial measurement unit coordinate system. The lower right i represents the image of the frame, and the above description can be referred to. The upper right i indicates that the relative rotation is obtained by the inertial measurement unit.

After adjusting the time point corresponding to each frame of image in the calibration video, the time sequence in the calibration video and the first rotation matrix between each two adjacent frames of images can be obtained byAnd (4) showing. Similarly, calibrating the second rotation matrix between the time sequence and each two adjacent frames of images in the video can be achieved byAnd (4) showing. Wherein time + dt may represent a time series formed by updating the time delay between the visual system and the inertial measurement unit.

For the same time period, such as the time period between the first frame image and the second frame image, the corresponding first rotation matrix can be obtainedAnd a second rotation matrixWherein the first rotation matrixThe angle representing the station at the vision system describes how the camera system has undergone rotation during the time period between one image frame and a second image frame, while the second image frame is rotatedRotation matrixThe angle representing the station at the inertial measurement unit describes how the camera system has undergone rotation during the time period between one frame of image and a second frame of image. First rotation matrixAnd a second rotation matrixThe difference between the first frame image and the second frame image represents the rotation error generated in the period between the first frame image and the second frame image. Wherein the rotation error can be represented by the degree of difference between the two matrices.

Through the process, the difference degree between the first rotation matrix and the second rotation matrix corresponding to each two adjacent frames of images can be calculated, and the difference degrees corresponding to each two adjacent frames of images are superposed to obtain the rotation error corresponding to the time delay dt. And the first updating value obtained by each updating is the time delay dt obtained by each updating.

According to the method provided by the embodiment of the invention, the time point corresponding to each frame of image in the calibration video is adjusted and calibrated according to the first update value obtained by updating each time. And acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted. And determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images. The first updating values with different time delays are screened based on the rotation error determined by the first updating values with different time delays, the first updating values which can meet the first preset condition are determined to be used as the reference values of the time delays, and the first preset condition can be used for measuring whether the rotation error determined based on the first updating values of the time delays is small enough, so that the determined reference values can be more accurate.

With reference to the above description, in an embodiment, the embodiment of the present invention does not specifically limit the manner of determining the rotation error corresponding to the first update value according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images, and includes but is not limited to: calculating the corresponding shaft angle difference value of each two adjacent frames of images according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images; and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

Wherein, the above calculation process can refer to the following formula (1):

in the above-mentioned formula (1),representing an axis angle determined from a first rotation matrix between the ith frame image and the (i + 1) th frame image,indicating an axis angle determined by a second rotation matrix corresponding to the two frame images of the i-th frame image and the i + 1-th frame image. The difference between the two axial angles is the axial angle difference corresponding to the two adjacent frames of images, and n +1 represents the total number of frames in the calibration video.

According to the method provided by the embodiment of the invention, the shaft angle difference value corresponding to each two adjacent frames of images is calculated according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images. And calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images. The first updating values with different time delays are screened based on the rotation errors determined by the first updating values with different time delays, the first updating values which can meet the first preset condition are determined to be used as reference values of the time delays, the first preset condition can be used for judging whether the rotation errors determined by the first updating values based on the time delays are small enough, the first updating values of the time delays are repeatedly updated in a first time delay interval according to a preset step length until the first updating values which can meet the first preset condition are determined to be used as the reference values of the time delays, and therefore the situation that calibration errors occur under the condition that the time stamp interval between a vision system and an inertial measurement unit is too large is avoided, and the determined reference values can be more accurate.

With reference to the above description of the embodiments, in an embodiment, the embodiment of the present invention does not specifically limit the way of calibrating the time delay and the rotational external parameter between the vision system and the inertial measurement unit according to the reference value of the time delay, which includes but is not limited to: determining a second time delay interval according to the reference value of the time delay, and determining a second initial value of the time delay in the second time delay interval; updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system.

The above process of determining the reference value of the time delay is actually a process of roughly calibrating the time delay. Thus, in the content provided by the embodiment of the present invention, the range of the second delay interval is much smaller than the range of the first delay interval in the above process. Taking the reference value of the delay as rawdt for example, the second delay interval can be represented by [ rawdt-2 × m res, rawdt +2 × m res ]. Where res denotes the nominal resolution and m denotes an integer multiple.

Similar to the first preset condition, the second preset condition may be that the rotation error falls within a second preset range, or that the rotation error gradually converges to a certain value, which is not specifically limited in the embodiment of the present invention. It should be noted that, unlike the coarse calibration process for obtaining the reference value of the time delay, the embodiment of the present invention provides a process for calibrating the time delay and the external rotation parameter simultaneously, which is equivalent to a process of fine calibration again. Thus, the second preset condition may be set more severe than the setting criterion of the first preset condition. Wherein the more stringent setting criteria may be embodied in that the second predetermined condition may ensure that the determined rotational error is smaller than the rotational error determined by the first predetermined condition.

According to the method provided by the embodiment of the invention, the second time delay interval is determined according to the reference value of the time delay, and the second initial value of the time delay is determined in the second time delay interval. Updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system. The time delay is roughly calibrated firstly, and then the time delay and the rotating external parameter are finely calibrated, so that the optimization range of the time delay is narrowed, the calculated amount can be reduced, and the calibration efficiency is improved.

In addition, the corresponding rotation external parameter can be determined based on the second updating value with different time delays, and then the corresponding rotation error is determined based on the second updating value with different time delays and the corresponding rotation external parameter, so that the time delays and the rotation external parameters are simultaneously screened, the second updating value and the rotation error which can meet the second preset condition are determined and serve as the calibration result, and the second preset condition can be used for judging whether the rotation error determined based on the first updating value with time delays is small enough, so that the determined calibration result can be more accurate.

In combination with the above description of the embodiments, in an embodiment, the embodiment of the present invention does not specifically limit the manner of obtaining the reference value of the external parameter of rotation between the vision system and the inertial measurement unit according to the second updated value of the time delay, which includes but is not limited to: determining a hand-eye calibration equation set according to the second updating value of the time delay, wherein the external rotation parameter in the hand-eye calibration equation set is an unknown quantity; and solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter.

From the rotation transformation relationship, a hand-eye calibration formula AX ═ XB can be constructed, which, in combination with the contents of the above example, can be specifically the following formula (2):

in the above-mentioned formula,andreference is made to the contents of the above examples, R_{b_c}I.e. the extrinsic parameter to be solved, and likewise a matrix. It should be noted that, in the following description,and R_{b_c}Are each a 3 x 3 matrix. The transformation process of the hand-eye calibration can refer to fig. 4.

It can be known from the content of the above embodiment that, on the premise that the second update value of the time delay is known, the first rotation matrix between every two adjacent frames of images in the calibration video and the second rotation matrix corresponding to every two adjacent frames of images can be obtained by solving. Therefore, a hand-eye calibration equation can be correspondingly constructed for every two adjacent frames of images in the calibration video, and a hand-eye calibration equation set can be obtained. And the number of the equations in the hand-eye calibration equation set is the total frame number in the calibration video minus 1.

The process of solving the equation can be converted into the following least square problem, and the following formula (3) can be referred to specifically:

in the above formula, it is actually a transposed matrix that multiplies the left matrix by the right matrix in formula (2). In an ideal situation, if the shaft angle difference is 0, the multiplication of the two should result in an identity matrix. Due to the rotation error, the identity matrix is usually not obtained, and the product size of the two can indicate the rotation error. In the formula (3), the first and second groups,representing robust kernel functions, i.e. weights, which can be calculated from practiceTo make the settings.

To find R to some extent_{b_c}Can minimize the cumulative value of the errors, and can minimize the cumulative value of the errors_{b_c}Namely the rotation external parameters needed to be solved. Based on the principle, when the hand-eye calibration equation set is solved to obtain the reference value of the rotation external parameter, the solution can be performed in an iterative optimization mode, and the solution can also be performed in a mode of converting the rotation external parameter into an axial angle or a quaternion solution linear equation set. Considering that the solution process of the quaternion is relatively fast, for convenience of understanding, the embodiment of the present invention linearly solves the hand-eye calibration equation set in a quaternion manner, where the specific process is as follows:

firstly, converting the hand-eye calibration equation set into a form of a quaternion left-multiplication matrix and a quaternion right-multiplication matrix, and specifically referring to the following formula (4):

in the above-mentioned formula (4),indicating the conversion of a quaternion to a 4 x 4 left-multiplied matrix,indicating the conversion of a quaternion to a 4 x 4 right-multiplying matrix.

The following homogeneous equation set can be obtained by accumulating the linear equations at the 1 st to nth time, and the following equation (5) can be referred to specifically:

in the above equation (5), the coefficient matrix Q in the homogeneous equation set_nThe dimension is 4n x 4, and since the number of frames used for calibration is typically large, it may reach hundreds of thousands of frames. Thus, for Q_nThe singular value decomposition takes a lot of time, and according to the derivation, the equation set Q_n*q_{b_c}0 andthe same is true. Thus, equation (5) can be converted to solveWherein the content of the first and second substances,dimension (d) of 4 x 4, the singular value decomposition is performed at a high speed.

With reference to the foregoing embodiment, after the reference value of the rotation external parameter between the vision system and the inertial measurement unit is obtained according to the second updated value of the time delay, the rotation error between the vision system and the inertial measurement unit may be determined according to the second updated value of the time delay and the reference value of the rotation external parameter. In conjunction with the contents of the above embodiments, the error formula for determining the rotation error can be referred to as the following formula (6):

of course, in practical implementation, in combination with the above hand-eye calibration formula, the error formula for determining the rotation error may also have various modifications, for example, the following formula (7) may be referred to:

according to the method provided by the embodiment of the invention, the hand-eye calibration equation set is determined according to the second updating value of the time delay, and the external rotation parameter in the hand-eye calibration equation set is an unknown quantity. And solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter. When the calibration result of the time delay and the rotation external parameter is obtained, a linear solving method is adopted, so that the situation that the obtained calibration result is not accurate enough and even the subsequent calibration fails due to the fact that the time delay and the rotation external parameter are trapped in a local minimum value in the optimization process can be avoided.

With reference to the above example, after solving the errors corresponding to different values of the delay, a graph of the values of the delay and the errors may refer to fig. 5. And (3) taking values dt of all time delays during calibration as horizontal coordinates, and drawing a time delay and average error curve by taking the average error Err corresponding to each dt as vertical coordinates. Fig. 5 shows the result of a certain calibration in practice, and the abscissa is the time delay dt of the vision system corresponding to the inertial measurement unit, and the specific unit is second. The ordinate is the average error calculated using the above error calculation formula at different dt, and the specific unit is the arc value. The lower and sharper the lowest point value of the curve is, the higher the effect and the precision of the time delay calibration are. Under different calibration precision testing tools, such as euroc data, which mainly uses data shot by global exposure, and utilizes time delay and rotation external parameters calibrated by euroc data, the corresponding calibration results can refer to the following table 1:

TABLE 1

Calibrating serial numbers	1	2	3	4	5	6	7
								Time delay/ms	-0.5	-0.5	-0.5	-0.5	-0.5	-0.5	-0.5
Error/degree of rotating external parameter	0.309	0.272	0.106	0.360	0.296	0.192	0.114

By using the evo data, which is mainly data shot by using the rolling shutter exposure, and using the time delay and the rotation external parameter calibrated by the evo data, the corresponding calibration result can refer to the following table 2:

TABLE 2

Calibrating serial numbers	1	2	3	4	5	6	7
								Time delay/ms	-1.5	-1.5	-0.5	-1.5	-2.0	-1.5	-1.5
Error/degree of rotating external parameter	0.104	0.388	0.373	0.328	0.418	0.392	0.314

From the above tables 1 and 2, it can be known that the time delay between the visual system calibrated by euroc data and the inertial measurement unit is relatively stable, and is substantially 0.5ms when tested on the data with sufficient rotation. Whereas the evo data mostly have calibration results: the calibration error is within 2ms, and the calibration error is between-0.0015 and 0s, and occasionally-0.002 and 0.001. The calibration result of the evo data is not stable with euroc data, and the evo data possibly has a certain relation with that obtained by shooting the evo data by a rolling shutter camera. In addition, the error of the external rotation parameter is an angle difference, the error corresponding to the external rotation parameter calibrated by the euroc data is about 0.3 degrees, and the error between the results of the external rotation parameter calibrated by the evo data and the anti-shake group calibration is about 0.5 degrees. The evo data is shot by a rolling shutter camera, and the euroc data is shot by a global camera.

It should be understood that although the steps in the flowcharts of fig. 1 and 2 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1 and 2 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the other steps or stages.

In combination with the content of the foregoing embodiments, in one embodiment, as shown in fig. 6, there is provided a photographing system calibration apparatus including: a determination module 601 and a calibration module 602, wherein:

a determining module 601, configured to determine a reference value of the time delay according to a rotation error between the vision system and the inertial measurement unit;

a calibration module 602, configured to calibrate a time delay and a rotation parameter between a vision system and an inertial measurement unit according to a reference value of the time delay, where the vision system and the inertial measurement unit are coupled in the same shooting system, a rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used to represent a degree of inconsistency between a rotation state determined based on the vision system and a rotation state determined based on the inertial measurement unit.

In an embodiment, the determining module 601 is configured to update a first initial value of the time delay to obtain a first updated value of the time delay, determine a rotation error between the vision system and the inertial measurement unit according to the first updated value of the time delay, repeat the updating and determining processes to determine a rotation error meeting a first preset condition, and use a first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

In one embodiment, the determining module 601 includes:

the updating unit is used for updating the first initial value according to the preset step length until the updated first updated value is not positioned in the first time delay interval, and the first initial value is positioned in the first time delay interval;

the first determining unit is used for determining a rotation error corresponding to a first updating value obtained by updating each time;

and the second determining unit is used for determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

In one embodiment, the first determination unit includes:

the adjusting subunit is used for adjusting and calibrating the time point corresponding to each frame of image in the video according to the first update value obtained by each update;

the acquisition subunit is used for acquiring a first rotation matrix between two adjacent frames of images in the calibration video and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images;

and the determining subunit is used for determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images.

In one embodiment, the determining subunit is configured to calculate an axial angle difference value corresponding to each two adjacent frames of images according to a first rotation matrix between each two adjacent frames of images and a second rotation matrix corresponding to each two adjacent frames of images; and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

In one embodiment, the calibration module 602 includes:

a third determining unit, configured to determine a second time delay interval according to the reference value of the time delay, and determine a second initial value of the time delay in the second time delay interval;

and the acquisition unit is used for updating the second initial value of the time delay to obtain a second updated value of the time delay, acquiring a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay corresponding to the rotation error meeting the second preset condition and the reference value of the rotation error as the calibration result of the shooting system.

In an embodiment, the obtaining unit is configured to determine a hand-eye calibration equation set according to the second update value of the time delay, where a rotation external parameter in the hand-eye calibration equation set is an unknown quantity; and solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter.

For the specific definition of the photographing system calibration apparatus, reference may be made to the above definition of the photographing system calibration method, which is not described herein again. The modules in the calibration apparatus for a photographing system may be implemented in whole or in part by software, hardware, or a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing the time delay and the rotating external parameters. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a camera system calibration method.

Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

In one embodiment, the processor, when executing the computer program, further performs the steps of:

updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the first updated value of the time delay, repeating the updating and determining processes to determine the rotation error meeting a first preset condition, and taking the first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

In one embodiment, the processor, when executing the computer program, further performs the steps of: updating the first initial value according to a preset step length until the updated first updated value is not located in a first time delay interval, wherein the first initial value is located in the first time delay interval; determining a rotation error corresponding to a first updating value obtained by each updating; and determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

In one embodiment, the processor, when executing the computer program, further performs the steps of: adjusting and calibrating a time point corresponding to each frame of image in the video according to a first updating value obtained by updating each time; acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images; and determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images.

In one embodiment, the processor, when executing the computer program, further performs the steps of: calculating the corresponding shaft angle difference value of each two adjacent frames of images according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images; and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining a second time delay interval according to the reference value of the time delay, and determining a second initial value of the time delay in the second time delay interval; updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system.

In one embodiment, the processor, when executing the computer program, further performs the steps of: determining a hand-eye calibration equation set according to the second updating value of the time delay, wherein the external rotation parameter in the hand-eye calibration equation set is an unknown quantity; and solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

determining a reference value of the time delay according to a rotation error between the vision system and the inertia measurement unit;

calibrating the time delay and the rotation external parameter between the visual system and the inertial measurement unit according to the reference value of the time delay, wherein the visual system and the inertial measurement unit are coupled in the same shooting system, the rotation error is determined based on a calibration video, the calibration video is obtained by shooting the shooting system under rotation, and the rotation error is used for representing the degree of inconsistency between the rotation state determined based on the visual system and the rotation state determined based on the inertial measurement unit.

In one embodiment, the computer program when executed by the processor further performs the steps of: updating the first initial value of the time delay to obtain a first updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the first updated value of the time delay, repeating the updating and determining processes to determine the rotation error meeting a first preset condition, and taking the first updated value corresponding to the rotation error meeting the first preset condition as a reference value of the time delay.

In one embodiment, the computer program when executed by the processor further performs the steps of: updating the first initial value according to a preset step length until the updated first updated value is not located in a first time delay interval, wherein the first initial value is located in the first time delay interval; determining a rotation error corresponding to a first updating value obtained by each updating; and determining the minimum value of all the determined rotation errors, and taking the minimum value as the rotation error meeting the first preset condition.

In one embodiment, the computer program when executed by the processor further performs the steps of: adjusting and calibrating a time point corresponding to each frame of image in the video according to a first updating value obtained by updating each time; acquiring a first rotation matrix between two adjacent frames of images in the calibration video, and acquiring a second rotation matrix corresponding to the two adjacent frames of images in the calibration video after the time point is adjusted, wherein the second rotation matrix is determined based on the measurement value of the inertial measurement unit in the time period corresponding to the two adjacent frames of images; and determining a rotation error corresponding to the first updating value according to the first rotation matrix between every two adjacent frames of images and the second rotation matrix corresponding to every two adjacent frames of images.

In one embodiment, the computer program when executed by the processor further performs the steps of: calculating the corresponding shaft angle difference value of each two adjacent frames of images according to the first rotation matrix between each two adjacent frames of images and the second rotation matrix corresponding to each two adjacent frames of images; and calculating the rotation error corresponding to the first updating value according to the corresponding shaft angle difference of every two adjacent frames of images.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining a second time delay interval according to the reference value of the time delay, and determining a second initial value of the time delay in the second time delay interval; updating the second initial value of the time delay to obtain a second updated value of the time delay, obtaining a reference value of the rotation external parameter between the visual system and the inertial measurement unit according to the second updated value of the time delay, determining a rotation error between the visual system and the inertial measurement unit according to the second updated value of the time delay and the reference value of the rotation external parameter, repeating the processes of updating the time delay and determining the rotation external parameter and the rotation error corresponding to the updated time delay to determine the rotation error meeting a second preset condition, and taking the second updated value of the time delay and the reference value of the rotation error corresponding to the rotation error meeting the second preset condition as a calibration result of the shooting system.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining a hand-eye calibration equation set according to the second updating value of the time delay, wherein the external rotation parameter in the hand-eye calibration equation set is an unknown quantity; and solving the hand-eye calibration equation set to obtain a reference value of the rotating external parameter.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.