阅读说明：本技术 仿生眼多通道imu与相机硬件时间同步方法和装置 (Bionic eye multi-channel IMU and camera hardware time synchronization method and device ) 是由 黄强 陈晓鹏 苟思远 华承昊 孟非 高峻峣 于 2019-08-21 设计创作，主要内容包括：本发明公开了一种仿生眼多通道IMU与相机硬件时间同步方法和装置。所述方法包括：对IMU进行设置,利用IMU的输出频率作为硬件时间同步与对齐的周期基准,并使其在输出数据时同步输出脉冲信号；对相机进行设置,设定其图像采集的相关属性、固定曝光时长,并使其在接受外部触发脉冲后进行图像采集,利用来自IMU的输出脉冲信号产生对相机的触发脉冲信号；硬件同步单元在捕捉到IMU输出脉冲信号的同时将其按固定频率分频,并产生触发脉冲信号对所有通道的相机进行同步触发。本发明实现精准硬件同步与对齐,可以更快速、更精确的恢复出仿生眼系统所感知环境的深度信息及仿生眼系统运动时的实时三维位姿信息,值得推广。(The invention discloses a method and a device for time synchronization of a bionic eye multichannel IMU and camera hardware. The method comprises the following steps: setting an IMU, using the output frequency of the IMU as a periodic reference for hardware time synchronization and alignment, and synchronously outputting pulse signals when outputting data; setting a camera, setting relevant attributes and fixed exposure time of image acquisition of the camera, carrying out image acquisition after receiving an external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from an IMU (inertial measurement Unit); the hardware synchronization unit captures the IMU output pulse signal, divides the frequency of the IMU output pulse signal according to a fixed frequency, and generates a trigger pulse signal to synchronously trigger the cameras of all channels. The method and the device realize accurate hardware synchronization and alignment, can more quickly and accurately recover the depth information of the environment sensed by the bionic eye system and the real-time three-dimensional pose information of the bionic eye system during movement, and are worthy of popularization.)

1. The time synchronization method of the bionic eye multichannel IMU and the camera hardware is characterized by comprising the following steps:

s1, setting the IMU, using the output frequency of the IMU as the cycle reference of hardware time synchronization and alignment, and synchronously outputting pulse signals when outputting data;

S2, setting the camera, setting the relevant attribute of image acquisition and fixed exposure time, carrying out image acquisition after receiving external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

S3, capturing a data synchronization output pulse signal of the IMU by the hardware synchronization unit, marking a time stamp, a channel number and a serial number of the IMU data, dividing the frequency of the IMU output pulse signal by the hardware synchronization unit according to a fixed frequency while capturing the IMU output pulse signal, generating a trigger pulse signal to synchronously trigger the cameras of all channels, and recording the trigger time stamp and the serial number of the cameras;

S4, performing time synchronization of each channel sensor by using a hardware synchronization unit, uniformly managing the measurement data, the time stamp and the serial number of each channel, putting the sensor data, the time stamp and the serial number of the same channel in the same period in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

S5, receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, analyzing the data packets, generating time stamps and serial numbers of IMU data packets and trigger pulse signals, and calculating synchronous time compensation of IMUs of other channels by taking a main channel IMU as a sampling time synchronous reference;

S6, calculating by using the time stamp and the serial number of the trigger pulse signal and combining with fixed exposure time to obtain the real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

And S7, issuing the hardware time synchronization and the aligned IMU data packet and camera data packet according to channel classification to wait for the calling of other functional programs of the bionic eye.

2. the method of claim 1, wherein the step S1 of setting the IMU includes initializing, sending a configuration command through the communication interface, determining to output specific data according to the functional requirements of the bionic eye, and enabling to output pulse signals synchronously.

3. the method for time synchronizing the multi-channel IMU and the hardware of the camera according to claim 1, wherein the setting of the camera in S2 comprises enabling external trigger function, setting exposure, gain, and image property.

4. the method of claim 1, wherein in S3, the hardware synchronization unit comprises a microprocessor for storing, managing and processing data, time stamps and serial numbers, and a crystal oscillator for system time synchronization, and the hardware synchronization unit receives the IMU synchronization pulse as an interrupt signal and frequency-divides the output trigger pulse signal.

5. The method for time synchronizing the multi-channel IMU and the camera according to claim 1, wherein in the step S5, the embedded upper computer comprises an interface for receiving the data of the hardware synchronization unit and the camera data, a logic calculation unit for calculating and publishing the multi-channel data, and a memory.

6. a bionic eye multi-channel IMU and camera hardware time synchronization device is characterized by comprising:

The first setting module is used for setting the IMU, using the output frequency of the IMU as a cycle reference for hardware time synchronization and alignment, and enabling the IMU to synchronously output pulse signals when outputting data;

The second setting module is used for setting the camera, setting the relevant attributes of image acquisition and fixed exposure duration of the camera, acquiring the image after receiving an external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

the hardware synchronization unit is used for capturing the IMU output pulse signals, dividing the frequency of the IMU output pulse signals according to a fixed frequency, generating trigger pulse signals to synchronously trigger cameras of all channels, and recording the trigger time stamps and the serial numbers of the cameras;

The second processing module is used for carrying out time synchronization on the sensors of each channel by using the hardware synchronization unit, uniformly managing the measurement data, the time stamps and the serial numbers of each channel, putting the sensor data, the time stamps and the serial numbers of the same channel and the same period together in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

The third processing module is used for receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, generating time stamps and serial numbers of IMU data packets and trigger pulse signals after analyzing the data packets, and calculating synchronous time compensation of IMUs of other channels by taking the main channel IMU as a sampling time synchronous reference;

The fourth processing module is used for calculating by utilizing the time stamp and the serial number of the trigger pulse signal and combining the fixed exposure duration to obtain a real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

And the fifth processing module is used for issuing the hardware time synchronization and the aligned IMU data packet and the camera data packet according to the channel classification so as to wait for the calling of other functional programs of the bionic eye.

7. the biomimetic eye multi-channel IMU and camera hardware time synchronization apparatus of claim 6, wherein the first setting module is further configured to:

The setting of the IMU comprises initialization, sending a configuration command through a communication interface, determining and outputting specific data according to the function requirement of the bionic eye, and enabling to synchronously output pulse signals.

8. The biomimetic eye multi-channel IMU and camera hardware time synchronization apparatus of claim 6, wherein the second setting module is further configured to:

the settings for the camera include enabling external trigger functions, setting exposure, gain, image attributes.

9. the device as claimed in claim 6, wherein the hardware synchronization unit comprises a microprocessor for storing, managing and processing data, time stamp and serial number, and a crystal oscillator for system time synchronization, and the hardware synchronization unit receives IMU synchronization pulse as interrupt signal and frequency-divides output trigger pulse signal.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.

Technical Field

The invention relates to the technical field of multi-sensor fusion sensing, in particular to a method and a device for synchronizing time of a bionic eye multi-channel IMU (inertial measurement Unit) and camera hardware.

background

To achieve accurate perception of the environment and modeling of the environment map for positional navigation, it is necessary to fuse the measurement data of multiple/diverse sensors. The premise of data fusion of multiple sensors is synchronous acquisition of data of each sensor, so that time synchronization of data acquisition of each sensor is required.

The existing multi-sensor time synchronization method comprises software time synchronization and hardware time synchronization. The software time synchronization mode is limited by the precision of a synchronous clock of a computing carrier, and an operating system is often required to meet the requirement of hard real-time, which brings greater difficulty to program development and function realization in practical application.

The hardware time synchronization method generally adopts a GNSS (satellite navigation system) as a reference for time synchronization, or measures time difference data between different sensors, and then calibrates and compensates the time difference. The GNSS time synchronization method has the disadvantages of high cost (each node needs to be provided with GNSS equipment), limited installation (the node needs to be used in an outdoor non-sheltered environment), susceptibility to weather and environmental change, and the like. Moreover, GNSS receivers generally have a large size and are difficult to apply to a bionic eye system having a compact structure.

disclosure of Invention

In view of the defects in the prior art, the method and the device for synchronizing the time of the multi-channel IMU of the bionic eye and the hardware of the camera are provided for solving the problem of synchronous acquisition of data of a plurality of IMUs and a plurality of cameras in the bionic eye system.

in a first aspect, the application provides a method for time synchronization between a multi-channel IMU of a bionic eye and camera hardware, comprising the following steps:

S1, setting the IMU, using the output frequency of the IMU as the cycle reference of hardware time synchronization and alignment, and synchronously outputting pulse signals when outputting data;

s2, setting the camera, setting the relevant attribute of image acquisition and fixed exposure time, carrying out image acquisition after receiving external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

S3, capturing a data synchronization output pulse signal of the IMU by the hardware synchronization unit, marking a time stamp, a channel number and a serial number of the IMU data, dividing the frequency of the IMU output pulse signal by the hardware synchronization unit according to a fixed frequency while capturing the IMU output pulse signal, generating a trigger pulse signal to synchronously trigger the cameras of all channels, and recording the trigger time stamp and the serial number of the cameras;

S4, performing time synchronization of each channel sensor by using a hardware synchronization unit, uniformly managing the measurement data, the time stamp and the serial number of each channel, putting the sensor data, the time stamp and the serial number of the same channel in the same period in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

s5, receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, analyzing the data packets, generating time stamps and serial numbers of IMU data packets and trigger pulse signals, and calculating synchronous time compensation of IMUs of other channels by taking a main channel IMU as a sampling time synchronous reference;

S6, calculating by using the time stamp and the serial number of the trigger pulse signal and combining with fixed exposure time to obtain the real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

and S7, issuing the hardware time synchronization and the aligned IMU data packet and camera data packet according to channel classification to wait for the calling of other functional programs of the bionic eye.

optionally, in S1, the setting of the IMU includes initializing, sending a configuration command through the communication interface, determining to output specific data according to a functional requirement of the bionic eye, and enabling to output a pulse signal synchronously.

Optionally, in S2, the setting of the camera includes enabling an external trigger function, setting exposure, gain, and image attribute.

Optionally, in S3, the hardware synchronization unit includes a microprocessor for storing, managing and processing data, a timestamp and a serial number, and a crystal oscillator for performing system time synchronization, and the hardware synchronization unit receives the IMU synchronization pulse as an interrupt signal and frequency-divides and outputs a trigger pulse signal.

Optionally, in S5, the embedded upper computer includes an interface for receiving data of the hardware synchronization unit, receiving camera data, a logic calculation unit for calculating and publishing multi-channel data, and a memory.

In a second aspect, the present application further provides a device for synchronizing time between a multi-channel IMU of a bionic eye and camera hardware, comprising:

the first setting module is used for setting the IMU, using the output frequency of the IMU as a cycle reference for hardware time synchronization and alignment, and enabling the IMU to synchronously output pulse signals when outputting data;

The second setting module is used for setting the camera, setting the relevant attributes of image acquisition and fixed exposure duration of the camera, acquiring the image after receiving an external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

the hardware synchronization unit is used for capturing the IMU output pulse signals, dividing the frequency of the IMU output pulse signals according to a fixed frequency, generating trigger pulse signals to synchronously trigger cameras of all channels, and recording the trigger time stamps and the serial numbers of the cameras;

The second processing module is used for carrying out time synchronization on the sensors of each channel by using the hardware synchronization unit, uniformly managing the measurement data, the time stamps and the serial numbers of each channel, putting the sensor data, the time stamps and the serial numbers of the same channel and the same period together in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

the third processing module is used for receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, generating time stamps and serial numbers of IMU data packets and trigger pulse signals after analyzing the data packets, and calculating synchronous time compensation of IMUs of other channels by taking the main channel IMU as a sampling time synchronous reference;

The fourth processing module is used for calculating by utilizing the time stamp and the serial number of the trigger pulse signal and combining the fixed exposure duration to obtain a real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

and the fifth processing module is used for issuing the hardware time synchronization and the aligned IMU data packet and the camera data packet according to the channel classification so as to wait for the calling of other functional programs of the bionic eye.

Optionally, the first setting module is further configured to:

the setting of the IMU comprises initialization, sending a configuration command through a communication interface, determining and outputting specific data according to the function requirement of the bionic eye, and enabling to synchronously output pulse signals.

Optionally, the second setting module is further configured to:

the settings for the camera include enabling external trigger functions, setting exposure, gain, image attributes.

optionally, the hardware synchronization unit includes a microprocessor for storing, managing and processing data, a timestamp and a serial number, and a crystal oscillator for performing system time synchronization, and the hardware synchronization unit receives an IMU synchronization pulse as an interrupt signal and frequency-divides and outputs a trigger pulse signal.

in a third aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the above-mentioned method for time synchronizing the multi-channel IMU of the bionic eye and the hardware of the camera.

compared with the prior art, the invention has the beneficial effects that:

1. Hardware time synchronization for bionic eye multi-channel IMU and camera data acquisition is achieved, compared with single-channel IMU and camera data acquisition and fusion, the multi-channel IMU and camera data acquisition and fusion can restore depth information of a sensed environment more quickly and accurately, and meanwhile each channel IMU and camera can perform real-time pose measurement on independent movement of each eye.

2. For the time difference between the IMUs of different channels, compensation is performed by combining time difference measurement and Bayesian estimation, and the time synchronization precision between the IMUs of different channels is further improved on the basis of consistent IMU sampling frequency.

3. the acquisition and processing of sensor data such as an IMU (inertial measurement Unit) and a camera are separated in a hardware synchronization unit and an embedded upper computer, so that the problem of time synchronization errors caused by inconsistent clock oscillators between the two devices is solved, and the advantage of rapidly processing a large amount of acquired data by the embedded upper computer is combined.

4. The accurate correspondence between the multichannel IMU data and the image frame data is realized through a hardware synchronization unit and corresponding software.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:

Fig. 1 is a block diagram illustrating a method for time synchronization between a multi-channel IMU of a bionic eye and camera hardware according to an embodiment of the present disclosure;

FIG. 2 is a flowchart of a hardware synchronization unit according to an embodiment of the present disclosure;

fig. 3 is a flowchart of a program of an embedded upper computer according to an embodiment of the present application;

Fig. 4 is a timing diagram illustrating a method for time synchronization between a multi-channel IMU and camera hardware according to an embodiment of the present disclosure.

Detailed Description

in order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

it should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Referring to fig. 1-4, the method for time synchronization of a multi-channel IMU (inertial measurement unit) of a bionic eye and camera hardware provided by the invention comprises the following steps:

S1, setting the IMU, using the output frequency of the IMU as the cycle reference of hardware time synchronization and alignment, and synchronously outputting pulse signals when outputting data;

S2, setting the camera, setting the relevant attribute of image acquisition and fixed exposure time, carrying out image acquisition after receiving external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

S3, capturing a data synchronization output pulse signal of the IMU by the hardware synchronization unit, marking a time stamp, a channel number and a serial number of the IMU data, dividing the frequency of the IMU output pulse signal by the hardware synchronization unit according to a fixed frequency while capturing the IMU output pulse signal, generating a trigger pulse signal to synchronously trigger the cameras of all channels, and recording the trigger time stamp and the serial number of the cameras;

s4, performing time synchronization of each channel sensor by using a hardware synchronization unit, uniformly managing the measurement data, the time stamp and the serial number of each channel, putting the sensor data, the time stamp and the serial number of the same channel in the same period in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

s5, receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, analyzing the data packets, generating time stamps and serial numbers of IMU data packets and trigger pulse signals, and calculating synchronous time compensation of IMUs of other channels by taking a main channel IMU as a sampling time synchronous reference;

S6, calculating by using the time stamp and the serial number of the trigger pulse signal and combining with fixed exposure time to obtain the real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

And S7, issuing the hardware time synchronization and the aligned IMU data packet and camera data packet according to channel classification to wait for the calling of other functional programs of the bionic eye.

The setting of the IMU comprises initialization, sending a configuration command through a communication interface, determining and outputting specific data according to the function requirement of the bionic eye, and enabling to synchronously output pulse signals. The settings for the camera include enabling external trigger functions, setting exposure, gain, image attributes. The hardware synchronization unit comprises a microprocessor used for storing, managing and processing data, timestamps and serial numbers and a crystal oscillator used for carrying out system time synchronization, and receives IMU synchronization pulses as interrupt signals and outputs trigger pulse signals in a frequency division mode. The embedded upper computer comprises an interface for receiving the data of the hardware synchronization unit and the camera data, a logic calculation unit for calculating and distributing multi-channel data, and a memory.

In this embodiment, the selected IMU is a high-precision micro sensor, the available communication interfaces include SPI, I2C, and UART, the IMU can enable output of a pulse signal while data acquisition, and the data output frequency is selectable.

The method selects a global shutter CCD industrial camera, an image data transmission interface is a USB interface, an image acquisition mode and image acquisition parameters are selectable, and an external trigger interface is provided.

a processing chip with a 16MHz clock crystal oscillator is selected as a microprocessor of a hardware synchronization unit, an external interrupt pin can be enabled, a TTL signal can be output, a plurality of UART interfaces are used for receiving and sending data, and an interface conversion chip and a USB interface are arranged.

an embedded upper computer is selected as a logic calculation and storage unit of the control system, available communication interfaces comprise USB2.0, USB 3.0, RS232, UART, CAN and the like, and the embedded upper computer CAN store and rapidly process data and programs and is a control platform for realizing multiple functions by the bionic eye.

The IMU is connected with the hardware synchronization unit through a high baud rate UART interface, the hardware synchronization unit enables an external interrupt pin, IMU data synchronization output pulse signals are used as interrupt sources, frequency division of the interrupt sources is converted into TTL trigger pulse signals, the camera external trigger interface is connected with the hardware synchronization unit, the TTL trigger pulse signals are waited to be received, a 16MHz crystal oscillator is used as a clock crystal oscillator of a hardware synchronization unit system timer, and the hardware synchronization unit and the camera are respectively connected with the embedded upper computer through corresponding data transmission interfaces. The block diagram of the system is shown in fig. 1.

the specific scheme of the embodiment is as follows:

the IMU is set through the hardware synchronization unit, an IMU configuration instruction is sent to the IMU through the UART interface, pulse signals can be synchronously output through data, the sampling output frequency, the UART baud rate, the output data structure and the like of the IMU are set, in the embodiment, the sampling output frequency is 200Hz, the UART baud rate is 115200, and the output data comprise IMU measurement data, a timestamp, a serial number, verification information and the like.

the camera is initialized through the control system, an external trigger function is enabled, the exposure time of the camera is set to be fixed time, the exposure time is set, the image acquisition time of the camera can be always aligned with the image acquisition time of a certain frame of the IMU, the image gain, the image and other attributes are set, and the image data is ensured to meet the requirements of the bionic eye program calling.

After IMU and camera setting are accomplished, hardware synchronization unit empties the buffer zone and initializes, enable outside interrupt function, open the system timer simultaneously, get into outside interrupt program after receiving the synchronous pulse signal from IMU, the timestamp of mark IMU data, channel number and sequence number, simultaneously according to the frequency division output camera trigger pulse signal of setting for, carry out synchronous trigger to the camera of all channels, the camera begins to carry out image acquisition after receiving the trigger signal pulse, get into the exposure stage, the timestamp and the sequence number of mark trigger pulse signal are as timestamp and the sequence number that the camera triggered, read buffer IMU data after withdrawing outside interrupt program, empty buffer zone data after the transfer of subchannel, ensure that next cycle data is IMU's new measured data.

The hardware synchronization unit processes the data, the time stamp and the serial number in the same period and the same channel to generate a data packet with a new data structure, and then transmits the data packet to an embedded upper computer of the control system through a data transmission channel. The embedded upper computer firstly checks the data packet after receiving the data packet, and then divides the data packet to perform CRC (cyclic redundancy check) of IMU (inertial measurement unit) data so as to effectively solve the problems of data errors and the like in the high-frequency transmission process. And after the check is finished, data analysis is carried out, IMU data and corresponding timestamps are sorted and stored according to channel numbers, IMU data timestamp calculation, IMU time synchronization compensation calculation, data format conversion and the like among different channels are carried out, IMU data packets are generated after hardware synchronization alignment and are issued to wait for calling of other functional programs. A flow chart of the hardware synchronization unit procedure is shown in fig. 2.

And the embedded upper computer simultaneously receives data such as images obtained by triggering the camera in the period, analyzes the data to obtain a camera triggering timestamp, calculates the timestamp of the camera image, generates a camera data packet together with the camera image data, and issues the camera data packet to wait for calling of other functional programs. A flowchart of the embedded upper computer program is shown in fig. 3.

system timing diagram As shown in FIG. 4, time stamp t of IMU_imuAnd time stamp t of camera image_camcan be calculated according to the following formula:

t_imu＝t_sample+T_d

t_cam＝t_trigger+T_w+T_e

wherein, t_sampleIs the IMU data sampling time, T_dthe pulse width duration of the IMU synchronous output pulse signal is a fixed duration; t is t_triggeris the camera trigger time, T_wis the pulse width duration, T, of the trigger pulse signal of the camera_eis the exposure duration, T, of the camera image acquisition_wAnd T_eis a fixed value, which is a set camera-related attribute.

And performing synchronous time compensation on the multi-channel IMU on the premise that the sampling frequencies of all the channels IMUs are consistent and the channels IMUs are synchronously electrified. Because the sampling time error of the IMU is subject to the characteristic of normal distribution expected to be 0 under the fixed frequency, the sampling time of the IMU of the main channel is taken as the sampling time synchronization reference, and the Bayesian estimation is carried out on the sampling time of the IMU of other channels, thereby improving the time synchronization precision of the IMUs of different channels.

The IMU sampling time of the main channel is known as T₁With standard deviation of error of sampling time of σ₁The IMU sampling time and the sampling time error standard deviation of other channels are sequentially (T)₁，σ₁)，(T₂，σ₂)，……，(T_n，σ_n) The sample time compensation algorithm is as follows:

……

the scheme of the invention can ensure that the time synchronization precision of the sensor data between the multichannel IMU and the camera is within 0.1 ms.

based on the same technical conception, the application also provides a bionic eye multi-channel IMU and camera hardware time synchronization device, which is characterized by comprising:

The first setting module is used for setting the IMU, using the output frequency of the IMU as a cycle reference for hardware time synchronization and alignment, and enabling the IMU to synchronously output pulse signals when outputting data;

The second setting module is used for setting the camera, setting the relevant attributes of image acquisition and fixed exposure duration of the camera, acquiring the image after receiving an external trigger pulse, and generating a trigger pulse signal for the camera by using an output pulse signal from the IMU;

The hardware synchronization unit is used for capturing the IMU output pulse signals, dividing the frequency of the IMU output pulse signals according to a fixed frequency, generating trigger pulse signals to synchronously trigger cameras of all channels, and recording the trigger time stamps and the serial numbers of the cameras;

The second processing module is used for carrying out time synchronization on the sensors of each channel by using the hardware synchronization unit, uniformly managing the measurement data, the time stamps and the serial numbers of each channel, putting the sensor data, the time stamps and the serial numbers of the same channel and the same period together in a queue mode to generate a new data packet, and sending the new data packet to an embedded upper computer of the bionic eye control system;

The third processing module is used for receiving and checking a new data packet by using an embedded upper computer of the bionic eye control system, reading image data streams acquired by cameras of all channels, generating time stamps and serial numbers of IMU data packets and trigger pulse signals after analyzing the data packets, and calculating synchronous time compensation of IMUs of other channels by taking the main channel IMU as a sampling time synchronous reference;

The fourth processing module is used for calculating by utilizing the time stamp and the serial number of the trigger pulse signal and combining the fixed exposure duration to obtain a real time stamp of image output, and packaging the real time stamp and the image data into a camera data packet;

and the fifth processing module is used for issuing the hardware time synchronization and the aligned IMU data packet and the camera data packet according to the channel classification so as to wait for the calling of other functional programs of the bionic eye.

optionally, the first setting module is further configured to:

The setting of the IMU comprises initialization, sending a configuration command through a communication interface, determining and outputting specific data according to the function requirement of the bionic eye, and enabling to synchronously output pulse signals.

Optionally, the second setting module is further configured to:

The settings for the camera include enabling external trigger functions, setting exposure, gain, image attributes.

Optionally, the hardware synchronization unit includes a microprocessor for storing, managing and processing data, a timestamp and a serial number, and a crystal oscillator for performing system time synchronization, and the hardware synchronization unit receives an IMU synchronization pulse as an interrupt signal and frequency-divides and outputs a trigger pulse signal.

based on the same technical concept, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for time synchronization between the multi-channel IMU of the bionic eye and the hardware of the camera is performed.

computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data.

examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.

the above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.