阅读说明：本技术 用于车辆的时间同步系统和方法 (Time synchronization system and method for vehicle ) 是由 王�华 王丰雷 闫振强 于 2019-03-29 设计创作，主要内容包括：本申请实施例公开了用于车辆的时间同步系统和方法。所述车辆包括卫星导航设备、至少一个激光雷达(LiDAR)设备、至少一个相机和控制器。所述时间同步方法包括：所述卫星导航设备发送协调世界时(UTC)和秒脉冲(PPS)信号到所述控制器和所述至少一个激光雷达设备,所述卫星导航设备包括第一时钟；基于所述秒脉冲信号和所述协调世界时,通过所述控制器将所述控制器的第二时钟与所述卫星导航设备的所述第一时钟同步；基于所述秒脉冲信号和所述协调世界时,通过所述至少一个激光雷达设备将所述至少一个激光雷达设备的第三时钟与所述卫星导航设备的所述第一时钟同步；以及所述至少一个相机将所述至少一个相机的第四时钟与所述控制器的所述第二时钟同步。(The embodiment of the application discloses a time synchronization system and a time synchronization method for a vehicle. The vehicle includes satellite navigation equipment, at least one LiDAR (LiDAR) device, at least one camera, and a controller. The time synchronization method comprises the following steps: the satellite navigation device transmitting coordinated Universal Time (UTC) and Pulse Per Second (PPS) signals to the controller and the at least one lidar device, the satellite navigation device including a first clock; synchronizing, by the controller, a second clock of the controller with the first clock of the satellite navigation device based on the pulse-of-seconds signal and the coordinated universal time; synchronizing, by the at least one lidar device, a third clock of the at least one lidar device with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time; and the at least one camera synchronizing a fourth clock of the at least one camera with the second clock of the controller.)

1. A method of time synchronization for a vehicle comprising a satellite navigation device, at least one lidar device, at least one camera, and a controller, the method comprising:

the satellite navigation device sends a coordinated universal time and second pulse signal to the controller and the at least one laser radar device, wherein the second pulse signal contains information of a whole second of the coordinated universal time, and the satellite navigation device comprises a first clock;

based on the pulse-per-second signal and the coordinated universal time, the controller synchronizes a second clock of the controller with the first clock of the satellite navigation device;

based on the pulse-per-second signal and the coordinated universal time, the at least one lidar device synchronizing a third clock of the at least one lidar device with the first clock of the satellite navigation device; and

the at least one camera synchronizes a fourth clock of the at least one camera with the second clock of the controller.

2. The method of claim 1, wherein the at least one camera synchronizing the fourth clock of the at least one camera with the second clock of the controller comprises:

the at least one camera sending an initial data packet to the controller, the initial data packet including a first timestamp based on the fourth clock of the at least one camera, the first timestamp indicating a time at which the at least one camera sent the initial data packet;

after receiving the initial data packet, the controller sending a modified data packet to the at least one camera, the modified data packet including a second timestamp and a third timestamp based on the second clock, the second timestamp indicating a time at which the controller received the initial data message, the third timestamp indicating a time at which the controller sent the modified data packet;

after receiving the modified data packet, the at least one camera recording a fourth timestamp based on the fourth clock of the at least one camera, the fourth timestamp indicating a time at which the at least one camera received the modified data packet from the controller;

based on the first, second, third, and fourth timestamps, the at least one camera determines a deviation between the second clock and the fourth clock; and

based on the deviation, the at least one camera synchronizes the fourth clock with the second clock.

3. The method of claim 2, wherein the at least one camera determining the offset between the second clock and the fourth clock based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp comprises:

based on the first, second, third, and fourth timestamps, the at least one camera determines a one-way transmission time of the initial or modified data packet between the controller and the at least one camera; and

based on the transmission time, the at least one camera determines the deviation.

4. The method of claim 1, further comprising:

the satellite navigation apparatus receiving navigation signals from one or more satellites; and

based on the navigation signal, the satellite navigation device determines the pulse-per-second signal and the coordinated universal time.

5. The method of claim 1, wherein the pulse-per-second signal and the coordinated universal time are sent to the controller or the at least one lidar apparatus simultaneously.

6. The method of claim 1, wherein the universal time coordinated is sent to the controller or the at least one lidar device after a period of time after the pulse-per-second signal is sent to the controller or the at least one lidar device.

7. The method of claim 1, wherein the vehicle further comprises an inertial navigation system;

the method further comprises the following steps:

the inertial navigation system receiving the pulse-per-second signal and the coordinated universal time from the satellite navigation device; and

based on the pulse-per-second signal and the coordinated universal time, the inertial navigation system synchronizes a fifth clock of the inertial navigation system with the first clock of the satellite navigation device.

8. The method of claim 1, wherein the PPS signal is amplified or photo-electrically isolated when the PPS signal is sent to the controller or the at least one lidar.

9. The method according to any one of claims 1-8, wherein the PPS signal is processed by masking high frequency noise in the PPS signal when the PPS signal is sent to the controller or the at least one lidar.

10. The method of claim 9, wherein the high frequency noise in the pulse per second signal is shielded using a magnetic loop.

11. A time synchronization system for a vehicle, comprising:

a satellite navigation device comprising a first clock, the satellite navigation device configured to transmit a pulse-per-second signal and a coordinated universal time, the pulse-per-second signal comprising whole second information of the coordinated universal time;

a controller configured to:

receiving the pulse-of-seconds signal and the coordinated universal time from the satellite navigation device; and

synchronizing a second clock of the controller with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time;

at least one lidar device configured to:

receiving the pulse-of-seconds signal and the coordinated universal time from the satellite navigation device; and

synchronizing a third clock of the at least one lidar device with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time; and

at least one camera configured to synchronize a fourth clock of the at least one camera with the second clock of the controller.

12. The system of claim 11, wherein the at least one camera synchronizing the fourth clock of the at least one camera with the second clock of the controller comprises:

the at least one camera sending an initial data packet to the controller, the initial data packet including a first timestamp based on the fourth clock of the at least one camera, the first timestamp indicating a time at which the at least one camera sent the initial data packet;

after receiving the initial data packet, the controller sending a modified data packet to the at least one camera, the modified data packet including a second timestamp and a third timestamp based on the second clock, the second timestamp indicating a time at which the controller received the initial data message, the third timestamp indicating a time at which the controller sent the modified data packet;

after receiving the modified data packet, the at least one camera recording a fourth timestamp based on the fourth clock of the at least one camera, the fourth timestamp indicating a time at which the at least one camera received the modified data packet from the controller;

based on the first, second, third, and fourth timestamps, the at least one camera determines a deviation between the second clock and the fourth clock; and

based on the deviation, the at least one camera synchronizes the fourth clock with the second clock.

13. The system of claim 12, wherein the at least one camera determining the offset between the second clock and the fourth clock based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp comprises:

based on the first, second, third, and fourth timestamps, the at least one camera determines a one-way transmission time of the initial or modified data packet between the controller and the at least one camera; and

based on the transmission time, the at least one camera determines the deviation.

14. The system of claim 11, the satellite navigation device further configured to:

receiving navigation signals from one or more satellites; and

determining the pulse-of-seconds signal and the coordinated universal time based on the navigation signal.

15. The system of claim 11, wherein the pulse-per-second signal and the coordinated universal time are sent to the controller or the at least one lidar apparatus simultaneously.

16. The system of claim 11, wherein the universal time coordinated is sent to the controller or the at least one lidar apparatus after a period of time that the pulse-per-second signal is sent to the controller or the at least one lidar apparatus.

17. The system of claim 11, further comprising an inertial navigation system, the inertial navigation device configured to:

receiving the pulse-of-seconds signal and the coordinated universal time from the satellite navigation device; and

synchronizing a fifth clock of the inertial navigation system with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time.

18. The system of claim 11, wherein the pulse per second signal is amplified or optically isolated when sent to the controller or the at least one lidar.

19. The system of any of claims 11-18, wherein the PPS signal is processed by masking high frequency noise in the PPS signal when the PPS signal is sent to the controller or the at least one lidar.

20. The system as recited in claim 19, wherein the high frequency noise in the pulse per second signal is shielded using a magnetic loop.

Technical Field

The present application relates generally to time synchronization and, more particularly, to a system and method for time synchronization of multiple sensors in a vehicle.

Background

An autonomous vehicle is a vehicle that can sense its surroundings and can travel with little or no human input. Autonomous vehicles typically incorporate various sensors to sense the surrounding environment, such as radar, computer vision (e.g., cameras), LiDAR (light detection and ranging) devices, acoustic locators, devices with location technology, odometers, and Inertial Measurement Units (IMUs). The positioning, environmental perception and control of the autonomous vehicle may be achieved based on the cooperation of a plurality of sensors. Such cooperation is essential for the automatic driving technique. Time synchronization of multiple sensors is a key operation to achieve smooth collaboration among multiple sensors. Accordingly, it is desirable to provide systems and methods that can effectively and accurately synchronize clocks of multiple sensors in an autonomous vehicle.

Disclosure of Invention

One of the embodiments of the present application provides a time synchronization method for a vehicle. The vehicle including satellite navigation equipment, at least one LiDAR (LiDAR) device, at least one camera, and a controller, wherein the method comprises: the satellite navigation device transmitting coordinated Universal Time (UTC) and Pulse Per Second (PPS) signals to the controller and the at least one lidar device, the PPS signals containing whole second information of the coordinated universal time, the satellite navigation device including a first clock; based on the pulse-per-second signal and the coordinated universal time, the controller synchronizes a second clock of the controller with the first clock of the satellite navigation device; based on the pulse-per-second signal and the coordinated universal time, the at least one lidar device synchronizing a third clock of the at least one lidar device with the first clock of the satellite navigation device; and the at least one camera synchronizing a fourth clock of the at least one camera with the second clock of the controller.

In some embodiments, the at least one camera synchronizing the fourth clock of the at least one camera with the second clock of the controller comprises: the at least one camera sending an initial data packet to the controller, the initial data packet including a first timestamp based on the fourth clock of the at least one camera, the first timestamp indicating a time at which the at least one camera sent the initial data packet; after receiving the initial data packet, the controller sending a modified data packet to the at least one camera, the modified data packet including a second timestamp and a third timestamp based on the second clock, the second timestamp indicating a time at which the controller received the initial data message, the third timestamp indicating a time at which the controller sent the modified data packet; after receiving the modified data packet, the at least one camera recording a fourth timestamp based on the fourth clock of the at least one camera, the fourth timestamp indicating a time at which the at least one camera received the modified data packet from the controller; based on the first, second, third, and fourth timestamps, the at least one camera determines a deviation between the second clock and the fourth clock; and based on the deviation, the at least one camera synchronizes the fourth clock with the second clock.

In some embodiments, based on the first, second, third, and fourth timestamps, the at least one camera determining the offset between the second clock and the fourth clock comprises: based on the first, second, third, and fourth timestamps, the at least one camera determines a one-way transmission time of the initial or modified data packet between the controller and the at least one camera; and based on the transmission time, the at least one camera determines the deviation.

In some embodiments, the time synchronization method further comprises: the satellite navigation apparatus receiving navigation signals from one or more satellites; and based on the navigation signal, the satellite navigation device determines the pulse-per-second signal and the coordinated universal time.

In some embodiments, the pulse-per-second signal and the coordinated universal time are sent to the controller or the at least one lidar device simultaneously.

In some embodiments, the universal time coordinated is sent to the controller or the at least one lidar device after the second pulse signal is sent to the controller or the at least one lidar device for a period of time.

In some embodiments, the vehicle further comprises an Inertial Navigation System (INS). The method further comprises the following steps: the inertial navigation system receiving the pulse-per-second signal and the coordinated universal time from the satellite navigation device; and based on the pulse-per-second signal and the coordinated universal time, the inertial navigation system synchronizes a fifth clock of the inertial navigation system with the first clock of the satellite navigation device.

In some embodiments, the pulse per second signal is amplified or opto-electrically isolated when the pulse per second signal is sent to the controller or the at least one lidar.

In some embodiments, the pulse-per-second signal is processed by masking high frequency noise in the pulse-per-second signal when the pulse-per-second signal is sent to the controller or the at least one lidar.

In some embodiments, the high frequency noise in the pulse-per-second signal is shielded using a magnetic loop.

One of the embodiments of the present application provides a time synchronization system for a vehicle, including: a satellite navigation device comprising a first clock, the satellite navigation device configured to transmit a pulse-per-second (PPS) signal and a Universal Time Coordinated (UTC), the PPS signal comprising an entire second of information of the UTC; a controller configured to: receiving the pulse-of-seconds signal and the coordinated universal time from the satellite navigation device; and synchronizing a second clock of the controller with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time; at least one lidar device configured to: receiving the pulse-of-seconds signal and the coordinated universal time from the satellite navigation device; and synchronizing a third clock of the at least one lidar device with the first clock of the satellite navigation device based on the pulse-per-second signal and the coordinated universal time; and at least one camera configured to synchronize a fourth clock of the at least one camera with the second clock of the controller.

Additional features of the present application will be set forth in part in the description which follows. Additional features of some aspects of the present application will be apparent to those of ordinary skill in the art in view of the following description and accompanying drawings, or in view of the production or operation of the embodiments. The features of the present disclosure may be realized and attained by practice or use of the methods, instrumentalities and combinations of the various aspects of the particular embodiments described below.

Drawings

The present application will be further described by way of exemplary embodiments. These exemplary embodiments will be described in detail by means of the accompanying drawings. These embodiments are non-limiting exemplary embodiments in which like reference numerals represent similar structures throughout the several views of the drawings, and wherein:

FIG. 1 is a schematic diagram of an exemplary time synchronization system shown in accordance with some embodiments of the present application;

FIG. 2 is a schematic diagram of exemplary hardware and/or software components of a computing device, shown in accordance with some embodiments of the present application;

FIG. 3 is a schematic diagram of exemplary hardware and/or software components of a mobile device shown in accordance with some embodiments of the present application;

FIG. 4 is a flow diagram of an exemplary process for time synchronization shown in accordance with some embodiments of the present application; and

fig. 5 is a flow diagram of an exemplary process for synchronizing a clock of at least one camera with a clock of a controller, shown in accordance with some embodiments of the present application.

Detailed Description

The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a particular application and its requirements. It will be apparent to those skilled in the art that various modifications to the disclosed embodiments are possible, and that the general principles defined in this application may be applied to other embodiments and applications without departing from the spirit and scope of the application. Thus, the present application is not limited to the described embodiments, but should be accorded the widest scope consistent with the claims.

The terminology used in the description presented herein is for the purpose of describing particular example embodiments only and is not intended to limit the scope of the present application. As used herein, the singular forms "a", "an" and "the" may include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, components, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, and/or groups thereof.

These and other features, aspects, and advantages of the present application, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description of the accompanying drawings, all of which form a part of this specification. It is to be understood, however, that the drawings are designed solely for the purposes of illustration and description and are not intended as a definition of the limits of the application. It should be understood that the drawings are not to scale.

Flow charts are used herein to illustrate operations performed by systems according to some embodiments of the present application. It should be understood that the operations in the flow diagrams may be performed out of order. Rather, various steps may be processed in reverse order or simultaneously. Also, one or more other operations may be added to the flowcharts. One or more operations may also be deleted from the flowchart.

The systems and methods of the present application may be applied to different scenarios. Under these scenarios, the clocks of the different components need to be synchronized. For example, the systems and methods herein may be applied to time synchronization of different components (e.g., sensors) in a vehicle. The vehicle may be applied to land, sea, aerospace, etc., or any combination thereof. The vehicle may include a taxi, a private car, a windmill, a bus, a train, a bullet train, a high-speed rail, a subway, a ship, an airplane, a spaceship, a hot air balloon, an unmanned vehicle, a bicycle, a tricycle, a motorcycle, an autonomous vehicle, etc., or any combination thereof. As another example, the systems and methods herein may be applied to time synchronization of different components (e.g., sensors) in a smart device. The smart devices may include smart home devices (e.g., smart lighting devices, controllers for smart appliances, smart monitoring devices, smart televisions, smart cameras, walkie talkies, etc.), smart wearable devices (e.g., smart bracelets, smart pedals, smart glasses, smart helmets, smart watches, smart clothing, smart backpacks, smart accessories, etc.), smart mobile devices (e.g., smartphones, Personal Digital Assistants (PDAs), gaming devices, navigation devices, point of sale (POS) devices, etc.), robots, etc., or any combination.

There is a need and/or a demand for efficiently and accurately synchronizing clocks of multiple sensors in a vehicle (e.g., an autonomous vehicle). To this end, the systems and methods for time synchronization herein combine hardware time synchronization (e.g., based on a Pulse Per Second (PPS) signal and coordinated Universal Time (UTC)) with software time synchronization (e.g., using Network Time Protocol (NTP)) to synchronize clocks of multiple sensors in a vehicle. Sensors (e.g. lidar equipment, Inertial Navigation Systems (INS), controllers of the vehicle) that acquire data at a relatively high frequency (e.g. greater than 20HZ) may perform hardware time synchronization, as the accuracy requirements for time synchronization in this case are relatively high. Sensors (e.g., cameras) that acquire data at a relatively low frequency (e.g., less than 10HZ) may be software time synchronized because the accuracy requirements for time synchronization are relatively low in this case.

For the sake of brevity, the description of the methods and/or systems for time synchronization in this application may be taken as an example of a vehicle. It should be noted that the methods and/or systems for time synchronization described below are merely some examples or embodiments. It will be apparent to those of ordinary skill in the art that the methods and/or systems for time synchronization of the present application may be applied to other similar situations, such as smart devices.

Fig. 1 is a schematic diagram of an example time synchronization system according to some embodiments of the present application. Time synchronization system 100 may include at least one lidar device 110, at least one camera 120, a positioning system 130, a satellite navigation device 140, an Inertial Navigation System (INS) (also known as an Inertial Measurement Unit (IMU))150, a controller 160, a storage device 170, and a network 180. In some embodiments, at least one lidar device 110, at least one camera 120, a satellite navigation device 140, an inertial navigation system 150, a controller 160, and a storage device 170 may be included in the vehicle.

In some embodiments, lidar device 110 may include a laser emitter, scanning and optical components, and photosensitive components. The laser emitter may be configured to emit a laser beam having a particular wavelength. For example, the laser emitter may emit a laser beam having a wavelength ranging from 600nm to 1000 nm. In some embodiments, lidar device 110 may be configured to obtain a set of points that includes a set of points for at least two points associated with an object (e.g., a person, animal, tree, barricade, building, or vehicle) within a range of lidar apparatus 110 (e.g., a distance of 500 meters from the vehicle). A point in the set of points may include a three-dimensional (3D) coordinate of the point and a reflection intensity of the point. The reflected intensity of the spot may refer to the intensity of the laser beam reflected by the spot.

In some embodiments, lidar apparatus 110 may emit at least two laser beams and rotate at an angular velocity such that the at least two laser beams impinge on an object within a distance (e.g., 500 meters) from the vehicle such that lidar device 110 may obtain a set of points that includes at least two points associated with the object. For example, lidar device 110 may obtain a set of points when lidar device 110 completes a 360 ° rotation.

In some embodiments, because objects around the vehicle may be constantly changing, lidar device 110 may constantly emit a laser beam and constantly obtain a set of points associated with the objects around the vehicle. In some embodiments, lidar device 110 may transmit the set of points to storage device 170 and/or controller 160.

The camera 120 may be configured to obtain one or more images related to objects (e.g., people, animals, trees, roadblocks, buildings, or vehicles) within a range of the camera 120 (e.g., a distance of 500 meters from the vehicle). As used in this application, an image may be a still image, a video, a streaming video, or a video frame obtained from a video. In some embodiments, the camera 120 may be a digital camera, a web camera, a smartphone, a tablet, a laptop, a video game console with a web camera, a camera with multiple lenses, or the like. In some embodiments, the camera 120 may be configured to capture two-dimensional (2D) images and/or three-dimensional (3D) images. For example, the camera 120 may be a still camera, pan-tilt-zoom camera, motion camera, stereo camera, structured light camera, time-of-flight camera, and the like. In some embodiments, the camera 120 may also be a camera equipped with a time-of-flight device, Kinect sensor, 3D laser scanner, photographic scanner, or the like.

In some embodiments, the camera 120 may rotate to capture images including objects around the vehicle. For example, the camera 120 may obtain an image when the camera 120 completes a 360 ° rotation. In some embodiments, the camera 120 may continuously capture images related to objects around the vehicle because the objects around the vehicle may continuously change. In some embodiments, the acquisition of the set of points of lidar device 110 and the capture of the image of camera 120 may be simultaneous, such that the points in the set of points may correspond to pixels in the image. In some embodiments, the camera 120 may send the images to the storage device 170 and/or the controller 160.

Positioning system 130 may include one or more satellites, such as satellite 130-1, satellite 130-2, and satellite 130-3. In some embodiments, the positioning system 130 may be a Global Positioning System (GPS), a global navigation satellite system (GLONASS), a COMPASS navigation system (COMPASS), a beidou navigation satellite system, a galileo positioning system, a quasi-zenith satellite system (QZSS), or the like. Positioning system 130 may transmit the navigation signals to satellite navigation device 140 via a wireless connection.

Satellite navigation device 140 may determine position information and time information for satellite navigation device 140 by processing the navigation signals. The location information may be in the form of coordinates, such as latitude and longitude coordinates, and the like.

The inertial navigation system 150 may continuously determine the position, direction, and velocity (direction and velocity of movement) of the moving object by dead reckoning using a processor and inertial sensors, such as motion sensors (e.g., accelerometers) and rotation sensors (e.g., gyroscopes), without external reference. In some embodiments, the inertial sensor may be supplemented with a barometric altimeter. In some embodiments, the inertial sensor may be supplemented by a magnetic sensor (e.g., magnetometer) and/or a speed measurement device.

In some embodiments, the controller 160 may process information and/or data to perform one or more functions described herein. For example, controller 160 may interpret sensory information (e.g., a set of points from lidar device 110, images from camera 120, position information and time information from satellite navigation device 140, and bearing, speed, and acceleration from inertial navigation system 150) to control the speed and driving direction of the autonomous vehicle, and identify suitable navigation paths, obstacles, and related markers. For another example, controller 160 may synchronize a clock of controller 160 with a clock of satellite navigation device 140.

In some embodiments, the controller 160 may be local or remote. For example, controller 160 may access information and/or data stored in camera 120, lidar device 110, satellite navigation device 140, inertial navigation system 150, and/or storage device 170 via network 180. As another example, controller 160 may be directly connected to camera 120, lidar device 110, satellite navigation device 140, inertial navigation system 150, and/or storage device 170 to access stored information and/or data. In some embodiments, the controller 160 may be implemented on a computing device 200 having one or more components shown in FIG. 2 herein, or on a mobile device 300 having one or more components shown in FIG. 3 herein.

In some embodiments, controller 160 may include one or more processing engines (e.g., a single core processing engine or a multi-core processor). Merely by way of example, the controller 160 may include one or more hardware processors such as a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an application specific instruction set processor (ASIP), an image processing unit (GPU), a physical arithmetic processing unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computer (RISC), a microprocessor, or the like, or any combination thereof.

Storage device 170 may store data and/or instructions. In some embodiments, storage device 170 may store data obtained from controller 160, camera 120, lidar device 110, satellite navigation device 140, and/or inertial navigation system 150. For example, the storage device 170 may store images captured by the camera 120. As another example, storage device 170 may store a set of points obtained from lidar device 110. In some embodiments, storage device 170 may store data and/or instructions that may be executed or used to perform the exemplary methods described herein. For example, the memory device 170 may store instructions that the controller 160 may execute to control an autonomous vehicle. In some embodiments, the storage device 170 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), and the like, or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid state drives, and the like. Exemplary removable memory may include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tape, and the like. Exemplary volatile read and write memories can include Random Access Memory (RAM). Exemplary RAM may include Dynamic Random Access Memory (DRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), Static Random Access Memory (SRAM), thyristor random access memory (T-RAM), and zero capacitance random access memory (Z-RAM), among others. Exemplary read-only memories may include mask read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (dvd-ROM), and the like. In some embodiments, the storage device 170 may execute on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-tiered cloud, and the like, or any combination thereof.

In some embodiments, storage device 170 may be connected to network 180 to synchronize one or more components in system 100 (e.g., controller 160, camera 120, lidar device 110, and/or satellite navigation device 140, inertial navigation system 150) with time. One or more components in the time synchronization system 100 may access data or instructions stored in the storage device 170 via the network 180. In some embodiments, storage device 170 may be directly connected to or in communication with one or more components in time synchronization system 100 (e.g., controller 160, camera 120, lidar device 110, and/or satellite navigation device 140, inertial navigation system 150). In some embodiments, storage device 170 may be part of lidar device 110, camera 120, satellite navigation device 140, inertial navigation system 150, or controller 160.

Network 180 may facilitate the exchange of information and/or data. In some embodiments, one or more components in time synchronization system 100 (e.g., controller 160, camera 120, lidar device 110, storage device 170, positioning system 130, satellite navigation device 140, and inertial navigation system 150) may send/obtain information and/or data from other components in time synchronization system 100 via network 180. For example, satellite navigation device 140 may receive navigation signals from positioning system 130 via network 180. As another example, the camera 120 may send data packets to the controller 160 via the network 180. In some embodiments, the network 180 may be any one of, or any combination of, a wired network or a wireless network. By way of example only, network 180 may include a cable network, a wireline network, a fiber optic network, a telecommunications network, an intranet, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a bluetooth network, a zigbee network, a Near Field Communication (NFC) network, a mobile network, or the like, or any combination thereof. In some embodiments, network 180 may include one or more network access points. For example, network 180 may include wired or wireless network access points, such as base stations and/or internet exchange points 180-1, 180-2, through which one or more components of time synchronization system 100 may connect to network 180 to exchange data and/or information.

In some embodiments, the components in the time synchronization system 100 may be connected or in communication with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a Local Area Network (LAN), a Wide Area Network (WAN), bluetooth, zigbee network, Near Field Communication (NFC), etc., or any combination thereof.

Fig. 2 is a schematic diagram of exemplary hardware and/or software components of a computing device in which lidar device 110, camera 120, satellite navigation device 140, inertial navigation system 150, and/or controller 160 may be implemented according to some embodiments of the present application. As shown in FIG. 2, computing device 200 may include a processor 210, memory 220, input/output (I/O)230, and communication ports 240.

The processor 210 (e.g., logic circuitry) may execute computer instructions (e.g., program code) and perform functions in accordance with the techniques described herein. For example, the processor 210 may include an interface circuit 210-a and a processing circuit 210-b therein. Interface circuit 210-a may be configured to receive electronic signals from a bus (not shown in fig. 2) that encode/include structured data and/or instructions for processing by the processing circuit. The processing circuitry may perform logical computations and then determine the conclusion, result, and/or instruction encoding as electrical signals. The interface circuit may then issue the electrical signals from the processing circuit via the bus.

The computer instructions may include, for example, routines, programs, objects, components, data structures, procedures, modules, and functions that perform the particular functions described herein. For example, the controller 160 may be implemented on the computing device 200. Processor 210 may synchronize the clock of controller 160 with the clock of satellite navigation device 140. In some embodiments, processor 210 may include one or more hardware processors, such as microcontrollers, microprocessors, Reduced Instruction Set Computers (RISC), Application Specific Integrated Circuits (ASICs), application specific instruction set processors (ASIPs), Central Processing Units (CPUs), Graphics Processing Units (GPUs), Physical Processing Units (PPUs), microcontroller units, Digital Signal Processors (DSPs), Field Programmable Gate Arrays (FPGAs), high-order RISC machines (ARMs), Programmable Logic Devices (PLDs), any circuit or processor capable of executing one or more functions, or the like, or any combination thereof.

For illustration only, only one processor is depicted in computing device 200. It should be noted, however, that the computing device 200 in the present application may also include multiple processors, and that operations and/or method steps performed thereby, such as one processor described in the present application, may also be performed by multiple processors, either jointly or separately. For example, if in the present application, the processors of computing device 200 perform steps a and B, it should be understood that steps a and B may also be performed jointly or independently by two or more different processors of computing device 200 (e.g., a first processor performing step a, a second processor performing step B, or a first and second processor performing steps a and B jointly).

Memory 220 may store data/information obtained from lidar device 110, camera 120, satellite navigation device 140, inertial navigation system 150, controller 160, and/or any other component of time synchronization system 100. In some embodiments, memory 220 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), and the like, or any combination thereof. For example, mass storage may include a magnetic disk, optical disk, solid state disk, and the like. Removable memory may include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tape, and the like. Volatile read and write memory can include Random Access Memory (RAM). RAM may include Dynamic RAM (DRAM), double-data-rate synchronous dynamic RAM (DDRSDRAM), Static RAM (SRAM), thyristor RAM (T-RAM), zero-capacitance (Z-RAM), and the like. The read-only memory may include mask read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (dvd-ROM), and the like. In some embodiments, memory 220 may store one or more programs and/or instructions to perform the example methods described herein.

I/O230 may input and/or output signals, data, information, and the like. In some embodiments, I/O230 may enable a user to interact with controller 160. In some embodiments, I/O230 may include input devices and output devices. Exemplary input devices may include a keyboard, mouse, touch screen, microphone, etc., or any combination thereof. Exemplary output devices may include a display device, speakers, printer, projector, etc., or any combination thereof. Examples of a display device may include a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) based display, a flat panel display, a curved screen, a television device, a Cathode Ray Tube (CRT), a touch screen, and the like, or any combination thereof.

The communication port 240 may be connected to a network (e.g., network 180) to facilitate data communication. For example, the controller 160 may be implemented on the computing device 200. Communication port 240 may establish a connection between controller 160 and lidar 110, camera 120, satellite navigation device 140, inertial navigation system 150, or storage device 170. The connection may be a wired connection, a wireless connection, any other communication connection that may enable data transmission and/or reception, and/or any combination of these connections. The wired connection may include, for example, an electrical cable, an optical cable, a telephone line, etc., or any combination thereof. The wired connection may include, for example, an electrical cable, an optical cable, a telephone line, etc., or any combination thereof. The wireless connection may include, for example, Bluetooth^TMLink, Wi-Fi^TMLink, WiMax^TMLinks, WLAN links, zigbee links, mobile network links (e.g., 3G, 4G, 5G, etc.), and the like or combinations thereof. In some embodiments, communicationThe port 240 may be and/or include a standardized communication port such as RS232, RS485, and the like.

Fig. 3 is a schematic diagram illustrating exemplary hardware and/or software components of a mobile device on which the controller 160 may be implemented according to some embodiments of the present application. As shown in FIG. 3, mobile device 300 may include a communication platform 310, a display 320, a Graphics Processing Unit (GPU)330, a Central Processing Unit (CPU)340, I/O350, memory 360, and storage 390. In some embodiments, any other suitable component, including but not limited to a system bus or a controller (not shown), may also be included in mobile device 300. In some embodiments, an operating system 370 (e.g., iOS)^TM、Android^TM、WindowsPhone^TMEtc.) and one or more application programs 380 may be downloaded from storage 390 to memory 360 and executed by CPU 340. The application 380 may include a browser or any other suitable mobile application for receiving and presenting information related to the time synchronization system 100 or other information from the controller 160. User interaction with the information stream may be accomplished through I/O350 and provided to controller 160 and/or other components of time synchronization system 100 via network 180. For example only, the status of the task may be displayed in the user terminal 140 via the display 320. As another example, a user may enter tasks via I/O350.

To implement the various modules, units, and their functions described herein, a computer hardware platform may be used as the hardware platform for one or more of the components described herein. A computer with user interface components may be used to implement a Personal Computer (PC) or any other type of workstation or terminal device. If programmed properly, the computer can also act as a server.

Those of ordinary skill in the art will appreciate that when components of the time synchronization system 100 execute, the components may execute via electrical and/or electromagnetic signals. For example, when the controller 160 processes a task such as making a determination or identifying information, the controller 160 may operate logic circuits in its processor to process such a task. When the controller 160 receives data (e.g., data packets) from the camera 120, a processor of the controller 160 may receive an electrical signal encoding/including the data. The processor of controller 160 may receive electrical signals through one or more information exchange ports. If the controller 160 communicates with the camera 120 via a wired network, the information exchange port may be physically connected to a cable. If the controller 160 is in communication with the camera 120 via a wireless network, the information exchange port of the controller 160 may be one or more antennas that may convert electrical signals to electromagnetic signals. Within an electronic device, such as laser radar device 110, camera 120, satellite navigation device 140, inertial navigation system 150, and/or controller 160, when a processor thereof processes instructions, the processor issues instructions and/or performs actions, the instructions and/or actions being performed via electrical signals. For example, when the processor retrieves or saves data from a storage medium (e.g., storage device 170), it may send electrical signals to a read/write device of the storage medium, which may read or write the structural data in the storage medium. The structured data may be transmitted to the processor in the form of electrical signals via a bus of the electronic device. Herein, an electrical signal may refer to one electrical signal, a series of electrical signals, and/or at least two discrete electrical signals.

Fig. 4 is a flow diagram of an exemplary process for time synchronization shown in accordance with some embodiments of the present application. In some embodiments, process 400 may be implemented in time synchronization system 100 shown in FIG. 1. For example, at least one operation (e.g., operation 450) of process 400 performed by a component (e.g., camera 120) of time synchronization system 100 may be stored in a storage medium (e.g., storage device 170, memory 220 of camera 120) in the form of instructions and may be invoked and/or executed by the component (e.g., processor 210 of camera 120). The operations of the illustrated process 400 presented below are intended to be illustrative. In some embodiments, process 400 may be accomplished with one or more additional operations not described, and/or one or more operations not discussed. Additionally, the order in which the operations of process 400 are illustrated in FIG. 4 and described below is not intended to be limiting.

In 410, satellite navigation device 140 may receive navigation signals from positioning system 130 (e.g., satellite 130-1, satellite 130-2, and/or satellite 130-3) via a wireless connection.

In some embodiments, the satellites in positioning system 130 carry stable atomic clocks that are synchronized to each other and to a ground clock. A satellite in positioning system 130 (e.g., satellite 130-1, satellite 130-2, or satellite 130-3) may continuously broadcast a navigation signal (radio signal, with a modulated carrier). The navigation signal may be a pseudorandom code (sequence of 1 s and 0 s) that may be processed by a receiver (e.g., satellite navigation device 140). The navigation signal may include a transmission time of the satellite transmitting the navigation signal (according to a clock of the satellite) and a position of the satellite at the transmission time.

Based on the navigation signal, satellite navigation device 140 may synchronize a first clock of satellite navigation device 140 with a clock of positioning system 130 at 420.

In some embodiments, satellite navigation device 140 may measure the time of arrival of the navigation signal upon receipt of the navigation signal (from the clock of satellite navigation device 140 itself, also referred to as the first clock). Based on the time of transmission and the time of arrival, satellite navigation device 140 may determine a time of flight (TOF) value of the navigation signal corresponding to one satellite. In some embodiments, satellite navigation device 140 may determine TOF values corresponding to at least four satellites. The satellite navigation device 140 may synchronize the first clock with the clock of the satellite by determining a clock offset between the first clock and the clock of the satellite and determine its own three-dimensional position (e.g., longitude, latitude, and altitude).

At 430, based on the navigation signal and/or the first clock, the satellite navigation device 140 may determine a Universal Time Coordinated (UTC) and a Pulse Per Second (PPS) signal.

At 440, satellite navigation device 140 may transmit coordinated universal time and second pulse signals to at least one of controller 160, at least one lidar device 110, and inertial navigation system 150.

The pulse per second signal and the coordinated universal time may be synchronized with the first clock. A one second pulse is an electrical signal having a width of less than one second, and a sharp rising or falling edge.

In some embodiments, satellite navigation device 140 may periodically transmit a pulse-per-second signal and/or coordinate universal time. For example, satellite navigation device 140 may send a second pulse signal to a component (e.g., controller 160, lidar device 110, or inertial navigation system 150) once per second. In this case, the pulse-per-second signal may include the entire second of the universal time coordinated. For example, a sharp rising or falling edge of the pulse-per-second signal may indicate an entire second of information when the world is coordinated.

In some embodiments, satellite navigation device 140 may transmit National Marine Electronics Association (NMEA) statements that include coordinated universal time (e.g., in the form of hhmms.

At 450, based on the pulse-per-second signal and the coordinated universal time, controller 160, at least one lidar device 110, and/or inertial navigation system 150 may synchronize their own clocks (e.g., a second clock of controller 160, a third clock of at least one lidar device 110, and a fourth clock of inertial navigation system 150) with the first clock, respectively. In some embodiments, controller 160, at least one lidar device 110, and/or inertial navigation system 150 may synchronize its own clock with the first clock using coordinated universal time and reset a counter associated with its own clock using the second pulse signal.

In some embodiments, if time synchronization system 100 includes two or more lidar devices, satellite navigation device 140 may transmit the PPS signal and the UTC to the two or more lidar devices. The two or more lidar devices may synchronize their own clocks with the first clock based on the PPS signal and the UTC, respectively.

In some embodiments, there may be errors in time synchronization based on the pulse-per-second signal and the coordinated universal time due to, for example, accumulation of errors within one period of transmitting the pulse-per-second signal, delays in transmitting the pulse-per-second signal and/or the coordinated universal time, reception and/or analysis of the navigation signal to determine failure of the pulse-per-second signal and the coordinated universal time, hardware related errors, and the like.

In some embodiments, the error of time synchronization based on the pulse-per-second signal may be accumulated in one cycle. The cumulative error of time synchronization within 1 second is acceptable in some cases. Satellite navigation device 140 may increase the frequency at which the pulse-per-second signal is transmitted if it is desired to further reduce the accumulated error. For example, satellite navigation device 140 may transmit the pulse-per-second signal at a frequency greater than 1 HZ.

In some embodiments, satellite navigation device 140 may transmit coordinated universal time (e.g., NMEA statements) after transmitting the pulse-per-second signal for a period of time to increase robustness of time synchronization and reduce stress on the time synchronization of lidar device 110, inertial navigation system 150, and/or controller 160, which may reduce errors in time synchronization.

In some embodiments, it may be difficult for satellite navigation device 140 to receive navigation signals from positioning system 130 when satellite navigation device 140 is located in certain locations (e.g., underwater, underground, indoors, in tunnels, etc.). In these cases, satellite navigation device 140 does not receive satellite signals and cannot send coordinated universal time (e.g., NMEA statements) and second pulse signals to other portions of the vehicle (e.g., controller 160). In this case, inertial navigation system 150 may determine a simulated pulse-per-second signal based on a clock of inertial navigation system 150 and send the simulated pulse-per-second signal to at least one lidar device 110 and/or controller 160.

In some embodiments, the pulse-per-second signal may be a transistor-transistor logic (TTL) signal that may be signal attenuated and affected by noise from other signals when transmitted between two devices (e.g., from satellite navigation device 140 to at least one of lidar device 110, controller 160, or inertial navigation system 150). To reduce signal attenuation, the pulse-per-second signal may be amplified (e.g., by a relay amplifier) or opto-electrically isolated when sent to controller 160, inertial navigation system 150, or at least one of lidar 110. To reduce the noise effects of other signals, the PPS signal may be processed by masking (e.g., using a magnetic loop) high frequency noise in the PPS signal when it is sent to the controller 160, the inertial navigation system 150, or at least one of the lidar 110.

In 460, the at least one camera 120 may synchronize a clock (e.g., a fifth clock) of the at least one camera 120 with a clock (e.g., a second clock) of the controller 160.

In some embodiments, the at least one camera 120 may synchronize the fifth clock with the second clock using Network Time Protocol (NTP). Details regarding the synchronization of the fifth clock with the second clock may be found elsewhere in the application (e.g., in conjunction with the description of fig. 5).

In some embodiments, if the time synchronization system 100 includes two or more cameras, the two or more cameras may each synchronize their own clock to a second clock using NTP.

In some embodiments, hardware time synchronization (e.g., based on a pulse-per-second signal and coordinated universal time) may enable higher precision (e.g., sub-microsecond level of error) time synchronization. However, the scalability and flexibility of hardware time synchronization is limited by the different hardware standards and the complex arrangement of hardware. The criteria for software time synchronization (e.g., using NTP) may be relatively uniform. However, the accuracy of software time synchronization (e.g., errors on the order of milliseconds) is low.

The process 400 for time synchronization of multiple sensors in a vehicle combines hardware time synchronization with software time synchronization. Sensors (e.g., at least one lidar device 110, inertial navigation system 150, and controller 160) that acquire data at a relatively high frequency (e.g., greater than 20HZ) may perform hardware time synchronization (e.g., operations 440 and 450) because in this case, the accuracy requirements for time synchronization are high. A sensor (e.g., the at least one camera 120) that obtains data at a relatively low frequency (e.g., below 10HZ) may perform software time synchronization (e.g., operation 460), as the accuracy of the time synchronization need not be as high in this case. The design improves the overall efficiency of time synchronization, saves resources and meets the requirements of all equipment in the vehicle.

In the present application, the time synchronization of the above-mentioned components is taken as an example, and is provided for illustrative purposes only, and is not intended to limit the scope of the present application. In some embodiments, the process 400 for time synchronization in the present application may be applied to other components, such as other sensors in a vehicle, such as radar, sonic locators, odometers, and the like.

It should be noted that the foregoing description is provided for illustrative purposes only, and is not intended to limit the scope of the present application. Various changes and modifications can be made by one of ordinary skill in the art in light of the teachings of this application. However, such changes and modifications do not depart from the scope of the present application.

Fig. 5 is a flow diagram of an exemplary process for synchronizing a clock of at least one camera with a clock of a controller, shown in accordance with some embodiments of the present application. In some embodiments, process 500 may be implemented in time synchronization system 100 shown in FIG. 1. For example, the process 500 may be stored in a storage medium (e.g., the storage device 170, the memory 220 of the camera 120) in the form of instructions and may be invoked and/or executed by the camera 120 (e.g., the processor 210 of the camera 120). The operations of the illustrated process 500 presented below are intended to be illustrative. In some embodiments, process 500 may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. Additionally, the order in which the operations of process 500 are illustrated in fig. 5 and described below is not intended to be limiting. In some embodiments, operation 450 of process 400 in fig. 4 may be performed based on process 500.

At 510, the at least one camera 120 may send an initial data packet to the controller 160. The initial data packet may include a first timestamp based on a fifth clock of the at least one camera 120. The first timestamp indicates a time at which the at least one camera 120 sent the initial data packet.

At 520, the at least one camera 120 may receive the modified data packet of the controller 160. The modification packet may include a second time stamp and a third time stamp based on the second clock of the controller 160. The second timestamp indicates the time at which the controller 160 received the initial data packet. The third timestamp indicates the time at which the modified packet was sent by the controller 160.

At 530, the at least one camera 120 may record a fourth timestamp according to a fifth clock of the at least one camera 120 after receiving the modification data packet. The fourth timestamp indicates a time at which the at least one camera 120 received the modified data packet from the controller 160.

At 540, based on the first time stamp, the second time stamp, the third time stamp, and the fourth time stamp, the at least one camera 120 may determine a deviation between the second clock and the fifth clock.

Based on the first, second, third, and fourth timestamps, the at least one camera 120 may determine a one-way transmission time of a data packet (e.g., an initial data packet or a modified data packet) between the controller 160 and the at least one camera 120. Based on the transmission time, the at least one camera 120 may determine the deviation.

For example, the at least one camera 120 may determine the deviation based on equation (1) below:

T_delay＝[(T₄-T₁)-(T₃-T₂)]/2 (1)

wherein, it is assumed that the transmission time of the data packet from the controller 160 to the at least one camera 120 is equal to the transmission time from the at least one camera 120 to the controller 160; t is_delayRefers to the one-way transmission time of the data packets between the controller 160 and the at least one camera 120; t1, T2, T3, and T4 refer to a first timestamp, a second timestamp, a third timestamp, and a fourth timestamp, respectively.

The at least one camera 120 may determine the deviation based on equation (2) below:

T_offset＝T₂-(T₁+T_delay) (2),

wherein T is_offsetRefers to the offset between the second clock and the fifth clock.

Based on the deviation, the at least one camera 120 may synchronize the fifth clock with the second clock at 550.

In some embodiments, the at least one camera 120 may send an initial data packet to the controller 160 at intervals (e.g., once per second).

In some embodiments, if the time synchronization system 100 includes two or more cameras, the two or more cameras may synchronize their own clocks with the second clock based on the process 500, respectively.

It should be noted that the foregoing description is provided for illustrative purposes only, and is not intended to limit the scope of the present application. Various changes and modifications can be made by one of ordinary skill in the art in light of the teachings of this application. However, such changes and modifications do not depart from the scope of the present application.

After time synchronizing the components in the time synchronization system 100 using the systems and/or methods (e.g., process 400 and/or process 500) of the present application, the accuracy of the time synchronization may be verified.

For example, data (e.g., a set of points) obtained by at least one lidar device 110 may be time-stamped according to a clock after synchronization of at least one lidar device 110. Controller 160 may read data obtained by at least one lidar device 110 in real-time and compare a timestamp of the real-time data to a current time of controller 160 to determine a deviation between a clock of at least one lidar device 110 and a clock of controller 160. The result is a deviation of about 0.3 ms.

For another example, a Light Emitting Diode (LED) is controlled to be turned on or off based on a pulse per second signal transmitted once per second. The sharp rising edge of the pulse-per-second signal indicates the entire second of the coordinated universal time. The response time of the LED is about 10-6 s. For example, after the power of the LED is set to a high level (e.g., a voltage greater than 3.5 volts), the LED is controlled to turn on at the edge of the sharp rise of the pulse signal every second. After a period of time (e.g., shorter than the width of the second pulse signal), the LED is controlled to be off and the power to the LED is set to a low level (e.g., a voltage less than 0.3 volts). The high and low levels are set to determine the duration that the LEDs are on. The at least one camera 120 may be controlled to capture an image of the LED at the full second according to the clock after the synchronization of the at least one camera 120. In this way, the result is that the cameras 120 are able to capture images when the LEDs are on, and the error in the time synchronization of at least one camera 120 is about 10 ms.

Having thus described the basic concept, it will be apparent to those skilled in the art from this detailed application that the foregoing disclosure is to be considered as illustrative only and is not limiting. Various modifications, improvements and adaptations of the present application may occur to those skilled in the art, although they are not explicitly described herein. Such modifications, improvements, and adaptations through the present application are intended to be suggested and are intended to be within the spirit and scope of the exemplary embodiments of the present application.

Also, this application uses specific language to describe embodiments of the application. For example, "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the application. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the application may be combined as appropriate.

Moreover, those of ordinary skill in the art will understand that aspects of the present application may be illustrated and described in terms of several patentable species or situations, including any new and useful combination of processes, machines, articles, or materials, or any new and useful improvement thereof. Accordingly, various aspects of the present application may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software. The above hardware or software may be referred to as a "unit", "module", or "system". Furthermore, aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media, with computer-readable program code embodied therein.

A computer readable signal medium may include a propagated data signal with computer program code embodied therewith, for example, on baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including electro-magnetic, optical, and the like, or any suitable combination. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code on a computer readable signal medium may be propagated over any suitable medium, including radio, cable, fiber optic cable, RF, etc., or any combination of the preceding.

Computer program code required for operation of various portions of the present application may be written in any one or more programming languages, including a subject oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C + +, C #, VB.NET, Python, and the like, a conventional programming language such as C, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, a dynamic programming language such as Python, Ruby, and Groovy, or other programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any network format, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as a software as a service (SaaS).

Additionally, the order in which elements and sequences of the processes described herein are processed, the use of alphanumeric characters, or the use of other designations, is not intended to limit the order of the processes and methods described herein, unless explicitly claimed. While various presently contemplated embodiments of the invention have been discussed in the foregoing disclosure by way of example, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments herein. For example, although implementations of the various components described above may be embodied in a hardware device, they may also be implemented by software-only solutions, such as installing the described system on an existing server or mobile device.

Similarly, it should be noted that in the preceding description of embodiments of the present application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed features are more features than are expressly recited in each claim. Indeed, less than all of the features of a single embodiment disclosed above are claimed.