阅读说明：本技术 用于基于障碍物检测校正高清地图的系统和方法 (System and method for correcting high-definition maps based on obstacle detection ) 是由 冯路 马腾 于 2018-11-15 设计创作，主要内容包括：本申请的实施例提供了用于校正高清地图的系统和方法。该系统可以包括通信接口,该通信接口被配置为接收由激光雷达捕获的场景的点云数据。该系统还可以包括被配置为存储所述点云数据的存储器,以及所述至少一个处理器。所述至少一个处理器可以被配置为在所述点云数据中定位由所述至少一个障碍物引起的至少一个孔洞。所述至少一个处理器还被配置用于对于所述至少一个孔洞,估计所述场景在没有所述至少一个障碍物的情况下被捕获的无障碍点云数据,并通过使用无障碍点云数据修复接收到的点云数据来校正高清地图。(Embodiments of the present application provide systems and methods for correcting high definition maps. The system may include a communication interface configured to receive point cloud data of a scene captured by a lidar. The system may also include a memory configured to store the point cloud data, and the at least one processor. The at least one processor may be configured to locate at least one hole in the point cloud data caused by the at least one obstacle. The at least one processor is further configured for, for the at least one hole, estimating unobstructed point cloud data of the scene captured without the at least one obstruction, and correcting the high definition map by repairing the received point cloud data using the unobstructed point cloud data.)

1. A system for correcting a high definition map, comprising:

a communication interface configured to receive point cloud data of a scene captured by a lidar;

a memory configured to store the point cloud data; and

at least one processor configured to:

detecting at least one obstacle from the point cloud data;

locating at least one hole in the point cloud data caused by the at least one obstacle;

for the at least one hole, estimating unobstructed point cloud data of the scene captured without the at least one obstruction; and

and correcting the high-definition map by repairing the received point cloud data by using the barrier-free point cloud data.

2. The system of claim 1, wherein the memory is configured to store a neural network, and the at least one processor is configured to detect the at least one obstacle using the neural network.

3. The system of claim 2, wherein the neural network is a convolutional neural network.

4. The system of claim 2, wherein the at least one processor is further configured to:

creating a two-dimensional projection image based on the point cloud data, the two-dimensional projection image comprising at least two pixels; and

determining, using the neural network, at least two attributes associated with each pixel, the attributes indicating a likelihood that the pixel corresponds to the at least one obstacle.

5. The system of claim 4, wherein the at least one processor is further configured to:

identifying a plurality of segments from the two-dimensional projection image, each segment comprising a plurality of pixels having a plurality of attributes indicating that the plurality of pixels corresponds to one of the at least one obstacle; and

determining a region of aggregation in the two-dimensional projection image, the region of aggregation containing all identified segments.

6. The system of claim 1, wherein to locate the at least one hole in the point cloud data, at least one processor is further configured to:

determining a plane where each hole is located; and

the boundaries of each hole are determined.

7. The system of claim 6, wherein the at least one processor is further configured to estimate the obstacle-free point cloud data based on a plane in which the holes lie.

8. The system of claim 1, wherein the received point cloud data comprises at least two frames of point cloud captured as the lidar moves along a trajectory, and the at least one processor is configured to:

repairing each point cloud frame with the corresponding barrier-free point cloud data; and

aggregating the repaired point cloud frames.

9. A method for correcting a high-definition map comprises the following steps:

receiving point cloud data of a scene captured by a laser radar through a communication interface;

detecting, by the at least one processor, at least one obstacle from the point cloud data;

positioning, by the at least one processor, in at least one hole caused by at least one obstacle;

estimating, by the at least one processor, unobstructed point cloud data captured without the at least one obstruction for the scene of the at least one hole; and

revising, by the at least one processor, the high-definition map and restoring the received point cloud data by utilizing the unobstructed point cloud data.

10. The method of claim 9, wherein the at least one obstacle is detected using the neural network.

11. The method of claim 10, wherein the neural network is a convolutional neural network.

12. The method of claim 10, further comprising:

creating a two-dimensional projection image from the point cloud data, the two-dimensional projection image comprising at least two pixels; and

determining, using the neural network, at least two attributes associated with each pixel, the attributes indicating a likelihood that the pixel corresponds to the at least one obstacle.

13. The method of claim 12, further comprising:

identifying a plurality of segments from the two-dimensional projection image, each segment comprising a plurality of pixels having a plurality of attributes indicating that the plurality of pixels corresponds to one of the at least one obstacle; and

determining a region of aggregation in the two-dimensional projection image, the region of aggregation containing all identified segments.

14. The method of claim 9, wherein to locate the at least one hole in the point cloud data, the point cloud data further comprises:

determining a plane where each hole is located; and

the boundaries of each hole are determined.

15. The method of claim 14, wherein the unobstructed point cloud data is estimated from a plane in which the hole lies.

16. The method of claim 1, wherein the received point cloud data comprises at least two frames of point clouds captured as the lidar moves along a trajectory, wherein correcting the high definition map further comprises:

repairing each point cloud frame with the corresponding barrier-free point cloud data; and

aggregating the repaired point cloud frames.

17. A non-transitory computer readable medium having stored thereon computer program instructions, wherein the computer program when executed by at least one processor causes the processor to perform a method for correcting a high definition map, the method comprising:

receiving point cloud data of a scene captured by a laser radar;

detecting at least one obstacle from the point cloud data;

locating at least one hole caused by at least one obstacle in the point cloud data;

for the at least one hole, estimating unobstructed point cloud data of the scene captured without the at least one obstruction; and

correcting the high-definition map by repairing the received point cloud data with the obstacle-free point cloud data.

18. The non-transitory, machine-readable storage medium of claim 17, wherein the detecting the at least one obstacle using a neural network, the method further comprises:

creating a two-dimensional projection image from the point cloud data, the two-dimensional projection image comprising at least two pixels; and

determining, using the neural network, at least two attributes associated with each pixel, the attributes indicating a likelihood that the pixel corresponds to the at least one obstacle.

19. The non-transitory computer-readable medium of claim 18, wherein the method further comprises:

identifying a plurality of segments from the two-dimensional projection image, each segment comprising a plurality of pixels having a plurality of attributes indicating that the plurality of pixels corresponds to one of the at least one obstacle; and

determining a region of aggregation in the two-dimensional projection image, the region of aggregation containing all identified segments.

20. The non-transitory computer-readable medium of claim 17, further comprising, to locate the at least one hole in the point cloud data:

determining a plane where each hole is located; and

the boundary of each of the holes is determined,

wherein the unobstructed point cloud data is estimated based on a plane in which the hole lies.

Technical Field

The present application relates to systems and methods for correcting high definition maps in the presence of obstacles in a scene, and more particularly to systems and methods for updating high definition maps based on obstacle detection.

Background

Autonomous driving techniques rely heavily on accurate maps. For example, the accuracy of navigation maps is critical to the function of autonomous vehicles, such as localization, environmental recognition, decision making, and control. High-definition maps may be obtained by aggregating images and information acquired by various sensors, detectors, and other devices mounted on the vehicle while the vehicle is in motion. For example, a vehicle may be equipped with a plurality of integrated sensors, such as a light detection and transmission system (lidar), a Global Positioning System (GPS) receiver, one or more inertial measurement unit (inertial measurement unit) sensors, and one or more cameras to capture features of the road or surrounding objects on which the vehicle is traveling. The captured data may include, for example, center line or boundary line coordinates of a lane, coordinates of an object, and images, such as a building, another vehicle, a landmark, a pedestrian, or a traffic sign.

In particular, the laser radar measures the distance to a target by irradiating the target with a pulsed laser and measuring the reflected pulse with a sensor. The difference in laser return time and wavelength can then be used to construct a digital three-dimensional (3-D) representation of the object. The data captured by the lidar is referred to as point cloud data. During the survey, various objects on the road may obstruct the lidar's view. Since the pulsed laser is reflected by an obstacle before reaching the road, the captured point cloud data may be distorted. A high-resolution map reconstructed using such distorted point cloud data may not accurately reflect the captured scene.

Embodiments of the present application address the above-mentioned problems by systems and methods for correcting high-definition maps based on segmentation of point cloud data and compensation of distortion caused by obstacles.

Disclosure of Invention

Embodiments of the present application provide a system for correcting a high definition map. The system may include a communication interface configured to receive point cloud data of a scene captured by a lidar. The system may also include a memory configured to store the point cloud data, and at least one processor. The at least one processor may be configured to detect at least one obstacle from the point cloud data and locate at least one hole in the point cloud data caused by the at least one obstacle. The at least one processor is further configured to estimate, for the at least one hole, unobstructed point cloud data of the scene captured without the at least one obstruction, and correct the high definition map by repairing the received point cloud reception data with the unobstructed point cloud data.

The embodiment of the application also provides a method for correcting the high-definition map. The method may include receiving, via a communication interface, point cloud data of a scene captured by a lidar. The method may also include detecting, by the at least one processor, at least one obstacle from the point cloud data, and locating, by the at least one processor, at least one hole caused by the at least one obstacle. The method may also include estimating, by at least one processor, unobstructed point cloud data of the scene captured without the at least one obstruction for the at least one hole, and the at least one processor correcting the high definition map by repairing the received point cloud data with the unobstructed point cloud data.

Embodiments of the present application also provide a non-transitory computer readable medium having stored thereon computer program instructions, which when executed by at least one processor, cause the processor to perform a method for correcting a high definition map. The method may include receiving point cloud data of a scene captured by a lidar. The method may also include detecting at least one obstacle from the point cloud data, and locating, by at least one processor, at least one hole in the point cloud data caused by the at least one obstacle. The method may further include, for the at least one hole, estimating unobstructed point cloud data of the scene captured without the at least one obstruction, and correcting the high definition map by repairing the received point cloud data with the unobstructed point cloud data.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

Drawings

FIG. 1 shows a schematic diagram of an exemplary vehicle equipped with sensors, according to an embodiment of the present application.

Fig. 2 illustrates a block diagram of an exemplary server for correcting high definition maps, according to an embodiment of the present application.

Fig. 3 shows a flow diagram of an exemplary method for correcting a high definition map according to an embodiment of the application.

Fig. 4 illustrates an exemplary Convolutional Neural Network (CNN) for segmenting point cloud data, according to embodiments of the present application.

Fig. 5 illustrates an exemplary method of estimating unobstructed point cloud data according to an embodiment of the application.

Detailed Description

The present application is described in detail by way of exemplary embodiments, which will be described in detail by way of illustration. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

FIG. 1 shows a schematic diagram of an exemplary vehicle 100 having at least two sensors 140 and 150, according to an embodiment of the present application. Consistent with some embodiments, vehicle 100 may be a survey vehicle configured for acquiring data for building high definition maps or 3-D city modeling. It is contemplated that the vehicle 100 may be an electric vehicle, a fuel cell vehicle, a hybrid vehicle, or a conventional internal combustion engine vehicle. The vehicle 100 may have a body 110 and at least one wheel 120. The body 110 may be any body type, such as a sports vehicle, a coupe, a sedan, a pick-up truck, a wagon, a Sport Utility Vehicle (SUV), a minivan, or a switch car. In some embodiments, the vehicle 100 may include a pair of front wheels and a pair of rear wheels, as shown in FIG. 1. However, it is contemplated that the vehicle 100 may have more or fewer wheels or equivalent structures that enable the vehicle 100 to move around. The vehicle 100 may be configured as all-wheel drive (AWD), front-wheel drive (FWR), or rear-wheel drive (RWD). In some embodiments, the vehicle 100 may be configured to be operated by an operator of the occupied vehicle, remotely controlled by the operator, and/or actively driven.

As shown in fig. 1, the vehicle 100 may be equipped with a sensor 140 mounted to the body 110 via a mounting structure 130. The mounting structure 130 may be an electromechanical device mounted or otherwise attached to the body 110 of the vehicle 100. In some embodiments, the mounting structure 130 may use screws, adhesives, or other mounting mechanisms. The vehicle 100 may be additionally equipped with sensors 150 inside or outside the body 110 using any suitable mounting mechanism. It is contemplated that the manner in which the sensors 140 or 150 are mounted on the vehicle 100 is not limited by the example shown in fig. 1, and may be modified depending on the type of sensors 140 and 150 and/or vehicle 100 to achieve the desired perception performance.

In some embodiments, the sensors 140 and 150 may be configured to capture data as the vehicle 100 travels along a trajectory. Consistent with the present application, the sensor 140 may be a lidar configured to scan the surrounding environment and acquire a point cloud. The laser radar measures the distance to a target by irradiating the target with a pulsed laser and measuring the reflected pulse with a sensor. The difference in laser return time and wavelength can then be used to create a digital 3-D representation of the target. The light used for lidar scanning may be ultraviolet, visible, or near infrared. Lidar is particularly suitable for high definition map mapping, since a narrow laser beam can map physical features with very high resolution. In some embodiments, the lidar may capture a point cloud. The point cloud data may include a set of data points on an outer surface of the target. The point cloud may be processed to construct a 3-D model of the object. The sensor 140 may continuously capture point cloud data as the vehicle 100 travels along the trajectory. Each set of point cloud data captured at a certain point in time is called a point cloud frame. The point cloud frames may be sent to the server 160 in real time (e.g., by streaming) or together after the vehicle 100 completes the entire trajectory.

As shown in fig. 1, the vehicle 100 may additionally be equipped with sensors 150, the sensors 150 may include sensors used in navigation units, such as a GPS receiver and one or more inertial measurement unit sensors. GPS is a global navigation satellite system that provides geographic positioning and time information to GPS receivers. An inertial measurement unit is an electronic device that uses various inertial sensors, such as accelerometers and gyroscopes, and sometimes magnetometers, to measure and provide specific forces, angular rates of the vehicle, and sometimes magnetic fields around the vehicle. By combining GPS receivers and inertial measurement unit sensors, the sensors 150 can provide real-time pose information of the vehicle 100 as the vehicle 100 travels, including the position and orientation (e.g., euler angles) of the vehicle 100 at each point in time. Consistent with the present application, the sensor 150 may measure pose information at the same point in time that the sensor 140 captures the point cloud frame. Thus, pose information can be associated with the respective point cloud frame. In some embodiments, a combination of the point cloud frames and their associated pose information may be used to locate the vehicle 100.

Consistent with the present application, the sensors 140 and 150 may communicate with a server 160. in some embodiments, the server 160 may be a local physical server, a cloud server (as shown in FIG. 1), a virtual server, a distributed server, or any other suitable computing device consistent with the present application, the server 160 may obtain data from the sensors 140 and 150. the server 160 may be via a network, such as a wireless local area network (W L AN), a Wide Area Network (WAN), a wireless network such as radio waves, a cellular network, a satellite communication network, and/or a local or short range wireless network (e.g., Bluetooth)^TM) Communicate with sensors 140 and 150, and/or other components of vehicle 100. Server 160 may store a high definition map. In some embodiments, a high-definition map may be initially constructed using point cloud data acquired by the sensors 140.

Consistent with the present application, the server 160 may also be responsible for correcting the high-definition map to remove any distortion of the captured point cloud data due to obstacles. In some embodiments, the obstacle may include any object that blocks the path of the laser light of the sensor 140, such that the laser light is reflected by the object before reaching the road. For example, the obstacle may be a pedestrian, a temporary stand, or a parked or moving vehicle, such as an automobile, motorcycle, or bicycle, etc. The server 160 may separate the point cloud data reflected by the obstacle from the rest of the point cloud data, estimate the correct point cloud data when the obstacle is not present, and correct the high definition map using the estimated point cloud data.

For example, fig. 2 illustrates a block diagram of an exemplary server 160 for correcting high definition maps according to embodiments of the present application. In some embodiments, as shown in fig. 2, server 160 may include a communication interface 202, a processor 204, memory 206, and storage 208. In some embodiments, the server 160 may have different modules, such as Integrated Circuit (IC) chips, in a single device, or separate devices with dedicated functionality. In some embodiments, one or more components of the server 160 may be located in the cloud, or may alternatively be located in a single location (e.g., within the vehicle 100 or within a mobile device) or distributed locations. The components of server 160 may be in an integrated device or distributed in different locations, but in communication with each other via a network (not shown).

Communication interface 202 may be via a communication cable, a wireless local area network (W L AN), a Wide Area Network (WAN), a wireless network such as a radio wave, a cellular network, and/or a local or short range wireless network (e.g., Bluetooth)^TM) Or other communication methods to transmit data to and receive data from components such as sensors 140 and 150. in some embodiments, communication interface 202 may be AN Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide a data communication connection.a further example, communication interface 202 may be a local area network (L AN) card to provide a data communication connection with a compatible L AN. a wireless link may also be implemented by communication interface 202. in such implementations, communication interface 202 may transmit and receive electrical, electromagnetic, or optical signals over the network that carry digital data streams representing various types of information.

Consistent with some embodiments, the communication interface 202 may receive data, such as point cloud data 203 captured by the sensor 140. In some embodiments, the communication interface 202 may also receive pose information (not shown) captured by the sensors 150. In some embodiments, the communication interface 202 may also receive a neural network 205 from the training device 180. In some embodiments, training device 180 may have a similar structure as server 160, including, for example, a processor, memory, storage device, and the like. In some embodiments, training device 180 may be part of server 160. The training device 180 may be configured to train the neural network 205 using sample data. The neural network 205 may learn to perform cognitive tasks during the training process. For example, the neural network 205 may be trained to detect obstacles based on the initial point cloud data captured by the lidar. In some embodiments, the neural network 205 may be trained to perform segmentation tasks, e.g., separating a portion of the point cloud data 203 related to an obstacle from the rest of the point cloud data 203. Consistent with the present application, the neural network 205 may be any suitable learning model, including but not limited to a Convolutional Neural Network (CNN). In some embodiments, the training of the neural network 205 may determine one or more model parameters, such as the weight, size, shape, and structure of the convolution kernel.

The communication interface 202 may also provide the received point cloud data 203 and the neural network 205 to a memory 208 for storage or to a processor 204 for processing. The communication interface 202 may also receive the corrected high definition map created by the processor 204 and provide the corrected map to any local component in the vehicle 100 or any remote device via a network.

The processor 204 may comprise any suitable type of general or special purpose microprocessor, digital signal processor, or microcontroller. The processor 204 may be configured as a separate processor module dedicated to correcting high definition maps. Alternatively, the processor 204 may be configured to share a processor module for performing other functions unrelated to generating and correcting high definition maps.

As shown in fig. 2, the processor 204 may include a plurality of modules, such as a point cloud segmentation unit 210, a hole location unit 212, an unobstructed point cloud estimation unit 214, a map correction unit 216, and so on. These modules (and any corresponding sub-modules or sub-units) may be hardware units (e.g., portions of an integrated circuit) of the processor 204 that are designed to be used with other components or software units implemented by the processor 204 by executing at least a portion of a program. The program may be stored on a computer readable medium and when executed by the processor 204, it may perform one or more functions. Although FIG. 2 shows all of the units 210 within one processor 204 and 216, it is contemplated that the units may be distributed among multiple processors that are close or remote from each other.

The point cloud segmentation unit 210 may be configured to segment the received point cloud data 203 to detect one or more obstacles. The obstacle may be a pedestrian, a vehicle, or other object in the laser path of the sensor 140. In some embodiments, the point cloud segmentation unit 210 may be configured to project the point cloud data 203 onto a two-dimensional (2-D) plane and obtain a two-dimensional projection image accordingly. For example, the two-dimensional projection image may be a top view. In some embodiments, the two-dimensional projection image may include a plurality of pixels or super-pixels (including auxiliary pixels). The point cloud segmentation unit 210 may also extract various features from the two-dimensional projection image. Each type of feature may correspond to a channel. In some embodiments, the point cloud segmentation unit 210 may apply the neural network 205 to the extracted features to determine attributes of the pixels, which may indicate a likelihood that the pixels belong to an obstacle. For example, the attributes may include a distance between the pixel and a center of the obstacle, a confidence level, a likelihood that the pixel is part of the obstacle, and a height of the obstacle.

There may be a plurality of segments of pixels corresponding to the obstacle based on the attribute predictions for the pixels. For example, each segment may include a plurality of pixels having an attribute indicating that the pixel is part of an obstacle, e.g., a likelihood greater than a threshold. In some embodiments, some segments belonging to the same obstacle may still be separated from each other, and some gaps may exist between the segments. In some embodiments, the point cloud segmentation unit 210 may aggregate those segments by modifying the prediction results of the pixels that form the gap.

Then, the point cloud generating unit 210 may separate the point cloud data corresponding to the obstacle from the remaining point cloud data. Thus, the remaining point cloud data contains at least two holes caused by obstacles. In some implementations, these holes may be later filled with estimated point cloud data to compensate for data distortion caused by obstacles.

In some embodiments, the hole locating unit 212 may be configured to locate holes in the point cloud data 203. In some embodiments, locating the hole includes determining a plane in which the hole lies and a bounding box that surrounds the obstacle. Although the positioning of the holes is described in the present application, it is envisaged that alternatively obstacles may be positioned instead of holes and the functions performed may be equivalent.

The unobstructed point cloud estimation unit 214 may estimate point cloud data to fill the hole as if the lidar measurements were not obstructed by an obstacle. In some embodiments, the laser path may be simulated based on the planar position of the obstacle. The laser path may then be extended across the obstacle to find the point where it makes contact with the road. Point cloud data for the point may be estimated.

The map correction unit 216 may be configured to correct the high definition map using the estimated point cloud data. In some embodiments, the map correction unit 216 may "fix" the hole with the estimated point cloud data. In some embodiments, the repair is confined within a bounding box that surrounds the obstacle. Repairing may include replacing, modifying, relocating, or otherwise manipulating the data. In some embodiments, the point cloud data 203 may include at least two frames of point clouds captured as the lidar moves along the trajectory. As described above, the map correction unit 216 may repair each point cloud frame with the corresponding unobstructed point cloud data and aggregate the repaired point cloud frames. The point cloud data frames after fusion repair can generate a high-definition map which can better cover the hole area.

Memory 206 and storage 208 may comprise any suitable type of mass storage to store any type of information that processor 204 may need to operate. The memory 206 and storage 208 may be volatile or non-volatile, magnetic, semiconductor, tape, optical, erasable, non-erasable, or other types of storage devices or tangible (i.e., non-transitory) computer-readable media, including but not limited to read-only memory, flash memory, dynamic random access memory, and static random access memory. The memory 206 and/or storage 208 may be configured to store one or more computer programs that may be executed by the processor 204 to perform the map correction functions disclosed herein. For example, the memory 206 and/or storage 208 may be configured to store a program that may be executed by the processor 204 to correct a high-definition map based on segmenting point cloud data captured by a lidar.

The memory 206 and/or storage 208 may be further configured to store information and data used by the processor 204. For example, the memory 206 and/or storage 208 may be configured to store various types of data (e.g., point cloud data, pose information, etc.) captured by the sensors 140 and 150, as well as high definition maps. The memory 206 and/or storage 208 may also store intermediate data such as the neural network 205, and estimated point clouds and the like. Various types of data may be permanently stored, periodically removed, or ignored immediately after processing each data frame.

Fig. 3 illustrates a flow diagram of an exemplary method 300 for correcting a high definition map, according to an embodiment of the application. In some embodiments, the method 300 may be implemented by a map correction system that includes the server 160 or the like. However, the method 300 is not limited to this exemplary embodiment. The method 300 may include steps S302-S320 as described below. It should be understood that some steps may be optional to perform the present application as provided herein. Further, some steps may be performed simultaneously or in a different order than shown in fig. 3.

In step S302, point cloud data captured by the sensor 140 may be received. In some embodiments, the vehicle 100 may be dispatched to make a survey trip to capture data for building a high-definition map. As the vehicle 100 moves along the trajectory, the sensor 140 may capture point cloud data of the surrounding scene. In some embodiments, the point cloud data may be comprised of a plurality of frames, each captured at a particular point in time when the vehicle is at a particular location on the trajectory. Consistent with the present application, there may be other objects on the road that obstruct the laser path of the lidar, resulting in distortion of the acquired point cloud data.

In step S304, the server 160 may project the 3-D point cloud data received in step S304 onto a plane to create a two-dimensional projection image. In some embodiments, the projection is vertically downward such that the two-dimensional projection image obtained is a top view. Projecting the 3-D point cloud data into the 2-D image may remove depth information and simplify the segmentation process.

Steps S306-S312 collectively perform segmentation of the point cloud data to separate distorted data corresponding to the obstacle from the remaining undistorted point cloud data. In some embodiments, the segmentation may be based on a machine learning model, such as a neural network. Neural networks, also known as artificial neural networks, are computational models that simulate biological neural networks to accomplish learning tasks. For example, neural networks may be trained to learn which portion of the point cloud data is reflected from obstacles other than roads obtained from lasers.

FIG. 4 illustrates an exemplary convolutional neural network 400 for segmenting point cloud data according to embodiments of the application in some embodiments, convolutional neural network 400 may include an encoder 420 and a decoder 430. Each of encoder 420 and decoder 430 may include a plurality of different layers (not shown.) examples of different layers may include one or more convolutional layers, non-linear operator layers (e.g., modified linear unit (Re L u) functions, sigmoid functions, or hyperbolic tangent functions), pooling or sub-sampling layers, fully-connected layers, and/or final loss layers.

The number of different layers in between from the input layer to the output layer can become very large, increasing the complexity of the architecture of the convolutional neural network model.

As used herein, the convolutional neural network model used by the disclosed segmentation method may refer to any neural network model that is described, adapted, or modified based on the structure of the convolutional neural network. For example, convolutional neural network models used for segmentation in embodiments of the present application may optionally include intermediate layers between the input layer and the output layer, such as one or more deconvolution layers, upsampling or uppooling layers, pixel direction prediction layers, and/or copy and crop algorithm layers.

Returning to fig. 3, in step S306, the server 160 may extract a feature from each pixel of the two-dimensional projection image. Examples of extracted features may include pixel values, texture around the pixel, contrast with neighboring pixels, relative position of the pixel in the image, and so forth. Convolutional neural network 400 may receive as input extracted features 402. In some embodiments, the features may be arranged as a W x H C matrix, where W is the pixel row number, H is the pixel column number, and C is the channel feature number.

In step S308, convolutional neural network 400 may be applied to the extracted features to predict pixel attributes. For example, the attributes may include a distance 404 between the pixel and a center of the obstacle, a confidence level 406, a probability 408 representing a likelihood that the pixel is part of the obstacle, and a height of the obstacle 410.

In step S310, the server 160 may identify a segment including a plurality of pixels having an attribute indicating that the pixel is part of an obstacle. In some embodiments, the attribute may be compared to some predetermined threshold to determine whether the pixel should be classified as an obstacle. For example, a pixel with a probability 408 of 95% may be classified as part of an obstacle after being compared to a threshold of 90%. Alternatively or additionally, pixels having a distance 404 from the center of the obstacle that is less than 5 pixels wide may be marked as being part of the obstacle.

Pixel-by-pixel prediction can sometimes result in adjacent but disjointed segments. For example, there may be some sporadic non-obstacle pixels or a small gap between two connected segments. At step S312, the server 160 may aggregate the segments by modifying the prediction results for those pixels in the gap. For example, pixel-by-pixel prediction may find a segment corresponding to a pedestrian's head and an adjacent segment corresponding to her torso. However, pixels of the pedestrian's neck region may be predicted not to be part of an obstacle initially because, for example, certain attributes of the pixels do not meet the criteria for classifying them as corresponding to an obstacle. However, by clustering, the head segment and the torso segment may be connected by modifying the pixels between the segments that correspond to the obstacle.

Due to the obstacle, a hole remains in the point cloud data. In step S314, the server 160 may locate the hole based on the detected obstacle. In some embodiments, the server 160 may first determine a bounding box for the obstacle, i.e., the boundary of the corresponding hole. The server 160 may further determine a first plane of the point cloud outside the bounding box. At the same time, the server 160 determines a second plane ("reference plane") of the point cloud data near the origin (where the sensor 140 is located) and determines the angle between the first and second planes. If the angle is less than the threshold, the server 160 sets the first plane as the target plane.

In step S316, the server 160 may estimate obstacle-free point cloud data for the hole if no obstacle exists. Fig. 5 illustrates an exemplary method of estimating unobstructed point cloud data according to an embodiment of the application. As shown in fig. 5, the vehicle 500 may be between plane b1 and plane b 2. The laser is emitted by the lidar at the origin O. When there is no obstacle in the laser path, the laser will collide with the road, for example at point a, and be reflected by the road. Due to the presence of the vehicle 500, the laser path may hit point B in plane B1 and be reflected by the vehicle 500. Without the vehicle 500, the laser path OB would have extended and eventually hit the road at point B'. Similarly, another laser path may strike plane b1 at point C, and once extended, it will strike the road at point C'. Accordingly, the acquired cloud data corresponding to points C and B may be used to estimate unobstructed point cloud data corresponding to points C 'and B'. In some embodiments, each point within the target bounding box (between point 501 and point 502) in plane b1 may find a corresponding point on the road. Accordingly, the point data obtained from the plane b1 may be assigned to or otherwise used to estimate the unobstructed point cloud data for the corresponding point on the road.

In step S318, the server 160 may repair the hole using the estimated obstacle-free point cloud data. For example, the corresponding unobstructed point cloud data can be used to fill in holes, or otherwise modify the data for holes. In step S320, the repair may be filtered through the target bounding box. For example, as shown in FIG. 5, the hole being repaired is limited to the projection of the target bounding box (between point 501 and point 502) onto the road. Filtering repairs may reduce noise and improve the correction results.

Another aspect of the application relates to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform a method as described above. The computer readable medium includes volatile or nonvolatile, magnetic, semiconductor, tape, optical, erasable, non-erasable, or other types of computer readable medium or computer readable storage device. For example, a computer-readable medium as in the present application may be a storage device or a storage module having stored thereon computer instructions. In some embodiments, the computer readable medium may be a disk or flash drive having computer instructions stored thereon.

It will be apparent that various modifications and variations can be made in the system and related methods of the present application by those of ordinary skill in the art. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the system and associated method of the present application.

It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.