阅读说明：本技术 后向车道检测覆盖 (Backward lane detection coverage ) 是由 M·W·特哈亚尔 A·J·里德 于 2018-12-28 设计创作，主要内容包括：后方车道显示系统包括：图像传感器,所述图像传感器被配置成捕获包括车辆前方的车道标记的一个或多个图像；一个或多个显示镜；以及处理器,所述处理器与所述图像传感器和所述显示镜通信,其中所述处理器被配置成处理所述一个或多个图像以预测车道标记将出现在所述车辆后方的位置；以及在所述一个或多个显示镜中显示预测的车道标记的表示。(The rear lane display system includes: an image sensor configured to capture one or more images including lane markings forward of a vehicle; one or more display mirrors; and a processor in communication with the image sensor and the display mirror, wherein the processor is configured to process the one or more images to predict where lane markings will appear behind the vehicle; and displaying a representation of the predicted lane markings in the one or more display mirrors.)

1. A rear lane display system, comprising:

an image sensor configured to capture one or more images including lane markings forward of a vehicle;

one or more display mirrors; and

a processor in communication with the image sensor and the display mirror, wherein the processor is configured to:

processing the one or more images to predict a location at which the lane marker will appear behind the vehicle; and

displaying a representation of the predicted lane markings in the one or more display mirrors.

2. The rear lane display system of claim 1 wherein the image sensor comprises a camera having an infrared light source.

3. The rear lane display system of claim 1 or 2, wherein the one or more display mirrors comprise an interior rearview mirror assembly.

4. The rear lane display system of any of the preceding claims, wherein the one or more display mirrors comprise a side exterior rear view mirror assembly.

5. The rear lane display system of any of the preceding claims, wherein the one or more display mirrors comprise:

a partially reflective and partially transmissive electro-optic mirror element; and

a display configured to provide coverage including the predicted lane markings.

6. The rear lane display system of any of the preceding claims, wherein the processor is configured to process the one or more images captured from the image sensor by:

receiving a current image from the image sensor;

determining coordinates of lane marker edge points in the current image;

modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image; and

fitting a line to the modified coordinates for display in the one or more display mirrors.

7. The rear lane display system of any of the preceding claims wherein determining coordinates of lane marker edge points in the current image comprises identifying lane marker positions with a lane marker detector.

8. The rear lane display system of any of the preceding claims, wherein modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image comprises:

establishing a coordinate system of the vehicle, wherein an x-axis extends to the left and right sides of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to the front and rear of the vehicle;

rotating the coordinates about x, y, and z axes; and

translating the coordinate along the z-axis.

9. The rear lane display system of any of the preceding claims, wherein modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image comprises removing lane marker edge points that exceed a predetermined distance from the vehicle.

10. The rear lane display system of any of the preceding claims wherein fitting a line to the modified coordinates for display in the one or more display mirrors comprises using a model comprising one or more of a pair of polynomials calculating X and Y values for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

11. A method for displaying rear lane markings, comprising:

capturing one or more images including lane markings forward of the vehicle;

processing the one or more images to predict a location at which the lane marker will appear behind the vehicle; and

displaying a representation of the predicted lane markings in one or more display mirrors.

12. The method of claim 11, comprising capturing the one or more images using a camera having an infrared light source.

13. The method according to claim 11 or 12, wherein displaying the representation of the predicted lane markings in one or more display mirrors comprises providing an overlay of the predicted lane markings on a partially reflective and partially transmissive electro-optic mirror element.

14. The method of any preceding claim, comprising displaying a representation of the predicted lane markings using an interior rearview mirror assembly.

15. The method of any preceding claim, comprising displaying a representation of the predicted lane markings using a side exterior rearview mirror assembly.

16. The method of any preceding claim, wherein processing the one or more images captured from the image sensor comprises:

receiving a current image from the image sensor;

determining coordinates of lane marker edge points in the current image;

modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image; and

fitting a line to the modified coordinates for display in the one or more display mirrors.

17. The method according to any of the preceding claims, wherein determining coordinates of lane marker edge points in the current image comprises identifying lane marker positions with a lane marker detector.

18. The method of any preceding claim, wherein modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image comprises:

establishing a coordinate system of the vehicle, wherein an x-axis extends to the left and right sides of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to the front and rear of the vehicle;

rotating the coordinates about x, y, and z axes; and

translating the coordinate along the z-axis.

19. The method of any preceding claim, wherein modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image comprises removing lane marker edge points that exceed a predetermined distance from the vehicle.

20. The method of any of the preceding claims, wherein fitting a line to the modified coordinates for display in the one or more display mirrors comprises using a model comprising one or more of a pair of polynomials that calculate x and y values for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

Technical Field

The disclosed embodiments relate generally to a driver assistance system and method thereof, and more particularly to a system for rear lane marker recognition and tracking.

Background

Since the road surface behind the vehicle is usually poorly illuminated, it may be difficult to distinguish between the lane lines and the positions of other vehicles relative to the lane lines when using the rear view mirror.

Disclosure of Invention

The disclosed embodiments relate to a rear lane display system, which includes: an image sensor configured to capture one or more images including lane markings forward of a vehicle; one or more display mirrors; and a processor in communication with the image sensor and the display mirror, wherein the processor is configured to: processing the one or more images to predict a location at which the lane marker will appear behind the vehicle; and displaying a representation of the predicted lane markings in the one or more display mirrors.

The image sensor may include a camera having an infrared light source.

The one or more display mirrors may comprise an interior rearview mirror assembly.

The one or more display mirrors may comprise a side exterior rear view mirror assembly.

The one or more display mirrors may include: a partially reflective and partially transmissive electro-optic mirror element; and a display configured to provide coverage of the predicted lane markings.

The processor may be configured to process one or more images captured from the image sensor by: receiving a current image from the image sensor; determining coordinates of lane marker edge points in the current image; modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image; and fitting a line to the modified coordinates for display in the one or more display mirrors.

Determining coordinates of lane marker edge points in the current image may include identifying lane marker positions with a lane marker detector.

Modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image may comprise: establishing a coordinate system of the vehicle, wherein an x-axis extends to the left and right sides of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to the front and rear of the vehicle; rotating the coordinates about the x, y, and z axes; and translating the coordinate along the z-axis.

Modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image may include removing lane marker edge points that exceed a predetermined distance from the vehicle.

Fitting a line to the modified coordinates for display in the one or more display mirrors may comprise using a model comprising one or more of a pair of polynomials that calculate X and Y values for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

The disclosed embodiments also relate to a method for displaying rear lane markings, comprising: capturing one or more images including lane markings forward of the vehicle; processing the one or more images to predict a location at which the lane marker will appear behind the vehicle; and displaying a representation of the predicted lane markings in one or more display mirrors.

The method may include capturing the one or more images using a camera having an infrared light source.

The method may include providing coverage of the predicted lane markings on the partially reflective and partially transmissive electro-optic mirror elements.

The method may include displaying a representation of the predicted lane markings using an interior rearview mirror assembly.

The method may include displaying a representation of the predicted lane markings using a side exterior rearview mirror assembly.

Processing the one or more images captured from the image sensor may include: receiving a current image from the image sensor; determining coordinates of lane marker edge points in the current image; modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image; and fitting a line to the modified coordinates for display in the one or more display mirrors.

Determining coordinates of lane marker edge points in the current image may include identifying lane marker positions with a lane marker detector.

Modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image may comprise: establishing a coordinate system of the vehicle, wherein an x-axis extends to the left and right sides of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to the front and rear of the vehicle; rotating the coordinates about the x, y, and z axes; and translating the coordinate along the z-axis.

Modifying the coordinates to reflect a change in vehicle orientation between receiving the current image and a previous image may include removing lane marker edge points that exceed a predetermined distance from the vehicle.

Fitting a line to the modified coordinates for display in the one or more display mirrors may comprise using a model comprising one or more of a pair of polynomials that calculate x and y values for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

These and other aspects, objects, and features of the present disclosure will be understood and appreciated by those skilled in the art upon studying the following specification, claims, and appended drawings. It will also be appreciated that features of each example disclosed herein may be used in combination with or as an alternative to features of other examples.

Drawings

The foregoing aspects and other features of the embodiments are explained in the following description, taken in connection with the accompanying drawings, wherein:

FIG. 1 illustrates a block diagram of an exemplary vehicle control system, in accordance with the disclosed embodiments;

FIG. 2 depicts a diagram of a vehicle incorporating the disclosed embodiments;

FIG. 3 depicts various components of an example external rear view mirror assembly;

FIG. 4 shows an example of an interior rearview mirror assembly;

FIG. 5 illustrates a process for predicting and displaying the appearance of rear lane markings;

FIG. 6 illustrates an exemplary forward display of a system for recognizing left and right lane markings;

FIG. 7 illustrates an exemplary view of a simulated lane line overlaid on an interior rearview display;

FIG. 8 depicts a left simulated lane line overlaid on a left external rear view display; and

FIG. 9 depicts a right side simulated lane line overlaid on a right side external rear view display.

Detailed Description

Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of method steps and apparatus components related to vehicle systems for determining lane markings. Accordingly, the apparatus components and method steps have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by the word "comprising … … a" does not, without further limitation, exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

It will be appreciated that embodiments of the invention described herein may be comprised of one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the vehicle system and method for determining lane markings described herein. The non-processor circuits may include, but are not limited to, signal drivers, clock circuits, power source circuits, and user input devices. Thus, these functions may be interpreted as steps of a method of determining lane markings.

Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs), in which each function or some combinations of certain of the functions are implemented as custom logic. It is understood that any number of methods may be used in combination. Thus, methods and means for these functions have been described herein. Moreover, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and integrated circuits with minimal experimentation.

FIG. 1 illustrates a block diagram of an exemplary vehicle control system 100, in accordance with the disclosed embodiments. The control system 100 may include a forward image sensor 105A, a rearward image sensor 105B, an image controller 110, a processor 115, a lane marker detector 120, a lane departure detector 125, a rearward lane projection circuit 130, and a vehicle interface 140. The lane marker detector 120 includes a left buffer 135 for storing left lane marker edge points and a right buffer 140 for storing right lane marker edge points.

The forward image sensor 105A and the backward image sensor 105B may be cameras including charge coupled devices or CMOS devices. The forward image sensor 105A and the backward image sensor 105B may also include one or more light sources, such as near infrared light sources, and may generally operate to provide a series of image frames. Exemplary image sensors may include those described in the following patents: U.S. patent No. 7,432,967 entitled "Control Circuit for Image Array Sensors"; united states patent No. 6,008,486 entitled "Wide Dynamic Range Optical Sensor"; united states patent No. 7,567,291 entitled "Vehicle Vision System"; U.S. Pat. No. 7,683,326 entitled "Vehicle Vision System with High Dynamic Range"; united states patent No. 7,423,821 entitled "Improved Vision System"; and united states patents nos. 8,289,430 and 8,305,471, both entitled "Improved Imaging Device"; all of these patents are incorporated by reference herein in their entirety.

The image controller 110 may control and receive data from the forward image sensor 105A and the backward image sensor 105B, and may also be used to perform pre-processing functions such as image auto-exposure, dynamic range compression, filtering, and color calculation.

The processor 115 may be a microprocessor that is discreetly configured or instantiated using, for example, one or more components, such as an FPGA, or a combination of discrete components and an FPGA. The processor 115 may request images from the image controller 106 and may also be enabled to directly control and receive images from the forward image sensor 105A and the backward image sensor 105B. The lane marker detector 120 and the lane departure detector 125 receive image frames generated by the forward image sensor 105A, wherein the lane marker detector 120 is operable to identify lane marker positions on the road and the lane departure detector 125 is operable to detect a departure from the driving lane. The rear lane projection circuit 130 may operate on data from one or more of the lane marker detector 120 and the lane departure detector 125 to provide an image of the lane behind the vehicle that will be superimposed on the image captured by the rear image sensor 105B.

In some embodiments, the lane marking detector 120, the deviation detector 125, and the rear lane projection circuit 130 may be embodied in computer code embodied on a medium readable by the processor 115, and the processor 115 may be configured to perform lane marking and lane deviation detection and provide a rear projection of the lane. In other embodiments, the lane marker detector 120, the lane departure detector 125, and the rear lane projection circuit 130 may be embodied as discrete circuits including one or more processors, memories, state machines, ASICs, FPGAs, or other components that may be operable to perform the functions described herein.

Typical systems for lane marker detection and lane departure detection may include those described in the following patents: U.S. Pat. No. 7,881,839 entitled "Image Acquisition and Processing Systems for Vehicle Equipment Control"; united states patent No. 8,543,254 entitled "Vehicular Imaging System and Method for Determining road Width"; U.S. patent No. 9,098,751 entitled "System and Method for Periodic Lane Marker Identification and Tracking"; and U.S. patent No. 9,230,183 entitled "Automatic Vehicle Equipment Monitoring, Warning, and Control System"; all of these patents are incorporated by reference herein in their entirety. Information from the lane marker detector 120 may be projected for viewing, for example, using a heads-up display.

Fig. 6 illustrates an exemplary display of recognized left and right lane markings 605, 610 that may be viewed through or displayed on the windshield of the vehicle.

The vehicle interface 140 is operable to receive various vehicle parameters from other vehicle systems, including parameters such as speed, pitch, roll, yaw, steering wheel position, and vehicle direction, and provide these parameters to the lane marker detector 120, the lane departure detector 125, and the rearward lane projection circuitry 130.

FIG. 2 depicts a diagram of a vehicle 200 incorporating the disclosed embodiments. The vehicle 200 may include: a left exterior mirror assembly 205, a right exterior mirror assembly 210, a center high stop light 215, a-pillars 220a, 220B, B-pillars 225a, 225B, C-pillars 230a, 230B, and an interior mirror assembly 235. It should be understood that the exterior rear view mirror assemblies 205, 210 and interior rear view mirror assembly 235 may be automatic dimming electro-optic mirrors and may incorporate one or more displays, as will be explained below. It should also be understood that although the forward image sensor 105A, the image controller 110, the processor 115, the lane marker detector 120, and the lane departure detector 125 may be positioned in any suitable location, in some embodiments, the forward image sensor 105A, the image controller 110, the processor 115, the lane marker detector 120, and the lane departure detector 125 may be located near, on, or within any of the following: a left exterior mirror assembly 205, a right exterior mirror assembly 210, a center high stop light 215, a-pillars 220a, 220B, B-pillars 225a, 225B, C-pillars 230a, 230B, and an interior mirror assembly 235.

In some embodiments, forward image sensor 105A may be located on a mount of interior rearview mirror assembly 235, and in one or more embodiments, image controller 110 and processor 115 may be co-located with forward image sensor 105A. In one or more embodiments, the rearward facing image sensor 105B may be located in the center high stop light 215 or in a mounting 245 on the rear of the vehicle 200.

Fig. 3 depicts various components of an example of a left side exterior rear view mirror assembly 205 in accordance with the disclosed embodiments. The left side exterior rear view mirror assembly 205 can include a electro-optic mirror element 305 and an exterior rear view display 310 mounted within a housing 315. The left side exterior rear view mirror assembly 205 can also include one or more filters 320, electrical connections, mounts, and other components for mounting the exterior rear view mirror assembly 300 to the vehicle 200 and for providing a side rear view of the vehicle 200. The electro-optic mirror element 305 may be partially reflective and partially transmissive such that information displayed by the external rear view display 310 is visible. The external rearview display 310 may utilize an LCD, LED, OLED, plasma, DLP, or any suitable display technology, and at least operates to provide an overlay of information that complements the reflected image provided by the electro-optic mirror element 305. In some embodiments, the external rear view display 310 may provide a partial rear side view that may be provided by the rearward image sensor 105B and may include a predicted lane marking indication, while in other embodiments, the external rear view display 310 may provide an entire rear side view that includes a predicted lane marking indication. It should be understood that the right side exterior rear view mirror assembly 210 may include the same components and may provide the same partial or complete view.

Fig. 4 illustrates an example of an interior rearview mirror assembly 235 in accordance with the disclosed embodiments. The interior rearview mirror assembly 235 can include an electro-optic element 405, front and rear shields 410, 415 surrounding an interior rearview display 420, a housing 425, and a mounting member 430. The electro-optic mirror element 405 of the interior rearview mirror assembly 235 may be partially reflective and partially transmissive so that information displayed by the interior rearview display 410 is visible to a user. The internal rearview display 410 may utilize LCD, LED, OLED, plasma, DLP, or any suitable display technology and at least operates to provide an overlay of information that complements the reflective image provided by the electro-optic mirror element 405. In some embodiments, the interior rearview display 410 may occupy only a portion of the surface of the electro-optic mirror element 405 and display a partial rear view provided by the rearward image sensor 105B that includes the predicted lane marking indication, while in other embodiments, the interior rearview display 410 may occupy most or substantially all of the surface of the electro-optic mirror element 405 and may provide an entire rear view provided by the rearward image sensor 105B that includes the predicted lane marking indication. Exemplary interior rearview Mirror assemblies can include those described in U.S. patent No. 9,598,018 entitled "Display Mirror Assembly" and U.S. patent application No. 2017/00880055 entitled "Full Display Mirror", all of which are incorporated herein by reference in their entirety.

As mentioned above, the rear lane projection circuit 130 may operate on data from one or more of the lane marker detector 120 and the lane departure detector 125 to provide a display of the lane behind the vehicle 200. For purposes of the disclosed embodiment, the coordinate system relative to the vehicle 200 is defined in terms of a three-dimensional coordinate system having an origin at a point on the ground centered between the two rear tires of the vehicle 200. Each point in the coordinate system is defined with respect to the origin in terms of (x, y, z), where the x-axis extends to the left and right of the vehicle, where x is the distance to the right of the center of the vehicle, and-x is the distance to the left of the center of the vehicle, the y-axis extends below and above the vehicle, where y is the distance below ground level, and-y is the distance above ground level, and the z-axis extends to the front and rear of the vehicle, where z is the distance toward the front of the vehicle 200, and-z is the distance toward the rear of the vehicle 200. The position and orientation of the rearward facing image sensor 105B may be calibrated relative to the coordinate system of the vehicle. In addition, the focal length, optical center, and lens distortion coefficient of the backward image sensor 105B can also be calibrated.

Fig. 5 shows a process 500 for predicting where lane markings will appear behind the vehicle 200 and displaying the lane markings to a user.

As shown in block 505, the lane marker detector 120 receives an image frame from the forward image sensor 105A. As each new frame is received, the coordinates of a point on the left lane marker in the image frame (referred to as the left edge point) are determined within the coordinate system relative to the vehicle 200 and stored in the left buffer 135, as shown in block 510. The coordinates of a point on the right lane marker in the image frame (referred to as the right edge point) are determined within the coordinate system relative to the vehicle 200 and stored in the right buffer 140, as shown in block 515. As each frame is received, the rear lane projection circuit 130 calculates the amount of time between receiving the current frame and the previous frame, as shown in block 520.

In some embodiments, forward image sensor 105, image controller 110, or processor 115 may timestamp frames when generating or retrieving the frames, and lane marker detector 120 may use the timestamps to calculate the amount of time between successive frames. In other embodiments, the lane marker detector 120 may reference a clock signal, or perform some other operation to calculate the amount of time between receiving frames.

As shown in block 525, the lane marker detector 120 performs the following operations on the coordinates of each edge point stored in the left buffer 135 and the right buffer 140 to reflect the vehicle orientation and position change between receiving the current frame and the previous frame:

the coordinates of each edge point are rotated around the y-axis (yaw) using the following matrix

The coordinates of each edge point are rotated around the x-axis (pitch) using the following matrix

Rotate each edge point around the z-axis using the following matrix (roll); and

each edge point is translated along the z-axis using the following matrix (velocity)

Alternatively, the operation on the coordinates of each edge point stored in the left buffer 135 and the right buffer 140 may be compiled into a single matrix multiplication:

as shown in block 530, points in left bumper 135 and right bumper 140 that exceed a certain distance from the vehicle due to vehicle travel may then be removed from the bumpers. By way of non-limiting example, points more than 50 meters behind the vehicle 200 may be removed.

The model may then be used to fit a line to the modified coordinates in each buffer independently, as shown in blocks 535 and 540. For example, in some embodiments, a pair of polynomials to calculate the X and Y values for a given z value may be used. For example, f can be used_x(z)＝a₃z³+a₂z²+a₁z+a₀And f_y(z)＝b₃z³+b₂z²+b₁z+b₀. In other embodiments, a least squares regression, a linear regression, a Theil-Sen estimator, or any suitable modeling technique may be utilized to fit a line to the modified coordinates.

The simulated fit line may then be drawn on the view displayed on the internal rearview display 410, as shown in block 545. In some embodiments, the projection matrix may be used to determine the pixels of the rear view display 410 that correspond to points of the simulated fit line. FIG. 7 illustrates an exemplary view shown on the internal rearview display 410 including simulated fit lines 705, 710.

Alternatively, the line of modified coordinates fitted into the left bumper may be overlaid in the view provided by the left exterior rear view mirror assembly 205, as shown in fig. 8, and the line of modified coordinates fitted into the right bumper may be overlaid in the view provided by the right exterior rear view mirror assembly 210, as shown in fig. 9. Other image operations may also be performed, such as enhancing contrast around the simulated fit line, or increasing brightness.

It should be noted that the embodiments described herein may be used alone or in any combination thereof. It should be understood that the foregoing description is only illustrative of embodiments. Various alternatives and modifications can be devised by those skilled in the art without departing from the embodiments. Accordingly, the present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

Various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings. However, all such and similar modifications of the teachings of the disclosed embodiments will still fall within the scope of the disclosed embodiments.

The various features of the different embodiments described herein are interchangeable one with another. Various described features and any known equivalents may be mixed and matched in accordance with the principles of the present disclosure to construct additional embodiments and techniques.

Furthermore, it may be more advantageous to use some of the features of the example embodiments without the corresponding use of other features. Accordingly, the foregoing description should be considered as merely illustrative of the principles of the disclosed embodiments and not in limitation thereof.