阅读说明：本技术 车辆数据增强 (Vehicle data enhancement ) 是由 古萨姆·肖林格 S·孙达尔 于 2021-04-13 设计创作，主要内容包括：本公开提供了“车辆数据增强”。一种系统,其包括处理器和存储器,所述存储器包括指令,所述指令将由处理器执行以从车辆接收一个或多个图像的指令,其中包括在所述车辆中的计算机中的第一深度神经网络未能确定所述一个或多个图像中的第一对象的取向。所述指令可包括用于进行以下操作的另外的指令：用小样本图像转换器生成多个修改图像,其中所述修改图像各自包括基于所述第一对象的修改对象。所述指令可包括用于进行以下操作的另外的指令：基于所述多个修改图像来重新训练所述深度神经网络以确定所述第一对象的取向,以及将所述重新训练的深度神经网络下载到所述车辆。(The present disclosure provides "vehicle data enhancement". A system comprising a processor and a memory, the memory including instructions to be executed by the processor to receive one or more images from a vehicle, wherein a first deep neural network included in a computer in the vehicle fails to determine an orientation of a first object in the one or more images. The instructions may include further instructions for: generating, with a small sample image converter, a plurality of modified images, wherein the modified images each include a modified object based on the first object. The instructions may include further instructions for: retraining the deep neural network based on the plurality of modified images to determine an orientation of the first object, and downloading the retrained deep neural network to the vehicle.)

1. A method, comprising:

receiving one or more images from a vehicle, wherein a first deep neural network included in a computer in the vehicle fails to determine an orientation of a first object in the one or more images;

generating a plurality of modified images with a small sample image-to-image converter, wherein the modified images each include a modified object based on the first object;

training a second deep neural network based on the plurality of modified images to determine the orientation of the modified object; and

downloading the trained second deep neural network to the vehicle.

2. The method of claim 1, wherein the small sample image-to-image converter is based on generating a countermeasure network and is trained using a plurality of real-world images and a plurality of composite images to convert one or more input images to the plurality of modified images based on the one or more images from the vehicle.

3. The method of claim 2, further comprising training the small sample image-to-image converter based on the plurality of real-world images and a plurality of composite images, the plurality of composite images including a vehicle trailer and ground truth data regarding an angle of each vehicle trailer relative to the vehicle for each real-world image, respectively.

4. The method of claim 3, wherein the plurality of real-world images and the plurality of composite images comprise a plurality of weather and lighting conditions, including day, night, rain, snow, and a plurality of sun directions.

5. The method of claim 4, wherein the plurality of vehicle trailers in the plurality of real world images and the plurality of composite images comprises a plurality of vehicle trailer types with a plurality of trailer components configured in various configurations.

6. The method of claim 1, wherein the first object is a vehicle trailer and determining the orientation of the first object comprises determining an angle of the vehicle trailer relative to the vehicle for each modified image, respectively.

7. The method of claim 1, wherein the plurality of modified images each include ground truth data including a respective angle of the vehicle trailer relative to the vehicle.

8. The method of claim 1, wherein each modified image includes the vehicle trailer oriented at an angle relative to the vehicle equal to the angle of the object appearing in the input image.

9. The method of claim 1, wherein each modification object based on the first object comprises a vehicle trailer component included in the first object having the same configuration as the vehicle trailer component included in the first object.

10. The method of claim 1, wherein the first and second deep neural networks comprise convolutional layers and fully-connected layers, wherein the convolutional layers process input image data and the fully-connected layers process output from the convolutional layers to determine an angle of the object relative to a vehicle.

11. The method of claim 1, wherein the small sample image-to-image converter comprises a content encoder layer, a class encoder layer, and a decoder layer, wherein the content encoder layer generates potential residual features based on an input object type and an object orientation, and the decoder layer generates an output image based on the class encoder layer.

12. The method of claim 1, further comprising operating a vehicle by inputting an image to the second deep neural network to determine a trailer angle of an attached trailer.

13. The method of claim 1, further comprising operating a vehicle by determining a vehicle path based on the input image including an attached trailer.

14. The method of claim 12, further comprising operating the vehicle by commanding a vehicle powertrain, vehicle steering, and vehicle brakes to operate the vehicle along the vehicle path that applies a force to the attached trailer to move the attached trailer to a target position in a target orientation.

15. A system comprising a computer programmed to perform the method of any of claims 1-14.

Technical Field

The present disclosure relates generally to vehicle sensors.

Background

Vehicles may be equipped with computing devices, networks, sensors, and controllers to acquire data about the environment of the vehicle and operate the vehicle based on the data. Vehicle sensors may provide data about a route to be traveled and objects to be avoided in a vehicle environment. Operation of the vehicle may rely on obtaining accurate and timely data about objects in the vehicle's environment while the vehicle is operating on the road.

Disclosure of Invention

The vehicle may be equipped to operate in an autonomous mode and an occupant operating mode. Semi-autonomous or fully autonomous mode means an operating mode in which the vehicle may be partially or fully operated by a computing device that is part of a system with sensors and controllers. The vehicle may be occupied or unoccupied, but in either case, the vehicle may be partially or fully operated without occupant assistance. For purposes of this disclosure, an autonomous mode is defined as a mode in which each of vehicle propulsion (e.g., via a powertrain including an internal combustion engine and/or an electric motor), braking, and steering are controlled by one or more vehicle computers; in the semi-autonomous mode, the vehicle computer controls one or both of vehicle propulsion, braking, and steering. In the non-autonomous mode, none of these are computer controlled.

A computer in the vehicle may be programmed to acquire data about the environment external to the vehicle and use the data to determine a vehicle path on which to operate the vehicle in an autonomous or semi-autonomous mode. The vehicle may be operated on the road by determining commands to instruct powertrain, braking and steering components of the vehicle to operate the vehicle to travel along the path based on the path of the vehicle. The data about the external environment may include the location of one or more objects in the environment surrounding the vehicle, such as a vehicle, a pedestrian, or a vehicle trailer. The data may be processed by a computer in the vehicle and the data may be used to operate the vehicle.

A computer in the vehicle may determine a vehicle path on which to operate the vehicle by inputting vehicle sensor data into a deep neural network trained to recognize and locate objects. For example, the deep neural network may be trained to determine the angle of the vehicle trailer relative to the vehicle. The computer may use the angle data to determine a vehicle path that allows the vehicle to reverse and park the trailer. Deep neural networks may be trained based on a variety of trailer types using sensor data acquired under a variety of environmental conditions. While deep neural networks are trained for a variety of trailer types and a variety of environmental conditions, in actual operation a new trailer type with new vehicle trailer components may be encountered. When this occurs, the deep neural network may not be able to determine angle data about the new trailer type. The techniques discussed herein may use sensor data corresponding to determining a fault with angle data for a new trailer type to generate a plurality of images based on the faulty sensor data. The plurality of generated images may be used to retrain the deep neural network to determine angle data about the new trailer type under a variety of environmental conditions. The retrained deep neural network may be downloaded to the vehicle and used to operate the vehicle based on determining angle data about the new trailer type. Disclosed herein is a method comprising: receiving one or more images from a vehicle, wherein a deep neural network in a computer included in the vehicle fails to determine an orientation of a first object in the one or more images; generating a plurality of modified images with a small sample (few-shot) image converter, wherein the modified images each include a modified object based on the first object; and retraining a deep neural network based on the plurality of modified images to determine the orientation of the modified object. The retrained deep neural network may be downloaded to the vehicle. The small sample image converter may be based on generating a countermeasure network and trained using a plurality of real-world images and a plurality of composite images to convert one or more input images to the plurality of modified images based on the one or more images from the vehicle. The small sample image converter may be trained based on the plurality of real-world images and a plurality of composite images including the vehicle trailer and ground truth data regarding an angle of each vehicle trailer relative to the vehicle for each real-world image, respectively. The plurality of real-world images and the plurality of composite images may include a plurality of weather and lighting conditions, including day, night, rain, snow, and a plurality of sun directions.

The plurality of vehicle trailers in the plurality of real world images and the plurality of composite images may include a plurality of vehicle trailer types with a plurality of trailer components configured in various configurations. The first object may be a vehicle trailer and determining the orientation of the first object may comprise determining an angle of the vehicle trailer relative to the vehicle for each modified image separately. The plurality of modified images may each include ground truth data including a respective angle of the vehicle trailer relative to the vehicle. Each modified image may include the vehicle trailer oriented at an angle relative to the vehicle equal to the angle of the object appearing in the input image. Each modification object may be based on the first object and include a vehicle trailer component included in the first object having the same configuration as the vehicle trailer component included in the first object. The deep neural network may include a convolutional layer and a fully-connected layer, wherein the convolutional layer processes input image data and the fully-connected layer processes output from the convolutional layer to determine an angle of the object relative to the vehicle. The small sample image converter may include a content encoder layer, a class encoder layer, and a decoder layer, wherein the content encoder layer generates potential residual features based on an input object type and an object orientation, and the decoder layer generates an output image based on the class encoder layer. The vehicle may be operated by inputting images to the retrained deep neural network to determine a trailer angle of the attached trailer. The vehicle may be operated by determining a vehicle path based on the input image including the attached trailer. The vehicle may be operated by commanding the vehicle powertrain, vehicle steering, and vehicle brakes to operate the vehicle along the vehicle path that applies a force to the attached trailer to move the attached trailer to a target position in a target orientation.

A computer readable medium storing program instructions for performing some or all of the above method steps is also disclosed. Also disclosed is a computer programmed to perform some or all of the above method steps, the computer comprising computer equipment programmed to: receiving one or more images from a vehicle, wherein a deep neural network in a computer included in the vehicle fails to determine an orientation of a first object in the one or more images; generating a plurality of modified images with a small sample image converter, wherein the modified images each include a modified object based on the first object; and retraining a deep neural network based on the plurality of modified images to determine the orientation of the modified object. The retrained deep neural network may be downloaded to the vehicle. The small sample image converter may be based on generating a countermeasure network and trained using a plurality of real-world images and a plurality of composite images to convert one or more input images to the plurality of modified images based on the one or more images from the vehicle. The small sample image converter may be trained based on the plurality of real-world images and a plurality of composite images including the vehicle trailer and ground truth data regarding an angle of each vehicle trailer relative to the vehicle for each real-world image, respectively. The plurality of real-world images and the plurality of composite images may include a plurality of weather and lighting conditions, including day, night, rain, snow, and a plurality of sun directions.

The instructions may include further instructions wherein the plurality of vehicle trailers in the plurality of real world images and the plurality of composite images may include a plurality of vehicle trailer types with a plurality of trailer components configured in various configurations. The first object may be a vehicle trailer and determining the orientation of the first object may comprise determining an angle of the vehicle trailer relative to the vehicle for each modified image separately. The plurality of modified images may each include ground truth data including a respective angle of the vehicle trailer relative to the vehicle. Each modified image may include the vehicle trailer oriented at an angle relative to the vehicle equal to the angle of the object appearing in the input image. Each modification object may be based on the first object and include a vehicle trailer component included in the first object having the same configuration as the vehicle trailer component included in the first object. The deep neural network may include a convolutional layer and a fully-connected layer, wherein the convolutional layer processes input image data and the fully-connected layer processes output from the convolutional layer to determine an angle of the object relative to the vehicle. The small sample image converter may include a content encoder layer, a class encoder layer, and a decoder layer, wherein the content encoder layer generates potential residual features based on an input object type and an object orientation, and the decoder layer generates an output image based on the class encoder layer. The vehicle may be operated by inputting images to the retrained deep neural network to determine a trailer angle of the attached trailer. The vehicle may be operated by determining a vehicle path based on the input image including the attached trailer. The vehicle may be operated by commanding the vehicle powertrain, vehicle steering, and vehicle brakes to operate the vehicle along the vehicle path that applies a force to the attached trailer to move the attached trailer to a target position in a target orientation.

Drawings

Fig. 1 is a diagram of an exemplary communication infrastructure system.

Fig. 2 is an illustration of an exemplary vehicle trailer.

Fig. 3 is an illustration of two exemplary vehicle trailers.

FIG. 4 is a diagram of an exemplary system for training a deep neural network.

Fig. 5 is a diagram of a small sample image converter.

FIG. 6 is a diagram of a system for retraining a deep neural network using a small sample image converter.

FIG. 7 is a flow diagram of an exemplary process for retraining a deep neural network.

Detailed Description

Fig. 1 is an illustration of a traffic infrastructure system 100 that includes a vehicle 101 that can operate in autonomous ("autonomous" by itself means "fully autonomous" in this disclosure), semi-autonomous, and occupant operation (also referred to as non-autonomous) modes. One or more vehicle 101 computing devices 105 may receive data from sensors 110 regarding the operation of vehicle 100. The computing device 105 may operate the vehicle 101 in an autonomous mode, a semi-autonomous mode, or a non-autonomous mode.

Computing device 105 includes a processor and memory such as is known. Further, the memory includes one or more forms of computer-readable media and stores instructions executable by the processor for performing various operations including as disclosed herein. For example, the computing device 105 may include programming to operate one or more of vehicle braking, propulsion (e.g., control acceleration of the vehicle 101 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior and/or exterior lights, etc., and to determine whether and when the computing device 105 (rather than a human operator) controls such operations.

Computing device 105 may include or be communicatively coupled to more than one computing device (e.g., a controller or the like (e.g., a powertrain controller, a brake controller, a steering controller, etc.) included in vehicle 101 for monitoring and/or controlling various vehicle components 115), for example, via a vehicle communication bus as described further below. The computing device 105 is typically arranged for communication over a vehicle communication network (e.g., including a bus in the vehicle 101, such as a Controller Area Network (CAN), etc.); additionally or alternatively, the vehicle 101 network may include wired or wireless communication mechanisms such as are known, for example, ethernet or other communication protocols.

Computing device 105 may transmit and/or receive messages to and/or from various devices in the vehicle (e.g., controllers, actuators, sensors (including sensor 110), etc.) via the vehicle network. Alternatively or additionally, where computing device 105 actually includes multiple devices, a vehicle communication network may be used for communication between the devices, represented in this disclosure as computing device 105. Further, as mentioned below, various controllers or sensing elements (such as sensors 110) may provide data to computing device 105 via a vehicle communication network.

Additionally, computing device 105 may be configured to communicate with server 125 (e.g., a cloud server or a remote server computer) over a vehicle-to-infrastructure (V-to-I) interface via network 120, including allowing computing device 105 to communicate via, for example, a wireless internet, as described belowOr the network 120 of the cellular network, and the server 125. Thus, computing device 105 may be configured to include a computing device configured to utilize various wired and/or wireless networking technologies (e.g., cellular, wireless, etc.),And wired and/or wireless packet networks), processors, memories, transceivers, etc. The computing device 105 may be configured to communicate with other vehicles 101 using a vehicle-to-vehicle (V-to-V) network (e.g., in accordance with Dedicated Short Range Communications (DSRC) and/or the like), such as formed on a mobile ad hoc network basis among nearby vehicles 101 or formed via V-to-I communications over an infrastructure-based network. Computing device 105 also includes non-volatile memory such as is known. The computing device 105 may record the data by storing the data in non-volatile memory for later retrieval and transmission to the server 125 or user mobile device via the vehicle communication network. The server computer 125 may also serve as the computing device 105 included in the roadside infrastructure, or an edge computing node, where the edge computing node is included in a fixedA computing device 105 in or on an infrastructure element (such as a pole, bridge, wall, building, etc.) and which acquires sensor data and communicates with the vehicle 101 via a V-to-I interface or the like in a local portion of one or more of the roads, parking lots, or parking structures, etc.

As already mentioned, programming for operating one or more vehicle 101 components (e.g., braking, steering, propulsion, etc.) without human operator intervention is typically included in instructions stored in memory and executable by a processor of computing device 105. Using data received in the computing device 105 (e.g., sensor data from the sensors 110, the server computer 125, etc.), the computing device 105 may make various determinations and/or control various vehicle 101 components and/or operations to operate the vehicle 101 without a driver. For example, the computing device 105 may include programming to adjust vehicle 101 operating behaviors (i.e., physical manifestations of vehicle 101 operation) such as speed, acceleration, deceleration, steering, etc., as well as strategic behaviors (i.e., control of operating behaviors that are typically conducted in a manner expected to achieve safe and efficient travel of the route), such as distance between vehicles and/or amount of time between vehicles, lane changes, minimum clearance between vehicles, left turn cross-over path minima, arrival time at a particular location, and intersection (no-signal) minimum arrival time to pass through an intersection.

The term controller as used herein includes computing devices that are typically programmed to monitor and/or control a particular vehicle subsystem. Examples include powertrain controllers, brake controllers, and steering controllers. The controller may be, for example, a known Electronic Control Unit (ECU), possibly including additional programming as described herein. The controller may be communicatively connected to the computing device 105 and receive instructions from the computing device to actuate the subsystems according to the instructions. For example, the brake controller may receive instructions from the computing device 105 to operate the brakes of the vehicle 101.

The one or more controllers for the vehicle 101 may include known Electronic Control Units (ECUs) or the like, including by way of non-limiting example, one or more powertrain controllers, one or more brake controllers, and one or more steering controllers. Each of the controllers may include a respective processor and memory and one or more actuators. The controller may be programmed and connected to a vehicle 101 communication bus, such as a Controller Area Network (CAN) bus or a Local Interconnect Network (LIN) bus, to receive instructions from the computing device 105 and control the actuators based on the instructions.

The sensors 110 may include a variety of devices known to provide data via a vehicle communication bus. For example, a radar fixed to a front bumper (not shown) of vehicle 101 may provide a distance from vehicle 101 to a next vehicle in front of vehicle 101, or a Global Positioning System (GPS) sensor disposed in vehicle 101 may provide geographic coordinates of vehicle 101. Vehicle position and orientation data may also be provided by Inertial Measurement Unit (IMU) sensors. The IMU includes an accelerometer configured to record three-dimensional motion of the vehicle 101. For example, the range provided by the radar and/or other sensors 110 and/or the geographic coordinates provided by the GPS and/or IMU sensors may be used by the computing device 105 to autonomously or semi-autonomously operate the vehicle 101.

The vehicle 101 is typically a land-based vehicle 101 (e.g., a passenger car, a light truck, etc.) capable of autonomous and/or semi-autonomous operation and having three or more wheels. Vehicle 101 includes one or more sensors 110, a V-to-I interface, a V-to-V interface, a computing device 105, and vehicle components 115 (including one or more controllers). The sensors 110 may collect data related to the vehicle 101 and the environment in which the vehicle 101 is operating. By way of example and not limitation, sensors 110 may include, for example, altimeters, cameras, lidar, radar, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, pressure sensors, hall sensors, optical sensors, voltage sensors, current sensors, mechanical sensors (such as switches), and the like. The sensors 110 may be used to sense the operating environment of the vehicle 101, for example, the sensors 110 may detect phenomena such as weather conditions (rain, outside ambient temperature, etc.), road grade, road location (e.g., using road edges, lane markings, etc.), or the location of a target object, such as a neighboring vehicle 101. The sensors 110 may also be used to collect data, including dynamic vehicle 101 data related to the operation of the vehicle 101, such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels applied to vehicle components 115 in the vehicle 101, connectivity between the vehicle components 115, and accurate and timely performance of the vehicle components 115.

Fig. 2 is an illustration of a trailer 200 attached to a vehicle 101. The trailer 200 is an unpowered vehicle that the vehicle 101 may tow. The trailer 200 includes a tongue 202, the tongue 202 including a trailer coupling 204. The trailer hitch 204 connects the trailer 200 to the vehicle 101 by placing the trailer hitch 204 over the trailer hitch 206 to removably connect the trailer hitch 204 to the trailer hitch 206. Trailer ball 206 is attached to vehicle 101 by ball seat 208. The orientation of the trailer 200 relative to the vehicle 101 to which it is connected may be described by the trailer angle 210. The trailer angle 210 is defined as the angle of the trailer centerline 212 relative to the vehicle centerline 214.

The trailer centerline 212 and the vehicle centerline 214 are lines (sometimes referred to as axes) that correspond to the horizontal longitudinal center of the trailer 200 or vehicle 101, respectively, i.e., the centerlines 212, 214 are parallel to the ground (typically the road supporting the trailer 200 and vehicle 101). Thus, the trailer centerline 212 is determined by: a line passing through the geometric center of the top-down image of the trailer 200 and the lateral center of the trailer coupler 204 is constructed, i.e., the trailer centerline 212 extends longitudinally through the trailer 200, assuming the trailer 200 is or can be considered to be generally rectangular (i.e., a geometric rectangle can be assembled to represent the trailer 200), the point at which the centerline 212 is determined is the lateral center of the trailer 200, i.e., the centerline 212 bisects the trailer 200 in the longitudinal direction. Similarly, the vehicle centerline 214 is determined by: a line is constructed through the lateral center of the top-down image of the vehicle 101, considering the vehicle 101 as being generally rectangular, with the centerline 214 further passing through the trailer ball 206 attached to the vehicle 101. Trailer angle 210 is determined by the angle of trailer centerline 212 and vehicle centerline 214; the vertex of angle 210 is formed by the intersection of trailer centerline 212 and vehicle centerline 214 at the center of trailer ball 206.

The trailer angle 210 of the trailer 200 attached to the vehicle 101 may be determined by the computer 105 in the vehicle 101 to operate the vehicle 101 with the attached trailer 200. For example, the trailer angle 210 may be estimated by processing images of the trailer 200, and the trailer angle 210 may be used to reverse the vehicle 101 with the attached trailer 200. Determining the trailer angle 210 of the attached trailer 200 may allow the computer 105 in the vehicle 101 to determine a command to be sent to the vehicle 101 steering component that will cause the vehicle 101 to tow the trailer 200 to retract the trailer 200 to a selected location, such as a parking space.

The video sensor 216 may be included in the vehicle 101. The video sensor 216 may be oriented to provide a field of view that includes a view of the trailer 200 (including the trailer coupling 204, the ball seat 208 attached to the vehicle 101, and the environment on either side of the trailer 200). In some examples, a second video sensor 216 may be included to acquire images including more of the environment on both sides of the trailer 200, and a third video sensor 216 may be included to acquire images from the rear of the trailer 200. The computer 105 may determine a target location of the trailer 200, such as a location of a parking space, dock, or ramp, for example, a location for parking and/or loading or unloading the trailer 200, based on the image 300 acquired by the video sensor 216. For example, the target location may be determined by processing the image 300 with a deep neural network. Based on the determined target position and the determined trailer angle 210, the computer 105 may determine a vehicle path on which to operate the vehicle 101 that will turn the attached trailer 200 in the appropriate direction at the appropriate rate to position the trailer at the target position. It is well known that reversing the vehicle 101 with the trailer 200 attached may cause the trailer to turn in a direction opposite to the direction of turning of the vehicle 101. Thus, the vehicle path determined by the computer 105 to move the trailer 200 into the target position may require, for example, forward and backward motion of the vehicle 101.

Fig. 3 is an illustration of two video images 300a, 300b (collectively referred to as images 300), wherein each video image 300 is acquired by a video sensor 216 included in the vehicle 101. Each video image 300 includes a respective image of a trailer 200a, 200b (collectively, trailers 200). The trailer 200 may have a number of different components configured in a number of different configurations. By trailer part is herein meant a conventional trailer part such as a tongue bar, a box board, a wheel, etc. In video image 300a, the trailer 200a is configured with an a-shaped tongue 202a, while in video image 300b, the trailer 200b is configured with a straight tongue 202 b. The trailer 200a is configured with a bed 314 without sides or doors, while the trailer 200b is configured with sides 320 and a hoistway door 322. The trailer may also be configured to carry a variety of different cargo. The range of cargo may include earth or stone, boxes, furniture, machinery, and the like. Cargo may include vehicles such as tractors, automobiles, and ships. Different types of cargo may require different configurations of trailer 200 components and may change the appearance of the trailer 200 in the video image. For example, changing the appearance of the trailer 200 in the video image may result in the computer 105 not being able to identify the trailer 200 because different trailers have different components.

Different configurations and appearances of the trailer components may present challenges in determining the trailer angle 210 relative to the vehicle 101. The trailer angle 210 may be determined by processing the image 300 using a deep neural network, as will be discussed below in conjunction with fig. 3. Determining the trailer angle 210 with the deep neural network based on the real-world image may depend on the deep neural network having been trained using a training data set that includes images of the trailer 200 constructed with components having the same configuration as the trailer 200 in the real-world image. Having trailer cargo different from that included in the training image may result in the deep neural network failing to determine the trailer angle. Successful determination of the trailer angle 210 is defined as determining that the trailer angle 210 is within +/-5 degrees of the trailer angle 210 that is obtainable by measuring the trailer angle 210 in the real world using physical instruments such as scales and protractors. Measuring the trailer angle with a physical instrument 210 may be used as ground truth for training the deep neural network. The inability to determine the trailer angle 210 includes the depth neural network being unable to determine the trailer angle 210 for a given input image at all.

Generally, determining the trailer angle 210 based on the input image 300 is made even more difficult by changing environmental conditions. The image 300 may include a variety of environmental conditions, including lighting and weather. Lighting conditions may range from day to night, and all conditions in between. The sunlight may cast shadows and introduce highlights that vary depending on the direction of the sunlight relative to the trailer 200. Weather conditions may include rain, snow, fog, and the like. The success rate of determining the trailer angle 210 may be increased by training a deep neural network with images 300 that include environmental conditions that will be encountered in processing images 300 acquired while operating in the real world.

The techniques discussed herein may improve the performance of the computer 105 included in the vehicle 101 in determining the trailer angles 210, 216. These techniques include storing a plurality of images 300 corresponding to a failure to determine the trailer angle 210 by a first deep neural network included in the computer 105 in the vehicle 101. The stored images are transmitted to a cloud-based computer and used to train a small sample image converter as defined and described below with respect to fig. 5 to generate a plurality of training images corresponding to the configuration of the trailer 200 that appears in the fault images 210, 216. The training images are used to retrain the deep neural network to determine the trailer angle 210 based on the trailer 200 configuration appearing in the fault image 210. The retrained deep neural network is downloaded to the computer 105 in the vehicle 101 for inputting images of the vehicle sensors 216 (including images of the trailer 200) and determining the trailer angle 210. The vehicle 101 may then operate using the retrained deep neural network. In examples where the computer 105 in the vehicle 101 includes a deep neural network that is unable to determine the trailer angle 210, a retrained deep neural network may be used in place of the first deep neural network.

Fig. 4 is a diagram of a deep neural network system 400. The deep neural network system 400 includes a deep neural network 402, a training image 404, ground truth 406 corresponding to the training image 404, and a trailer angle 210 output from the deep neural network 402. The deep neural network 402 may include: a plurality of convolutional layers that process input image data including training images 404; and a plurality of fully-connected layers that process outputs from the convolutional layers. The deep neural network 402 may input training image 404 data from the video sensors 216 included in the vehicle 101, or input a composite training image 404 generated by the photorealistic rendering software. In this example, the video sensor 216 is configured to provide a field of view that includes the trailer 200, as discussed above with respect to fig. 2. The first deep neural network 402 is trained by processing training images 404, the training images 404 including the trailer 200 to estimate the trailer angles 210, 216.

The first deep neural network 402 is trained by processing a data set that includes a large number (> 1000) of training images 404 that include multiple trailer 200 types under multiple environmental conditions. Each image in the data set has a corresponding ground truth 406 that specifies the trailer angle 210 of the trailer 200 in the image. The ground truth 406 is data about the training image 404 that is determined by a process independent of the deep neural network 402. The ground truth 406 is considered to represent a real-world measurement. For example, the ground truth 406 trailer angle 210 may be estimated through manual inspection of the images, i.e., the trailer angle 210 in the image data is estimated, for example, using instruments including scales and protractors on a hard copy of the training image 404. In other examples, the ground truth 406 trailer angle 210 may be estimated by measuring the trailer angle 210 in the real world using instruments such as scales and protractors on the real world vehicle 101 and trailer 200 imaged by the video sensor 216. In training, the trailer angle 210 from processing the input training image 404 is backpropagated and compared to the ground truth 406 corresponding to the input training image 404. The back propagation may calculate a loss function based on the trailer angle 210 and the corresponding ground truth 406. The loss function is a mathematical function that maps values such as trailer angle 210 to real numbers corresponding to costs. In this example, the cost may be determined as the difference in degrees between the determined trailer angle 210 and the corresponding trailer angle 210 in the ground truth 406. The loss function determines how close the trailer angle 210 is to the ground truth and is used to adjust the parameters or weights that control the deep neural network 402.

Parameters controlling the processing of the deep neural network 402 are changed until the output trailer angle 210 results match the ground truth 406 for each of the plurality of training images 404 in the training data set. As can be appreciated from various applications of image or pattern recognition using neural networks, the parameters or weights include coefficients of a convolution kernel used to extract features from the input image data. The parameters or weights also include coefficients used by linear and/or nonlinear equations included in the fully-connected layer. The fully connected layer processes the extracted image features output by the convolutional layer. Training the deep neural network 402 typically includes systematically altering these parameters or weights and comparing the output results with expected results corresponding to the ground truth 406. As a result of varying the parameters or weights in a plurality of trials over a plurality of input images, a set of parameters or weights may be determined that achieves a result of matching with the ground truth.

The number of training images 404 in the training dataset may be increased by generating a composite image including the trailer 200 using photorealistic rendering software. Realism is defined as appearing to the observer as if they were viewing a picture of a real scene. Photorealistic rendering software is a software program that can create an image that appears to be captured by a real camera viewing a real scene. An exemplary photorealistic rendering software is a fantasy engine produced by imperial digital corporation of krey, north carolina (zip code 27518).

The photorealistic rendering software inputs data describing the scene in a mathematical format. For example, the trailer 200 to be rendered may be described as a collection of geometric shapes and surface textures. Each trailer type may have a different set of geometries and surface textures. The position and orientation of each trailer 200 to be rendered is also specified. The position and orientation of each trailer 200 may be converted to a trailer angle 210 by specifying the position and field of view of the virtual camera used to determine the rendering process. The trailer angle 210 determined by the input data to the photorealistic rendering software is the ground truth data to be used with the output image data to train the deep neural network 402.

Image data output from the photorealistic rendering software program may be input to the generation countermeasure network to increase the number and realism of images in the training image dataset. The generating an antagonistic network is a neural network that includes a generating network that modifies the input image and a network of discriminators trained to determine whether the modified image is similar to a real image. Image similarity may be determined by comparing images using image subtraction, where a first image is subtracted from a second image, and then the absolute or squared difference between the two images is summed. A small absolute difference or squared difference (< 1% of the total pixel value sum) indicates a similar image. Image similarity may also be determined based on correlation techniques that associate regions of the first image with regions of the second image. A high correlation (> 90%) between two images indicates similar images. The training generates a countermeasure network to make sufficiently realistic modifications to the input composite image for the discriminator network to determine it as "true". The generation countermeasure network can be trained to modify the input image to simulate the effects of different environmental conditions. For example, the generation of the antagonistic network may be trained to modify the composite image of the trailer rendered in full sunlight to appear as if it is raining when generating a realistic image. Generating a countering network may also improve the realism of the composite image by adding visual features to the composite image to make it appear as if it were acquired by the real-world video sensor 216.

As discussed above with respect to fig. 2, the different appearances of the trailer 200 in the image 300 used for training may result in the inability of the deep neural network 402 to determine the trailer angles 210, 216. When the computer 105 determines that the deep neural network 402 cannot determine the trailer angles 210, 216, the first action that the computer 105 may take is to store the fault-causing image 300 and acquire a second image 300 from the vehicle sensors. The second image 300 is processed by the first deep neural network 302 to determine whether the problem is a temporary problem with the first image 300 or whether the second image 300 also causes a malfunction. If the second image 300 also results in the trailer angles 210, 216 not being determined, the second image 300 is stored by the computer 105 and the third image 210 is acquired and processed. If the third image 200, 205 also causes a fault, the computer 105 stores the third image 200, 205 and may repeat the process of acquiring and processing the image 200, 205 multiple times (>10) to ensure that the fault persists and multiple images including the trailer 200 are acquired that cause the failure of the deep neural network 402.

Fig. 5 is an illustration of a small sample image converter 500. The small sample image converter 500 is a generative confrontation network modified to allow training of the small sample image converter 500 using a training data set comprising a plurality of image classes 1 to n (IC1, ICn)512, 514. An image category is defined as a set of images, where each image includes trailers 200 having the same configuration at a plurality of trailer angles 210 under a plurality of environmental conditions. Each image CLASS 512, 514 is input to a CLASS encoder (CLASS1, CLASS) 508, 510. The class encoders 508, 510 are neural networks, each of which includes a convolutional encoder layer followed by an average pooling layer. Each class encoder 508, 510 outputs a content subcode in an encoded format corresponding to the trailer 200 configuration. Content potential (CLC)516 is input to a mean processor (MN)518 that determines the mean and variance of content potential 516. The mean processor outputs a potential class vector (LCV)522 that includes the mean and variance of each content potential 516.

In training, an output potential class vector 522 determined based on the image classes 512, 514 in the training dataset is sent to a Decoder (DEC) 524. The potential class vectors 522 are processed in a decoder 524 with a plurality of fully connected neural networks to form parameters to be used by an affine transformation processor in the decoder 524. The test input Image (IN)504 is input to a content encoder (CONT)502, and the content encoder 502 encodes the content of the test input image 504 and outputs a potential residual feature (LRF) 506. The content encoder 502 includes a plurality of convolutional content encoder layers. The potential residual features 506 are data corresponding to the components of the trailer 200 described above with respect to fig. 2 and 3. The potential residual features 506 are output as a feature map that encodes the locations of the potential residual features 506 into a 16x16 array.

Decoder 524 includes a decoder layer that performs an affine transform on the potential residuals to map features included in potential residual features 522 into a format corresponding to the class potential vectors. Affine transformations include translation, rotation, and scaling operations applied to a 16x16 array. The affine transformation operation maps the potential residual features 522 extracted from the input image 504 to image features corresponding to one of the image classes 512, 514, where the parameters applied to translation, rotation, and scaling are determined by training. During training, the output Modified Image (MI)526 is backpropagated for comparison with the input training image 404 and image classes 512, 514. The output modified image 526 is tested by comparing the trailer angle 210 with the trailer angle of the input image 504 while comparing the trailer component in the output modified image 526 with the trailer component in one of the image categories 512, 514. When the small sample image converter 500 completes training, the output modified image 526 will include objects having the same object orientation as the objects in the input training image 404.

The small sample image converter 500 is pre-trained with known image classes to achieve satisfactory image conversion quality. The small sample image converter 500 is then retrained by inputting a stored image 300 uploaded from the computer 105 in the vehicle 101 that corresponds to the first deep neural network 302 determining a failure of the trailer angle 210 by replacing one of the source image categories 512, 514 with a new category consisting of images from the unseen failed image. The small sample image converter 500 is pre-trained for known source image classes to achieve satisfactory image conversion quality for known trailer 200 classes. After this pre-training, the faulty or unseen image is used as a new class to fine tune or retrain the small sample image converter 500. A fault or missing image is used to retrain the small sample image converter network 500 that has been pre-trained. This retrains the small sample image converter 500 to map the input image 504 into the feature space corresponding to the stored image 300. Input images 504 corresponding to a plurality of trailer angles 210 and a plurality of environmental conditions are input to the small sample image converter 500. After retraining, the small sample image converter 500 can generate a modified image 526 corresponding to the failure image for the image of the trailer 200. The modified image 526 includes images of the trailer 200 at a plurality of trailer angles 210 at a plurality of environmental conditions corresponding to the trailer angle and environmental condition of the input image 504, while including features of the trailer in the stored images 200, 205. The output modified image 526 is used as a second training data set for the second deep neural network 610.

Fig. 6 is a diagram of a deep neural network retraining system 600. The deep neural network retraining system 600 includes a small sample IMAGE converter (FEW-SHOT IMAGE TRANSLATOR)500 and a Deep Neural Network (DNN) 610. The deep neural network 610 may be a copy of the deep neural network 402 included in the vehicle 101. Deep neural network 610 may be a software program executing on a cloud-based computer (e.g., server 125) that communicates with computer 105 included in vehicle 101 via network 120. Training the deep neural network 610 with the output modified image 526 may retrain the original deep neural network 610 to determine the trailer angle 210 in the image including the trailer 200 with the component from the image 300 corresponding to the failure of the first deep neural network 302. The image 300 corresponding to the failure of the first deep neural network 302 is input as an UNSEEN image (UNSEEN IMAGES)604 into the small sample image converter. The unseen images 604 are input to the small sample image converter 500 as input category images 512, 514 to train the small sample image converter 500 to modify the input training images 404 to include image features, such as trailer components and environmental conditions that make them appear to be the unseen images 604.

The training image 404 is included in a training data set used to train the deep neural network 402. Since the training images 404 were acquired from the training dataset, the training images 404 will include corresponding ground truth 406 with respect to the trailer angle 210. The small sample image converter 500 converts the training images 404 into output images 512 while retaining the ground truth 406 corresponding thereto in each training image 404. The small sample image converter will modify the appearance of each input training image 404 to match the appearance of the unseen image 604 while preserving the trailer angle 210 in the training image 404. The output modified image 526 may be matched to ground truth corresponding to the input training image 404 that generated the output modified image 526, thereby providing ground truth for training the retrained deep neural network 610.

When the computer 105 has acquired and stored a plurality of images that result in the inability of the deep neural network 402 to determine the trailer angles 210, 216, the stored plurality of images is input to the small sample image converter 500 as unseen images 604. The small sample image converter 500 is a software program executing on a cloud-based management computing device computer (e.g., server 125). Which may be accessed by a computer 105 in vehicle 101 through a network 120. The small sample image converter 500 may input an unseen image 604 that includes the newly configured trailer 200 that is not included in the training image 404 used to train the first deep neural network 402. The small sample image converter 500 may then output a plurality of second training data sets of modified images 526 in response to the training images 404 from the training data sets, the modified images including the trailer 200 configured like the newly configured trailer 200 at a plurality of trailer angles 210 corresponding to the training images 404 in the first training data set. The modified image 526 output from the small sample image converter 500 is referred to as the modified image 526 because, as discussed below with respect to fig. 5, the training image 404 is processed by the small sample image converter 500 to change the appearance of the trailer 200 in the input training image 404 based on the unseen image 604.

A second training data set comprising modified images 526 comprising images of the trailer 200 modified to appear in a new configuration may be added to the first training data set and used to retrain the original deep neural network 610. The retrained deep neural network 610 is a software program executing on a cloud-based management computer (e.g., server 125). The retrained deep neural network 610 is a copy of the first deep neural network 302. Retraining the original deep neural network 610 to determine the trailer angles 210, 216 using both the training image 404 and the modified image 526 allows the retrained deep neural network 610 to determine the trailer angles 210 for all trailers 200 in both the training image 404 and the modified image 526.

After training using both the training image 404 and the modified image 526, the retrained deep neural network 610 may be downloaded to the computer 105 in the vehicle 101 via the network 120 to determine the trailer angle 210 based on the input images 300 acquired by the video sensor 216 included in the vehicle. In examples where computer 105 does not include deep neural network 402, downloading the retrained deep neural network corresponds to a first installation. In an example where the computer 105 in the vehicle 101 includes the first deep neural network 402, the retrained deep neural network 610 corresponds to an update of the deep neural network 402 and replaces the deep neural network 402. The replacement may be accomplished by replacing parameters or weights that are programming data for the deep neural network 402. Because the retrained deep neural network 610 is trained using both the training images 404 and the modified images 526, the retrained deep neural network 610 can successfully determine the trailer angle 210 for trailer 200 configurations, including new trailer 200 configurations that cause the deep neural network 402 to fail.

Fig. 7 is a flow diagram of a process 700 for retraining and downloading a retrained deep neural network 610 to a vehicle 101. Process 700 may be implemented by a processor of a computing device, for example, taking information from sensors as input, and executing commands, and outputting object information. Process 700 includes multiple blocks that may be performed in the order shown. The process 700 may alternatively or additionally include fewer blocks, or may include blocks performed in a different order.

The process 700 begins at block 702, where a first deep neural network 302 included in the vehicle 101 stores an unseen image 604 corresponding to a determination of a failure of the trailer angle 210 in an image 300 acquired by a video sensor 216 included in the vehicle 101.

At block 704, the computer 105 in the vehicle 101 uploads the unseen image 604 to the small sample image converter 500. The small sample image converter 500 inputs the unseen image 604 into an image class 512, 513 that trains the small sample image converter 500 to modify the input image to match the appearance of the unseen image 604. Matching the appearance of the unseen image 604 includes modifying the input image to include the trailer 200 components included in the unseen image 604.

At block 706, the small sample image converter 500 generates the modified image 526 by inputting the training image 404 and converting the training image 404 to the modified image 526, as discussed above with respect to fig. 5 and 6. The output modified image 526 will include the trailer 200 having a trailer angle 210 that matches the trailer angle 210 in the input image 504 while including a trailer 200 component that matches the trailer 200 component in the unseen image 604.

At block 708, the output modified image 526 is combined with the ground truth 406 from the training image 404 to train the retrained deep neural network 610. The output modified image 526 is combined with the training image 404 to train the retrained deep neural network 610 to determine the trailer angle 210 in both the training image 404 and the modified image 526. In this manner, the retrained deep neural network 610 will be trained to process all trailer 200 types identified by the deep neural network 402, as well as new trailer 200 types corresponding to the unseen image 604 that caused the deep neural network 402 to fail.

At block 710, the retrained deep neural network 610 is downloaded to the computer 105 included in the vehicle 101. The retrained deep neural network 610 may be used by the computer 105 to successfully determine the trailer angle 210 for the trailer 200 types, including the trailer 200 type that caused the first deep neural network 302 to fail. In examples where the computer 105 does not include the first deep neural network 402, downloading the retrained deep neural network corresponds to a first installation. In an example where the computer 105 in the vehicle 101 includes the first deep neural network 402, the retrained deep neural network 610 corresponds to an update of the deep neural network 402 and replaces or reprograms the deep neural network 402.

The retrained deep neural network 610 may input the image 404 including the trailer 200 and determine the trailer angle 210. The computer 105 may use the determined trailer angle 210 to determine a vehicle path for operating the vehicle that positions the attached trailer 200 at the determined target location, as discussed above with respect to fig. 2. For example, the computer 105 may determine a target position and orientation of the trailer 200 based on processing the input image 300. For example, the target location may be a parking space or a ramp. The computer 105 may use a kinematic simulation program or the like to predict the motion of the trailer 200 in response to the motion of the tractor amount 101. The kinematic simulation program uses mathematical models of the trailer 200 and the vehicle 101 to simulate the motion of the trailer 200 in response to forces applied based on the starting trailer angle 210. The computer 105 may determine a vehicle path to apply a force to the trailer 200 to move the trailer 200 to a target position in a target orientation based on the starting trailer angle 210. The computer 105 may transmit commands to the vehicle driveline, vehicle steering, and vehicle brakes to operate the vehicle 101 on the vehicle path and thereby apply the determined force to the trailer 200 and move it to the target position in the target orientation. After block 710, the process 700 ends.

Computing devices, such as those discussed herein, typically each include commands that are executable by one or more computing devices, such as those identified above, and for performing the blocks or steps of the processes described above. For example, the process blocks discussed above may be embodied as computer-executable commands.

The computer-executable commands may be compiled or interpreted from a computer program created using a variety of programming languages and/or techniques, including but not limited to: java (Java)^TMC, C + +, Python, Julia, SCALA, Visual Basic, Java Script, Perl, HTML, and the like. In general, a processor (e.g., a microprocessor) receives commands, e.g., from a memory, a computer-readable medium, etc., and executes the commands, thereby performing one or more processes, including one or more of the processes described herein. Such commands and other data may be stored in files and transmitted using a variety of computer-readable media. A file in a computing device is typically a collection of data stored on a computer-readable medium, such as a storage medium, random access memory, or the like.

Computer-readable media includes any medium that participates in providing data (e.g., commands) that may be read by a computer. Such a medium may take many forms, including but not limited to, non-volatile media, and the like. Non-volatile media includes, for example, optical or magnetic disks and other persistent memory. Volatile media includes Dynamic Random Access Memory (DRAM), which typically constitutes a main memory. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

The computer 105 may be the computer 105 as described above, or may include special-purpose electronic circuitry including an ASIC made for a particular operation, e.g., an ASIC for processing sensor data and/or transmitting sensor data. In another example, the computer 140 may include an FPGA, which is an integrated circuit manufactured to be configurable by a user. Generally, a hardware description language such as VHDL (very high speed integrated circuit hardware description language) is used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, ASICs are manufactured based on VHDL programming provided before manufacture, while logic components internal to the FPGA may be configured based on VHDL programming stored, for example, in memory electrically connected to the FPGA circuitry. In some examples, a combination of processors, ASICs, and/or FPGA circuitry may be included in computer 105.

Unless explicitly indicated to the contrary herein, all terms used in the claims are intended to be given their ordinary and customary meaning as understood by those skilled in the art. In particular, the use of singular articles such as "a," "the," "said," etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

The term "exemplary" is used herein in a sense that it represents an example, e.g., a reference to "exemplary widget" should be read to refer only to an example of a widget.

The adverb "about" modifying a value or result means that the shape, structure, measured value, determination, calculation, etc. may deviate from the geometry, distance, measured value, determination, calculation, etc. described with certainty due to imperfections in materials, machining, manufacturing, sensor measurements, calculations, processing time, communication time, etc.

In the drawings, like reference numerals designate like elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc., described herein, it should be understood that although the steps or blocks of such processes, etc., have been described as occurring in a certain sequential order, such processes may be practiced with the described steps performed in an order other than the order described herein. It is also understood that certain steps may be performed simultaneously, that other steps may be added, or that certain steps described herein may be omitted. In other words, the description of processes herein is provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claimed invention.

According to the present invention there is provided a system comprising a management computing device having: a processor; and a memory, the memory including instructions executable by the processor to: receiving one or more images from a vehicle, wherein a first deep neural network included in a computer in the vehicle fails to determine an orientation of a first object in the one or more images; generating a plurality of modified images with a small sample image-to-image converter, wherein the modified images each include a modified object based on the first object; training a second deep neural network based on the plurality of modified images to determine the orientation of the first object; and downloading the trained second deep neural network to the vehicle.

According to one embodiment, a small sample image-to-image converter is trained based on generating a countermeasure network and using a plurality of real world images and a plurality of composite images to convert one or more input images to the plurality of modified images based on the one or more images from the vehicle.

According to one embodiment, the instructions comprise further instructions for: training the small sample image-to-image converter based on the plurality of real-world images and a plurality of composite images each including a vehicle trailer and ground truth data regarding an angle of each respective vehicle trailer relative to the vehicle.

According to one embodiment, the plurality of real world images and the plurality of composite images comprise a plurality of weather and lighting conditions, including day, night, rain, snow, and a plurality of sun directions.

According to one embodiment, the plurality of vehicle trailers in the plurality of real world images and the plurality of composite images comprise a plurality of vehicle trailer types with a plurality of trailer components configured in various configurations.

According to one embodiment, the first object is a vehicle trailer and determining the orientation of the first object comprises determining an angle of the vehicle trailer relative to the vehicle for each modified image, respectively.

According to one embodiment, the plurality of modified images each comprise ground truth data comprising a respective angle of the vehicle trailer relative to the vehicle.

According to one embodiment, each modified image comprises a vehicle trailer oriented at an angle relative to the vehicle equal to the angle of the object appearing in the input image.

According to one embodiment, each modification object based on the first object comprises a vehicle trailer component comprised in the first object having the same configuration as the vehicle trailer component comprised in the first object.

According to one embodiment, the first and second deep neural networks comprise convolutional layers and fully-connected layers, wherein the convolutional layers process input image data and the fully-connected layers process output from the convolutional layers to determine an angle of the object relative to the vehicle.

According to one embodiment, the small sample image-to-image converter includes a content encoder layer, a class encoder layer, and a decoder layer, wherein the content encoder layer generates potential residual features based on an input object type and an object orientation, and the decoder layer generates an output image based on the class encoder layer.

According to one embodiment, the instructions comprise further instructions for: operating a vehicle by inputting an image to the second deep neural network to determine a trailer angle of an attached trailer.

According to one embodiment, the instructions comprise further instructions for: the vehicle is operated by commanding the vehicle powertrain, vehicle steering, and vehicle brakes to operate the vehicle along a vehicle path that applies a force to the attached trailer to move the attached trailer to a target position in a target orientation.

According to the invention, a method comprises: receiving one or more images from a vehicle, wherein a first deep neural network included in a computer in the vehicle fails to determine an orientation of a first object in the one or more images; generating a plurality of modified images with a small sample image-to-image converter, wherein the modified images each include a modified object based on the first object; training a second deep neural network based on the plurality of modified images to determine the orientation of the modified object; and downloading the trained second deep neural network to the vehicle.

In one aspect of the invention, a small sample image-to-image converter is based on generating a countermeasure network and trained using a plurality of real world images and a plurality of composite images to convert one or more input images to the plurality of modified images based on the one or more images from the vehicle.

In one aspect of the invention, the method includes training the small sample image-to-image converter based on the plurality of real-world images and a plurality of composite images including the vehicle trailer and ground truth data regarding an angle of each vehicle trailer relative to the vehicle for each real-world image, respectively.

In one aspect of the invention, the plurality of real world images and the plurality of composite images include a plurality of weather and lighting conditions including day, night, rain, snow, and a plurality of sun directions.

In one aspect of the invention, the plurality of vehicle trailers in the plurality of real world images and the plurality of composite images comprise a plurality of vehicle trailer types with a plurality of trailer components configured in various configurations.

In one aspect of the invention, the first object is a vehicle trailer and determining the orientation of the first object comprises determining an angle of the vehicle trailer relative to the vehicle for each modified image, respectively.

In one aspect of the invention, the plurality of modified images each comprise ground truth data comprising a respective angle of the vehicle trailer relative to the vehicle.