阅读说明：本技术 自动泊车决策模型优化系统及方法 (Automatic parking decision model optimization system and method ) 是由 张素民 支永帅 包智鹏 何睿 杨志 孟志伟 于 2021-08-25 设计创作，主要内容包括：本发明公开了一种自动泊车决策模型优化系统及方法,所述方法包括：步骤1,采集车辆周围的泊车环境信息,识别可用停车位,选择是否泊车,若是则执行下一步骤；步骤2,基于环境信息和停车位识别结果计算模型决策信息；步骤3,当选择自动泊车时,使用模型决策信息指导车辆进行自动泊车,当选择手动泊车时,执行下一步骤；步骤4,基于用户决策信息进行手动泊车操作,并模拟基于模型决策信息的自动泊车,若模拟结果不满足泊车终止条件,执行下一步骤；步骤5,对用户决策信息和环境信息进行关联,根据关联信息对自动泊车决策模型进行重复训练,利用更新的自动泊车决策模型进行自动泊车；本发明能够在各种环境下进行自动泊车,自适应能力强。(The invention discloses an automatic parking decision model optimization system and method, wherein the method comprises the following steps: step 1, collecting parking environment information around a vehicle, identifying available parking spaces, selecting whether to park, and executing the next step if the available parking spaces are the parking spaces; step 2, calculating model decision information based on the environment information and the parking space identification result; step 3, when automatic parking is selected, the model decision information is used for guiding the vehicle to automatically park, and when manual parking is selected, the next step is executed; step 4, performing manual parking operation based on the user decision information, simulating automatic parking based on the model decision information, and executing the next step if the simulation result does not meet the parking termination condition; step 5, correlating the user decision information with the environment information, repeatedly training the automatic parking decision model according to the correlation information, and automatically parking by using the updated automatic parking decision model; the invention can automatically park in various environments and has strong self-adaptive capability.)

1. The automatic parking decision model optimization method is characterized by comprising the following steps of:

step 1, collecting parking environment information around a vehicle, identifying an available parking space by using a parking space identification model, selecting whether to park according to an identification result, and repeatedly identifying the available parking space until parking is selected if parking is not selected;

step 2, the automatic parking decision model determines model decision information according to the parking space recognition result and the environment information;

step 3, when the user selects automatic parking, inputting the model decision information into the control module to guide the vehicle to carry out automatic parking operation; when the user selects manual parking, executing step 4;

step 4, controlling the vehicle to perform manual parking according to the user decision information, simulating an automatic parking process guided by the model decision information, if the automatic parking process meets the parking termination condition, not processing the user decision information, and if the automatic parking process does not meet the parking termination condition, executing step 5;

step 5, performing correlation processing on the user decision information and the environmental information around the vehicle;

and 6, acquiring the associated data of the user decision information and the environment information of the plurality of vehicles, and repeatedly training the automatic parking model based on the associated data to obtain the optimized automatic parking decision model.

2. The automated parking decision model optimization method of claim 1, wherein the parking space recognition model comprises a CNN network, a candidate area network, and a prediction network connected in sequence;

the CNN network is used for extracting a pyramid feature map of a semantic segmentation image of the environmental information under different scales;

the candidate area network takes the pyramid feature map as input, and respectively uses convolution kernels of 1 × 4, 1 × 3 and 1 × 2 to identify the interested area with the corresponding length-width ratio in each pyramid feature map;

the prediction network takes the pyramid feature map and the regions of interest as input, uses an ROI pooling layer to map each region of interest to a corresponding region of a corresponding pyramid feature map, divides the region into a fixed number of parts, uses a maximum pooling operation to obtain a feature map with a fixed size, and predicts the type and frame information of the parking space through two layers of full-connection networks.

3. The automatic parking decision model optimization method according to claim 1, wherein the environment information includes a road route map, a road structure map, status information of each obstacle, sideline information of a parking space, and distance information between the parking space and surrounding obstacles, the obstacles include dynamic obstacles and static obstacles, and the status information includes pose and shape of the obstacles;

the identification result comprises the type, size and position of the parking space, and the type of the parking space comprises a convex parking space, a linear parking space and an oblique parking space;

the model decision information and the user decision information both comprise the opening degree of an accelerator pedal, the opening degree of a brake pedal and the steering wheel angle.

4. The automated parking decision model optimization method of claim 1, wherein the parking end conditions include vehicle physical constraints, obstacle avoidance constraints, range of motion constraints, parking end condition constraints, and time constraints.

5. The automatic parking decision model optimization method according to claim 1, wherein the automatic parking decision model is composed of a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer and a fully-connected layer, which are connected in sequence;

semantic segmentation image I of environment information for convolutional layer and pooling layer_i,jAnd user decision information (u)_i,j,v_i,j,ω_i,j) As input, the features of environment information from detail to whole are extracted in sequence, and the steering wheel angle is respectively mapped from the extracted features by the full connection layerOpening degree of accelerator pedalAnd brake pedal opening degreeModel decision information output by an automated parking decision modelAnd user decision information (u)_i,j,v_i,j,ω_i,j) Inter loss L_iGradient descending is carried out, model parameters are continuously updated, and an optimized automatic parking decision model is obtained;

the loss function of the automatic parking decision model is as follows:

wherein L is_iIs the loss value of the ith iteration, N is the total number of samples of each iteration, j is the sample number used for model training, u_i,jThe actual value of the steering wheel angle input by the user,steering wheel angle, v, predicted for an automated parking decision model_i,jThe actual value of the accelerator pedal opening degree input by the user,accelerator pedal opening, omega, predicted for an automated parking decision model_i,jThe actual value of the brake pedal opening degree input by the user,and the opening degree of the brake pedal is predicted by the automatic parking decision model.

6. A system for optimizing an automated parking decision model using the method of any of claims 1-5, comprising:

the environment information acquisition module is used for acquiring the surrounding environment information when the vehicle parks;

the parking space identification module is used for identifying available parking spaces;

the parking decision module is used for calculating model decision information based on the parking environment information and the parking space identification result;

the user decision acquisition module is used for acquiring decision information input by a user;

the vehicle-user interaction module is used for displaying the parking space identification result and interacting the user with the user decision acquisition module;

the control module is used for guiding the vehicle to perform parking operation according to the user decision information or the model decision information;

the comparison module is used for simulating an automatic parking process guided by model decision information and judging whether the process meets a parking termination condition;

the correlation module is used for performing correlation processing on the user decision information and the parking environment information;

and the external communication module is used for acquiring the user decision information and the parking environment information association data, and updating the parameters of the automatic parking decision model based on the association data to obtain the optimized automatic parking decision model.

Technical Field

The invention belongs to the technical field of intelligent driving, and particularly relates to an automatic parking decision model optimization system and method.

Background

With the development of deep learning technology, in recent years, more and more scientific researchers develop the automatic parking system based on data driving by training the machine learning model, the quality of the training data set directly influences the performance of the automatic parking system in the model training process, and in practical application, the scientific researchers invest huge manpower, material resources and financial resources in the collection, arrangement and annotation of the training data set, but the effect is poor; although training data can be generated in a simulation environment, the simulation environment cannot completely simulate a real parking environment, the obtained training data quality is poor, the process of generating a training data set in an actual parking scene is tedious and tedious, and data capable of improving the performance of a model in a specific or extreme use environment is difficult to collect.

The method comprises the following steps that a machine learning model for automatic parking is trained, thousands of samples or even millions of samples are needed, therefore, whether a large amount of high-quality training data can be obtained becomes an important limiting factor for the use of a machine learning technology, along with the complication of the machine learning model, a larger training data set needs to be constructed, so that the trained model has enough generalization capability, namely the model can well represent on unseen data, if the training data set is not large enough, the model is over-fitted, namely the model well represents on the training data set, but can not be well generalized to other unseen scenes, and therefore, data of additional scenes need to be added into the training data set to enrich the training data set; however, there are huge technical obstacles to obtaining training data in a real scene, for example, the automatic parking system needs to identify data required by an effective parking space and a parking decision model from the acquired image, and if there is no large amount of training data sets, the machine learning model in the automatic parking system cannot accurately and robustly identify the required information, so that the automatic parking system cannot perform automatic parking in some unusual scenes.

Disclosure of Invention

The embodiment of the invention aims to provide an automatic parking decision model optimization system, which is connected with a plurality of vehicles by using a vehicle networking technology, can obtain a large amount of parking space identification training data and associated data of user decision information and environment information in various driving scenes, has a simple data acquisition process and low cost, is less influenced by the driving preference of individual drivers, and can construct an automatic parking decision model with good adaptability in a real driving scene.

The embodiment of the invention aims to provide an automatic parking decision model optimization method, which can automatically acquire decision information input by a user in a specific scene in the vehicle driving process, circularly update parameters in a vehicle-mounted automatic parking decision model based on the decision information and improve the generalization capability and the applicability of the vehicle-mounted automatic parking decision model.

In order to solve the technical problem, the technical scheme adopted by the invention is that the automatic parking decision model optimization method comprises the following steps:

step 1, collecting parking environment information around a vehicle, identifying an available parking space by using a parking space identification model, selecting whether to park according to an identification result, and repeatedly identifying the available parking space until parking is selected if parking is not selected;

step 2, the automatic parking decision model determines model decision information according to the parking space recognition result and the environment information;

step 3, when the user selects automatic parking, inputting the model decision information into the control module to guide the vehicle to carry out automatic parking operation; when the user selects manual parking, executing step 4;

step 4, controlling the vehicle to perform manual parking according to the user decision information, simulating an automatic parking process guided by the model decision information, if the automatic parking process meets the parking termination condition, not processing the user decision information, and if the automatic parking process does not meet the parking termination condition, executing step 5;

step 5, performing correlation processing on the user decision information and the environmental information around the vehicle;

and 6, acquiring the associated data of the user decision information and the environment information of the plurality of vehicles, and repeatedly training the automatic parking model based on the associated data to obtain the optimized automatic parking decision model.

Further, the parking space identification model comprises a CNN network, a candidate area network and a prediction network which are connected in sequence;

the CNN network extracts a pyramid feature map of a semantic segmentation image of the environmental information under different scales;

the candidate area network takes the pyramid feature map as input, and respectively uses convolution kernels of 1 × 4, 1 × 3 and 1 × 2 to identify the interested areas with corresponding length-width ratios in the feature maps;

the prediction network uses the pyramid feature map and the regions of interest as input, uses the ROI pooling layer to map each region of interest to a corresponding region of a corresponding pyramid feature map, divides the region into a fixed number of parts, uses the maximum pooling operation to obtain the feature map with a fixed size, and predicts the type and frame information of the parking space through two layers of full-connection networks.

Further, the environment information includes a road route map, a road structure map, state information of each obstacle, sideline information of a parking space, and distance information between the parking space and surrounding obstacles, the obstacles include dynamic obstacles and static obstacles, and the state information includes poses and shapes of the obstacles;

the identification result comprises the type, size and position of the parking space, and the type of the parking space comprises a convex parking space, a linear parking space and an oblique parking space;

the model decision information and the user decision information both comprise the opening degree of an accelerator pedal, the opening degree of a brake pedal and the steering wheel angle.

Further, the parking termination condition comprises vehicle physical constraint, obstacle avoidance constraint, activity range constraint, parking termination condition constraint and time constraint.

Furthermore, the automatic parking decision model consists of a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a third convolution layer and a full-connection layer which are connected in sequence;

semantic segmentation image I of environment information for convolutional layer and pooling layer_i,jAnd user decision information (u)_i,j,v_i,j,ω_i,j) As input, the features of environment information from detail to whole are extracted in sequence, and the steering wheel angle is respectively mapped from the extracted features by the full connection layerOpening degree of accelerator pedalAnd brake pedal opening degreeModel decision information output by an automated parking decision modelAnd user decision information (u)_i,j,v_i,j,ω_i,j) Inter loss L_iGradient descending is carried out, model parameters are continuously updated, and an optimized automatic parking decision model is obtained;

the loss function of the automatic parking decision model is as follows:

wherein L is_iIs the loss value of the ith iteration, N is the total number of samples of each iteration, j is the sample number used for model training, u_i,jThe actual value of the steering wheel angle input by the user,steering wheel angle, v, predicted for an automated parking decision model_i,jThe actual value of the accelerator pedal opening degree input by the user,accelerator pedal opening, omega, predicted for an automated parking decision model_i,jThe actual value of the brake pedal opening degree input by the user,and the opening degree of the brake pedal is predicted by the automatic parking decision model.

An automated parking decision model optimization system, comprising:

the environment information acquisition module is used for acquiring the surrounding environment information when the vehicle parks;

the parking space identification module is used for identifying available parking spaces;

the parking decision module is used for calculating model decision information based on the parking environment information and the parking space identification result;

the user decision acquisition module is used for acquiring decision information input by a user;

the vehicle-user interaction module is used for displaying the parking space identification result and interacting the user with the user decision acquisition module;

the control module is used for guiding the vehicle to perform parking operation according to the user decision information or the model decision information;

the comparison module is used for simulating an automatic parking process guided by model decision information and judging whether the process meets a parking termination condition;

the correlation module is used for performing correlation processing on the user decision information and the parking environment information;

and the external communication module is used for acquiring the user decision information and the parking environment information association data, and updating the parameters of the automatic parking decision model based on the association data to obtain the optimized automatic parking decision model.

The invention has the beneficial effects that: the data generation method provided by the embodiment of the invention does not need a driver to drive a vehicle specially in an actual parking scene to obtain training data, and acquires the associated data of the vehicle parking environment and user decision information in daily life by the way and generally through the Internet of vehicles system, so that the acquisition cost of the training data is lower, the popularization of the training data is wider, most of actual parking environments can be covered, the automatic parking model obtained based on the training can be used for accurately parking the vehicle in various parking environments, and the generalization capability is strong.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a flow chart of data collection according to an embodiment of the present invention.

Fig. 2 is a parking space recognition training data processing flow chart.

Fig. 3 is a structural view of the automatic parking system of the present invention.

Fig. 4 is a schematic view of a parking process for several typical parking spaces, where fig. 4 (a) is a schematic view of a parking process for a linear parking space, fig. 4 (b) is a schematic view of a parking process for a convex parking space, and fig. 4 (c) is a schematic view of a parking process for an oblique parking space.

Fig. 5 is a schematic diagram of a manual parking process and a virtual parking process.

Fig. 6 is a block diagram of an automatic parking decision model.

Fig. 7 is a structural diagram of a parking space recognition model.

Fig. 8 is a schematic view of the aspect ratio of different types of parking spaces.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 3, the automatic parking decision model optimization system includes a parking space recognition module, the parking space recognition module recognizes an available parking space on the roadside based on ultrasonic radar data, camera image data, etc. obtained by a vehicle-mounted sensor, and uses technologies such as pure ultrasonic radar parking space recognition or panoramic parking space recognition, etc. to transmit a parking space recognition result to a vehicle-user interaction module for display, the vehicle-user interaction module is a vehicle-mounted display screen, the vehicle-mounted display screen displays the parking space recognition result and options, the options include parking at this place and parking at this place, parking includes manual parking and automatic parking, a user can select according to actual needs, and when the vehicle-user interaction module selects not to park at this place, the vehicle-user interaction module informs the parking space recognition module to repeat parking space recognition operation until the user selects parking, when a user selects to park, the vehicle-user interaction module informs the environment information acquisition module to send parking environment information to the automatic parking decision-making model, and model decision-making information for controlling automatic parking is calculated, wherein the model decision-making information comprises a steering wheel turning angle, an accelerator pedal opening and a brake pedal opening.

When the user selects automatic parking, the automatic parking decision model sends the model decision information to the control module to guide the vehicle to automatically park; when the user selects manual parking, the user decision information acquisition module acquires user decision information input by the user according to current environment information, sends the user decision information to the control module and guides the vehicle to perform parking operation, and meanwhile, the user decision information acquisition module and the model decision information acquisition module respectively send the user decision information and the model decision information to the comparison module.

The comparison module simulates model decision information to guide parking operation in the current parking environment, as shown in fig. 5, the comparison module can display barrier information (square H) around a vehicle, position information (black vehicle) of the vehicle guided by user decision information, position information (dotted line square frame) of a virtual vehicle guided by the model decision information, parking boundaries FG and BE and a parking space boundary ACDB in real time, if a parking result in the simulated parking process does not meet a parking termination condition, the automatic parking decision model is considered to BE incapable of parking operation in the current parking scene, at this time, the comparison module sends the user decision information to the association module, otherwise, the automatic parking decision model is considered to BE capable of guiding the vehicle to complete parking operation, and the user decision information does not need to BE stored.

The association module associates the user decision information with the environmental information in the current scene, specifically, the association module aligns parking environmental information acquired by different types of sensors in a time domain, for example, image data at time t is I_tAnd the ultrasonic radar data is R_tAnd then extracting user decision information (steering wheel angle, accelerator pedal opening, brake pedal opening and the like) at corresponding moment, associating the decision information input by the user with the ambient environment information by the method, and transmitting the associated information to an external communication module through a network interface.

The external communication module comprises a cloud database and a cloud server, the cloud database is used for storing the correlation information of the user decision information and the environment information to form a training data set, and the cloud server is used for updating the parameters of the automatic parking decision model by using the training data set and sending the updated parameters to the vehicle-mounted automatic parking decision model.

The cloud database is connected with a plurality of vehicles by using a vehicle networking technology, a vehicle road cooperation technology and the like, receives decision information and associated information of environment information of a plurality of drivers in each actual parking environment, omits the process of specially acquiring parking environment information and user decision information, reduces the cost of data acquisition, avoids the influence of the driving preference of individual drivers on a decision model, has huge number of constructed training data sets, covers a plurality of parking scenes in actual life, can generate a more feasible and efficient parking decision under similar parking scenes by using the automatic parking decision model trained by using the automatic parking decision model, and has stronger applicability.

As shown in fig. 1, the method for optimizing an automatic parking decision model by using the automatic parking decision model optimization system includes the following steps:

s1, collecting environmental information around the vehicle in the parking process by using a sensor mounted on the vehicle, and identifying the parking space in the parking environment by using a parking space identification model;

after a user starts parking space search, available parking spaces are identified by using an ultrasonic radar parking space identification mode, a panoramic parking space identification mode or a combination mode of the ultrasonic radar parking space identification mode and the panoramic parking space identification mode, and the identification result of the parking spaces comprises the types of the parking spaces (convex parking spaces, linear parking spaces or oblique parking spaces and the like), the sizes and the positions of the parking spaces and the like; effective parking space identification is a key ring in an automatic parking system, specifically, a deep learning-based method can be adopted to identify parking spaces, a large number of high-quality parking space identification training data sets need to be constructed in order to enable the automatic parking system to identify more types of parking spaces, and the parking space identification training data under unusual scenes are added into the original parking space identification training data sets, so that the parking space identification scenes are enriched, and the parking space identification capability of the automatic parking system is improved;

the sensor comprises one or more of an ultrasonic radar, a panoramic camera and a high-dynamic-range forward-looking camera;

in the implementation process, 8 round-looking cameras are mounted on the vehicle to provide a 360-degree round-looking visible image with a range of 250 meters, or a plurality of cameras are mounted on the vehicle and comprise a front wide camera, a front narrow camera, a rear camera, a plurality of front side cameras and a plurality of rear side cameras; or an ultrasonic radar or a millimeter wave radar is assembled on the vehicle, or the environmental information around the vehicle in the parking process is obtained by utilizing the vehicle networking and vehicle road cooperation technology;

the environment information around the vehicle comprises a road route map, a road structure map, state information of each obstacle, ultrasonic radar data based on a vehicle-mounted sensor, camera image data and the like, wherein the obstacles comprise dynamic obstacles (such as pedestrians, vehicles and the like) and static obstacles (such as road edges, railings, vehicles parked around parking spaces and the like), the state information comprises the poses, the shapes and the like of the obstacles, the ultrasonic radar data comprises distance information between the parking spaces and the obstacles such as the surrounding vehicles and the like, and the camera image data comprises sideline information and the like of the parking spaces;

s2, displaying the parking space recognition result on the vehicle-user interaction module, and enabling the user to select whether to park or not based on the selection, if the user is not satisfied with the currently recognized parking space, selecting not to park, and repeating the step 1 until the user selects a proper parking space;

s3, when a user selects to park at the position, the collected vehicle surrounding environment information is sent to an automatic parking decision model, and in view of the fact that the automatic parking decision models corresponding to the three parking space types have the same structure and different model parameters, the automatic parking decision model selects corresponding model parameters according to the parking space recognition result transmitted in the step 1 and calculates model decision information based on the current vehicle surrounding environment information, wherein the model decision information comprises steering wheel turning angles, brake pedal opening degrees, accelerator pedal opening degrees and the like;

s4, when the user selects automatic parking, the model decision information is sent to a control module, and the control on variables such as an accelerator, a brake and a steering wheel corner is executed to complete automatic parking operation; if the user selects manual parking, sending the model decision information to the comparison module, and executing S5;

s5, collecting user decision information input by a user according to the environmental information around the vehicle, controlling an accelerator, a brake and a steering wheel corner by the control module according to the user decision information, completing manual parking operation, and sending the user decision information to the comparison module;

the user decision information includes but is not limited to accelerator pedal opening, brake pedal opening, steering wheel angle, etc.;

s6, the comparison module carries out parking simulation operation based on the model decision information, if the simulation process meets the automatic parking termination condition, the model decision information is considered to be capable of guiding the vehicle to carry out automatic parking, the automatic parking decision model can be competent for a parking task in the current scene, the user decision data in the current scene is not target training data, collection and storage are not needed, and the user decision information at the moment is abandoned;

if the vehicle collides with surrounding obstacles, road edges and the like in the virtual parking process (namely the automatic parking process does not meet the parking termination condition), the model decision information calculated by the automatic parking decision model in the scene is considered to be seriously deviated from the parking willingness of the user and can not be competent for parking operation, and at the moment, the user decision information and the parking environment information are sent to the association module;

the user decision information is a parking decision made by the user according to the current parking environment, and the model decision information is model decision information calculated by the automatic parking decision model according to the current vehicle surrounding environment information;

the parking termination condition comprises vehicle physical constraint, obstacle avoidance constraint, activity range constraint, parking termination condition constraint and time constraint, the vehicle physical constraint is that the maximum speed of the vehicle, the maximum Ackerman of the vehicle and the maximum rotating speed of a steering wheel do not exceed a set threshold value, the obstacle avoidance constraint is that the vehicle cannot be cut or rubbed or collided in the parking process, the activity range constraint is that the vehicle can only run in a driving area in the parking process and cannot run into an area outside the activity range, the parking termination condition constraint is that the vehicle body cannot exceed a parking space borderline after the parking is completed, meanwhile, a user is ensured to have enough space for getting on and off the vehicle, the time constraint is that the duration time of the parking process cannot be too long, otherwise, other vehicles can be influenced;

s7, aligning parking environment information (video information and radar information) acquired by different types of sensors in a time domain, associating user decision information (steering wheel angle, accelerator pedal and brake pedal) with the parking environment information to form a sequence pair with a front-back sequence in a time dimension, and storing a data set obtained by association;

and S8, obtaining the associated data of the plurality of vehicles by using technologies such as vehicle networking, vehicle road cooperation and the like, repeatedly training the automatic parking decision model by the external communication module based on the associated data set, and updating the model parameters to obtain the optimized automatic parking decision model.

The parking space types commonly seen in life include a linear parking space, a convex parking space and an oblique parking space, the parking spaces in different pixels are different in size at different visual angles (for example, a far parking space looks smaller, a near parking space looks larger, and the like), in order to improve the speed and the accuracy of parking space identification, the embodiment of the invention designs a parking space identification model based on Faster R-CNN, the structure of which is shown in FIG. 7, and the parking space identification model comprises a CNN network, a candidate area network and a prediction network which are sequentially connected, firstly carries out semantic segmentation on parking environment information by using an ICNet algorithm, extracts feature maps of input semantic segmentation images at different scales by using the CNN network, the CNN network takes a VGG16 network as a main network, takes a layer set with feature maps with the same size as a section, and the last layer of each section is respectively represented as C1, C2, C3, C4 and C5, pyramid features corresponding to C2, C3, C4 and C5 are represented as P2, P3, P4 and P5, P5 is obtained by performing 1 × 1 convolution operation on C5, P4 is obtained by adding an upsampling result of P5 to 1 × 1 convolution operation of C4, P3 is obtained by adding an upsampling result of P4 to 1 × 1 convolution operation of C3, P2 is obtained by adding an upsampling result of P3 to 1 × 1 convolution operation of C2, the pyramid features have higher-layer semantic information and smaller receptive fields at the same time, and the CNN network can obtain pyramid feature maps with different receptive fields and different sizes.

The shape of the linear parking space and the convex parking space is rectangular, the shape of the diagonal parking space is parallelogram, and no convolution kernel of the parallelogram exists in convolution, so that the diagonal parking space needs to be filled to form a rectangle, and the rectangle contains information of the diagonal parking space; as shown in fig. 8, the width-to-length ratio of the linear parking space to the convex parking space may be approximately 1: 4 and 1: and 3, rounding the width-length ratio of the expanded diagonal parking space to 1: in order to improve efficiency and accuracy of parking space identification, in the embodiment of the present invention, 1 × 4, 1 × 3, and 1 × 2 convolution kernels are respectively used at the upper end of a candidate area network to respectively extract image features with corresponding aspect ratios in feature images, the image features are input to a classification layer and a regression layer to respectively obtain types and candidate areas of suspected parking spaces, the generated candidate areas have receptive fields with different scales and different aspect ratios, and the candidate areas are regions of interest (ROIs).

Inputting the pyramid characteristics and the interesting regions into a prediction network, enabling the prediction network to use an ROI pooling layer to perform maximum pooling on the candidate interesting regions extracted by the candidate network, mapping ROIs with different sizes to corresponding regions of feature maps with corresponding sizes, dividing the regions into a fixed number of parts, obtaining the feature maps with the fixed sizes in each obtained part by using maximum pooling operation, and finally predicting the feature maps through two layers of full-connection networks to obtain the types of parking spaces and frame information of the parking spaces.

The candidate area network in the original faster R-CNN assumes that only one feature map is provided with a plurality of anchor frames, and the feature map has a fixed receptive field shape, but the method has poor effect when the size of the anchor frame is changed in a large range, and in order to solve the problem, the embodiment of the invention adds a pyramid feature extraction layer in the parking space recognition model shown in FIG. 7 to generate feature maps with multiple sizes, increase the receptive field of the model and improve the effectiveness of the model; meanwhile, considering the shape corresponding to the type of the parking space, the parking space recognition model generates candidate regions with different length-width ratios by using convolution kernels of 1 × 4, 1 × 3 and 1 × 2, the generated candidate regions have receptive fields with different scales, and the size and the shape of the generated candidate regions are similar to those of the actual parking space, so that the accuracy and the efficiency of parking space recognition can be improved.

As shown in fig. 2, the training process of the parking space recognition model is as follows:

s1-1, collecting environmental data around the vehicle by using a vehicle-mounted sensor, and constructing a parking space recognition training data set;

using a forward-looking dynamic range camera to obtain image information of a surrounding environment, or using a plurality of cameras to obtain 360-degree around-looking information of a vehicle, or using a wide-view forward-looking camera, a narrow-view forward-looking camera, a rear-viewing camera, a forward-looking side camera and a rear-viewing side camera to jointly obtain the around-looking information of the vehicle, splicing the information to obtain the around-looking information of the vehicle, or using vehicle-mounted ultrasonic waves or millimeter wave radars to obtain parking environment information around the vehicle;

s1-2, preprocessing environmental information around the vehicle;

the LIME algorithm is used for carrying out illumination balance processing on parking environment information around the vehicle, each frame of image can obtain a good illumination effect so as to improve the contrast of the environment information in the image, meanwhile, in order to reduce noise points in the image, Gaussian filtering is used for processing the image, the image is cut into 190 multiplied by 100 to obtain a preprocessed image, and an ICNet algorithm is used for carrying out semantic segmentation on the preprocessed image;

s1-3, constructing a parking space recognition model, and taking the semantic segmentation image, artificially labeled parking space frame information and parking space type codes as input training parking space recognition models;

the parking space bounding box information comprises a diagonal intersection point position (x, y) of a parking space, the length and the width of the parking space, the type code of the parking space is '1', '2' or '3', '1' corresponds to a linear parking space, '2' corresponds to a convex parking space, and '3' corresponds to an oblique parking space;

s1-4, calculating the confidence coefficient of each parking space recognition result, and when the confidence coefficient is higher than 0.8, indicating that the parking space recognition model can effectively recognize the parking space in the environment, wherein the parking environment information cannot further improve the parking space recognition model at the moment, and abandoning the data; when the confidence coefficient is lower than 0.8, the parking space recognition effect of the parking space recognition model in the environment is poor, or the parking space cannot be recognized in the environment, the parking environment data in the scene is added into the training data set, and the parking space recognition model is repeatedly trained.

The automatic parking decision model structure is shown in fig. 6 and comprises a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a third convolution layer and a full connection layer which are connected in sequence, wherein convolution operation and pooling operation are used for extracting features from details to the whole, and the full connection layer is used for respectively mapping a steering wheel angle, an accelerator pedal opening and a brake pedal opening from the extracted features.

In the process of training an automatic parking decision model, firstly, semantic segmentation is carried out on the collected parking environment information by using a semantic segmentation algorithm (ICNet algorithm and the like), and then a semantic segmentation image I is obtained_i,jAnd user decision information (u)_i,j,v_i,j,ω_i,j) Training parking decisions as inputs pi_θ(u_i,j,v_i,j,ω_i,j|I_i,j) By model decision output to an automatic parking decision modelInformationAnd user decision information (u)_i,j,v_i,j,ω_i,j) Inter loss L_iGradient descending is carried out, the parameter theta is continuously updated, and an optimized automatic parking decision model is obtained; the automatic parking decision models corresponding to different parking space types are the same, and the model parameters theta are different.

Wherein u is_i,jThe actual value of the steering wheel angle input by the user,steering wheel angle, v, predicted for an automated parking decision model_i,jThe actual value of the accelerator pedal opening degree input by the user,accelerator pedal opening, omega, predicted for an automated parking decision model_i,jThe actual value of the brake pedal opening degree input by the user,the automatic parking decision model has the advantages that the opening degree of a brake pedal is predicted by the automatic parking decision model, theta is a parameter which needs to be learned by the automatic parking decision model, the automatic parking decision model has the same structure for different types of parking spaces, but the parameter theta is different, and in practical application, corresponding model parameters can be selected according to the recognized parking space types, so that the parking performance of the automatic parking decision model in different types of parking spaces is improved.

In order to accelerate the learning speed of the automatic parking decision model, the invention uses a linear rectification function (ReLUs) as an activation function after convolution, and the loss functions of the model comprise a steering wheel angle loss, an accelerator pedal opening loss and a brake pedal opening loss:

L_ifor the loss value of the ith iteration,n is the total number of samples per iteration and j is the sample number used for model training.

And in the training process of the automatic parking decision model, a small-batch gradient descent optimization method is adopted to iteratively update the network weight, the number of small-batch samples is 100, and the training is stopped when iteration reaches a preset iteration number or loss is detected not to descend.

In actual use, users with different driving experiences have different views on the vehicle-mounted automatic parking system, the users with rich driving experiences pay more attention to the parking efficiency, the users often feel that partial functions of the automatic parking system are redundant, the automatic parking algorithm consumes longer time, and actual requirements cannot be met; the automatic parking system has the advantages that users with less driving experience are prone to the loss of parking skills, parking efficiency is not the main problem which disturbs the users, but the existing automatic parking system is poor in applicability and cannot meet parking requirements in most parking scenes, the reason is that high-quality training samples of an automatic parking decision model are few, the automatic parking decision model is only suitable for several common parking scenes, the generalization capability is poor, parking spaces cannot be effectively identified in some special parking scenes, or vehicles are guided to be automatically parked, although simulation technology is used for simulating some extreme parking scenes currently, the simulation scene precision is not high, the problem of transplantation to a real scene exists, time consumption is long, the data sample richness is poor, and the adaptability of the trained automatic parking decision model is poor.

The embodiment of the invention utilizes technologies such as vehicle networking and the like to acquire the vehicle surrounding environment information under various actual scenes in real time, repeatedly trains the parking space recognition model based on the information, can accurately give the type of the parking space, improves the efficiency and the precision of parking space recognition, and simultaneously acquires the user decision information input by drivers with rich vehicle starting experience under various parking scenes in real time, and takes the user decision information as the standard decision under the specific parking scene to enrich the training data set of the automatic parking decision model, accelerates the acquisition progress of the training data, reduces the acquisition cost of the training data, and obtains the training data covering most actual road conditions, integrates the thought of various vehicle starting experience users, and the automatic parking decision model trained by the training data has stronger adaptability, the automatic parking system can accurately and efficiently park in various parking environments, and improves the application range and efficiency of the automatic parking system.

Examples

When an emergency situation is met in the automatic parking process, such as a tricycle is driven from the rear in the roadside side parking process, the tricycle is stopped on an automatic parking route, collision can occur if parking is carried out according to an original route, the tricycle is kept in place and can be blocked, the road congestion is caused, at the moment, a user needs to intervene to carry out manual parking, the tricycle is interacted with the tricycle, the tricycle is driven into a parking space after no obstacle is determined, in the vehicle intervention process of the user, a vehicle-mounted sensor records user decision information, associated data of parking environment information and the user decision information are established and stored, the associated data are transmitted to a cloud database through a vehicle-mounted network and the like, model parameters of an automatic parking decision model are updated, and in the subsequent parking process, if similar problems are met, the automatic parking decision model can perform more humanization on the tricycle, And the parking decision is more quick and efficient.

The stored parking environment information and user decision information associated data are used as training samples, an automatic parking decision model to be trained is repeatedly trained to obtain an optimized automatic parking decision model of the vehicle, the training data is extremely low in acquisition cost and high in accuracy, acquired sample data can be uploaded to a cloud database through a vehicle networking technology, the parking scene database is greatly enriched, the automatic parking decision model in the parking system is continuously trained and updated on the basis of the database, the automatic parking decision model has better adaptability when being applied on line, and the vehicle can output parking decision information suitable for the parking environment when the parking environment is complex.

The method can be used for acquiring parking environment information and user model decision information in an actual parking environment, and then establishing and storing an incidence relation between the parking environment information and the user model decision information.

Fig. 4 is a schematic diagram of 3 typical parking scenarios, wherein the direction of the arrow indicates the direction of vehicle motion.

(a) Linear parking space

A straight parking space is a normal lateral parking space, and is mostly seen in parking lots at two sides of a road or with limited space, as shown in fig. 4 (a), the parking space is a rectangular area including four points ABCD, a vehicle is required to be parked in the area completely, the vehicle is parked at the inner side of a dotted line AB, the head of the vehicle returns after the vehicle is parked stably, the vehicle body is parallel to the line AB and the line CD, the vehicle cannot press a line BD and a line AC during parking, and the rear edge of the vehicle cannot sweep the line CD during parking, because the CD edge of a part of the straight parking space is not a mark line on the ground but a wall body, if the line CD is swept, namely the rear edge of the vehicle is rubbed against the wall body, the direction is not allowed during parking.

The method comprises the steps that a vehicle-mounted camera is used for detecting relative positions (coordinate positions and angle positions) between a parking space, a point B and a straight line AB and a vehicle in a parking process, an ultrasonic radar is used for detecting positions of the vehicle and possible obstacles around the vehicle, collision with other objects in the parking process is prevented, when the vehicle is parked on a narrow road side, a vehicle-mounted sensor is used for detecting possible coming vehicles behind the vehicle, whether the following parking operation can affect normal running of the rear vehicle or not is judged, and after the vehicle is determined to be parked, a signal (turning light, deceleration and the like) is sent to the rear, so that the situation that the rear vehicle occupies a parking area of the vehicle to cause parking failure is avoided.

In the process of parking in fig. 4 (a), a vehicle detects that an obstacle O is running at a certain speed behind the vehicle, if the vehicle continues to park without any measures, the obstacle O may run to a parking route of the vehicle, so that parking cannot be normally performed, if the vehicle stops parking after finding the obstacle O, when the obstacle O is close to the vehicle, the method is feasible, if the obstacle O is far away from the vehicle, parking efficiency is low, and the driving experience is not met, at this time, an automatic parking decision model may not perform parking processing, a user needs to take over the control right of the vehicle, and the vehicle is reminded to slowly courtesy behind by a horn or a turn light, and the vehicle is manually operated to park with the current parking intention.

Through collecting parking environment information and driver's decision-making information (horn information, indicator information, steering wheel corner information, brake pedal aperture and accelerator pedal aperture etc.) in the whole process, construct the data sample that comes the car and interact and park with the rear under this scene, convey this data sample to high in the clouds database.

(b) Convex parking space

Convex parking spaces are usually found in relatively large parking lots and are often symmetrically distributed on two sides of a road of the parking lot, as shown in fig. 4 (B), the parking spaces are typical convex parking spaces, the parking spaces are rectangular areas contained by four points, namely A1B1C1D1, parking is required to be completed in the areas, the vehicle head should not exceed A1B1, and the vehicle tail should not exceed C1D1, during parking, the vehicle cannot sweep the line A1C1 and the line B1D1, in actual life, although parking marking lines are drawn in the parking lots, some drivers may park part of the vehicle in the parking spaces where the vehicle is to be parked, and although normal parking of the vehicle is not affected, the automatic parking decision model may encounter difficulties during execution.

As shown in (B) of fig. 4, a box M indicates an occupied parking space part, and in the parking process, the vehicle needs to avoid the position M so as to avoid collision, in this case, the automatic parking system cannot guide the vehicle to park well, a vehicle user can select manual parking according to needs, and manually park the vehicle at a proper position in the A1B1C1D1, in the whole parking process, the vehicle-mounted sensor senses and records the parking environment, and makes one-to-one correspondence with the user decision information of the vehicle in the time dimension to form sample data in the parking scene, and the automatic parking decision model is trained by collecting the sample data, so that the automatic parking system can adopt a proper parking decision to perform effective parking when encountering a similar scene.

(c) Diagonal parking space

The oblique parking spaces are mostly found on the roadside and the parking lot, the parking area is a parallelogram area which is contained by four points A2B2C2D2 in (C) of fig. 4, the vehicle is required to be finally parked in the area, the vehicle body is parallel to the side A2C2 and the side D2B2, and the vehicle needs to observe the surrounding environment all the time during parking, so that the parking safety is ensured.

As described above, in three common parking scenes, the attention points of the automatic parking model in each scene are different, specifically, in a straight-line parking space parking scene, the automatic parking decision model mainly focuses on a side rear vehicle and a vertex B, in a convex parking space parking scene, the automatic parking decision model mainly focuses on the vertices a1 and B1 of the parking space and other vehicles which may intrude into the parking space, and in a diagonal parking space parking scene, the automatic parking decision model mainly focuses on the side rear vehicle and a line segment A2C 2.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.