阅读说明：本技术 一种基于lstm神经网络算法的无人机路面病害检测方法 (Unmanned aerial vehicle pavement disease detection method based on LSTM neural network algorithm ) 是由 方宏远 王念念 马铎 胡浩帮 庞高兆 于 2021-07-13 设计创作，主要内容包括：本发明公开了一种基于LSTM神经网络算法的无人机路面病害检测方法,包括如下步骤：S1、获取无人机姿态监测数据,构建无人机时间序列数据集,并按比例分为训练集,验证集和测试集；S2、构建LSTM神经网络模型；S3、初始化模型,将训练集数据输入到模型中,设置超参数,进行模型训练；S4、模型调参,得到最优模型；S5、将测试集数据输入到最优模型进行测试,输出各项数值评价指标,判断是否达到预期值；S6、现场检测,判断评价指标能否达到所需要求。LSTM神经网络模型算法可用于对无人机姿态时间序列信息的预测,从而得到无人机距离路面病害的距离,偏角等信息,对路面病害的精确测量提供帮助该方法检测效率快,准确度高,鲁棒性好。(The invention discloses an unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm, which comprises the following steps: s1, acquiring attitude monitoring data of the unmanned aerial vehicle, constructing an unmanned aerial vehicle time sequence data set, and dividing the unmanned aerial vehicle time sequence data set into a training set, a verification set and a test set according to a proportion; s2, constructing an LSTM neural network model; s3, initializing the model, inputting the training set data into the model, setting the hyper-parameters, and performing model training; s4, model parameter adjustment to obtain an optimal model; s5, inputting the test set data into the optimal model for testing, outputting each numerical value evaluation index, and judging whether the numerical value evaluation index reaches an expected value; and S6, detecting on site, and judging whether the evaluation index can meet the required requirement. The LSTM neural network model algorithm can be used for predicting the attitude time sequence information of the unmanned aerial vehicle, so that the information such as the distance and the deflection angle between the unmanned aerial vehicle and a road surface disease is obtained, and the method is high in detection efficiency, high in accuracy and good in robustness and helps for accurate measurement of the road surface disease.)

1. An unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm is characterized by comprising the following steps:

s1, acquiring attitude monitoring data of the unmanned aerial vehicle, constructing an unmanned aerial vehicle time sequence data set, and dividing the unmanned aerial vehicle time sequence data set into a training set, a verification set and a test set according to a proportion;

s2, constructing an LSTM neural network model;

s3, initializing the model, inputting the training set data into the model, setting the hyper-parameters, and performing model training;

s4, model parameter adjustment to obtain an optimal model;

s5, inputting the test set data into the optimal model for testing, outputting each numerical value evaluation index, and judging whether the numerical value evaluation index reaches an expected value;

and S6, detecting on site, and judging whether the evaluation index can meet the required requirement.

2. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: in step S1, the acquiring the unmanned aerial vehicle attitude monitoring data and constructing the unmanned aerial vehicle time series dataset specifically includes: acquiring yaw angle, pitch angle, roll angle, longitude, latitude, altitude, barometer height and direction data by using a GPS, a speedometer, a gyroscope, a magnetic compass and a barometer sensor; and (4) considering different acquisition frequencies of all the sensors, writing a python program to enable all the data to correspond to a uniform time point.

3. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: in step S1, the proportionally dividing into training sets, the verification set and the test set specifically include: according to the following steps of 6: 2: the scale of 2 is divided into a training set, a validation set and a test set.

4. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: the LSTM neural network model in step S2 includes a forgetting gate, an input gate, an output gate, and a memory unit; the forgetting gate selectively forgets the input of the previous node; the input gate is used for judging the importance of the previous node information and selectively memorizing the previous node information; the output gate determines the output information.

5. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: the hyper-parameters in the step S3 are the learning rate, the total iteration number, the number of stacks of memory blocks, the number of small batches, and the number of predicted steps.

6. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: in the step S4, the parameter adjustment of the model to obtain the optimal model specifically includes: inputting verification set data into the model, adjusting the hyper-parameters in sequence, and obtaining the optimal hyper-parameters and the optimal model according to the loss values and the change curves of the accuracy rates of the models under different hyper-parameters.

7. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: each numerical evaluation index in the step S5 includes accuracy, mean square error, and mean absolute error;

the mean square error is an expectation value of the square of the difference between the parameter estimation value and the parameter true value, and the formula is as follows:

the average absolute error is an average of absolute errors, and the formula is as follows:

8. the unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm according to claim 1, characterized in that: in step S6, the step of performing the field test to determine whether the evaluation index can meet the required requirement specifically includes: and controlling the unmanned aerial vehicle to fly, returning unmanned aerial vehicle attitude data acquired by various sensors to a high-performance computer terminal for detection, and judging whether the evaluation index can meet the required requirement.

9. The unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm, according to claim 8, is characterized in that: the evaluation indexes mainly comprise prediction efficiency, mean square error, average absolute error and robustness.

Technical Field

The invention relates to the technical field of deep learning neural network and pavement disease detection, in particular to an unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm.

Background

With the continuous development of economy, the total mileage of the Chinese highway leaps the first in the world. However, various diseases can occur on the road surface under the influence of long-term vehicle loading and freeze-thaw cycles. The traditional detection method is manual detection or detection vehicle detection, which wastes time and labor and is huge in cost. The invention provides an unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm by combining continuity, time sequence and interactivity of flight tracks.

The traditional unmanned aerial vehicle attitude prediction model has the problems of large simplification and few consideration factors. In order to solve the problems of gradient loss and gradient explosion in the long-sequence training process and the long-term dependence of a general recurrent neural network, the algorithm of the invention simultaneously uses the attitude, position, acceleration and other information from sensors such as an IMU and the like as the input of a trajectory prediction model, so that the predicted value is more consistent with the change rule of a real trajectory. And training the established unmanned aerial vehicle attitude prediction model based on the LSTM by adjusting the learning rate, the learning step length and other super parameters to obtain an optimal model.

Patent document with application number 201410608597.8 discloses an unmanned aerial vehicle control method based on a PID neural network, and the method adopts a PID neural network control algorithm with a basic form of 2 x 3 x 1, so that a small unmanned aerial vehicle flight control model with good anti-interference capability, good robustness and high control precision is obtained. However, the neural network does not consider the influence of the continuous time on the model output, and therefore, the prediction accuracy is low.

Aiming at the problems of continuity, time sequence and the like of the flight attitude of the unmanned aerial vehicle, the invention adopts an LSTM neural network algorithm to obtain an unmanned aerial vehicle attitude monitoring model, and improves the accuracy of judging the target distance of the unmanned aerial vehicle to the diseases during the road disease detection.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides the unmanned aerial vehicle pavement disease detection method based on the LSTM neural network algorithm, the method is high in detection efficiency, high in accuracy and good in robustness, and the problems in the background art are solved.

In order to achieve the purpose, the invention provides the following technical scheme: an unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm comprises the following steps:

s1, acquiring attitude monitoring data of the unmanned aerial vehicle, constructing an unmanned aerial vehicle time sequence data set, and dividing the unmanned aerial vehicle time sequence data set into a training set, a verification set and a test set according to a proportion;

s2, constructing an LSTM neural network model;

s3, initializing the model, inputting the training set data into the model, setting the hyper-parameters, and performing model training;

s4, model parameter adjustment to obtain an optimal model;

s5, inputting the test set data into the optimal model for testing, outputting each numerical value evaluation index, and judging whether the numerical value evaluation index reaches an expected value;

and S6, detecting on site, and judging whether the evaluation index can meet the required requirement.

Preferably, in step S1, the acquiring the unmanned aerial vehicle attitude monitoring data and constructing the unmanned aerial vehicle time series data set specifically includes: acquiring yaw angle, pitch angle, roll angle, longitude, latitude, altitude, barometer height and direction data by using a GPS, a speedometer, a gyroscope, a magnetic compass and a barometer sensor; and (4) considering different acquisition frequencies of all the sensors, writing a python program to enable all the data to correspond to a uniform time point.

Preferably, in step S1, the proportionally dividing into the training set and the verifying set and the testing set specifically include: according to the following steps of 6: 2: the scale of 2 is divided into a training set, a validation set and a test set.

Preferably, the LSTM neural network model in step S2 includes a forgetting gate, an input gate, an output gate, and a memory unit; the forgetting gate selectively forgets the input of the previous node; the input gate is used for judging the importance of the previous node information and selectively memorizing the previous node information; the output gate determines the output information.

Preferably, the hyper-parameters in step S3 are learning rate, total iteration number, stack number of memory blocks, small batch number, and predicted step number.

Preferably, in step S4, the parameter tuning of the model to obtain the optimal model specifically includes: inputting verification set data into the model, adjusting the hyper-parameters in sequence, and obtaining the optimal hyper-parameters and the optimal model according to the loss values and the change curves of the accuracy rates of the models under different hyper-parameters.

Preferably, the numerical evaluation indexes in step S5 include accuracy, mean square error, and mean absolute error;

the mean square error is an expectation value of the square of the difference between the parameter estimation value and the parameter true value, and the formula is as follows:

the average absolute error is an average of absolute errors, and the formula is as follows:

wherein, N represents the number of the real values (or predicted values), and i represents the number corresponding to the real values (or predicted values).

Preferably, in step S6, the field test and the determination of whether the evaluation index can meet the required requirement specifically include: and controlling the unmanned aerial vehicle to fly, returning unmanned aerial vehicle attitude data acquired by various sensors to a high-performance computer terminal for detection, and judging whether the evaluation index can meet the required requirement.

Preferably, the evaluation index mainly includes prediction efficiency, mean square error, mean absolute error and robustness.

The invention has the beneficial effects that: the LSTM network of the invention is an excellent variant model of the recurrent neural network, and inherits the characteristics of most recurrent neural networks. By adding a CEC (constant error carrousel) module, the error is ensured to flow in a constant form in the network, and the problem of gradient disappearance caused by gradual reduction in the gradient back propagation process is solved. The LSTM network updates the state of the memory unit through the forgetting gate, the input gate and the output gate, so that the long-term dependence problem existing in the recurrent neural network is improved, and the performance of the LSTM network is better than that of a time recurrent neural network and a hidden Markov model; as a nonlinear model, the LSTM network can be used as a complex nonlinear unit for constructing a larger deep neural network, so that more time series characteristics are extracted, and a more excellent prediction result is obtained. The LSTM neural network algorithm can be used for predicting the attitude time sequence information of the unmanned aerial vehicle, so that the information such as the distance and the deflection angle of the unmanned aerial vehicle from a road surface disease is obtained, and the accurate measurement of the road surface disease is facilitated.

Drawings

FIG. 1 is a schematic flow diagram of the process of the present invention;

FIG. 2 is a schematic diagram of the LSTM neural network model structure of the present invention;

fig. 3 is a schematic diagram of the prediction result of the present invention, wherein (a) is a curve of the true value and the predicted value of the pitch angle, (b) is a curve of the true value and the predicted value of the roll angle, and (c) is a curve of the true value and the predicted value of the yaw angle.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1-3, the present invention provides a technical solution: the main structure of the network model is a memory block, and the network model comprises four main parts, namely a forgetting gate, an input gate, an output gate and a memory unit. The method has the advantages of high detection efficiency, high accuracy and good robustness, the specific steps are shown in the flow chart of figure 1, and the steps are as follows:

step one, acquiring unmanned aerial vehicle attitude monitoring data, constructing an unmanned aerial vehicle time sequence data set, and performing the following steps according to the data of 6: 2: the proportion of 2 is divided into a training set, a verification set and a test set, and the specific process is as follows:

in particular, neural network algorithms require a large amount of data to train. The invention is an unmanned aerial vehicle attitude prediction model, so that unmanned aerial vehicle attitude monitoring data needs to be collected. Through a large amount of training, the LSTM neural network learns the influence of various parameters in the monitoring data on the predicted value, and therefore the unmanned aerial vehicle attitude prediction model is obtained. And the network is tuned and tested through the verification set and the test set data. Therefore, the unmanned aerial vehicle time sequence data set is the first step of model construction and plays a crucial role in the construction of the unmanned aerial vehicle attitude prediction model.

The method for acquiring the pavement disease image comprises the following steps: attitude data of actual flight of the unmanned aerial vehicle;

because the acquisition frequency of each sensor is different, a python program is written, so that each data corresponds to a uniform time point.

In particular, the LSTM network requires various items of data at the same point in time. Therefore, the data is augmented using linear interpolation such that the data is one-to-one for each time point.

Specifically, the attitude monitoring data of the unmanned aerial vehicle comes from sensors such as a GPS, a speedometer, a gyroscope, a magnetic compass and a barometer, and mainly comprises data such as a yaw angle, a pitch angle, a roll angle, longitude, latitude, altitude, the height of the barometer and direction.

Specifically, the functions of the various sensors are as follows:

the accelerometer is used for providing acceleration force borne by the unmanned aerial vehicle in the directions of three axes XYZ;

the gyroscope sensor can monitor the angular velocity of three axes, so that the change rate of the angle during pitching, rolling and yawing can be monitored;

the magnetic compass can provide the sense of direction for unmanned aerial vehicle. It can provide the data of the magnetic field of the device in each axial direction of XYZ. Then the relevant data will be imported into the algorithm of the microcontroller to provide the heading angle relevant to magnetic north pole, and then the information can be used for detecting the geographical position;

the principle of the barometer operation is to convert the altitude by using the atmospheric pressure. The pressure sensor can detect the atmospheric pressure of the earth. The data provided by the barometer can assist the drone in navigating to the desired altitude. Accurately estimating ascending and descending speeds;

the GPS module is one of global navigation systems, which is a satellite navigation 2 system in the united states, and can obtain position information of the unmanned aerial vehicle body through the GPS.

Step two, constructing an LSTM neural network model;

the main structure of the LSTM neural network (Long Short Term Memory network, which is a special recurrent neural network) algorithm is a Memory block, which mainly includes a forgetting gate, an input gate, an output gate, and a Memory unit. Specifically, the function of each part of the memory block is as follows:

the input gate is used for judging the importance of the previous node information and selectively memorizing the information. The information input from the previous layer at each moment firstly passes through an input gate which determines whether information is input into the memory block of the layer at the moment;

for the information in the memory block at each moment, the forgetting gate is responsible for selectively forgetting the information;

the output gate determines the output information, namely whether information is output from the layer of memory block at each moment;

the forgetting gate, the input gate and the output gate jointly form a memory unit, which is the minimum structural unit in the LSTM neural network and is responsible for overall processing of long-time sequence data.

As shown in FIG. 2, the structure of the LSTM neural network model is schematically shown, wherein "×" represents scaling information, "+" represents adding information, ". sigma" represents Sigmoid function layer, tanh represents hyperbolic tangent function layer, and h represents_t-1Representing the output of the last memory block, X_tRepresents the layer input, h_tRepresenting the output of the memory block. During the information transmission, the state of each transmission unit is determinedThe core of the LSTM network runs through the entire structure. In this process, invariance to information transmission is ensured by some linear action.

Specifically, through the forgetting gate, the input gate and the output gate, the LSTM neural network can add and remove information transmitted in the cells, and selectively allow the information to pass through, thereby achieving the purpose of managing information transmission.

Specifically, in the training process, the conventional RNN model is more likely to be updated according to the weight direction at the end of the sequence, i.e., the influence of the farther sequence input on the weight change is smaller. Therefore, the training results are often biased towards new information, i.e. less capable of longer memory function. In order to overcome the problem of long-term error disappearance, some limitations need to be made, and the error at the time t is calculated as follows:

δ_j(t)＝f_j'(s_j(t))δ_j(t+1)w_jj

in order to prevent the error from changing, the correlation coefficient can be forced

f_j'(s_j(t))w_jj＝1

The following can be obtained:

this indicates that the activation function is linear. In order to keep the error from changing, let f often_j(x)＝x，w_jjA constant error stream is thus obtained, also called cec (constant error carrousel).

And step three, initializing the model, inputting the training set data into the model, setting the hyper-parameters, and performing model training.

And initializing the model by adopting a transfer learning method according to the LSTM neural network model constructed in the step two. And (4) importing an LSTM neural network model according to the unmanned aerial vehicle time sequence data set (training set) constructed in the step one. And setting hyper-parameters and training the model.

Specifically, for the initialization of random parameters, the transfer learning is to transfer labeled data or knowledge structures from related fields, and to complete or improve the learning effect of a target field or task, thereby having the effects of accelerating the training efficiency and improving the detection accuracy.

Specifically, based on the high bandwidth and multithreading parallel computing characteristics of the GPU, the deep learning algorithm has to be performed on a high performance computer. Meanwhile, in order to guarantee the real-time performance of prediction and the portability of equipment, the LSTM neural network algorithm runs on an airborne Nvidia Jetson nano development board, and prediction data are transmitted to the unmanned aerial vehicle in real time.

Step four, model parameter adjustment to obtain an optimal model

And introducing verification set data for testing according to the model obtained in the step three. And adjusting the hyper-parameters according to the sequence of the learning rate, the total iteration times, the stacking number of the memory blocks, the small batch number and the predicted step number. The most important hyper-parameters are the learning rate and the total number of iterations, and therefore, should be adjusted preferentially. Comparing the loss values of the models under different hyper-parameters and the change curves of the accuracy, wherein the curves are shown in fig. 3, fig. 3(a) is a pitch angle-time curve, fig. 3(b) is a rolling angle-time curve, fig. 3(c) is a yaw angle-time curve, a solid line represents a real value, a dotted line represents a predicted value, searching for the optimal hyper-parameter, and determining the optimal model.

Specifically, the main hyper-parameters are the learning rate, the total iteration number, the stacking number of memory blocks, the small batch number, the predicted step number, and the like.

Specifically, the learning rate determines the speed of weight update. In the process of model training, the model is easy to cross an optimal value due to overlarge learning rate, so that overfitting is caused; too little learning rate can slow the gradient descent process too slowly. The learning rate needs to be set based on experience and constant experimentation.

In particular, the number of small batches determines the direction in which the number gradient decreases. The small batch quantity is too small, so that the sample difference is too large easily, and convergence is difficult. An excessively large number of small batches will make the gradient direction substantially stable, but will easily fall into a locally optimal solution, reducing accuracy.

Specifically, the optimal hyper-parameter is a model with the minimum loss value and the maximum accuracy, namely the optimal model, obtained by adjusting the model under different hyper-parameter values.

And step five, inputting the test set data into the optimal model obtained in the step four for testing, outputting each numerical value evaluation index, and judging whether the expected value is reached.

Specifically, each numerical evaluation index includes accuracy, mean square error and mean absolute error;

the mean square error is an expectation value of the square of the difference between the parameter estimation value and the parameter true value, and the formula is as follows:

the average absolute error is an average of absolute errors, and the formula is as follows:

specifically, the requirement is satisfied when the mean square error and the mean absolute error are less than 0.1.

Step six, field detection is carried out to judge whether the evaluation index can meet the required requirement

And controlling the unmanned aerial vehicle to fly, returning the attitude data of the unmanned aerial vehicle acquired by various sensors to a high-performance computer terminal, detecting by adopting the optimal model obtained in the fifth step, and judging whether the evaluation index can meet the required requirement.

Specifically, the evaluation index mainly includes prediction efficiency, mean square error, mean absolute error, robustness, and the like.

In particular, robustness refers to the ability of the prediction system to maintain performance under uncertain disturbances. Therefore, the model should be tested under different conditions to judge the robustness of the model.

Specifically, when the prediction efficiency is higher than the data acquisition speed, the mean square error and the average absolute error are less than 0.1, that is, the requirement is met.

The invention provides an unmanned aerial vehicle pavement disease detection method based on an LSTM neural network algorithm, the LSTM neural network algorithm suitable for road disease detection is researched and developed, model training is carried out based on unmanned aerial vehicle sensor monitoring data, the accuracy and robustness of a model are improved, and the automatic prediction of the unmanned aerial vehicle posture is realized.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and/or modifications of the invention can be made, and equivalents and modifications of some features of the invention can be made without departing from the spirit and scope of the invention.