阅读说明：本技术 视频去雨的方法、装置、设备及计算机可读存储介质 (Method, device and equipment for removing rain from video and computer readable storage medium ) 是由 刘家瑛 杨文瀚 魏晨 杨帅 郭宗明 于 2018-05-30 设计创作，主要内容包括：本发明提供了一种视频去雨的方法、装置、设备及计算机可读存储介质。该方法包括：搭建进行视频去雨的循环深度神经网络模型；对循环深度神经网络模型进行训练,以获得优化后的循环深度神经网络模型；采用优化后的循环深度神经网络模型对待处理的含雨视频进行去雨操作。由于循环深度神经网络模型能够编码相邻帧之间的背景信息,抽取更具有表示性的特征,能够整合含雨视频中的时域和空域上的冗余信息,所以能够达到很好的视频去雨的效果,使恢复出的视频更加准确和清晰。(The invention provides a method, a device and equipment for removing rain from a video and a computer readable storage medium. The method comprises the following steps: building a circulating deep neural network model for video rain removal; training the circulation depth neural network model to obtain an optimized circulation depth neural network model; and carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model. Because the cycle depth neural network model can encode the background information between adjacent frames, extract the characteristic with representation more, can integrate the redundant information on time domain and airspace in the video containing rain, so can reach the effect of good video rain removal, make the video that resumes more accurate and clear.)

1. a method for video rain removal, comprising:

Building a circulating deep neural network model for video rain removal;

training the circulation depth neural network model to obtain an optimized circulation depth neural network model;

and carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model.

2. The method according to claim 1, wherein the building of the cyclic depth neural network model for video rain removal specifically comprises:

Building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video;

Constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image;

Constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image;

Building a rain removing sub-network, wherein the rain removing sub-network generates a predicted raindrop image of the current frame image according to the spatial domain characteristics of the current frame image;

Constructing a reconstruction sub-network, wherein the reconstruction sub-network obtains a prediction detail characteristic diagram of the current frame image according to the integrated time domain characteristic of the current frame image;

Building a rain removal reconstruction joint sub-network, wherein the rain removal reconstruction joint sub-network generates a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image;

The convolutional neural network, the degradation discrimination sub-network, the fusion sub-network, the rain removal sub-network, the reconstruction sub-network, and the rain removal reconstruction joint sub-network are part of a cyclic depth neural network model in sequence.

3. The method according to claim 2, wherein the training of the cycle deep neural network model to obtain the optimized cycle deep neural network model specifically comprises:

Adding a mixed raindrop model to each rainless video of a training set, wherein each video added with the mixed raindrop model is each training sample in the training set;

And inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model.

4. The method according to claim 3, wherein the hybrid raindrop model is specifically: o is_t＝(1-α_t)(B_t+S_t)+α_tA_t；

Wherein, O_trepresenting the t-th frame image, B, in said rain-containing video_tShowing the rain-free background map corresponding to the t-th frame image, S_tShows a raindrop pattern corresponding to the t-th frame image, A_tRepresenting a low-transparency raindrop occlusion diagram corresponding to the t-th frame image; alpha is alpha_tthen representing a raindrop shielding area diagram corresponding to the t-th frame image;

the combined loss function is specifically: l is_all＝L_joint+λ_dL_detect+λ_cL_rect+λ_rL_removal；

Wherein L is_jointAs a joint loss function, L_detectDetective loss function for raindrop sheltering_rectFor reconstruction of the loss function, L_removalAs a function of rain loss, λ_d，λ_cAnd λ_rAnd respectively blocking rain removal detection loss functions for raindrops, and reconstructing weights corresponding to the loss functions and the rain removal loss functions.

5. The method according to any one of claims 1-4, wherein after performing a rain removal operation on the rain-containing video to be processed by using the optimized cyclic depth neural network model, the method further comprises:

And integrating the predicted rain-free background image of each frame corresponding to the rain-containing video to be processed to form a predicted rain-free video.

6. A video de-raining apparatus, comprising:

the building module is used for building a circulating depth neural network model for video rain removal;

the training module is used for training the circulation depth neural network model to obtain an optimized circulation depth neural network model;

And the rain removing module is used for carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating depth neural network model.

7. The device according to claim 6, characterized in that the building module is specifically configured to:

Building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video; constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image; constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image; building a rain removing sub-network, wherein the rain removing sub-network generates a predicted raindrop image of the current frame image according to the spatial domain characteristics of the current frame image; constructing a reconstruction sub-network, wherein the reconstruction sub-network obtains a prediction detail characteristic diagram of the current frame image according to the integrated time domain characteristic of the current frame image; building a rain removal reconstruction joint sub-network, wherein the rain removal reconstruction joint sub-network generates a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image; the convolutional neural network, the degradation discrimination sub-network, the fusion sub-network, the rain removal sub-network, the reconstruction sub-network, and the rain removal reconstruction joint sub-network are part of a cyclic depth neural network model in sequence.

8. The apparatus of claim 7, wherein the training module is specifically configured to:

Adding a mixed raindrop model to each rainless video of a training set, wherein each video added with the mixed raindrop model is each training sample in the training set; and inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model.

9. the apparatus according to claim 8, wherein the hybrid raindrop model is specifically: o is_t＝(1-α_t)(B_t+S_t)+α_tA_t；

wherein, O_trepresenting the t-th frame image, B, in said rain-containing video_tshowing the rain-free background map corresponding to the t-th frame image, S_tShows a raindrop pattern corresponding to the t-th frame image, A_tRepresenting a low-transparency raindrop occlusion diagram corresponding to the t-th frame image; alpha is alpha_tThen representing a raindrop shielding area diagram corresponding to the t-th frame image;

The combined loss function is specifically: l is_all＝L_joint+λ_dL_detect+λ_cL_rect+λ_rL_removal；

Wherein L is_jointAs a joint loss function, L_detectDetective loss function for raindrop sheltering_rectFor reconstruction of the loss function, L_removalAs a function of rain loss, λ_d，λ_cAnd λ_rAnd respectively blocking rain removal detection loss functions for raindrops, and reconstructing weights corresponding to the loss functions and the rain removal loss functions.

10. The apparatus of any one of claims 6-9, further comprising:

And the integration module is used for integrating the predicted rain-free background image of each frame corresponding to the rain-containing video to be processed to form a predicted rain-free video.

11. A video de-raining apparatus, comprising:

A memory, a processor, and a computer program;

Wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-5.

12. A computer-readable storage medium, having stored thereon a computer program for execution by a processor to perform the method of any one of claims 1-5.

Technical Field

The embodiment of the invention relates to the technical field of video processing, in particular to a method, a device and equipment for removing rain from a video and a computer readable storage medium.

Background

Videos shot in severe weather conditions are often faced with a series of problems such as image blur, image content coverage, etc. Raindrops, the most common factor affecting rain-containing video, often cause occlusion and blurring of local areas. This signal distortion and loss of detail can affect the performance of many outdoor vision applications that use high quality video as input. Therefore, the detection and removal of raindrops in the video is of great significance.

The existing video rain removing method utilizes information redundancy of a space domain and a time domain, and is based on a physical modeling method, such as video rain removing by utilizing the directivity and color characteristics of raindrops. Or video de-raining is performed by using dynamic time domain information, such as continuity of a video background and randomness of raindrops.

However, these methods do not fully consider the relationship between the temporal and spatial redundant information, which results in inaccurate and clear video restored by the existing video rain-removing method.

Disclosure of Invention

The embodiment of the invention provides a method, a device and equipment for removing rain from a video and a computer readable storage medium, which solve the technical problem that the video recovered by the video rain removing method in the prior art is inaccurate and clear.

in a first aspect, an embodiment of the present invention provides a method for removing rain from a video, including:

Building a circulating deep neural network model for video rain removal;

Training the circulation depth neural network model to obtain an optimized circulation depth neural network model;

And carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model.

Further, according to the method, the building of the cyclic depth neural network model for video rain removal specifically includes:

building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video;

Constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image;

constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image;

Building a rain removing sub-network, wherein the rain removing sub-network generates a predicted raindrop image of the current frame image according to the spatial domain characteristics of the current frame image;

Constructing a reconstruction sub-network, wherein the reconstruction sub-network obtains a prediction detail characteristic diagram of the current frame image according to the integrated time domain characteristic of the current frame image;

Building a rain removal reconstruction joint sub-network, wherein the rain removal reconstruction joint sub-network generates a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image;

the convolutional neural network, the degradation discrimination sub-network, the fusion sub-network, the rain removal sub-network, the reconstruction sub-network, and the rain removal reconstruction joint sub-network are part of a cyclic depth neural network model in sequence.

further, in the method, the training of the cycle deep neural network model to obtain the optimized cycle deep neural network model specifically includes:

Adding a mixed raindrop model to each rainless video of a training set, wherein each video added with the mixed raindrop model is each training sample in the training set;

And inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model.

Further, in the method, the mixed raindrop model specifically includes: o is_t＝

(1-α_t)(B_t+S_t)+α_tA_t；

Wherein, O_tRepresenting the t-th frame image, B, in said rain-containing video_tShowing the rain-free background map corresponding to the t-th frame image, S_tShows a raindrop pattern corresponding to the t-th frame image, A_tRepresenting a low-transparency raindrop occlusion diagram corresponding to the t-th frame image; alpha is alpha_tThen representing a raindrop shielding area diagram corresponding to the t-th frame image;

The combined loss function is specifically: l is_all＝L_joint+λ_dL_detect+λ_cL_rect+λ_rL_removal；

wherein L is_jointas a joint loss function, L_detectDetective loss function for raindrop sheltering_rectfor reconstruction of the loss function, L_removalAs a function of rain loss, λ_d，λ_cAnd λ_rAnd respectively blocking rain removal detection loss functions for raindrops, and reconstructing weights corresponding to the loss functions and the rain removal loss functions.

Further, the method, after performing a rain removing operation on the rain-containing video to be processed by using the optimized cyclic depth neural network model, further includes:

and integrating the predicted rain-free background image of each frame corresponding to the rain-containing video to be processed to form a predicted rain-free video.

In a second aspect, an embodiment of the present invention provides a video rain removing device, including:

The building module is used for building a circulating depth neural network model for video rain removal;

The training module is used for training the circulation depth neural network model to obtain an optimized circulation depth neural network model;

and the rain removing module is used for carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating depth neural network model.

Further, as for the apparatus described above, the building module is specifically configured to:

Building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video; constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image; constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image; building a rain removing sub-network, wherein the rain removing sub-network generates a predicted raindrop image of the current frame image according to the spatial domain characteristics of the current frame image; constructing a reconstruction sub-network, wherein the reconstruction sub-network obtains a prediction detail characteristic diagram of the current frame image according to the integrated time domain characteristic of the current frame image; building a rain removal reconstruction joint sub-network, wherein the rain removal reconstruction joint sub-network generates a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image; the convolutional neural network, the degradation discrimination sub-network, the fusion sub-network, the rain removal sub-network, the reconstruction sub-network, and the rain removal reconstruction joint sub-network are part of a cyclic depth neural network model in sequence.

Further, in the apparatus as described above, the training module is specifically configured to:

Adding a mixed raindrop model to each rainless video of a training set, wherein each video added with the mixed raindrop model is each training sample in the training set; and inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model.

Further, in the above apparatus, the mixed raindrop model specifically includes: o is_t＝(1-α_t)(B_t+S_t)+α_tA_t；

Wherein, O_trepresenting the t-th frame image, B, in said rain-containing video_tShowing the rain-free background map corresponding to the t-th frame image, S_tShows a raindrop pattern corresponding to the t-th frame image, A_tRepresenting a low-transparency raindrop occlusion diagram corresponding to the t-th frame image; alpha is alpha_tThen representing a raindrop shielding area diagram corresponding to the t-th frame image;

The combined loss function is specifically: l is_all＝L_joint+λ_dL_detect+λ_cL_rect+λ_rL_removal；

wherein L is_jointAs a joint loss function, L_detectDetective loss function for raindrop sheltering_rectfor reconstruction of the loss function, L_removalAs a function of rain loss, λ_d，λ_cand λ_rAnd respectively blocking rain removal detection loss functions for raindrops, and reconstructing weights corresponding to the loss functions and the rain removal loss functions.

Further, the apparatus as described above, further comprising:

And the integration module is used for integrating the predicted rain-free background image of each frame corresponding to the rain-containing video to be processed to form a predicted rain-free video.

In a third aspect, an embodiment of the present invention provides a video rain removing device, including:

a memory, a processor, and a computer program;

Wherein the computer program is stored in the memory and configured to be executed by the processor to implement a method as claimed in any one of the above.

In a fourth aspect, embodiments of the invention provide a computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement a method as described in any one of the above.

the embodiment of the invention provides a method, a device and equipment for removing rain from a video and a computer readable storage medium, wherein a circulating deep neural network model for removing rain from the video is built; training the circulation depth neural network model to obtain an optimized circulation depth neural network model; and carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model. Because the cycle depth neural network model can encode the background information between adjacent frames, extract the characteristic with representation more, can integrate the redundant information on time domain and airspace in the video containing rain, so can reach the effect of good video rain removal, make the video that resumes more accurate and clear.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of a first embodiment of a video rain removal method according to the present invention;

FIG. 2 is a flowchart of a second embodiment of a video rain removal method according to the present invention;

FIG. 3 is a schematic structural diagram of a first video rain removing apparatus according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a second embodiment of the video rain removal apparatus of the present invention;

Fig. 5 is a schematic structural diagram of a video rain removing apparatus according to a first embodiment of the present invention.

Detailed Description

in order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

Fig. 1 is a flowchart of a first embodiment of a video rain removing method according to the present invention, and as shown in fig. 1, an execution subject of the embodiment is a video rain removing device, and the video rain removing device may be integrated in a computer, a notebook computer, a server, or an apparatus with independent computing and processing capabilities, and the video rain removing method provided in the embodiment includes the following steps.

Step 101, building a cyclic depth neural network model for video rain removal.

specifically, in this embodiment, a cyclic deep neural network model is built to perform video rain removal operation. Firstly, spatial domain characteristics of each frame of image in the rain-containing video are extracted through a convolutional neural network in a cyclic depth neural network model. And then obtaining the predicted raindrop sheltering area image by the spatial domain characteristics of each frame of image through a degradation judgment sub-network in the circulating depth neural network model, and fusing the degradation characteristics, the spatial domain characteristics and the integration time domain characteristics of the predicted raindrop sheltering area image of each frame of image through a fusion sub-network in the circulating depth neural network model. And distinguishing the raindrop textures from the background by utilizing the spatial characteristics of each frame of image through a rain removing sub-network in the cycle depth neural network model. And reconstructing lost information in the raindrop shielding region by utilizing the integrated time domain characteristics of the previous frame of image through a reconstruction sub-network in the cyclic depth neural network model. And finally, restoring the clear background of each frame through a reconstruction joint sub-network in the cycle depth neural network model.

the convolutional neural network, the degradation discrimination subnetwork, the fusion subnetwork, the rain removal subnetwork, the reconstruction subnetwork and the rain removal reconstruction joint subnetwork are part of the cycle depth neural network model in sequence.

And 102, training the circulation depth neural network model to obtain the optimized circulation depth neural network model.

Specifically, in this embodiment, each training sample is input into the cycle deep neural network model to train each parameter in the cycle deep neural network model, so as to obtain the optimized cycle deep neural network model.

The cycle depth neural network model may be trained by using a gradient descent algorithm, or may be trained by using other algorithms, which is not limited in this embodiment.

And 103, carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model.

Specifically, in this embodiment, the actual rain-containing video is input into the optimized cyclic depth neural network model in frames, the optimized cyclic depth neural network model performs a rain removing operation on each frame of rain-containing image to obtain a predicted rain removing image of each frame of image, and the predicted rain removing images of each frame are integrated to form the predicted rain-free video.

According to the method for removing rain from the video, a cyclic depth neural network model for removing rain from the video is built; training the circulation depth neural network model to obtain an optimized circulation depth neural network model; and carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model. Because the cycle depth neural network model can encode the background information between adjacent frames, extract the characteristic with representation more, can integrate the redundant information on time domain and airspace in the video containing rain, so can reach the effect of good video rain removal, make the video that resumes more accurate and clear.

Fig. 2 is a flowchart of a second video rain removing method according to the present invention, and as shown in fig. 2, the video rain removing method provided in this embodiment is further refined in steps 101 to 102 on the basis of the first video rain removing method according to the present invention, and further includes a step of integrating a predicted no-rain background image of each frame corresponding to a video containing rain to be processed to form a predicted no-rain video.

Step 201, a cyclic depth neural network model for video rain removal is built.

Further, in this embodiment, the cycle deep neural network model includes: a convolutional neural network, a degradation discrimination subnetwork, a fusion subnetwork, a rain removal subnetwork, a reconstruction subnetwork, and a rain removal reconstruction joint subnetwork. Namely, the convolutional neural network, the degradation discrimination subnetwork, the fusion subnetwork, the rain removal subnetwork, the reconstruction subnetwork, and the rain removal reconstruction joint subnetwork are part of the cycle depth neural network model in turn.

Further, in this embodiment, the building of the cyclic depth neural network model for video rain removal specifically includes:

Firstly, a convolutional neural network is built, and spatial domain features of a current frame image of the rain-containing video are extracted by the convolutional neural network.

Specifically, for each frame image O in the rain-containing video image_tFirstly, a Convolutional Neural Network (CNN) is used to extract the spatial domain characteristics of the current frame image of the rain-containing video, which is expressed as F_tWherein t represents that the current frame image is the t-th frame image.

The structure of the convolutional neural network is the prior art, and is not described herein again.

secondly, a degradation discrimination sub-network is built, and the degradation discrimination sub-network is used for discriminating the spatial domain characteristic F of the current frame image_tIntegration temporal feature H with previous frame image_t-1Obtaining a predicted raindrop occlusion region map of a current frame image

Specifically, in this embodiment, a degradation discrimination subnetwork in the cycle deep neural network model is constructed. The structure of the degradation discrimination subnetwork is the same as that of the single-frame convolutional neural network. Spatial domain characteristic F of current frame image_tIntegration temporal feature H with previous frame image_t-1Inputting the raindrop occlusion regions into a degradation judgment sub-network together to distinguish raindrop occlusion regions from other regions in the current frame image to obtain a predicted raindrop occlusion region map of the current frame image

Thirdly, constructing a fusion sub-network, wherein the fusion sub-network is based on the spatial domain characteristics F of the current frame image_tIntegration temporal feature H of previous frame image_t-1And predicting raindrop occlusion region map of current frame imageConstraint feature f_t,4Obtaining an integrated time-domain feature H of a current frame image_t。

wherein the raindrop shielding region map is predictedconstraint feature f_t,4Is formed byAnd performing multilayer convolution to obtain the product.

Specifically, in this embodiment, a fusion sub-network in the cycle deep neural network model is constructed, and a gating cycle unit is applied to the structure of the fusion sub-network. To progressively fuse internal features, a read gate r is included in the fused sub-network_tand an update gate z_tWherein the read gate r_texpressed as formula (1), update gate z_tThe expression is shown as formula (2).

r_t＝ReLU(W_rF_t+U_rH_t-1+V_rf_t,4) (1)

Wherein W_r，U_r，V_ris the convolutional layer weight, f_t,4for areas shielded by raindropsThe characteristic of the constraint, ReLU (·), represents a linear rectifying unit.

z_t＝σ(W_zF_t+U_zH_t-1+V_zf_t,4) (2)

Wherein W_z，U_z，V_zis the convolutional layer weight; σ (-) denotes the activation function.

Integrated reading gate r_tand an update gate z_tAnd (4) obtaining the integrated time domain characteristics of the current frame image by information, wherein the integrated time domain characteristics are expressed by an expression (3) and an expression (4).

Wherein W_h，U_hAre convolutional layer weights. tanh (. cndot.) is hyperbolicThe function of the tangent is a function of the tangent,Representing a bitwise multiplication operation.

Then, a rain removing sub-network is built, and the rain removing sub-network is used for removing rain according to the spatial domain characteristics F of the current frame image_tGenerating a predicted raindrop map of a current frame image

specifically, in the implementation, a rain removing sub-network in the cyclic deep neural network model is built, and the structure of the rain removing sub-network is the same as that of the single-frame convolutional neural network. Rain removing subnetwork utilizing spatial domain characteristic F of current frame image_tDifferentiating raindrop texture from background to produce a predicted raindrop map

Then, a reconstruction sub-network is built, and the reconstruction sub-network integrates the time domain characteristics H according to the current frame image_tobtaining the prediction detail characteristic map of the current frame image

specifically, in this embodiment, a reconstruction sub-network in the cycle deep neural network model is constructed, and the structure of the reconstruction sub-network is the same as that of the single-frame convolutional neural network. Reconstruction sub-network utilizes integrated time domain characteristics of current frame image to reconstruct and predict raindrop sheltering area imagePredicting the predicted detail characteristic map of the current frame imageWhere E (-) denotes a high pass filter.

And finally, building a rain removal reconstruction joint sub-network, wherein the rain removal reconstruction joint sub-network predicts a detailed feature map according to the current frame imagegenerating a predicted rainless background map of a current frame image

Specifically, in this embodiment, a rain removal reconstruction joint sub-network in the cyclic depth neural network model is constructed, the structure of the rain removal reconstruction joint sub-network is the same as that of the single-frame convolutional neural network, and the rain removal reconstruction joint sub-network is based on the predicted detail feature map of the current frame imageGenerating a predicted rainless background map of a current frame image

Step 202, adding the mixed raindrop model to each rainless video of the training set, wherein each video added with the mixed raindrop model is each training sample in the training set.

Further, in the present embodiment, in order to describe the phenomenon that the low-transparency raindrops completely cover the background and better describe the shielding of the raindrops from the background, a mixed raindrop model is proposed. The mixed raindrop model is expressed by equation (5):

O_t＝(1-α_t)(B_t+S_t)+α_tA_t (5)

Wherein, O_tRepresenting the t-th frame of the video containing rain, B_tShowing the rain-free background map corresponding to the t-th frame image, S_tShows a raindrop pattern corresponding to the t-th frame image, A_tAnd showing a low-transparency raindrop occlusion diagram corresponding to the t-th frame image. Alpha is alpha_tThe raindrop occlusion region map corresponding to the t-th frame image is represented. Alpha is alpha_tIs expressed by the formula (6):

Wherein the content of the first and second substances,Ω_sIndicating a low transparency raindrop area.

specifically, in this embodiment, the mixed raindrop model is added to each rainless video of the training set frame by frame. Each video added to the hybrid raindrop model is each training sample in the training set.

And 203, inputting each training sample into the cycle depth neural network model, constraining by the combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm to adjust each parameter in the cycle depth neural network model to obtain the optimized cycle depth neural network model.

wherein, the combination loss function is specifically expressed as shown in formula (7):

L_all＝L_joint+λ_dL_detect+λ_cL_rect+λ_rL_removal (7)

Wherein L is_jointas a joint loss function, L_detectDetective loss function for raindrop sheltering_rectFor reconstruction of the loss function, L_removalas a function of rain loss, λ_d，λ_cAnd λ_rand respectively blocking rain removal detection loss functions for raindrops, and reconstructing weights corresponding to the loss functions and the rain removal loss functions.

For lambda_d，λ_cAnd λ_rThe value of (a) may be 0.001, 0.0001 and 0.0001, respectively, or other suitable values, which is not limited in this embodiment.

wherein a joint loss function L_jointRain drop blocking rain removal detection loss function L_detectrebuilding the loss function L_rectand a rain loss function L_removalRespectively expressed by formula (8), formula (9), formula (10) and formula (11):

WhereinRepresenting a predicted rainless background map of the t-th frame, b_tRepresenting an actual rain-free background map for the t-th frame.

L_detect＝log(∑_k＝1,2exp(f_t,2(k)))-α_t (9)

Wherein f is_t,2Features of raindrop occlusion region, alpha, representing the t-th frame extracted by the degradation discrimination subnetwork_tShowing a map of the actual raindrop occlusion area.

Wherein E (-) denotes a high-pass filter.

wherein the content of the first and second substances,Predicted raindrop pattern for the t-th frame, s_tis the real raindrop image of the t-th frame.

And 204, carrying out rain removing operation on the rain-containing video to be processed by adopting the optimized circulating deep neural network model.

in this embodiment, the implementation manner of step 204 is the same as the implementation manner of step 103 in the first embodiment of the video rain removal method of the present invention, and details are not repeated here.

And step 205, integrating the predicted rain-free background image of each frame corresponding to the rain-containing video to be processed to form the predicted rain-free video.

Further, in this embodiment, the predicted no-rain background image of each frame predicted by the optimized cyclic depth neural network model is integrated to obtain a predicted no-rain video corresponding to the rain-containing video to be processed.

it will be appreciated that after the predicted rain-free video is formed, the predicted rain-free video is compared to the actual rain-free video to evaluate the accuracy of the method of rain removal for the video.

The method for removing rain from a video provided by this embodiment specifically includes, when building a cyclic depth neural network model for removing rain from a video: building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video; constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image; constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image; building a rain removing sub-network, and generating a predicted raindrop image of the current frame image by the rain removing sub-network according to the spatial domain characteristics of the current frame image; constructing a reconstruction sub-network, and acquiring a prediction detail characteristic diagram of the current frame image by the reconstruction sub-network according to the integrated time domain characteristic of the current frame image; the rain-removing reconstruction joint sub-network is built, the rain-removing reconstruction joint sub-network generates a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image, background information between adjacent frames can be coded, more representational characteristics can be extracted, and redundant information on a time domain and a space domain in a rain-containing video can be integrated, so that a good rain-removing effect of the video can be achieved, and the restored video is more accurate and clear.

The method for removing rain from a video provided by this embodiment trains a cycle depth neural network model to obtain an optimized cycle depth neural network model, and specifically includes: adding the mixed raindrop model into each rainless video of the training set, wherein each video added with the mixed raindrop model is each training sample in the training set; and inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model. Compared with an additive raindrop model, the hybrid raindrop model can better describe the complex image degradation condition in a real scene, so that the optimized cycle depth neural network model is more suitable for the rain removal of a rain-containing video in the real scene. And when the circulation depth neural network model is optimized, a combined loss function is added for constraint, so that the training time of the circulation depth neural network model is effectively shortened, the trained circulation depth neural network model is optimized, and the recovered video is accurate and clear.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The foregoing program may be stored in a readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Fig. 3 is a schematic structural diagram of a first video rain removing device according to an embodiment of the present invention, and as shown in fig. 3, the video rain removing device provided in this embodiment includes: a building module 31, a training module 32 and a rain removing module 33.

The building module 31 is configured to build a cyclic depth neural network model for video rain removal. And the training module 32 is configured to train the cycle deep neural network model to obtain an optimized cycle deep neural network model. And a rain removing module 33, configured to perform a rain removing operation on the rain-containing video to be processed by using the optimized cyclic depth neural network model.

The video rain removing device provided in this embodiment may implement the technical solution of the method embodiment shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 4 is a schematic structural diagram of a second video rain removing device according to an embodiment of the present invention, and as shown in fig. 4, the video rain removing device provided in this embodiment further includes, on the basis of the first video rain removing device according to the embodiment of the present invention: the module 41 is integrated.

Further, the building module 31 is specifically configured to: building a convolutional neural network, wherein the convolutional neural network extracts the spatial domain characteristics of the current frame image of the rain-containing video; constructing a degradation judgment sub-network, wherein the degradation judgment sub-network obtains a predicted raindrop shielding area image of the current frame image according to the spatial domain feature of the current frame image and the integrated time domain feature of the previous frame image; constructing a fusion sub-network, wherein the fusion sub-network obtains the integration time domain characteristic of the current frame image according to the spatial domain characteristic of the current frame image, the integration time domain characteristic of the previous frame image and the constraint characteristic of the predicted raindrop occlusion region image of the current frame image; building a rain removing sub-network, and generating a predicted raindrop image of the current frame image by the rain removing sub-network according to the spatial domain characteristics of the current frame image; constructing a reconstruction sub-network, and acquiring a prediction detail characteristic diagram of the current frame image by the reconstruction sub-network according to the integrated time domain characteristic of the current frame image; building a rain removal reconstruction joint sub-network, and generating a prediction rain-free background image of the current frame image according to the prediction detail characteristic image of the current frame image by the rain removal reconstruction joint sub-network; the convolutional neural network, the degradation discrimination subnetwork, the fusion subnetwork, the rain removal subnetwork, the reconstruction subnetwork, and the rain removal reconstruction joint subnetwork are sequentially part of the cycle depth neural network model.

Further, the training module 32 is specifically configured to: adding the mixed raindrop model into each rainless video of the training set, wherein each video added with the mixed raindrop model is each training sample in the training set; and inputting each training sample into the cycle depth neural network model, constraining by a combined loss function, and training the cycle depth neural network model by using a gradient descent algorithm so as to adjust each parameter in the cycle depth neural network model and obtain the optimized cycle depth neural network model.

The mixed raindrop model is specifically represented by formula (5). The combination loss function is specifically shown in formula (7).

Further, the integrating module 41 is configured to integrate the predicted no-rain background image of each frame corresponding to the to-be-processed rain-containing video to form a predicted no-rain video.

the video rain removing device provided in this embodiment may implement the technical solution of the method embodiment shown in fig. 2, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 5 is a schematic structural diagram of a first video rain removing device according to an embodiment of the present invention, and as shown in fig. 5, a video rain removing device according to an embodiment of the present invention further includes: a memory 51, a processor 52 and a computer program.

Wherein the computer program is stored in the memory 51 and configured to be executed by the processor 52 to implement the method of the first embodiment of the video de-raining method of the present invention or the method of the second embodiment of the video de-raining method of the present invention. The relevant description may be understood by referring to the relevant description and effect corresponding to the steps in fig. 1 to fig. 2, and redundant description is not repeated here.

In the present embodiment, the memory 51 and the processor 52 are connected by a bus 53.

Embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the method in the first embodiment of the video rain removing method or the second embodiment of the video rain removing method.

finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.