阅读说明：本技术 视频审核方法、视频审核装置和电子设备 (Video auditing method, video auditing device and electronic equipment ) 是由 孙斌 焦大原 刘亚萍 于 2020-04-22 设计创作，主要内容包括：本申请公开了一种视频审核方法、视频审核装置和电子设备,涉及视频审核技术领域。其中,视频审核方法包括：获取待审核视频的N个图像帧,所述N为大于1的整数；获取所述N个图像帧的帧间差异变化值,所述帧间差异变化值用于表征相邻两图像帧之间的差异度；选择目标帧间差异变化值对应的图像帧作为审核帧,所述目标帧间差异变化值为所述N个图像帧的帧间差异变化值中,不大于相邻的帧间差异变化值的帧间差异变化值。本申请通过获取待审核视频的若干个图像帧的帧间差异变化值,并以此作为选择审核帧的依据,能够保证所选择的审核帧具有较高的稳定性,从而能够提高视频审核的效果,解决了现有技术中存在的视频审核效果较差的问题。(The application discloses a video auditing method, a video auditing device and electronic equipment, and relates to the technical field of video auditing. The video auditing method comprises the following steps: acquiring N image frames of a video to be audited, wherein N is an integer greater than 1; acquiring interframe difference change values of the N image frames, wherein the interframe difference change values are used for representing the difference degree between two adjacent image frames; and selecting an image frame corresponding to a target interframe difference change value as an audit frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than an adjacent interframe difference change value in the interframe difference change values of the N image frames. According to the method and the device, the interframe difference change values of a plurality of image frames of the video to be audited are obtained and are used as the basis for selecting the audit frame, so that the selected audit frame can be ensured to have higher stability, the video audit effect can be improved, and the problem that the video audit effect is poor in the prior art is solved.)

1. A video auditing method, characterized in that the method comprises:

acquiring N image frames of a video to be audited, wherein N is an integer greater than 1;

acquiring interframe difference change values of the N image frames, wherein the interframe difference change values are used for representing the difference degree between two adjacent image frames;

and selecting an image frame corresponding to a target interframe difference change value as an audit frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than an adjacent interframe difference change value in the interframe difference change values of the N image frames.

2. The method according to claim 1, wherein the selecting the image frame corresponding to the target interframe difference change value as the audit frame comprises:

fitting the interframe difference change values of the N image frames to form an interframe difference change curve;

and taking the image frame corresponding to the minimum value point of the interframe difference change curve as an audit frame.

3. The method according to claim 2, wherein the using the image frame corresponding to the minimum value point of the frame-to-frame difference variation curve as an audit frame comprises:

and taking the image frame corresponding to the minimum value point of which the interframe difference change value is greater than or equal to the first threshold value in the interframe difference change curve as the audit frame.

4. The method according to any one of claims 1 to 3, further comprising:

if the interval between a first audit frame of the audit frame and a first image frame of the N image frames is greater than or equal to a second threshold value, adding a second image frame as the audit frame, wherein the second image frame is the image frame which is positioned between the first audit frame and the first image frame in the N image frames;

the first audit frame is a first audit frame of the audit frame, and the first image frame is a first image frame of the N image frames; or, the first audit frame is the last audit frame of the audit frames, and the first image frame is the last image frame of the N image frames.

5. The method according to any one of claims 1 to 3, wherein the method for calculating the interframe difference change value comprises:

acquiring gray channel information of two adjacent image frames;

and calculating the pixel difference of each pixel point between two adjacent image frames according to the gray channel information, and adding and averaging the calculated pixel differences to obtain the interframe difference change value.

6. The method according to any one of claims 1 to 3, wherein the acquiring N image frames of a video to be audited comprises:

acquiring a video to be audited;

searching whether a video with similarity meeting preset conditions with the video to be audited exists in a preset resource library, wherein the video stored in the preset resource library is a video associated with label information;

and if the preset resource library does not have a video with the similarity meeting the preset condition with the video to be audited, acquiring N image frames of the video to be audited.

7. The method according to any one of claims 1 to 3, further comprising:

auditing the audit frame to obtain an audit result of the audit frame;

and creating the label information of the audit frame according to the audit result of the audit frame.

8. The method of claim 7, further comprising:

and under the condition that the audit frame has a risk frame, displaying the risk frame in a preset display area, wherein the risk frame is associated with risk label information.

9. The method of claim 8, further comprising:

and if unfilled picture areas exist in the display area, selecting a risk-free frame from the audit frame, and displaying the risk-free frame in the unfilled picture areas.

10. The method of claim 8, further comprising:

auditing the audio of the video to be audited;

and displaying the risk audio segment in the display area under the condition that the risk audio segment exists in the audio, wherein the risk audio segment is associated with risk label information and time information.

11. A video review apparatus, comprising:

the first acquisition module is used for acquiring N image frames of a video to be audited, wherein N is an integer greater than 1;

the second obtaining module is used for obtaining interframe difference change values of the N image frames, and the interframe difference change values are used for representing the difference degree between two adjacent image frames;

and the selecting module is used for selecting the image frame corresponding to the target interframe difference change value as an auditing frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than the adjacent interframe difference change value in the interframe difference change values of the N image frames.

12. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

13. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 10.

Technical Field

The application relates to a data processing technology, in particular to the technical field of video auditing, and specifically relates to a video auditing method, a video auditing device and electronic equipment.

Background

With the development of internet technology, the number of videos on the internet is increasing day by day, and the efficiency of video review and the accuracy of video review become more and more important. In the video auditing process, auditing can be performed frame by frame, and auditing frames can also be extracted for auditing. In the prior art, the audit frame is generally extracted at time intervals, however, the stability of the extracted audit frame is difficult to guarantee, which results in poor video audit effect.

Disclosure of Invention

The application provides a video checking method, a video checking device and electronic equipment.

According to a first aspect, the present application provides a video review method, the method comprising:

acquiring N image frames of a video to be audited, wherein N is an integer greater than 1;

acquiring interframe difference change values of the N image frames, wherein the interframe difference change values are used for representing the difference degree between two adjacent image frames;

and selecting an image frame corresponding to a target interframe difference change value as an audit frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than an adjacent interframe difference change value in the interframe difference change values of the N image frames.

According to a second aspect, the present application provides a video auditing apparatus, comprising:

the first acquisition module is used for acquiring N image frames of a video to be audited, wherein N is an integer greater than 1;

the second obtaining module is used for obtaining interframe difference change values of the N image frames, and the interframe difference change values are used for representing the difference degree between two adjacent image frames;

and the selecting module is used for selecting the image frame corresponding to the target interframe difference change value as an auditing frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than the adjacent interframe difference change value in the interframe difference change values of the N image frames.

According to a third aspect, the present application provides an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods of the first aspect.

According to a fourth aspect, the present application provides a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of the first aspects.

According to the technology of the application, because the interframe difference change value can reflect the change degree of the video picture, the selected review frame can be ensured to have higher stability by acquiring the interframe difference change values of a plurality of image frames of the video to be reviewed and taking the interframe difference change values as the basis for selecting the review frame, so that the video review effect can be improved, and the problem of poor video review effect in the prior art is solved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

fig. 1 is a schematic flow chart of a video auditing method according to a first embodiment of the present application;

FIG. 2 is a diagram illustrating selection of a review frame based on a minimum value point of a frame-to-frame difference variation curve according to a first embodiment of the present application;

FIG. 3 is a diagram illustrating a threshold puncturing strategy according to a first embodiment of the present application;

FIG. 4 is a schematic diagram of a frame-filling strategy according to a first embodiment of the present application;

FIG. 5 is a schematic illustration of an audit result presentation according to a first embodiment of the present application;

FIG. 6 is a schematic view of a video review flow according to a first embodiment of the present application;

fig. 7 is a schematic structural diagram of a video auditing apparatus according to a second embodiment of the present application;

fig. 8 is a block diagram of an electronic device for implementing a video auditing method according to an embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

First embodiment

As shown in fig. 1, the present application provides a video auditing method, which includes the following steps:

step 101: and acquiring N image frames of the video to be audited.

The N is an integer greater than 1, and the N image frames may be all image frames of the video to be audited, or may be a part of image frames selected from all image frames of the video to be audited according to a certain step length or interval. If the video to be audited is long, in order to reduce the required calculation time, a part of image frames can be selected from all the image frames of the video to be audited as the N image frames according to a certain step length or interval. The image frames of the video to be audited can be decoded through the Ffmpeg to obtain the N image frames.

The video auditing is based on extracting auditing frames from the video to be audited, and the number of the auditing frames and the quality of the auditing frames are two important indexes for evaluating the frame extracting effect of the video. The number of the audit frames affects the efficiency of video audit, if the number of the audit frames is too large, too many repeat frames may exist, which will cause the waste of storage resources and computing power; in the prior art, generally, the quality of the audit frame is extracted at a time interval, which is difficult to guarantee, motion blur frames or too many repeat frames may be generated, and image frames of a key scene are easy to miss, so that risks are easy to miss, and the video audit effect is poor. Therefore, the robustness of the audit frame extraction method needs to be improved.

In view of the above technical problems, the present application provides a new method for extracting an audit frame to improve the stability of the audit frame and reduce the number of repeated frames.

Step 102: and acquiring the interframe difference change values of the N image frames.

The inter-frame difference change value is used for representing the difference degree between two adjacent image frames. Alternatively, the inter-frame difference variation value can reflect the variation degree of the video picture, in other words, the inter-frame difference variation value can reflect the stability of the video picture. The better the stability of the video picture, the smaller the change degree of the video picture and the smaller the change value of the difference between frames.

The N image frames correspond to N-1 inter-frame difference change values, e.g., the first inter-frame difference change value represents a degree of difference between the first image frame and the second image frame, the second inter-frame difference change value represents a degree of difference between the second image frame and the third image frame, … …, and so on, and the last inter-frame difference change value represents a degree of difference between the penultimate image frame and the last image frame. For example, if the duration of the video to be audited is 10 seconds, and the frame rate is 30 frames/second, the total frame amount of the video to be audited is 300 frames, and the video to be audited may include 299 interframe difference change values.

Step 103: and selecting an image frame corresponding to a target interframe difference change value as an audit frame, wherein the target interframe difference change value is an interframe difference change value which is not greater than an adjacent interframe difference change value in the interframe difference change values of the N image frames.

After the interframe difference change values of the N image frames are obtained, an auditing frame with good stability can be selected from the N image frames according to the interframe difference change values of the N image frames. Specifically, an interframe difference change value not greater than an adjacent interframe difference change value may be selected as a target interframe difference change value, and an image frame corresponding to the target interframe difference change value may be used as an audit frame. For example, if the 5 th interframe difference variation value is not greater than the 4 th interframe difference variation value and is not greater than the 6 th interframe difference variation value, the 5 th interframe difference variation value is the target interframe difference variation value. Since the 5 th interframe difference change value represents the difference degree between the 5 th image frame and the 6 th image frame, the 5 th image frame can be used as an audit frame, and the 6 th image frame can also be used as an audit frame.

For most videos, each image frame presents a dynamic picture in turn, so the variation value of the difference between frames generally presents a variation trend of fluctuating up and down, and the variation value of the difference between frames is rarely a constant value. Therefore, in general, the target inter-frame difference variation value is smaller than the adjacent inter-frame difference variation value.

For a video with only small local change of the picture, the inter-frame difference change values are all smaller, so that for a video with only small local change of the picture, the image frame corresponding to the target inter-frame difference change value may have more repeated frames.

In view of this, in this embodiment, the first threshold may be set, and only the image frame corresponding to the target inter-frame difference change value greater than or equal to the first threshold may be used as the audit frame. If the target interframe difference change value is smaller than the first threshold value, the image frame corresponding to the target interframe difference change value is likely to be a repeated frame and can be ignored. Therefore, the number of the review frames can be effectively controlled, and more repeated frames are prevented from being extracted, so that the video review efficiency can be further improved.

Because the target interframe difference change value is not greater than the adjacent interframe difference change value, the stability of the image frame corresponding to the target interframe difference change value is higher than that of the adjacent image frame to a great extent, so that the stability of the audit frame can be ensured, and the fuzzy frame can be prevented from being extracted. And a plurality of inter-frame difference change values generally exist among the target inter-frame difference change values, so that the extracted review frames are generally not repeated, more repeated frames can be avoided being extracted, and the number of the review frames can be effectively controlled. In addition, since the image frames with good stability can be basically acquired, the whole video to be audited can be completely covered, and the image frames of the key scene are not easy to miss.

Therefore, according to the technology of the application, because the interframe difference change value can reflect the change degree of the video image, the stability of the audit frame can be ensured by acquiring the interframe difference change values of a plurality of image frames of the video to be audited and taking the interframe difference change values as the basis for selecting the audit frame, the number of the audit frame can be effectively controlled, the image frames of the key scene are not easy to omit, the video audit effect can be improved, and the problem that the video audit effect is poor in the prior art is solved.

In the present application, the inter-frame difference change value can be measured from the dimension of the pixel difference, and a calculation method of the inter-frame difference change value is provided below.

Optionally, the method for calculating the interframe difference change value includes:

acquiring gray channel information of two adjacent image frames;

and calculating the pixel difference of each pixel point between two adjacent image frames according to the gray channel information, and adding and averaging the calculated pixel differences to obtain the interframe difference change value.

Wherein D is_avgI, j, t represent the horizontal coordinate, vertical coordinate and image frame number of image frame pixel,being the pixel value at a certain coordinate in the gray scale image,andrespectively, the height and width of the grayscale image.

The pixels of the image can be determined according to the RGB channel information of the image, and the pixel values determined according to the RGB channel information can be regarded as precise pixel values, or pixel absolute values, but the calculation amount involved is large. In the application, the interframe difference change values of the N image frames are used as a basis for selecting the audit frame, and the purpose of obtaining the interframe difference change values of the N image frames is to select the audit frame with better stability, so that the interframe difference change values in the application are not required to be absolute numerical values.

In view of this, in this embodiment, the pixel difference of each pixel between two image frames can be calculated from the gradation channel information of the image frames, and the amount of calculation for calculating the pixel difference from the gradation channel information is small and occupies only 1/3 of the amount of calculation for calculating the pixel difference from the RGB channel information.

In this embodiment, the pixel differences obtained by calculation are added and averaged to obtain a floating point number, which is the inter-frame difference variation value.

Optionally, the selecting an image frame corresponding to the target inter-frame difference change value as an audit frame includes:

fitting the interframe difference change values of the N image frames to form an interframe difference change curve;

and taking the image frame corresponding to the minimum value point of the interframe difference change curve as an audit frame.

In the embodiment, the interframe difference change values of the N image frames are fitted to form an interframe difference change curve, so that the change intensity of the whole video image can be reflected more intuitively and clearly, and the minimum value point of the interframe difference change curve is the position of the target interframe difference change value. As can be seen, in this embodiment, by fitting the inter-frame difference change values of the N image frames to form an inter-frame difference change curve, the position of the target inter-frame difference change value can be determined more directly from the inter-frame difference change curve.

For example, as shown in fig. 2, the maximum point in the inter-frame difference variation curve corresponds to the video frame with the largest variation, and the minimum point (the position of the circular point in fig. 2) corresponds to the most stable video frame.

In the method, when the interframe difference change values of the N image frames are fitted to form an interframe difference change curve, a filtering operator can be added to smooth the interframe difference change curve in consideration of the existence of noise. For example, a filtering algorithm such as a mean filtering algorithm or a hanning window can be used, the filtering effect depends on the size of the filtering window, the larger the filtering window is, the smoother the filtering result is, the smaller the number of the corresponding obtained image frames is, and the size of the filtering window can be dynamically adjusted according to the test result. After the filtering is finished, the image frame corresponding to the minimum value point of the interframe difference change curve obtained by filtering can be used as an audit frame.

Optionally, the taking the image frame corresponding to the minimum value point of the interframe difference change curve as an audit frame includes:

and taking the image frame corresponding to the minimum value point of which the interframe difference change value is greater than or equal to the first threshold value in the interframe difference change curve as the audit frame.

For a video with a small local change in picture, the inter-frame difference change values are all small, so that for a video with a small local change in picture, there may be more repeated frames in the image frame corresponding to the minimum value point.

In view of this, in this embodiment, the first threshold may be set, and only the image frame corresponding to the minimum value point in the inter-frame difference variation curve whose inter-frame difference variation value is greater than or equal to the first threshold may be used as the audit frame. If there is a minimum value point smaller than the first threshold in the inter-frame difference variation curve, the image frame corresponding to the minimum value point is likely to be a repeated frame and may be ignored.

In this embodiment, a strategy for reducing repeated frames by setting a first threshold to screen out minimum value points smaller than the first threshold, which may also be referred to as a threshold frame deletion strategy.

For the interframe difference variation curve, a frame deletion threshold line can be directly added to the curve, and the interframe difference variation value corresponding to the frame deletion threshold line is the first threshold.

As an example, as shown in fig. 3, a frame erasure threshold line (i.e., a dashed line in fig. 3) is added to the inter-frame difference variation curve, and assuming that the first threshold is 0.6, the inter-frame difference variation value corresponding to the frame erasure threshold line is 0.6. In this way, all five minimum value points below the frame deletion threshold line can be ignored, and all image frames corresponding to four minimum value points above the frame deletion threshold line can be used as audit frames.

For a video with only small local changes in picture, it is possible that all minimum value points in the inter-frame difference change curve are smaller than the first threshold, and for this case, the picture of the video is in a substantially static state, so any frame image of the video can be used as an audit frame.

Therefore, the embodiment can effectively control the number of the review frames, avoid extracting more repeated frames and further improve the video review efficiency.

Optionally, the method further includes:

if the interval between a first audit frame of the audit frame and a first image frame of the N image frames is greater than or equal to a second threshold value, adding a second image frame as the audit frame, wherein the second image frame is the image frame which is positioned between the first audit frame and the first image frame in the N image frames;

the first audit frame is a first audit frame of the audit frame, and the first image frame is a first image frame of the N image frames; or, the first audit frame is the last audit frame of the audit frames, and the first image frame is the last image frame of the N image frames.

In the initial segment or the final segment of the video, the variation curve of the inter-frame difference may be monotonously increased or monotonously decreased for a longer time, so that the key scenes in the initial segment or the final segment of the video are easier to miss.

In view of this, it can be considered to perform frame complementing in the initial segment or the final segment of the video, i.e. to extract the second image frame as the review frame in the initial segment or the final segment of the video. The second image frame may be an image frame located at a middle point of an initial segment or an end segment of the video, or an image frame with a fixed frame number.

As an example, as shown in fig. 4, the initial segment in the inter-frame difference variation curve appears monotonously increasing for a long time, and the final segment in the inter-frame difference variation curve appears monotonously decreasing for a long time. Therefore, the second image frame (the position of the square point in fig. 4) is extracted as the review frame again in both the initial segment and the final segment of the inter-frame difference change curve. Thus, the number of audit frames is a total of 6 frames.

In the embodiment, by selecting the audit frame in the initial section or the final section of the video, omission of key scenes in the initial section or the final section of the video can be effectively avoided.

Optionally, the acquiring N image frames of the video to be audited includes:

acquiring a video to be audited;

searching whether a video with similarity meeting preset conditions with the video to be audited exists in a preset resource library, wherein the video stored in the preset resource library is a video associated with label information;

and if the preset resource library does not have a video with the similarity meeting the preset condition with the video to be audited, acquiring N image frames of the video to be audited.

The video on the internet has high repetition degree, and the same video can be converted into a plurality of videos through conversion of resolution, picture proportion, format and the like, but the contents of the videos are approximately the same. If the repeated videos are all audited, machine resources and human resources are wasted, and auditing timeliness is affected.

In view of this, the method and the device for checking the videos can store the checked videos in the preset resource library, create the tag information for the checked videos, the tag information can be used for representing the checking result of the checked videos, the tag information of the checked videos can be stored in the tag library, and the storage address of the checked videos can be recorded in the tag information, so that the tag information is associated with the checked videos, and the videos can be conveniently checked subsequently.

Therefore, before the video to be audited is audited, the preset resource library can be searched for whether the audited video similar to the video to be audited exists or not, if the audited video similar to the video to be audited is searched, repeated auditing of the video to be audited is not needed, and therefore auditing time is saved and auditing efficiency is improved.

If the audited video similar to the video to be audited is not retrieved, the video to be audited can be audited by adopting the video auditing method.

The above-mentioned search process is a process of video similarity search, for example, the video similarity search can be performed by means of URL (Uniform Resource Locator) comparison, hash value calculation, or based on deep learning.

After the audit of the video to be audited is completed, tag information can be created for the video, the video is stored in a preset resource library, and the tag information of the video is stored in a tag library.

Note that the tag information may include tag information in a video dimension, or may include tag information in an image frame dimension. If the tag information includes tag information of image frame dimensions, the corresponding image frame may also be stored in a picture library, and the image frame may be associated with the tag information of the image frame, and the tag information of the image frame may also be stored in the tag library.

Optionally, the method further includes:

auditing the audit frame to obtain an audit result of the audit frame;

and creating the label information of the audit frame according to the audit result of the audit frame.

In the application, a machine audit policy algorithm can be used for auditing each audit frame, wherein the machine audit policy algorithm is a deep learning recognition model which is trained in advance according to various risk data, such as a multi-classification model, a target detection model, a character recognition model, a face recognition model, a similar retrieval model and the like.

When the audit frame is audited, each policy algorithm can be adopted to audit the audit frame in parallel, each policy algorithm can obtain the audit result of the audit frame, for example, the face recognition policy algorithm can perform face risk detection on the audit frame, the character recognition policy algorithm can perform character risk detection on the audit frame, and the like. The auditing results of auditing the auditing frames by each policy algorithm can be divided into two types of risk and risk-free, and the auditing frames without risk can be called risk-free frames, and the auditing frames with risk can be called risk frames. A risk policy hit number threshold or a risk policy hit frame number threshold can be set to judge whether the video is a risk video or not and whether further manual review is needed or not.

In this embodiment, the tag information of the audit frame may be stored in the tag library, and is associated with the audit frame, and the storage address of the audit frame may be recorded in the tag information of the audit frame. In the embodiment, the tag information of the audit frame is created according to the audit result of the audit frame, on one hand, the tag information is beneficial to providing audit reference for manual audit; on the other hand, the method is beneficial to providing reference for similar retrieval.

Further, the creating of the tag information of the audit frame according to the audit result of the audit frame includes:

and under the condition that the risk frame exists in the audit frame, creating risk label information of the risk frame according to the audit result of the risk frame.

In this embodiment, the risk tag information of the risk frame may be stored in the tag library and associated with the risk frame, and the storage address of the risk frame may be recorded in the risk tag information of the risk frame. In the embodiment, the risk label information is established on the risk frame in a targeted manner, so that the storage resource can be saved, and the auditing efficiency can be improved.

Optionally, the method further includes:

and creating the label information of the video to be audited according to the audit result of the audit frame.

In this embodiment, the tag information of the video to be audited may be stored in a tag library, and associated with the video to be audited. In the embodiment, the tag information of the video to be audited is created according to the audit result of the audit frame, so that on one hand, audit reference can be provided for manual audit; on the other hand, the method is beneficial to providing reference for similar retrieval.

In the application, after the machine audit is performed on the video to be audited, manual audit can be further performed. In order to improve the auditing efficiency of manual auditing, the following embodiments are also provided.

Optionally, the method further includes:

and under the condition that the audit frame has a risk frame, displaying the risk frame in a preset display area, wherein the risk frame is associated with risk label information.

In this embodiment, under the condition that the audit frame has the risk frame, the risk frame can be only displayed in the preset display area, and the risk frame can be associated with the risk label information for the audit staff to view. Further, the risk frame may be associated with time information indicating a temporal position of the risk frame in the video.

The preset display area can be understood as a manual review display area, and the risk label information can be independently displayed in the display area or can be triggered to be displayed in the display area by clicking and the like.

The risk frames may be presented in the display area in the form of a multi-grid (e.g., nine-grid, six-grid, etc.) photo wall, so that auditors can quickly audit the risk frames without having to click to view the frames one by one. Meanwhile, the risk tag information associated with the risk frame can also provide an audit reference for auditors.

Optionally, the method further includes:

and if unfilled picture areas exist in the display area, selecting a risk-free frame from the audit frame, and displaying the risk-free frame in the unfilled picture areas.

In this embodiment, the number of risk frames may not fill the picture area of the display area, for example, the risk frames are displayed in the display area in the form of a squared photo wall, and the number of risk frames is 5, so that there are 4 unfilled picture areas in the display area. In order to enable the auditor to obtain more related information of the video to be audited so as to improve the auditing effect of the video, 4 risk-free frames can be selected from the auditing frames and displayed in the unfilled picture area.

In addition, if the number R of the risk frames is larger than the number S of the picture areas which can be filled, the S frames can be hierarchically sampled from the risk frames, otherwise, all the risk frames are selected, and S-R frame non-risk frames are selected from the audit frames.

Optionally, the image frames displayed in the display area are sorted by time stamp. Specifically, the selected risk frames and non-risk frames may be sorted according to the frame time stamps and then uniformly filled into the display area. Therefore, the video display has better time continuity, and the video content can be quickly understood by the auditor, so that the audit effect can be improved.

In the application, besides the video picture is audited, the audio obtained by decoding ffmpeg can be audited by machine policy, for example, including no-sound identification, audio character identification, audio retrieval and the like. Besides displaying image frames, the display area can also display audio audit bars, and audio wall risk references are provided for auditors. The audio audit bar may provide complete audio or may provide partial audio segments, such as risk audio segments.

Optionally, the method further includes:

auditing the audio of the video to be audited;

and displaying the risk audio segment in the display area under the condition that the risk audio segment exists in the audio, wherein the risk audio segment carries risk label information and time information.

In the embodiment, the risk audio frequency section is displayed in the display area and carries the risk label information and the time information, so that the auditing reference can be more intuitively provided for the auditors, and the video auditing efficiency is improved.

As an example, as shown in fig. 5, the presentation area includes a squared photo wall area a and an audio wall area b. The photo wall area a is filled with nine display frames, i.e., display frame 1 to display frame 9. The audio wall area b is displayed with an audio audit bar c, and also displayed with risk label information (e.g., "vulgar" in fig. 5) and time information for the risk audio segment (e.g., 00:12 in fig. 5).

When a click operation for the presentation frame is received, the presentation frame may be popped up and enlarged, and auxiliary information such as Optical Character Recognition (OCR) information, timestamp information, risk tag information, and the like in the presentation frame may also be popped up.

When a click operation for the audio audit bar c is received, audio can be played. Specifically, the audio may be played from a time position corresponding to the click operation.

The display area also comprises an audit result area d and a video playing area e. The audit result area d can list risk label information after the video machine audit strategy audit, display a specific timestamp, and display a specific hit risk standard regulation when a click operation for the risk label information is received, wherein the specific hit risk standard regulation can be associated with a risk system knowledge base to meet the requirement of quick query on the risk standard during manual audit. The video playing area e can provide a double-speed playing function of the video, so that the auditors can click and watch the video when reviewing.

In the application, as shown in fig. 6, a video to be audited may be obtained from a service end through a server, an audit frame of the video to be audited is extracted and machine audit is performed, specifically, various machine audit strategies are audited based on an optimized audit frame, multi-risk multi-label fusion is performed on an audit result, and then display frame sampling (i.e., fusion sampling in fig. 6) is performed according to a requirement of a photo wall. After the server-side finishes checking the video to be checked, the checking result of the video can be sent to the client-side, and the client-side can display the checking result in the form of a photo wall and an audio wall. It should be noted that, after the end of the audit of the video to be audited by the server, the audit result of the video may also be displayed locally at the server.

It should be noted that, various optional embodiments in the video auditing method in the present application may be implemented in combination with each other, or may be implemented separately, and the present application is not limited thereto.

The above embodiments of the present application have at least the following advantages or benefits:

in the method and the device, the interframe difference change value can reflect the change degree of the video picture, so that the stability of the audit frame can be ensured by acquiring the interframe difference change value of a plurality of image frames of the video to be audited and taking the interframe difference change value as the basis for selecting the audit frame, the number of the audit frame can be effectively controlled, the image frames of the key scene are not easy to omit, the video audit effect can be improved, and the problem of poor video audit effect in the prior art is solved.

Second embodiment

As shown in fig. 7, the present application provides a video auditing apparatus 200, including:

a first obtaining module 201, configured to obtain N image frames of a video to be audited, where N is an integer greater than 1;

a second obtaining module 202, configured to obtain an interframe difference change value of the N image frames, where the interframe difference change value is used to represent a difference between two adjacent image frames;

the selecting module 203 is configured to select an image frame corresponding to a target inter-frame difference change value as an audit frame, where the target inter-frame difference change value is an inter-frame difference change value that is not greater than an adjacent inter-frame difference change value among the inter-frame difference change values of the N image frames.

Optionally, the selecting module 203 includes:

the fitting submodule is used for fitting the interframe difference change values of the N image frames to form an interframe difference change curve;

and the selection submodule is used for taking the image frame corresponding to the minimum value point of the interframe difference change curve as an audit frame.

Optionally, the selection submodule is specifically configured to:

and taking the image frame corresponding to the minimum value point of which the interframe difference change value is greater than or equal to the first threshold value in the interframe difference change curve as the audit frame.

Optionally, the video auditing apparatus 200 further includes:

an adding module, configured to add a second image frame as the review frame if an interval between a first review frame of the review frame and a first image frame of the N image frames is greater than or equal to a second threshold, where the second image frame is an image frame, of the N image frames, located between the first review frame and the first image frame;

the first audit frame is a first audit frame of the audit frame, and the first image frame is a first image frame of the N image frames; or, the first audit frame is the last audit frame of the audit frames, and the first image frame is the last image frame of the N image frames.

Optionally, the video auditing apparatus 200 further includes:

the third acquisition module is used for acquiring the gray channel information of two adjacent image frames;

and the calculation module is used for calculating the pixel difference of each pixel point between two adjacent image frames according to the gray channel information, and adding and averaging the pixel differences obtained by calculation to obtain the interframe difference change value.

Optionally, the first obtaining module 201 includes:

the first obtaining submodule is used for obtaining a video to be audited;

the retrieval submodule is used for retrieving whether a video with the similarity meeting the preset condition with the video to be audited exists in a preset resource library, and the video stored in the preset resource library is the video associated with the label information;

and the second obtaining submodule is used for obtaining N image frames of the video to be audited if the video with the similarity meeting the preset condition with the video to be audited does not exist in the preset resource library.

Optionally, the video auditing apparatus 200 further includes:

the first auditing module is used for auditing the auditing frame to obtain an auditing result of the auditing frame;

and the creating module is used for creating the label information of the audit frame according to the audit result of the audit frame.

Optionally, the video auditing apparatus 200 further includes:

and the first display module is used for displaying the risk frame in a preset display area under the condition that the audit frame has the risk frame, and the risk frame is associated with risk label information.

Optionally, the video auditing apparatus 200 further includes:

and the second display module is used for selecting a risk-free frame from the audit frame and displaying the risk-free frame in the unfilled picture area if the unfilled picture area exists in the display area.

Optionally, the video auditing apparatus 200 further includes:

the second auditing module is used for auditing the audio of the video to be audited;

and the third display module is used for displaying the risk audio segment in the display area under the condition that the audio has the risk audio segment, and the risk audio segment is associated with risk label information and time information.

The video auditing device 200 provided by the application can realize each process in the video auditing method embodiment, and can achieve the same beneficial effects, and for avoiding repetition, the details are not repeated here.

According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.

Fig. 8 is a block diagram of an electronic device according to a video auditing method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.

As shown in fig. 8, the electronic apparatus includes: one or more processors 601, memory 602, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 8 illustrates an example of a processor 601.

The memory 602 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by at least one processor to cause the at least one processor to perform a video review method provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the video review method provided herein.

The memory 602 is used as a non-transitory computer readable storage medium and can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules (for example, the first obtaining module 201, the second obtaining module 202, and the selecting module 203 shown in fig. 7) corresponding to the video auditing method in the embodiment of the present application. The processor 601 executes various functional applications and data processing of the problem analysis apparatus by running non-transitory software programs, instructions and modules stored in the memory 602, that is, implementing the video auditing method in the above method embodiments.

The memory 602 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device of the video review method, and the like. Further, the memory 602 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 602 optionally includes memory located remotely from the processor 601, and these remote memories may be connected to the electronic device of the video review method via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the video auditing method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected by a bus or other means, and fig. 8 illustrates the connection by a bus as an example.

The input device 603 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device of the video review method, such as an input device like a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. The output devices 604 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

According to the technical scheme of the embodiment of the application, because the interframe difference change value can reflect the change degree of the video image, the stability of the audit frame can be ensured by acquiring the interframe difference change values of a plurality of image frames of the video to be audited and taking the interframe difference change values as the basis for selecting the audit frame, the number of the audit frame can be effectively controlled, the image frames of the key scene are not easy to miss, the video audit effect can be improved, and the problem that the video audit effect is poor in the prior art is solved.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.