阅读说明：本技术 一种视频广告播出监测方法 (Video advertisement broadcasting monitoring method ) 是由 甘涛 陈俞舟 何艳敏 罗瑜 于 2019-11-18 设计创作，主要内容包括：本发明公开了一种视频广告播出监测方法,包括视频哈希库构建和广告检测两个阶段；本发明在视频哈希库构建中,对慢录视频中的每一帧,提取颜色和灰度的两级特征,其中颜色为粗粒度特征,灰度为细粒度特征,在此基础上进行两级位数适中的局部敏感哈希映射；在广告检测中,采用两级哈希匹配方法,并通过直线拟合和直线匹配分别对广告播放的开始时间和结束时间进行检测,具有处理速度快、检测精度高的特点；同时,通过采取特征区域选择、判定重复帧、取头片段、取尾片段等操作,在保证精度的情况下,进一步提高了处理速度。(The invention discloses a video advertisement broadcasting monitoring method, which comprises two stages of video hash library construction and advertisement detection; in the construction of a video hash library, two-stage characteristics of color and gray level are extracted for each frame in a slow-recording video, wherein the color is a coarse-grained characteristic, and the gray level is a fine-grained characteristic, and on the basis, local sensitive hash mapping with moderate two-stage digit is performed; in the advertisement detection, a two-stage Hash matching method is adopted, and the start time and the end time of the advertisement playing are respectively detected through straight line fitting and straight line matching, so that the method has the characteristics of high processing speed and high detection precision; meanwhile, by adopting the operations of selecting the characteristic region, judging the repeated frame, fetching the head segment, fetching the tail segment and the like, the processing speed is further improved under the condition of ensuring the precision.)

1. A video advertisement broadcasting monitoring method is characterized by comprising two stages of video hash library construction and advertisement detection;

the video hash library construction comprises the following steps:

A1. initialization: let N_LInitializing a video hash table to be empty for inputting the total frame number of the slow recording video, and numbering z for the frames of the slow recording video_L＝0；

A2. Taking a video frame: taking a frame of video from the slow-recording video, updating the frame number of the slow-recording frame to be z_L+1；

A3. Two-stage feature extraction: selecting a characteristic region, and extracting a first-stage characteristic vector H from the image of the characteristic region in RGB space and YUV space respectively_aAnd a second level feature vector H_b；

A4. And (3) judging repeated frames: if the second-level feature vectors of the current frame and the previous frame are the same, turning to step A2; otherwise go to step A5;

A5. and (4) two-stage Hash warehousing: for the extracted feature vector H_aAnd H_bPerforming two-stage local sensitive hash mapping, generating a hash table entry, and writing the frame number data of the slow recording frame into a video hash library;

A6. determining whether to take a complete video frame: if the current slow recording frame number z_LLess than the total number of video frames N_LGo to step a 2; otherwise, ending the video hash library construction process;

the advertisement detection comprises the following steps:

B1. taking a head fragment: initializing the binary set R to be null, setting the first detection seconds s, and extracting one frame as a wide frame from the 1 st frame in the first Q frames of the advertisement video at intervals of P framesThe total number of the obtained advertisement head test frames is recorded as N_AWherein Q is sxf_A，f_AFrame rate for advertisement video;

B2. two-stage hash matching: processing the advertisement head test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

B3. and (3) straight line fitting: each doublet (z) in R_A,z_L) Viewed as a point (x, y) on a plane, where z_AAnd z_LFinding in a plane at least through N, corresponding to x and y, respectively, of the abscissa_A2, marking the total number of the found straight lines meeting the condition as M, wherein the straight lines have the slope of 1 and the width of 3 pixels;

B4. judging whether the fitting is successful: if M is equal to 0, ending the advertisement detection process, outputting the advertisement which is not broadcasted, otherwise executing step B5;

B5. determining the head position: defining an advertisement playing start time position array Head _ t and a straight line intercept array b with the length of M +1, and calculating the mth playing start time position Head _ t [ M ] of the advertisement to be detected for each M satisfying that M is more than or equal to 1 and less than or equal to M:

Head_t[m]＝b[m]/f_L

wherein, b [ m ]]For the intercept of the m-th line found on the y-axis, f_LThe frame rate of the slow video recording is;

B6. taking a tail segment:

b6-1. initial definition: defining an advertisement playing ending time position array Tail _ t and a frame position array Tail _ f with the length of M +1, and recording the total number of the advertisements to be detected as

b6-2. taking tail and initializing: setting k as the block serial number of the advertisement tail detection, and initializing k to be 0; let t be the number of point pairs found, z_maxFor the maximum value of the ordinate in the point pair found, t is initialized to 0, z_maxSet the value of 0, and set the tuple set R toSetting to be empty;

b6-3, let z_sNumbering the starting frame of the tail segment to be fetched

B6-4, let z_eNumbering the end frame of the tail segment to be fetched

B6-5 at z th of advertisement video_sTo z_eIn the frame, from z_sExtracting one frame as an advertisement tail test frame every P frames at the beginning of the frame, and recording the total frame number of the obtained advertisement tail test frames as N'_A；

B7. Two-stage hash matching: processing the advertisement tail test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

B8. straight line matching: for each doublet (z) in the doublet set R_A,z_L) The method comprises the following steps: if z_L-z_A-b[m]If | is less than or equal to 2, where | is absolute value operation, then updating t to t +1, and calculating z_max＝max(z_max,z_L) Wherein, max (z)_max,z_L) Is to take z_maxAnd z_LThe larger of the two;

B9. and (3) judging whether matching is finished:

b9-1, if t is more than or equal to N'_APer 2, let the m-th element Tail _ f [ m ] of Tail _ f]＝z_maxGo to step B9-3, otherwise let Tail _ f [ m]＝b[m]+Q；

B9-2, updating k to k +1, and judging if

b9-3, updating M to be M +1, judging, if M is less than or equal to M, turning to the step B6-2, otherwise, turning to the step B10;

B10. determining the tail position: for each M which satisfies that M is more than or equal to 1 and less than or equal to M, calculating the mth playing end time position Tail _ t [ M ] of the advertisement to be detected:

Tail_t[m]＝Tail_f[m]/f_L

and judging the integrity of the mth broadcast of the advertisement to be detected: if it is

2. The method for monitoring the broadcast of video advertisements as recited in claim 1, wherein the step a3 of two-stage feature extraction comprises the following sub-steps:

a3-1. feature region selection:

a3-1-1, removing the boundary, namely setting the length and the width of the current video frame as H and W respectively, and selecting a region which takes the center point of the video frame as the center, has the length of α H and the width of α W as a characteristic region, wherein α is a preset scale factor;

a3-1-2. size scaling: scaling a feature region image to a length H₀Width of W₀A fixed size of (d);

a3-2, first-stage feature extraction:

a3-2-1. interval division: dividing the color value range of 0-255 into I different chromaticity intervals, wherein I is the number of preset chromaticity intervals;

a3-2-2. interval number statistics: let g¹、g²And g³R, G and B color channels for the characteristic region image, for the j-th channel g^j(j is 1,2 and 3), distributing each pixel to different chromaticity intervals according to the value of the pixel, and counting the number of pixel points falling into each interval I (I is more than or equal to 1 and less than or equal to I)

A3-2-3. feature formation: for the channel g^j(j-1, 2,3), and all of them

A3-3, second-stage feature extraction:

a3-3-1. spatial transformation: converting the feature region image processed in the step A3-1 from an RGB space to a YUV space;

a3-3-2. feature formation: scanning the Y component from left to right and from top to bottom, and splicing each scanning value in sequence to form a column vector to finally obtain the length H₀×W₀Second-level feature vector H of_b。

3. The video advertisement broadcasting monitoring method according to claim 2, wherein the step a5 two-stage hash warehousing specifically comprises the following sub-steps:

a5-1. two-level hash mapping: feature vector H is divided using a standard locality sensitive hashing algorithm_aAnd H_bRespectively mapped to a number of bits of K_aThe hash key value hash _ key1 and the number of bits K_bThe hash key value hash _ key2, wherein

a5-2, generating a primary table entry: searching the table entry with the key value of hash _ key1 in the hash table, and if the search is successful, finding the linked list L1 corresponding to the content of the table entry; otherwise, creating an Item1 with a key value of hash _ key1 and an empty linked list L1, and enabling the content of the Item1 to be the first address of the linked list L1;

a5-3, generating a secondary table entry: searching a list item with a key value of hash _ key2 in a linked list L1, and if the searching is successful, finding a linked list L2 corresponding to the content of the list item; otherwise, creating an Item2 with a key value of hash _ key2 and an empty linked list L2, and enabling the content of the Item2 to be the first address of the linked list L2;

a5-4, data warehousing: creating a content as current slow recording frame number z_LAnd adds it to the linked list L2.

4. The video advertisement playout monitoring method of claim 3, wherein said two-stage hash matching method specifically comprises the sub-steps of:

b2-1, taking a video frame from the advertisement test frame, setting a frame counter with an initial value of 0, and adding 1 to the frame counter when taking a video frame;

b2-2. feature region selection: according to step A3-1, each advertisement test frame is converted to a length of H₀Width of W₀The characteristic region image of (1);

b2-3, two-stage feature extraction: according to the steps A3-2 and A3-3, respectively extracting a first-level feature vector H_a' and second level feature vector H_b′；

B2-4. decision repeat frame: if the second-level feature vectors of the current frame and the previous frame are the same, turning to the step B2-1, otherwise, turning to the step B2-5;

b2-5. two-stage HashMap: according to the step A5-1, using a standard locality sensitive hashing algorithm, the feature vector H is divided into_a' and H_b' separately mapped to a number of bits of K_aThe hash key value hash _ key 1' and the number of bits is K_bHash _ key 2';

b2-6, first-stage hash matching: searching the table entry with the key value of hash _ key1 'in the hash table, if the search is successful, finding the linked list L1' corresponding to the table entry content, and going to step B2-7; otherwise go to step B2-1;

b2-7, second-stage hash matching:

b2-7-1, searching the list item with the key value of hash _ key2 ' in the linked list L1 ', if the searching is successful, finding the linked list L2 ' corresponding to the content of the list item, and going to the step B2-7-4, otherwise going to the step B2-7-2;

b2-7-2, calculating the Hamming distance between the key value of each table entry of the linked list L1 'and the hash _ key 2', and finding the minimum value D in all the calculated Hamming distances_min；

B2-7-3 if D_minGreater than a preset key value distance threshold D_thGo to step B2-1, else find D_minThe linked list L2' corresponding to the content of the table entry;

b2-7-4. for each entry in the linked list L2': take out its contents, i.e. slow-recording frame number z_LWill z_LFrame number z in original advertisement to be detected with current test frame_AForm a binary group (z)_A,z_L) Adding the binary group into a binary group set R;

b2-8, if the current count value of the frame counter is less than the total frame number of the advertisement test frames, go to step B2-1, otherwise, end the two-stage hash matching process.

5. The video advertisement playout monitoring method of claim 1, wherein the first detection seconds s has a value range of: s is more than or equal to 3 and less than or equal to 8.

6. The method of claim 2, wherein the scale factor α is selected from the range of 0.75- α -1.

7. The video advertisement broadcasting monitoring method according to claim 2, wherein the value range of the chromaticity interval number I is: i is more than or equal to 30 and less than or equal to 60.

8. A video advertisement playout monitoring method as claimed in claim 3, being characterized in thatCharacterized in that the bit number factor β_aAnd β_bThe value ranges of the compounds are respectively not less than 0.1 and not more than β_aNot more than 0.2 and not more than 0.001 not more than β_b≤0.003。

9. The video ad playout monitoring method of claim 4, wherein said key value distance threshold D_thThe value range is as follows: d is not less than 5_th≤15。

Technical Field

The invention belongs to the technical field of digital video processing, and particularly relates to a video advertisement broadcasting monitoring method.

Background

Currently, broadcasting has become the most popular mass publicity and entertainment media, and many businesses also broadcast their product advertisements by means of television broadcasting. In television advertising, both the radio and the merchant are very concerned about whether each advertisement that is requested to be broadcast is on-spot. In addition, most television advertisements have a short duration, and whether the advertisement is played completely is also an important issue. The traditional television advertisement monitoring is mainly completed by manual monitoring, namely, the playing start time and the playing end time of the advertisement to be played are observed manually, and whether the advertisement is played completely is judged. In current automated broadcast systems, the monitoring is accomplished by a computer executing advertisement monitoring software.

One implementation idea of the advertisement monitoring and broadcasting software is to take out the advertisements in the advertisement library one by one and search in a broadcasted video record (slow recording) file so as to determine the starting time and the ending time of the broadcasting. In order to detect whether an advertisement exists in a slow recording, the conventional method is to use the same algorithm to extract the characteristics of the slow recording and the advertisement to be detected, then to take out a characteristic segment with the same length as the advertisement from the slow recording starting point, compare the characteristic segment with the advertisement characteristic segment, if the comparison is unsuccessful, move the starting point backwards by one frame, and take out the next segment for comparison until the end of the slow recording file. One of the biggest defects of the method is that the comparison times are too many, so that the detection time is too long, and real-time processing cannot be realized; another approach is to use a hash retrieval technique to retrieve ads in the slow-record. Although this kind of method can solve the problem of detection speed, it still has a deficiency for the advertisement monitoring application.

The traditional hash retrieval technology mainly faces two difficulties: on one hand, the detection accuracy depends on the selection of the features, and the features with high accuracy are often high in calculation complexity. The advertisement monitoring and broadcasting has high requirements on detection speed and precision, so that a proper feature extraction and retrieval method needs to be found. On the other hand, the traditional Hash retrieval technology focuses on determining whether the target advertisement exists, but the detection of the broadcasting position of the target advertisement, especially the detection of the actual ending position under the condition that the advertisement is not completely broadcasted, is lack of an efficient solution; modern hash retrieval techniques often use deep networks to extract features and construct hash codes. Although the method improves the detection precision, the training time of the deep network is too long, so that the method is difficult to meet the service requirement that the radio station needs to count the actual broadcasting situation of the advertisement in time on the broadcasting day.

In a word, the problem of detection speed and detection precision is not well solved in the current broadcast monitoring related technology so as to meet the broadcast monitoring requirement of the radio station advertisement.

Disclosure of Invention

Aiming at the defects in the prior art, the invention provides a high-performance video advertisement broadcasting monitoring method, which adopts a new feature extraction and retrieval algorithm to improve the processing speed while ensuring the detection precision.

In order to achieve the purpose of the invention, the invention adopts the technical scheme that:

a video advertisement broadcasting monitoring method comprises two stages of video hash library construction and advertisement detection;

the video hash library construction comprises the following steps:

A1. initialization: let N_LInitializing a video hash table to be empty for inputting the total frame number of the slow recording video, and numbering z for the frames of the slow recording video_L＝0；

A2. Taking a video frame: taking a frame of video from the slow-recording video, updating the frame number of the slow-recording frame to be z_L+1；

A3. Two-stage feature extraction: selecting a characteristic region, and extracting a first-stage characteristic vector H from the image of the characteristic region in RGB space and YUV space respectively_aAnd a second level feature vector H_b；

A4. And (3) judging repeated frames: if the second-level feature vectors of the current frame and the previous frame are the same, turning to step A2; otherwise go to step A5;

A5. and (4) two-stage Hash warehousing: for the extracted feature vector H_aAnd H_bTo carry outTwo-stage local sensitive hash mapping is carried out, a hash table entry is generated, and the slow recording frame number data is written into a video hash library;

A6. determining whether to take a complete video frame: if the current slow recording frame number z_LLess than the total number of video frames N_LGo to step a 2; otherwise, ending the video hash library construction process;

the advertisement detection comprises the following steps:

B1. taking a head fragment: initializing a binary set R to be null, setting a first detection second number s, extracting one frame from the first frame 1 in the first Q frames of the advertisement video at intervals of P frames as an advertisement head test frame, and recording the total number of the obtained advertisement head test frames as N_AWherein Q is sxf_A，f_AFrame rate for advertisement video;

B2. two-stage hash matching: processing the advertisement head test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

B3. and (3) straight line fitting: each doublet (z) in R_A,z_L) Viewed as a point (x, y) on a plane, where z_AAnd z_LFinding in a plane at least through N, corresponding to x and y, respectively, of the abscissa_A2, marking the total number of the found straight lines meeting the condition as M, wherein the straight lines have the slope of 1 and the width of 3 pixels;

B4. judging whether the fitting is successful: if M is equal to 0, ending the advertisement detection process, outputting the advertisement which is not broadcasted, otherwise executing step B5;

B5. determining the head position: defining an advertisement playing start time position array Head _ t and a straight line intercept array b with the length of M +1, and calculating the mth playing start time position Head _ t [ M ] of the advertisement to be detected for each M satisfying that M is more than or equal to 1 and less than or equal to M:

Head_t[m]＝b[m]/f_L

wherein, b [ m ]]For the intercept of the m-th line found on the y-axis, f_LThe frame rate of the slow video recording is;

B6. taking a tail segment:

b6-1. initial definition: defining an advertisement playing ending time position array Tail _ t and a frame position array Tail _ f with the length of M +1, and recording the total number of the advertisements to be detected as

Setting m to be 1;

b6-2. taking tail and initializing: setting k as the block serial number of the advertisement tail detection, and initializing k to be 0; let t be the number of point pairs found, z_maxFor the maximum value of the ordinate in the point pair found, t is initialized to 0, z_maxWhen the value is equal to 0, setting the tuple set R to be null;

b6-3, let z_sNumbering the starting frame of the tail segment to be fetched

B6-4, let z_eNumbering the end frame of the tail segment to be fetched

B6-5 at z th of advertisement video_sTo z_eIn the frame, from z_sExtracting one frame as an advertisement tail test frame every P frames at the beginning of the frame, and recording the total frame number of the obtained advertisement tail test frames as N'_A；

B7. Two-stage hash matching: processing the advertisement tail test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

B8. straight line matching: for each doublet (z) in the doublet set R_A,z_L) The method comprises the following steps: if z_L-z_A-b[m]If | is less than or equal to 2, where | is absolute value operation, then updating t to t +1, and calculating z_max＝max(z_max,z_L) Wherein, max (z)_max,z_L) Is to take z_maxAnd z_LThe larger of the two;

B9. and (3) judging whether matching is finished:

b9-1, if t is more than or equal to N'_APer 2, let the m-th element Tail _ f [ m ] of Tail _ f]＝z_maxGo to step B9-3, otherwise let Tail _ f [ m]＝b[m]+Q；

B9-2, updating k to k +1, and judging if

Go to step B6-3, otherwise go to step B9-3;

b9-3, updating M to be M +1, judging, if M is less than or equal to M, turning to the step B6-2, otherwise, turning to the step B10;

B10. determining the tail position: for each M which satisfies that M is more than or equal to 1 and less than or equal to M, calculating the mth playing end time position Tail _ t [ M ] of the advertisement to be detected:

Tail_t[m]＝Tail_f[m]/f_L

and judging the integrity of the mth broadcast of the advertisement to be detected: if it is

The advertisement is broadcast completely, otherwise the advertisement is broadcast incompletely.

Further, the two-stage feature extraction in step a3 specifically includes the following sub-steps:

a3-1. feature region selection:

a3-1-1, removing the boundary, namely setting the length and the width of the current video frame as H and W respectively, and selecting a region which takes the center point of the video frame as the center, has the length of α H and the width of α W as a characteristic region, wherein α is a preset scale factor;

a3-1-2. size scaling: scaling a feature region image to a length H₀Width of W₀A fixed size of (d);

a3-2, first-stage feature extraction:

a3-2-1. interval division: dividing the color value range of 0-255 into I different chromaticity intervals, wherein I is the number of preset chromaticity intervals;

a3-2-2. interval number statistics: let g¹、g²And g³R, G and B color channels for the characteristic region image, for the j-th channel g^j(j is 1,2 and 3), distributing each pixel to different chromaticity intervals according to the value of the pixel, and counting the number of pixel points falling into each interval I (I is more than or equal to 1 and less than or equal to I)

A3-2-3. feature formation: for the channel g^j(j-1, 2,3), and all of them

Splicing to obtain the characteristic vector of the channel

And splicing the eigenvectors of each channel to finally obtain a first-stage eigenvector H_a＝(H¹,H²,H³)^T；

A3-3, second-stage feature extraction:

a3-3-1. spatial transformation: converting the feature region image processed in the step A3-1 from an RGB space to a YUV space;

a3-3-2. feature formation: scanning the Y component from left to right and from top to bottom, and splicing each scanning value in sequence to form a column vector to finally obtain the length H₀×W₀Second-level feature vector H of_b。

Further, the step a5 two-stage hash binning specifically includes the following sub-steps:

a5-1. two-level hash mapping: feature vector H is divided using a standard locality sensitive hashing algorithm_aAnd H_bRespectively mapped to a number of bits of K_aThe hash key value hash _ key1 and the number of bits K_bThe hash key value hash _ key2, wherein

β_aAnd β_bIs a preset number factor of bits and is,

is a rounding-down operation;

a5-2, generating a primary table entry: searching the table entry with the key value of hash _ key1 in the hash table, and if the search is successful, finding the linked list L1 corresponding to the content of the table entry; otherwise, creating an Item1 with a key value of hash _ key1 and an empty linked list L1, and enabling the content of the Item1 to be the first address of the linked list L1;

a5-3, generating a secondary table entry: searching a list item with a key value of hash _ key2 in a linked list L1, and if the searching is successful, finding a linked list L2 corresponding to the content of the list item; otherwise, creating an Item2 with a key value of hash _ key2 and an empty linked list L2, and enabling the content of the Item2 to be the first address of the linked list L2;

a5-4, data warehousing: creating a content as current slow recording frame number z_LAnd adds it to the linked list L2.

Further, the two-stage hash matching method specifically includes the following sub-steps:

b2-1, taking a video frame from the advertisement test frame, setting a frame counter with an initial value of 0, and adding 1 to the frame counter when taking a video frame;

b2-2. feature region selection: according to step A3-1, each advertisement test frame is converted to a length of H₀Width of W₀The characteristic region image of (1);

b2-3, two-stage feature extraction: according to the steps A3-2 and A3-3, respectively extracting a first-level feature vector H_a' and second level feature vector H_b′；

B2-4. decision repeat frame: if the second-level feature vectors of the current frame and the previous frame are the same, turning to the step B2-1, otherwise, turning to the step B2-5;

b2-5. two-stage HashMap: according to the step A5-1, using a standard locality sensitive hashing algorithm, the feature vector H is divided into_a' and H_b' separately mapped to a number of bits of K_aHash key value hash _ ke ofy 1' and a number of bits K_bHash _ key 2';

b2-6, first-stage hash matching: searching the table entry with the key value of hash _ key1 'in the hash table, if the search is successful, finding the linked list L1' corresponding to the table entry content, and going to step B2-7; otherwise go to step B2-1;

b2-7, second-stage hash matching:

b2-7-1, searching the list item with the key value of hash _ key2 ' in the linked list L1 ', if the searching is successful, finding the linked list L2 ' corresponding to the content of the list item, and going to the step B2-7-4, otherwise going to the step B2-7-2;

b2-7-2, calculating the Hamming distance between the key value of each table entry of the linked list L1 'and the hash _ key 2', and finding the minimum value D in all the calculated Hamming distances_min；

B2-7-3 if D_minGreater than a preset key value distance threshold D_thGo to step B2-1, else find D_minThe linked list L2' corresponding to the content of the table entry;

b2-7-4. for each entry in the linked list L2': take out its contents, i.e. slow-recording frame number z_LWill z_LFrame number z in original advertisement to be detected with current test frame_AForm a binary group (z)_A,z_L) Adding the binary group into a binary group set R;

b2-8, if the current count value of the frame counter is less than the total frame number of the advertisement test frames, turning to the step B2-1, otherwise ending the two-stage hash matching process.

Further, the value range of the first detection second number s is as follows: s is more than or equal to 3 and less than or equal to 8.

Further, the value range of the scale factor α is 0.75- α -1.

Further, the value range of the chromaticity interval number I is: i is more than or equal to 30 and less than or equal to 60.

Further, the bit number factor β_aAnd β_bThe value ranges are respectively equal to or less than 0.1 and equal to β_aNot more than 0.2 and not more than 0.001 not more than β_b≤0.003。

Further, the key value distance threshold D_thValue ofThe range is as follows: d is not less than 5_th≤15。

The invention has the beneficial effects that: in the construction of a video hash library, two-stage characteristics of color and gray level are extracted for each frame in a slow-recording video, wherein the color is a coarse-grained characteristic, and the gray level is a fine-grained characteristic, and on the basis, local sensitive hash mapping with moderate two-stage digit is performed; in the advertisement detection, a two-stage Hash matching method is adopted, and the start time and the end time of the advertisement playing are respectively detected through straight line fitting and straight line matching, so that the method has the characteristics of high processing speed and high detection precision; meanwhile, by adopting the operations of selecting the characteristic region, judging the repeated frame, fetching the head segment, fetching the tail segment and the like, the processing speed is further improved under the condition of ensuring the precision.

Drawings

Fig. 1 is a schematic flow chart of a video advertisement broadcasting monitoring method according to the present invention.

FIG. 2 is a diagram of 3 fitted straight lines found during the commercial detection phase according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.

The embodiment of the invention provides a video advertisement broadcasting monitoring method, which comprises two stages of video hash library construction and advertisement detection, wherein the advertisement detection stage detects the head position and the tail position of an advertisement in sequence.

In the embodiment, a slow recording video of 24 hours of a certain day of a radio station and an advertisement video with the duration of 15 seconds are used as test samples, and the video frame rate f_L＝f_AAnd (5) detecting the advertisements in the slow-recording video by adopting the monitoring method of the invention at 30 frames/second.

First, the video hash library construction steps in the present invention will be described in further detail.

The video hash library construction steps specifically include the following steps a1 to a 6:

A1. initialization: let N_LInitializing a video hash table to be empty for inputting the total frame number of the slow recording video, and numbering z for the frames of the slow recording video_L＝0；

In this embodiment, the total number of frames N of the currently input slow-recording video_L＝2592000。

A2. Taking a video frame: taking a frame of video from the slow-recording video, updating the frame number of the slow-recording frame to be z_L+1, i.e. z_L＝z_L+1；

A3. Two-stage feature extraction: selecting a characteristic region, and extracting a first-stage characteristic vector H from the image of the characteristic region in RGB space and YUV space respectively_aAnd a second level feature vector H_b；

In this embodiment, the two-stage feature extraction specifically includes the following sub-steps:

a3-1. feature region selection:

a3-1-1, removing the boundary, namely setting the length and the width of the current video frame as H and W respectively, and selecting a region which takes the center point of the video frame as the center, has the length of α H and the width of α W as a characteristic region, wherein α is a preset scale factor;

in the embodiment, the length H of the current video frame is 1280, the width W is 720, the scale factor α has a value range of 0.75 ≦ α ≦ 1, and here α has a value of 0.8.

A3-1-2. size scaling: scaling a feature region image to a length H₀Width of W₀A fixed size of (d);

in the present embodiment, the length H of the feature region image after scaling₀320, width W₀＝240。

A3-2, first-stage feature extraction:

a3-2-1. interval division: dividing the color value range of 0-255 into I different chromaticity intervals, wherein I is the number of preset chromaticity intervals;

in this embodiment, the value of the chromaticity interval number I is 30 or more and I or less 60 or less, where I is 50.

A3-2-2. interval number statistics: let g¹、g²And g³R, G and B color channels for the characteristic region image, for the j-th channel g^j(j is 1,2 and 3), distributing each pixel to different chromaticity intervals according to the value of the pixel, and counting the number of pixel points falling into each interval I (I is more than or equal to 1 and less than or equal to I)

A3-2-3. feature formation: for the channel g^j(j-1, 2,3), and all of them

Splicing to obtain the characteristic vector of the channel

And splicing the eigenvectors of each channel to finally obtain a first-stage eigenvector H_a＝(H¹,H²,H³)^T；

A3-3, second-stage feature extraction:

a3-3-1. spatial transformation: converting the feature region image processed in the step A3-1 from an RGB space to a YUV space;

a3-3-2. feature formation: scanning the Y component from left to right and from top to bottom, and splicing each scanning value in sequence to form a column vector, and finally obtaining a second-stage feature vector H with the length of 320 multiplied by 240_b。

A4. And (3) judging repeated frames: if the second-level feature vectors of the current frame and the previous frame are the same, turning to step A2; otherwise go to step A5;

A5. and (4) two-stage Hash warehousing: extracting feature vector H_aAnd H_bPerforming two-stage local sensitive hash mapping, generating a hash table entry, and writing the frame number data of the slow recording frame into a video hash library;

in this embodiment, the two-stage hash-binning specifically includes the following sub-steps:

a5-1. two-level hash mapping: feature vector H is divided using a standard locality sensitive hashing algorithm_aAnd H_bRespectively mapped to a number of bits K_aHash key value hash _ key1 and bit number K_bThe hash key value hash _ key2, wherein

β_aAnd β_bIs a preset number factor of bits and is,is a rounding-down operation;

in the present embodiment, the bit number factor β_aAnd β_bThe value ranges of the compounds are respectively not less than 0.1 and not more than β_a≤0.2，0.001≤β_bLess than or equal to 0.003, here β_aIs 0.16, β_bIs taken to be 0.00167, so

A5-2, generating a primary table entry: searching the table entry with the key value of hash _ key1 in the hash table, and if the search is successful, finding the linked list L1 corresponding to the content of the table entry; otherwise, creating an Item1 with a key value of hash _ key1 and an empty linked list L1, and enabling the content of the Item1 to be the first address of the linked list L1;

a5-3, generating a secondary table entry: searching a list item with a key value of hash _ key2 in a linked list L1, and if the searching is successful, finding a linked list L2 corresponding to the content of the list item; otherwise, creating an Item2 with a key value of hash _ key2 and an empty linked list L2, and enabling the content of the Item2 to be the first address of the linked list L2;

a5-4, data warehousing: creating a content as current slow recording frame number z_LAnd adds it to the linked list L2.

A6. Determining whether to take a complete video frame: if the current slow recording frame number z_LLess than the total number of video frames N_LWhen the result is 2592000, go to step a 2; otherwise, ending the video hash library construction process.

And the whole video hash library occupies 148MB of storage space.

The advertisement detection step of the present invention will be described in further detail below.

The advertisement detection step specifically includes the following steps B1 to B8:

B1. taking a head fragment: initializing a binary set R to be null, setting a first detection second number s, extracting one frame from the first frame 1 in the first Q frames of the advertisement video at intervals of P frames as an advertisement head test frame, and recording the total number of the obtained advertisement head test frames as N_AWherein Q is sxf_A，f_AFrame rate for advertisement video;

in this embodiment, the number of seconds of head detection s is in the range of 3 ≦ s ≦ 8, where s is 5, so Q ≦ sxf_AWhen the number of the first Q frames of the advertisement video is 150 and one frame is extracted from the 1 st frame in the first Q frames of the advertisement video as the advertisement head test frame in each interval P frame of 2 frames, the extraction is completed, and the total number of the obtained test frames is N_A＝Q/(P+1)＝150/(2+1)＝50；

B2. Two-stage hash matching: processing the advertisement head test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

in this embodiment, the two-stage hash matching method specifically includes the following sub-steps:

b2-1, taking a video frame from the advertisement test frame, setting a frame counter with an initial value of 0, and adding 1 to the frame counter when taking a video frame;

b2-2. feature region selection: step A3-1 of constructing a video Hash library, each advertisement test frame is converted into a length H₀Width of W₀The characteristic region image of (1);

in this example, after transformation, theLength H of feature region image₀320, width W₀＝240。

B2-3, two-stage feature extraction: respectively extracting a first-level feature vector H according to steps A3-2 and A3-3 of video hash library construction_a' and second level feature vector H_b′；

In the present embodiment, the extracted first-stage feature vector H_a' dimension 150, second-level feature vector H_bThe dimension of' is 76800 dimensions.

B2-4. decision repeat frame: if the second-level feature vectors of the current frame and the previous frame are the same, turning to the step B2-1, otherwise, turning to the step B2-5;

b2-5. two-stage HashMap: step A5-1 of constructing a video hash library, using a standard locality sensitive hashing algorithm, to convert the feature vector H into a vector_a' and H_b' separately mapped to a number of bits of K_aThe hash key value hash _ key 1' and the number of bits is K_bHash _ key 2';

in this embodiment, the number of bits of the hash key value hash _ key1 'is 24 bits, and the number of bits of the hash key value hash _ key 2' is 128 bits.

B2-6, first-stage hash matching: searching the table entry with the key value of hash _ key1 'in the hash table, if the search is successful, finding the linked list L1' corresponding to the table entry content, and going to step B2-7; otherwise go to step B2-1;

b2-7, second-stage hash matching:

b2-7-1, searching the list item with the key value of hash _ key2 ' in the linked list L1 ', if the searching is successful, finding the linked list L2 ' corresponding to the content of the list item, and going to the step B2-7-4, otherwise going to the step B2-7-2;

b2-7-2, calculating the Hamming distance between the key value of each table entry of the linked list L1 'and the hash _ key 2', and finding the minimum value D in all the calculated Hamming distances_min；

B2-7-3 if D_minGreater than a preset key value distance threshold D_thGo to step B2-1, else find D_minThe linked list L2' corresponding to the content of the table entry;

in this embodiment, the key value distance threshold D_thHas a value range of T not less than 5₀15 or less, where D_thIs 10; if D is_minIf the value is greater than the threshold value 10, the step B2-1 is carried out, otherwise D is found_minThe linked list L2' corresponding to the content of the table entry;

b2-7-4. for each entry in the linked list L2': take out its contents, i.e. slow-recording frame number z_LWill z_LFrame number z in original advertisement to be detected with current test frame_AForm a binary group (z)_A,z_L) Adding the binary group into a binary group set R;

b2-8, if the current count value of the frame counter is less than the total frame number 50 of the advertisement test frames, go to step B2-1, otherwise, end the two-stage hash matching process.

B3. And (3) straight line fitting: each doublet (z) in R_A,z_L) Viewed as a point (x, y) on a plane, where z_AAnd z_LFinding in a plane at least through N, corresponding to x and y, respectively, of the abscissa_A50/2-25 points, the slope is 1, the width is 3 pixels of straight lines, and the total number of the straight lines which meet the condition is found is recorded as M;

in the present embodiment, a total of 3 straight lines satisfying the condition are found;

B4. judging whether the fitting is successful: if M is equal to 0, ending the advertisement detection process, outputting the advertisement which is not broadcasted, otherwise executing step B5;

B5. determining the head position: defining an advertisement playing start time array Head _ t and a straight line intercept array b with the length of M +1 ═ 4, and calculating the mth playing start time position Head _ t [ M ] of the advertisement to be detected for each M meeting the condition that M is more than or equal to 1 and less than or equal to 3:

Head_t[m]＝b[m]/f_L

wherein, b [ m ]]For the intercept of the m-th line found on the y-axis, f_L30 is the frame rate of the slow recording video;

the 3 straight lines obtained at this point are shown in fig. 2, wherein the abscissa in the figure represents the frame number of the advertisement video, and the ordinate represents the detected frame number of the corresponding slow recording video; graph (a) corresponds to m being 1, resulting in b [1] ═ 1565321, Head _ t [1] ═ 52177.4, corresponding to a commercial presentation start time of 14:29: 37.4; graph (b) corresponds to m2, resulting in b [2] ═ 1925977, Head _ t [2] ═ 64199.2, corresponding to ad playout start time of 17:49: 59.2; graph (c) corresponds to m being 3, resulting in b [3] ═ 2279592, Head _ t [3] ═ 75986.4, and a corresponding commercial presentation start time of 21:06: 26.4.

B6. Taking a tail segment:

b6-1. initial definition: defining an advertisement playing ending frame position array Tail _ f and a time position array Tail _ t with the length of M + 1-4, and the total frame number of the advertisement to be detectedSetting m to be 1;

b6-2. taking tail and initializing: setting k as the block serial number of the advertisement tail detection, and initializing k to be 0; let t be the number of point pairs found, z_maxFor the maximum value of the ordinate in the point pair found, t is initialized to 0, z_maxWhen the value is equal to 0, setting the tuple set R to be null;

b6-3, let z_sNumbering the starting frame of the tail segment to be fetched

In the present embodiment, namely

When k is 0, since 450- (k +2) 150. gtoreq.0, z_s＝450-(k+1)150+1＝301；

B6-4, let z_eNumbering the end frame of the tail segment to be fetched

In this embodiment, when k is 0, since 450- (k +2)150 ≧ 0, z_e＝301+150-1＝450；

B6-5 at z th of advertisement video_sTo z_eIn the frameFrom z th_sExtracting one frame as an advertisement tail test frame every P frames at the beginning of the frame, and recording the total frame number of the obtained advertisement tail test frames as N'_A；

In the present embodiment, when k is 0, in the z-th of the advertisement video_s301 to z_e450 frames, from z_sStarting with 301 frames, extracting one frame as an advertisement tail test frame at an interval P of 2 frames to obtain the total frame number N 'of the advertisement tail test frame'_A＝50；

B7. Two-stage hash matching: processing the advertisement tail test frame according to a two-stage Hash matching method to obtain a group of frame numbers z of slow recording frames_LAnd test frame number z_AForming binary group pairs, and adding the group pairs into a binary group set R;

B8. straight line matching: for each doublet (z) in the doublet set R_A,z_L) The method comprises the following steps: if z_L-z_A-b[m]If | is less than or equal to 2, where | is absolute value operation, then t is updated to t +1, i.e. t ═ t +1, and z is calculated_max＝max(z_max,z_L) Wherein, max (z)_max,z_L) Is to take z_maxAnd z_LThe larger of the two;

B9. and (3) judging whether matching is finished:

b9-1, if t is more than or equal to N'_A(t is equal to or greater than 25), let the mth element Tail _ f [ m ] of Tail _ f]＝z_maxGo to step B9-3, otherwise let Tail _ f [ m]＝b[m]+Q；

When k is 0 and m is 1, because t is 48>25, let Tail _ f [1] be 1565772, and go to step B9-3;

when k is 0 and m is 2, because t is 48>25, let Tail _ f [2] be 1926428, and go to step B9-3;

when k is 0 and m is 3, because t is 48>25, let Tail _ f [3] be 2280043, and go to step B9-3;

b9-2, updating k to k +1, and judging if

Go to step B6-3, otherwise go to step B9-3;

b9-3, updating M to be M +1, judging, if M is less than or equal to M, namely M is less than or equal to 3, turning to the step B6-2, otherwise, turning to the step B10;

B10. determining the tail position: for each m which satisfies that m is more than or equal to 1 and less than or equal to 3, calculating the mth playing end time position Tail _ t [ m ] of the advertisement to be detected:

Tail_t[m]＝Tail_f[m]/f_L

namely, it is

Tail_t[m]＝Tail_f[m]/30

And judging the integrity of the mth broadcast of the advertisement to be detected: if it isThe m-th broadcast of the advertisement is complete, otherwise the broadcast of the advertisement is incomplete.

Tail _ t [1] when k is 0 and m is 1]＝Tail_f[1]1565772/30-52192.4, the corresponding end time of the advertisement broadcast is 14:29: 52.4; due to the fact that

If the broadcast time is less than 0.5, the 1 st broadcast of the advertisement is complete;

tail _ t [2] when k is 0 and m is 2]＝Tail_f[2]The broadcast end time of the corresponding advertisement is 17:50:14.3, wherein the broadcast end time is 1926428/30-64214.3; due to the fact that

If the broadcast time is less than 0.5, the 2 nd broadcast of the advertisement is complete;

tail _ t [3] when k is 0 and m is 3]＝Tail_f[3]The broadcast end time of the corresponding advertisement is 21:06:41.4, wherein the broadcast end time is 2280043/30-76001.4; due to the fact that

Less than 0.5, the 3 rd playout of the advertisement is complete.

The method of the invention is evaluated in terms of detection accuracy and speed.

1. Detection accuracy

(1) Broadcasted advertisement video

The detection results of the advertisement videos in the above embodiments are shown in table 1.

Table 1 advertisement broadcasting test results

In the table, "manual positioning" represents the start time and the end time of advertisement broadcasting recorded in a manual mode, and "detection position" represents the start time and the end time of advertisement broadcasting detected by the method of the invention, wherein the formats of the "start time" and the "end time" are both "hour: minute: second", wherein the hour and the minute are accurate to integers, and the second is accurate to one bit after the decimal point. It can be seen that 3 plays of the advertisement are detected in their entirety, and that the detected times all deviate from the artificially marked times by less than 1 second, with an average deviation of 0.52 seconds.

(2) Non-broadcast advertisement video

In addition, 50 advertisements which are not in slow recording are tested, the advertisement duration is 15 seconds, the frame rate is 30 frames/second, and the detection results are 'not broadcast', which is consistent with the actual situation.

2. Detecting speed

The simulation program of the method is written by Python, the test machine is an Intel (R) i5 processor, the main frequency is 2.4GHz, and the internal memory is 8 GB. The average time to perform the video hash library construction and commercial detection is shown in table 2.

Table 2 video hash library construction and execution time (unit: second) for advertisement detection

It can be seen that the commercial detection is fast, taking only 0.54 seconds on average to complete the detection in 24 hour slow recording for both commercials that are shown and commercials that are not shown. The video hash library is time-consuming to construct, but the operation is only required to be performed once, and once the video hash library is constructed, the video hash library can be used for detecting all advertisements.

It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Those skilled in the art can make various other specific changes and combinations based on the teachings of the present invention without departing from the spirit of the invention, and these changes and combinations are within the scope of the invention.