阅读说明：本技术 弹幕显示方法及装置 (Bullet screen display method and device ) 是由 王�琦 于 2020-06-12 设计创作，主要内容包括：本发明提供一种弹幕显示方法,包括：获取待处理的视频；依据视频中每一帧的图像特征计算视频图像的时域显著度和空域显著度；依据所述时域显著度和空域显著度生成二值化掩膜,所述二值化掩膜中包括无遮挡区域和遮挡区域；依据预设的弹幕的位置信息,判断所述弹幕在对应二值化掩膜中的无遮挡区域,将处于无遮挡区域的弹幕进行隐藏。本发明弹幕显示方法与现有的技术局限于以人为主的固定场景的方法相比,可应用于非固定场景,非固定目标类型的视频中,应用范围更广。(The invention provides a bullet screen display method, which comprises the following steps: acquiring a video to be processed; calculating the time domain significance and the space domain significance of the video image according to the image characteristics of each frame in the video; generating a binarization mask according to the time domain significance and the space domain significance, wherein the binarization mask comprises an unoccluded area and an occluded area; and judging the non-shielding area of the bullet screen in the corresponding binaryzation mask according to the preset position information of the bullet screen, and hiding the bullet screen in the non-shielding area. Compared with the prior art which is limited to the man-made fixed scene, the bullet screen display method can be applied to the non-fixed scene and the non-fixed target type video, and has wider application range.)

1. A bullet screen display method is characterized in that: the method comprises the following steps:

acquiring a video to be processed;

calculating the time domain significance and the space domain significance of the video image according to the image characteristics of each frame in the video;

generating a binarization mask according to the time domain significance and the space domain significance, wherein the binarization mask comprises an unoccluded area and an occluded area;

and judging the non-shielding area of the bullet screen in the corresponding binaryzation mask according to the preset position information of the bullet screen, and hiding the bullet screen in the non-shielding area.

2. The bullet screen display method of claim 1, wherein: the calculating the time domain saliency and the spatial saliency of the video image according to the image characteristics of each frame in the video comprises the following steps:

calculating the spatial domain significance of each frame of the video according to the brightness, the chrominance and the texture characteristics;

selecting two frames with a preset frame interval from a video to be processed, and extracting optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame in the two frames; and calculating the unit modulus value of the optical flow according to the optical flow feature, and taking the unit modulus value of the optical flow as the time domain saliency.

3. The bullet screen display method of claim 2, wherein: the calculating a unit modulus value of an optical flow as a time domain saliency, from optical flow characteristics of a foreground object of a subsequent frame with respect to a foreground object of a previous frame among two frames spaced by a predetermined frame, includes:

dividing each frame into a preset number of regions to be processed according to the scene type, calculating the distance between the regions to be processed of the same scene type in the two frames, and dividing the two frames into a set of frames of the same scene when the distance is within a preset threshold value;

selecting two frames of a frame set of the same scene at intervals of a preset frame, extracting the optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame, and calculating the unit modulus value of the optical flow as the time domain saliency.

4. The bullet screen display method of claim 3, wherein: the generating of the binarization mask according to the time domain significance and the space domain significance comprises the following steps:

weighting the spatial domain significance and the time domain significance to obtain a significance score;

sequencing the regions to be processed in the same frame according to the significance scores, and selecting a plurality of sequenced regions to be processed as regions of interest;

and generating a binary mask according to the region of interest.

5. The bullet screen display method of claim 3, wherein: the selecting two frames of the frame set of the same scene, which are separated by a predetermined frame, and the extracting of the optical flow characteristics of the foreground object of the next frame relative to the foreground object of the previous frame comprises:

and sending the images of the previous frame and the next frame into an optical flow extraction network to obtain the optical flow characteristics of the foreground object of the next frame relative to the foreground object of the previous frame.

6. The bullet screen display method of claim 2, wherein: the calculating the spatial saliency of each frame of the video according to the brightness, the chrominance and the texture characteristics comprises the following steps:

dividing a video image into a plurality of pixel blocks;

calculating the average value of the brightness of the current pixel block as a first average brightness, and calculating the average value of the brightness of the pixel blocks around the current pixel block as a second average brightness;

calculating the average value of the difference values of the first average brightness and the second average brightness as the brightness significance of the current pixel block;

calculating the average value of the chroma of the current pixel block as a first average chroma, and calculating the average value of the chroma of the pixel blocks around the current pixel block as a second average chroma;

calculating the average value of the difference values of the first average chroma and the second average chroma as the chroma significance of the current pixel block;

calculating the average value of the texture features of the current pixel block as a first average texture feature, and calculating the average value of the texture features of the pixel blocks around the current pixel block as a second average texture feature;

calculating the average value of the difference values of the first average textural feature and the second average textural feature as the textural feature significance of the current pixel block;

and performing weighted calculation on the brightness significance, the chroma significance and the texture feature significance to obtain the spatial domain significance of the current pixel block.

7. A bullet screen display device which characterized in that: the method comprises the following steps:

the video preprocessing module is used for acquiring a video to be processed and calculating the time domain significance and the space domain significance of a video image according to the image characteristics of each frame in the video;

the bullet screen processing module is used for generating a binarization mask according to the time domain significance and the space domain significance, and the binarization mask comprises an unoccluded area and an occluded area;

bullet screen display module for according to the positional information of predetermined bullet screen, judge bullet screen does not shelter from the region or shelter from the region in corresponding binary mask, will be in and do not shelter from regional bullet screen and hide, will be in and shelter from regional bullet screen and carry out normal demonstration.

8. The bullet screen display device of claim 7 wherein: the video preprocessing module comprises:

the spatial domain significance submodule is used for calculating the spatial domain significance of each frame of the video according to the brightness, the chroma and the texture characteristics;

the time domain saliency submodule selects two frames with preset frame intervals from a video to be processed and extracts the optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame in the two frames; and calculating the unit modulus value of the optical flow according to the optical flow feature, and taking the unit modulus value of the optical flow as the time domain saliency.

9. The bullet screen display device of claim 8 wherein: the temporal saliency sub-module comprises:

the scene segmentation unit is used for segmenting each frame into a preset number of regions to be processed according to the scene type, calculating the distance between the regions to be processed of the same scene type in the two frames, and dividing the two frames into a set of frames of the same scene when the distance is within a preset threshold value;

and the saliency calculation unit is used for selecting two frames which are separated by a preset frame in the frame set of the same scene, extracting the optical flow characteristics of the foreground object of the next frame relative to the foreground object of the previous frame, and calculating the unit modulus value of the optical flow as the time domain saliency.

Technical Field

The invention relates to the technical field of bullet screen display, in particular to a bullet screen display method and device.

Background

The current bullet screen display technology needs to determine a specific area in advance, shield the bullet screen in the specific area, and display the bullet screen outside the specific area. However, this processing manner fails if an area in the video image, which requires attention to be focused, appears outside a specific area. Based on the current bullet screen display technology, it is difficult to form effective processing for a variety of image contents that need to be processed.

Disclosure of Invention

The bullet screen display method and the bullet screen display device provided by the invention can effectively process bullet screens in non-fixed scenes.

In a first aspect, the present invention provides a bullet screen display method, including:

acquiring a video to be processed;

calculating the time domain significance and the space domain significance of the video image according to the image characteristics of each frame in the video;

generating a binarization mask according to the time domain significance and the space domain significance, wherein the binarization mask comprises an unoccluded area and an occluded area;

and judging the non-shielding area of the bullet screen in the corresponding binaryzation mask according to the preset position information of the bullet screen, and hiding the bullet screen in the non-shielding area.

Optionally, the calculating the temporal saliency and the spatial saliency of the video image according to the image feature of each frame in the video includes:

calculating the spatial domain significance of each frame of the video according to the brightness, the chrominance and the texture characteristics;

selecting two frames with a preset frame interval from a video to be processed, and extracting optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame in the two frames; and calculating the unit modulus value of the optical flow according to the optical flow feature, and taking the unit modulus value of the optical flow as the time domain saliency.

Optionally, the calculating a unit modulus value of the optical flow as the temporal saliency includes, from the optical flow characteristics of the foreground object of the next frame with respect to the foreground object of the previous frame among two frames spaced by a predetermined frame:

dividing each frame into a preset number of regions to be processed according to the scene type, calculating the distance between the regions to be processed of the same scene type in the two frames, and dividing the two frames into a set of frames of the same scene when the distance is within a preset threshold value;

selecting two frames of a frame set of the same scene at intervals of a preset frame, extracting the optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame, and calculating the unit modulus value of the optical flow as the time domain saliency.

Optionally, the generating a binarization mask according to the temporal saliency and the spatial saliency includes:

weighting the spatial domain significance and the time domain significance to obtain a significance score;

sequencing the regions to be processed in the same frame according to the significance scores, and selecting a plurality of sequenced regions to be processed as regions of interest;

and generating a binary mask according to the region of interest.

Optionally, the selecting two frames separated by a predetermined frame in the set of frames of the same scene, and the extracting optical flow features of foreground objects of a subsequent frame relative to foreground objects of a previous frame includes:

and sending the images of the previous frame and the next frame into an optical flow extraction network to obtain the optical flow characteristics of the foreground object of the next frame relative to the foreground object of the previous frame.

Optionally, the calculating the spatial saliency of each frame of the video according to the luminance, chrominance and texture features includes:

dividing a video image into a plurality of pixel blocks;

calculating the average value of the brightness of the current pixel block as a first average brightness, and calculating the average value of the brightness of the pixel blocks around the current pixel block as a second average brightness;

calculating the average value of the difference values of the first average brightness and the second average brightness as the brightness significance of the current pixel block;

calculating the average value of the chroma of the current pixel block as a first average chroma, and calculating the average value of the chroma of the pixel blocks around the current pixel block as a second average chroma;

calculating the average value of the difference values of the first average chroma and the second average chroma as the chroma significance of the current pixel block;

calculating the average value of the texture features of the current pixel block as a first average texture feature, and calculating the average value of the texture features of the pixel blocks around the current pixel block as a second average texture feature;

calculating the average value of the difference values of the first average textural feature and the second average textural feature as the textural feature significance of the current pixel block;

and performing weighted calculation on the brightness significance, the chroma significance and the texture feature significance to obtain the spatial domain significance of the current pixel block.

The bullet screen display method of the invention divides and divides the picture blocks according to the target detection and the scene segmentation, and respectively calculates the spatial domain significance of the foreground block and the time domain significance of the background block for weighting to obtain the region of interest of the audience. Through judging the position relation in bullet curtain and this region, change the display mode of bullet curtain, promote to watch and experience. The bullet screen display method does not need to train a specific type of detection model aiming at the video type, does not need to set the shielding-free content in advance according to the video content, and can be applied to non-fixed scenes and non-fixed target type videos compared with the prior art which is limited to the man-made fixed scene, so that the application range is wider.

In a second aspect, the present invention provides a bullet screen display device, comprising:

the video preprocessing module is used for acquiring a video to be processed and calculating the time domain significance and the space domain significance of a video image according to the image characteristics of each frame in the video;

the bullet screen processing module is used for generating a binarization mask according to the time domain significance and the space domain significance, and the binarization mask comprises an unoccluded area and an occluded area;

bullet screen display module for according to the positional information of predetermined bullet screen, judge bullet screen does not shelter from the region or shelter from the region in corresponding binary mask, will be in and do not shelter from regional bullet screen and hide, will be in and shelter from regional bullet screen and carry out normal demonstration.

Optionally, the video pre-processing module includes:

the spatial domain significance submodule is used for calculating the spatial domain significance of each frame of the video according to the brightness, the chroma and the texture characteristics;

the time domain saliency submodule selects two frames with preset frame intervals from a video to be processed and extracts the optical flow characteristics of a foreground object of a next frame relative to a foreground object of a previous frame in the two frames; and calculating the unit modulus value of the optical flow according to the optical flow feature, and taking the unit modulus value of the optical flow as the time domain saliency.

Optionally, the temporal saliency module comprises:

the scene segmentation unit is used for segmenting each frame into a preset number of regions to be processed according to the scene type, calculating the distance between the regions to be processed of the same scene type in the two frames, and dividing the two frames into a set of frames of the same scene when the distance is within a preset threshold value;

and the saliency calculation unit is used for selecting two frames which are separated by a preset frame in the frame set of the same scene, extracting the optical flow characteristics of the foreground object of the next frame relative to the foreground object of the previous frame, and calculating the unit modulus value of the optical flow as the time domain saliency.

The bullet screen display device divides picture blocks according to target detection and scene segmentation, calculates the spatial domain significance of a foreground block and the temporal domain significance of a background block respectively, and weights the spatial domain significance and the temporal significance to obtain the region of interest of audiences. Through judging the position relation in bullet curtain and this region, change the display mode of bullet curtain, promote to watch and experience. The bullet screen display device does not need to train a specific type of detection model aiming at the video type, does not need to set the shielding-free content in advance according to the video content, and can be applied to non-fixed scenes and non-fixed target type videos compared with the prior art which is limited to a man-made fixed scene, so that the bullet screen display device has a wider application range.

Drawings

FIG. 1 is a flowchart illustrating a bullet screen display method according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating a method for determining an area of interest in a bullet screen display method according to an embodiment of the present invention;

fig. 3 is a flowchart illustrating a method for changing a bullet screen display mode in a bullet screen display method according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.