阅读说明：本技术 基于视频流的直播交互方法及其相应的装置 (Live broadcast interaction method based on video stream and corresponding device thereof ) 是由 于川 余蒙 吴晓东 吴昊 曹立永 于 2016-12-19 设计创作，主要内容包括：本分案申请提供一种基于视频流的直播交互方法及装置,所述方法包括：终端识别出所接收的各视频流的每个图像帧中的特征物；当连续多个图像帧包含预设的所述特征物的特征时,确定该些图像帧各自所含特征之间的特征变化信息；将所确定的特征变化信息与预设的动作事件参数模板进行匹配,当该特征变化信息与一个所述的参数模板相匹配时,产生该参数模板相对应的动作事件；当识别到产生某一特定动作事件,依据映射关系表确定与之对应的响应指令；以便终端执行该指令以向直播间的至少一个用户发送响应该事件的反馈信息；其中,所述终端预存有所述动作事件与其响应指令的映射关系。本申请可以增强用户与直播方的互动性,提升用户体验。(The divisional application provides a live broadcast interaction method and device based on video streaming, and the method comprises the following steps: the terminal identifies features in each image frame of each received video stream; when a plurality of continuous image frames contain the preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames; matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template; when a specific action event is identified to be generated, determining a corresponding response instruction according to the mapping relation table; so that the terminal executes the instruction to send feedback information in response to the event to at least one user of the live broadcast room; and the terminal prestores the mapping relation between the action event and the response instruction thereof. The method and the device can enhance the interactivity of the user and the live broadcast party and improve the user experience.)

1. A live broadcast interaction method based on video streaming is applied to game live broadcast, and the method runs on a terminal and comprises the following steps:

in an application scene of live game, determining a game role of a current live game event;

identifying the shape change of the device held by the game role and the position change of the arm or the body of the game role to determine the killing action, and recording the completion of one killing action when the action is completed;

comparing the description parameter items of the killing actions with a preset action event parameter template, and if the description parameter items of the killing actions are matched with the preset action event parameter template, generating corresponding action events according to the characteristic change information determined by the identified killing actions;

and when the action event reaches the preset generation times, determining a preset response instruction for the action event, and executing the response instruction to send feedback information responding to the action event to at least one user in the live broadcast room.

2. The live interactive method based on video streaming according to claim 1, wherein the determining of the response command preset for the action event when the action event reaches the predetermined generation times comprises:

when the game role is recognized to finish one killing, the preset response instruction is a praise and counts the killing action;

when recognizing that the terminal completes two killing actions, the preset response instruction is to send a gift, and the preset response instruction is to continue counting the killing actions so that the terminal can display the feedback information of the sent gift or the praise and the times of killing completion of the game role on the user interface when making corresponding reaction for the killing actions.

3. The live interaction method based on video streaming according to claim 1, wherein the executing of the response instruction to send feedback information responding to the action event to at least one user of the live broadcast room comprises:

a delivery instruction to deliver an electronic gift to an anchor user, the instruction resulting in generating notification information sent to at least the anchor user containing a delivery of the electronic gift; when the game role finishes continuous killing for multiple times, the terminal automatically identifies the dynamic change information of the game role and automatically sends an electronic gift such as an achievement display or a virtual article to the game role, and when the terminal executes the instruction, at least one user in a live broadcast room sends sending notification information containing the electronic gift;

or

Sending a sending instruction of a preset text to a main broadcasting user, wherein the instruction causes the generation of notification information at least sent to the main broadcasting user and containing the text; the preset text information can be used for recording the occurrence frequency of dynamic change information in a live video stream or identifying the attribute and the number of virtual articles dispatched for identifying certain dynamic change information;

or

Broadcasting a broadcasting instruction of a preset text to a live broadcast room, wherein the instruction causes a plurality of users in the live broadcast room to receive notification information containing the text; and the notification information fed back when the response instruction is executed is used for being sent to all live broadcast room users, so that all the users are ensured to receive the notification information.

4. The live broadcast interaction method based on video streaming as claimed in claim 3, wherein when the game character completes a plurality of consecutive clicks, the terminal displays the number of clicks of the game character on the user interface, and when the user misses the click scene of the game character due to temporary leaving or other things, the corresponding interaction is made through the notification information and the click result information displayed on the user interface.

5. The live interactive method based on video streaming according to claim 1, wherein the parameter description items of the killing action are: the coordinate change of the arms of the killer, the coordinate change of the body center point and the shape change of the sword;

the feature change parameter description items of the feature are as follows: the coordinate change of the feature a, the coordinate change of the feature B and the shape change of the feature B of the feature A;

the characteristic object A is a killer, the characteristic a is the arm of the killer, the characteristic B is the body central point of the killer, the characteristic object B is a sword, and the parameter description items are correspondingly as follows: the coordinate change of the arms of the killer, the coordinate change of the body center point and the shape change of the sword.

6. A live broadcast interaction method based on video streaming is applied to live broadcast and comprises the following steps:

the terminal identifies features in each image frame of each received video stream; when a plurality of continuous image frames contain the preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames; matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template;

when a specific action event is identified to be generated, determining a corresponding response instruction according to the mapping relation table; so that the terminal executes the instruction to send feedback information in response to the event to at least one user of the live broadcast room; the terminal prestores a mapping relation between the action event and a response instruction thereof so as to determine the response instruction according to the action event.

7. The live video streaming-based interaction method as claimed in claim 6, wherein the determining feature change information between features included in each of the plurality of image frames when the plurality of consecutive image frames include a predetermined feature of the feature object comprises:

when the fact that a plurality of continuous image frames contain preset reference features of the feature objects is identified, change information of the feature objects among different image frames is further identified, the change information of the feature objects is position change, shape change, position and shape change of the character characters, article characters or movable parts, which are represented by the plurality of continuous image frames of the video stream, and when the plurality of continuous image frames are played, specific features of the feature objects contained in the plurality of continuous image frames are subjected to position change or shape change, and the specific features are determined to be feature change information among the features contained in the plurality of image frames;

wherein, the feature is any one of a character, an article character and an activity part in the video stream suitable for being described by features, and the corresponding feature change information is the position and/or shape change information of the character, the article character or the activity part represented by a plurality of continuous image frames of the video stream.

8. The live interaction method based on video streaming according to claim 6, wherein the response instruction is in any form as follows:

a serving instruction for serving an electronic gift to a anchor user, the instruction resulting in generating notification information sent to at least the anchor user containing a serving of the electronic gift;

a sending instruction for sending a preset text to a anchor user, the instruction causing generation of notification information containing the text to be sent at least to the anchor user;

broadcast instructions for broadcasting a preset text to a live room, the instructions causing a plurality of users of the live room to receive notification information containing the text.

9. The live interaction method based on video streaming according to claim 6, further comprising:

and when the video stream is subjected to image recognition, the selected feature is subjected to dynamic information change recognition and recorded so that the terminal executes a corresponding response instruction according to the recognition result to feed back corresponding notification information to a user in a live broadcast room.

10. The live broadcast interaction method based on the video stream as claimed in claim 9, wherein the features listed in the feature list are displayed as thumbnails corresponding to the features in the video stream, and the feature list further includes reference features preset for the features so as to determine initial states of the features in video stream images to determine the feature change information.

11. A terminal, comprising:

the display is used for displaying a user interface and realizing man-machine interaction;

one or more processors;

a memory;

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors; the one or more programs are configured to drive the one or more processors, the processors being configured to perform the method of any of claims 1 to 10.

Technical Field

The invention relates to the technical field of internet, in particular to a live broadcast interaction method based on video streaming and a corresponding device thereof.

Background

Today, network technologies are becoming more sophisticated, and network interactive live broadcasting is becoming known. Live interactive network refers generally to live interactive content that has undergone the process from early text interaction (chat communication), to voice interaction, to video interaction. Because the simple definition of the interaction is interaction and mutual influence, the network interactive live broadcast is difficult to realize in the traditional broadcast and television live broadcast. The network live broadcast inherits and develops the characteristics of the internet because of being based on the network, and the biggest characteristic of the network live broadcast is the initiative increase of audiences/audiences and unprecedented interactivity in the live broadcast process.

In the prior art, the user watches the performance of the anchor and dispatches the gift according to the wonderful performance of the anchor, thereby realizing interactivity. For another example, when a user watches the performance of the game anchor, a certain game role wins or continuously clicks, and the like, at this time, the user correspondingly sends gifts or praises to realize interaction, the process needs to manually judge whether the anchor completes a corresponding action or whether the game role completes a certain specific action, manual operation is added, and the intelligence is low.

Therefore, in the prior art, the interaction between the user and the anchor of the anchor terminal is realized, the operation is complex, the intelligence is low, the interestingness is lacked, and the watching experience of the user is reduced.

Disclosure of Invention

In view of the foregoing problems, the present invention provides a live broadcast interaction method based on video stream and a corresponding apparatus thereof.

The invention also provides a mobile terminal for executing the live broadcast interaction method based on the video stream.

In order to solve the problems, the invention adopts the following technical scheme:

in a first aspect, the present invention provides a live broadcast interaction method based on video streaming, including the following steps:

carrying out image recognition on the received video stream of the anchor user, and acquiring feature change information of a feature object in the video stream so as to generate a corresponding action event;

determining a preset response instruction for the action event according to the action event;

and executing the response instruction to send feedback information responding to the action event to at least one user of the live broadcast room.

With reference to the first aspect, in a first implementation manner of the first aspect, the step of performing image recognition on the received video stream of the anchor user to obtain feature change information of a feature object in the video stream, so as to generate a corresponding action event includes the following specific steps:

identifying features in each image frame of the received video streams;

when a plurality of continuous image frames contain the preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames;

and matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template.

With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the feature variation information and the parameter template have the same parameter description items, and matching between the feature variation information and the parameter template is achieved by comparing the parameter description items corresponding to each other.

With reference to the first aspect, in a third implementation manner of the first aspect, the feature is any one of a character, an item character, and an activity part in the video stream that is suitable for being characterized, and the corresponding feature change information is position and/or shape change information that the character, the item character, or the activity part represents in a plurality of consecutive image frames of the video stream.

With reference to the first aspect, in a fourth implementation manner of the first aspect, when performing image recognition on the video stream image, a reference feature preset for a feature is called so as to determine an initial state of the feature in the video stream image, so as to determine the feature change information.

With reference to the first aspect, in a fifth implementation manner of the first aspect, the method further includes the following steps:

and receiving selection of one or more of the feature lists provided by the live broadcast by a user, determining the selected features as features for subsequent image recognition, and displaying the features listed in the feature list by using the thumbnail corresponding to the features in the video stream.

With reference to the first aspect, in a sixth implementation manner of the first aspect, the method further includes the following steps:

and receiving the configuration of a response instruction aiming at the preset action event, and performing associative storage on the action event and the corresponding response instruction so as to determine the response instruction according to the action event.

With reference to the first aspect, in a seventh implementation manner of the first aspect, in the step of determining, according to the action event, a response instruction preset for the action event, the response instruction is determined if and only if the action event reaches a predetermined generation number.

With reference to the first aspect, in an eighth implementation manner of the first aspect, the response instruction is any of:

a serving instruction for serving an electronic gift to a anchor user, the instruction resulting in generating notification information sent to at least the anchor user containing a serving of the electronic gift;

a sending instruction for sending a preset text to a anchor user, the instruction causing generation of notification information containing the text to be sent at least to the anchor user;

broadcast instructions for broadcasting a preset text to a live room, the instructions causing a plurality of users of the live room to receive notification information containing the text.

In a second aspect, the present invention provides a live broadcast interaction device based on video streaming, including:

an identification module: the video processing system is used for carrying out image recognition on the received video stream of the anchor user, and acquiring the characteristic change information of the characteristic object in the video stream so as to generate a corresponding action event;

a determination module: the response instruction is used for determining a response instruction preset for the action event according to the action event;

a sending module: and the response instruction is executed to send feedback information responding to the action event to at least one user of the live broadcast room.

With reference to the second aspect, in a first implementation manner of the second aspect, the identifying module specifically includes:

an identification unit: for identifying features in each image frame of the received video streams;

a determination unit: when a plurality of continuous image frames contain preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames;

a matching unit: the action event processing module is used for matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template.

With reference to the first implementation manner of the second aspect, in a second implementation manner of the second aspect, the feature variation information and the parameter template have the same parameter description items, and matching between the feature variation information and the parameter template is achieved by comparing the corresponding parameter description items.

With reference to the second aspect, in a third implementation manner of the second aspect, the feature is any one of a character, an item character, and an activity part in the video stream that is suitable for being characterized, and the corresponding feature change information is position and/or shape change information of the character, the item character, or the activity part represented in a plurality of consecutive image frames of the video stream.

With reference to the second aspect, in a fourth implementation manner of the second aspect, when performing image recognition on the video stream image, a reference feature preset for a feature is called so as to determine an initial state of the feature in the video stream image, so as to determine the feature change information.

With reference to the second aspect, in a fifth implementation manner of the second aspect, the method further includes the following modules:

selecting a module: the video stream processing device is used for receiving user selection of one or more of feature lists provided by a live broadcast room, determining the selected features as features for subsequent image recognition, and displaying the features listed in the feature lists by thumbnails corresponding to the features in the video stream.

With reference to the second aspect, in a sixth implementation manner of the second aspect, the method includes the following steps:

a configuration unit: the configuration module is used for receiving a response instruction aiming at the preset action event, and performing associative storage on the action event and the corresponding response instruction so as to determine the response instruction according to the action event.

With reference to the second aspect, in a seventh implementation manner of the second aspect, the determining module determines the response instruction if and only if the action event reaches a predetermined generation number.

With reference to the second aspect, in an eighth implementation manner of the second aspect, the response instruction is any of the following instructions:

a serving instruction for serving an electronic gift to a anchor user, the instruction resulting in generating notification information sent to at least the anchor user containing a serving of the electronic gift;

a sending instruction for sending a preset text to a anchor user, the instruction causing generation of notification information containing the text to be sent at least to the anchor user;

broadcast instructions for broadcasting a preset text to a live room, the instructions causing a plurality of users of the live room to receive notification information containing the text.

In a third aspect, the present invention provides a mobile terminal, including:

the touch-sensitive display is used for displaying a user interface and realizing man-machine interaction;

one or more processors;

a memory;

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors; the one or more programs may be configured to cause the one or more processors to construct modules for performing the methods described above.

Compared with the prior art, the technical scheme of the invention at least has the following advantages:

the invention provides a live broadcast interaction method based on video stream and a corresponding device thereof, which can automatically judge the state of a playing event in the current video stream by identifying the action information of a character role or an article role in the current video stream by utilizing an image identification technology, so that a terminal can make corresponding reaction according to the identified dynamic change information and the interaction with a user at a watching end, and the condition that each link needs to be confirmed by people is avoided.

Specifically, the method and the device perform image recognition on dynamic change information of a character or an article character or a movable part in the current video stream, generate a corresponding action event according to the dynamic change information, and determine a corresponding response instruction according to the action event to execute the instruction so as to send feedback information responding to the action event to at least one user in a live broadcast room.

The implementation of the invention can apply the image recognition technology to the live broadcast technology, automatically recognize the dynamic change of the characters or articles in the current video stream picture, know the progress or the link of the current video stream playing event through the image recognition technology and automatically send the corresponding feedback information, realize the automatic interaction of the live broadcast, and do not need to manually confirm each event link, thereby improving the freshness of the user, and the invention has simple operation and high flexibility.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flow chart of an embodiment of a live broadcast interaction method based on video streaming according to the present invention;

FIG. 2 is a flow chart of an embodiment of a live broadcast interaction method based on video streaming according to the present invention;

fig. 3 is a schematic structural diagram of a first embodiment of a live broadcast interaction device based on video streaming according to the present invention;

fig. 4 is a schematic structural diagram of a second embodiment of a live broadcast interaction device based on video streaming according to the present invention;

fig. 5 is a block diagram of a mobile terminal part according to the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.

In some of the flows described in the present specification and claims and in the above-described figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations being numbered, e.g., S11, S12, etc., merely to distinguish between various operations, and the order of the operations by themselves is not meant to imply any order of execution. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

It will be understood by those of ordinary skill in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The image classification display method is mainly suitable for terminals with communication functions, such as smart phone terminals or smart tablet terminals, is not limited to the type of an operating system, and can be operating systems such as Android, IOS, WP, Saiban and the like.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, in a live broadcast interaction method based on a video stream provided by the present invention, in a specific implementation manner, the method specifically includes the following steps:

and S11, performing image recognition on the received video stream of the anchor user, and acquiring the characteristic change information of the characteristic object in the video stream to generate a corresponding action event.

In the embodiment of the present invention, it is preferable that the following specific steps are performed to obtain feature change information of a feature object in the video stream, so as to generate a corresponding action event:

firstly, identifying features in each image frame of each received video stream;

the feature object is a reference object selected to extract dynamic change information of the feature object to determine a state of a play event in a current video stream, and may be any one of a character role, an article role, and an activity component in the video stream, for example, a game character role of a current live broadcast room anchor or a current live broadcast game event, or a current live broadcast event participation role.

Secondly, when a plurality of continuous image frames contain the preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames;

when the feature in the video stream is identified, the dynamic change process of the feature is specifically identified. When the plurality of continuous image frames are identified to contain the preset reference feature of the feature, change information of the feature between different image frames is further identified, wherein the change information of the feature is the change of the position, the change of the shape, the change of the position and the change of the shape of the character, the object character or the movable part represented by the plurality of continuous image frames of the video stream.

Specifically, when a plurality of consecutive image frames are played, a specific feature included in the image frames is changed in position or shape, and is determined as feature change information between features included in the image frames.

In one possible application scenario, for example, when a game event is currently live, when a killing action of a certain game character is recognized, the recognition process may specifically determine the completion of the killing action by recognizing a shape change of a sword or other device held by the game character and a position change of an arm or a body of the game character, and when the action is completed, record a characteristic change information of completing one killing action, that is, the game character.

And thirdly, matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template.

In the embodiment of the present invention, the generation of the corresponding action event from the determined feature change information is specifically realized by matching the description parameter items of each other. The characteristic change information and the parameter template have the same parameter description items, and matching between the characteristic change information and the parameter template is realized by comparing the corresponding parameter description items.

For example, in the application scenario of the live game, the parameter description items of the killing action may be: the coordinate change of the arms of the killer, the coordinate change of the body central point and the shape change of the sword, and the corresponding feature change parameter description items of the feature are as follows: in this embodiment, the feature a is a killer, the feature a is an arm, the feature B is a body center point, and the feature B is a sword, that is, the parameter description items correspond to: the coordinate change of the arms of the killer, the coordinate change of the body center point and the shape change of the sword. When this dynamically changing information of the feature is identified, it is determined that a killing action is completed for the feature.

And S12, determining a preset response instruction for the action event according to the action event.

In the embodiment of the invention, the terminal prestores the mapping relation between the action event and the response instruction thereof so as to determine the response instruction according to the action event. When a specific action event is identified to be generated, a corresponding response instruction is determined according to the mapping relation table, so that the terminal executes the instruction to send feedback information responding to the event to at least one user in the live broadcast room.

The mapping relation comprises a corresponding relation between the attribute of the action event and the response instruction, in one possible design, the mapping relation also comprises a corresponding relation between the sending times of the action event and the response instruction, the corresponding relation is used for representing that the response instruction is determined if and only if the action event reaches the preset generation times, and in one possible design, the mapping relation comprises the response instruction with different generation times of the specific action event corresponding to different times.

For example, in the application scenario of live game, when it is recognized that the game character M completes one click, the preset response command is one click and counts the click, and when it is recognized that the game character M completes two clicks, the preset response command is to dispatch one gift and continue counting the click so that the terminal responds to the click, and the feedback information of the dispatched gift or the click and the number of times of the click completed by the game character M are displayed on the user interface.

Preferably, when the configuration of the response command for the preset action event is received, the action event and the corresponding response command are stored in an associated manner, so as to refresh the mapping relationship in real time, so as to determine the response command according to the action event.

And S13, executing the response instruction to send feedback information responding to the action event to at least one user in the live broadcast room.

Preferably, the response instruction in the embodiment of the present invention is specifically in any form as follows:

a delivery instruction for delivering an electronic gift to a host user, the instruction causing generation of notification information sent to at least the host user containing the delivery of the electronic gift;

for example, in live game broadcasting, when a game role completes continuous killing, the terminal automatically identifies the dynamic change information of the game role and automatically sends an electronic gift such as an achievement display or a virtual article to the game role, when the terminal executes the instruction, at least one user in the live broadcasting room sends sending notification information containing the electronic gift, preferentially, the killing frequency of the game role is also displayed on a user interface, and when the user misses the killing scene of the game role because of temporary departure or other things, the user can also make corresponding interaction through the notification information and the killing achievement information displayed on the user interface, so that the interactivity between the user and the live broadcasting party is enhanced, and the freshness of the user is improved.

Secondly, a sending instruction used for sending a preset text to a main broadcasting user, wherein the instruction causes the generation of notification information at least sent to the main broadcasting user and containing the text;

the preset text information when the response instruction is executed may be information such as the number of occurrences of dynamic change information in a live video stream or the attribute and number of virtual items dispatched to identify a certain dynamic change information. The text information is used for informing the user of knowing the dynamic change process and the feedback information which is made by the terminal in response to the dynamic change process, so that the user does not need to manually confirm the progress of the current live event in real time and can guess the occurred event according to the notification information, the user can follow the progress of the live event in real time, and the user experience is improved.

And thirdly, broadcasting instructions for broadcasting preset texts to the live broadcast room, wherein the instructions cause a plurality of users in the live broadcast room to receive notification information containing the texts.

And the notification information fed back when the response instruction is executed is used for being sent to all live broadcast room users, so that all the users are ensured to receive the notification information.

It should be noted that the type of the response instruction according to the embodiment of the present invention may also be other instructions, and is not limited to the above three instructions, and is not limited herein.

Referring to fig. 2, in another embodiment of the present invention, a step S14 is further included to receive a user selection of one or more features in a feature list provided in a live broadcast, determine the selected features as features to be subsequently subjected to image recognition, and display the features listed in the feature list in a thumbnail corresponding to the features in the video stream.

In the embodiment of the present invention, the video stream may include a plurality of features, and when the plurality of features appear on the same screen at the same time, the user may select to pay attention to the dynamic change information of a certain feature, for example, in a live broadcast process of a certain event, the user only wants to pay attention to the performance of a competitor who the user wants to pay attention to, and at this time, the user may pay attention to one or more selected feature lists.

Specifically, the terminal provides a user interaction interface of the selected features so that a user can select one or more of the feature lists, the terminal determines the selected features as features for subsequent image recognition, and when the video stream is subjected to image recognition, dynamic information change recognition is performed on the selected features and recording is performed so that the terminal executes corresponding response instructions according to recognition results to feed back corresponding notification information to a user in a live broadcast room.

And displaying the features listed in the feature list by using the thumbnail corresponding to the feature in the video stream. The feature list further comprises reference features preset for the features so as to determine the initial state of the features in the video stream image and determine the feature change information.

For example, when the user selects one or more features as the features to be recognized, the terminal determines the initial state of the features to be recognized according to the feature list, and determines the subsequent dynamic change information on the basis of the initial state as the reference datum point for dynamic information change.

Referring to fig. 3, to further illustrate the live interaction method based on video stream according to the present invention, a live interaction apparatus based on video stream is provided, which includes, in one embodiment, an identifying module 11, a determining module 12, and a sending module 13, and further includes, in the embodiment shown in fig. 4, a selecting module 14. Wherein the content of the first and second substances,

the identification module 11: the method is used for carrying out image recognition on the received video stream of the anchor user and acquiring the characteristic change information of the characteristic object in the video stream so as to generate the corresponding action event.

In the embodiment of the present invention, it is preferable that the following specific units acquire feature change information of a feature object in the video stream, so as to generate a corresponding action event:

an identification unit: for identifying features in each image frame of the received video streams;

the feature object is a reference object selected to extract dynamic change information of the feature object to determine a state of a play event in a current video stream, and may be any one of a character role, an article role, and an activity component in the video stream, for example, a game character role of a current live broadcast room anchor or a current live broadcast game event, or a current live broadcast event participation role.

A determination unit: when a plurality of continuous image frames contain preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames;

when the feature in the video stream is identified, the dynamic change process of the feature is specifically identified. When the plurality of continuous image frames are identified to contain the preset reference feature of the feature, change information of the feature between different image frames is further identified, wherein the change information of the feature is the change of the position, the change of the shape, the change of the position and the change of the shape of the character, the object character or the movable part represented by the plurality of continuous image frames of the video stream.

Specifically, when a plurality of consecutive image frames are played, a specific feature included in the image frames is changed in position or shape, and is determined as feature change information between features included in the image frames.

In one possible application scenario, for example, when a game event is currently live, when a killing action of a certain game character is recognized, the recognition process may specifically determine the completion of the killing action by recognizing a shape change of a sword or other device held by the game character and a position change of an arm or a body of the game character, and when the action is completed, record a characteristic change information of completing one killing action, that is, the game character.

A matching unit: the action event processing module is used for matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template.

In the embodiment of the present invention, the generation of the corresponding action event from the determined feature change information is specifically realized by matching the description parameter items of each other. The characteristic change information and the parameter template have the same parameter description items, and matching between the characteristic change information and the parameter template is realized by comparing the corresponding parameter description items.

For example, in the application scenario of the last live game, the parameter description items of the killing action may be: the coordinate change of the arms of the killer, the coordinate change of the body central point and the shape change of the sword, and the corresponding feature change parameter description items of the feature are as follows: in this embodiment, the feature a is a killer, the feature a is an arm, the feature B is a body center point, and the feature B is a sword, that is, the parameter description items correspond to: the coordinate change of the arms of the killer, the coordinate change of the body center point and the shape change of the sword. When this dynamically changing information of the feature is identified, it is determined that a killing action is completed for the feature.

The determination module 12: and determining a preset response instruction for the action event according to the action event.

In the embodiment of the invention, the terminal prestores the mapping relation between the action event and the response instruction thereof so as to determine the response instruction according to the action event. When a specific action event is identified to be generated, a corresponding response instruction is determined according to the mapping relation table, so that the terminal executes the instruction to send feedback information responding to the event to at least one user in the live broadcast room.

The mapping relation comprises a corresponding relation between the attribute of the action event and the response instruction, in one possible design, the mapping relation also comprises a corresponding relation between the sending times of the action event and the response instruction, the corresponding relation is used for representing that the response instruction is determined if and only if the action event reaches the preset generation times, and in one possible design, the mapping relation comprises the response instruction with different generation times of the specific action event corresponding to different times.

For example, in the application scenario of live game, when it is recognized that the game character M completes one click, the preset response command is one click and counts the click, and when it is recognized that the game character M completes two clicks, the preset response command is to dispatch one gift and continue counting the click so that the terminal responds to the click, and the feedback information of the dispatched gift or the click and the number of times of the click completed by the game character M are displayed on the user interface.

Preferably, when the configuration unit receives the configuration of the response instruction for the preset action event, the action event and the corresponding response instruction are subjected to associative storage, so as to refresh the mapping relationship in real time, so that the response instruction is determined according to the action event.

The sending module 13: and the response instruction is executed to send feedback information responding to the action event to at least one user of the live broadcast room.

Preferably, the response instruction in the embodiment of the present invention is specifically in any form as follows:

a delivery instruction for delivering an electronic gift to a host user, the instruction causing generation of notification information sent to at least the host user containing the delivery of the electronic gift;

for example, in live game broadcasting, when a game role completes continuous killing, the terminal automatically identifies the dynamic change information of the game role and automatically sends an electronic gift such as an achievement display or a virtual article to the game role, when the terminal executes the instruction, at least one user in the live broadcasting room sends sending notification information containing the electronic gift, preferentially, the killing frequency of the game role is also displayed on a user interface, and when the user misses the killing scene of the game role because of temporary departure or other things, the user can also make corresponding interaction through the notification information and the killing achievement information displayed on the user interface, so that the interactivity between the user and the live broadcasting party is enhanced, and the freshness of the user is improved.

Secondly, a sending instruction used for sending a preset text to a main broadcasting user, wherein the instruction causes the generation of notification information at least sent to the main broadcasting user and containing the text;

the preset text information when the response instruction is executed may be information such as the number of occurrences of dynamic change information in a live video stream or the attribute and number of virtual items dispatched to identify a certain dynamic change information. The text information is used for informing the user of knowing the dynamic change process and the feedback information which is correspondingly made by the terminal in response to the dynamic change process, so that the user does not need to manually confirm the progress of the current live event in real time and can presume the occurred event and the like according to the notification information, the user can follow the progress of the live event in real time, and the user experience is improved.

And thirdly, broadcasting instructions for broadcasting preset texts to the live broadcast room, wherein the instructions cause a plurality of users in the live broadcast room to receive notification information containing the texts.

And the notification information fed back when the response instruction is executed is used for being sent to all live broadcast room users, so that all the users are ensured to receive the notification information.

It should be noted that the type of the response instruction according to the embodiment of the present invention may also be other instructions, and is not limited to the above three instructions, and is not limited herein.

Referring to fig. 4, in another embodiment of the present invention, a selecting module 14 is further included for receiving a user selection of one or more features in a feature list provided in the live broadcast, determining the selected features as features to be subsequently subjected to image recognition, and displaying the features listed in the feature list in a thumbnail corresponding to the features in the video stream.

In the embodiment of the present invention, the video stream may include a plurality of features, and when the plurality of features appear on the same screen at the same time, the user may select to pay attention to the dynamic change information of a certain feature, for example, in a live broadcast process of a certain event, the user only wants to pay attention to the performance of a competitor who the user wants to pay attention to, and at this time, the user may pay attention to one or more selected feature lists.

Specifically, the terminal provides a user interaction interface of the selected features so that a user can select one or more of the feature lists, the terminal determines the selected features as features for subsequent image recognition, and when the video stream is subjected to image recognition, dynamic information change recognition is performed on the selected features and recording is performed so that the terminal executes corresponding response instructions according to recognition results to feed back corresponding notification information to a user in a live broadcast room.

And displaying the features listed in the feature list by using the thumbnail corresponding to the feature in the video stream. The feature list further comprises reference features preset for the features so as to determine the initial state of the features in the video stream image and determine the feature change information.

For example, when the user selects one or more features as the features to be recognized, the terminal determines the initial state of the features to be recognized according to the feature list, and determines the subsequent dynamic change information on the basis of the initial state as the reference datum point for dynamic information change.

With reference to the foregoing embodiments, the greatest effect of the present invention is that the present invention performs image recognition on a received video stream of an anchor user to obtain dynamic change information of features in the video stream, such as characters, roles of articles, or moving parts, generates a corresponding action event when recognizing that the dynamic change information of a certain feature in a pre-stored feature list matches a parameter of a preset action event, and executes a preset response instruction corresponding to the action event according to the action event to send feedback information responding to the action event to at least one user in a live broadcast room.

Through the implementation of the invention, the terminal can automatically identify the change process or progress of the live broadcast event in the live broadcast room, and execute corresponding automatic interactive operation according to the identified change process of the event, thereby avoiding the need of user confirmation of the progress of each event, and enhancing the flexibility and the easy operability of the scheme. In addition, the user can interact with the anchor according to the notification information after the automatic interaction sent by the terminal, so that the interactivity between the user and the live broadcast party is enhanced, the freshness of the user is improved, and the user experience is improved.

An embodiment of the present invention further provides a mobile terminal, as shown in fig. 5, for convenience of description, only a portion related to the embodiment of the present invention is shown, and details of the specific technology are not disclosed. The terminal may be any terminal device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, etc., taking the terminal as the mobile phone as an example:

fig. 5 is a block diagram illustrating a partial structure of a mobile phone related to a terminal provided in an embodiment of the present invention. Referring to fig. 5, the handset includes: touch sensitive display 0513, processor 0511, memory 0514 and the like. Those skilled in the art will appreciate that the handset configuration shown in fig. 5 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The following describes each component of the mobile phone in detail with reference to fig. 5:

the memory 0514 may be used to store software programs and modules, and the processor 0511 executes various functional applications and data processing of the mobile phone by operating the software programs and modules stored in the memory 0514. The memory 0514 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory 0514 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The touch sensitive display 0513 may comprise two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor, and can receive and execute commands sent by the processor. In addition, touch sensitive displays may be implemented using various types of resistive, capacitive, infrared, and surface acoustic waves.

Touch sensitive Display 0513 may be used to Display information entered by or provided to a user, as well as various menus of the handset, such as information editing interfaces, etc. touch sensitive Display 0513 may include a Display panel, and optionally, may be configured in the form of a liquid Crystal Display (L acquired Crystal Display, L CD), Organic light Emitting Diode (O L ED), etc. further, when a touch operation is detected on or near touch sensitive Display 0513, a type of touch event is communicated to the processor to determine the type of touch event, and the processor then provides a corresponding visual output on the touch sensitive Display according to the type of touch event.

The handset may also include at least one sensor 0512, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel according to the brightness of ambient light, and a proximity sensor that turns off the display panel and/or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.

The processor 0511 is a control center of the mobile phone, connects each part of the whole mobile phone by various interfaces and lines, and executes various functions and processes data of the mobile phone by operating or executing software programs and/or modules stored in the memory 0514 and calling data stored in the memory 0514, thereby carrying out integral monitoring on the mobile phone. Optionally, processor 0511 may include one or more processing units; preferably, the processor 0511 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 0511.

The mobile phone further includes a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the processor 0511 through a power management system, so that functions of managing charging, discharging, power consumption management and the like are realized through the power management system.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which are not described herein.

In the embodiment of the present invention, the processor 0511 included in the terminal further has the following functions:

carrying out image recognition on the received video stream of the anchor user, and acquiring feature change information of a feature object in the video stream so as to generate a corresponding action event;

determining a preset response instruction for the action event according to the action event;

and executing the response instruction to send feedback information responding to the action event to at least one user of the live broadcast room.

The step of performing image recognition on the received video stream of the anchor user to acquire feature change information of a feature object in the video stream so as to generate a corresponding action event comprises the following specific steps:

identifying features in each image frame of the received video streams;

when a plurality of continuous image frames contain the preset characteristics of the characteristic objects, determining characteristic change information among the characteristics contained in the image frames;

and matching the determined characteristic change information with a preset action event parameter template, and generating an action event corresponding to the parameter template when the characteristic change information is matched with one parameter template.

The characteristic change information and the parameter template have the same parameter description items, and matching between the characteristic change information and the parameter template is realized by comparing the corresponding parameter description items.

The feature is any one of a character, an article character and an activity part in the video stream suitable for being described by features, and the corresponding feature change information is the position and/or shape change information of the character, the article character or the activity part represented by a plurality of continuous image frames of the video stream.

When the video stream image is subjected to image recognition, reference features preset for the feature are called so as to determine the initial state of the feature in the video stream image, and the feature change information is determined.

And receiving selection of one or more of the feature lists provided by the live broadcast by a user, determining the selected features as features for subsequent image recognition, and displaying the features listed in the feature list by using the thumbnail corresponding to the features in the video stream.

And receiving the configuration of a response instruction aiming at the preset action event, and performing associative storage on the action event and the corresponding response instruction so as to determine the response instruction according to the action event.

In the step of determining a response command preset for the action event according to the action event, the response command is determined if and only if the action event reaches a predetermined generation number.

The response instruction is any instruction as follows:

a serving instruction for serving an electronic gift to a anchor user, the instruction resulting in generating notification information sent to at least the anchor user containing a serving of the electronic gift;

a sending instruction for sending a preset text to a anchor user, the instruction causing generation of notification information containing the text to be sent at least to the anchor user;

broadcast instructions for broadcasting a preset text to a live room, the instructions causing a plurality of users of the live room to receive notification information containing the text.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing embodiments of the image classification display method, and are not described herein again.

In the embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method for displaying pictures by classification can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

It will be understood by those skilled in the art that all or part of the steps in the various image classification display methods of the foregoing embodiments may be implemented by a program to instruct associated hardware, where the program may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

It will be understood by those skilled in the art that all or part of the steps in the method for displaying classified pictures in the above embodiments may be implemented by hardware related to instructions of a program, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk.

While the mobile terminal provided by the present invention has been described in detail, for those skilled in the art, the idea of the embodiment of the present invention may be changed in the specific implementation and application scope, and in summary, the content of the present description should not be construed as limiting the present invention.