阅读说明：本技术 一种用于带有步进天车娃娃机的随动方式手势控制方法及系统 (Follow-up gesture control method and system for doll with stepping crown block ) 是由 杜国铭 于美国 冯大志 于 2020-04-21 设计创作，主要内容包括：本发明提出一种用于带有步进天车娃娃机的随动方式手势控制方法及系统,所述方法包括如下步骤：步骤1、利用摄像头采集用户手势操作图像信息,通过CNN卷积神经网络模型对采集到的用户手势操作图像信息进行手势识别,从而得到手势坐标；步骤2、将手势坐标转换成步进天车空间坐标,更新当前手势状态；步骤3、与娃娃机系统进行通信,获取娃娃机状态以及通知娃娃机当前手势状态,从而完成娃娃机的控制操作；所述方法降低了玩家游戏操作难度并增加了手势控制娃娃机产品多样性。(The invention provides a follow-up gesture control method and a follow-up gesture control system for a doll with a stepping crown block, wherein the method comprises the following steps of: step 1, acquiring user gesture operation image information by using a camera, and performing gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate; step 2, converting the gesture coordinates into stepping crown block space coordinates, and updating the current gesture state; step 3, communicating with the doll machine system to obtain the state of the doll machine and inform the doll machine of the current gesture state, thereby completing the control operation of the doll machine; the method reduces the difficulty of game operation of the player and increases the diversity of products of the gesture control doll machine.)

1. A follow-up gesture control method for a doll with a stepping crown block, the stepping crown block being capable of moving according to coordinate positions, characterized in that: the method comprises the following steps:

step 1, acquiring user gesture operation image information by using a camera, and performing gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

step 2, converting the gesture coordinates into stepping crown block space coordinates, and updating the current gesture state;

step 3, communicating with the doll machine system to obtain the state of the doll machine and inform the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

the step 2 specifically comprises the following steps:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4;

step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, if the gesture coordinates fall in the image operation area, executing the step 2.3, and if the gesture coordinates do not fall in the image operation area, executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained;

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

2. The method of claim 1, wherein: the effective gesture types include palm open, palm closed, and fist making gestures.

3. The method of claim 2, wherein: the step 1 specifically comprises the following steps:

step 1.1: acquiring an image by a monocular camera;

step 1.2: the CNN convolutional neural network identifies images;

step 1.3: the recognition result is that the palm is opened, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; the recognition result is that the palm is closed, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; if the recognition result is a fist-making gesture, the step 1.4 is carried out;

step 1.4: judging the effectiveness of the fist making gesture, and entering step 1.6 if the fist making gesture is inside the palm identified last time according to the current fist making gesture coordinate, or entering step 1.5 if the fist making gesture is not inside the palm;

step 1.5: if no gesture is detected, the gesture coordinates are invalid;

step 1.6: and recognizing the end of the task.

4. The method of claim 3, wherein: and the coordinate anti-shaking adopts a distance judgment method, compares the coordinate with the gesture coordinate of the previous frame, specifies a distance threshold, updates the gesture coordinate when the distance threshold is larger than the threshold, and uses the gesture coordinate of the previous frame when the distance threshold is smaller than the threshold.

5. The method of claim 1, wherein: the step 3 specifically comprises the following steps:

step 3.1: waiting for the doll to request connection;

step 3.2: the step 3.3 is executed after the connection is successful and the step 3.1 is executed after the connection is failed;

step 3.3: waiting for a game starting command requested by the doll machine;

step 3.4: step 3.5 is executed after receiving the doll machine game starting instruction, and step 3.3 is executed after not receiving the doll machine game starting instruction;

step 3.5: waiting for receiving a doll command, wherein the command is a gesture state request to execute the step 3.6, and the command is a game ending execution step 3.7;

step 3.6: the current gesture state is returned to the doll machine;

step 3.7: and finishing the task and performing receipt.

6. A follow-up gesture control system for a doll with a stepper crown block that is capable of moving in accordance with coordinate positions, characterized by: the system comprises a gesture recognition module, a coordinate conversion module and a communication module;

the gesture recognition module acquires user gesture operation image information by using a camera, and performs gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

the coordinate conversion module is used for converting the gesture coordinate into a space coordinate of the stepping crown block and updating the current gesture state;

the communication module is used for communicating with the doll machine system, acquiring the state of the doll machine and informing the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

the method comprises the following steps of converting the gesture coordinate into a stepping crown block space coordinate, and updating the current gesture state, wherein the method specifically comprises the following steps:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4;

step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, if the gesture coordinates fall in the image operation area, executing the step 2.3, and if the gesture coordinates do not fall in the image operation area, executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained;

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

Technical Field

The invention belongs to the technical field of gesture recognition control, and particularly relates to a follow-up gesture control method and system for a doll with a stepping crown block.

Background

Gesture control doll technique is various, most technique is developed based on degree of depth camera, this technique is more mature, nevertheless the cost is too high and leads to the marketing effect not good enough, part gesture control is based on monocular camera development, it is with low costs, detection effect is accurate, nevertheless the control mode of any kind of technique gesture control doll is controlled based on original mechanical rocker mode, the gesture simulates mechanical rocker in the space and accomplishes the overhead traveling crane action all around, this kind of operation mode part player needs the adaptation for the first time to get to the hand, when first experiencing, experience effect is not good enough.

The control mode of the existing doll gesture control module is a rocker type gesture control mode, and has the following defects: (1) the whole operation flow has certain complexity; (2) the central position of the space control area needs to be searched, namely the stop position, otherwise, the front, rear, left and right operation areas cannot be determined; (3) the crown block is moved at a short distance, a novice player is not easy to operate, for example, fine adjustment of the crown block can be completed only by 'gesture leftwards' and then 'gesture middle' switching, and certain requirements are imposed on the speed and the position of gesture translation.

Disclosure of Invention

The invention aims to solve the problems in the prior art and provides a follow-up gesture control method and a follow-up gesture control system for a doll with a stepping crown block.

The invention is realized by the following technical scheme, the invention provides a gesture control method in a follow-up mode for a doll with a stepping crown block, the stepping crown block can move according to a coordinate position, and the method comprises the following steps:

step 1, acquiring user gesture operation image information by using a camera, and performing gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

step 2, converting the gesture coordinates into stepping crown block space coordinates, and updating the current gesture state;

step 3, communicating with the doll machine system to obtain the state of the doll machine and inform the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

the step 2 specifically comprises the following steps:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4;

step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, if the gesture coordinates fall in the image operation area, executing the step 2.3, and if the gesture coordinates do not fall in the image operation area, executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained;

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

Further, the valid gesture types include palm open, palm closed, and fist making gestures.

Further, the step 1 specifically comprises:

step 1.1: acquiring an image by a monocular camera;

step 1.2: the CNN convolutional neural network identifies images;

step 1.3: the recognition result is that the palm is opened, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; the recognition result is that the palm is closed, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; if the recognition result is a fist-making gesture, the step 1.4 is carried out;

step 1.4: judging the effectiveness of the fist making gesture, and entering step 1.6 if the fist making gesture is inside the palm identified last time according to the current fist making gesture coordinate, or entering step 1.5 if the fist making gesture is not inside the palm;

step 1.5: if no gesture is detected, the gesture coordinates are invalid;

step 1.6: and recognizing the end of the task.

Further, the coordinate anti-shake method adopts a distance judgment method, compares the coordinate with the coordinate of the gesture of the previous frame, specifies a distance threshold, updates the gesture coordinate when the distance threshold is larger than the threshold, and uses the coordinate of the gesture of the previous frame when the distance threshold is smaller than the threshold.

Further, the step 3 specifically includes:

step 3.1: waiting for the doll to request connection;

step 3.2: the step 3.3 is executed after the connection is successful and the step 3.1 is executed after the connection is failed;

step 3.3: waiting for a game starting command requested by the doll machine;

step 3.4: step 3.5 is executed after receiving the doll machine game starting instruction, and step 3.3 is executed after not receiving the doll machine game starting instruction;

step 3.5: waiting for receiving the doll command, wherein the command is a gesture state request to execute the step 3.6, and the command is a game end to execute the step 3.7

Step 3.6: the current gesture state is returned to the doll machine;

step 3.7: and finishing the task and performing receipt.

The invention also provides a follow-up gesture control system for the doll with the stepping crown block, wherein the stepping crown block can move according to the coordinate position, and the system comprises a gesture recognition module, a coordinate conversion module and a communication module;

the gesture recognition module acquires user gesture operation image information by using a camera, and performs gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

the coordinate conversion module is used for converting the gesture coordinate into a space coordinate of the stepping crown block and updating the current gesture state;

the communication module is used for communicating with the doll machine system, acquiring the state of the doll machine and informing the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

the method comprises the following steps of converting the gesture coordinate into a stepping crown block space coordinate, and updating the current gesture state, wherein the method specifically comprises the following steps:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4;

step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, if the gesture coordinates fall in the image operation area, executing the step 2.3, and if the gesture coordinates do not fall in the image operation area, executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained;

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

The invention has the following beneficial effects:

1. customized development is carried out on the doll machine with the stepping crown block;

2. the game operation difficulty of the player is reduced;

3. the product diversity of the gesture control doll machine is increased.

Drawings

FIG. 1 is a block diagram of the overall components of a follow-up gesture control system for a doll with a stepper crown block according to the present disclosure;

FIG. 2 is a flow chart of gesture recognition;

FIG. 3 is a flow chart of coordinate transformation;

FIG. 4 is a schematic diagram of an image operation area;

FIG. 5 is a schematic diagram of relative coordinates of a gesture within an image manipulation area;

fig. 6 is a communication flow diagram;

fig. 7 is a simplified diagram of a gesture-controlled doll.

Detailed Description

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention provides a follow-up gesture control method for a doll with a stepping crown block, wherein the stepping crown block can move according to a coordinate position, and the method comprises the following steps:

step 1, acquiring user gesture operation image information by using a camera, and performing gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

with reference to fig. 2, the step 1 specifically includes:

step 1.1: acquiring an image by a monocular camera;

step 1.2: the CNN convolutional neural network identifies images;

step 1.3: the recognition result is that the palm is opened, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; the recognition result is that the palm is closed, the gesture coordinate is effective, the gesture coordinate is stored and the coordinate is processed for anti-shake, and the step 1.6 is carried out; if the recognition result is a fist-making gesture, the step 1.4 is carried out;

step 1.4: judging the effectiveness of the fist making gesture, and entering step 1.6 if the fist making gesture is inside the palm identified last time according to the current fist making gesture coordinate, or entering step 1.5 if the fist making gesture is not inside the palm;

step 1.5: if no gesture is detected, the gesture coordinates are invalid;

step 1.6: and recognizing the end of the task.

And the coordinate anti-shaking adopts a distance judgment method, compares the coordinate with the gesture coordinate of the previous frame, specifies a distance threshold, updates the gesture coordinate when the distance threshold is larger than the threshold, and uses the gesture coordinate of the previous frame when the distance threshold is smaller than the threshold.

Step 2, converting the gesture coordinates into stepping crown block space coordinates, and updating the current gesture state;

with reference to fig. 3, the step 2 specifically includes:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4; the effective gesture types include palm open, palm closed, and fist making gestures.

Step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, as shown in fig. 4, if the gesture coordinates fall in the image operation area, executing the step 2.3, otherwise executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained; as shown in fig. 5, the upper left corner of the image operation area is a starting point 0, a maximum value 100 on the horizontal axis, and a maximum value 100 on the vertical axis, and the calculation formula is as follows:

it is known that: image width W, image height H, image gesture coordinates px, py, invalid region distance d.

Solving the following steps: the relative coordinates x, y of the gesture within the image manipulation zone.

Solution: x ═ (px-d)/(W-2 d);

y＝(py–d)/(H–2d)；

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

Step 3, communicating with the doll machine system to obtain the state of the doll machine and inform the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

with reference to fig. 6, the step 3 specifically includes:

step 3.1: waiting for the doll to request connection;

step 3.2: the step 3.3 is executed after the connection is successful and the step 3.1 is executed after the connection is failed;

step 3.3: waiting for a game starting command requested by the doll machine;

step 3.4: step 3.5 is executed after receiving the doll machine game starting instruction, and step 3.3 is executed after not receiving the doll machine game starting instruction;

step 3.5: waiting for receiving the doll command, wherein the command is a gesture state request to execute the step 3.6, and the command is a game end to execute the step 3.7

Step 3.6: the current gesture state is returned to the doll machine;

step 3.7: and finishing the task and performing receipt.

With reference to fig. 1, the present invention further provides a follow-up gesture control system for a doll with a stepping crown block, wherein the stepping crown block can move according to a coordinate position, and the system comprises a gesture recognition module, a coordinate conversion module and a communication module;

the gesture recognition module acquires user gesture operation image information by using a camera, and performs gesture recognition on the acquired user gesture operation image information through a CNN convolutional neural network model to obtain a gesture coordinate;

the coordinate conversion module is used for converting the gesture coordinate into a space coordinate of the stepping crown block and updating the current gesture state;

the communication module is used for communicating with the doll machine system, acquiring the state of the doll machine and informing the doll machine of the current gesture state, thereby completing the control operation of the doll machine;

the method comprises the following steps of converting the gesture coordinate into a stepping crown block space coordinate, and updating the current gesture state, wherein the method specifically comprises the following steps:

step 2.1: judging whether the gesture type is valid, if so, performing the step 2.2, and if not, performing the step 2.4;

step 2.2: judging whether the gesture coordinates fall in an image operation area, defining a rectangular area formed after 4 edges of the image are inwards shifted for a certain distance as the image operation area, if the gesture coordinates fall in the image operation area, executing the step 2.3, and if the gesture coordinates do not fall in the image operation area, executing the step 2.4;

step 2.3: converting the image gesture coordinates into stepping crown block space coordinates: the gesture center is defined as an image gesture coordinate, and the relative position of the horizontal axis and the vertical axis of the gesture center in the image operation area is calculated, so that the relative coordinate of the gesture in the image operation area, namely the space coordinate of the stepping crown block, is obtained;

step 2.4: updating the current gesture state;

step 2.5: and (4) finishing.

The method and system of the present invention are described below by way of example.

The doll machine capable of being provided with the follow-up gesture control system requires that the crown block has a stepping function, the crown block can move according to relative coordinates, the main control logic is controlled by a doll machine host system, and fig. 7 is a schematic diagram of the follow-up gesture control doll machine.

And (3) implementing installation and debugging steps:

step 1: installing a camera component of a gesture control system, selecting a position right above the doll at the installation position, and irradiating downwards;

step 2: defining a gesture operation area, opening a camera by using software, adjusting the size of an image to 320X240, ensuring that the left end point and the right end point of a moving area of a crown block of the doll machine correspond to the left end point and the right end point of a gesture control area, A corresponds to C, B corresponds to D, acquiring an image by the camera to be slightly larger than the C point and the D point, and adjusting the height of the camera by observing the image to meet the requirement;

and step 3: installing a gesture coordinate conversion module into a control cabinet of a doll machine, wherein a serial port line of the gesture coordinate conversion module is connected with a serial port line of a main board of the doll machine;

and 4, step 4: and connecting the camera line to the gesture coordinate conversion module.

The following gesture control method and system for a doll with a stepping crown block provided by the invention are introduced in detail, and the principle and the implementation mode of the invention are explained by applying specific examples, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.