Nail key point detection method and device, electronic equipment and storage medium
阅读说明：本技术 指甲关键点检测方法、装置、电子设备及存储介质 (Nail key point detection method and device, electronic equipment and storage medium ) 是由 张知行 于 2020-04-27 设计创作，主要内容包括：本公开关于一种指甲关键点检测方法、装置、电子设备及存储介质,其中方法通过获取具有指甲区域的待检测图像,并将待检测图像输入关键点检测模型中进行关键点提取,从而得到与指甲区域中指甲的方向对应的指甲关键点坐标,由于关键点检测模型是采用包括了角度损失项和距离损失项的预设损失函数进行训练后得到的,从而保证了模型输出结果的稳定性与准确性,即得到的指甲关键点坐标具有稳定的方向性,因此,避免了传统技术中因指甲关键点存在方向抖动,而导致难以完成素材贴合的问题。(The method obtains an image to be detected with a nail area, inputs the image to be detected into a key point detection model to extract key points, and obtains nail key point coordinates corresponding to the direction of a nail in the nail area.)
1. A nail key point detection method is characterized by comprising the following steps:
acquiring an image to be detected, wherein the image to be detected comprises a nail area;
inputting the image to be detected into a key point detection model to obtain a nail key point coordinate corresponding to the direction of the nail in the nail region, wherein the key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function comprises an angle loss term and a distance loss term.
2. The nail key point detection method according to claim 1, wherein the training method of the key point detection model includes:
acquiring a nail sample image, wherein the nail root key point coordinates and the nail tip key point coordinates are marked in the nail sample image;
extracting a first predicted nail key point coordinate corresponding to the marked nail tip key point coordinate and a second predicted nail key point coordinate corresponding to the marked nail root key point coordinate in the nail sample image by using the neural network;
determining a loss value according to the preset loss function, the marked nail tip key point coordinates and the nail root key point coordinates, and the first prediction nail key point coordinates and the second prediction nail key point coordinates;
and training the neural network according to the loss value to obtain the key point detection model.
3. The nail keypoint detection method of claim 2, wherein said determining a loss value from said preset loss function, said annotated nail tip keypoint coordinates and said nail root keypoint coordinates, and said first predicted nail keypoint coordinates and said second predicted nail keypoint coordinates, comprises:
according to the angle loss item, calculating the angle loss between the first prediction fingernail key point coordinate and the second prediction fingernail key point coordinate and the marked fingernail tip key point coordinate and the fingernail root key point coordinate;
calculating distance loss between the first predicted nail key point coordinate and the second predicted nail key point coordinate and the marked nail tip key point coordinate and the nail root key point coordinate according to the distance loss item;
determining a sum of the angular loss and the distance loss as the loss value.
4. The nail keypoint detection method of claim 3, wherein said calculating, according to said angular loss term, an angular loss between said first and second predicted nail keypoint coordinates and said annotated nail tip keypoint coordinates and said nail root keypoint comprises:
determining a first connection between the first predicted nail keypoint coordinate and the second predicted nail keypoint coordinate;
determining a second connecting line between the marked nail tip key point coordinates and the nail root key points;
and calculating an included angle between the first connecting line and the second connecting line according to the angle loss item, and determining the included angle as the angle loss.
5. The nail keypoint detection method of claim 4, wherein said angular loss term is:
wherein x is a nail sample image, x1、y1Is the nail tip key point coordinate, x, marked in the plane rectangular coordinate system mapped by the nail sample image2、y2The coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,extracting, for the neural network, a first predicted nail tip keypoint coordinate corresponding to the nail tip keypoint coordinate from a planar orthogonal coordinate system mapped to the nail sample image,second predicted nail base keypoint coordinates corresponding to the nail base keypoint coordinates extracted from a planar rectangular coordinate system mapped by the nail sample image for the neural network.
6. The nail keypoint detection method of claim 3, wherein said calculating, according to said distance loss term, a distance loss between said first and second predicted nail keypoint coordinates and said annotated nail tip and root keypoint coordinates comprises:
calculating a first distance between the first predicted nail key point coordinate and the labeled nail tip key point coordinate according to the distance loss item, and calculating a second distance between the second predicted nail key point coordinate and the labeled nail root key point coordinate;
determining a sum of the first distance and the second distance as the distance loss.
7. The nail keypoint detection method of claim 6, wherein said distance loss term is:wherein x is a nail sample image, x1、y1Is the nail tip key point coordinate, x, marked in the plane rectangular coordinate system mapped by the nail sample image2、y2The coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,extracting, for the neural network, a first predicted nail tip keypoint coordinate corresponding to the nail tip keypoint coordinate from a planar orthogonal coordinate system mapped to the nail sample image,and extracting a second predicted nail key point coordinate corresponding to the nail root key point coordinate from a plane rectangular coordinate system mapped by the nail sample image by the neural network, wherein lambda represents weight.
8. A nail key point detecting device, comprising:
the image acquisition module to be detected is configured to execute acquisition of an image to be detected, wherein the image to be detected comprises a nail region;
the nail key point coordinate detection module is configured to input the image to be detected into a key point detection model to obtain nail key point coordinates corresponding to the direction of the nail in the nail region, the key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function comprises an angle loss item and a distance loss item.
9. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the nail keypoint detection method of any one of claims 1 to 7.
10. A storage medium in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform the nail keypoint detection method of any one of claims 1 to 7.
Technical Field
The present disclosure relates to the field of deep learning technologies, and in particular, to a nail key point detection method and apparatus, an electronic device, and a storage medium.
Background
With the development of deep learning techniques, virtual nail art is more and more common in short video applications or camera applications. Currently, a common virtual nail art is based on nail key points to guide the attachment of directional materials.
In the related art, the current key point detection scheme can accurately detect the nail key points, but the finally calculated nail key points are difficult to avoid having direction jitter, so that the material attachment with directionality is difficult to complete.
Disclosure of Invention
The present disclosure provides a nail key point detection method, apparatus, electronic device, and storage medium, to at least solve the problem in the related art that it is difficult to avoid directional jitter of calculated nail key points. The technical scheme of the disclosure is as follows:
according to a first aspect of embodiments of the present disclosure, there is provided a nail keypoint detection method, comprising:
acquiring an image to be detected, wherein the image to be detected comprises a nail area;
inputting an image to be detected into a key point detection model to obtain a nail key point coordinate corresponding to the direction of a nail in a nail region, wherein the key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function comprises an angle loss item and a distance loss item.
In one embodiment, a method for training a keypoint detection model includes: acquiring a nail sample image, wherein the nail root key point coordinates and the nail tip key point coordinates are marked in the nail sample image; extracting a first predicted nail key point coordinate corresponding to the marked nail tip key point coordinate and a second predicted nail key point coordinate corresponding to the marked nail root key point coordinate in the nail sample image by adopting a neural network; determining a loss value according to a preset loss function, the marked nail tip key point coordinates and nail root key point coordinates, and the first prediction nail key point coordinates and the second prediction nail key point coordinates; and training the neural network according to the loss value to obtain a key point detection model.
In one embodiment, determining a loss value according to a preset loss function, the labeled nail tip and nail root key point coordinates, and the first and second predicted nail key point coordinates comprises: according to the angle loss item, calculating the angle loss between the first prediction fingernail key point coordinate and the second prediction fingernail key point coordinate and the marked fingernail tip key point coordinate and the fingernail root key point coordinate; calculating the distance loss between the first predicted fingernail key point coordinate and the second predicted fingernail key point coordinate and the marked fingernail tip key point coordinate and the fingernail root key point coordinate according to the distance loss item; determining the sum of the angle loss and the distance loss as the loss value.
In one embodiment, calculating an angular loss between the first and second predicted nail keypoint coordinates and the annotated nail tip keypoint coordinates and nail root keypoints according to an angular loss term comprises: determining a first connection between the first predicted fingernail keypoint coordinate and the second predicted fingernail keypoint coordinate; determining a second connecting line between the marked nail tip key point coordinates and the nail root key points; and calculating an included angle between the first connecting line and the second connecting line according to the angle loss term, and determining the included angle as the angle loss.
In one embodiment, the angle loss term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,second predicted nail base keypoint coordinates corresponding to nail root keypoint coordinates extracted from a planar rectangular coordinate system mapped from a nail sample image for a neural network。
In one embodiment, calculating a distance loss between the first predicted nail keypoint coordinate and the second predicted nail keypoint coordinate and the annotated nail tip keypoint coordinate and the nail root keypoint coordinate according to a distance loss term comprises: calculating a first distance between the first predicted fingernail key point coordinate and the labeled fingernail tip key point coordinate according to the distance loss item, and calculating a second distance between the second predicted fingernail key point coordinate and the labeled fingernail root key point coordinate; the sum of the first distance and the second distance is determined as a distance loss.
In one embodiment, the distance penalty term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,and b, extracting a second predicted nail key point coordinate corresponding to the nail root key point coordinate from a plane rectangular coordinate system mapped by the nail sample image by the neural network, wherein lambda represents weight.
In one embodiment, before training the neural network according to the loss value, the method further includes: determining the size of lambda, wherein the size of lambda is used for controlling the accuracy weight of the angle or the accuracy weight of the distance in the output result of the neural network.
In one embodiment, training the neural network based on the loss values comprises: judging whether the neural network converges according to the loss value; if the neural network is converged, determining that the training of the neural network is finished, updating parameters of the neural network, and determining the neural network after the parameters are updated as a key point detection model; if the neural network is not converged, adjusting parameters of the neural network according to a loss value, and returning to execute the step of extracting a first predicted nail key point coordinate corresponding to the labeled nail tip key point coordinate and a second predicted nail key point coordinate corresponding to the labeled nail root key point coordinate in the nail sample image by using the neural network until the neural network is converged.
According to a second aspect of the embodiments of the present disclosure, there is provided a nail key point detecting device including:
the image acquisition module to be detected is configured to execute acquisition of an image to be detected, wherein the image to be detected comprises a nail region;
the nail key point coordinate detection module is configured to input an image to be detected into a key point detection model to obtain nail key point coordinates corresponding to the direction of a nail in the nail region, the key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function comprises an angle loss item and a distance loss item.
In one embodiment, the nail key point coordinate detecting module includes: a sample image acquisition sub-module configured to perform acquisition of a nail sample image in which a nail root key point coordinate and a nail tip key point coordinate are labeled; a feature extraction sub-module configured to perform extraction of a first predicted nail keypoint coordinate corresponding to the annotated nail tip keypoint coordinate and a second predicted nail keypoint coordinate corresponding to the annotated nail root keypoint coordinate in the nail sample image using a neural network; a loss value determination submodule configured to perform a loss value determination according to a preset loss function, the labeled nail tip key point coordinates and nail root key point coordinates, and the first predicted nail key point coordinates and the second predicted nail key point coordinates; and the model training submodule is configured to train the neural network according to the loss value to obtain a key point detection model.
In one embodiment, the loss value determination sub-module includes: an angle loss calculation unit configured to perform calculating an angle loss between the first and second predicted nail keypoint coordinates and the annotated nail tip and root keypoint coordinates according to an angle loss term; a distance loss calculation unit configured to perform calculating a distance loss between the first and second predicted nail keypoint coordinates and the annotated nail tip and root keypoint coordinates according to a distance loss term; a loss value calculation unit configured to perform determination of a sum of the angle loss and the distance loss as a loss value.
In one embodiment, the angle loss calculation unit is configured to perform: determining a first connection between the first predicted fingernail keypoint coordinate and the second predicted fingernail keypoint coordinate; determining a second connecting line between the marked nail tip key point coordinates and the nail root key points; and calculating an included angle between the first connecting line and the second connecting line according to the angle loss term, and determining the included angle as the angle loss.
In one embodiment, the angle loss term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,deriving nail sample images for neural networksAnd a second predicted fingernail key point coordinate corresponding to the fingernail root key point coordinate extracted from the mapped plane rectangular coordinate system.
In one embodiment, the distance loss calculation unit is configured to perform: calculating a first distance between the first predicted fingernail key point coordinate and the labeled fingernail tip key point coordinate according to the distance loss item, and calculating a second distance between the second predicted fingernail key point coordinate and the labeled fingernail root key point coordinate; the sum of the first distance and the second distance is determined as a distance loss.
In one embodiment, the distance penalty term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,and b, extracting a second predicted nail key point coordinate corresponding to the nail root key point coordinate from a plane rectangular coordinate system mapped by the nail sample image by the neural network, wherein lambda represents weight.
In one embodiment, the model training submodule further includes a parameter determining unit configured to perform determining a magnitude of λ, the magnitude of λ being used to control an accuracy weight of an angle or an accuracy weight of a distance in the neural network output result.
In one embodiment, the model training submodule is configured to perform: judging whether the neural network converges according to the loss value; if the neural network is converged, determining that the training of the neural network is finished, updating parameters of the neural network, and determining the neural network after the parameters are updated as a key point detection model; if the neural network is not converged, adjusting parameters of the neural network according to the loss value, and returning to execute the step of extracting a first predicted nail key point coordinate corresponding to the marked nail tip key point coordinate and a second predicted nail key point coordinate corresponding to the marked nail root key point coordinate in the nail sample image by using the neural network until the neural network is converged.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions such that the electronic device performs the nail keypoint detection method described in any embodiment of the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a storage medium having instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the nail keypoint detection method described in any one of the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program stored in a readable storage medium, from which at least one processor of a device reads and executes the computer program, causing the device to perform the nail keypoint detection method described in any of the first aspects.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects: the method comprises the steps of obtaining an image to be detected with a nail region, inputting the image to be detected into a key point detection model for key point extraction, and obtaining nail key point coordinates corresponding to the direction of a nail in the nail region.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
Fig. 1 is a schematic diagram illustrating an output of keypoint detection in the conventional art according to an exemplary embodiment.
Fig. 2 is an application environment diagram illustrating a nail keypoint detection method according to an exemplary embodiment.
FIG. 3 is a flow diagram illustrating a nail keypoint detection method, according to an exemplary embodiment.
FIG. 4 is a flowchart illustrating the training steps of the keypoint detection model, according to an exemplary embodiment.
FIG. 5 is a flowchart illustrating steps for determining a loss value according to an exemplary embodiment.
FIG. 6 is a graphical illustration of a comparison of the output of nail keypoint detection, shown according to an exemplary embodiment.
Fig. 7 is a block diagram illustrating a nail keypoint detection apparatus according to an exemplary embodiment.
Fig. 8 is an internal block diagram of an electronic device shown in accordance with an example embodiment.
Fig. 9 is an internal block diagram of a server according to an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
In the conventional technology, loss function calculation and back propagation are usually performed on the output coordinates of the model and the real coordinates by using average absolute value difference, and the distance between the coordinates is optimized, so that only the accuracy of the output coordinates of the key points of the model can be restricted, but the accuracy of the direction cannot be restricted, as shown in fig. 1, when the nail key point coordinate detection is performed by applying the model, the model may output two results of a first type of coordinates and a second type of coordinates corresponding to the same input under different random noises. Since the connecting line 10 of the first type coordinates and the connecting line 20 of the second type coordinates correspond to the same loss function (not shown in the figures, it is understood that the two end points of the connecting line are corresponding coordinates, for example, the two end points of the connecting line 10 are the first type coordinates), and the loss value between the two end points and the connecting line 30 of the real coordinates is the same, the direction of the nail key point coordinates output during the model application will continuously jitter between the first type coordinates and the second type coordinates, and the material attachment is difficult to complete.
Based on this, the present disclosure provides a nail key point detection method, which can be applied in an application environment as shown in fig. 2. Wherein the terminal 210 interacts with the server 220 through the network. Specifically, the terminal 210 may be various devices having an image capturing function, such as but not limited to various smart phones, tablet computers, cameras, and portable image capturing devices, and the server 220 may be implemented by an independent server or a server cluster formed by a plurality of servers. In this embodiment, the method for detecting a nail key point may be applied to the server 220, specifically, the terminal 210 is configured to acquire an image to be detected, where the image to be detected includes a nail region, the terminal 210 sends the acquired image to be detected to the server 220 through a network, and the server 220 inputs the image to be detected into a key point detection model to obtain a nail key point coordinate corresponding to a direction of a nail in the nail region, where the key point detection model is obtained by pre-training a neural network using a preset loss function, and the preset loss function at least includes an angle loss function. Because the angle loss function exists in the loss function for training the neural network, the accuracy of the direction can be effectively restricted, and the trained key point detection model can output nail key point coordinates with stable directions.
The nail key point detection method provided by the disclosure can also be applied to the terminal 210, specifically, a trained key point detection model is loaded in the terminal 210, the terminal 210 can also collect an image to be detected, wherein the image to be detected comprises a nail region, the terminal 210 performs key point detection on the collected image to be detected through the key point detection model to obtain nail key point coordinates corresponding to the direction of a nail in the nail region, wherein the key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function at least comprises an angle loss function. Because the angle loss function exists in the loss function for training the neural network, the accuracy of the direction can be effectively restrained, and the nail key point coordinate with stable direction can be output by loading the trained key point detection model in the terminal.
Fig. 3 is a flowchart illustrating a nail key point detecting method according to an exemplary embodiment, which is illustrated in fig. 3, and includes the following steps, taking the method as an example for a terminal.
In step S310, an image to be detected is acquired.
The image to be detected is an image to be subjected to nail key point detection, and specifically, the image to be detected includes a nail region. In this embodiment, treat that the image of detecting can be the image that has the nail region that the terminal was gathered, for example, to the scene of virtual beautiful nail, when the user need carry out virtual beautiful nail, then through terminal acquisition including the image of waiting to detect of user's nail region to the terminal carries out subsequent processing to the image of detecting of gathering, thereby carries out the material laminating, in order to accomplish virtual beautiful nail.
In step S320, the image to be detected is input into the key point detection model, and the nail key point coordinates corresponding to the direction of the nail in the nail region are obtained.
The key point detection model is obtained by adopting a preset loss function to train a neural network in advance, and the preset loss function comprises an angle loss term and a distance loss term. Specifically, the neural Network may be implemented based on a basic feature extraction Network framework, where the basic feature extraction Network framework includes, but is not limited to, a Network such as a VGG (Visual Geometry Group Network) or a hour glass Network; the angle loss item is used for constraining the angle or the direction of the output result of the model in the process of training the model, and the distance loss item is used for constraining the position of the output result of the model in the process of training the model, so that the stability and the accuracy of the output result of the model are ensured. The nail key point coordinates are the coordinate positions in the nail that have a decisive influence on the direction of the material attachment. The direction of the nail may refer to a growing direction of the nail, such as a direction from a nail root to a nail tip, and may be a direction opposite to the growing direction, or another direction, which is not limited in this embodiment. In this embodiment, the terminal detects the acquired image to be detected through the key point detection model, so as to obtain the nail key point coordinates corresponding to the direction of the nail in the nail region in the image to be detected, that is, the direction of the obtained nail key point coordinates is the same as the direction of the nail in the nail region, and therefore, material fitting can be performed based on the obtained nail key point coordinates to complete virtual nail beautifying.
According to the method for detecting the key points of the fingernails, the image to be detected with the fingernail region is obtained, the image to be detected is input into the key point detection model for key point extraction, and therefore the key point coordinates corresponding to the directions of the fingernails in the fingernail region are obtained.
In an exemplary embodiment, as shown in fig. 4, the method for training the keypoint detection model may include the following steps:
in step S410, a nail sample image is acquired.
The nail sample image is training data for training a neural network, and the nail sample image is marked with a nail root key point coordinate and a nail tip key point coordinate, specifically, the nail root key point coordinate may be a nail root vertex coordinate or a nail root central point coordinate, and the nail tip key point coordinate may be a nail tip vertex coordinate or a nail tip central point coordinate. In the present embodiment, the nail root key point coordinates and the nail tip key point coordinates are marked in the acquired nail sample image, so that the neural network can learn the directional characteristics of the nail.
In step S420, predicted nail keypoint coordinates in the nail sample image are extracted using a neural network.
The predicted nail key point coordinates are output results obtained after characteristic detection is carried out on the nail sample images through a neural network, and specifically the predicted nail key point coordinates comprise first predicted nail key point coordinates corresponding to the marked nail tip key point coordinates and second predicted nail key point coordinates corresponding to the marked nail root key point coordinates. In this embodiment, the key point extraction is performed on the nail sample image through the neural network, so as to obtain a first predicted nail key point coordinate corresponding to the labeled nail tip key point coordinate in the nail sample image, and a second predicted nail key point coordinate corresponding to the labeled nail root key point coordinate.
In step S430, a loss value is determined according to a preset loss function, the labeled nail root key point coordinates and nail tip key point coordinates, and the predicted nail key point coordinates.
The preset loss function comprises an angle loss term and a distance loss term, wherein the angle loss term is specifically used for representing the angle loss between a first predicted nail key point coordinate and a second predicted nail key point coordinate which are predicted and a nail root key point coordinate and a nail tip key point coordinate which are marked in a nail sample image; the distance loss term is used for representing the distance loss between the predicted first and second predicted nail key point coordinates and the labeled nail root and tip key point coordinates in the nail sample image. In this embodiment, a corresponding loss value is obtained according to a preset loss function, the labeled nail root key point coordinates and nail tip key point coordinates, and the corresponding first predicted nail key point coordinates and second predicted nail key point coordinates.
In step S440, the neural network is trained according to the loss value, and a keypoint detection model is obtained.
Specifically, whether the neural network converges or not can be judged according to the loss value, when the neural network converges is determined, the neural network training is completed, the parameters of the neural network are updated, and the neural network with the updated parameters is determined as the key point detection model. When the neural network is determined not to be converged, parameters of the neural network can be adjusted according to the loss value, the neural network continues to be trained on the basis of a back propagation method, namely, the step of extracting the predicted nail key point coordinates in the nail sample image by using the neural network is returned to be executed until the neural network is converged.
In the embodiment, the nail root key point coordinates and the nail tip key point coordinates are marked in the nail sample image used in the network training process, and the loss function comprising the angle loss item and the distance loss item is used in the training process, so that the model can learn the directional characteristics of the nail, and further, when the model is used for detecting the nail key point, the nail key point coordinates corresponding to the direction of the nail in the image to be detected can be output, and the stability of the output result of the model is ensured.
In an exemplary embodiment, as shown in fig. 5, in step S430, a loss value is determined according to a preset loss function, the labeled nail root key point coordinates and nail tip key point coordinates, and the first predicted nail key point coordinates and the second predicted nail key point coordinates, which may be specifically implemented by the following steps:
in step S431, an angle loss between the first and second predicted nail key point coordinates and the labeled nail root and tip key point coordinates is calculated according to a preset angle loss term.
Specifically, since the angle loss term is used to represent the angle loss between the predicted nail key point coordinate and the annotated nail root key point coordinate and the nail tip key point coordinate in the nail sample image, that is, the angle between the connecting line of the model output key point coordinate and the connecting line of the annotated coordinate, the angle is usually determined by two intersecting straight lines, one straight line is determined by at least two points, and in this embodiment, the predicted nail key point coordinate includes the first predicted nail key point coordinate and the second predicted nail key point coordinate, therefore, the first connecting line between the two is determined by the first predicted nail key point coordinate and the second predicted nail key point coordinate, the second connecting line between the two is determined by the nail root key point coordinate and the nail tip key point coordinate, and the included angle between the first connecting line and the second connecting line is calculated according to the angle loss term, the included angle is the angle loss between the predicted nail key point coordinate and the marked nail root key point coordinate and the nail tip key point coordinate.
In step S432, distance loss between the predicted nail key point coordinates and the labeled nail root key point coordinates and nail tip key point coordinates is calculated according to the distance loss term in the preset loss function.
Since the distance loss term is used to represent the distance loss between the predicted nail key point coordinate and the labeled nail root key point coordinate and the nail tip key point coordinate in the nail sample image, that is, the distance between the model output key point coordinate and the labeled coordinate, in this embodiment, taking manhattan distance as an example for explanation, the distance loss between the predicted nail key point coordinate and the labeled nail root key point coordinate and the nail tip key point coordinate is: and the sum of absolute wheel base distances between the first predicted nail key point coordinate and the second predicted nail key point coordinate and the nail root key point coordinate and the nail tip key point coordinate on a standard coordinate system. Specifically, a first distance between a first predicted nail key point coordinate and a labeled nail tip key point coordinate is calculated according to a distance loss term, a second distance between a second predicted nail key point coordinate and a labeled nail root key point coordinate is calculated, and the sum of the first distance and the second distance is determined as distance loss.
In step S433, the sum of the angle loss and the distance loss is determined as a loss value.
Specifically, the loss value for training the neural network may be the sum of the angle loss and the distance loss. So that the trained model can output results with stable direction and accurate distance.
In an exemplary embodiment, the angle loss term may specifically be implemented by the following expression:
the distance loss term may be specifically implemented by the following expression:it represents the manhattan distance between the predicted nail key point coordinates and the marked nail root key point coordinates and nail tip key point coordinates; the predetermined loss function is then: l ═ Lθ(x)+Ld. Wherein x is a nail sample image, x1、y1For nail tips marked in a planar rectangular coordinate system mapped on nail sample imagesCoordinates of key points, x2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,for a second predicted nail key point coordinate corresponding to a nail root key point coordinate extracted by a neural network from a plane rectangular coordinate system mapped by a nail sample image, λ represents a weight, and the initial value of λ is usually 1, in a model training process, the accuracy weight of an angle or the accuracy weight of a distance in a model output result can be controlled by adjusting the proportion of λ, that is, the accuracy and the stability of a direction or the accuracy of a coordinate position of a final output result of the model can be more inclined or more inclined by adjusting the proportion of λ, for example, if λ is increased, the direction of the output result is more accurate, but the coordinate position has a larger deviation value, so the detail of λ can be adjusted in the training process of the model.
In an exemplary embodiment, the distance loss term may be specifically implemented by the following expression:that is, the mean square difference is used to replace the mean absolute value in the above formula, and other similar alternative ways may also be used, which is not limited in this embodiment.
As the neural network is trained by using the above loss function, so that the model can learn the directional characteristic of the labeled nail, as shown in fig. 6, although the connection line 100 of the first type coordinate and the connection line 200 of the second type coordinate correspond to the same loss function (the specific coordinate point is not shown in the figure, it can be understood that the two end points of the connection line are corresponding coordinate points, for example, the two end points of the connection line 100 are the first type coordinates, wherein the connection line 300 is the connection line between the labeled nail root key point coordinate and the nail tip key point coordinate), and the values of the connection line 100 and the connection line 200 corresponding to the second term (i.e. distance loss) of the loss function are also the same, due to the control of the first term (i.e. angle loss) of the loss function, the model outputs the second type coordinate and brings a lower loss function value, after sufficient training, when the nail key point coordinate detection is performed on the model detection image after applying training, the model is more prone to outputting the second type of coordinates, so that stability and accuracy of the output result of the model in the direction are guaranteed, and the problem that the direction of a nail key point is shaken in the directional virtual nail fitting process is effectively solved.
It should be understood that although the various steps in the flow charts of fig. 1-6 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-6 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
Fig. 7 is a block diagram illustrating a nail keypoint detection apparatus according to an exemplary embodiment. Referring to fig. 7, the apparatus includes an image acquisition module 701 to be detected and a nail key point coordinate detection module 702.
The image acquisition module 701 to be detected is configured to execute acquisition of an image to be detected, wherein the image to be detected comprises a nail region;
a nail key point coordinate detection module 702 configured to input an image to be detected into a key point detection model to obtain nail key point coordinates corresponding to a direction of a nail in the nail region, wherein the key point detection model is obtained by pre-training a neural network by using a preset loss function, and the preset loss function includes an angle loss term and a distance loss term.
In an exemplary embodiment, nail keypoint coordinate detection module 702 comprises: a sample image acquisition sub-module configured to perform acquisition of a nail sample image in which a nail root key point coordinate and a nail tip key point coordinate are labeled; a feature extraction sub-module configured to perform extraction of a first predicted nail keypoint coordinate corresponding to the annotated nail tip keypoint coordinate and a second predicted nail keypoint coordinate corresponding to the annotated nail root keypoint coordinate in the nail sample image using a neural network; a loss value determination submodule configured to perform a loss value determination according to a preset loss function, the labeled nail tip key point coordinates and nail root key point coordinates, and the first predicted nail key point coordinates and the second predicted nail key point coordinates; and the model training submodule is configured to train the neural network according to the loss value to obtain a key point detection model.
In an exemplary embodiment, the loss value determination submodule includes: an angle loss calculation unit configured to perform calculating an angle loss between the first and second predicted nail keypoint coordinates and the annotated nail tip and root keypoint coordinates according to an angle loss term; a distance loss calculation unit configured to perform calculating a distance loss between the first and second predicted nail keypoint coordinates and the annotated nail tip and root keypoint coordinates according to a distance loss term; a loss value calculation unit configured to perform determination of a sum of the angle loss and the distance loss as a loss value.
In an exemplary embodiment, the angle loss calculation unit is configured to perform: determining a first connection between the first predicted fingernail keypoint coordinate and the second predicted fingernail keypoint coordinate; determining a second connecting line between the marked nail tip key point coordinates and the nail root key points; and calculating an included angle between the first connecting line and the second connecting line according to the angle loss term, and determining the included angle as the angle loss.
In an exemplary embodiment, the angle loss term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,and extracting second predicted nail key point coordinates corresponding to the nail root key point coordinates from the planar rectangular coordinate system mapped by the nail sample image for the neural network.
In an exemplary embodiment, the distance loss calculation unit is configured to perform: calculating a first distance between the first predicted fingernail key point coordinate and the labeled fingernail tip key point coordinate according to the distance loss item, and calculating a second distance between the second predicted fingernail key point coordinate and the labeled fingernail root key point coordinate; the sum of the first distance and the second distance is determined as a distance loss.
In an exemplary embodiment, the distance penalty term is:wherein x is a nail sample image, x1、y1Is the coordinate of the key point of the nail tip, x, marked in a plane rectangular coordinate system mapped by the nail sample image2、y2For the coordinates of the critical point of the nail root marked in the plane rectangular coordinate system mapped by the nail sample image,first predicted nail tip keypoint coordinates corresponding to the nail tip keypoint coordinates extracted by the neural network from a planar rectangular coordinate system mapped by the nail sample image,and b, extracting a second predicted nail key point coordinate corresponding to the nail root key point coordinate from a plane rectangular coordinate system mapped by the nail sample image by the neural network, wherein lambda represents weight.
In an exemplary embodiment, the model training submodule further comprises a parameter determination unit configured to perform determining a magnitude of λ, the magnitude of λ being used to control an accuracy weight of an angle or an accuracy weight of a distance in the neural network output result.
In an exemplary embodiment, the model training submodule is configured to perform: judging whether the neural network converges according to the loss value; if the neural network is converged, determining that the training of the neural network is finished, updating parameters of the neural network, and determining the neural network after the parameters are updated as a key point detection model; if the neural network is not converged, adjusting parameters of the neural network according to the loss value, and returning to execute the step of extracting a first predicted nail key point coordinate corresponding to the marked nail tip key point coordinate and a second predicted nail key point coordinate corresponding to the marked nail root key point coordinate in the nail sample image by using the neural network until the neural network is converged.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 8 is a block diagram illustrating an apparatus Z00 for a nail keypoint detection method, according to an exemplary embodiment. For example, device Z00 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, and the like.
Referring to fig. 8, device Z00 may include one or more of the following components: a processing component Z02, a memory Z04, a power component Z06, a multimedia component Z08, an audio component Z10, an interface to input/output (I/O) Z12, a sensor component Z14 and a communication component Z16.
The processing component Z02 generally controls the overall operation of the device Z00, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component Z02 may include one or more processors Z20 to execute instructions to perform all or part of the steps of the method described above. Further, the processing component Z02 may include one or more modules that facilitate interaction between the processing component Z02 and other components. For example, the processing component Z02 may include a multimedia module to facilitate interaction between the multimedia component Z08 and the processing component Z02.
The memory Z04 is configured to store various types of data to support operations at device Z00. Examples of such data include instructions for any application or method operating on device Z00, contact data, phonebook data, messages, pictures, videos, etc. The memory Z04 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component Z06 provides power to the various components of the device Z00. The power component Z06 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device Z00.
The multimedia component Z08 comprises a screen between the device Z00 and the user providing an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component Z08 includes a front facing camera and/or a rear facing camera. When device Z00 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and/or the rear-facing camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component Z10 is configured to output and/or input an audio signal. For example, the audio component Z10 includes a Microphone (MIC) configured to receive external audio signals when the device Z00 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory Z04 or transmitted via the communication component Z16. In some embodiments, the audio component Z10 further includes a speaker for outputting audio signals.
The I/O interface Z12 provides an interface between the processing component Z02 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly Z14 includes one or more sensors for providing status assessment of various aspects to the device Z00. For example, sensor assembly Z14 may detect the open/closed state of device Z00, the relative positioning of the components, such as the display and keypad of device Z00, sensor assembly Z14 may also detect a change in the position of one component of device Z00 or device Z00, the presence or absence of user contact with device Z00, the orientation or acceleration/deceleration of device Z00, and a change in the temperature of device Z00. The sensor assembly Z14 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly Z14 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly Z14 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component Z16 is configured to facilitate wired or wireless communication between device Z00 and other devices. Device Z00 may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component Z16 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component Z16 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the device Z00 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as the memory Z04, comprising instructions executable by the processor Z20 of the device Z00 to perform the above method. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Fig. 9 is a block diagram illustrating an apparatus S00 for a nail keypoint detection method, according to an example embodiment. For example, the device S00 may be a server. Referring to FIG. 9, device S00 includes a processing component S20 that further includes one or more processors and memory resources represented by memory S22 for storing instructions, e.g., applications, that are executable by processing component S20. The application program stored in the memory S22 may include one or more modules each corresponding to a set of instructions. Further, the processing component S20 is configured to execute instructions to perform the above-described nail keypoint detection method.
The device S00 may also include a power supply component S24 configured to perform power management of the device S00, a wired or wireless network interface S26 configured to connect the device S00 to a network, and an input-output (I/O) interface S28. The device S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.
In an exemplary embodiment, there is also provided a storage medium comprising instructions, such as the memory S22 comprising instructions, executable by the processor of the device S00 to perform the above method. The storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.