阅读说明：本技术 心胸比确定方法、装置、设备、存储介质及计算机设备 (Cardiothoracic ratio determining method, cardiothoracic ratio determining device, cardiothoracic ratio determining equipment, storage medium and computer equipment ) 是由 邹彤 周越 王少康 陈宽 于 2019-10-31 设计创作，主要内容包括：本发明公开了一种心胸比确定方法、装置、设备、存储介质及计算机设备，可以获得胸部正位图像的特征图；将所述特征图输入预设的心胸比关键点确定模型中，获得所述胸部正位图像上心缘关键点的位置概率与所述胸部正位图像上胸廓关键点的位置概率；根据所述心缘关键点的位置概率与所述胸廓关键点的位置概率，确定所述胸部正位图像的心胸比。本发明通过预设的心胸比关键点确定模型对心缘关键点和胸廓关键点进行精准定位的技术手段，解决了现有技术中心缘关键点和胸廓关键点定位不稳定的技术问题，进而达到了精准计算心胸比的技术效果。(The invention discloses a method, a device, equipment, a storage medium and computer equipment for determining a heart-chest ratio, which can obtain a characteristic diagram of a chest orthostatic image; inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image; and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points. According to the invention, the technical means that the model accurately positions the heart edge key points and the thoracic cavity key points is determined through the preset heart-chest ratio key points, the technical problem that the center edge key points and the thoracic cavity key points are unstable in positioning in the prior art is solved, and the technical effect of accurately calculating the heart-chest ratio is achieved.)

1. A cardiothoracic ratio determination method, comprising:

obtaining a feature map of a chest orthopaedics image;

inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image;

and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points.

2. The method of claim 1, wherein the inputting the feature map into a predetermined cardiothoracic ratio keypoint determination model to obtain the position probability of the cardiac edge keypoints and the position probability of the thoracic edge keypoints on the chest ortho image comprises:

inputting the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter to obtain the position probability of a first number of heart edge key points and the position probability of a second number of thoracic key points.

3. The method according to claim 2, wherein the first number of the first location probabilities of the first number of the second number.

4. The method of claim 3, wherein determining the cardiothoracic ratio in the chest ortho image based on the location probability of the cardiac edge keypoints and the location probability of the thoracic keypoints comprises:

determining the transverse diameter of the heart according to the position probability of the first number of heart edge key points;

determining the transverse diameter of the thorax according to the position probability of the second number of the thoracic key points;

and dividing the heart transverse diameter by the thoracic transverse diameter to determine the cardiothoracic ratio in the chest orthostatic image.

5. The method of claim 4, wherein the first number is 2 and the second number is 2, and wherein the first convolution parameter comprises: the convolution kernel size is 3 × 3 and the number of output channels is 4;

and/or the presence of a gas in the gas,

determining a heart transverse diameter according to the position probability of the first number of heart edge key points, including:

determining the positions of the 2 heart-edge key points according to the position probabilities of the 2 heart-edge key points;

determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart;

determining the transverse diameter of the thorax according to the position probability of the second number of the thoracic key points, comprising:

determining the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points;

and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

6. The method of any one of claims 1 to 5, wherein the obtaining a feature map of the chest ortho image comprises:

obtaining initial images of the chest orthostatic image at a plurality of different scales;

and fusing the initial images under the different scales to obtain a characteristic image.

7. The method according to claim 6, wherein the fusing the initial maps at the plurality of different scales to obtain a feature map comprises:

the initial images under the different scales are up-sampled according to a first up-sampling parameter, and a third number of matrix images are obtained, wherein the scales of the matrix images of the third number are the same;

element summation is carried out on the third number of matrix diagrams to obtain element diagrams;

and upsampling the element graph according to a second upsampling parameter to obtain a characteristic graph.

8. A cardiothoracic ratio determination apparatus, comprising: a characteristic map obtaining unit, a position probability obtaining unit and a cardiothoracic ratio determining unit,

the characteristic map obtaining unit is used for obtaining a characteristic map of the chest orthostatic image;

the position probability obtaining unit is used for inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image;

and the cardio-thoracic ratio determining unit is used for determining the cardio-thoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points.

9. The apparatus according to claim 8, wherein the location probability obtaining unit is specifically configured to input the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter to obtain the location probabilities of the first number of cardiac edge key points and the second number of thoracic key points.

10. The apparatus according to claim 9, wherein the first number of the first location probabilities of the first number of the second number.

11. The apparatus of claim 10, wherein the cardiothoracic ratio determining unit comprises: a heart transverse diameter determining subunit, a thoracic transverse diameter determining subunit and a cardiothoracic ratio determining subunit,

the heart transverse diameter determining subunit is configured to determine a heart transverse diameter according to the position probability of the first number of heart edge key points;

the thoracic transverse diameter determining subunit is configured to determine the thoracic transverse diameter according to the position probabilities of the second number of thoracic key points;

and the cardiothoracic ratio determining subunit is used for determining the quotient of the heart transverse diameter and the thoracic transverse diameter as the cardiothoracic ratio in the chest orthostatic image.

12. The apparatus of claim 11, wherein the first number is 2 and the second number is 2, and wherein the first convolution parameter comprises: the convolution kernel size is 3 × 3 and the number of output channels is 4;

and/or the presence of a gas in the gas,

the heart transverse diameter determining subunit is specifically configured to determine the positions of the 2 heart edge key points according to the position probabilities of the 2 heart edge key points; determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart;

the thoracic transverse diameter determining subunit is specifically used for determining the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points; and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

13. The apparatus according to any one of claims 8 to 12, wherein the feature map obtaining unit includes: an initial map acquisition subunit and a feature map acquisition subunit,

the initial image obtaining subunit is used for obtaining initial images of the chest orthotopic image at a plurality of different scales;

and the characteristic map obtaining subunit is used for fusing the initial maps under the different scales to obtain a characteristic map.

14. A storage medium having stored thereon computer-executable instructions which, when loaded and executed by a processor, carry out a method of cardiothoracic ratio determination as set forth in any one of claims 1 to 7.

15. A computer device comprising a processor, a memory and a program stored on the memory and executable on the processor, the processor when executing the program implementing at least the cardiothoracic ratio determination method as claimed in any of claims 1 to 7.

16. A cardiothoracic ratio determination apparatus, comprising: a chest positive image receiving device, a processor, a memory, a communication bus and an output device, the memory having stored thereon a program executable on the processor,

the processor is respectively in communication connection with the chest normal position image receiving device, the memory and the output device through the communication bus;

the chest positive image receiving device receives a chest positive image;

the processor, when executing a program, at least implements a cardiothoracic ratio determination method as claimed in any one of claims 1 to 7 above;

and the output device obtains and outputs the cardiothoracic ratio determined by the processor.

Technical Field

The invention relates to the field of image processing, in particular to a method, a device, equipment, a storage medium and computer equipment for determining a heart-chest ratio.

Background

The cardiothoracic ratio is the ratio of the maximum transverse diameter of the heart to the maximum transverse diameter of the thorax. Technicians can manually measure cardio-thoracic ratio on a client's chest orthostatic image, however manual measurement is time consuming, inefficient and not easy to save.

The existing digital image processing method can segment the lung fields and the heart areas in the chest orthostatic image, determine key points for calculating the cardio-thoracic ratio and calculate to obtain the cardio-thoracic ratio. However, the segmentation of the lung fields and the heart regions by the digital image processing method is easily affected by factors such as image contrast, brightness, gray scale distribution and the like, and different machines, exposure time, imaging modes and conditions of the client can affect the quality of the final chest orthotopic image, so that the final segmentation effect is affected, the key points of the heart-chest ratio are inaccurately positioned, and the calculated heart-chest ratio has a large error.

Disclosure of Invention

In view of the foregoing problems, embodiments of the present invention provide a method, an apparatus, a device, a storage medium, and a computer device for determining a cardiothoracic ratio, which overcome the foregoing problems or at least partially solve the foregoing problems, and the technical solutions are as follows:

a cardiothoracic ratio determination method, comprising:

obtaining a feature map of a chest orthopaedics image;

inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image;

and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points.

Optionally, the inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key point on the chest ortho-position image and the position probability of the thoracic contour key point on the chest ortho-position image includes:

inputting the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter to obtain the position probability of a first number of heart edge key points and the position probability of a second number of thoracic key points.

Optionally, the position probability of each cardiac edge key point in the position probabilities of the first number of cardiac edge key points corresponds to one cardiac edge key point, and the cardiac edge key points corresponding to the position probabilities of the cardiac edge key points are different, the position probability of each thoracic edge key point in the position probabilities of the second number of thoracic edge key points corresponds to one thoracic edge key point, and the thoracic edge key points corresponding to the position probabilities of the thoracic edge key points are different.

Optionally, the determining the cardiothoracic ratio in the chest orthotopic image according to the position probability of the cardiac edge keypoints and the position probability of the thoracic rib keypoints includes:

determining the transverse diameter of the heart according to the position probability of the first number of heart edge key points;

determining the transverse diameter of the thorax according to the position probability of the second number of the thoracic key points;

and dividing the heart transverse diameter by the thoracic transverse diameter to determine the cardiothoracic ratio in the chest orthostatic image.

Optionally, the first number is 2, the second number is 2, and the first convolution parameter includes: the convolution kernel size is 3 × 3 and the number of output channels is 4.

And/or the presence of a gas in the gas,

determining a heart transverse diameter according to the position probability of the first number of heart edge key points, including:

determining the positions of the 2 heart-edge key points according to the position probabilities of the 2 heart-edge key points;

determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart;

determining the transverse diameter of the thorax according to the position probability of the second number of the thoracic key points, comprising:

determining the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points;

and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

Optionally, the obtaining a feature map of the chest orthostatic image includes:

obtaining initial images of the chest orthostatic image at a plurality of different scales;

and fusing the initial images under the different scales to obtain a characteristic image.

Optionally, the fusing the initial maps at the multiple different scales to obtain a feature map includes:

the initial images under the different scales are up-sampled according to a first up-sampling parameter, and a third number of matrix images are obtained, wherein the scales of the matrix images of the third number are the same;

element summation is carried out on the third number of matrix diagrams to obtain element diagrams;

and upsampling the element graph according to a second upsampling parameter to obtain a characteristic graph.

A cardiothoracic ratio determination apparatus, comprising: a characteristic map obtaining unit, a position probability obtaining unit and a cardiothoracic ratio determining unit,

the characteristic map obtaining unit is used for obtaining a characteristic map of the chest orthostatic image;

the position probability obtaining unit is used for inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image;

and the cardio-thoracic ratio determining unit is used for determining the cardio-thoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points.

Optionally, the position probability obtaining unit is specifically configured to input the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter, and obtains a first number of position probabilities of the cardiac edge key points and a second number of position probabilities of the thoracic edge key points.

Optionally, the position probability of each cardiac edge key point in the position probabilities of the first number of cardiac edge key points corresponds to one cardiac edge key point, and the cardiac edge key points corresponding to the position probabilities of the cardiac edge key points are different, the position probability of each thoracic edge key point in the position probabilities of the second number of thoracic edge key points corresponds to one thoracic edge key point, and the thoracic edge key points corresponding to the position probabilities of the thoracic edge key points are different.

Optionally, the cardiothoracic ratio determining unit includes: a heart transverse diameter determining subunit, a thoracic transverse diameter determining subunit and a cardiothoracic ratio determining subunit,

the heart transverse diameter determining subunit is configured to determine a heart transverse diameter according to the position probability of the first number of heart edge key points;

the thoracic transverse diameter determining subunit is configured to determine the thoracic transverse diameter according to the position probabilities of the second number of thoracic key points;

and the cardiothoracic ratio determining subunit is used for determining the quotient of the heart transverse diameter and the thoracic transverse diameter as the cardiothoracic ratio in the chest orthostatic image.

Optionally, the first number is 2, the second number is 2, and the first convolution parameter includes: the convolution kernel size is 3 × 3 and the number of output channels is 4;

and/or the presence of a gas in the gas,

the heart transverse diameter determining subunit is specifically configured to determine the positions of the 2 heart edge key points according to the position probabilities of the 2 heart edge key points; determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart;

the thoracic transverse diameter determining subunit is specifically used for determining the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points; and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

Optionally, the feature map obtaining unit includes: an initial map acquisition subunit and a feature map acquisition subunit,

the initial image obtaining subunit is used for obtaining initial images of the chest orthotopic image at a plurality of different scales;

and the characteristic map obtaining subunit is used for fusing the initial maps under the different scales to obtain a characteristic map.

Optionally, the feature map obtaining subunit includes: a matrix map obtaining subunit, an element map obtaining subunit and an upsampling subunit,

the matrix map obtaining subunit is configured to perform upsampling on the initial maps at the multiple different scales according to a first upsampling parameter, so as to obtain a third number of matrix maps, where the scales of the third number of matrix maps are the same;

the element map obtaining subunit is configured to perform element summation on the third number of matrix maps to obtain an element map;

and the up-sampling subunit is used for up-sampling the element diagram according to a second up-sampling parameter to obtain a characteristic diagram.

A storage medium having stored therein computer-executable instructions that, when loaded and executed by a processor, carry out a cardiothoracic ratio determination method as claimed in any one of the preceding claims.

A computer device comprising a processor, a memory and a program stored on the memory and executable on the processor, the processor when executing the program at least implementing a cardiothoracic ratio determination method as claimed in any preceding claim.

A cardiothoracic ratio determination device comprising: a chest positive image receiving device, a processor, a memory, a communication bus and an output device, the memory having stored thereon a program executable on the processor,

the processor is respectively in communication connection with the chest normal position image receiving device, the memory and the output device through the communication bus;

the chest positive image receiving device receives a chest positive image;

the processor, when executing a program, at least implements the cardiothoracic ratio determination method as defined in any of the above;

and the output device obtains and outputs the cardiothoracic ratio determined by the processor.

By means of the technical scheme, the method, the device, the equipment, the storage medium and the computer equipment for determining the cardiothoracic ratio can obtain a characteristic diagram of a chest orthostatic image; inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image; and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points. According to the embodiment of the invention, the technical means that the model accurately positions the heart edge key points and the thoracic cavity key points is determined through the preset heart-chest ratio key points, the technical problem that the center edge key points and the thoracic cavity key points are not stably positioned in the prior art is solved, and the technical effect of accurately calculating the heart-chest ratio is achieved.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is a flow chart of a cardiothoracic ratio determination method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a chest orthostatic image provided by an embodiment of the invention;

FIG. 3 is a flow chart of another cardiothoracic ratio determination method provided by an embodiment of the invention;

FIG. 4 is a flow chart illustrating another cardiothoracic ratio determination method provided by an embodiment of the present invention;

FIG. 5 is an illustrative diagram of obtaining an elemental map from a matrix map according to an embodiment of the invention;

FIG. 6 is a schematic flow chart of a method for obtaining a cardiothoracic ratio key point determination model according to an embodiment of the present invention;

FIG. 7 is a flow chart of another cardiothoracic ratio determination method according to an embodiment of the present invention

FIG. 8 is a diagram illustrating the location probability of a cardiac edge keypoint or the location probability of a thoracic keypoint according to an embodiment of the present invention;

FIG. 9 is a schematic diagram illustrating a coordinate axis pattern provided by an embodiment of the invention;

FIG. 10 is a flow chart illustrating another cardiothoracic ratio determination method provided by an embodiment of the present invention;

FIG. 11 is a schematic diagram of a cardiothoracic ratio image provided by an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of a cardiothoracic ratio determining apparatus according to an embodiment of the present invention;

FIG. 13 is a schematic diagram illustrating extension of a chest film cassette of a cardiothoracic ratio determination apparatus provided by an embodiment of the present invention;

FIG. 14 is a schematic diagram illustrating the retrieval of a chest film cassette of a cardiothoracic ratio determination apparatus provided by an embodiment of the present invention;

fig. 15 shows a connection diagram of a processor, a memory and a communication bus of a cardiothoracic ratio determination device according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As shown in fig. 1, a method for determining a cardiothoracic ratio according to an embodiment of the present invention may include:

and S100, obtaining a characteristic diagram of the chest orthostatic image.

Specifically, as shown in fig. 2, the chest ortho image may be a human chest ortho image obtained by applying Digital Radiography (DR) or Computed Radiography (CR) techniques to a detector using X-ray penetrating radiation. The chest orthostatic image may include posterior anterior (PA bit) and anterior posterior (AP bit). The PA position refers to a conventional positive position, namely, the user faces to the detector, and the chest is tightly attached to the detector to take a picture to obtain a chest positive position image; the AP position refers to an unconventional station position, namely that a user faces back to the detector, and the back is tightly attached to the detector to take a picture to obtain a chest normal position image. The detector comprises an X-ray machine. The scale may be the image size of the image at a certain resolution. The chest orthophoto scale can be the scale of the X-ray image output by the X-ray machine. For example, the resolution of the chest ortho image output by the X-ray machine may be 512 × 512.

It can be understood that, since the scales of the chest normal position images output by the X-ray machines of different types are different, a person skilled in the art can adjust the scales of the chest normal position images output by the X-ray machines to a required scale, and then execute the steps of the embodiment of the present invention.

Optionally, as shown in fig. 3, in another method for determining a cardiothoracic ratio according to an embodiment of the present invention, step S100 may include:

and S110, obtaining initial images of the chest orthostatic image under a plurality of different scales.

According to the embodiment of the invention, the initial images of the chest orthostatic image under a plurality of different scales can be obtained through the feature extraction network. The feature extraction network may include convolutional neural networks including Resnext-50, Resnext-101, Resnext-152, and Densenet. For example, with Resnext-50, embodiments of the present invention may obtain initial maps 1/32, 1/16, 1/8, and 1/4 at the scale of the chest normal image, respectively. The embodiment of the invention can obtain the image characteristics of the same characteristic point of the chest orthostatic image under different scales. For example, an embodiment of the present invention may obtain image features of the same keypoint in the initial map at four scales in the initial maps 1/32, 1/16, 1/8 and 1/4, which are scales of the breast orthostatic image, respectively.

And S120, fusing the initial images under the different scales to obtain a characteristic image.

Specifically, in the embodiment of the present invention, after feature extraction is performed on a plurality of initial images with different scales through an image feature extraction network, information such as edges, shapes, contours, local features, and the like is comprehensively processed to obtain a feature image. The image Feature extraction network may include FPN (Feature Pyramid Networks). For example, based on the above example of obtaining the initial map of the chest normal image at four scales through Resnext-50, the embodiment of the present invention may fuse the four initial maps of 1/32, 1/16, 1/8, and 1/4, which are scales of the chest normal image respectively, into one feature map through Panoptic FPN.

It is understood that the feature map of the embodiment of the present invention may also be an initial feature map. Specifically, in the embodiment of the present invention, an initial map of one scale in the initial maps of the chest orthostatic image under a plurality of different scales may be obtained through a feature extraction network as a feature map.

Based on the method shown in fig. 3, as shown in fig. 4, in another method for determining a cardiothoracic ratio according to an embodiment of the present invention, step S120 may include:

s121, performing upsampling on the plurality of initial images under different scales according to a first upsampling parameter to obtain a third number of matrix images, wherein the scales of the third number of matrix images are the same.

Specifically, in the embodiment of the present invention, the feature maps under a plurality of different scales may be upsampled according to different first upsampling parameters, so as to obtain a third number of matrix maps. Wherein the third number is related to the number of initial graphs. For example, after the initial maps at four different scales are upsampled according to different first upsampling parameters, four matrix maps are obtained.

The first upsampling parameter may be determined according to a scale of the initial map. For ease of understanding, the description is made herein by way of example: if the four initial maps 1/32, 1/16, 1/8 and 1/4 with the scales respectively being the scales of the chest positive image need to be up-sampled according to different first up-sampling parameters, the initial map 1/32 with the scale being the scale of the chest positive image can be up-sampled according to the first up-sampling parameters of 3 convolution times and 2 x bilinear, the obtained matrix map has the scale of 1/4 of the scale of the chest positive image, the initial map 1/16 with the scale being the scale of the chest positive image is up-sampled according to the first up-sampling parameters of 2 convolution times and 2 x bilinear, the obtained matrix map has the scale of 1/4 of the scale of the chest positive image, the initial map 1/8 with the scale being the scale of the chest positive image is up-sampled according to the first up-sampling parameters of 1 convolution time and 2 x bilinear, the obtained matrix map has the scale of 1/4 of the scale of the chest positive image, the initial map of 1/4 of the scale of the chest positive image is upsampled according to the first upsampling parameter of 0 convolution and 2 x bilinear, the obtained matrix map has the scale of 1/4 of the scale of the chest positive image, and the obtained matrix maps have the same scale and are all 1/4 of the scale of the chest positive image.

And S122, performing element summation on the matrix diagrams of the third quantity to obtain element diagrams.

In particular, the element summation may be adding numbers at the same position in the third number of matrix images. The embodiment of the invention can take the result of element summation of the third number of matrix diagrams as the element diagram. For ease of understanding, this is illustrated herein in connection with FIG. 5: if the third number of matrix maps are fig. 5(a) and 5(b), respectively, then the numbers at the same positions of fig. 5(a) and 5(b) are added, and the resulting elemental map can be as shown in fig. 5 (c). It is noted that the dimensions of the element map are the same as the dimensions of the matrix map. For ease of understanding, the dimensions of the matrix map and the element map shown in fig. 5 are both 3 × 3, and in practice, the dimensions of the element map may be 1/4 of the dimensions of the breast normal image.

And S123, performing upsampling on the element diagram according to the second upsampling parameter to obtain a characteristic diagram.

In particular, the second upsampling parameter may be determined from the scale of the chest ortho image and the scale of the elemental map. For example, when the scale of the element map is 1/4 of the scale of the chest normal image, the embodiment of the present invention may perform upsampling on the element map according to the second upsampling parameter of 1 convolution and 4 × bilinear to obtain the feature map with the same scale as the chest normal image.

S200, inputting the feature map into a preset cardiothoracic ratio key point determination model, and obtaining the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image.

The preset cardiothoracic ratio key point determination model can be a convolutional neural network model. The preset cardiothoracic ratio key point determination model can perform machine learning by using feature maps of the positions of the labeled cardiac edge key points and the positions of the labeled thoracic key points, so as to learn the image features of the positions of the cardiac edge key points and the positions of the thoracic key points. The embodiment of the invention can obtain a preset cardiothoracic ratio key point determination model according to the method.

As shown in fig. 6, a method for obtaining a cardiothoracic ratio key point determination model according to an embodiment of the present invention may include:

s010, obtaining a training initial image of at least one chest orthostatic training image under multiple scales, wherein the chest orthostatic training image is marked with the positions of heart edge key points and the positions of thoracic cavity key points;

s020, fusing the training initial images under the multiple scales to obtain a training characteristic image;

s030, performing machine learning according to the training feature diagram to obtain a heart-chest ratio key point determination model, wherein the input of the heart-chest ratio key point determination model is as follows: and (3) a training characteristic diagram obtained by fusing training initial diagrams under multiple scales, wherein the output of the cardiothoracic ratio key point determination model is as follows: the location probability of the cardiac edge keypoints and the location probability of the thoracic keypoints.

Specifically, the cardiothoracic ratio key point determination model performs machine learning on the training feature map, which may be performed by performing machine learning on image features that mark positions of the cardiac edge key points and positions of the thoracic key points in the training feature map of the chest orthostatic training image, where the image features may include image geometric features and gray value distribution. In actual use, the cardiothoracic ratio key point determination model can output the position probability of each cardiac edge key point and the position probability of each thoracic key point corresponding to the feature map of the chest orthostatic image according to the image features of the positions of the cardiac edge key points and the positions of the thoracic key points obtained by machine learning.

Optionally, based on the method shown in fig. 1, as shown in fig. 7, in another method for determining a cardiothoracic ratio provided in an embodiment of the present invention, step S200 may include:

s210, inputting the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter, and the position probability of a first number of heart edge key points and the position probability of a second number of thoracic key points are obtained.

In particular, the first convolution parameters may include a convolution kernel size and a number of output channels. The first number is equal to the number of output channels in the first convolution parameter. For example, if the number of output channels in the first convolution parameter is 4, the cardiothoracic ratio keypoint determination model convolves the feature map according to the first convolution parameter, and the total number of the position probabilities of the cardiac edge keypoints and the thoracic edge keypoints is 4, that is, the sum of the first number and the second number is the number of output channels in the first convolution parameter.

Normally, 2 cardiac-edge keypoints and 2 thoracic keypoints can be displayed in the chest ortho-position image, so that in an alternative embodiment of the present invention, the first number is 2, the second number is 2, and the first convolution parameter includes: the convolution kernel size is 3 × 3 and the number of output channels is 4.

The position probabilities of all the cardiac edge key points in the position probabilities of the cardiac edge key points of the first number are corresponding to one cardiac edge key point, the cardiac edge key points corresponding to the position probabilities of all the cardiac edge key points are different, the position probabilities of all the thoracic edge key points in the position probabilities of the thoracic edge key points of the second number are corresponding to one thoracic edge key point, and the thoracic edge key points corresponding to the position probabilities of all the thoracic edge key points are different.

Specifically, the position probability of each of the first number of the position probabilities of the heart edge key points is only a probability prediction of each position of one heart edge key point in the chest normal position image, where each position in the chest normal position image may be each pixel point in the chest normal position image. Therefore, when the first number is 2, the probability of the positions of the 2 heart edge key points can be respectively predicted according to the probability of the positions of the 2 heart edge key points in the chest normal position image. The position probability of each of the second number of thoracic keypoints is predicted only by the probability of each position of one thoracic keypoint in the thoracic orthotopic image, where each position in the thoracic orthotopic image may be each pixel point in the thoracic orthotopic image. Therefore, when the second number is 2, the probability of the positions of the 2 thoracic key points can be predicted respectively according to the probability of the positions of the 2 thoracic key points in the chest orthostatic image. For example, the position probability of the corresponding cardiac edge keypoint or the position probability of the corresponding thoracic keypoint in the chest ortho image may be a position probability map as shown in fig. 8, wherein the probability at each position in the position probability map shown in fig. 8 is the probability that the position is a position of a cardiac edge keypoint or a thoracic edge keypoint, and the probabilities at each position in the position probability map add up to 1. It should be noted that, for the convenience of understanding, the scale of the position probability map shown in fig. 8 is 5 × 5, and in practical use, the scale of the position probability map of the heart edge key points or the thorax key points may be the same as that of the chest orthotopic image.

Optionally, in the embodiment of the present invention, the process of convolving the feature map by the preset cardiothoracic ratio keypoint determination model according to a preset first convolution parameter to obtain the position probabilities of the first number of the cardiac edge keypoints and the second number of the thoracic key points may include:

convolving the feature map according to a preset first convolution parameter by using a preset cardiothoracic ratio key point determination model to obtain a position information map of a first number of heart edge key points and a second number of thoracic key points;

normalizing the position information graphs of the first number of the heart edge key points to obtain position information graphs of the first number of the heart edge key points;

and normalizing the position information maps of the second number of thoracic key points to obtain the position information maps of the second number of thoracic key points.

Optionally, in the embodiment of the present invention, the position information maps of the first number of cardiac edge key points and the position information maps of the second number of thoracic key points may be normalized by using a softmax function, so as to obtain the position information maps of the first number of cardiac edge key points and the position information maps of the second number of thoracic key points.

S300, determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points.

According to the embodiment of the invention, a coordinate axis can be set for the chest orthostatic image, and the cardio-thoracic ratio of the chest orthostatic image is determined according to the coordinates of each heart edge key point and each thoracic contour key point in the coordinate axis. For example, the coordinate axis may be as shown in fig. 9, with the top left corner of the chest orthostatic image as the origin, the horizontal axis as X, and the vertical axis as Y.

Based on the method shown in fig. 7, as shown in fig. 10, in another method for determining a cardiothoracic ratio according to an embodiment of the present invention, step S300 may include:

and S310, determining the transverse diameter of the heart according to the position probability of the first number of the heart edge key points.

Optionally, step S310 may include:

determining the positions of the 2 heart-edge key points according to the position probabilities of the 2 heart-edge key points;

determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart.

Specifically, the embodiment of the present invention may determine the coordinates of the positions of the 2 cardiac edge key points according to the position probabilities of the 2 cardiac edge key points, and calculate the transverse diameter of the heart according to the coordinates of the positions of the 2 cardiac edge key points. For example, based on the coordinate axes shown in fig. 9, when the coordinates of the position of the edge key point a are (X1, Y1) and the coordinates of the position of the edge key point B are (X2, Y2), the transverse diameter of the heart is an absolute value from X2 to X1. It is understood that the transverse diameter of the heart is the transverse distance of 2 key points of the heart edge, and the transverse diameter of the heart can have different algorithms according to different coordinate axes, and the invention is not further limited herein.

And S320, determining the transverse diameter of the thoracic cage according to the position probability of the second number of thoracic key points.

Optionally, step S320 may include:

determining the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points;

and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

Specifically, the embodiment of the present invention may determine the coordinates of the positions of 2 thoracic key points according to the position probabilities of the 2 thoracic key points, and calculate the transverse diameter of the thoracic cage according to the coordinates of the positions of the 2 thoracic key points. For example, based on the coordinate axes shown in fig. 9, if the coordinates of the position of thoracic key point C are (X3, Y3) and the coordinates of the position of thoracic key point D are (X4, Y4), the transverse diameter of the thoracic cage is the absolute value of X3 to X4. It is understood that the transverse diameter of the thorax is the transverse distance of 2 key points of the thorax. The cross-radius of the thorax may have different algorithms according to different coordinate axes, and the present invention is not limited herein.

And S330, dividing the heart transverse diameter by the thoracic transverse diameter to determine the cardiothoracic ratio in the chest orthostatic image.

For ease of understanding, the description is made herein by way of example: based on the coordinate axes shown in fig. 9, if the coordinates of the position of the heart edge key point a are (X1, Y1), the coordinates of the position of the heart edge key point B are (X2, Y2), the coordinates of the position of the thoracic key point C are (X3, Y3), and the coordinates of the position of the thoracic key point D are (X4, Y4), the determined cardiothoracic ratio is a quotient of the absolute value of X2-X1 divided by the absolute value of X3-X4.

According to the method for determining the cardiothoracic ratio, provided by the embodiment of the invention, a characteristic diagram of a chest orthostatic image can be obtained; inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image; and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points. According to the embodiment of the invention, the technical means that the model accurately positions the heart edge key points and the thoracic cavity key points is determined through the preset heart-chest ratio key points, the technical problem that the center edge key points and the thoracic cavity key points are not stably positioned in the prior art is solved, and the technical effect of accurately calculating the heart-chest ratio is achieved.

Optionally, after step S300, the embodiment of the present invention may further output a cardiothoracic ratio image labeling the positions of the cardiac edge key points and the thoracic contour key points on the chest ortho-position image. The cardiothoracic ratio image may be as shown in fig. 11.

Corresponding to the above method embodiment, an embodiment of the present invention further provides a cardiothoracic ratio determining apparatus, which may be configured as shown in fig. 12, and includes: a feature map obtaining unit 100, a location probability obtaining unit 200, and a cardiothoracic ratio determining unit 300.

The feature map obtaining unit 100 is configured to obtain a feature map of the chest orthophoria image.

Specifically, as shown in fig. 2, the chest ortho image may be a human chest ortho image obtained by applying Digital Radiography (DR) or Computed Radiography (CR) techniques to a detector using X-ray penetrating radiation. The chest orthostatic image may include posterior anterior (PA bit) and anterior posterior (AP bit). The PA position refers to a conventional positive position, namely, the user faces to the detector, and the chest is tightly attached to the detector to take a picture to obtain a chest positive position image; the AP position refers to an unconventional station position, namely that a user faces back to the detector, and the back is tightly attached to the detector to take a picture to obtain a chest normal position image. The detector comprises an X-ray machine. The scale may be the image size of the image at a certain resolution. The chest orthophoto scale can be the scale of the X-ray image output by the X-ray machine. For example, the resolution of the chest ortho image output by the X-ray machine may be 512 × 512.

Optionally, the feature map obtaining unit 100 may include: an initial graph obtaining subunit and a feature graph obtaining subunit.

The initial image obtaining subunit is used for obtaining initial images of the chest orthotopic image at a plurality of different scales.

The initial map obtaining subunit can obtain initial maps of the chest orthostatic image at a plurality of different scales through a feature extraction network. The feature extraction network may include convolutional neural networks including Resnext-50, Resnext-101, Resnext-152, and Densenet.

And the characteristic map obtaining subunit is used for fusing the initial maps under the different scales to obtain a characteristic map.

Specifically, the feature map obtaining subunit may perform feature extraction on the initial maps at a plurality of different scales through an image feature extraction network, and then perform comprehensive processing on information such as edges, shapes, contours, local features, and the like to obtain the feature map. The image Feature extraction network may include FPN (Feature Pyramid Networks).

Optionally, the feature map obtaining subunit may include: the matrix image acquisition unit comprises a matrix image acquisition subunit, an element image acquisition subunit and an up-sampling subunit.

The matrix map obtaining subunit is configured to perform upsampling on the initial maps at the multiple different scales according to a first upsampling parameter, and obtain a third number of matrix maps, where the scales of the third number of matrix maps are the same.

Specifically, the matrix map obtaining subunit may perform upsampling on the feature maps of the plurality of different scales according to different first upsampling parameters, so as to obtain a third number of matrix maps. Wherein the third number is related to the number of initial graphs.

The first upsampling parameter may be determined according to a scale of the initial map.

And the element map obtaining subunit is configured to perform element summation on the third number of matrix maps to obtain an element map.

In particular, the element summation may be adding numbers at the same position in the third number of matrix images. The element map obtaining subunit may obtain, as the element map, a result of element summing the third number of matrix maps.

And the up-sampling subunit is used for up-sampling the element diagram according to a second up-sampling parameter to obtain a characteristic diagram.

In particular, the second upsampling parameter may be determined from the scale of the chest ortho image and the scale of the elemental map.

The position probability obtaining unit 200 is configured to input the feature map into a preset cardiothoracic ratio key point determination model, and obtain a position probability of a cardiac edge key point on the chest orthostatic image and a position probability of a thoracic contour key point on the chest orthostatic image.

The preset cardiothoracic ratio key point determination model can be a convolutional neural network model. The preset cardiothoracic ratio key point determination model can perform machine learning by using feature maps of the positions of the labeled cardiac edge key points and the positions of the labeled thoracic key points, so as to learn the image features of the positions of the cardiac edge key points and the positions of the thoracic key points.

Optionally, the position probability obtaining unit 200 is specifically configured to input the feature map into a preset cardiothoracic ratio key point determination model, so that the preset cardiothoracic ratio key point determination model convolves the feature map according to a preset first convolution parameter, and obtains a first number of position probabilities of the cardiac edge key points and a second number of position probabilities of the thoracic edge key points.

In particular, the first convolution parameters may include a convolution kernel size and a number of output channels. The first number is equal to the number of output channels in the first convolution parameter.

Normally, 2 cardiac-edge keypoints and 2 thoracic keypoints can be displayed in the chest ortho-position image, so that in an alternative embodiment of the present invention, the first number is 2, the second number is 2, and the first convolution parameter includes: the convolution kernel size is 3 × 3 and the number of output channels is 4.

The position probabilities of all the cardiac edge key points in the position probabilities of the cardiac edge key points of the first number are corresponding to one cardiac edge key point, the cardiac edge key points corresponding to the position probabilities of all the cardiac edge key points are different, the position probabilities of all the thoracic edge key points in the position probabilities of the thoracic edge key points of the second number are corresponding to one thoracic edge key point, and the thoracic edge key points corresponding to the position probabilities of all the thoracic edge key points are different.

Specifically, the position probability of each of the first number of the position probabilities of the heart edge key points is only a probability prediction of each position of one heart edge key point in the chest normal position image, where each position in the chest normal position image may be each pixel point in the chest normal position image. Therefore, when the first number is 2, the probability of the positions of the 2 heart edge key points can be respectively predicted according to the probability of the positions of the 2 heart edge key points in the chest normal position image. The position probability of each of the second number of thoracic keypoints is predicted only by the probability of each position of one thoracic keypoint in the thoracic orthotopic image, where each position in the thoracic orthotopic image may be each pixel point in the thoracic orthotopic image. Therefore, when the second number is 2, the probability of the positions of the 2 thoracic key points can be predicted respectively according to the probability of the positions of the 2 thoracic key points in the chest orthostatic image.

Optionally, the position probability obtaining unit 200 may make the preset cardiothoracic ratio key point determination model perform convolution on the feature map according to a preset first convolution parameter, so as to obtain the position information maps of the first number of cardiac edge key points and the second number of thoracic key points; normalizing the position information graphs of the first number of the heart edge key points to obtain position information graphs of the first number of the heart edge key points; and normalizing the position information maps of the second number of thoracic key points to obtain the position information maps of the second number of thoracic key points.

Optionally, in the embodiment of the present invention, the position information maps of the first number of cardiac edge key points and the position information maps of the second number of thoracic key points may be normalized by using a softmax function, so as to obtain the position information maps of the first number of cardiac edge key points and the position information maps of the second number of thoracic key points.

The cardiothoracic ratio determining unit 300 is configured to determine the cardiothoracic ratio of the chest orthotopic image according to the position probability of the cardiac edge key points and the position probability of the thoracic key points.

According to the embodiment of the invention, a coordinate axis can be set for the chest orthostatic image, and the cardio-thoracic ratio of the chest orthostatic image is determined according to the coordinates of each heart edge key point and each thoracic contour key point in the coordinate axis.

Optionally, the cardiothoracic ratio determining unit 300 includes: the heart transverse diameter determining subunit, the thoracic transverse diameter determining subunit and the cardiothoracic ratio determining subunit.

And the heart transverse diameter determining subunit is used for determining the heart transverse diameter according to the position probability of the first number of the heart edge key points.

Optionally, the heart transverse diameter determination subunit is specifically configured to determine the positions of the 2 heart edge key points according to the position probabilities of the 2 heart edge key points; determining the distance between the positions of the 2 heart edge key points as the transverse diameter of the heart.

Specifically, the heart transverse diameter determining subunit may determine coordinates of the positions of the 2 cardiac edge key points according to the position probabilities of the 2 cardiac edge key points, and calculate the heart transverse diameter according to the coordinates of the positions of the 2 cardiac edge key points.

And the transverse diameter of the thoracic cage determining subunit is used for determining the transverse diameter of the thoracic cage according to the position probability of the second number of key points of the thoracic cage.

Optionally, the thoracic transverse diameter determining subunit is specifically configured to determine the positions of 2 thoracic key points according to the position probabilities of the 2 thoracic key points; and determining the distance between the positions of the 2 thoracic key points as the transverse diameter of the thoracic cage.

Specifically, the thoracic transverse diameter determining subunit may determine coordinates of the positions of the 2 thoracic key points according to the position probabilities of the 2 thoracic key points, and calculate the thoracic transverse diameter according to the coordinates of the positions of the 2 thoracic key points.

And the cardiothoracic ratio determining subunit is used for determining the quotient of the heart transverse diameter and the thoracic transverse diameter as the cardiothoracic ratio in the chest orthostatic image.

The device for determining the cardiothoracic ratio can obtain a characteristic diagram of a chest orthostatic image; inputting the feature map into a preset cardiothoracic ratio key point determination model to obtain the position probability of the heart edge key points on the chest orthostatic image and the position probability of the thoracic key points on the chest orthostatic image; and determining the cardiothoracic ratio of the chest orthostatic image according to the position probability of the heart edge key points and the position probability of the thoracic key points. According to the embodiment of the invention, the technical means that the model accurately positions the heart edge key points and the thoracic cavity key points is determined through the preset heart-chest ratio key points, the technical problem that the center edge key points and the thoracic cavity key points are not stably positioned in the prior art is solved, and the technical effect of accurately calculating the heart-chest ratio is achieved.

The storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the method for determining a cardiothoracic ratio is implemented.

The computer device provided by the embodiment of the invention comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor at least realizes the cardiothoracic ratio determination method according to any one of the above items when executing the program.

Optionally, as shown in fig. 13 to fig. 15, a cardiothoracic ratio determining apparatus provided in an embodiment of the present invention may include: a chest positive image receiving device, a processor 50, a memory 60, a communication bus 70 and an output device, the memory 60 having stored thereon a program operable on the processor 50,

the processor 50 is communicatively connected to the chest ortho image receiving device, the memory 60 and the output device, respectively, via the communication bus 70;

the chest positive image receiving device receives a chest positive image;

the processor 50, when executing a program, at least implements the cardiothoracic ratio determination method as defined in any of the above;

the output device obtains and outputs the cardiothoracic ratio determined by the processor 50.

Fig. 15 is a schematic diagram illustrating a connection between the processor 50, the memory 60, and the communication bus 70 according to an embodiment of the present invention.

Wherein, as shown in fig. 13 to 14, the chest normal position image receiving apparatus may include: a chest radiography housing 10 and an operating mechanism, and the chest orthopaedics image receiving device can also comprise a scanning device or a photographing device. The operating mechanism of the embodiment of the present invention may first drive the chest film holding box 10 to extend from the cardiothoracic ratio determining apparatus. Specifically, the cardiothoracic ratio determining apparatus may be provided with an access 20, and the chest film cassette 10 may be extended or retracted from the access 20. After the chest radiography housing 10 is extended from the entrance 20 of the cardiothoracic ratio determining apparatus, the user can first put the chest orthophoto image into the chest radiography housing 10. The processor 50 then retracts the chest film cassette 10 back into the cardiothoracic ratio determination apparatus by controlling the operating mechanism. Then the scanning device or the photographing device can scan or photograph the chest positive position image in the chest radiography housing 10, so as to obtain the chest positive position image and send the chest positive position image to the processor 50, and after the processor 50 performs the cardiothoracic ratio determination method as described in any one of the above items on the chest positive position image, the cardiothoracic ratio of the chest positive position image is output through the output device.

Alternatively, the output device may be a display screen 30 and/or a printer.

Optionally, the cardiothoracic ratio determining device may further include an input device, wherein the input device may be a key. It will be appreciated that the input device may also be a touch screen.

The user may control the cardiothoracic ratio determining device by means of a key or a touch screen, for example: the user first controls the apparatus to extend the chest radiography housing 10 and, after placing the chest orthostatic image, controls the chest radiography housing 10 to retract and perform the process according to any of the methods described above.

Wherein, the output heart-chest ratio can be automatically output and can also be output according to the operation of the user. For example, the result of printing is output from a printer, which may be provided inside the cardiothoracic ratio determining apparatus and output printing paper through a printing paper output port 40 as shown in fig. 13 to 14.

Of course, the cardiothoracic ratio determining device may also comprise a data interface to obtain the chest ortho image from other devices via the data interface. The data interface may be a USB interface, a bluetooth communication interface, a Wi-Fi communication interface, or the like, which is not limited in the present invention.

The cardiothoracic ratio determination apparatus comprises a processor and a memory, wherein the above feature map obtaining unit 100, the position probability obtaining unit 200, the cardiothoracic ratio determination unit 300, and the like are stored in the memory as program units, and the processor executes the above program units stored in the memory to realize corresponding functions.

The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more than one kernel can be set, and the kernel parameters are adjusted to accurately position the edge key points and the thorax key points.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.

An embodiment of the present invention provides a storage medium having a program stored thereon, which when executed by a processor, implements a cardiothoracic ratio determination method as described in any one of the above.

An embodiment of the present invention provides a processor, where the processor is configured to execute a program, where the program executes the method for determining a cardiothoracic ratio as described in any one of the above.

An embodiment of the present invention provides an apparatus, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, the steps of the cardiothoracic ratio determination method as described in any one of the above are implemented.

The device herein may be a server, a PC, a PAD, a mobile phone, etc.

The present application further provides a computer program product adapted to perform a program initialized with the cardiothoracic ratio determination method steps when executed on a data processing device.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.