阅读说明：本技术 板层角膜移植ar手术导航系统及方法 (Lamellar cornea transplantation AR operation navigation system and method ) 是由 潘俊君 李方红 葛璞 于 2019-10-30 设计创作，主要内容包括：本发明提供一种板层角膜移植AR手术导航系统及方法。该板层角膜移植AR手术导航系统包括依次连接的摄像单元、处理单元以及AR单元；摄像单元用于获取板层角膜移植手术过程中的视频影像；处理单元用于对视频影像中的每一帧图像中的角膜轮廓进行分割,得到多帧角膜轮廓图像,将多帧角膜轮廓图像划分为多个关键帧角膜轮廓图像与多个遮挡帧角膜轮廓图像,当当前帧角膜轮廓图像为遮挡帧角膜轮廓图像时,恢复当前帧角膜轮廓图像被遮挡的运动信息并对关键帧角膜轮廓图像进行变形以重建当前帧角膜轮廓图像,再控制AR单元在当前帧角膜轮廓图像中叠加一标记以标注出角膜轮廓以及多个角膜缝针位置,从而指导医生及时准确的找到角膜缝针位置,提高手术成功率。(The invention provides a navigation system and a navigation method for an AR operation of lamellar cornea transplantation. The lamellar cornea transplantation AR operation navigation system comprises a camera shooting unit, a processing unit and an AR unit which are sequentially connected; the camera shooting unit is used for acquiring a video image in the lamellar cornea transplantation operation process; the processing unit is used for segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images, dividing the plurality of frames of cornea contour images into a plurality of key frame cornea contour images and a plurality of shielding frame cornea contour images, when the current frame cornea contour images are the shielding frame cornea contour images, recovering the shielded motion information of the current frame cornea contour images and deforming the key frame cornea contour images to reconstruct the current frame cornea contour images, and then controlling the AR unit to superpose a mark in the current frame cornea contour images to mark the cornea contour and the positions of a plurality of cornea sewing needles, so that a doctor is guided to find the positions of the cornea sewing needles timely and accurately, and the success rate of the operation is improved.)

1. A lamellar cornea transplantation AR operation navigation system, characterized by comprising: an imaging unit (10), a processing unit (20) connected to the imaging unit (10), and an AR unit (30) connected to the processing unit (20);

the AR unit (30) is used for overlaying a mark in the current frame cornea contour image through AR so as to mark out a cornea contour and a plurality of cornea suture needle positions;

the camera unit (10) is used for acquiring a video image in the lamellar cornea transplantation operation process;

the processing unit (20) is used for segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images, dividing the plurality of frames of cornea contour images into a plurality of key frame cornea contour images and a plurality of shielding frame cornea contour images, wherein the key frame cornea contour images are completely visible images of the cornea contour, and the shielding frame cornea contour images are images of the cornea contour which are shielded,

when the current frame cornea contour image is a shielding frame cornea contour image, restoring lost motion information due to shielding between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow supplement network, deforming the key frame cornea contour image through a supplement optical flow output by the optical flow supplement network to reconstruct the current frame cornea contour image, and then controlling an AR unit (30) to superpose a mark in the current frame cornea contour image through AR so as to mark out a cornea contour and a plurality of cornea sewing needle positions;

when the current frame cornea contour image is the key frame cornea contour image, the AR unit (30) is controlled to superpose a mark in the current frame cornea contour image through AR so as to mark out the cornea contour and a plurality of cornea sewing needle positions.

2. The lamellar cornea transplant AR surgical navigation system of claim 1, characterized in that the processing unit (20) segments the corneal profile by a U-Net segmentation network.

3. The AR surgical navigation system for lamellar cornea transplantation as in claim 1, characterized in that the specific steps of the processing unit (20) dividing the corneal profile image are: calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the segmentation of each frame of cornea contour image₀(x₀，y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁，n₂，n₃，n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

judging whether the cornea contour image is a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, wherein the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}Is the threshold value of the symmetry proportion of the key frame cornea contour image.

4. The lamellar cornea transplant AR surgical navigation system of claim 1, characterized in that the processing unit (20) deforms the key frame corneal profile image by a deformation formula:

whereinI and j are respectively the abscissa and ordinate of a predetermined position in the image, y_ijIs the value after the deformation of the optical flow,andfor the two-channel optical flow value at the preset position, k is a linear interpolation kernel function, and m and v are constants.

5. The lamellar cornea transplantation AR surgical navigation system of claim 1, characterized in that the processing unit (20) further reconstructs a current frame cornea contour image by a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is:wherein, I_keyFor the key frame corneal profile image adjacent to the current frame corneal profile image,to supplement the optical flow complemented by the optical flow complementing network,is the true value of the optical flow,is I_keyAlong the edgeThe alignment operation with the current frame cornea contour image,is I_keyAlong the edgeAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t，keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,a mask is calculated for the extended space, an indication of a dot product operation.

6. A navigation method for an AR operation of lamellar cornea transplantation is characterized by comprising the following steps:

step S1, acquiring a video image in the lamellar cornea transplantation operation process, and segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images;

step S2, dividing the multi-frame cornea contour image into a plurality of key frame cornea contour images and a plurality of occlusion frame cornea contour images; the key frame cornea contour image is an image in which a cornea contour is completely visible; the cornea contour image of the occlusion frame is an image of the occluded cornea contour;

step S3, when the current frame cornea contour image is a sheltered frame cornea contour image, restoring the motion information lost due to sheltering between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow mending network, and deforming the key frame cornea contour image through the mending optical flow output by the optical flow mending network so as to reconstruct the current frame cornea contour image;

when the current frame cornea contour image is the key frame cornea contour image, performing step S4;

step S4, a mark is superimposed on the current frame corneal profile image by AR to mark out the corneal profile and the positions of the plurality of corneal stitches.

7. The method for navigating a lamellar cornea transplantation AR surgery of claim 6, wherein in step S1, the corneal contour is segmented by a U-Net segmentation network.

8. The navigation method for lamellar cornea transplantation AR surgery of claim 6, wherein the specific step of dividing the cornea contour image in step S2 is: step S21, calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the division position of each frame of cornea contour image₀(x₀，y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

step S22, respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁，n₂，n₃，n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

step S23, the cornea contour image is judged to be a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, and the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}Is the threshold value of the symmetry proportion of the key frame cornea contour image.

9. The AR surgical navigation method for lamellar cornea transplantation of claim 6, wherein in step S3, the keyframe corneal profile image is deformed by a deformation formula:

wherein i and j are respectively the abscissa and ordinate of a preset position in the image, y_ijIs the value after the deformation of the optical flow,andfor the two-channel optical flow value at the preset position, k is a linear interpolation kernel function, and m and v are constants.

10. The method for navigating the AR surgery for lamellar keratoplasty of claim 6, wherein in step S3, the current frame corneal profile image is further reconstructed by a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is:wherein, I_keyFor the key frame corneal profile image adjacent to the current frame corneal profile image,to supplement the optical flow complemented by the optical flow complementing network,is the true value of the optical flow,is I_keyAlong the edgeThe alignment operation with the current frame cornea contour image,is I_keyAlong the edgeAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t，keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,a mask is calculated for the extended space, an indication of a dot product operation.

Technical Field

The invention relates to the field of ophthalmic surgery, in particular to a navigation system and a navigation method for lamella corneal transplantation AR surgery.

Background

Corneal disease is one of the most important blinding eye diseases worldwide, and the incidence rate in China is second to cataract and is about 11.5/10000. With the development of technology, Deep lamellar keratoplasty (DALK) is the only effective treatment means for infectious corneal ulcer and other diseases that cannot be treated by drugs, lamellar keratoplasty is an operation performed with partial tissues of the cornea as the operation object, only the superficial tissues of the cornea with pathological changes are excised, the recipient cornea with a relatively intact Deep layer is still kept as a transplant bed, then corneal materials with the same size and thickness are taken as superficial corneal lamellae and sewn on the wound surface of the cornea of a patient, and the lamellar keratoplasty does not disturb the intraocular tissues, so that the lamellar keratoplasty can effectively reduce the risk of endothelial rejection and reduce the occurrence of surgical complications such as astigmatism.

However, the lamellar cornea transplantation operation has strict requirements on the technique and experience of doctors, and as the lamellar cornea transplantation operation has complex scene, long operation time, great interference caused by blood pollution and shielding of surgical instruments in the operation process and quite fuzzy outline boundary of the cornea, the position of a cornea suture needle is difficult to position in the lamellar cornea transplantation process, the distribution of the cornea suture position is uneven, the stress of the transplanted cornea is uneven, and various postoperative complications are caused, so that serious shortage of trained medical personnel and cornea donors easily causes corneal diseases to be difficult to be timely and effectively treated, and serious consequences such as blindness are caused.

Augmented Reality (AR) is a new technology that fuses computer-generated virtual graphics into the real-world view seen by a user, who can obtain additional information from a simulation model depicted by a computer, thereby augmenting the real environment. Augmented reality has three major characteristics: virtual combination, real-time interaction and three-dimensional matching. The operation navigation is to accurately correspond the image data before or during the operation of the patient to the anatomical structure of the patient on the operation bed, track the surgical instrument during the operation and update and display the position of the surgical instrument on the image of the patient in a virtual probe mode in real time, so that a doctor can clearly know the position of the surgical instrument relative to the anatomical structure of the patient, and the surgical operation is quicker, more accurate and safer. Augmented Reality surgical Navigation (Augmented Reality Navigation) is to fuse virtual information into a real surgical environment, so that a doctor can directly see a scene after registration and superposition, real-time surgical guidance is provided for the doctor, an accurate decision is made in time, and surgical risks are reduced.

Disclosure of Invention

The invention aims to provide a lamina corneal transplantation AR surgery navigation system which can track the corneal contour in real time, guide doctors to find the position of a corneal suture needle timely and accurately and improve the success rate of surgery.

The invention also aims to provide a navigation method for the lamella corneal transplantation AR operation, which can track the corneal contour in real time, guide doctors to find the position of a corneal suture needle timely and accurately and improve the success rate of the operation.

In order to achieve the above object, the present invention provides a lamellar cornea transplantation AR surgery navigation system, comprising: the system comprises an image pickup unit, a processing unit connected with the image pickup unit and an AR unit connected with the processing unit;

the AR unit is used for overlaying a mark in the current frame cornea contour image through AR so as to mark out a cornea contour and a plurality of cornea sewing needle positions;

the camera shooting unit is used for acquiring a video image in the lamellar cornea transplantation operation process;

the processing unit is used for segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images, dividing the plurality of frames of cornea contour images into a plurality of key frame cornea contour images and a plurality of shielding frame cornea contour images, wherein the key frame cornea contour images are images with completely visible cornea contours, the shielding frame cornea contour images are images with shielded cornea contours,

when the current frame cornea contour image is a shielding frame cornea contour image, restoring lost motion information due to shielding between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow supplement network, deforming the key frame cornea contour image through a supplement optical flow output by the optical flow supplement network to reconstruct the current frame cornea contour image, and controlling an AR unit to superpose a mark in the current frame cornea contour image through AR so as to mark out a cornea contour and a plurality of cornea sewing needle positions;

when the current frame cornea contour image is the key frame cornea contour image, the AR unit is controlled to superpose a mark in the current frame cornea contour image through AR so as to mark out the cornea contour and a plurality of cornea sewing needle positions.

The processing unit segments the corneal profile through a U-Net segmentation network.

The specific steps of dividing the cornea contour image by the processing unit are as follows: calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the segmentation of each frame of cornea contour image₀(x₀，y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁，n₂，n₃，n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

judging whether the cornea contour image is a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, wherein the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}Is the threshold value of the symmetry proportion of the key frame cornea contour image.

The processing unit deforms the key frame cornea contour image through a deformation formula, wherein the deformation formula is as follows:

wherein i and j are respectively the abscissa and ordinate of a preset position in the image, y_ijIs the value after the deformation of the optical flow,andfor the two-channel optical flow value at the preset position, k is a linear interpolation kernel function, and m and v are constants.

The processing unit is also used for reconstructing a current frame cornea contour image through a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is:wherein, I_keyFor the key frame corneal profile image adjacent to the current frame corneal profile image,to supplement the optical flow complemented by the optical flow complementing network,is the true value of the optical flow,is I_keyRadix astragaliThe alignment operation with the current frame cornea contour image,is I_keyRadix astragaliAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t，keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,a mask is calculated for the extended space, an indication of a dot product operation.

The invention also provides an AR surgery navigation method for lamellar cornea transplantation, which comprises the following steps:

step S1, acquiring a video image in the lamellar cornea transplantation operation process, and segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images;

step S2, dividing the multi-frame cornea contour image into a plurality of key frame cornea contour images and a plurality of occlusion frame cornea contour images; the key frame cornea contour image is an image in which a cornea contour is completely visible; the cornea contour image of the occlusion frame is an image of the occluded cornea contour;

step S3, when the current frame cornea contour image is a sheltered frame cornea contour image, restoring the motion information lost due to sheltering between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow mending network, and deforming the key frame cornea contour image through the mending optical flow output by the optical flow mending network so as to reconstruct the current frame cornea contour image;

when the current frame cornea contour image is the key frame cornea contour image, performing step S4;

step S4, a mark is superimposed on the current frame corneal profile image by AR to mark out the corneal profile and the positions of the plurality of corneal stitches.

In step S1, the corneal contour is segmented by a U-Net segmentation network.

The specific steps of dividing the cornea contour image in step S2 are as follows: step S21, calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the division position of each frame of cornea contour image₀(x₀，y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

step S22, respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁，n₂，n₃，n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

step S23, the cornea contour image is judged to be a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, and the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}Is the threshold value of the symmetry proportion of the key frame cornea contour image.

In step S3, the keyframe corneal profile image is deformed by a deformation formula, where the deformation formula is:

wherein i and j are respectively the abscissa and ordinate of a preset position in the image, y_ijIs the value after the deformation of the optical flow,andfor the two-channel optical flow value at the preset position, k is a linear interpolation kernel function, and m and v are constants.

In step S3, reconstructing a current frame cornea contour image by a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is:wherein, I_keyFor the key frame corneal profile image adjacent to the current frame corneal profile image,to supplement the optical flow complemented by the optical flow complementing network,is the true value of the optical flow,is I_keyAlong the edgeThe alignment operation with the current frame cornea contour image,is I_keyAlong the edgeAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t，keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,a mask is calculated for the extended space, an indication of a dot product operation.

The invention has the beneficial effects that: the invention relates to a lamellar cornea transplantation AR operation navigation system, which comprises: the system comprises an image pickup unit, a processing unit connected with the image pickup unit and an AR unit connected with the processing unit; the camera shooting unit is used for acquiring a video image in the lamellar cornea transplantation operation process; the processing unit is used for segmenting a cornea contour in each frame of image in a video image to obtain a plurality of frames of cornea contour images, dividing the plurality of frames of cornea contour images into a plurality of key frame cornea contour images and a plurality of shielding frame cornea contour images, wherein the key frame cornea contour images are images with completely visible cornea contours, the shielding frame cornea contour images are images with shielded cornea contours, when the current frame cornea contour images are shielding frame cornea contour images, the optical flow complementing network restores motion information lost due to shielding between the current frame cornea contour images and one key frame cornea contour image adjacent to the current frame cornea contour images, the key frame cornea contour images are deformed through complementing optical flows output by the optical flow complementing network to reconstruct the current frame cornea contour images, and then the AR unit is controlled to superimpose a mark in the current frame cornea contour images through AR to mark out the cornea contours and a plurality of sewing needles A location; when the current frame cornea contour image is the key frame cornea contour image, the AR unit is controlled to superpose a mark in the current frame cornea contour image through the AR so as to mark out the cornea contour and a plurality of cornea sewing needle positions, so that the cornea contour can be tracked in real time, a doctor is guided to find out the cornea sewing needle positions timely and accurately, and the operation success rate is improved. The navigation method for the laminated cornea transplantation AR operation can track the cornea outline in real time, guides doctors to find the position of a cornea suture needle timely and accurately, and improves the success rate of the operation.

Drawings

For a better understanding of the nature and technical aspects of the present invention, reference should be made to the following detailed description of the invention, taken in conjunction with the accompanying drawings, which are provided for purposes of illustration and description and are not intended to limit the invention.

In the drawings, there is shown in the drawings,

FIG. 1 is a schematic diagram of the lamellar cornea grafting AR surgery navigation system of the present invention;

FIG. 2 is a flow chart of the navigation method of the lamellar cornea transplantation AR operation of the invention.

Detailed Description

To further illustrate the technical means and effects of the present invention, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings.

Referring to fig. 1, the present invention provides a lamellar cornea transplantation AR operation navigation system, comprising: an imaging unit 10, a processing unit 20 connected to the imaging unit 10, and an AR unit 30 connected to the processing unit 20;

the AR unit 30 is used for overlaying a mark in the current frame cornea contour image through AR to mark out the cornea contour and a plurality of cornea sewing needle positions;

the camera unit 10 is used for acquiring a video image in the lamellar cornea transplantation operation process;

the processing unit 20 is configured to segment the corneal contour in each frame of image in the video image to obtain a plurality of frames of corneal contour images, divide the plurality of frames of corneal contour images into a plurality of key frame corneal contour images and a plurality of occlusion frame corneal contour images, where the key frame corneal contour images are images in which the corneal contour is completely visible, and the occlusion frame corneal contour images are images in which the corneal contour is occluded,

when the current frame cornea contour image is the occlusion frame cornea contour image, restoring lost motion information due to occlusion between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow supplementation network, deforming the key frame cornea contour image through supplementation optical flow output by the optical flow supplementation network to reconstruct the current frame cornea contour image, and then controlling an AR unit 30 to superimpose a mark in the current frame cornea contour image through AR to mark out a cornea contour and a plurality of cornea sewing needle positions;

when the current frame corneal profile image is the key frame corneal profile image, the AR unit 30 is controlled to superimpose a mark to mark out the corneal profile and the plurality of corneal stitch positions in the current frame corneal profile image by AR.

It should be noted that, because the lamellar cornea transplantation operation scene is relatively complex, the main interference factor is blood pollution and the shielding of operation equipment, the invention divides the cornea contour in each frame image in the video image, divides the cornea contour image polluted by blood and the cornea contour image shielded by operation equipment into shielding frame cornea contour images, divides the completely visible image of the cornea contour into key frame cornea contour images, when the current frame cornea contour image is the shielding frame cornea contour image, restores the lost motion information between the current frame cornea contour image and the key frame cornea contour image adjacent to the current frame cornea contour image through the optical flow complementing network, and deforms the key frame cornea contour image through the complementing optical flow output by the optical flow complementing network to reconstruct the current frame cornea contour image, a mark is superposed in the current frame cornea contour image through the AR to mark out the cornea contour and the positions of a plurality of cornea sewing needles, so that the cornea contour can be tracked in real time, a doctor is guided to find out the positions of the cornea sewing needles timely and accurately, and the success rate of the operation is improved.

Specifically, the processing unit 20 segments the corneal profile through a U-Net segmentation network. The U-Net is a network specially used for processing medical image segmentation, the medical image processing has the characteristics of small sample, high positioning precision requirement and the like, and the U-Net can realize the precise positioning of a target area and the fusion of multi-scale context information under the condition of less training data and reserve richer detail information.

Specifically, the specific steps of the processing unit 20 dividing the cornea contour image are as follows: calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the segmentation of each frame of cornea contour image₀(x₀，y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁，n₂，n₃，n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

judging whether the cornea contour image is a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, wherein the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}A threshold value of the symmetry proportion of the key frame cornea contour image; therefore, the invention can screen out the image with the completely visible cornea contour as the key frame cornea contour image, and simultaneously screen out the image with the relatively serious cornea contour shielding degree (larger than a threshold value) as the shielding frame cornea contour image.

Specifically, a key frame corneal contour image adjacent to the current frame corneal contour image refers to a key frame corneal contour image closest to the current frame corneal contour image in the video image time sequence.

Specifically, the optical flow completion network is composed of an encoding network and a decoding network, the middle layer uses a hole convolution to increase the receptive field, and information with a larger receptive field is obtained as far as possible on the premise of not influencing time complexity. The optical flow completion network only carries out down sampling twice, changes the scale of the image into one fourth of the original scale, can keep richer detail information, and ensures that the completed optical flow is clearer as much as possible. The input of the optical flow completion network is a two-channel optical flow data and a mask (SDM) (the value of a part to be reconstructed is 1, and the value of the other part is 0), and the output of the network is a completed two-channel optical flow.

Specifically, the processing unit 20 deforms the key frame corneal profile image by a deformation formula, where the deformation formula is:

wherein i and j are respectively the abscissa and ordinate of a preset position in the image, y_ijIs the value after optical flow deformation (i.e. optical flow deformation from the preset position of the key frame cornea contour image to the same position of the occlusion frame cornea contour image),andfor two channel luminous flux values at preset positions (i, j), k is a linear interpolation kernel function, and m and v are constants.

Further, k may be defined as k (p, q) ═ max (0,1- | p |) max (0,1- | q |), and p and q may be specifically selected according to actual conditions.

Specifically, the processing unit 20 further reconstructs a current frame cornea contour image through a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is;wherein, I_keyThe key frame cornea contour image adjacent to the current frame cornea contour image, that is, the key frame cornea contour image closest to the current frame cornea contour image in time series,to supplement the optical flow complemented by the optical flow complementing network,the true value of the optical flow refers to the original non-occlusion time and I of the current frame cornea contour image^keyThe flow of light in between the two,is I_keyAlong the edgeThe alignment operation with the current frame cornea contour image,is I_keyAlong the edgeAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t，keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,for expanding a spatial computation mask (ESDM) (an original mask has a value of 1 only in a portion to be reconstructed, and values near a boundary are large and values near a center are small when values of other portions are 0, and an expanded spatial computation mask is obtained by slightly enlarging a non-zero region based on the original mask, and values of enlarged elements are obtained by using y^dWhere d is a distance between the current pixel and the boundary of the region to be compensated, γ is a constant less than 1), indicating a dot product operation.

Specifically, the processing unit 20 performs ellipse fitting on a foreground point set obtained from a segmentation result of a current frame corneal profile image, controls the AR unit 30 to mark out a corneal profile by using the fitted ellipse as a mark, and performs equal division and uniform sampling on the fitted ellipse to mark out a plurality of uniformly distributed corneal suture needle positions.

Further, the number of corneal stitch locations may be 32.

Specifically, the processing unit 20 further determines the accuracy of the superposition of the fitted ellipse and the corneal profile through a corneal profile tracking formula; the cornea contour tracing formula is as follows:

wherein, G and F are respectively a true value ellipse and a uniform sampling point on a fitting ellipse, d (G, F) is the distance between two points of G and F, and belongs to_ellipseAs the superposition accuracy of the fitted ellipse and the corneal contour, belongs to_ellibseThe lower the stack the more accurate.

Referring to fig. 2, based on the navigation system for AR surgery of lamellar keratoplasty, the present invention further provides a navigation method for AR surgery of lamellar keratoplasty, comprising the following steps:

step S1, acquiring a video image in the lamellar cornea transplantation operation process, and segmenting the cornea contour in each frame of image in the video image to obtain a plurality of frames of cornea contour images;

step S2, dividing the multi-frame cornea contour image into a plurality of key frame cornea contour images and a plurality of occlusion frame cornea contour images; the key frame cornea contour image is an image in which a cornea contour is completely visible; the cornea contour image of the occlusion frame is an image of the occluded cornea contour;

step S3, when the current frame cornea contour image is a sheltered frame cornea contour image, restoring the motion information lost due to sheltering between the current frame cornea contour image and a key frame cornea contour image adjacent to the current frame cornea contour image through an optical flow mending network, and deforming the key frame cornea contour image through the mending optical flow output by the optical flow mending network so as to reconstruct the current frame cornea contour image;

when the current frame cornea contour image is the key frame cornea contour image, performing step S4;

step S4, a mark is superimposed on the current frame corneal profile image by AR to mark out the corneal profile and the positions of the plurality of corneal stitches.

It should be noted that, because the lamellar cornea transplantation operation scene is relatively complex, the main interference factor is blood pollution and the shielding of operation equipment, the invention divides the cornea contour in each frame image in the video image, divides the cornea contour image polluted by blood and the cornea contour image shielded by operation equipment into shielding frame cornea contour images, divides the completely visible image of the cornea contour into key frame cornea contour images, when the current frame cornea contour image is the shielding frame cornea contour image, restores the lost motion information between the current frame cornea contour image and the key frame cornea contour image adjacent to the current frame cornea contour image through the optical flow complementing network, and deforms the key frame cornea contour image through the complementing optical flow output by the optical flow complementing network to reconstruct the current frame cornea contour image, a mark is superposed in the current frame cornea contour image through the AR to mark out the cornea contour and the positions of a plurality of cornea sewing needles, so that the cornea contour can be tracked in real time, a doctor is guided to find out the positions of the cornea sewing needles timely and accurately, and the success rate of the operation is improved.

Specifically, in step S1, the corneal contour is segmented by a U-Net segmentation network. The U-Net is a network specially used for processing medical image segmentation, the medical image processing has the characteristics of small sample, high positioning precision requirement and the like, and the U-Net can realize the precise positioning of a target area and the fusion of multi-scale context information under the condition of less training data and reserve richer detail information.

Specifically, the step S2 of dividing the corneal profile image includes the specific steps of: step S21, calculating the center point coordinate p of each frame of cornea contour image through the cornea contour foreground point at the division position of each frame of cornea contour image₀(x₀,y₀)，x₀And y₀Is defined as

Wherein, P is a point set of the cornea contour obtained by segmentation, and n is the number of the point set;

step S22, respectively counting the central points p₀Number of foreground points (n) in the four quadrants of (1)₁,n₂,n₃,n₄) And calculating the symmetric proportion of the four quadrants:

wherein n is₁Is the number of foreground points in the first quadrant, n₂Is the number of foreground points in the second quadrant, n₃Is the number of foreground points in the third quadrant, n₄Is the number of foreground points in the fourth quadrant, r_LRIs the bilateral symmetry proportion of four quadrants, r_UDIs the up-down symmetrical proportion of four quadrants;

step S23, the cornea contour image is judged to be a key frame cornea contour image or a shielding frame cornea contour image through a screening formula, and the screening formula is as follows:

wherein, I_tIs an image of the contour of the cornea,in order to mask the frame of the corneal profile image,for the key frame corneal profile image, n_thresIs a threshold value of the number of point sets, r_{thres_occ}Threshold value r for the symmetry ratio of the corneal profile image of the occlusion frame_{thres_key}A threshold value of the symmetry proportion of the key frame cornea contour image; therefore, the invention can screen out the image with the completely visible cornea contour as the key frame cornea contour image, and simultaneously screen out the image with the relatively serious cornea contour shielding degree (larger than a threshold value) as the shielding frame cornea contour image.

Specifically, in step S3, the key frame corneal contour image adjacent to the current frame corneal contour image is a key frame corneal contour image closest to the current frame corneal contour image in the video image time sequence.

Specifically, the optical flow completion network is composed of an encoding network and a decoding network, the middle layer uses a hole convolution to increase the receptive field, and information with a larger receptive field is obtained as far as possible on the premise of not influencing time complexity. The optical flow completion network only carries out down sampling twice, changes the scale of the image into one fourth of the original scale, can keep richer detail information, and ensures that the completed optical flow is clearer as much as possible. The input of the optical flow completion network is a two-channel optical flow data and a mask (SDM) (the value of a part to be reconstructed is 1, and the value of the other part is 0), and the output of the network is a completed two-channel optical flow.

Specifically, in step S3, the keyframe corneal contour image is deformed by a deformation formula, where the deformation formula is:

wherein i and j are respectively the abscissa and ordinate of a preset position in the image, y_ijIs the value after optical flow deformation (i.e. optical flow deformation from the preset position of the key frame cornea contour image to the same position of the occlusion frame cornea contour image),andfor two channel luminous flux values at preset positions (i, j), k is a linear interpolation kernel function, and m and v are constants.

Further, k may be defined as k (p, q) ═ max (0,1- | p |) max (0,1- | q |), and p and q may be specifically selected according to actual conditions.

Specifically, in step S3, the current frame corneal contour image is reconstructed by using a joint loss function; the joint loss function comprises a deformation loss function and an extended space calculation loss function;

the joint loss function is: l ═ α L_warp+(1-α)L_esdlWherein L is a joint loss function, I_warpAs a function of deformation loss, L_esdlCalculating a loss function for the extended space, alpha being greater than 0 and less than 1;

the deformation loss function is:wherein, I_keyThe key frame cornea contour image adjacent to the current frame cornea contour image, that is, the key frame cornea contour image closest to the current frame cornea contour image in time series,to supplement the optical flow complemented by the optical flow complementing network,the true value of the optical flow refers to the original non-occlusion time and I of the current frame cornea contour image^keyThe flow of light in between the two,is I_keyAlong the edgeThe alignment operation with the current frame cornea contour image,is I_keyAlong the edgeAligning the current frame cornea contour image with the current frame cornea contour image;

the extended space computation loss function is:wherein f is_t,keyCorneal profile image and I for the current frame generated by FlowNet2 network_keyThe flow of light in between the two,to expand the spatial computation mask (ESDM) (the original mask has a value of 1 only in the portion to be reconstructed, and values near the boundary are large and values near the center are small when values of other portions are 0,the expanded space calculation mask is obtained by slightly enlarging non-zero region based on original mask, and the value of enlarged element is determined by gamma^dWhere d is a distance between the current pixel and the boundary of the region to be compensated, γ is a constant less than 1), indicating a dot product operation.

Specifically, in step S4, ellipse fitting is performed on the foreground point set obtained from the segmentation result of the current frame corneal profile image, the fitted ellipse is used as a mark to mark out the corneal profile, and the fitted ellipse is equally divided and uniformly sampled to mark out a plurality of uniformly distributed corneal suture needle positions.

Further, the number of corneal stitch locations may be 32.

Specifically, in step S4, the accuracy of superimposing the fitted ellipse on the corneal contour is further determined by a corneal contour tracing formula; the cornea contour tracing formula is as follows:

wherein, G and F are respectively a true value ellipse and a uniform sampling point on a fitting ellipse, d (G, F) is the distance between two points of G and F, and belongs to_ellipseAs the superposition accuracy of the fitted ellipse and the corneal contour, belongs to_ellipseThe lower the stack the more accurate.

In summary, the lamellar cornea transplantation AR surgical navigation system of the present invention includes: the system comprises an image pickup unit, a processing unit connected with the image pickup unit and an AR unit connected with the processing unit; the camera shooting unit is used for acquiring a video image in the lamellar cornea transplantation operation process; the processing unit is used for segmenting a cornea contour in each frame of image in a video image to obtain a plurality of frames of cornea contour images, dividing the plurality of frames of cornea contour images into a plurality of key frame cornea contour images and a plurality of shielding frame cornea contour images, wherein the key frame cornea contour images are images with completely visible cornea contours, the shielding frame cornea contour images are images with shielded cornea contours, when the current frame cornea contour images are shielding frame cornea contour images, the optical flow complementing network restores motion information lost due to shielding between the current frame cornea contour images and one key frame cornea contour image adjacent to the current frame cornea contour images, the key frame cornea contour images are deformed through complementing optical flows output by the optical flow complementing network to reconstruct the current frame cornea contour images, and then the AR unit is controlled to superimpose a mark in the current frame cornea contour images through AR to mark out the cornea contours and a plurality of sewing needles A location; when the current frame cornea contour image is the key frame cornea contour image, the AR unit is controlled to superpose a mark in the current frame cornea contour image through the AR so as to mark out the cornea contour and a plurality of cornea sewing needle positions, so that the cornea contour can be tracked in real time, a doctor is guided to find out the cornea sewing needle positions timely and accurately, and the operation success rate is improved. The navigation method for the laminated cornea transplantation AR operation can track the cornea outline in real time, guides doctors to find the position of a cornea suture needle timely and accurately, and improves the success rate of the operation.

As described above, it will be apparent to those skilled in the art that other various changes and modifications may be made based on the technical solution and concept of the present invention, and all such changes and modifications are intended to fall within the scope of the appended claims.