阅读说明：本技术 胃部消化道下手术器械追踪和实时预警的方法和系统 (Method and system for tracking and real-time early warning of surgical instruments under stomach and digestive tract ) 是由 宋霜 谭佳豪 王焦乐 于 2021-08-26 设计创作，主要内容包括：本发明提供了一种胃部消化道下手术器械追踪和实时预警的方法及系统,该方法包括图像预处理,获得可直接处理的核心视频图像信息；手术器械识别与分割,采用HSV颜色空间变换和阈值识别、最大轮廓提取与器械分割,获得器械末端位置点；手术器械追踪预警,包括：位置标尺变换、角点检测、光流跟踪、实时预警。本发明的技术方案利用单目内窥镜采集到的视频数据,通过HSV颜色分割有效识别出器械本体,再进一步通过标尺变换以及角点检测来追踪器械的位移和软组织的形变,通过分析位移及形变大小来设置安全阈值,达到安全预警的功能,有效保障了手术过程中医生的安全操作和患者的人身安全。(The invention provides a method and a system for tracking and early warning an operating instrument under a stomach digestive tract in real time, wherein the method comprises the steps of preprocessing an image to obtain core video image information which can be directly processed; identifying and segmenting surgical instruments, namely obtaining the tail end position points of the instruments by adopting HSV color space transformation and threshold identification, maximum contour extraction and instrument segmentation; early warning is tracked to surgical instruments includes: position scale transformation, angular point detection, optical flow tracking and real-time early warning. According to the technical scheme, the device body is effectively identified by HSV color segmentation by utilizing video data acquired by a monocular endoscope, the displacement of the device and the deformation of soft tissues are tracked by scale transformation and angular point detection, and a safety threshold is set by analyzing the displacement and the deformation, so that the function of safety early warning is achieved, and the safety operation of a doctor and the personal safety of a patient in the operation process are effectively guaranteed.)

1. A method for tracking and real-time early warning of a surgical instrument under a stomach digestive tract is characterized by comprising the following steps: it includes:

step S1, image preprocessing is carried out on the acquired real-time image in the monocular endoscopy to obtain core video image information which can be directly processed;

step S2, identifying and segmenting surgical instruments, and obtaining the tail end position points of the instruments by adopting HSV color space transformation and threshold identification, maximum contour extraction and instrument segmentation;

step S3, tracking and warning the surgical instrument, including: converting the two-dimensional image characteristics into three-dimensional displacement and deformation through scale transformation to obtain the length occupied by each pixel point, and calculating the moving distance of the tail end position point under different frames by taking the length as a unit so as to obtain the displacement size of the tail end position point of the instrument; determining a specified number of characteristic points on the soft tissue through angular point detection; tracking the characteristic points by means of optical flow, and acquiring the displacement and deformation of the instrument and the soft tissue at the starting moment in real time; and setting a safety threshold value to judge the pulling state: and when the end position point of the instrument and the whole deformation of the soft tissue are simultaneously greater than the set threshold value, judging that the state is unsafe, and sending out an early warning signal.

2. The method for tracking and real-time early warning of surgical instruments under the gastrointestinal tract of claim 1, wherein: the step S1 image preprocessing includes: the image is converted from an endoscope acquisition format to a format required by image post-processing, then a calibration experiment is carried out by means of internal reference and external reference of the existing fisheye endoscope, distortion correction of a monocular fisheye endoscope camera is completed, finally image cutting is carried out, the area of soft tissue clamped by a surgical instrument in the image is reserved, and core video image information capable of being directly processed is obtained.

3. The method for tracking and real-time early warning of surgical instruments under the gastrointestinal tract of claim 2, wherein: step S2 includes: determining the threshold ranges of hue H, saturation S and lightness V applicable to the instrument in an experimental environment by using HSV color space transformation to obtain the rough outline of the instrument; and by means of maximum contour extraction, segmenting and drawing the contour of the instrument and the circumscribed minimum rectangle, and meanwhile, determining the middle point of the short side of the rectangle corresponding to the tail end of the instrument by means of prior information that the tail end of the instrument always appears near the central point of the image, and representing the position point of the tail end of the instrument by the position of the middle point.

4. The method for tracking and real-time early warning of surgical instruments under the gastrointestinal tract of claim 3, wherein: the instrument tip is identified by the steps of:

four end points (p) of the minimum circumscribed rectangle of the input as the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a If ED (mid (p) is present₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Conversely, the other three conditions can be used for deducing the position of the tail end of the instrument in the same way.

5. The method for tracking and real-time early warning of surgical instruments under the gastrointestinal tract of claim 4, wherein: the safety threshold is set through a relation function of tension and displacement deformation obtained in an animal in-vitro experiment;

in the animal in-vitro experiment, firstly, a monocular camera is connected with a computer to record images, the brightness of a light source and the focal length of the camera are adjusted to obtain clear images, then, an operation clamp is used for clamping and pulling a soft tissue bulge, the interaction process of surgical instruments and biological soft tissue is simulated, and finally, the whole operation process is recorded and filed, so that a simulated operation video set is obtained;

the simulated operation video set comprises a plurality of sections of effective instrument dragging videos, and meanwhile, by means of displacement and tension relations acquired through multiple times of experiment acquisition, a scatter diagram of a plurality of groups of experiment data is fitted to obtain a relation function of tension and displacement deformation, so that a safety threshold value is determined.

6. The method for tracking and real-time early warning of surgical instruments under the gastrointestinal tract of claim 5, wherein: the optical flow tracking comprises a soft tissue deformation safety identification algorithm, the starting judgment of the soft tissue deformation safety identification algorithm is based on the integral displacement of the feature points, when the integral displacement of the feature points of three continuous frames is greater than a specified threshold value and has a tendency of increasing, the detection is started, and the safety judgment of deformation is carried out; the soft tissue deformation safety identification algorithm is judged to be ended according to prior information in the pulling process, when the pulling fracture occurs, the characteristic point displacement is suddenly reduced along with the rebound of the soft tissue, and the ending moment is judged by comparing the displacement change conditions of adjacent frames and the interval frames.

7. The system of surgical instruments pursuit and real-time early warning under stomach alimentary canal, its characterized in that: it includes:

the image preprocessing module is used for preprocessing the acquired real-time image in the monocular endoscopy surgery to obtain core video image information which can be directly processed;

the surgical instrument recognition and segmentation module is used for obtaining the position point of the tail end of the instrument by adopting HSV color space transformation and threshold recognition, maximum contour extraction and instrument segmentation;

the surgical instrument tracking early warning module converts the two-dimensional image characteristics into three-dimensional displacement and deformation through scale transformation to obtain the length occupied by each pixel point, and calculates the moving distance of the tail end position point under different frames by taking the length as a unit so as to obtain the displacement size of the tail end position point of the surgical instrument; determining a specified number of characteristic points on the soft tissue through angular point detection; tracking the characteristic points by means of optical flow, and acquiring the displacement and deformation of the instrument and the soft tissue at the starting moment in real time; and setting a safety threshold value to judge the pulling state: and when the end position point of the instrument and the whole deformation of the soft tissue are simultaneously greater than the set threshold value, judging that the state is unsafe, and sending out an early warning signal.

8. The system for surgical instrumentation tracking and real-time early warning under the gastrointestinal tract of claim 7, wherein: the image preprocessing module converts an image from an endoscope acquisition format to a format required by image post-processing, performs a calibration experiment by means of internal reference and external reference of the fisheye endoscope to finish distortion correction of a monocular fisheye endoscope camera to obtain an image which is not affected by distortion, and finally performs image cutting to reserve an area of a surgical instrument clamping soft tissues in the image to obtain core video image information which can be directly processed.

9. The system for surgical instrumentation tracking and real-time early warning under the gastrointestinal tract of claim 8, wherein: the surgical instrument identification and segmentation module determines the threshold ranges of hue H, saturation S and lightness V applicable to the instrument under the experimental environment by using HSV color space transformation to obtain the rough outline of the instrument; extracting by means of the maximum contour, segmenting and drawing the contour of the instrument and the circumscribed minimum rectangle, determining the middle point of the short side of the rectangle corresponding to the tail end of the instrument by means of the prior information of the tail end of the instrument always appearing near the central point of the image, and representing the position point of the tail end of the instrument by the position of the middle point;

the surgical instrument identification and segmentation comprises an instrument tip reasoning module, wherein the instrument tip reasoning module determines the instrument tip by adopting the following steps:

four end points (p) of the minimum circumscribed rectangle of the input as the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a If ED (mid (p) is present₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Conversely, the other three conditions can be used for deducing the position of the tail end of the instrument in the same way.

10. The system for surgical instrumentation tracking and real-time early warning under the gastrointestinal tract of claim 9, wherein: the surgical instrument tracking early warning module comprises a soft tissue deformation safety recognition module, the soft tissue deformation safety recognition module recognizes based on the integral displacement of the feature points, and when the integral displacement of the feature points of three continuous frames is larger than a specified threshold value and has a trend of increasing, the soft tissue deformation safety recognition module starts to detect and judges the safety of deformation;

the soft tissue deformation safety identification module carries out ending judgment according to prior information in the dragging process: and comparing the displacement change conditions of the adjacent frame and the inter-frame to judge the ending time.

Technical Field

The invention relates to the technical field of medical instruments, in particular to a method and a system for tracking and early warning an operating instrument under a stomach digestive tract in real time.

Background

In order to solve the problem of small operation space of single-instrument endoscopic cutting surgery, people begin to research and develop a multi-instrument-assisted endoscopic minimally invasive surgery robot method, and by applying auxiliary instruments, a scalpel obtains larger operation space and the safety of the surgery is ensured. Among them, the most important auxiliary instrument is a surgical forceps. In the operation, the operation forceps fully expose submucosa through the operations of clamping, lifting and the like to the alimentary canal mucosa, thereby providing a great operable space for endoscopic operation.

However, the introduction of auxiliary instruments also brings additional safety hazards to endoscopic minimally invasive surgery. The endoscopic surgery seriously depends on the fine operation of a doctor, but because the feedback of operation force is lacked and the touch information is lacked in the operation process, the doctor is difficult to reasonably evaluate the safety of the operations of clamping, pulling and the like of the auxiliary instrument, the misoperation is easy to cause the tearing of soft tissues, the tissue hemorrhage is caused, and the life safety of a patient is threatened. Therefore, in the traditional endoscopic minimally invasive surgery, a doctor needs to ensure smooth operation and also needs to be careful to ensure the safety of interaction between the instrument and soft tissues, so that the burden of the doctor is greatly increased, the safety of the endoscopic surgery is reduced, and the further development of the endoscopic minimally invasive surgery is hindered. In response to these problems, a new direction in current research is how to remotely drive biopsy devices outside the patient's body using magnetic fields. Therefore, in order to ensure the safety of interaction between the auxiliary instrument and soft tissues in the multi-instrument endoscope minimally invasive surgery, a set of instrument tracking and interaction safety early warning method for the digestive endoscope minimally invasive surgery robot is developed, which is a key problem for ensuring the safety in the surgery.

Disclosure of Invention

Aiming at the technical problems, the invention discloses a method and a system for tracking and real-time early warning of a surgical instrument under a stomach digestive tract, which realize the functions of real-time tracking and safety early warning in the process of a stomach digestive tract operation, reduce the burden of using a robot for operation by a doctor in the operation process and ensure the safety of a patient in the operation process.

In contrast, the technical scheme adopted by the invention is as follows:

a method for tracking and real-time early warning of surgical instruments under a gastrointestinal tract comprises the following steps:

step S1, image preprocessing is carried out on the acquired real-time image in the monocular endoscopy to obtain core video image information which can be directly processed;

step S2, identifying and segmenting surgical instruments, and obtaining the tail end position points of the instruments by adopting HSV color space transformation and threshold identification, maximum contour extraction and instrument segmentation;

step S3, tracking and warning the surgical instrument, including: converting the two-dimensional image characteristics into three-dimensional displacement and deformation through scale transformation to obtain the length occupied by each pixel point, and calculating the moving distance of the tail end position point under different frames by taking the length as a unit so as to obtain the displacement size of the tail end position point of the instrument; determining a specified number of characteristic points on the soft tissue through angular point detection; tracking the characteristic points by means of optical flow, and acquiring the displacement and deformation of the instrument and the soft tissue at the starting moment in real time; and setting a safety threshold value to judge the pulling state: when the tail end position point of the instrument and the whole deformation of the soft tissue are simultaneously larger than the set threshold value, the state is judged to be unsafe, and an early warning signal is sent out, so that the signal can be sent out to require the instrument to stop clamping and pulling states immediately.

Further, step S1 specifically includes: image format conversion, fisheye distortion correction, central area image retention and clipping.

As a further improvement of the present invention, the step S1 image preprocessing includes: the method comprises the steps of converting an image from an endoscope acquisition format to a format required by image post-processing, performing a calibration experiment by means of internal parameters and external parameters of the existing fisheye endoscope to finish distortion correction of a monocular fisheye endoscope camera to obtain an image hardly affected by distortion, finally performing image cutting, reserving an area of a surgical instrument clamping soft tissues in the image, and obtaining core video image information capable of being directly processed.

As a further improvement of the present invention, step S2 includes: determining the threshold ranges of hue H, saturation S and lightness V applicable to the instrument in an experimental environment by using HSV color space transformation to obtain the rough outline of the instrument; and by means of maximum contour extraction, segmenting and drawing the contour of the instrument and the circumscribed minimum rectangle, and meanwhile, determining the middle point of the short side of the rectangle corresponding to the tail end of the instrument by means of prior information that the tail end of the instrument always appears near the central point of the image, and representing the position point of the tail end of the instrument by the position of the middle point.

As a further development of the invention, the instrument tip is determined by the following steps:

four end points (p) of the minimum circumscribed rectangle of the input as the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a Consider the distance from the midpoint of the two short edges to the midpoint of the image, if there is ED (mid (p)₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Conversely, the other three conditions can be used for deducing the position of the tail end of the instrument in the same way.

As a further improvement of the invention, the in vitro animal experiment is carried out, and the tools required by the in vitro animal experiment comprise a monocular endoscope, standard surgical clamps, animal stomach organs, two groups of motor driving units and one group of stress detection units.

As a further improvement of the invention, the safety threshold is set by a relation function of tension and displacement deformation obtained in an animal in-vitro experiment;

in the animal in-vitro experiment, firstly, a monocular camera is connected with a computer to record images, the brightness of a light source and the focal length of the camera are adjusted to obtain clear images, then, an operation clamp is used for clamping and pulling a soft tissue bulge, the interaction process of surgical instruments and biological soft tissue is simulated, and finally, the whole operation process is recorded and filed, so that a simulated operation video set is obtained;

the simulated operation video set comprises a plurality of sections of effective instrument dragging videos, and meanwhile, by means of displacement and tension relations acquired through multiple times of experiment acquisition, a scatter diagram of a plurality of groups of experiment data is fitted to obtain a relation function of tension and displacement deformation, so that a safety threshold value is determined.

Further, the monocular endoscope is an operation standard 1/10' CCD lens.

As a further improvement of the invention, in the optical flow tracking, based on the size of the whole displacement of the feature points, when the whole displacement of the feature points of three continuous frames is greater than a specified threshold and tends to become larger, detection is started, and safety judgment of deformation is performed; when the fracture is dragged, the displacement of the feature point generates a sudden drop along with the rebound of soft tissue, and the judgment is finished by comparing the displacement change conditions of adjacent frames and the inter-phase frames.

Furthermore, in the optical flow tracking, the soft tissue deformation safety identification and judgment starting algorithm can perform judgment based on the overall displacement of the feature points, and under the condition that external vibration is not considered, when the overall displacement of the feature points of three continuous frames is greater than a specified threshold and tends to increase, detection is started to perform deformation safety judgment.

Further, in the optical flow tracking, for the judgment ending algorithm for soft tissue deformation safety identification, a specific idea is that according to prior information in the dragging process: along with the displacement of soft tissues, the average displacement of the feature points of each frame is increased along with the increase of the number of frames; however, when the fracture is dragged, the displacement of the characteristic point can generate a sudden drop along with the rebound of soft tissue, and the ending time is judged by comparing the displacement change conditions of adjacent frames and the frames between the adjacent frames.

The invention also discloses a system for tracking and real-time early warning of surgical instruments under the stomach and digestive tract, which comprises the following components:

the image preprocessing module is used for preprocessing the acquired real-time image in the monocular endoscopy surgery to obtain core video image information which can be directly processed;

the surgical instrument recognition and segmentation module is used for obtaining the position point of the tail end of the instrument by adopting HSV color space transformation and threshold recognition, maximum contour extraction and instrument segmentation;

the surgical instrument tracking early warning module converts the two-dimensional image characteristics into three-dimensional displacement and deformation through scale transformation to obtain the length occupied by each pixel point, and calculates the moving distance of the tail end position point under different frames by taking the length as a unit so as to obtain the displacement size of the tail end position point of the surgical instrument; determining a specified number of characteristic points on the soft tissue through angular point detection; tracking the characteristic points by means of optical flow, and acquiring the displacement and deformation of the instrument and the soft tissue at the starting moment in real time; and setting a safety threshold value to judge the pulling state: and when the end position point of the instrument and the whole deformation of the soft tissue are simultaneously greater than the set threshold value, judging that the state is unsafe, and sending out an early warning signal.

As a further improvement of the invention, the image preprocessing module converts the image from the endoscope acquisition format to the format required by the image post-processing, then carries out calibration experiments by means of internal reference and external reference of the fish-eye endoscope, finishes the distortion correction of the monocular fish-eye endoscope camera, obtains an image which is not affected by distortion, and finally carries out image cutting, reserves the area of soft tissue clamped by surgical instruments in the image, and obtains the core video image information which can be directly processed.

As a further improvement of the invention, the surgical instrument recognition and segmentation module determines the threshold ranges of hue H, saturation S and lightness V applicable to the instrument in an experimental environment by using HSV color space transformation to obtain a rough contour of the instrument; and by means of maximum contour extraction, segmenting and drawing the contour of the instrument and the circumscribed minimum rectangle, and meanwhile, determining the middle point of the short side of the rectangle corresponding to the tail end of the instrument by means of prior information that the tail end of the instrument always appears near the central point of the image, and representing the position point of the tail end of the instrument by the position of the middle point.

As a further improvement of the present invention, the surgical instrument recognition and segmentation includes an instrument tip inference module, which determines the instrument tip by using the following steps:

four end points (p) of the minimum circumscribed rectangle of the input as the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a Consider the distance from the midpoint of the two short edges to the midpoint of the image, if there is ED (mid (p)₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Conversely, the other three conditions can be used for deducing the position of the tail end of the instrument in the same way.

As a further improvement of the invention, the surgical instrument tracking and early warning module comprises a soft tissue deformation safety identification module, the soft tissue deformation safety identification module identifies based on the size of the overall displacement of the feature points, and when the overall displacement of the feature points of three continuous frames is greater than a specified threshold value and has a tendency of increasing, the soft tissue deformation safety identification module starts to detect and judge the safety of deformation; the soft tissue deformation safety identification module carries out ending judgment according to prior information in the dragging process: and comparing the displacement change conditions of the adjacent frame and the inter-frame to judge the ending time.

The invention also discloses a device which comprises a processor and a memory which are connected, wherein the processor is used for executing the computer program stored in the memory so as to execute the method for tracking and real-time early warning of the surgical instrument under the gastrointestinal tract.

The invention also discloses a computer readable storage medium comprising a computer program for performing the method of sub-gastric-digestive-tract surgical instrument tracking and real-time early warning as described in any one of the above when the computer program runs on a computer.

Compared with the prior art, the invention has the beneficial effects that:

according to the technical scheme, the device body is effectively identified by HSV color segmentation by utilizing video data acquired by a monocular endoscope, the displacement of the device and the deformation of soft tissues are tracked by scale transformation and angular point detection, and a safety threshold is set by analyzing the displacement and the deformation, so that the function of safety early warning is achieved, and the safety operation of a doctor and the personal safety of a patient in the operation process are effectively guaranteed.

Drawings

Fig. 1 is a flow chart of a method for tracking and real-time early warning of a surgical instrument under a gastrointestinal tract according to an embodiment of the invention.

Fig. 2 is a surgical instrument tip algorithm derivative diagram of a method for tracking and real-time early warning of a surgical instrument under a gastrointestinal tract according to an embodiment of the invention.

Detailed Description

The following examples are given for the detailed implementation and specific operation of the present invention, but the scope of the present invention is not limited to the following examples.

As shown in fig. 1, a method for tracking and real-time early warning of a surgical instrument under a stomach digestive tract comprises the following steps:

(1) construction of endoscopic minimally invasive surgery simulation platform

In order to simulate the instrument endoscope minimally invasive surgery platform required by the project, the invention builds a minimum double-instrument endoscope surgery simulation platform based on a monocular camera, a pig stomach digestive tract, an operating forceps and the like, and the platform has the functions of basically assisting instruments to clamp and pull biological soft tissues and the like. The pig stomach digestive tract simulates the patient's digestive tract soft tissue, which has similar soft body characteristics to biological soft tissue. The simulation platform adopts the surgical forceps as an auxiliary instrument to be responsible for clamping, pulling and other operations on the surface mucosa of the soft tissue, provides a larger operation space for the operation and assists the cutting of the surgical knife; simulating biological soft tissue such as tumor, intestinal mucosa and the like by adopting soft tissue on the inner wall of the pig digestive tract; shooting and recording an operation image by adopting a monocular camera, and collecting data; a camera USB interface is adopted to connect a computer and store data; the light source is adopted to illuminate the operation area, and brightness compensation is provided for the camera.

(2) Animal in vitro experiment and monocular video data acquisition:

based on a simulation experiment platform, the monocular camera is connected with a computer to record images, the brightness of a light source and the focal length of the camera are adjusted to obtain clear images, then the operation forceps are used for clamping and pulling the soft tissue bulges, the interaction process of the operation forceps and biological soft tissues is simulated, and finally the whole operation process is recorded and filed, so that a simulated operation video set is obtained. Based on a simulation platform, the invention simulates endoscopic surgery operation, collects 20 accumulated effective instrument pulling videos, and fits a scatter diagram of a plurality of groups of experimental data by means of displacement and tension relations acquired by multiple experiments, thereby finally preliminarily obtaining a relation function of tension and displacement deformation.

(3) Video preprocessing:

in order to effectively utilize and obtain interactive data in a video, firstly, a video image needs to be converted from an endoscope acquisition format to an image processing required format, then distortion correction of a monocular fisheye endoscope camera is carried out to obtain an image which is hardly affected by distortion, and finally, image cutting is carried out to reserve the middle area of the image so as to obtain core video image information which can be directly processed.

(4) Surgical instrument identification based on HSV color segmentation:

after video data which can be directly processed is obtained, the image is required to be converted into an HSV color space, the value ranges of hue H, saturation S and lightness V which are applicable to the instrument in an experimental environment are determined through image processing operations such as filtering, corrosion and expansion by means of a designed sliding adjusting function, the value is adjusted through multiple experiments, the optimal range which accords with the actual effect is finally obtained, the contour of the surgical clamp is obtained, other noise points are filtered, and the position of the central point of the instrument end tool is described through maximum contour extraction and the midpoint of the short edge of the tail end of the circumscribed rectangle, so that the tail end position of the surgical instrument is obtained.

Wherein, the position of the tail end of the surgical instrument adopts the following derivation algorithm:

as shown in FIG. 2, the input is the four endpoints (p) of the minimum bounding rectangle of the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a Further analysis, consider the distance from the midpoint of the two short edges to the midpoint of the image, if any, ED (mid (p)₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Otherwise, the other three conditions can be deduced to the position of the end of the surgical instrument in the same way.

(6) Early warning is tracked to surgical instruments specifically includes:

a) scale transformation of three-dimensional depth:

because the monocular camera is adopted for image data acquisition, the actual size of the instrument displacement needs to be detected by means of an additional reference. In the image data, since the instrument used is a circular tube, it is most obvious to perform size conversion with the diameter of the instrument as a reference. The length occupied by each pixel point can be obtained by measuring the diameter of the known instrument to be 2.3mm and by means of the minimum circumscribed rectangle obtained before, the moving distance of the tail end position point under different frames can be calculated by taking the length as a unit, and therefore the displacement size of the tail end position point of the instrument can be obtained. The method for determining the displacement through the size transformation avoids the defects that the depth information cannot be acquired by a single eye and the deviation is caused by different depths, and can still well complete the actual displacement measurement of the tail end position point.

b) And (3) HARRI corner detection:

as the instrument pulls the soft tissue, the deformation directions and the deformation sizes of the soft tissue at different positions are different. In order to better describe the whole deformation of the soft tissue, firstly, HARRI angular point detection is carried out on the soft tissue, 200 characteristic points are detected, wherein most of the characteristic points are on the soft tissue and are used for qualitatively judging the whole deformation size of the soft tissue.

c) Optical flow tracking:

and then, obtaining the moving distance of the pixel point of each characteristic point by means of optical flow tracking, and obtaining the deformation size representing the whole soft tissue according to the previous size transformation and summation average. In practical application, the feature corner points of the soft tissue can be represented by green dots, and the red lines represent the optical flow displacement from the starting positions of the feature points to the real-time positions.

d) Deformation safety early warning:

the displacement and the deformation of the instrument and soft tissue at the relative starting moment are obtained in real time through angular point detection and optical flow tracking, and a safety threshold is set to judge the dragging state: when the tail end position point of the instrument and the whole deformation of the soft tissue are simultaneously larger than the set threshold value, the state is judged to be unsafe, and the instrument is required to stop clamping and pulling states immediately.

And (3) setting the safety threshold value by using a relation function of the tension and the displacement deformation obtained in the step (2), so that the pulling state can be conveniently judged.

The method for tracking the surgical instrument under the gastrointestinal tract and early warning in real time determines an algorithm for starting and ending operation in the process of detecting the soft tissue pulled by the instrument. The starting judgment algorithm of the soft tissue deformation safety identification module can judge based on the integral displacement of the feature points, and under the condition that external vibration is not considered, when the integral displacement of the feature points of three continuous frames is greater than a specified threshold value and has a trend of increasing, the module starts detection to judge the safety of deformation. For the judgment finishing algorithm of the soft tissue deformation safety identification module, the specific idea is that the algorithm is based on prior information in the pulling process: along with the displacement of soft tissues, the average displacement of the feature points of each frame is increased along with the increase of the number of frames; however, when the fracture is dragged, the displacement of the characteristic point can generate a sudden drop along with the rebound of soft tissue, and the ending time is judged by comparing the displacement change conditions of adjacent frames and the frames between the adjacent frames.

In this embodiment, for the determination of the number of the instruments, the image may be subjected to inverse binarization and appropriate erosion and expansion operations under the precondition that the instrument color and the like are the same in basic conditions by using the optimal value range of the previous HSV image segmentation, and the number of the instruments may be determined by determining the number of the largest connected regions in the binary image that are larger than the threshold value.

The embodiment utilizes the video data that monocular endoscope gathered, cuts apart effectively to discern the apparatus body through HSV colour, further tracks the displacement of apparatus and the deformation of soft tissue through scale transform and angular point detection again, sets up the safety threshold value through analysis displacement and deformation size, reaches the function of safety precaution, has effectively ensured doctor's safety operation and patient's personal safety in the operation process.

The embodiment also discloses a system for tracking and real-time early warning of surgical instruments under the gastrointestinal tract, which comprises:

the image preprocessing module is used for preprocessing the acquired real-time image in the monocular endoscopy, converting the image from an endoscope acquisition format to a format required by image post-processing, performing a calibration experiment by means of internal reference and external reference of the fisheye endoscope to finish distortion correction of the monocular fisheye endoscope camera to obtain an image hardly affected by distortion, and finally performing image cutting to reserve a main area of an image instrument for clamping soft tissues to obtain core video image information capable of being directly processed.

The surgical instrument recognition and segmentation module determines the threshold ranges of hue H, saturation S and lightness V applicable to the instrument under the experimental environment by using HSV color space transformation to obtain the rough outline of the instrument; and by means of maximum contour extraction, segmenting and drawing the contour of the instrument and the circumscribed minimum rectangle, and simultaneously determining the middle point of the short side of the rectangle corresponding to the tail end of the instrument by means of prior information and an algorithm, and representing the position point of the tail end of the instrument by the position of the middle point.

The surgical instrument tracking early warning module converts the two-dimensional image characteristics into three-dimensional displacement and deformation through scale transformation to obtain the length occupied by each pixel point, and calculates the moving distance of the tail end position point under different frames by taking the length as a unit so as to obtain the displacement size of the tail end position point of the surgical instrument; determining a specified number of characteristic points on the soft tissue through angular point detection; tracking the characteristic points by means of optical flow, and acquiring the displacement and deformation of the instrument and the soft tissue at the starting moment in real time; and setting a safety threshold value to judge the pulling state: and when the end position point of the instrument and the whole deformation of the soft tissue are simultaneously greater than the set threshold value, judging that the state is unsafe, and sending out an early warning signal.

The surgical instrument identification and segmentation comprises an instrument tip reasoning module, wherein the instrument tip reasoning module determines the instrument tip by adopting the following steps:

four end points (p) of the minimum circumscribed rectangle of the input as the surgical instrument₀,p₁,p₂,p₃) Then, the Euler distances of any adjacent sides are taken for comparison, if ED (p)₀,p₃)<ED(p₀,p₁) Then, the minimum bounding rectangle short side is determined to be (p)₀,p₃) And (p)₁,p₂) (ii) a Consider the distance from the midpoint of the two short edges to the midpoint of the image, if there is ED (mid (p)₀,p₃),O_c)<ED(mid(p₁,p₂),O_c) Then the short edge of the end of the instrument is determined as the midpoint (p)₀,p₃) The corresponding instrument end point is the middle point mid (p) of the short side₀,p₃) (ii) a Conversely, the other three conditions can be used for deducing the position of the tail end of the instrument in the same way.

When the system is applied specifically, a video data set can be constructed under the isolated gastric digestive tract of an animal, and the identification process of the surgical instrument can be completed by performing operations such as HSV color segmentation on related videos; meanwhile, scale transformation is carried out on the two-dimensional video data, the motion displacement and the state of the real-time instrument are obtained based on optical flow tracking, the comprehensive deformation of the soft tissue is quantitatively analyzed, a threshold value is finally set to achieve an early warning function, and real-time state tracking and early warning of interaction of the surgical instrument and the soft tissue are achieved.

The invention also discloses a device which comprises a processor and a memory which are connected, wherein the processor is used for executing the computer program stored in the memory so as to execute the method for tracking and real-time early warning of the surgical instrument under the gastrointestinal tract.

The invention also discloses a computer readable storage medium comprising a computer program which, when run on a computer, performs the method of surgical instrument tracking and real-time early warning under the gastrointestinal tract as described in any one of the above.

The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.