阅读说明：本技术 路径定位方法、信息显示装置、存储介质及集成电路芯片 (Path positioning method, information display device, storage medium and integrated circuit chip ) 是由 邱锡彦 陈品元 刘浩澧 魏国珍 于 2021-01-29 设计创作，主要内容包括：本发明提供一种路径定位方法、信息显示装置、存储介质及集成电路芯片,该方法根据一信息显示装置提供的与一实际手术对象相关的多个影像图片找出一呈现一病灶的目标影像图片并在病灶上决定一手术目标点,且建构出呈现该手术目标点的一三维虚拟影像,并将该三维虚拟影像叠合在该实际手术对象影像上以显示一叠合后影像,且于该叠合后影像上产生一对应于该实际手术对象的表面的一实际入刀点的虚拟入刀点,并使该虚拟入刀点与该手术目标点连成一入刀路径,并显示由该手术目标点朝该虚拟入刀点方向沿着该入刀路径从该入刀点向外直线延伸的一入刀导引路径。由此,可协助手术操作者规划手术路径,而有助于手术的顺利进行并提高手术成功率。(The invention provides a path positioning method, an information display device, a storage medium and an integrated circuit chip, wherein the method finds out a target image picture presenting a focus according to a plurality of image pictures related to an actual operation object provided by an information display device, determines an operation target point on the focus, constructs a three-dimensional virtual image presenting the operation target point, superimposes the three-dimensional virtual image on the actual operation object image to display a superimposed image, generates a virtual knife entering point corresponding to an actual knife entering point on the surface of the actual operation object on the superimposed image, connects the virtual knife entering point and the operation target point into a knife entering path, and displays a knife entering guide path extending from the knife entering point outwards along the knife entering path from the operation target point towards the virtual knife entering point. Therefore, the surgical path planning device can assist an operator in planning a surgical path, is favorable for smooth operation and improves the success rate of the operation.)

1. A method for positioning a surgical path, comprising:

a: the information display device provides a plurality of image pictures related to an actual operation object, so as to find out a target image picture from the plurality of image pictures and display the target image picture, wherein the target image picture presents a focus positioned in the actual operation object, and the target image picture can be clicked to determine an operation target point on the focus;

b: the three-dimensional image construction module of the information display device constructs a three-dimensional virtual image which presents the operation target point and corresponds to the actual operation object according to the target image picture and the plurality of image pictures which present the focus and the operation target point;

c: when the information display device shoots the actual surgical object and displays the shot actual surgical object image, the image superposition module of the information display device superposes the three-dimensional virtual image on the actual surgical object image to generate a superposed image presenting the surgical target point, and the information display device is made to display the superposed image; and

d: the knife entering path generating module of the information display device generates a virtual knife entering point corresponding to the actual knife entering point on the surface of the actual surgical object on the superposed image according to the surgical target point and the plurality of image pictures presented on the superposed image, connects the virtual knife entering point and the surgical target point into a straight line, and displays a knife entering guide path extending from the surgical target point to the virtual knife entering point along the straight line outwards from the virtual knife entering point.

2. The method as claimed in claim 1, wherein the three-dimensional virtual image includes a first three-dimensional virtual image corresponding to the actual surgical object and a second three-dimensional virtual image corresponding to the surgical target point, the image overlay module overlays the first three-dimensional virtual image and the second three-dimensional virtual image on the actual surgical object image to generate the overlaid image representing the surgical target point, and the information display device displays the overlaid image to display the actual position of the surgical target point.

3. The method as claimed in claim 1, wherein the knife-entering detection module of the information display device detects whether a surgical instrument appears in the superimposed image displayed by the information display device, and determines whether the tip of the surgical instrument and the virtual knife-entering point are superimposed, determines whether the tip of the surgical instrument and the surgical target point are superimposed, determines whether the surgical instrument and the knife-entering guide path are superimposed and/or determines whether a spatial angle of the surgical instrument and a spatial angle of the knife-entering guide path are superimposed, and outputs feedback information according to the determination result.

4. The method as claimed in claim 1, wherein when the knife-in detection module of the information display device detects that a surgical instrument appears in the superimposed image displayed by the information display device, the knife-in detection module instantly generates a virtual surgical instrument corresponding to the surgical instrument, and correspondingly displays the virtual surgical instrument in the superimposed image according to the position of the surgical instrument in the superimposed image; and the knife entering detection module judges whether the tip point of the virtual surgical instrument is superposed with the virtual knife entering point, judges whether the tip point of the virtual surgical instrument is superposed with the surgical target point, judges whether the virtual surgical instrument is superposed with the knife entering guide path and/or judges whether the space angle of the virtual surgical instrument is superposed with the space angle of the knife entering guide path, and outputs feedback information according to the judgment result.

5. The method according to claim 3 or 4, wherein the feedback information comprises at least one of image feedback, sound feedback, and tactile feedback, the image feedback comprises at least one of color change and new information, the sound feedback comprises an audio prompt, and the tactile feedback is that the information display device vibrates the wearable electronic device by transmitting a feedback signal to the wearable electronic device.

6. An information display device, comprising:

a storage unit in which a plurality of image pictures related to an actual surgical object are stored;

a display;

an image capturing unit; and

a processor electrically connected to the storage unit, the display and the image capturing unit and operable to find a target image picture from the plurality of image pictures stored in the storage unit and to cause the display to display the target image picture, the target image picture representing a lesion located inside the actual surgical object, and the processor causing the target image picture to be selectable for determining a surgical target point on the lesion; and the processor comprises a three-dimensional image construction module, an image superposition module and a cutter-entering path generation module, wherein

The three-dimensional image construction module constructs a three-dimensional virtual image which presents the operation target point and corresponds to the actual operation object according to the target image picture and the plurality of image pictures which present the focus and the operation target point;

when the image overlapping module judges that the display displays the actual operation object image of the actual operation object shot by the image capturing unit, the three-dimensional virtual image is overlapped on the actual operation object image to generate an overlapped image presenting the operation target point, and the display is enabled to display the overlapped image;

the knife entering path generating module generates a virtual knife entering point corresponding to the actual knife entering point on the surface of the actual surgical object on the superposed image according to the surgical target point displayed on the superposed image and the plurality of image pictures stored in the storage unit, connects the virtual knife entering point and the surgical target point into a straight line, and enables the display to display a knife entering guide path extending from the virtual knife entering point to the outside along the straight line from the surgical target point to the virtual knife entering point.

7. The information display apparatus according to claim 6, wherein the three-dimensional virtual image includes a first three-dimensional virtual image corresponding to the actual surgical object and a second three-dimensional virtual image corresponding to the surgical target point, the image superimposing module superimposes the first three-dimensional virtual image and the second three-dimensional virtual image on the actual surgical object image to generate the superimposed image representing the surgical target point, and causes the display to display the superimposed image to display an actual position of the surgical target point.

8. The information display device of claim 6, wherein the processor further comprises a knife-entering detection module, the knife-entering detection module determines that the image capturing unit captures a surgical instrument, determines whether the tip of the surgical instrument overlaps the virtual knife-entering point, determines whether the tip of the surgical instrument overlaps the surgical target, determines whether the surgical instrument overlaps the knife-entering guide path, and/or determines whether the spatial angle of the surgical instrument overlaps the spatial angle of the knife-entering guide path, and outputs feedback information according to the determination result.

9. The information display device of claim 6, wherein the processor further comprises a knife-in detection module, the knife-in detection module determines that when the image capturing unit captures a surgical instrument, the knife-in detection module instantly generates a virtual surgical instrument corresponding to the surgical instrument, and correspondingly displays the virtual surgical instrument in the superimposed image according to a position of the surgical instrument in a space; and the knife-entering detection module judges whether the tip point of the virtual surgical instrument is superposed with the virtual knife-entering point, judges whether the tip point of the virtual surgical instrument is superposed with the surgical target point, judges whether the virtual surgical instrument is superposed with the knife-entering guide path and/or judges whether the space angle of the virtual surgical instrument is superposed with the space angle of the knife-entering guide path, and outputs feedback information according to the judgment result.

10. The information display device of claim 8 or 9, wherein the feedback information comprises at least one of visual feedback, audio feedback, and tactile feedback, the visual feedback comprises at least one of color change and new information, the audio feedback comprises an audio prompt, and the tactile feedback is generated by the processor transmitting a feedback signal to the wearable electronic device in a wired or wireless manner to vibrate the wearable electronic device.

11. A computer-readable storage medium storing a program for causing a computer to execute the surgical path positioning method according to any one of claims 1 to 5.

12. An asic chip, wherein a three-dimensional image constructing module, an image superimposing module and a knife path generating module are integrated, and when the asic chip is applied to an electronic device, the electronic device can perform the surgical path positioning method according to claim 1 or 2.

13. The asic chip of claim 12, further incorporating a knife detection module, such that when the asic chip is applied to the electronic device, the electronic device can perform the surgical path positioning method of any of claims 3 to 5.

Technical Field

The present invention relates to a surgical positioning method, and more particularly, to a surgical path positioning method, an information display device, a computer-readable storage medium, and an asic chip.

Background

At present, brain tomography images are taken firstly to confirm an operation target (focus), such as a hydrocephalus part, and the operation target needs to be confirmed, and the current method mainly comprises the following steps: 1. finding an approximate position of a surgical target by DICOM image pictures obtained by Computed Tomography (CT), Magnetic Resonance Imaging (MRI), ultrasonography (Ultrasound imaging), and the like; 2. instantly displaying the DICOM image picture by using an optical navigation system to find the position of the operation target; 3. the DICOM image picture is directly overlapped on the operation object by using the Augmented Reality (AR) or Mixed Reality (MR) technology. However, after finding out the surgical target, since the current system cannot display the quantitative information of the DICOM image in real time, the physician still needs to find out the surgical access point and the surgical access path from the surgical access point to the surgical target by himself or herself with the experience of the physician.

Disclosure of Invention

The present invention is directed to a surgical path positioning method, an information display device, a computer readable storage medium and an asic chip for implementing the method, which can overlay a three-dimensional virtual image of a surgical object on the surgical object, automatically determine a knife entry point according to a surgical target point on the surgical object, generate a knife entry path from the knife entry point to the surgical target point, and generate a knife entry guide path extending outward from the knife entry point, so that an operator can enter a surgical object from the knife entry point along the knife entry guide path and keep the surgical instrument moving to the surgical target point along the correct knife entry path, thereby facilitating the smooth operation.

The surgical path positioning method of the present invention includes: (A) an information display device provides a plurality of image pictures related to an actual operation object, so as to find out a target image picture from the image pictures and display the target image picture, wherein the target image picture presents a focus positioned in the actual operation object, and the target image picture can be clicked to determine an operation target point on the focus; (B) a three-dimensional image constructing module of the information display device constructs a three-dimensional virtual image which presents the operation target point and corresponds to the actual operation object according to the target image picture and the image pictures which present the focus and the operation target point; (C) when the information display device shoots the actual surgical object and displays the shot actual surgical object image, an image superposition module of the information display device superposes the three-dimensional virtual image on the actual surgical object image to generate a superposed image presenting the surgical target point, and the information display device is made to display the superposed image; and (D) a knife entering path generating module of the information display device generates a virtual knife entering point corresponding to the actual knife entering point on the surface of the actual surgical object on the superposed image according to the surgical target point and the image pictures presented on the superposed image, connects the virtual knife entering point and the surgical target point into a straight line, and displays a knife entering guide path extending from the virtual knife entering point to the outside along the straight line from the surgical target point to the virtual knife entering point.

In some embodiments of the present invention, the three-dimensional virtual image includes a first three-dimensional virtual image corresponding to the actual surgical object and a second three-dimensional virtual image corresponding to the surgical target point, the image overlay module overlays the first three-dimensional virtual image and the second three-dimensional virtual image on the actual surgical object image to generate the overlaid image representing the surgical target point, and the display displays the overlaid image to display the actual position of the surgical target point.

In some embodiments of the present invention, a knife entering detection module of the information display device detects whether a surgical instrument appears in the superimposed image frame displayed by the information display device, and determines whether a tip of the surgical instrument and the virtual knife entering point are superimposed, determines whether a tip of the surgical instrument and the surgical target point are superimposed, determines whether the surgical instrument and the knife entering guide path are superimposed and/or determines whether a spatial angle of the surgical instrument and a spatial angle of the knife entering guide path are superimposed, and outputs a feedback information according to a determination result.

In some embodiments of the present invention, when a knife-entering detection module of the information display device detects that a surgical instrument appears in the superimposed image displayed by the information display device, the knife-entering detection module instantly generates a virtual surgical instrument corresponding to the surgical instrument, and correspondingly displays the virtual surgical instrument in the superimposed image according to the position of the surgical instrument in the superimposed image; and the knife-entering detection module judges whether a tip point of the virtual surgical instrument is superposed with the virtual knife-entering point, judges whether a tip point of the virtual surgical instrument is superposed with the surgical target point, judges whether the virtual surgical instrument is superposed with the knife-entering guide path and/or judges whether the space angle of the virtual surgical instrument is superposed with the space angle of the knife-entering guide path, and outputs feedback information according to the judgment result.

In some embodiments of the present invention, the feedback information includes at least one of image feedback, sound feedback, and tactile feedback, the image feedback includes at least one of color change and new information, the sound feedback includes sound prompt, and the tactile feedback is that the information display device vibrates the wearable electronic device by transmitting a feedback signal to the wearable electronic device.

Furthermore, the information display device for realizing the method comprises a storage unit, a display, an image acquisition unit and a processor; wherein the storage unit stores a plurality of image pictures related to an actual operation object; the processor is electrically connected with the storage unit, the display and the image acquisition unit and can be operated to find a target image picture from the image pictures stored in the storage unit and make the display the target image picture, the target image picture presents a focus positioned in the actual operation object, and the processor makes the target image picture available for selection to determine an operation target point on the focus; the processor comprises a three-dimensional image constructing module, an image overlapping module and a knife-entering path generating module, wherein the three-dimensional image constructing module constructs a three-dimensional virtual image which presents the operation target point and corresponds to the actual operation object according to the target image picture and the image pictures which present the operation target point; when the image overlapping module judges that the display displays an actual operation object image of the actual operation object shot by the image capturing unit, the three-dimensional virtual image is overlapped on the actual operation object image to generate an overlapped image presenting the operation target point, and the display is made to display the overlapped image; the knife entering path generating module generates a virtual knife entering point corresponding to an actual knife entering point on the surface of the actual surgical object on the superposed image according to the surgical target point presented on the superposed image and the image pictures stored in the storage unit, connects the virtual knife entering point and the surgical target point to form a straight line, and enables the display to display a knife entering guide path extending from the virtual knife entering point to the outside along the straight line from the surgical target point to the virtual knife entering point.

In some embodiments of the present invention, the processor further includes a knife entering detection module, the knife entering detection module determines that the image capturing unit captures a surgical instrument, determines whether a tip of the surgical instrument overlaps the virtual knife entering point, determines whether a tip of the surgical instrument overlaps the surgical target point, determines whether the surgical instrument overlaps the knife entering guide path and/or determines whether a spatial angle of the surgical instrument overlaps the spatial angle of the knife entering guide path, and outputs a feedback message according to a determination result.

In some embodiments of the present invention, the processor further includes a knife entering detection module, and when the knife entering detection module determines that the image capturing unit captures a surgical instrument, the knife entering detection module instantly generates a virtual surgical instrument corresponding to the surgical instrument, and correspondingly displays the virtual surgical instrument in the superimposed image according to a position of the surgical instrument in the space; and the knife-entering detection module judges whether a tip point of the virtual surgical instrument is superposed with the virtual knife-entering point, judges whether a tip point of the virtual surgical instrument is superposed with the surgical target point, judges whether the virtual surgical instrument is superposed with the knife-entering guide path and/or judges whether the space angle of the virtual surgical instrument is superposed with the space angle of the knife-entering guide path, and outputs feedback information according to the judgment result.

In some embodiments of the present invention, the feedback information includes at least one of an image feedback, an audio feedback, and a tactile feedback, the image feedback includes at least one of a color change and new information, the audio feedback includes an audio prompt, and the tactile feedback is that the processor transmits a feedback signal to a wearable electronic device in a wired or wireless manner to vibrate the wearable electronic device.

In addition, the computer readable storage medium for implementing the method of the present invention stores a software program including a three-dimensional image constructing module, an image superimposing module, a knife-in path generating module, and a knife-in detecting module, and when the software program is loaded into and executed by an electronic device, the electronic device can perform the surgical path positioning method as described above.

In addition, the invention realizes the application specific integrated circuit chip of the above method, wherein a three-dimensional image constructing module, an image overlapping module, a knife-entering path generating module and a knife-entering detecting module are integrated, and when the application specific integrated circuit chip is applied to an electronic device, the electronic device can complete the above operation path positioning method.

The invention has the beneficial effects that: the three-dimensional virtual image of the actual operation object is overlapped on the image of the actual operation object, and the knife entering point is automatically determined according to the focus presented by the target image picture and the operation target point, the straight line (knife entering path) advancing from the knife entering point to the operation target point and the knife entering guide path extending from the knife entering point to the outside straight line are generated, so that the operation operator can know the optimal operation knife entering path in advance, the tip point of the operation instrument can be accurately positioned at the knife entering point in the operation along the knife entering guide path, the knife entering point enters the actual operation object along the knife entering path and guides the tip point of the operation instrument to move to the operation target point along the correct knife entering path, and therefore, the operation operator is assisted in planning the operation path, the smooth operation is facilitated, and the operation success rate is improved.

Drawings

FIG. 1 is a principal flow diagram of one embodiment of a surgical path locating method of the present invention;

FIG. 2 is a flowchart illustrating the detection of whether the position and the movement path of the surgical instrument overlap with the planned knife entry point and the planned knife entry path according to the present embodiment;

FIG. 3 is a block diagram of the main hardware blocks and modules of an information display device according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a target image picture according to the present embodiment;

FIG. 5 is a schematic diagram of a three-dimensional virtual image corresponding to the actual surgical object and showing the position of the surgical target point generated in the present embodiment;

fig. 6 is a schematic diagram illustrating an overlapped image generated by overlapping the three-dimensional virtual image shown in fig. 5 on the actual operation object image according to the present embodiment;

FIG. 7 is a schematic diagram illustrating the embodiment further generating a knife-in guiding path extending from the virtual knife-in point along the straight line to the outside on the superimposed image;

FIG. 8 is a schematic diagram illustrating the present embodiment detecting whether the tip of a surgical instrument overlaps the actual cutting point;

FIG. 9 is a schematic diagram illustrating that the present embodiment displays a feedback message in an image feedback manner when detecting that the surgical instrument deviates from the knife-entering guide path;

fig. 10 is a schematic diagram illustrating that when the surgical instrument is detected to overlap the knife guiding path, a feedback message is displayed in an image feedback manner according to the embodiment.

Detailed Description

The invention is described in detail below with reference to the following figures and examples:

before the present invention is described in detail, it should be noted that in the following description, like elements are represented by like reference numerals.

Referring to fig. 1 and 2, the main process steps of an embodiment of the surgical path positioning method of the present invention are shown, and as shown in fig. 3, the method of the present embodiment is implemented by an information display device 3 of the present invention; the information display device 3 may be, but is not limited to, a desktop personal computer, a tablet computer, a smart phone or a head-mounted display device (e.g., Augmented Reality (AR) glasses, Augmented Reality (AR) head-mounted device (AR head set), etc.), and mainly includes a display 31, a storage unit 32, a processor 33 electrically connected to the display 31 and the storage unit 32, and an image capturing unit 34 electrically connected to the processor 33, such as a camera lens.

The processor 33 mainly includes a three-dimensional image construction module 331, an image superposition module 332, a tool-entering path generation module 333, and a tool-entering detection module 334, and the modules 331 to 334 may be software programs that can be loaded and executed by the processor 33 to complete the method flow shown in fig. 1 and 2; alternatively, the modules 331-334 may be integrated into one or more Application-specific integrated circuit (ASIC) chips or a Programmable Logic Device (PLD), so that the processor 33 including the ASIC chip(s) or the PLD can complete the method shown in fig. 1 and 2. Alternatively, the modules 331 to 334 may be firmware burned into the processor 33, and the processor 33 executes the firmware to complete the method flow shown in fig. 1 and 2.

Thus, when a physician needs to perform an operation on an actual operation object 1 (see fig. 6) of a patient, for example, the head of the patient, for example, the brain of the patient has a focus, for example, hydrocephalus, and the operation must be performed on the focus (hydrocephalus) portion of the brain to extract hydrocephalus, as shown in step S1 of fig. 1, the processor 33 of the information display apparatus 3 can read a plurality of image pictures related to the actual operation object (head) 1, which are pre-loaded and stored in the storage unit 32, according to the operation of the user, and make the display 31 display the image pictures according to the operation of the user. The image pictures are, for example, two-dimensional slice DICOM (Digital Imaging and Communications in Medicine) images obtained by the actual operation object (head) 1 through Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound Imaging (Ultrasound Imaging), etc., so that the image pictures can simultaneously or respectively include information of blood vessels, nerves, bones, etc.

Therefore, the physician can find out a target image picture as shown in fig. 4 from the image pictures displayed by the information display device 3, the target image picture presents a focus (water spot) 41 located inside (brain) the actual surgical object 1, and the processor 33 of the information display device 3 makes the target image picture available for clicking to determine a surgical target point 42 on the focus 41, that is, the processor 33 determines whether a target point is set on the focus 41, if so, determines the target point as the surgical target point 42 and displays the surgical target point 42 on the focus. In addition, the processor 33 can also automatically find the target image picture showing a lesion (water spot) 41 located inside the actual surgical object 1 (brain) from the image pictures by using a pre-trained artificial intelligence module (not shown), such as a neural network, and automatically set the surgical target point 42, and then the physician can decide whether to accept the surgical target point 42 set by the artificial intelligence module or perform fine adjustment on the surgical target point 42.

Next, in step S2 of fig. 1, the three-dimensional image construction module 331 of the processor 33 constructs a three-dimensional virtual image 5 corresponding to the actual surgical object 1 and representing the surgical target point 42 according to the target image picture representing the lesion 41 and the surgical target point 42 shown in fig. 4 and the image pictures stored in the storage unit 32, as shown in fig. 5. For example, the three-dimensional image construction module 331 can apply (but not limited to) Unity software to construct an initial three-dimensional virtual image corresponding to the actual surgical object 1 according to the image pictures, and the initial three-dimensional virtual image shows the position of the lesion 41, and then the three-dimensional image construction module 331 displays the surgical target 42 in the initial three-dimensional virtual image according to the positions of the lesion 41 and the surgical target 42 shown on the target image picture, so as to generate the three-dimensional virtual image 5 showing the surgical target 42.

Next, as shown in step S3 of fig. 1, when the image superimposing module 332 in the processor 33 finds that the image capturing unit 34 of the information display device 3 captures the actual surgical object 1 and displays the captured actual surgical object image 1 ', as shown in fig. 6, the image superimposing module 332 superimposes the three-dimensional virtual image 5 on the actual surgical object image 1', so as to generate a superimposed image 6 representing the surgical target 42, and causes the display 31 of the information display device 3 to display the superimposed image 6. Specifically, the three-dimensional virtual image includes a first three-dimensional virtual image corresponding to the actual surgical object 1 and a second three-dimensional virtual image corresponding to the surgical target 42, and the image overlay module 332 overlays the first three-dimensional virtual image and the second three-dimensional virtual image on the actual surgical object image 1' to generate the overlaid image 6 representing the surgical target 42, and the display 31 displays the overlaid image 6, so that the surgeon can know the actual position of the surgical target 42 from the displayed overlaid image 6.

Specifically, the image overlay module 332 can (but is not limited to) implement image overlay by using an optical positioning system, and the optical positioning system can use NDI Polaris Vicra optical tracking system, NDI Polaris Spectra optical tracking system, ART tracking system, Claronav MicronTracker, etc., but not limited thereto.

In addition, the image overlay module 332 can also apply an image positioning system developed by Vuforia augmented reality platform, for example, to replace the optical positioning system, and since the image overlay method is already known in the field of Augmented Reality (AR) and Mixed Reality (MR), it is not described in detail herein.

Then, as shown in step S4 of fig. 1 and fig. 6, the knife-entering path generating module 333 in the processor 33 generates a virtual knife-entering point 43 ' corresponding to an actual knife-entering point 43 on the virtual scalp of the superimposed image 6 according to the surgical target point 42 and the image pictures presented on the superimposed image 6, connects the virtual knife-entering point 43 ' with the surgical target point 42 to form a line 44, and generates and displays a knife-entering guide path 45 extending from the virtual knife-entering point 43 ' along the line 44 outward from the virtual knife-entering point 43 ' in the direction of the virtual knife-entering point 43 ', as shown in fig. 7. Wherein the line 44 is a knife-in path, and the knife-in guide path 45 can be an elongated line or a cylinder-shaped, conically-extending, thick line.

The virtual cutting point 43' is generated as follows. The surgeon can determine an actual cutting point 43 on the surface of the patient's head directly according to habits or judgments, or determine an actual cutting point 43 with reference to the surgical target point 42 in the superimposed image 6 displayed by the information display device 3; after the actual knife entering point 43 is determined, as shown in fig. 6, when the surgeon uses the tip 71 of a surgical instrument 7 to point the actual knife entering point 43, the knife entering path generating module 333 generates a virtual knife entering point 43' corresponding to the actual knife entering point 43 on the virtual scalp of the superimposed image 6. Then, the knife entering path generating module 333 generates the knife entering guiding path 45 according to the operation target point 42 and the virtual knife entering point 43'; the surgeon may consider whether to change the cutting point according to the cutting guide path 45, and if so, re-determine a cutting point, so that the cutting path generation module 333 re-generates a cutting guide path 45 according to the surgery target point 42 and the new actual cutting point 43. It should be noted that the tool-entering path generating module 333 of the present embodiment may also introduce artificial intelligence to automatically generate the virtual tool-entering point 43' by a pre-constructed artificial intelligence model. The information display device 3 also records the three-dimensional virtual image 5, the superimposed image 6, and the cutting guide path 45 generated on the superimposed image 6, which are generated in the above steps.

Thus, as shown in step S21 of fig. 2, when the display 31 of the information display device 3 displays the superimposed image 6 and the straight line (knife entering path) 44 on the superimposed image 6 and the knife entering guide path 45 extending outward from the straight line (knife entering path) 44 as shown in fig. 7, as shown in steps S22 and 8 of fig. 2, the knife entering detection module 334 of the processor 33 determines (detects) whether a surgical instrument 7 appears in the superimposed image 6 displayed on the display 31 through the image frames captured by the image capturing unit 34, if so, the knife entering detection module 334 further determines whether a sharp end 71 ' (corresponding to the sharp end 71 of the surgical instrument 7) of the surgical instrument image 7 ' appearing in the superimposed image 6 is superimposed with the virtual knife entering point 43 ' as shown in steps S23 and 9 of fig. 2, and/or determining whether the surgical instrument image 7 '(corresponding to the surgical instrument 7) overlaps the knife-in guide path 45, and/or determining whether the spatial angle of the surgical instrument image 7' (corresponding to the surgical instrument 7) overlaps the spatial angle of the knife-in guide path 45, and/or determining whether the tip point 71 '(corresponding to the tip point 71 of the surgical instrument image 7') approaches the surgical target 42 along the straight line 44 (knife-in path) and overlaps the surgical target 42, and performing step S24 to output a feedback message according to the determination result.

Specifically, the knife entering detection module 334 can obtain a space coordinate information of the surgical instrument 7 operated by the surgeon or the related personnel in real time through the optical positioning system or the image positioning system, so as to obtain the position of the surgical instrument 7 in the space, further determine whether the tip point 71 'of the surgical instrument image 7' is overlapped with the virtual knife entering point 43 'or the surgical target point 42, determine whether the path or the space angle of the surgical instrument image 7' is overlapped with the knife entering guide path 45 or the space angle thereof, and timely output the corresponding feedback information according to the determination result.

In addition, it should be mentioned that the three-dimensional image constructing module 331 can also cooperate with the cutting-in detecting module 334, when the cutting-in detecting module 334 detects that the surgical instrument 7 appears in the superimposed image 6 frame displayed on the display 31, the three-dimensional image constructing module 331 can immediately construct a virtual surgical instrument (virtual object, not shown) corresponding to the surgical instrument 7, and the cutting-in detecting module 334 displays the virtual surgical instrument in the superimposed image 6 frame according to a relative coordinate of the surgical instrument 7 relative to the actual surgical target 1, so that the shape and position of the virtual surgical instrument in the image frame are equivalent to the surgical instrument image 7 ', and the display 31 can instantly display whether the tip point of the virtual surgical instrument is superimposed on the virtual cutting-in point 43' or the surgical target point 42, and instantly displaying whether the path or the space angle of the virtual surgical instrument is overlapped with the knife entering guide path 45 or the space angle thereof, so as to present the overlapping relation between the surgical instrument 7 and the knife opening target and the knife entering path in a virtual visualization way.

Wherein the feedback information includes but is not limited to at least one of video feedback, audio feedback, and tactile feedback, and the video feedback includes but is not limited to at least one of color change and new information; taking fig. 9 as an example, when the tip point 71 ' of the surgical instrument image 7 ' is overlapped with the virtual knife entering point 43 ', and the knife entering detection module 334 determines whether the surgical instrument image 7 ' is overlapped with the knife entering guide path 45, a circle 9 appears at the upper right of the display frame of the display 31 and has a center point 91 at the midpoint of the circle 9, when the knife entering detection module 334 determines that the surgical instrument image 7 ' is not overlapped with the knife entering guide path 45, a line segment 92 deviating from the center point 91 appears in the circle 9, the length of the line segment 92 represents the distance of the surgical instrument 7 deviating from the knife entering guide path 45, that is, the size of the length represents the deviation of the spatial orientation of the surgical instrument 7 from the spatial orientation of the knife entering guide path 45. The direction of deflection of the line segment 92 then represents the direction of deflection of the surgical instrument 7 away from the knife guide path 45.

On the contrary, as shown in fig. 10, when the knife-in detection module 334 determines that the surgical instrument image 7' is overlapped with the spatial orientation of the knife-in guide path 45, the line segment 92 deviating from the central point 91 does not appear in the circle 9, but only the central point 91 is displayed, which represents that the surgical instrument image 7 is overlapped with the spatial orientation of the knife-in guide path 45 without deviating from the knife-in guide path 45. Thus, the physician holding the surgical instrument 7 can adjust the surgical instrument 7 by simultaneously observing changes in the line segment 92 within the circle 9 so that the surgical instrument 7 is in the correct surgical orientation.

The audio feedback includes audio prompts, such as whether the tip 71 visually representing the surgical instrument 7 is overlapped with the actual knife entering point 43 or the surgical target point 42, whether the surgical instrument 7 is overlapped with the knife entering guide path 45 or the spatial angle thereof, and the like, to produce audio information, such as the processor 33 enabling the speaker of the information display device 3 to output voices such as "surgical instrument has deviated from the knife entering guide path", "surgical instrument approaches the knife entering guide path", and "surgical instrument is overlapped with the knife entering guide path", and the doctor who guides the surgical instrument 7 by outputting the voices instantly can adjust the surgical instrument 7 at any time so as to make the surgical instrument 7 be in the correct surgical position or position.

The tactile feedback is, for example, that the processor 33 transmits a feedback signal to a wearable electronic device worn by a surgeon in a wired or wireless manner through a signal transmission interface of the information display device 3, so that the wearable electronic device generates vibration, and the surgeon wearing the wearable electronic device can more intuitively check whether the surgical instrument 7 is in a correct surgical knife entering position, position and/or path. The wearable electronic device may be, but is not limited to, a smart phone placed in a doctor's pocket, a smart watch worn on the hand, a smart bracelet or a wearable Beacon (Beacon), a remote control stick (similar to a game Wii remote control stick) held by the other hand (left hand) of the doctor, a smart necklace hung on the doctor's neck, smart glasses worn on the doctor's head, and the like.

In summary, in the above embodiment, the three-dimensional virtual image 5 of the actual surgical object 1 is overlapped on the actual surgical object image 1 ', the virtual knife entering point 43' is generated according to the surgical target point 42 displayed by the target image picture and the actual knife entering point 43 determined on the surface of the actual surgical object 1, the knife entering path (straight line) 44 advancing from the virtual knife entering point 43 to the surgical target point 42 and the knife entering guide path 45 extending from the virtual knife entering point 43 to the outside straight line are generated, so that the operator can know the optimal knife entering path in advance, so as to accurately position the tip point 71 of the surgical instrument 7 at the actual knife entering point 43 along the knife entering guide path 45 during the surgery, and then the actual knife entering point 43 enters the actual surgical object 1 along the knife entering path 44 and guides the tip point 71 of the surgical instrument 7 to move to the surgical target point 42 correctly, therefore, the surgical operation path planning device helps an operator to plan a surgical path, is beneficial to smooth operation and improves the success rate of the operation, and really achieves the efficacy and the purpose of the invention.

The above description is only for the preferred embodiment of the present invention, and it is not intended to limit the scope of the present invention, and any person skilled in the art can make further modifications and variations without departing from the spirit and scope of the present invention, therefore, the scope of the present invention should be determined by the claims of the present application.