阅读说明：本技术 一种ar荧光远程医疗手术导航系统及其控制方法 (AR fluorescence telemedicine operation navigation system and control method thereof ) 是由 李明 揭光锦 顾兆泰 李娜娜 安昕 张浠 于 2021-02-05 设计创作，主要内容包括：本发明公开了一种AR荧光远程医疗手术导航系统及其控制方法,通过将荧光导航系统中的荧光图像信号投影至目标区域,将肉眼不可见的解剖结构或特殊组织检测信息转化为手术视野中可见的真实图像,为手术操作提供指导；利用远程医疗技术,实时共享手术现场状况和患者信息等,便于医学交流和专家会诊,而且可以实时以AR形式将远程端反馈的带有指导标记的荧光合成图像显示在目标区域中；通过采用投影的技术手段,使得无需在显示屏幕与目标区域频繁转换视线,带来更人性化的体验和更安全高效的手术效果。(The invention discloses an AR fluorescence remote medical surgery navigation system and a control method thereof, wherein fluorescence image signals in the fluorescence navigation system are projected to a target area, so that the detection information of an anatomical structure or a special tissue which is invisible to naked eyes is converted into a real image which can be seen in a surgery visual field, and guidance is provided for surgery operation; by using a remote medical technology, the operation site condition, the patient information and the like are shared in real time, medical communication and expert consultation are facilitated, and a fluorescent synthetic image with a guide mark fed back by a remote end can be displayed in a target area in real time in an AR form; by adopting the projection technical means, the sight line does not need to be frequently switched between the display screen and the target area, and more humanized experience and safer and more efficient operation effects are brought.)

1. An AR fluoroscopic telemedicine surgical navigation system, comprising:

the illumination module is used for generating near-infrared excitation light to irradiate the target area, and exciting the fluorescent contrast agent gathered on the target area to enable the fluorescent contrast agent to emit near-infrared fluorescence;

the camera module is used for carrying out fluorescence imaging on the near infrared fluorescence emitted by the target area to obtain a fluorescence image signal;

the image processing module is used for processing the fluorescence image signal to obtain a fluorescence synthesis image;

and the projection module is used for projecting the fluorescent composite image to a target area to realize the accumulation of the fluorescent composite image information and the actual scene.

2. The AR fluoroscopic telemedicine surgical navigation system of claim 1, further comprising a terminal display module for displaying the fluoroscopic composite image.

3. The AR fluoroscope telemedicine surgical navigation system of any of claims 1 or 2, further comprising a remote interaction module to communicate with a remote end in real time; and transmitting the fluorescence synthetic image to a remote end, and receiving the fluorescence synthetic image with the guide mark fed back by the remote end.

4. The AR fluorescence telemedicine surgery navigation system of claim 3, wherein the remote interaction module comprises a local workstation and a system center for storing various background information, the remote end comprises an expert workstation and an AR wearable device for enhancing realism, and the local workstation, the expert workstation and the system center form a telemedicine system; the local workstation is connected with expert's workstation communication, and expert's workstation is connected with the wearable equipment of AR, and local workstation and expert's workstation all are connected with system center.

5. The AR fluoroscope telemedicine surgical navigation system of claim 1, wherein the illumination module generates near infrared excitation light with a peak value of 780nm to illuminate a target area.

6. The AR fluorescence telemedicine surgical navigation system of claim 1, wherein the camera module comprises an achromatic lens, a notch filter and a camera, and near-infrared fluorescence emitted from the target area enters the camera module, is collected by the achromatic lens, and is filtered of excitation light by the notch filter, leaving the fluorescence imaged on the camera.

7. The method for controlling the AR fluorescence telemedicine surgical navigation system of any claim 1 to 6, comprising the following steps:

s1: the illumination module generates near-infrared excitation light to irradiate a target area, and excites the fluorescent contrast agent gathered on the target area to enable the fluorescent contrast agent to emit near-infrared fluorescence;

s2: the camera module performs fluorescence imaging on near-infrared fluorescence emitted by a target area to obtain a fluorescence image signal;

s3: the image processing module processes the fluorescence image signal to obtain a fluorescence synthesis image;

s4: and the projection module projects the fluorescence composite image to a target area to realize the accumulation of the fluorescence composite image information and the actual scene.

8. The method for controlling the AR fluoroscopic telemedicine surgical navigation system of claim 7, wherein in step S4, the projection module further projects the fluoroscopic composite image with the guidance mark fed back from the remote end received by the remote interaction module to the target area, so as to realize the accumulation of the fluoroscopic composite image information and the actual scene.

9. The control method of the AR fluoroscopic telemedicine surgical navigation system according to claim 7, further comprising the processes of: and the terminal display module displays the fluorescent synthetic image.

Technical Field

The invention relates to the technical field of a fluorescence navigation system, in particular to an AR fluorescence remote medical operation navigation system and a control method thereof.

Background

Modern medical images are increasingly widely applied in clinic, and provide a great deal of important basis related to biological tissue characteristics for diagnosis, treatment and prognosis of diseases (prognosis is a medical term and is divided into natural prognosis and intervention prognosis). The medical imaging technology in the surgical operation can provide relevant structural and functional information of a focus in the operation process, assist doctors to realize preoperative planning, intraoperative real-time guidance and postoperative evaluation prognosis, and improve the operation safety and accuracy. However, medical images of surgical operations are often transmitted through screen display, and users need to frequently switch the view lines between the screen and the operation area, which affects the efficiency and accuracy of the operation position determination.

With the rapid development of communication technology, the traditional work experience and process may be changed dramatically, for example, in the medical field, the remote medical system becomes a hot spot. Telemedicine is currently used for both conferencing and teaching purposes, and in surgical applications local surgeons also have to switch the line of sight between the surgical field and the monitor to follow the expert instructions, which is laborious and error prone.

Therefore, the prior art still needs to be improved and developed.

Disclosure of Invention

The invention aims to provide an AR fluorescence remote medical operation navigation system and a control method thereof, and aims to solve the problems that the existing medical image is displayed and transmitted through a screen, the sight of a user needs to be frequently switched between the screen and an operation area, and the efficiency and the precision of operation position judgment are influenced.

The technical scheme of the invention is as follows: an AR fluoroscopic telemedicine surgical navigation system, comprising:

the illumination module is used for generating near-infrared excitation light to irradiate the target area, and exciting the fluorescent contrast agent gathered on the target area to enable the fluorescent contrast agent to emit near-infrared fluorescence;

the camera module is used for carrying out fluorescence imaging on the near infrared fluorescence emitted by the target area to obtain a fluorescence image signal;

the image processing module is used for processing the fluorescence image signal to obtain a fluorescence synthesis image;

and the projection module is used for projecting the fluorescent composite image to a target area to realize the accumulation of the fluorescent composite image information and the actual scene.

The AR fluorescence telemedicine operation navigation system further comprises a terminal display module used for displaying the fluorescence synthetic image.

The AR fluorescence telemedicine operation navigation system also comprises a remote interaction module which is communicated with a remote end in real time; and transmitting the fluorescence synthetic image to a remote end, and receiving the fluorescence synthetic image with the guide mark fed back by the remote end.

The AR fluorescence telemedicine operation navigation system comprises a remote interaction module, a remote end and a remote control module, wherein the remote interaction module comprises a local workstation and a system center used for storing various background information, the remote end comprises an expert workstation and AR wearable equipment used for enhancing reality, and the local workstation, the expert workstation and the system center form a telemedicine system; the local workstation is connected with expert's workstation communication, and expert's workstation is connected with the wearable equipment of AR, and local workstation and expert's workstation all are connected with system center.

The AR fluorescence telemedicine operation navigation system is characterized in that the illumination module generates near-infrared excitation light with the peak value of 780nm to irradiate a target area.

The AR fluorescence telemedicine operation navigation system, wherein, camera module includes achromatic lens, notch filter and camera, and the near-infrared fluorescence of target area transmission gets into in the camera module, is collected by achromatic lens to filter the exciting light through notch filter, leave fluorescence imaging in the camera.

A control method of the AR fluorescence telemedicine operation navigation system comprises the following steps:

s1: the illumination module generates near-infrared excitation light to irradiate a target area, and excites the fluorescent contrast agent gathered on the target area to enable the fluorescent contrast agent to emit near-infrared fluorescence;

s2: the camera module performs fluorescence imaging on near-infrared fluorescence emitted by a target area to obtain a fluorescence image signal;

s3: the image processing module processes the fluorescence image signal to obtain a fluorescence synthesis image;

s4: and the projection module projects the fluorescence composite image to a target area to realize the accumulation of the fluorescence composite image information and the actual scene.

In the control method of the AR fluorescence telemedicine surgical navigation system, in step S4, the projection module further projects the fluorescence composite image with the guidance mark fed back from the remote end and received by the remote interaction module to the target area, so as to realize accumulation of the fluorescence composite image information and the actual scene.

The control method of the AR fluorescence telemedicine operation navigation system further comprises the following processes: and the terminal display module displays the fluorescent synthetic image.

The invention has the beneficial effects that: the invention provides an AR fluorescence remote medical operation navigation system and a control method thereof, wherein fluorescence image signals in the fluorescence navigation system are projected to a target area, so that detection information of an anatomical structure or a special tissue which is invisible to naked eyes is converted into a real image which is visible in an operation visual field, and guidance is provided for operation; by using a remote medical technology, the operation site condition, the patient information and the like are shared in real time, medical communication and expert consultation are facilitated, and a fluorescent synthetic image with a guide mark fed back by a remote end can be displayed in a target area in real time in an AR form; by adopting the projection technical means, the sight line does not need to be frequently switched between the display screen and the target area, and more humanized experience and safer and more efficient operation effects are brought.

Drawings

FIG. 1 is a schematic workflow diagram of the AR fluorescence telemedicine surgical navigation system of the present invention.

Fig. 2 is a schematic diagram of a telemedicine system of the present invention.

FIG. 3 is a flowchart illustrating the steps of a method for controlling an AR fluorescence telemedicine surgical navigation system according to the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.

In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate orientations and positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be considered as limiting the present invention. Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.

In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; may be mechanically connected, may be electrically connected or may be in communication with each other; either directly or indirectly through intervening media, either internally or in any other relationship. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

In the present invention, unless otherwise expressly stated or limited, "above" or "below" a first feature means that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact with each other via another feature therebetween. Also, the first feature being "on," "above" and "over" the second feature includes the first feature being directly on and obliquely above the second feature, or merely indicating that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature includes the first feature being directly under and obliquely below the second feature, or simply meaning that the first feature is at a lesser elevation than the second feature.

The following disclosure provides many different embodiments or examples for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are merely examples and are not intended to limit the present invention. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples, such repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. In addition, the present invention provides examples of various specific processes and materials, but one of ordinary skill in the art may recognize applications of other processes and/or uses of other materials.

As shown in fig. 1, an AR fluoroscopic telemedicine surgical navigation system includes:

the illumination module 1 is used for generating near-infrared excitation light to irradiate the target area 7, and exciting the fluorescent contrast agent gathered on the target area 7 to enable the fluorescent contrast agent to emit near-infrared fluorescence;

the camera module 2 is used for carrying out fluorescence imaging on the near-infrared fluorescence emitted by the target area 7 to obtain a fluorescence image signal;

the image processing module 3 is used for processing the fluorescence image signal to obtain a fluorescence synthesis image;

the terminal display module 4 is used for displaying the fluorescent synthetic image;

and the projection module 5 is used for projecting the fluorescence composite image to a target area 7 so as to realize the accumulation of the fluorescence composite image information and the actual scene.

In some embodiments, the AR fluoroscopic telemedicine surgical navigation system further includes a remote interaction module 6, which communicates with the remote end in real time; and transmitting the fluorescence synthetic image to a remote end, and receiving the fluorescence synthetic image with the guide mark fed back by the remote end.

The image processing module 3 transmits the fluorescence composite image to the remote interaction module 6 by using the ultra-high definition, low-delay and high-broadband video transmission technology, and the remote interaction module 6 transmits the fluorescence composite image to a remote end; an expert at a remote end can enter an operating room as if being personally on the scene through the AR technology to guide and mark the operation; the fluorescence composite image with the guide mark is transmitted back to the image processing module 3 through the remote interaction module 6, and is projected and accumulated to the target area 7 through the projection module 5 as well as the fluorescence information invisible to naked eyes.

Wherein, the remote interactive module 6 comprises a local workstation and a system center, the remote end comprises an expert workstation and an AR wearable device, and the local workstation, the expert workstation and the system center form a remote medical system (as shown in FIG. 2); the operating doctor logs in the telemedicine system through the local workstation to start the remote operation function, and the consultation specialist logs in the specialist workstation to connect with the teleoperation through the key; after the communication between the local workstation and the expert workstation is established, the consultation expert can see the situation of the operation site as the clinical reality by means of the reality enhancing capability of the AR wearable equipment, communicate with the main doctor in real time and guide the operation; meanwhile, the consultation specialist can call background information stored in the system center, such as patient medical record information, examination reports, historical consultation records, real-time monitoring information of the surgical equipment and the like, and the information is displayed in front of eyes through the AR enhancement technology in a three-dimensional mode, so that the consultation judgment and guidance are facilitated for the specialist.

In some embodiments, the local workstation includes, but is not limited to, a video live broadcast module, a user registration module, a user login module, a communication establishment module, a file upload module, a device information access module, an operation guidance delivery module, and a system help module, and the functions of each module are as follows:

the video live broadcast module: and 3, transmitting the fluorescent synthetic image to the matched and connected expert workstation in real time by using a 5G high-bandwidth low-delay video transmission technology, and simultaneously realizing the function of a remote video conference.

A user registration module: the user information registration function is realized, the basic personal information, the unit information, the machine ID information provided by manufacturers and the like of the user are contained, and the system management and maintenance are facilitated.

A user login module: the system realizes the login function of the user, provides various login modes such as passwords, fingerprints and face recognition, and is convenient for system safety management.

A communication establishing module: through key confirmation, the one-to-one connection function of the local workstation and the expert workstation is realized, and efficient and real-time expert consultation and technology sinking base level are conveniently established.

The file uploading module: and local doctors can upload medical record information, examination reports, historical consultation records and other files of patients.

The equipment information access module: and the information access of other monitoring equipment in the operating room is realized, and the information is transmitted to a remote expert workstation for display.

An operation guidance delivery module: and extracting the operation guidance mark information transmitted by the remote expert workstation, transmitting the operation guidance mark information to the image processing unit for superposition processing, and then projecting and accumulating the operation guidance mark information to an actual scene through the projection module.

A system help module: the method helps the user to quickly obtain answers to common questions, difficult and complicated answering prompts, operation execution descriptions, after-sale maintenance information of the equipment and the like.

In some embodiments, the expert workstation includes, but is not limited to, a live video module, a user registration module, a user login module, a communication establishment module, a file download module, a device information reading module, an operation guidance input module, and a system help module, and the like, wherein the functions of the modules are as follows:

the video live broadcast module: and (3) receiving and displaying the fluorescent composite image transmitted by the matched local workstation in real time by using a 5G high-broadband low-delay video transmission technology, and simultaneously realizing the function of a remote video conference.

A user registration module: the user information registration function is realized, the basic personal information, the unit information, the machine ID information provided by manufacturers and the like of the user are contained, and the system management and maintenance are facilitated.

A user login module: the system realizes the login function of the user, provides various login modes such as passwords, fingerprints and face recognition, and is convenient for system safety management.

A communication establishing module: through key confirmation, the one-to-one connection function of the local workstation and the expert workstation is realized, and efficient and real-time expert consultation and technology sinking base level are conveniently established.

A file downloading module: the remote expert doctor can download the files of patient medical record information, examination report, historical consultation record and the like.

A device information reading module: and the information of other monitoring equipment in the operating room transmitted by the local workstation is read and displayed on the expert workstation, so that the consultation and judgment of experts are facilitated.

An operation guidance input module: the consultation specialist can guide and mark the operation and transmit a guide signal to the local workstation for projection.

A system help module: the method helps the user to quickly obtain answers to common questions, difficult and complicated answering prompts, operation execution descriptions, after-sale maintenance information of the equipment and the like.

In some embodiments, the system center includes, but is not limited to, a basic information management module, a user information management module, an expert information management module, a case information management module, a communication establishment management module, an equipment information monitoring module, a file management module, a system management module, a rights management module, a log management module, a version management module, a system backup module, a system recovery module, and the like, wherein the functions of the modules are as follows:

a basic information management module: basic information such as equipment machine ID information, sales information, after-sales maintenance information, and top system administrator information of each surgical navigation system is managed.

The user information management module: and managing registered general doctor user information or other equipment user information.

The expert information management module: and managing the registered VIP big expert information.

A medical record information management module: and managing the uploaded medical record information, examination reports, historical consultation records and other information of all patients.

The communication establishment management module: the communication establishment of the local workstation and the expert workstation of each projection device is managed, the smooth establishment of the communication is ensured, and the situations of one-to-many or many-to-one and the like are prevented.

Equipment information monitoring module: when the communication is established, the monitoring information of the operating room equipment transmitted by each local workstation is monitored and backed up, so that the history backtracking of medical accidents and the like is facilitated.

A file management module: and managing the uploading and downloading of files such as patient medical record information, examination reports, historical consultation records and the like.

The system management comprises the following management modules:

the authority management module: and managing the use authority of each common user and VIP big expert.

A log management module: and managing system logs, recording all behavior information generated by the system, facilitating debugging of the system and optimizing system performance.

And a version management module: historical version information of the management system.

A system backup module: and various historical system versions are backed up, so that emergency situations can be conveniently dealt with, and the robustness of the system is enhanced.

A system recovery module: and the system is restored to the system with the designated historical version, so that the system safety is enhanced.

In some embodiments, the illumination module 1 generates near infrared excitation light with a peak value of 780nm to illuminate the target region 7, and excites the fluorescent contrast agent collected on the target region 7, so that the fluorescent contrast agent emits near infrared fluorescence with a peak value of 835 nm.

In some embodiments, the image capturing module 2 includes an achromatic lens, a notch filter and a camera, and near-infrared fluorescence emitted from the target area 7 enters the image capturing module 2, is collected by the achromatic lens, and is filtered by the notch filter to remove excitation light, leaving fluorescence to be imaged on the camera.

The camera transmits a fluorescence image signal to the image processing module 3, the image processing module 3 performs algorithm processing on the extracted fluorescence image signal, an ICG fluorescence composite image of target tissues such as blood vessels, lymph and the like is output to the terminal display module 4 in real time and is transmitted to the projection module 5, and the projection module 5 projects the fluorescence composite image to an observation tissue (namely, a target area 7) to realize accumulation of image information and an actual scene.

As shown in fig. 3, a control method of the AR fluorescence telemedicine surgical navigation system includes the following steps:

s1: the illumination module 1 generates near-infrared excitation light to irradiate the target area 7, so as to excite the fluorescent contrast agent gathered on the target area 7 and enable the fluorescent contrast agent to emit near-infrared fluorescence;

s2: the camera module 2 performs fluorescence imaging on the near-infrared fluorescence emitted by the target area 7 to obtain a fluorescence image signal;

s3: the image processing module 3 processes the fluorescence image signal to obtain a fluorescence synthesis image;

s4: the projection module 5 projects the fluorescence composite image to the target area 7, so as to realize accumulation of fluorescence composite image information and an actual scene.

In some embodiments, in step S4, the projection module 5 further projects the fluorescence composite image with the guide mark received by the remote interaction module 6 and fed back from the remote end to the target area 7, so as to realize the accumulation of the fluorescence composite image information and the actual scene.

In some embodiments, the control method of the AR fluoroscopic telemedicine surgery navigation system further includes the following processes: and the terminal display module 4 displays the fluorescence synthesis image.

In the description herein, references to the description of the terms "one embodiment," "certain embodiments," "an illustrative embodiment," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.

Reference numerals:

a lighting module 1; a camera module 2; an image processing module 3; a terminal display module 4; a projection module 5; a remote interaction module 6; a target area 7.