阅读说明：本技术 ***辅助装置、***辅助方法和包括***辅助装置的内窥镜装置 (Insertion assisting device, insertion assisting method, and endoscope device including insertion assisting device ) 是由 西村博一 于 2017-06-21 设计创作，主要内容包括：本发明的一个实施方式提供按照内窥镜的插入状况提示可靠的辅助信息的插入辅助装置(100)。插入辅助装置具有：图像输入部(112),按时间序列生成的多个内窥镜图像被输入至图像输入部(112)中；状况判断部(113),其基于多个内窥镜图像判断内窥镜插入的状况；和辅助信息提示部(114),其用于基于状况判断部的判断结果,从与内窥镜插入手法相关联地准备的多个辅助信息中提示至少1个辅助信息。本发明还能够提供一种插入辅助方法和包括插入辅助装置的内窥镜装置(1)。(An embodiment of the present invention provides an insertion assisting device (100) that presents reliable assisting information in accordance with the insertion state of an endoscope. The insertion assisting device includes: an image input unit (112) to which a plurality of endoscope images generated in time series are input; a situation determination unit (113) that determines the situation of endoscope insertion based on a plurality of endoscope images; and an auxiliary information presentation unit (114) for presenting at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with the endoscope insertion technique, based on the determination result of the situation determination unit. The present invention can also provide an insertion assisting method and an endoscope apparatus (1) including the insertion assisting apparatus.)

1. An insertion aid, comprising:

an image input unit to which a plurality of endoscope images generated in time series are input;

a situation determination unit that determines a situation of endoscope insertion based on the plurality of endoscope images that are input; and

and an auxiliary information presentation unit configured to present at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with the endoscope insertion technique, based on a determination result of the situation determination unit.

2. The insertion aid of claim 1, wherein:

the situation determination unit determines a situation of endoscope insertion based on image change information of the plurality of endoscope images.

3. The insertion aid of claim 1 or 2, wherein:

the plurality of endoscopic images are endoscopic images generated by performing a large intestine endoscopy,

the status determination unit determines an insertion status of the large intestine endoscopy.

4. An insertion aid according to any one of claims 1 to 3, wherein:

the determination result of the state determination unit includes a determination result of whether or not insertion difficulty occurs during insertion of the endoscope,

the auxiliary information presentation unit presents at least 1 piece of auxiliary information corresponding to a situation in which insertion is difficult when the situation determination unit determines that the situation in which insertion is difficult has occurred.

5. An insertion aid according to any one of claims 1 to 4, wherein:

the auxiliary information presenting section presents at least one of information on an operation of the endoscope and information on a direction in which the endoscope should be inserted as the auxiliary information.

6. An insertion aid according to any one of claims 1 to 5, wherein:

further comprising an insertion state input section to which information on an insertion state of the endoscope is input,

the auxiliary information presenting unit determines the state of endoscope insertion based on the information on the insertion state of the endoscope in addition to the plurality of endoscope images.

7. The insertion aid of claim 6, wherein:

the information on the insertion state of the endoscope includes at least one of information on an insertion shape of an insertion portion of the endoscope and information on an insertion length of the insertion portion of the endoscope.

8. An insertion aid as claimed in any one of claims 1 to 7, wherein:

when the status determination unit determines that the visibility is in the course of the route but the field of view is not changed, the auxiliary information presentation unit presents, as the auxiliary information: information for prompting a rigidity changing operation, a straightening operation, an angle operation, or a twisting operation of an insertion portion of the endoscope.

9. An insertion aid as claimed in any one of claims 1 to 7, wherein:

when the situation determination unit determines that the excessively close state continues to exist, the auxiliary information presentation unit presents, as the auxiliary information: information for prompting the pulling-back operation, the angle operation, or the twisting operation of the insertion portion of the endoscope.

10. An insertion aid as claimed in any one of claims 1 to 7, wherein:

when the situation determination unit determines that the forward and backward movement are repeated in the same range, the auxiliary information presentation unit presents, as the auxiliary information: information for prompting a rigidity changing operation, a straightening operation, an angle operation, or a twisting operation of an insertion portion of the endoscope.

11. An insertion aid as claimed in any one of claims 1 to 7, wherein:

when the status determination unit determines that the direction of the intestinal lumen is invisible and the visual field is also unchanged, the auxiliary information presentation unit presents, as the auxiliary information: information for prompting the air supply operation or the air exhaust operation of the endoscope.

12. An insertion assistance method, comprising:

inputting a plurality of endoscope images generated in time series;

a step of determining a status of endoscope insertion based on the plurality of endoscope images inputted; and

and presenting at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with the endoscope insertion technique based on the determination of the situation.

13. An endoscopic device, comprising:

an insertion aid according to any one of claims 1 to 11; and

an endoscope having an insertion portion, the insertion portion having a bending portion actively bendable and a flexible tube portion passively bendable connected to a proximal end side of the bending portion.

Technical Field

The present invention relates to an insertion assisting device and an insertion assisting method for assisting insertion of an endoscope, and an endoscope device including the insertion assisting device.

Background

For insertion of a large intestine endoscope, a system and a method for assisting insertion have been proposed. For example, japanese patent nos. 4885388 and 5687583 disclose an endoscope insertion direction detection method and an endoscope insertion direction detection device that detect the intestinal lumen direction as the endoscope insertion direction by analysis of an endoscope image. In particular, the endoscope insertion direction detecting apparatus disclosed in japanese patent No. 5687583 has a display unit that displays the detected endoscope insertion direction.

For example, japanese patent No. 4855902 discloses a system including a display control unit that determines whether or not to cause a display unit to display insertion assistance information for assisting an insertion operation, based on a display period control value calculated from latest analysis data and past analysis data of an insertion state.

For example, japanese patent No. 4855912 discloses an endoscope insertion shape analysis system capable of detecting hyperextension of the large intestine and the like and smoothly performing an insertion operation of an insertion portion.

Disclosure of Invention

The above-described system and method, although providing auxiliary information for coping with various problems for various difficult situations occurring in the large intestine endoscope insertion operation, do not consider when any problem occurs in the entire endoscopy and what auxiliary information is required by the operator.

An assisting system for assisting the insertion operation of the large intestine endoscope is preferably configured to find the type of problem at an appropriate timing and provide appropriate assisting information, as in the case where a skilled physician gives appropriate advice when observing the situation of an unskilled physician, for example. Therefore, the assistance system needs to identify the situation in which the unskilled physician is, and if the situation is problematic, select and provide appropriate assistance information.

Accordingly, an object of the present invention is to provide an insertion assisting device, an insertion assisting method, and an endoscope device including the insertion assisting device, which present reliable assisting information in accordance with an insertion state of an endoscope.

According to an embodiment of the present invention, there can be provided an insertion assisting device including: an image input unit into which a plurality of endoscope images generated in time series are input; a situation determination unit that determines a situation of endoscope insertion based on the plurality of endoscope images that are input; and an auxiliary information presentation unit configured to present at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with the endoscope insertion technique, based on a determination result of the situation determination unit.

According to another embodiment of the present invention, there can be provided an insertion assisting method including: inputting a plurality of endoscope images generated in time series to an image input unit of an insertion assisting apparatus; a situation determination section of the insertion assisting device determines a situation of endoscope insertion based on the plurality of endoscope images input; and an auxiliary information presentation unit of the insertion assisting device presents at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with an endoscope insertion technique based on the determination of the situation.

According to still another embodiment of the present invention, there can be provided an endoscope apparatus including: the insertion aid; and an endoscope having an insertion portion, the insertion portion having a bending portion actively bendable and a flexible tube portion passively bendable connected to a root end side of the bending portion.

According to the present invention, it is possible to provide an insertion assisting device, an insertion assisting method, and an endoscope device including the insertion assisting device, which can present reliable assisting information in accordance with the insertion state of an endoscope.

Drawings

Fig. 1 is a block diagram showing an example of an endoscope apparatus according to an embodiment of the present invention.

Fig. 2 is a diagram showing an example of an endoscope and an insertion shape observation apparatus according to an embodiment of the present invention.

Fig. 3 is a diagram showing an example of a loop pattern of the insertion portion.

Fig. 4 is a diagram showing an example of an insertion assistance flow.

Fig. 5 is a diagram showing an example of an insertion status, a determination that insertion is difficult, and auxiliary information.

FIG. 6 shows an example of an insertion method using a non-loop method (loop-free bending method).

Detailed Description

Embodiments of the present invention will be described below with reference to the drawings. Next, an endoscope apparatus including a large intestine endoscope will be described as an example.

Fig. 1 is a block diagram showing an example of an endoscope apparatus 1 according to an embodiment of the present invention. The endoscope apparatus 1 includes: an endoscope 10 (here, a large intestine endoscope), an insertion shape observation device 20, a light source device 30, an endoscope control device 40, an input device 50, a display device 60, and an insertion support control device 100.

Fig. 2 is a diagram showing an example of the endoscope 10 and the insertion shape observation device 20. The endoscope 10 includes: a tubular insertion portion 11 insertable into an inserted body (here, large intestine); and an operation portion 15 provided at the root end side of the insertion portion 11. The insertion portion 11 has: a distal end hard portion 12; a bending portion 13 connected to the root end side of the distal end hard portion 12; and a flexible tube portion 14 connected to the root end side of the bending portion 13. The distal end hard portion 12 includes an illumination lens and an objective lens, which are not shown, and an image pickup device 19 shown in fig. 1. The bending portion 13 is a portion bent by the operation of the operation portion 15, and the bent shape thereof can be actively changed. The flexible tube portion 14 is a flexible, elongated tubular portion that can be passively bent. The operation unit 15 is provided with an angle operation knob 16 for performing a bending operation of the bending portion 13, and 1 or more buttons 17 for performing various operations including air supply, water supply, and suction operations. The operator can bend the bending portion 13 in any direction by operating the angle operation knob 16. The operation unit 15 is further provided with 1 or more switches 18, and functions such as still image recording, focus switching, and the like of the endoscope image are assigned to the switches 18 by setting the endoscope control device 40.

The endoscope 10 is connected to the light source device 30. The light source device 30 includes a laser light source, an LED light source, a xenon lamp, a halogen lamp, or the like. The light source device 30 has a function of supplying illumination light to an illumination lens of the endoscope 10. The light source device 30 may have a function of supplying air and water into the body cavity through the endoscope 10. The light source device 30 includes, for example, an air supply pump for supplying air to the endoscope 10. The operator operates the button 17 of the operation section 15 of the endoscope 10 to supply air from a nozzle (not shown) at the distal end of the endoscope or to pressurize a water supply tank connected to the endoscope 10 to supply water from the nozzle at the distal end of the endoscope.

The endoscope apparatus 1 includes an insertion shape observing apparatus 20. Fig. 1 and 2 illustrate a magnetic sensor type insertion shape observation device 20 as an example. The insertion shape observation device 20 includes, for example, a control unit 21, a receiving unit 22, and a transmitting unit 23 in an observation device main body 24. The plurality of source coils 25 provided in the insertion portion 11 of the endoscope 10 and the antenna 26 provided separately from the observation device body 24 also belong to a part of the insertion shape observation device 20.

As shown in fig. 2, a plurality of source coils 25 are provided in the insertion portion 11. The source coil 25 is an element for generating a magnetic field. The source coils 25 are disposed at intervals in the longitudinal direction between the bending portion 13 and the flexible tube portion 14 in order to detect the bending shape of the insertion portion 11 in the longitudinal direction (axial direction). Fig. 2 shows a configuration in which the source coil 25 is mounted in the insertion portion 11 in advance, but a probe having a built-in source coil may be inserted into a channel extending in the longitudinal direction in the insertion portion 11. The source coil 25 may be disposed entirely or only a part of the flexible tube portion 14.

The antenna 26 is used to detect the magnetic field generated by the source coil 25. The antenna 26 is separate from the observation device body 24 and the endoscope 10, and is disposed around an insertion object into which the insertion portion 11 of the endoscope 10 is insertable, by a holder or the like not shown.

The control unit 21 of the insertion shape observation device 20 is a control circuit that performs various calculations and the like in the insertion shape observation device 20 and performs display control of the display device 60. The receiving unit 22 is a receiving circuit that receives the magnetic field detection signal from the antenna 26. The transmitter 23 is a transmission circuit that transmits a drive signal to the source coil 25. In particular, the control unit 21 performs an arithmetic process based on a position estimation algorithm based on the magnetic field detection signal from the antenna 26.

The insertion shape observing apparatus 20 is not limited to this. The insertion shape observation device 20 may be any device as long as it can detect the bent state of the insertion portion 11, and may be configured by 1 or a combination of sensors using a change in the amount of light (light intensity) or a change in optical characteristics transmitted through a light guide member such as an optical fiber (optical fiber sensor), a sensor using electromagnetic waves (electromagnetic sensor), a sensor using ultrasonic waves (ultrasonic sensor), a sensor using strain (strain sensor), and a sensor using an X-ray absorbing material.

The flexible tube characteristic changing portion 70 is provided in the flexible tube portion 14 of the endoscope 10, and the bending rigidity of the portion of the flexible tube portion 14 where the flexible tube characteristic changing portion 70 is provided can be partially changed. The flexible tube characteristic changing section 70 may be, for example, a rigidity variable actuator whose hardness changes in accordance with an applied voltage when the voltage is applied by the endoscope control device 40. Fig. 2 shows 1 flexible tube characteristic changing portion 70, but the number of flexible tube characteristic changing portions 70 is not limited thereto. That is, a plurality of flexible tube characteristic changing portions 70 may be provided along the longitudinal direction of the flexible tube portion 14.

The endoscope control device 40 includes a control section 41, a drive section 42, and an image processing section 43. The insertion support control device 100 includes an insertion state input unit 111, an image input unit 112, a situation determination unit 113, and a support information presentation unit 114. The endoscope control device 40 and the insertion support control device 100 may be configured by a processor such as a CPU. That is, the aforementioned parts of the endoscope control device 40 and the insertion assistance control device 100 may be constituted by a processor such as a CPU. In this case, for example, various programs for causing the processor to function as each section are prepared in advance in an internal memory or an external memory, which are not shown, and the processor functions as each section of the control devices 40 and 100 by executing the programs. Alternatively, each part of the control devices 40 and 100 may be configured by a hardware Circuit including an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

In the present embodiment, the above-described respective parts of the insertion assistance control device 100, that is, the insertion state input part 111, the image input part 112, the situation determination part 113, and the assistance information presentation part 114 are included in the insertion assistance control device 100 configured separately from the endoscope control device 40 as the endoscope video image processor. However, the present invention is not limited thereto. That is, the processor or the hardware circuit functioning as each part of the insertion assistance control device 100 may be provided in 1 casing or a plurality of casings as long as it can function as each part. Of course, the endoscope control device 40 may be provided.

In the endoscope control device 40, the control unit 41 is a control circuit that controls various operations of the endoscope 10. The driving unit 42 is a driving circuit that transmits a signal for driving the image pickup device 19 of the endoscope 10. The image pickup element 19 can convert an optical signal into an electric signal. The image processing unit 43 is an image signal processing circuit that converts the electric signal converted by the image pickup device 19 into a video signal. The control unit 41 also performs dimming control of the light source device 30.

The insertion state of the insertion portion 11 is input with the bending shape information (bending angle, bending amount, curvature radius, etc.) of the insertion portion 11 from the insertion shape observation device 20 to the insertion state input portion 111 of the insertion assistance control device 100. Endoscope image information is input from the endoscope control apparatus 40 to the image input section 112. The status determination unit 113 determines the insertion status of the insertion unit 11 based on the information from the insertion status input unit 111 and the image input unit 112. The auxiliary information presentation unit 114 generates insertion auxiliary information based on the determination made by the situation determination unit 113 and displays the insertion auxiliary information on the display device 60.

The input device 50 is a commonly used input device such as a keyboard. Various commands for operating the endoscope apparatus 1 and the like are input to the input device 50. The input device 50 may be an operation panel provided in the endoscope control device 40 or a touch panel displayed on a display screen. The observation device body 24 and the insertion assistance control device 100 of the insertion shape observation device 20 are also provided with an input unit such as an operation panel or a touch panel, but these are not illustrated here.

The display device 60 is a conventional monitor such as a liquid crystal display. The display device 60 displays the endoscopic image subjected to the image processing by the image processing unit 43 under the display control by the control unit 41 of the endoscope control device 40. The display device 60 may display a three-dimensional image, character information, and the like relating to the curved shape of the insertion portion 11 based on the curved shape information from the insertion shape observation device 20 under display control by the control portion 21 of the insertion shape observation device 20. The display device 60 also displays the insertion assistance information generated by the assistance information presentation unit 114 of the insertion assistance control device 100 under the display control performed by the insertion assistance control device 100. The display device for displaying the endoscopic image, the display device for displaying the curved shape, and the like, and the display device for displaying the insertion assisting information may be the same or different from each other. That is, the insertion assistance information may be displayed on a display device different from the display device 60.

Next, a description will be given of a large intestine endoscope insertion operation which is normally performed by the endoscope apparatus 1.

The insertion portion 11 of the endoscope 10 of the endoscope apparatus 1 is inserted into the rectum and colon from the anus by the operator. The endoscope 10 converts light from an object in the intestinal tract into an electric signal by the image pickup device 19 of the distal end hard portion 12. The electric signal is transmitted to the endoscope control device 40. The image processing unit 43 of the endoscope control device 40 acquires the electric signal and performs processing for converting the acquired electric signal into a video signal. The control unit 41 causes the display device 60 to display an endoscopic image based on the video signal.

During the insertion, the control unit 21 of the insertion shape observation device 20 causes the transmission unit 23 to supply drive signals of different frequencies to the source coils 25. In this way, each source coil 25 generates a weak alternating magnetic field around it. That is, information on the position thereof is output from each source coil 25. The antenna 26 detects the magnetic field generated by the source coil 25 and transmits a detection signal to the receiver 22. The control unit 21 calculates the position coordinates of each source coil 25 based on the magnetic field detection signal received by the receiving unit 22, generates, for example, a three-dimensional image of the insertion unit 11 corresponding to the calculated position coordinates, and displays the three-dimensional image on the display device 60.

The large intestine endoscope insertion technique is difficult, and it takes time to learn the technique. For example, it is considered that the operator should go through 1000 cases to reach the cecum with a cecum arrival rate of more than 90% at the tip of the insertion portion. In general, when a non-skilled physician performs an examination during training of a procedure, a skilled physician (or a mentor) is required to perform guidance individually at the site. Endoscopy by unskilled physicians is time consuming and sometimes painful for the patient. And, it takes time for a skilled physician. Furthermore, many hospitals do not necessarily have a skilled physician who can guide endoscopy.

When the large intestine endoscope is inserted, various cases occur in which the insertion is difficult. For example, the operator needs to find the intestinal lumen direction again when getting lost the intestinal lumen direction that should be inserted, but this is not easy for a non-skilled physician. Further, there are cases where the insertion portion 11 inserted into the body cavity is bent or formed into various loop curves, and the operator does not advance the distal end of the insertion portion even if pushing the insertion portion 11 inward from the manipulator side, or the patient feels pain by applying a force from the insertion portion 11 to the intestinal wall to stretch it. For example, the sigmoid colon is a movable intestinal tract that is not fixed in the abdomen, and therefore, when a force is applied from the insertion portion 11 to the top of the sigmoid colon (so-called S-top), which is a curved portion of the sigmoid colon, to cause hyperextension, pain may be felt by the patient.

In such a situation, it is effective to eliminate the bending of the insertion portion 11 and to release the loop shape (for example, referred to as "straightening of the endoscope"). However, it is difficult for an operator who is not a skilled physician to accurately judge the situation and perform an appropriate operation at an appropriate timing.

For example, as a large intestine endoscope insertion technique, a method is known in which the insertion portion 11 is inserted by concentrically shortening and straightening the intestinal tract by intussuscepting the intestinal tract. This method is called Right turn shortening (Right turn shortening). The shaft holding shortening method and the Hooking corrugation method (Hooking the fold) are similar methods. In this procedure, the insertion section 11 is linearly advanced from the rectum to the S-top, and is formed into a rod shape when folded over the S-top. The straight line portion of the insertion portion 11 in the rod shape is referred to as a rotation axis. In the right turn shortening method, the S-top is shortened by combining the operation of withdrawing the insertion part 11 to the manipulator side and the operation of twisting the rotation shaft clockwise, and the sigmoid colon is intussuscepted and straightened.

As shown in fig. 3, the loop shape pattern of the intestinal tract 200 (e.g., sigmoid colon) at the time of insertion includes α loop shape, N loop shape, γ loop shape, etc. in order to insert the insertion portion 11 into the sigmoid colon by the Right turn shortening method (Right turn shortening), the N loop shape of the Right turn shaft is set as the initial loop shape pattern, from this state, α loop shape is formed when the insertion portion is turned Left (counterclockwise), γ loop shape (reverse α loop shape) is formed when the insertion portion is turned Right (clockwise), even if α loop shape and γ loop shape are formed, the loop shape can be released by the Right turn shortening method, the insertion can be advanced while nesting the sigmoid colon, the N loop shape can be reversed to form N loop shape when the insertion portion is turned Left, the reverse loop shape can be released while the reverse loop shape is turned Right (clockwise), the reverse loop shape can be formed when the insertion portion is turned Left (counterclockwise), the loop shape can be released from this state, the reverse loop shape can be formed when the loop shape is turned Left (clockwise), the loop shape is formed, the loop shape can be released, the loop shape can be pushed back when the insertion portion is rotated counterclockwise, the loop shape is formed, the loop shape is pushed, the Left, the loop shape is pushed by the Left turn forming operation (clockwise), and the Left turn operation is pushed, the Left turn operation of the loop shape is pushed, the Left turn of the loop shape.

Thus, the large intestine endoscope insertion method differs according to the loop shape pattern. The loop may be generated not only in the sigmoid colon but also in different parts such as the transverse colon. Therefore, there are various large intestine endoscope insertion techniques. Under such circumstances, it is desired to shorten the time required for an unskilled physician to learn a large intestine endoscopic insertion procedure, to improve learning efficiency, and to realize safer endoscopy. In this regard, the present embodiment provides the insertion assistance by the insertion assistance control device 100.

Next, an example of the endoscope insertion assistance according to the present embodiment will be described. Fig. 4 is a diagram showing an example of the insertion assistance flow according to the present embodiment.

In step S101, the insertion assistance control device 100 determines whether or not the insertion assistance is ended, that is, whether or not an insertion assistance end instruction is input. The termination of the insertion assistance is determined based on, for example, an end instruction (input from the outside) made when the distal end of the insertion portion reaches the cecum. When the operator wants to end the insertion assistance, the operator may input an end instruction from an input unit or the like, not shown, of the insertion assistance control apparatus 100 based on the endoscope image acquired by the endoscope 10, the curved shape of the insertion portion 11 acquired by the insertion shape observation apparatus 20, or the hand feeling of the operator. If it is determined that the insertion assist is to be ended (Yes), the process is ended. If it is determined that the insertion assist is not to be ended (No; No), the process proceeds to step S102.

In step S102, a situation determination is made as to the insertion situation of the insertion portion 11. A plurality of endoscope images generated in time series are input from the endoscope control apparatus 40 to the image input unit 112 of the insertion assistance control apparatus 100. For example, the endoscope 10 captures an endoscope image at 30fps, and the endoscope image is input from the endoscope control device 40 to the image input section 112. Of course, the input endoscopic image is not limited to this, and the number of images in 1 second period may be reduced to less than 30 frames by sampling. The input image is stored in a predetermined number, for example, 300 frames or more in an internal memory (or an external memory) in the image input unit 112 or the status determination unit 113, or in an internal memory (or an external memory), not shown, provided in the insertion assistance control device 100. The status determination unit 113 determines the endoscope insertion status based on the endoscope image input and stored in the image input unit 112. For example, the situation determination unit 113 performs situation determination using 300 frames of images, which are images 10 seconds before the current time.

Information on the insertion state of the endoscope 10 may be input from the insertion shape observation device 20 to the insertion state input unit 111 of the insertion assistance control device 100. The situation determination unit 113 may be configured to determine the endoscope insertion situation based on not only the endoscope image input and stored in the image input unit 112 but also the information on the insertion state input to the insertion state input unit 111.

For example, when an insertion operation (other than observation, treatment, and/or treatment) is performed in a large intestine endoscopy, the situation shown in the left column of fig. 5 may occur. The situation determination unit 113 determines which situation has occurred based on the endoscope image or based on the endoscope image and information about the insertion state. For example, the status determination unit 113 determines the endoscope insertion status based on the 300-frame endoscope image.

1. Without major problems, is being inserted

The situation determination unit 113 determines that the insertion unit 11 is being inserted without any significant problem when it is detected that the plurality of endoscopic images (imaging scenes) input to the image input unit 112 continuously change for a predetermined time (determination result 1). This is a situation in which the insertion portion 11 is advancing smoothly in the intestinal tract. Alternatively, in addition to the above-described aspect, the situation determination unit 113 may be configured to determine that the determination result is 1 when it is detected that the insertion shape and the insertion length of the insertion portion 11 continuously change within a predetermined time period based on the information on the insertion state of the endoscope 10.

2. The course of the intestinal cavity and the like can be seen but the visual field is not changed

The situation determination unit 113 determines that the course of the intestinal lumen or the like is visible but the field of view is not changed when it detects that the course of the intestinal lumen or the like is visible in the plurality of endoscopic images input to the image input unit 112 but the change of the image is slight (the degree to which the orientation of the distal end of the insertion portion is slightly changed) or the image is hardly changed (determination result 2). Alternatively, the situation determination unit 113 may be configured to determine the determination result 2 when it is detected from the information on the insertion state of the endoscope 10 that the change (amount of movement) on the distal end side of the insertion unit 11 is smaller than the operation (pushing-in operation or pulling-out operation) on the manipulator side of the insertion unit 11. In the case where the change of the tip side of the insertion portion 11 is small compared to the pushing-in operation or the pulling-out operation on the manipulator side, it is considered that the insertion portion 11 is bent or loop-like bent. The situation determination unit 113 may be configured to determine that the endoscope is the determination result 2 when the bending portion itself or the loop itself is detected based on the information on the insertion state of the endoscope 10.

3. The state of excessive approach persists

The situation determination unit 113 determines that the excessively close state continues when it is detected that the plurality of endoscopic images input to the image input unit 112 remain in the excessively close state or the red state for a predetermined time (for example, 10 seconds), that is, when it is detected that the 300-frame image is in the red state, for example (determination result 3). Here, the red state is a state in which the tip of the insertion portion is in contact with the mucous membrane and the entire image is reddish due to blood flow in the submucosal layer.

4. Repeating forward/backward in the same range (scene)

The situation determination unit 113 determines that, when it is detected that, of the plurality of endoscope images formed in time series input to the image input unit 112, an image substantially identical to the time t is captured at the time t + α, the insertion unit 11 returns to the position and situation at the time t due to, for example, a pull-back or the like (determination result 4) when it is determined that the images are repeatedly moved forward and backward in the same range (scene).

5. The trend of the intestinal cavity and the like cannot be seen due to the shrunken intestinal tract, and the visual field is not changed

When it is recognized that the intestinal tract is deflated from the plurality of endoscopic images input to the image input unit 112 and a slight change in the image (a slight change in the orientation of the distal end of the insertion portion) or almost no change in the image is detected, the situation determination unit 113 determines that the intestinal tract is deflated, and the direction of the intestinal lumen or the like is not visible, and the field of view is not changed (determination result 5).

6. Others (impossible to judge, excluded from insertion auxiliary objects, etc.)

If any of the above-described determination results 1 to 5 is not satisfied, the status determination unit 113 determines that it is a determination result 6. This applies to the case where it is impossible to judge or excluded from the insertion of the auxiliary object, or the like.

As described above, of the determination results 1 to 6, the determination results 1 and 2 may be determination results obtained based on the endoscope image input to the image input section 112, determination results obtained based on the information on the insertion state input to the insertion state input section 111, or determination results obtained based on a combination thereof, and the determination results 3 to 6 may be determination results obtained based on the endoscope image input to the image input section 112. By using the information on the insertion state for the situation determination, more accurate recognition can be performed.

In the above-described situation determination based on the endoscopic images, when the situation determination unit 113 performs the situation determination, for example, detection of an amount of change based on the correlation between the endoscopic images, scene recognition of the images, or a combination thereof may be used. The situation determination unit 113 performs situation determination by recognizing continuity and repetition of the same scene, presence or absence of important structures such as intestinal lumen and folds, and the like.

The situation determination unit 113 can use a learning result using a deep neural Network technology such as a recurrent neural Network (recurrent neural Network) or a 3D-CNN. Thus, it is possible to perform content recognition such as detection of a change amount based on the correlation between the endoscopic images and scene recognition of each image by learning. The situation determination unit 113 performs situation recognition (application of motion recognition) on the multi-frame image using, for example, a recurrent neural network. The situation determination unit 113 may perform determination using an image string of a plurality of frames (for example, 150 frames in 30 frames × 5 seconds, or sampling at intervals of several frames) and recognition based on learning for classifying situations (classification of determination results 1 to 5). Such a technique may be, for example, a technique disclosed in Noriki Nishida, high deki Nakayama in Pacific-Rim Symposium on Image and Video Technology (PSIVT 2015), "Multimodesty recognition using multi-stream recognition Neural Network" or "connected-current Neural Network による self " (self-movement recognition based on connected-current Neural Network) "published by Shenlongs et al in drawings センシングシンポジウム, 2016(Symposium on sensing Image Information, 2016).

As described above, in step S102, the situation determination unit 113 determines which of the determination results 1 to 6 the situation of endoscope insertion is based on the plurality of endoscope images generated in time series input to the image input unit 112 or on the plurality of endoscope images and the information on the insertion state input to the insertion state input unit 111.

In the next step S103, the situation determination unit 113 determines whether or not a situation in which insertion is difficult is encountered, that is, whether or not auxiliary information should be presented to the operator, based on the situation determination results (determination results 1 to 6) in step S102. The middle column of fig. 5 shows the result of determination (yes/no) as to whether or not insertion is difficult in step S103. In the case of the determination result 1, the insertion can be performed without a serious problem, and therefore, the insertion is not difficult. In the case of the determination result 6, the insertion is also considered to be difficult. Therefore, if the determination results are 1 and 6, the situation determination unit 113 determines in step S103 whether or not the situation is difficult to insert (no), and returns to step S101. In the case of any of the determination results 2 to 5, the situation determination unit 113 determines that the insertion portion 11 of the endoscope 10 cannot be smoothly inserted even if the insertion is continued in the current state, and is in a situation in which the insertion is difficult (yes). And then proceeds to step S104.

In step S104, the auxiliary information presentation unit 114 generates auxiliary information corresponding to the result of the situation determination in step S102. The auxiliary information presentation unit 114 has, for example, a database in which information to be presented (for example, character information, image information, or a combination thereof) is associated with the determination result of the situation determination unit 113, and selects presentation information corresponding to the determination result. That is, the auxiliary information presentation unit 114 presents at least 1 piece of auxiliary information from a plurality of pieces of auxiliary information prepared in association with the endoscope insertion technique based on the determination result of the situation determination unit 113. Then, in step S105, the auxiliary information presentation unit 114 causes the display device 60 to display the generated auxiliary information.

In the present embodiment, the determination results 2 to 5 are situations in which some problems occur during the large intestine endoscope insertion operation and it is considered that it is difficult for the operator to continue the insertion operation without any assistance. Next, representative problems of the determination results 2 to 5 and auxiliary information that can effectively solve the problems will be described. The auxiliary information is information corresponding to an operation performed based on experience when an operator who is experienced in large intestine endoscopy is faced with a situation in which insertion is difficult, for example. The right column of fig. 5 shows an example of the auxiliary information for the determination results 2 to 5.

Auxiliary information for determination result 2

When the direction of the intestinal lumen or the like is visible but the field of view is not changed, it is considered that the pushing force cannot be transmitted to the distal end of the insertion portion even if the operator performs the operation of pushing the insertion portion 11 back from the hand side. Therefore, the auxiliary information presentation unit 114 generates, for example, auxiliary information for prompting the rigid body changing operation and increasing the bending rigidity of the flexible tube portion 14 by the flexible tube characteristic changing unit 70. When the bending rigidity of the flexible tube portion 14 is increased, the flexible tube portion 14 is less likely to be bent, and therefore, the pushing force of the operator pushing the insertion portion 11 inward from the manipulator side is easily transmitted to the insertion portion distal end. Therefore, the insertion portion tip is easily advanced.

Alternatively, the assistant information presenting unit 114 generates assistant information for prompting the straightening of the endoscope. The auxiliary information for prompting the operator to straighten the endoscope may be a PULL-in (PULL) operation for pulling back the insertion portion 11 from the operator's hand side or the loop releasing operation described with reference to fig. 3. According to such auxiliary information, the operator who has previously performed the PUSH (PUSH) operation of pushing the insertion portion 11 inward from the manipulator switches the PUSH operation to the pull-back operation or the loop releasing operation. In this way, the insertion situation can be improved.

In consideration of the situation where the distal end of the insertion portion cannot be smoothly oriented in the intestinal lumen direction even when the direction of the intestinal lumen or the like is visible, the auxiliary information presentation portion 114 derives the operation direction in which the bending portion 13 is bent at a certain angle or the operation direction in which the insertion portion 11 is twisted, and takes the derived operation direction as the auxiliary information. Based on the auxiliary information, the operator bends the bending portion 13 by a predetermined angle in the derived operation direction or twists the insertion portion 11 from the operation hand side. By adopting such a manner, the insertion condition can be improved.

Auxiliary information for determination result 3

When the tip of the insertion portion is too close to the mucosal surface, the skilled practitioner, based on experience, pulls the insertion portion 11 from the manipulator side once to separate the tip of the insertion portion from the mucosal surface, and further operates the angle operation knob 16 of the manipulator 15 to bend the bending portion 13 at a certain angle or to turn the insertion portion 11 clockwise from the manipulator side to search for the intestinal lumen direction. On the other hand, an unskilled physician may sometimes be concerned that the inserted insertion portion 11 is pulled out and the appropriate retraction operation cannot be performed. In fact, even if slight detachment occurs, the shape of the large intestine up to the site of once successful insertion is already well suited to the shape of the insertion portion 11, and therefore reinsertion is often easy. Therefore, in the case of the determination result 3, it is useful to clearly remind the user of the pulling operation of the insertion portion 11 as the auxiliary information. The auxiliary information presentation unit 114 generates auxiliary information for prompting the pull operation. The operator can improve the insertion state by performing the pull operation based on the auxiliary information.

As shown in fig. 6, as one of the large intestine endoscope insertion methods, for example, a Non-loop method is known in which a loop of the insertion portion 11 is not formed and the insertion portion 11 is advanced without bending the intestinal tract 200. This method transmits torque to the intestine at the bent portion 11a of the insertion portion 11 by twisting the insertion portion 11 clockwise from the manipulator side. Then, the orientation of the crank-shaped bent intestinal tract is changed by slightly pulling out the insertion portion 11 toward the manipulator side, and the angle of the bent portion 13 is eliminated without colliding with the mucous membrane. Then, when the angle is removed so that the insertion portion 11 does not come off after the intestinal tract is rotated, the intestinal tract can be straightened. In the case of the determination result 3, the auxiliary information presentation unit 114 may present auxiliary information for smoothly performing an insertion operation by the Non-loop method, for example.

Auxiliary information for determination result 4

When the same range (scene) is repeatedly advanced/retreated, it is considered that the pushing force pushing the insertion portion 11 back from the manipulator side may not be sufficiently transmitted to the insertion portion distal end, similarly to the determination result 2. Therefore, it is useful to remind to change the bending rigidity or to straighten the endoscope. The auxiliary information presentation unit 114 generates auxiliary information for prompting, for example, a rigidity changing operation to be performed and for improving the bending rigidity of the flexible tube portion 14 by the flexible tube characteristic changing unit 70. Thus, the flexible tube portion 14 is less likely to be flexed, and the pushing force of the operator pushing the insertion portion 11 inward from the hand side can be transmitted to the distal end of the insertion portion, which facilitates the advancement of the distal end of the insertion portion. Alternatively, the auxiliary information presentation unit 114 generates auxiliary information for prompting the endoscope to be straightened (a pull-back operation, a loop releasing operation, and the like). The operator can improve the insertion state by performing the operation based on the auxiliary information.

For example, when the insertion operation is performed by the right-turn shortening method as described above, there is a case where, when the operation for shortening or straightening the intestinal tract is performed, the operation cannot be performed smoothly and the forward/backward movement is repeated in the same range (scene). This also belongs to the judgment result 4. In this case, the auxiliary information presentation unit 114 generates auxiliary information for presenting an operation of attempting to shorten or straighten the intestinal tract after the insertion unit 11 is inserted deeper, or generates auxiliary information for presenting that the operation of shortening or straightening is abandoned and switched to insertion by a PUSH (PUSH) operation, for example. The operator can improve the insertion state by performing the operation based on the auxiliary information.

Auxiliary information for determination result 5

In a situation where the intestinal lumen is deflated, it is sometimes necessary to perform an operation of blowing air (or water) into the endoscope 10 to appropriately open the intestinal lumen. On the other hand, it is known that insertion becomes more difficult when or after the intestinal tract is inflated due to excessive air supply. Therefore, it is preferable to prompt the opening of the intestinal lumen by air supply and stop air supply at a timing when an appropriate air supply amount is reached and the intestinal lumen can be identified.

In this way, in the case of the determination result 5, the auxiliary information presentation unit 114 generates auxiliary information for prompting, for example, that the intestinal lumen is opened by air supply and then the air supply is stopped at a timing when the appropriate air supply amount is reached and the intestinal lumen can be identified. The operator can improve the condition of the deflated intestinal lumen by performing the operation based on the auxiliary information. It is also useful to remind the user of the remaining air after the intestinal lumen is opened and the insertion portion 11 passes. The auxiliary information presentation unit 114 may generate auxiliary information for reminding that the gas is discharged after passing. Thus, the condition in which the intestinal lumen is deflated can be improved, and the insertion can be facilitated.

As is apparent from the determination result 5, the number of pieces of auxiliary information generated or selected by the auxiliary information presentation unit 114 for each of the determination results 2 to 5 is not limited to 1. The auxiliary information presentation unit 114 may generate or select a plurality of auxiliary information. The plurality of auxiliary information may be presented on the display device 60 at the same time, or may be presented on the display device 60 in sequence.

After step S105, the process returns to step S101, and the insertion assistance control device 100 determines whether or not to end the insertion assistance. If it is determined that the insertion assist is to be ended (Yes), the process ends. When it is determined that the insertion assist is not ended (No), the process repeats steps S102 to S105.

As described above, in the present embodiment, the auxiliary information presentation unit 114 presents at least 1 piece of auxiliary information from the plurality of pieces of auxiliary information prepared in association with the large intestine endoscope insertion technique based on the determination result of the situation determination unit 113. As the large intestine endoscope insertion technique, there are a technique of advancing the distal end of the insertion portion along the curved shape of the intestinal tract mainly by pushing (so-called loop insertion technique), a technique of inserting the insertion portion while shortening the intestinal tract while holding the longitudinal axis of the insertion portion (so-called axis holding shortening technique, the right turn shortening technique described above is the same as this technique), and the like. In the present embodiment, the auxiliary information presentation unit 114 can select and present appropriate auxiliary information from a plurality of pieces of auxiliary information prepared in association with the above methods, in accordance with the determination result of the insertion status.

According to the present embodiment, when the operator is faced with a situation in which the endoscope is difficult to insert when performing the endoscope insertion operation, the auxiliary information corresponding to the content difficult to insert can be presented in time. Therefore, it is possible to provide the operator with the support information according to the advice of the instructor even in a situation where the instructor is not present, and it is possible to obtain the support effect of performing the endoscopic examination more smoothly and assisting the skilled endoscope insertion.

In particular, when performing an endoscopic insertion operation of a large intestine, since a movable intestinal tract such as sigmoid colon is easily moved and has a curved portion such as S-top, a less experienced operator is likely to fall into a situation where insertion is difficult. In contrast, in the present embodiment, various difficult situations and coping methods assumed when performing an endoscope insertion operation for the large intestine are associated with each other and prepared in advance as auxiliary information in the database of the auxiliary information presentation unit 114, and therefore, it is possible to provide timely assistance and support when an assumed difficult situation occurs. By performing a large intestine endoscopy after the operator receives appropriate assistance and support, the pain of the patient due to excessive force applied to the intestinal tract from the insertion portion 11 can be reduced, or the examination time can be shortened.

The status determination unit 113 may perform status determination based on a plurality of endoscope images formed in time series input to the image input unit 112, and based on the plurality of endoscope images and information on the insertion state input to the insertion state input unit 111, and may perform status determination using detection information detected by various sensors including: a sensor that detects the amount of movement of the insertion portion 11 that is moved by a pushing operation or a pulling operation, a sensor that detects the amount of rotation of the insertion portion 11 that is rotated clockwise or counterclockwise, a sensor that detects the amount of bending of the bending portion 13 that is bent by an angle operation of the angle operation knob 16, a sensor that detects the amount of force applied to the insertion portion 11, and the like. By adopting the above-described method, more accurate situation determination can be performed, and reliable auxiliary information can be provided.

For example, when a plurality of sensors for detecting the force applied from the outside (for example, intestinal tract) to the insertion portion 11 are arranged along the longitudinal direction of the insertion portion 11, the situation determination portion 113 can use the force detected by each sensor for situation determination in addition to the endoscope image information when performing situation determination. By performing such determination, in the case of determination result 2 (the course of the intestinal lumen or the like is visible but the visual field is not changed), it can be more accurately determined whether or not determination result 2 is caused by the formation of loop flexure.

The auxiliary information presented by the auxiliary information presenting unit 114 is not limited to visual information, and may be auditory auxiliary information such as a sound emitted from a speaker not shown or the like, or combination information of visual auxiliary information and auditory auxiliary information.

The situation determined by the situation determination unit 113 is not limited to a state in which insertion is difficult. The situation determination unit 113 may detect and determine an excessively fast push-in operation, angle operation, or the like using the sensor or the like. In this case, the auxiliary information presented by the auxiliary information presenting unit 114 is information for suppressing such an operation.

The above description has been given by way of example of an endoscope apparatus including a large intestine endoscope, but the present invention is not limited to this. The concept of the present invention can also be applied to a lower gastrointestinal endoscope or an upper gastrointestinal endoscope other than a large intestine endoscope. The concept of the present invention can also be applied to an endoscope apparatus other than an endoscope apparatus having a medical endoscope, or a flexible tube insertion apparatus.

The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the gist thereof. The embodiments can be implemented in appropriate combinations within a possible range, in which case a combined effect can be obtained. The above embodiment includes inventions in various stages, and various inventions are extracted by appropriately combining a plurality of constituent elements as described above.