WO2016190607A1 - Système de lunettes intelligentes pour fournir une image d'aide à la chirurgie et procédé de fourniture d'image d'aide à la chirurgie à l'aide des lunettes intelligentes - Google Patents

Système de lunettes intelligentes pour fournir une image d'aide à la chirurgie et procédé de fourniture d'image d'aide à la chirurgie à l'aide des lunettes intelligentes Download PDF

Info

Publication number
WO2016190607A1
WO2016190607A1 PCT/KR2016/005312 KR2016005312W WO2016190607A1 WO 2016190607 A1 WO2016190607 A1 WO 2016190607A1 KR 2016005312 W KR2016005312 W KR 2016005312W WO 2016190607 A1 WO2016190607 A1 WO 2016190607A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
fluorescent
smart glass
real
infrared
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2016/005312
Other languages
English (en)
Korean (ko)
Inventor
김법민
김현구
김민지
오유진
허정석
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Korea University Research and Business Foundation
Original Assignee
Korea University Research and Business Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020150071967A external-priority patent/KR101667152B1/ko
Application filed by Korea University Research and Business Foundation filed Critical Korea University Research and Business Foundation
Publication of WO2016190607A1 publication Critical patent/WO2016190607A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays

Definitions

  • the present invention relates to a smart glass system for providing a surgical support image and a method for providing a surgical support image using a smart glass, and more particularly, to provide a surgical support image that allows a surgeon to more accurately identify a surgical site through a fluorescent image.
  • the present invention relates to a method for providing a surgical support image using a smart glass system and smart glass.
  • Sentinel lymph node is a lymph node in which cancer cells preferentially metastasize in the primary tumor and is an important indicator for determining lymph node metastasis. Therefore, if cancer cells are not found through the biopsy of the monitored lymph nodes, other lymph nodes are considered to have no metastases and no further surgery is performed.
  • lymph node In-vivo examination of lymph node through accurate search of the lymph node, which is an important indicator in determining the metastasis of cancer, can reduce postoperative sequelae such as lymphedema and minimize scarring on the patient's body. .
  • surveillance lymph nodes using targeted drugs have been used as a standard technique in early breast cancer or melanoma surgery.
  • a method of obtaining a visible light image using a blue dye and a visible light camera, and a near infrared fluorescence image using a near infrared fluorescent dye and a near infrared camera A method of obtaining and a method of obtaining a radiographic image by using a gamma imaging apparatus of radiopharmaceuticals integrated in a surveillance lymph node have been proposed.
  • the surgeon sees the surgical site of the actual patient and the surgical lymph nodes and the surgical site of the actual patient Matching can determine the extent to be excised.
  • the surgeon checks the fluorescence image on the monitor installed in the operating room with the eyes, and proceeds to the operation while looking at the operation site of the patient lying on the operating table. There is discomfort to operate. In particular, when looking at the actual surgical site of the patient there is a limit that can not be made precisely because the fluorescent portion is not identified.
  • Korean Patent Registration No. 10-13553408 in the 'guided surgery imaging system and method thereof', the gynecologist wears an affected image of a patient photographed by CT, MRI, and X-ray previously photographed.
  • the present invention discloses a technique for displaying surgery on a transparent display in a shape and looking at the affected part of the gynecologist and the actual affected part passing through the transparent display.
  • the method disclosed in the Korean Patent Registration uses a gyro sensor or converts the affected image to a specific point of the patient in order to match the pre-photographed affected image and the actual affected part seen through the transparent display.
  • the affected image does not exactly match the actual affected area, which is why Rather, it may interfere with recognition.
  • the present invention has been made to solve the above problems, a smart glass that provides a surgical support image that can proceed with the operation while visually screening the resection of cancer cells metastasized including the monitoring lymph nodes in real time It is an object of the present invention to provide a method for providing a surgical support image using a system and smart glasses.
  • the near-infrared fluorescence image provides the same effect as the actual surgical site of the patient by providing a surgeon to provide a surgical support image that allows the surgeon to focus the field of vision only on the patient's surgical site to check the surgical site
  • Another object is to provide a surgical support image providing method using a glass system and a smart glass.
  • the smart glass system for providing a surgical support image, wireless communication between the image processing module, the transparent display unit to display the image, the visible light camera, the visible light image captured by the visible light camera And a near-infrared photographing module having a smart glass having a glass communication unit for transmitting to the image processing module through the camera, a near-infrared camera, and a module communication unit for transmitting the near-infrared fluorescent image photographed by the near-infrared camera module to the image processing module.
  • the image processing module generates a real-time converted fluorescence image by converting the near-infrared fluorescence image to at least one of a photographing direction and a size of the visible light image based on the visible light image, and transmits the real-time converted fluorescence image to the smart glass.
  • the real-time conversion fluorescence image received through the glass communication unit is displayed on the transparent display unit of the smart glass, the operation support image, characterized in that the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. Achieved by a smart glass system.
  • At least three color fluorescent markers are arranged in a geometric structure at a position that can be photographed by the near infrared camera and the visible light camera around the surgical site of the patient;
  • the image processing module extracts the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively;
  • the real-time conversion fluorescence image may be generated by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image.
  • the near-infrared imaging module may be provided in the form of a head mount that can be worn on the head of the jikdo.
  • the near-infrared imaging module may further include a near-infrared light source for irradiating near-infrared light to enable imaging of the near-infrared fluorescent image by the near-infrared camera.
  • the visible light camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.
  • the above object according to another embodiment of the present invention in a method for providing a surgical support image using a smart glass having a transparent display unit, a visible light camera and a glass communication unit, the image is displayed, (a) to the visible light camera Photographing the visible light image in real time; (b) transmitting the visible light image to the image processing module through the glass communication unit; (c) photographing a near infrared fluorescence image by a near infrared camera; (d) transmitting the near infrared acid fluorescence image to the image processing module; (e) converting the near-infrared fluorescent image in real time to at least one of a direction and a size of the visible light image based on the visible light image in the image processing module to generate a real-time converted fluorescent image; (f) transmitting the real-time converted fluorescence image from the image processing module to the smart glass; and (g) displaying the real-time converted fluorescent image received through the glass communication unit on the transparent display unit of the smart glass.
  • Step (e) comprises: (e1) extracting the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively; (e2) generating the real-time converted fluorescence image by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image can do.
  • the near-infrared camera may be provided in the form of a head mount that can be worn on the head of the gypsy.
  • the visible light camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.
  • the image processing module in the smart glass system for providing a surgical support image, the image processing module, the transparent display unit for displaying the image, the near-infrared camera, and the image taken by the near-infrared camera module Smart glass having a glass communication unit for transmitting the near-infrared fluorescent image to the image processing module;
  • the image processing module generates a real-time converted fluorescent image by real-time converting the size of the near-infrared fluorescent image transmitted from the smart glass based on pre-registered reference information, and transmits the real-time converted fluorescent image to the smart glass;
  • the real-time conversion fluorescence image transmitted from the glass communication unit is displayed on the transparent display unit of the smart glass, and the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. It is achieved by providing a smart glass system.
  • a fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient;
  • the image processing module may generate the real-time converted fluorescence image by extracting the fluorescent marker from the near-infrared fluorescent image, comparing the extracted fluorescent marker with the reference information in real time to convert the size of the near-infrared fluorescent image.
  • the fluorescent markers are at least three or more arranged in a geometric structure;
  • the reference information includes at least two reference marker information for the fluorescent markers respectively extracted from an image photographed at different distances from the fluorescent markers;
  • the image processing module detects a distance from the near infrared camera by comparing the change in the geometry of the fluorescent marker extracted from the near infrared fluorescence image with the reference marker information, and detects the distance from the near infrared camera based on the distance from the near infrared camera.
  • the size of the image can be converted in real time.
  • the smart glass may further include a near-infrared light source for irradiating near-infrared light to enable imaging of the near-infrared fluorescent image by the near-infrared camera.
  • the near-infrared camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.
  • the above object is, according to another embodiment of the present invention, in a method for providing a surgical support image using a smart display having a transparent display unit, a near-infrared camera and a glass communication unit, the image is (a) the reference information is an image Registering with a processing module; (b) photographing a near infrared fluorescence image by the near infrared camera; (c) transmitting the near-infrared fluorescent image from the smart glass to the image processing module; (d) generating, by the image processing module, a real-time converted fluorescence image by converting the size of the near infrared fluorescence image based on the reference information; (e) transmitting the real-time converted fluorescence image from the image processing module to the smart glass; and (f) displaying the real-time converted fluorescence image received through the glass communication unit on the transparent display unit of the smart glass.
  • the reference information is an image Registering with a processing module
  • a fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient;
  • the step (d) may include (d1) extracting the fluorescent marker from the near infrared fluorescence image, and (d2) comparing the fluorescent marker extracted from the step (d1) with the reference information to determine the size of the near infrared fluorescence image. It may include the step of generating a real-time conversion fluorescence image by real-time conversion.
  • Step (a) is (a1) the at least two or more fluorescent markers are photographed at different distances from the fluorescent marker by the near-infrared camera, and (a2) from two or more images captured in the (a1) step, respectively.
  • At least two reference marker information for the extracted fluorescent marker is registered as the reference information;
  • step (d2) (d21) comparing the reference marker information with the change in the geometry of the fluorescent marker extracted in the step (d1) and detecting the distance from the near infrared camera, (d22) the ( and converting the size of the near-infrared fluorescent image in real time based on the distance from the near-infrared camera detected in step d22) to generate the real-time converted fluorescent image.
  • the near-infrared camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.
  • the near-infrared fluorescence image provides the same effect as that displayed on the actual surgical site of the patient, the surgeon can proceed to the operation while confirming the surgical site by concentrating his vision only on the surgical site of the patient.
  • FIG. 1 is a view showing an example of a surgical environment of a conventional operating room
  • FIG. 2 is a view showing the configuration of a smart glass system for providing a surgical support image according to a first embodiment of the present invention
  • FIG. 3 is a view showing an example of the configuration of a smart glass according to the first embodiment of the present invention
  • FIG. 4 is a diagram illustrating an example of a configuration of an image processing module according to a first embodiment of the present invention.
  • FIG. 5 and 6 are views for explaining a surgical support image providing method using a smart glass according to a first embodiment of the present invention
  • FIG. 7 is a diagram illustrating an example of an actual image provided by a method for providing a surgical support image using smart glasses according to a first embodiment of the present invention.
  • FIG. 8 is a diagram showing the configuration of a smart glass system for providing a surgical support image according to a second embodiment of the present invention.
  • FIG. 9 is a view showing an example of the configuration of a smart glass according to a second embodiment of the present invention.
  • FIG. 10 is a diagram illustrating an example of a configuration of an image processing module according to a second embodiment of the present invention.
  • 11 and 12 are views for explaining a surgical support image providing method using a smart glass according to a second embodiment of the present invention.
  • the present invention relates to a smart glass system for providing a surgical support image and a method for providing a surgical support image using a smart glass.
  • the smart glass system for providing a surgical support image according to the present invention comprises an image processing module, a transparent display unit displaying an image, a visible light camera, and a visible light image captured by the visible light camera through the image processing module.
  • a near-infrared photographing module having a smart glass having a glass communication unit for transmitting a light source, a near-infrared camera, and a module communication unit for transmitting a near-infrared fluorescent image photographed by the near-infrared camera module to the image processing module;
  • the image processing module generates a real-time converted fluorescence image by converting the near-infrared fluorescence image to at least one of a photographing direction and a size of the visible light image based on the visible light image, and transmits the real-time converted fluorescence image to the smart glass.
  • the real-time conversion fluorescence image received through the glass communication unit is displayed on the transparent display unit of the smart glass, characterized in that the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass.
  • the smart glass system according to the first embodiment of the present invention includes a smart glass 100, a near infrared ray imaging module 200, and an image processing module 310.
  • the smart glass 100 is provided in the form of glasses that can be worn by a doctor. That is, the smart glass 100 may include a glass frame 150 that forms a skeleton and a lens 160 that is fixed to the glass frame 150 to be positioned in front of a human eye when the glass frame 150 is worn. .
  • the smart glass 100 includes a transparent display unit 110, a visible light camera 120, and a glass communication unit 140.
  • the smart glass 100 may include a glass control unit 130 for controlling the re-function of the transparent display unit 110, visible light camera 120 and the glass communication unit 140.
  • the transparent display unit 110 is made of a transparent material to display an image on the surface.
  • the transparent display unit 110 includes the entire lens 160 constituting the smart glass 100.
  • the transparent display unit 110 may be formed only in a part of the lens 160. Of course, it can be prepared.
  • the transparent display unit 110 even if the wearer wears the smart glass 100 or the image is displayed on the transparent display unit 110, the actual image coming through the transparent display unit 110, that is, The actual image viewed by the eye can be viewed together with the image displayed on the transparent display unit 110. That is, when an image is displayed on the transparent display unit 110, an image displayed on the transparent display unit 110 may be overlapped with an actual image to obtain an effect.
  • the visible light camera 120 is installed in the smart glass 100 to take a visible light image.
  • the visible light camera 120 is installed in the glass frame 150 of the smart glass 100. It is installed to take a visible light image in the same direction as the eye gaze direction.
  • the glass communication unit 140 is connected to the image processing module 310 through wireless communication.
  • the glass communication unit 140 communicates with the image processing module 310 through TCP / IP-based Wi-Fi communication or Bluetooth communication, and other wireless communication is applicable.
  • TCP / IP-based Wi-Fi communication or Bluetooth communication and other wireless communication is applicable.
  • the glass controller 130 transmits the visible light image captured by the visible light camera 120 to the image processing module 310 in real time through the glass communication unit 140, the image processing module 310
  • the real-time conversion fluorescence image transmitted from the glass communication unit 140 and received through the transparent display unit 110 will be described, which will be described later.
  • the NIR module 200 may include a NIR camera 210 and a module communicator 230.
  • the near infrared camera 210 photographs a near infrared fluorescence image of the surgical site of the patient.
  • the module communicator 230 transmits the near infrared fluorescence image captured by the near infrared camera 210 to the image processing module 310.
  • the near-infrared imaging module 200 is provided in the form of a head mount that can be worn on the head. Accordingly, the gynecologist wears the smart glass 100 in the form of glasses, and proceeds to surgery while wearing the head-mounted near infrared imaging module 200 on the head.
  • the near-infrared light source 220 for irradiating near-infrared to enable the imaging of the near-infrared fluorescent image by the near-infrared camera 210 is installed in the near-infrared imaging module 200 of the head-mount type
  • the near-infrared imaging module 200 of the head-mount type For example,
  • the smart glass 100 and the near-infrared photographing module 200 are configured as independent devices.
  • the near-infrared imaging module 200 and the smart glass 100 in the form of a head mount may be provided in one device form.
  • the glass communication unit 140 and the module communication unit 230 which were independently configured, are integrated into one communication module 400 and have a near infrared fluorescent shape captured by the near infrared camera 210.
  • Under the control of the glass control unit 130 may be provided to be transmitted to the image processing module 310 through one communication module 400.
  • the on / off of the near infrared light source 220 may also be provided to operate under the control of the glass controller 130.
  • the image processing module 310 processes the visible light image transmitted from the smart glass 100 and the near infrared fluorescence image transmitted from the near infrared imaging module 200 to display on the transparent display unit 110 of the smart glass 100. Generate real-time converted fluorescence images to be.
  • the image processing module 310 and the field display unit 320 are configured as an information processing apparatus 300 such as a computer.
  • 4 is a diagram illustrating an example of a configuration of an image processing module 310 according to a first embodiment of the present invention.
  • the image processing module 310 may include a first communication unit 311, a second communication unit 312, an image matching unit 314, and a main control unit 313.
  • the first communication unit 311 is connected to the glass communication unit 140 of the smart glass 100 through wireless communication to receive a visible light image transmitted from the smart glass 100, the real-time conversion fluorescence image to the smart glass 100 send.
  • the second communication unit 312 is connected to the module communication unit 230 of the near infrared imaging module 200 to receive a near infrared fluorescence image transmitted from the near infrared imaging module 200.
  • the first communication unit 311 and the second communication unit 312 are glass of the smart glass 100.
  • the communication unit 140 and the near infrared imaging module 200 may be provided in accordance with the communication method of the module communication unit 230, respectively.
  • the first communication unit 311 may be connected to the glass communication unit 140 through wireless communication as described above
  • the second communication unit 312 may be connected to the module communication unit 230 through wired communication or wireless communication. Can be.
  • the glass communication unit 140 of the smart glass 100 and the module communication unit 230 of the near-infrared photographing module 200 are provided as one communication module 400 or independently provided as shown in FIG. 3.
  • the first communication unit 311 and the second communication unit 312 may be provided as one communication module.
  • the image matching unit 314 may display the near-infrared fluorescent image received through the second communication unit 312 based on the visible light image received through the first communication unit 311.
  • a real-time converted fluorescent image is generated by converting at least one of the photographing direction and the size in real time.
  • the visible light camera 120 is configured to photograph the visible light image in the same direction of view as the eye direction of the house of the house, so that the visible image captured by the visible camera 120 may be matched with the actual image viewed by the house of the house. have.
  • the installation position is different so that the distance between the image photographed by the visible light camera 120 and its direction or size, that is, the surgical site. Therefore, in order to match the near-infrared fluorescence shape photographed by the near-infrared camera 210 to the gaze of the gimmick, the near-infrared fluorescence image is converted in accordance with the visible light image corresponding to the gaze of the gimmick.
  • At least three color fluorescent markers are disposed in a geometric structure at a position that can be photographed by the near infrared camera 210 and the visible light camera 120 around the surgical site of the patient, and the image processing module is used therein.
  • the image matching unit 314 of 310 generates a real-time converted fluorescence image by converting a near infrared fluorescence image will be described below in more detail with reference to FIGS. 5 to 7.
  • the visible light image is photographed by the visible light camera 120 (S10)
  • the near infrared fluorescence image is photographed by the near infrared camera 210 (S30).
  • the visible light image photographed by the visible light camera 120 and the near infrared fluorescence image photographed by the near infrared camera 210 are transmitted to the image processing module 310 (S11 and S31).
  • the image matching unit 314 of the image processing module 310 extracts a color fluorescent marker from the visible light image and the near infrared fluorescence image (S20).
  • the color fluorescent marker is made of a color fluorescent material, and the visible light image extracts the position of the color fluorescent marker based on the color of the fluorescent color marker. In the near infrared fluorescent image, the position of the color fluorescent marker is determined based on the fluorescent material of the fluorescent color marker. Will be extracted.
  • FIG. 6 In the first embodiment of the present invention, as shown in FIG. 6, three color fluorescent markers are arranged in a triangular geometric structure around the patient's surgery, and FIG. 6A is extracted from the visible light image. The position of the color fluorescent marker M1 is shown, and FIG. 6B shows the position of the color fluorescent marker M2 extracted from the near infrared fluorescence image.
  • the color fluorescent markers M1 and M2 in the visible light image and the near infrared fluorescence image do not coincide with each other.
  • the near-infrared fluorescent image is converted so that the position of the color fluorescent marker M2 extracted from the image overlaps the position of the color fluorescent marker M1 extracted from the visible light image.
  • the color fluorescent marker M1 extracted from the near-infrared fluorescent image is extracted from the visible light image.
  • the conversion of the near-infrared fluorescent image may be rotated about three axes, or may be converted in size, and may be converted to three axes. This can be calculated through the geometric relationship between the color fluorescent markers, that is, the geometric relationship such as distance and angle.
  • the real-time conversion fluorescence image may go through various types of image processing processes known in advance so that the near-infrared fluorescence image of the original form can be easily visually recognized by the gynecologist.
  • the generated real-time converted fluorescence image as described above is transmitted to the smart glass 100 by the main control unit 313 through the first communication unit 311 (S23), the glass control unit 130 of the smart glass 100 is glass
  • the real-time conversion fluorescent image received through the communication unit 140 is displayed on the transparent display unit 110 (S12).
  • FIG. 7A illustrates a surgical site of a patient who is visible when a real-time conversion fluorescence image is not displayed on the transparent display unit 110, and FIG. 7B is displayed on the transparent display unit 110.
  • Figure 7 (c) is a surgical site of the patient visible to the eye of the patient in the state in which the real-time conversion fluorescence image is displayed on the transparent display 110.
  • the surgeon is able to proceed with the operation as if the fluorescent material is actually displayed on the surgical site of the patient, while performing the operation while watching the screen of the monitor provided in the operating room. The inconvenience of coming out of the way can be eliminated.
  • the gaze of the gimmick that changes according to the movement of the gimmick is corrected based on the visible light image taken by the visible light camera 120, thereby enabling more accurate display of the near infrared fluorescence image. do.
  • the main control unit 313 matches the real-time converted fluorescence image generated in step S22 and the visible light image with each other and displays them on the field display unit 320 installed at the surgery site, whereby other personnel or assistants at the surgery site are additionally added. It may be prepared to check this.
  • the near-infrared light source 220 is installed in the near-infrared imaging module 200 of the head mounted type as shown in FIG. 2, but the outside of the near-infrared imaging module 200 is an example. For example, it may be installed in a specific space inside the operating room.
  • the smart glass system according to the second embodiment of the present invention includes a smart glass 1000 and an image processing module 3100.
  • the smart glass 1000 is provided in the form of glasses that can be worn by a zip doctor. That is, the smart glass 1000 may include a glass frame 1500 that forms a skeleton and a lens 1600 that is fixed to the glass frame 1500 to be positioned in front of the human eye when the glass frame 1500 is worn. .
  • the smart glass 1000 includes a transparent display unit 1100, a near infrared camera, and a glass communication unit 1400.
  • the smart glass 1000 may include a glass control unit 1300 for controlling the functions of the transparent display unit 1100, the near infrared camera, and the glass communication unit 1400.
  • the transparent display unit 1100 is made of a transparent material and displays an image on the surface. According to the second embodiment of the present invention, the transparent display unit 1100 is composed of the entire lens 1600 constituting the smart glass 1000, but only a part of the lens 1600 may be provided with the transparent display. Of course.
  • the actual image passing through the transparent display unit 1100 that is, The actual image viewed by the eye of the house may be viewed together with the image displayed on the transparent display unit 1100. That is, when an image is displayed on the transparent display unit 1100, an image displayed on the transparent display unit 1100 overlaps an actual image, thereby obtaining an effect.
  • the near infrared camera is installed in the smart glass 1000 to take a near infrared fluorescence image of the surgical site of the patient.
  • the glass controller 1300 transmits the near infrared fluorescence image captured by the near infrared camera to the image processing module 3100 through the glass communication unit 1400.
  • the near-infrared camera is installed at a position capable of photographing a position corresponding to the gaze of the gimmick when the gimmick wears the smart glass 1000.
  • the glass control unit 1300 transmits the near-infrared fluorescent image captured by the near infrared camera in real time to the image processing module 3100 through the glass communication unit 1400, and transmits the image from the image processing module 3100.
  • the real-time conversion fluorescent image to be described later received through the glass communication unit 1400 is displayed on the transparent display unit 1100, which will be described later.
  • the image processing module 3100 processes the near-infrared fluorescent image transmitted from the smart glass 1000 to generate a real-time converted fluorescent image to be displayed on the transparent display unit 1100 of the smart glass 1000.
  • the image processing module 3100 and the field display unit 3200 are configured as an information processing apparatus 3000 such as a computer.
  • 10 is a diagram illustrating an example of a configuration of an image processing module 3100 according to a second embodiment of the present invention.
  • the image processing module 3100 may include a communication unit 3110, an image processing unit 3140, and a main control unit 3130.
  • the communication unit 3110 is connected to the glass communication unit 1400 of the smart glass 1000 through wireless communication to receive a near-infrared fluorescent image transmitted from the smart glass 1000, and transmits a real-time converted fluorescent image to the smart glass 1000. do.
  • the communication unit 3110 may be connected to the glass communication unit 1400 through wireless communication, for example, Wi-Fi or Bluetooth communication based on TCP / IP.
  • the image processor 3140 processes the near-infrared fluorescent image received through the glass communication unit 1400 to generate a real-time converted fluorescent image.
  • the real-time converted fluorescence image generated by the image processor 3140 is transmitted to the smart glass 1000 through the glass communication unit 1400.
  • the main controller 3130 controls the image processor 3140 such that the image processor 3140 generates a real-time converted fluorescent image based on pre-registered reference information.
  • the image processor 3140 may convert the real-time converted image. How to create.
  • the image processor 3140 generates a real-time converted fluorescence image by real-time converting at least one of the photographing direction and size of the near infrared fluorescence image transmitted from the smart glass 1000 based on the reference image.
  • the near-infrared camera is installed in the smart glass 1000 so as to correspond to the direction of eye gaze of the house, and according to the position of the near-infrared camera of this position, the near-infrared camera is in the same direction of gaze as the eye of the house of the house.
  • the near-infrared fluorescence image is taken, and the near-infrared fluorescence image photographed by the near-infrared camera coincides with the direction of gaze and the actual image seen by the surgeon.
  • a real-time conversion fluorescence image is generated by adjusting the size of an image portion (hereinafter referred to as a 'fluorescent image') represented by the near infrared fluorescent dye in the near infrared fluorescence image photographed by the near infrared camera, which is determined by the smart glass 1000.
  • a 'fluorescent image' an image portion represented by the near infrared fluorescent dye in the near infrared fluorescence image photographed by the near infrared camera, which is determined by the smart glass 1000.
  • the image processing module 3100 uses a fluorescent marker and pre-registered reference information to generate a real-time converted image.
  • the fluorescent marker is placed at a position that can be photographed by a near infrared camera around the surgical site of the patient.
  • three fluorescent markers are arranged in a geometric structure, for example, a triangular structure.
  • the image processor 3140 extracts the fluorescent marker from the near infrared fluorescence image and compares the extracted fluorescent marker with reference information to convert the size of the near infrared fluorescence image, that is, the above-described fluorescent image in real time.
  • the reference information may include at least 8 reference marker information for the fluorescent markers extracted from the image, which is drawn at different distances from the fluorescent marker.
  • the image processor 3140 is transmitted from the smart glass 1000.
  • the fluorescent image is extracted from the image and transmitted to the smart plastic to be displayed on the transparent display unit 1100.
  • the fluorescence image is displayed in the state overlapping with the surgical site, and the gynecologist adjusts the size of the fluorescence image while checking whether the fluorescence image overlaps with the actual size at the current position, and finally the size and extraction of the finally determined fluorescence image
  • the size of the fluorescent marker or the distance between the fluorescent markers is set as one reference marker information (hereinafter referred to as 'first reference marker information').
  • the image processor 3140 may determine the first reference marker information.
  • Another reference marker information (hereinafter referred to as 'second reference marker information') is generated and set in the same manner as the generation.
  • FIG. 12A illustrates an example of an image photographed at a close distance from the fluorescent marker
  • FIG. 12B illustrates an example image captured at a farther distance from the fluorescent marker.
  • the size of the fluorescent marker in the image photographed at a close distance is increased or the distance between the fluorescent markers is increased, and the size of the fluorescent marker photographed at a long distance is decreased or the distance between the fluorescent markers is closer.
  • the fluorescent marker is extracted from the near-infrared fluorescent image taken by the near-infrared camera during the surgical procedure, and the size of the fluorescent marker
  • the size of the target image that is, the fluorescent image, in the near infrared fluorescent image may be determined. For example, when the distance between the fluorescent markers extracted from the near infrared fluorescence image is located within the range between the distance between the fluorescent markers in the first reference marker information and the second reference marker information, the fluorescent image in the near infrared fluorescence image is determined according to the ratio of the corresponding distance. Size can be adjusted linearly.
  • the image processor 3140 compares the change in the geometrical structure of the fluorescent marker extracted from the near infrared fluorescence image, for example, the change in the size of the fluorescent marker, the change in the distance between the fluorescent markers, and the reference marker information.
  • the distance to the site is detected, and through this, the size of the near-infrared fluorescent image is converted in real time to generate a canal tube converted fluorescent image.
  • the registration process of the reference information as described above is performed.
  • three fluorescent markers are disposed around the surgical site of the patient.
  • the near-infrared camera of the smart glass 1000 captures an image to include the fluorescent marker at a first distance, for example, a distance close to the fluorescent marker (S50), the captured near-infrared image (hereinafter, 'first near-infrared image'). 'Is transmitted to the image processing module 3100 through the glass communication unit 1400.
  • the first near-infrared image photographed by the near-infrared camera may be one image, may be a real-time image, and is not limited to one image by the expression of near-infrared image.
  • the image processor 3140 of the image processing module 3100 extracts an image portion indicated by a fluorescent marker and a fluorescent image, that is, a near infrared fluorescent dye from the first near infrared image (S61).
  • the image processor 3140 may adjust information about the fluorescent marker, for example, information about a size or a distance between the markers and the size of the fluorescent image displayed on the transparent display unit 1100 as described above.
  • First reference marker information including information on the size of the determined fluorescent image is set (S62).
  • the near infrared camera of the smart glass 1000 captures an image to include the fluorescent marker at a second distance, for example, a distance further from the fluorescent marker (S51)
  • the captured near infrared image (hereinafter, 'second' The NIR image is transmitted to the image processing module 3100 through the glass communication unit 1400.
  • the second near infrared image captured by the near infrared camera may be a single image, may be a real time image, and is not limited to a single image by the expression of near infrared image.
  • the image processor 3140 of the image processing module 3100 extracts an image portion indicated by a fluorescent marker and a fluorescent image, that is, a near infrared fluorescent dye, from the second near infrared image (S63).
  • the image processor 3140 may adjust information about the fluorescent marker, for example, information about a size or a distance between the markers and the size of the fluorescent image displayed on the transparent display unit 1100 as described above.
  • second reference marker information including information on the size of the fluorescent image determined as (S64)
  • the first reference marker information and the second reference marker information are registered as the reference information.
  • the near-infrared camera captures the near-infrared fluorescence image in real time in the process of performing the actual concentrating (S52), and the captured near-infrared fluorescence image is real-time image processing module 3100. Is sent to.
  • the image processor 3140 of the image processing module 3100 extracts a fluorescent marker and a fluorescence image displayed by a near infrared fluorescence image, that is, a near infrared fluorescence dye from the near infrared fluorescence image (S65), and the first reference marker information and By comparing the second reference marker information and the extracted fluorescent marker, the size of the fluorescent image is adjusted to convert the near-infrared fluorescent image (S66), thereby generating a real-time converted fluorescent image (S67).
  • the real-time conversion fluorescence image generated in real time is transmitted to the smart glass 1000, the real-time conversion fluorescence image is displayed on the transparent display unit 1100 of the smart glass 1000 (S53), so that the collection of the transparent display
  • the real-time conversion fluorescence image displayed on the unit 1100 the real patient's surgery area beyond the transparent display unit 1100 is seen, whereby the real-time conversion fluorescence image displayed on the transparent display unit 1100 overlaps the actual operation site of the patient.
  • the same effect can be obtained as the colored fluorescent substance is labeled on the actual surgical site of the patient.
  • the near-infrared fluorescence image is corrected on the basis of the fluorescent marker, thereby more accurately displaying the near-infrared fluorescent image Becomes possible.
  • near infrared camera 220 near infrared light source
  • module communication unit 300 3000: information processing device
  • image processing module 311 first communication unit
  • main control unit 3140 image processing unit
  • 314 image matching unit 320, 3200: field display unit
  • the surgeon is applied to the field of providing surgical support images that can more accurately identify the surgical site through fluorescence images.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Heart & Thoracic Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biomedical Technology (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Robotics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Pathology (AREA)
  • Endoscopes (AREA)

Abstract

La présente invention concerne un système de lunettes intelligentes destiné à fournir une image d'aide à la chirurgie, ainsi qu'un procédé de fourniture de l'image d'aide à la chirurgie à l'aide des lunettes intelligentes. Le système de lunettes intelligentes selon la présente invention comprend : un module de traitement d'image ; des lunettes intelligentes ayant une unité d'affichage transparente pour afficher une image, une caméra à rayons visibles et une unité de communication de lunettes destinée à transmettre au module de traitement d'image, par communication sans fil, une image de rayons visibles photographiée par la caméra à rayons visibles et un module de photographie de rayons proche infrarouge ayant une caméra à rayons proche infrarouge, ainsi qu'une unité de communication de module destinée à transmettre une image fluorescente de rayons proche infrarouge capturée par le module de caméra à rayons proche infrarouge au module de traitement. Le module de traitement d'image produit une image fluorescente convertie en temps réel en convertissant, en temps réel, l'image fluorescente de rayons proche infrarouge sur la base de l'image de rayons visibles au moyen d'un élément parmi une direction de photographie et une taille de l'image de rayons visibles, puis transmet l'image fluorescente convertie en temps réel aux lunettes intelligentes et l'unité d'affichage transparente des lunettes intelligentes affiche l'image fluorescente convertie en temps réel reçue par le biais de l'unité de communication de lunettes, de sorte que l'image fluorescente convertie en temps réel puisse être superposée sur la vue d'un chirurgien en train d'opérer portant les lunettes intelligentes. Par conséquent, le chirurgien en train d'opérer peut effectuer la chirurgie tout en vérifiant visuellement, en temps réel, un site de résection, y compris un ganglion sentinelle, vers lequel des cellules cancéreuses se sont propagées et un effet d'une image fluorescente de rayons proche infrarouge apparaissant comme étant affichée sur un site chirurgical réel d'un patient est fourni, permettant ainsi au chirurgien en train d'opérer de concentrer sa propre vision uniquement sur un site chirurgical d'un patient et d'effectuer la chirurgie tout en vérifiant le site chirurgical.
PCT/KR2016/005312 2015-05-22 2016-05-19 Système de lunettes intelligentes pour fournir une image d'aide à la chirurgie et procédé de fourniture d'image d'aide à la chirurgie à l'aide des lunettes intelligentes Ceased WO2016190607A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR1020150071967A KR101667152B1 (ko) 2015-05-22 2015-05-22 수술 지원 영상을 제공하는 스마트 글라스 시스템 및 스마트 글라스를 이용한 수술 지원 영상 제공 방법
KR10-2015-0071967 2015-05-22
KR20160061294 2016-05-19
KR10-2016-0061294 2016-05-19

Publications (1)

Publication Number Publication Date
WO2016190607A1 true WO2016190607A1 (fr) 2016-12-01

Family

ID=57394105

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/005312 Ceased WO2016190607A1 (fr) 2015-05-22 2016-05-19 Système de lunettes intelligentes pour fournir une image d'aide à la chirurgie et procédé de fourniture d'image d'aide à la chirurgie à l'aide des lunettes intelligentes

Country Status (1)

Country Link
WO (1) WO2016190607A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107374730A (zh) * 2017-09-06 2017-11-24 东北大学 光学手术导航系统
WO2020014999A1 (fr) * 2018-07-20 2020-01-23 东北大学 Dispositif d'affichage de lumière invisible et système de guidage optique pour opérations
CN111481164A (zh) * 2019-01-25 2020-08-04 宏碁股份有限公司 用以获取荧光眼底图的方法及其装置
CN112704474A (zh) * 2020-11-30 2021-04-27 浙江大学医学院附属第一医院 一种用于检测甲状旁腺位置的ar检测眼镜
CN113194299A (zh) * 2021-07-01 2021-07-30 深圳市修远文化创意有限公司 一种智能医疗场景下的口腔治疗实时画面分享方法
CN115708717A (zh) * 2022-11-01 2023-02-24 上海微创医疗机器人（集团）股份有限公司 医生控制台以及显示模式的切换方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20070028351A (ko) * 2004-06-30 2007-03-12 하마마츠 포토닉스 가부시키가이샤 림프절 검출 장치
KR20130108320A (ko) * 2010-09-10 2013-10-02 더 존스 홉킨스 유니버시티 관련 애플리케이션들에 대한 일치화된 피하 해부구조 참조의 시각화
KR20130135476A (ko) * 2012-06-01 2013-12-11 의료법인 우리들의료재단 수술 유도영상 시스템 및 그 방법
KR20140112207A (ko) * 2013-03-13 2014-09-23 삼성전자주식회사 증강현실 영상 표시 시스템 및 이를 포함하는 수술 로봇 시스템
KR20150001756A (ko) * 2012-04-16 2015-01-06 칠드런스 내셔널 메디컬 센터 수술 및 중재 시술에서 추적 및 제어를 위한 듀얼-모드 스테레오 이미징 시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20070028351A (ko) * 2004-06-30 2007-03-12 하마마츠 포토닉스 가부시키가이샤 림프절 검출 장치
KR20130108320A (ko) * 2010-09-10 2013-10-02 더 존스 홉킨스 유니버시티 관련 애플리케이션들에 대한 일치화된 피하 해부구조 참조의 시각화
KR20150001756A (ko) * 2012-04-16 2015-01-06 칠드런스 내셔널 메디컬 센터 수술 및 중재 시술에서 추적 및 제어를 위한 듀얼-모드 스테레오 이미징 시스템
KR20130135476A (ko) * 2012-06-01 2013-12-11 의료법인 우리들의료재단 수술 유도영상 시스템 및 그 방법
KR20140112207A (ko) * 2013-03-13 2014-09-23 삼성전자주식회사 증강현실 영상 표시 시스템 및 이를 포함하는 수술 로봇 시스템

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107374730A (zh) * 2017-09-06 2017-11-24 东北大学 光学手术导航系统
WO2020014999A1 (fr) * 2018-07-20 2020-01-23 东北大学 Dispositif d'affichage de lumière invisible et système de guidage optique pour opérations
CN111481164A (zh) * 2019-01-25 2020-08-04 宏碁股份有限公司 用以获取荧光眼底图的方法及其装置
CN112704474A (zh) * 2020-11-30 2021-04-27 浙江大学医学院附属第一医院 一种用于检测甲状旁腺位置的ar检测眼镜
CN113194299A (zh) * 2021-07-01 2021-07-30 深圳市修远文化创意有限公司 一种智能医疗场景下的口腔治疗实时画面分享方法
CN115708717A (zh) * 2022-11-01 2023-02-24 上海微创医疗机器人（集团）股份有限公司 医生控制台以及显示模式的切换方法

Similar Documents

Publication Publication Date Title
WO2016190607A1 (fr) Système de lunettes intelligentes pour fournir une image d'aide à la chirurgie et procédé de fourniture d'image d'aide à la chirurgie à l'aide des lunettes intelligentes
KR101667152B1 (ko) 수술 지원 영상을 제공하는 스마트 글라스 시스템 및 스마트 글라스를 이용한 수술 지원 영상 제공 방법
WO2014088268A1 (fr) Appareil d'imagerie par rayons x et procédé de commande associé
US10021351B2 (en) Stereoscopic video imaging
WO2020101283A1 (fr) Dispositif d'assistance chirurgicale mettant en œuvre la réalité augmentée
WO2017111464A1 (fr) Appareil d'imagerie à rayons x, procédé de commande associé et détecteur de rayons x
WO2014077613A1 (fr) Robot pour procédure de repositionnement, et procédé pour commander son fonctionnement
WO2016195401A1 (fr) Système de lunettes 3d pour opération chirurgicale au moyen de la réalité augmentée
WO2011002209A2 (fr) Dispositif d'agrandissement monté sur la tête pour chirurgie
WO2013157673A1 (fr) Tomographie par cohérence optique et procédé associé
WO2013133470A1 (fr) Système d'imagerie thermique capable de pister une partie spécifique dans une image thermique, et procédé permettant de pister une partie spécifique au moyen dudit système
WO2021071336A1 (fr) Dispositif d'affichage à lunettes intelligentes basé sur la détection du regard
WO2020235914A1 (fr) Dispositif de mesure de l'état de la peau et de soins de la peau, et système de mesure de l'état de la peau et de soins de la peau associé
WO2020116669A1 (fr) Dispositif de nystagmographie portatif basé sur la réalité virtuelle et procédé de test diagnostique l'utilisant
WO2021112312A1 (fr) Outil de réalité augmentée pour chirurgie d'implant et procédé de visualisation d'informations de chirurgie d'implant
CN109688403A (zh) 一种应用于手术室内的裸眼3d人眼追踪方法及其设备
WO2018143651A1 (fr) Appareil de photographie de rétine et procédé de photographie de rétine faisant appel à celui-ci
WO2019074201A1 (fr) Dispositif d'imagerie à rayons x, détecteur de rayons x et système d'imagerie à rayons x
WO2019146904A1 (fr) Système de visiocasque pour obtenir une image de support chirurgical
WO2015119467A1 (fr) Dispositif d'imagerie pour radiographie
WO2022080855A1 (fr) Dispositif de mesure d'image médicale tridimensionnelle et système de correspondance d'image
WO2014011014A1 (fr) Appareil ophtalmique, et méthode de mesure d'un site de traitement pour l'appareil
CN209358681U (zh) 一种应用于手术室内的裸眼3d人眼追踪设备
KR102254456B1 (ko) 수술 지원 영상을 제공하는 헤드 마운트 시스템
US11744466B2 (en) Method and apparatus for localizing brain tissue regions connected with a brain function

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16800247

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16800247

Country of ref document: EP

Kind code of ref document: A1