WO2024102728A1 - Application de vision informatique pour permettre des mesures numériques en temps réel et un ciblage pendant l'arthroscopie - Google Patents

Application de vision informatique pour permettre des mesures numériques en temps réel et un ciblage pendant l'arthroscopie Download PDF

Info

Publication number
WO2024102728A1
WO2024102728A1 PCT/US2023/078944 US2023078944W WO2024102728A1 WO 2024102728 A1 WO2024102728 A1 WO 2024102728A1 US 2023078944 W US2023078944 W US 2023078944W WO 2024102728 A1 WO2024102728 A1 WO 2024102728A1
Authority
WO
WIPO (PCT)
Prior art keywords
instrument
image
real
dimension
profile
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2023/078944
Other languages
English (en)
Inventor
Miho Tanaka
Kartik Mangudi Varadarajan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Pictus Ai LLC
General Hospital Corp
Original Assignee
Pictus Ai LLC
General Hospital Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Pictus Ai LLC, General Hospital Corp filed Critical Pictus Ai LLC
Publication of WO2024102728A1 publication Critical patent/WO2024102728A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00203—Electrical control of surgical instruments with speech control or speech recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/06—Measuring instruments not otherwise provided for
    • A61B2090/061—Measuring instruments not otherwise provided for for measuring dimensions, e.g. length
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing

Definitions

  • This invention relates, in general, to systems and methods for enabling realtime, digital on screen measurements and targeting during a medical procedure such as arthroscopic surgery.
  • fixation devices implant
  • reconstructive tissue grafts are positioned based on visualization of known anatomic landmarks and intraoperative surgical measurements.
  • the present disclosure meets an important clinical need in medical procedures by providing systems and methods for enabling real-time, digital on screen measurements and targeting during a medical procedure such as arthroscopic surgery.
  • the present disclosure describes systems and methods, for enabling realtime digital measurements and targeting during a medical procedure such as arthroscopic surgery.
  • image processing methods either alone or in combination with data from positional sensors are used to detect the shape, key-points, and/or features of an object such as an instrument visible in the arthroscopy video.
  • the detected shape, key-points, and features are then used to track the instrument and present real-time on-screen information (such as an extendable virtual ruler, or variable virtual graft/drill diameter), which can be used for making quick and accurate digital measurements, and targeting decisions during arthroscopy surgery.
  • the image processing methods include both classical and modern deep learning-based computer vision methods.
  • the projected locations of instrument key-points on the image can be predicted via computer vision models, and combined with the predicted segmentation (via PnP matching, and 3D-to-2D matching) to predict the six degree-of-freedom (6DOF) pose (location/orientation) of the instrument. Once the 6DOF pose is known, it is combined with stored 3D geometric data of the instrument to place and appropriately scale the virtual overlays on the correct location on the image via coordinate transformations.
  • the disclosure provides a system for performing digital measurements during a medical procedure in a subject.
  • the system can include a camera, a display, an instrument, at least one memory, and at least one processor.
  • the camera can be positionable to capture image data corresponding to a treatment site in a subject.
  • the instrument can be dimensioned to be placed at the treatment site, and can include at least a first section having a first dimension between a first boundary and a second boundary.
  • the at least one memory can have stored thereon a configuration corresponding to the instrument, the configuration including the first dimension.
  • the at least one processor can be in electrical communication with the camera and the at least one memory.
  • the at least one processor can be configured to execute instructions stored in the at least one memory to: receive real-time image data from the camera, the real-time image data including at least a portion of the instrument; detect, in the realtime image data, a profile of the instrument; identify, on the instrument, the first boundary and the second boundary; determine a real-time measurement between a first set of points of the real-time image data, the real-time measurement being determined based at least in part on the first dimension; generate a visualization based on the realtime measurement; and display, at the display, the visualization and the real-time image data.
  • the first dimension is a diameter of the instrument.
  • the first boundary and the second boundary are visually-identifiable markings along the instrument.
  • the instrument includes a removable sleeve, wherein the first boundary and the second boundary are defined on the removable sleeve.
  • the visualization includes a first digital ruler, wherein the first digital ruler is overlaid onto the real-time image data at the display.
  • the instrument comprises an awl.
  • the at least one processor is further configured to execute instructions stored in the at least one memory to identify, in the real-time image data, a distal tip of the instrument.
  • a placement of the visualization on the display is based at least in part on a position of the distal tip.
  • the real-time measurement is a measurement of a radius extending from the distal tip.
  • the first set of points comprises a first point corresponding to a position of the distal tip at a first time, and a second point corresponding to a position of the distal tip at a second time, wherein the first measurement is a distance between the first point and the second point.
  • the instrument includes a first linear portion and a second linear portion, the second linear portion being substantially perpendicular to the first linear portion, the second portion defining a second dimension between a first end and a second end, wherein the configuration includes the second dimension, and wherein the at least one processor is further configured to execute instructions stored in the at least one memory to determine a second real-time measurement between a second first set of points of the real-time image data, the second real-time measurement being determined based at least in part on the second dimension.
  • the at least one processor is further configured to execute instructions stored in the at least one memory to receive an instrument input, and, based on the instrument input, select, from a plurality of instrument configurations stored in the at least one memory, the configuration.
  • the medical procedure is an arthroscopic knee procedure.
  • the instrument includes a first three- dimensional key point, wherein determining the real-time measurement includes predicting a location of the first three-dimensional key point on the real-time image.
  • determining the real-time measurement includes minimizing a difference between the profile of the instrument and a two- dimensional projection of a three-dimensional model of the instrument, and, based on the two-dimensional projection, determining a pose of the instrument relative to the camera, wherein the real-time measurement is based on the pose.
  • the pose of the instrument is a 6 degree of freedom pose.
  • the disclosure provides a method for performing digital measurements during a medical procedure in a subject.
  • the method comprises: receiving a first image, the first image including a visual representation of a first instrument and a tissue of a subject; detecting, within the first image, a profile of the first instrument, including an outer edge of the first instrument; comparing the profile of the first instrument to a two-dimensional projection of a three-dimensional model of the first instrument to determine a pose of the instrument relative to the camera; identifying a first key feature of the instrument; determining, based on the first key feature and the pose, a first dimension of the first image; generating an overlay including a first visualization based at least in part on the first dimension; and outputting, at a display, the overlay and the first image.
  • detecting a profile of the first image includes inputting the first image into a trained machine learning model, the trained machine learning model being trained on a first plurality of images, each of the first plurality of images including a first label identifying an instrument within the image, and a second label identifying a tissue within the image.
  • the first image is received in real-time from an arthroscope.
  • the method further comprises providing, prior to receiving the first image, a plurality of two-dimensional projections of the instrument, and selecting a first two-dimensional projection of the plurality of two-dimensional projections based on a similarity between the profile of the first instrument and the first two-dimensional projection, wherein performing the smoothing operation includes altering the profile to increase a similarity between the profile and the first two- dimensional projection.
  • the first key point is a first band positioned along the instrument, wherein the first dimension is a known distance between the first band and a second band positioned along the first instrument.
  • the first visualization is a digital ruler, wherein a scale of the digital ruler is determined at least in part based on the first dimension.
  • the method further comprises: receiving a second image, the second image including a visual representation of the first instrument and a tissue of the subject; identifying, in the second image, the first key point; determining, based at least partially on the first dimension, a measurement between a position of the first key point in the first image, and the first key point in the second image; and outputting, at the display, the measurement.
  • the medical procedure is an arthroscopic knee procedure.
  • the tissue is tissue of a joint.
  • the joint is a knee.
  • the disclosure provides a computer system comprising instructions stored on a non-transitory computer readable medium.
  • the instructions can cause at least one processor on a computer to: receive a first image, the first image including a visual representation of a first instrument and a tissue of a subject; detect, within the first image, a profile of the first instrument, including an outer edge of the first instrument; determine a predicted pose of the first instrument relative to the camera based on the profile; select a first two-dimensional projection of a three-dimensional model of the instrument based on the predicted pose; determine an actual pose of the instrument based at least in part on a comparison between the profile and the first two- dimensional projection; determine, based on the actual pose, a first dimension of the first image; generate an overlay including a first visualization based at least in part on the first dimension; and output, at a display, the overlay and the first visualization.
  • the tissue is tissue of a joint.
  • the joint is a knee.
  • the instructions stored on the non-transitory computer readable medium further cause at least one processor on a computer to: select, based on the comparison between the profile and the first two- dimensional projection, a second two-dimensional projection, minimizing a difference between the profile and the second two-dimensional projection, wherein the actual pose corresponds to a model pose of the second two-dimensional projection.
  • the disclosure provides a method for performing digital measurements during a medical procedure in a subject.
  • the method can include receiving a first image, the first image including a visual representation of a first instrument and a tissue of a subject.
  • a profile of the first instrument, including an outer edge of the first instrument, can be detected in the first image.
  • a smoothening operation can be performed on at least a first portion of the outer edge, wherein the smoothening operation at least partially alters a profile of the first instrument.
  • a first key feature of the instrument can be identified.
  • a first dimension of the first image can be determined, based on the first key feature.
  • An overlay can be generated including a first visualization based at least in part on the first dimension. The overlay and the first image can be output to the display.
  • detecting a profile of the first image includes inputting the first image into a trained machine learning model, the trained machine learning model being trained on a first plurality of images, each of the first plurality of images including a first label identifying an instrument within the image, and a second label identifying a tissue within the image.
  • the first image is received in real-time from an arthroscope.
  • the method further comprises: providing, prior to receiving the first image, a plurality of two-dimensional projections of the instrument, and selecting a first two-dimensional projection of the plurality of two-dimensional projections based on a similarity between the profile of the first instrument and the first two-dimensional projection, wherein performing the smoothing operation includes altering the profile to increase a similarity between the profile and the first two- dimensional projection.
  • the first key point is a first band positioned along the instrument, wherein the first dimension is a known distance between the first band and a second band positioned along the first instrument.
  • the first visualization is a digital ruler, wherein a scale of the digital ruler is determined at least in part based on the first dimension.
  • the method further comprises: receiving a second image, the second image including a visual representation of the first instrument and a tissue of the subject; identifying, in the second image, the first key point; determining, based at least partially on the first dimension, a measurement between a position of the first key point in the first image, and the first key point in the second image; and outputting, at the display, the measurement.
  • the medical procedure is an arthroscopic knee procedure.
  • the tissue is tissue of a joint.
  • the joint is a knee.
  • the disclosure provides a computer system comprising instructions stored on a non-transitory computer readable medium.
  • the instructions can cause at least one processor on a computer to: receive a first image, the first image including a visual representation of a first instrument and a tissue of a subject; detect, within the first image, a profile of the first instrument, including an outer edge of the first instrument; perform a smoothening operation on at least a first portion of the outer edge, wherein the smoothening operation at least partially alters a profile of the first instrument; identify a first key feature of the instrument; determine, based on the first key feature, a first dimension of the first image; generate an overlay including a first visualization based at least in part on the first dimension; and output, at a display, the overlay and the first visualization.
  • the tissue is tissue of a joint.
  • the joint is a knee.
  • Figure 1 shows measurements performed to describe a lesion or to identify optimal implant position by placing an instrument of known dimensions, (such as a hook probe) in the view of the arthroscope camera.
  • an instrument of known dimensions such as a hook probe
  • Figure 2 is a detailed side view showing a hook probe used in estimating distances in a treatment site using an arthroscope camera.
  • Figure 3 shows measurements using arthroscopic images wherein: Panel A shows measurement of ligament tear width; Panel B shows measurement of cartilage lesion width; Panel C shows measurement of patella overhang; Panel D shows measurement of patella to trochlea gap height; Panel E shows measurement of femur to tibia gap height; and Panel F shows measurement of femur to tibia gap height.
  • Figure 4 shows: in Panel A, post-surgery knee measurements using the methods of Ref. 7; and in Panel B, post-surgery knee measurements using the methods of Ref. 8.
  • Figure 5 shows a schematic of a non-limiting example system according to the present disclosure for performing digital measurements and/or targeting during a medical procedure in a subject.
  • Figure 6 shows an example of real-time data about relative angular orientation of the instrument with respect to the camera/camera plane in three- dimensional space obtained via positional sensors that can be used in the system of Figure 5
  • Figure 7 shows: in Panel A, an example of an extendable virtual ruler overlay of a method of the present disclosure; in Panel B, examples of virtual area fitting circles/ellipses of a method of the present disclosure; and in Panel C, examples of virtual variable drill sizes of a method of the present disclosure.
  • Figure 8 shows: in Panel A, a calibration image of a method of the present disclosure; in Panel B, predicted segmentation of a method of the present disclosure; and in Panel C, a predicted key-point heatmap of a method of the present disclosure.
  • Figure 9 shows a step of a method of the present disclosure in which the models are trained to segment both the outer profile of the instrument and internal features such as the edges of bands.
  • Figure 10 shows a step of a method of the present disclosure in which computer vision models are trained to detect the edges of internal features of the instrument.
  • Figure 10 Panel (A) shows a cropped probe image using segmentation mask
  • Figure 10 Panel (B) shows canny edge detection to identify edges of bands.
  • Figure 11 shows steps of a method of the present disclosure in which computer vision models are trained to output predicted segmentation masks of the probe hook on the video frames wherein the predicted segmentation masks were then used to crop the image, and Sobel filter applied to detect the probe band edges.
  • Panel A shows the original video frame
  • Panel B shows the detection of the probe hook, and predicted segmentation mask
  • Panel C shows the image cropped using predicted segmentation
  • Panels D and E show the detections I calculations.
  • Figure 12 shows in Panel A), an instrument 1200 tracked by computer vision in a method of the present disclosure; in Panel B), a variable diameter circle overlay placed on instrument tip in a method of the present disclosure; and in Panel C), a virtual ruler overlaid along instrument body of a method of the present disclosure.
  • Figure 13 shows steps of a method of the present disclosure wherein different types of on-screen information can be displayed to the user to enable quick and accurate digital measurements or targeting.
  • Virtual rulers are overlayed along the probe body and probe hook to enable quick measurements in Panels A and B. The user is able to rotate the virtual ruler about specific points to make measurements in directions not aligned with the probe body or probe hook in Panels C and D.
  • Figure 14 shows steps of a method of the present disclosure wherein the relative orientation (a) between a probe body and a camera plane may be displayed onscreen to help the user adjust probe orientation
  • Figure 15 shows steps of a method of the present disclosure wherein a probe tip point (point 1 , Figure 7) may be tracked across video frames to enable point-point on-screen measurements.
  • Figure 16 shows steps of a method of the present disclosure wherein the tip of a micro-fracture awl is tracked using the image processing steps.
  • Figure 17 shows steps of a method of the present disclosure wherein areas of interest are traced in the manner of Figure 16 to include, for example, the anterior cruciate ligament (ACL) footprint during surgery.
  • ACL anterior cruciate ligament
  • Figure 18 shows in Panels A, B, C, and D, embodiments of instruments according to the present disclosure each having a sleeve to improve computer vision based detection and tracking of the instruments.
  • Figure 19 is an example of hardware that can be used to implement a computer vision based detection and tracking of the instruments according to the present disclosure.
  • the present disclosure provides systems and methods for enabling real-time, digital on screen measurements and targeting during a medical procedure such as arthroscopic surgery.
  • the system includes a camera and light source attached to an arthroscope to capture the view inside the joint, an instrument (such as a hook probe, micro-fracture awl, drill guide etc.), a module to capture user inputs (e.g. via keyboard/mouse, physical buttons on camera/instrument, foot pedal, voice command etc.), optional positioning sensors placed on the arthroscope and/or the instrument, an image processing module, and a display to present information to the user (see Figure 5).
  • the positional sensors (if present) provide real-time data about relative angular orientation of the instrument with respect to the camera/camera plane in three- dimensional (3D) space (e.g., angles a, [3, y, see Figure 6).
  • the image processing module analyzes the video data coming from the arthroscopic camera and combines it with the user inputs, and orientation information from the positional sensors (if present) to track the six degree-of-freedom (6DOF) pose (location/orientation) of the instrument 70, and overlays appropriate visual information on the video data (see Figure 7).
  • the processed video data is then presented to the user via the display module (a computer screen, tablet, augmented reality glasses etc.).
  • the display module a computer screen, tablet, augmented reality glasses etc.
  • the requisite analysis/calculations are performed via image analysis methods alone.
  • the image processing module performs one or more of the following steps:
  • a calibration step to calculate camera matrix and account for distortion present in the arthroscopic images e.g., radial fish-eye distortion.
  • Image segmentation using a combination of deep-learning and/or classical computer vision methods to detect the shape/contour of the instrument This can be accomplished with specific segmentation models/algorithms trained to detect shape/contours of specific instruments. At the time of use, the specific models can be automatically selected based on user input about the instrument being used.
  • a key-point prediction step where 2D projected locations of pre-defined 3D key-points on the instrument are predicted on the arthroscopic image.
  • Data-output step that performs calculations and displays appropriate visual information on the processed video presented to the user.
  • Step 1 Camera calibration:
  • This step is performed to calculate camera intrinsic/extrinsic matrix, and perform distortion correction. Any of the known approaches to distortion correction, such as that described in [Ref. 12,13], can be used. A pre-saved file specific to the camera/lens system can be used for this calibration step. Alternatively, an intraoperative calibration step can be performed by imaging a calibration image such as a grid pattern ( Figure 8 Panel A).
  • computer vision models are trained to detect and predict the contour of the instrument 80 on the video frames.
  • models that can be used include deep learning based semantic and instance segmentation architectures such as Fully Convolutional Network (FCN), ll-Net, DeepLab v3, Mask R-CNN, SegFormer, etc. [Ref. 9,10],
  • FCN Fully Convolutional Network
  • ll-Net ll-Net
  • DeepLab v3 DeepLab v3
  • Mask R-CNN Mask R-CNN
  • SegFormer etc.
  • the output of such models is a prediction of image pixels that correspond to the class of the interest (in this case the instrument).
  • This model prediction is then used to create a predicted segmentation mask ( Figure 8 Panel B).
  • the models can be trained to segment either just the outer profile of the instrument 80 or both the outer profile of the instrument 85 and internal features such as the edges of bands 87 (see Figure 9).
  • deep learning-based methods may be used to predict segmentation mask for the outer profile of the instrument, and then used to crop out the instrument from the image.
  • Other computer vision methods such as canny edge detection, Sobel filter, etc. can then be applied to the cropped instrument image to detect the edges of internal features such as bands.
  • a Mask R-CNN instance segmentation model was trained via the opensource Detectron2 framework [Ref. 11 ], by providing several manually labelled images. Two class labels were included, (1 ) hook probe, and (2) tissue. Although the primary class of interest was the hook probe, including the background “tissue” class (which surrounds the hook probe) improved detection of the primary class.
  • the trained model outputs predicted segmentation masks of the probe hook 95 on the video frames.
  • the predicted segmentation masks were then used to crop the image, and Sobel filter applied to detect the probe band edges 97 (see Figure 10), using classical computer vision to detect probe band edges [e.g., canny edge detection http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/], [0073]
  • the segmentation mask can be used to improve the key-point prediction in Step 3. This can be done by either by providing the mask as another input channel along with the original image, or by using the image cropped with the segmentation mask an input for the key-point prediction.
  • Step 3 Key-points prediction:
  • a model predicts the projected locations of known 3D key-points of the instrument on the 2D arthroscopic image.
  • Key-points are generally 3D points distributed on the instrument surface in a region likely to be in the view of the camera, and adequately spread-out to maximize pose estimation accuracy in the step 4.
  • the original image cropped with segmentation mask from step 2 is presented as an input to an Hourglass network based deep CNN, which outputs a heatmap representing the prediction confidence for the key-point locations ( Figure 8 Panel C). The centroid of the heatmap is used as the predicted 2D projection of the known 3D key-point on the image.
  • Step 4 Prediction of instrument 6DOF pose:
  • the 6DOF pose of the instrument relative to the camera is predicted via a 3D- to-2D registration process.
  • First Perspective-n-Point (PnP) with RANSAC algorithm is applied to the 3D key-points and their 2D predictions obtained from step 3, to obtain an initial guess for the 6DOF pose of the instrument relative to the camera.
  • PnP First Perspective-n-Point
  • an edgebased heuristic procedure is used to minimize differences between the segmented mask of step 3, and 2D projection of the instrument placed at the initial pose.
  • the initial pose of the instrument is perturbed until a pose that minimizes differences (L1 or L2 norm) between the segmented mask and 2D projection of the instrument is identified. This final pose is used as the predicted 6DOF pose for the instrument.
  • Step 5 Data-output:
  • FIG. 11 Panel A shows the original video frame with a probe 100; Panel B shows the detection of the probe hook, and predicted segmentation mask; Panel C shows the image cropped using predicted segmentation; and Panels D and E show the detections / calculations.
  • Figure 12 shows in Panel A), an instrument 1200 tracked by computer vision; in Panel B), a variable diameter circle overlay placed on the instrument tip; and in Panel C), a virtual ruler overlaid along instrument body.
  • virtual rulers R1 , R2 are overlayed along the probe body 1310 and probe hook 1320 to enable quick measurements (see Figure 13 Panels A,B).
  • the user is able to rotate the virtual ruler about specific points to make measurements in directions not aligned with the probe body 1310 or probe hook 1320 (see Figure 13 Panels C, D).
  • the relative orientation (a) between probe body 1400 and camera plane may be displayed on-screen to help the user adjust probe orientation if they so desire (see Figure 14).
  • the probe tip of the probe 1500 may be tracked across video frames to enable point-point onscreen measurements (see Figure 15).
  • computer vision methods can be used to compensate for minor camera movements while probe is moved by user from one location to the next within the camera field of view.
  • FIG. 11 and 12 pertains to detection/tracking of a hook probe
  • instruments used for marking and/or drilling insertion sites for placement of fixation devices or reconstructive grafts can be tracked using computer vision. This is further illustrated with the example of an anterior cruciate ligament (ACL) reconstruction surgery.
  • ACL anterior cruciate ligament
  • surgeons need to select a graft of certain diameter (e.g., 8-12 mm), and mark or otherwise plan the appropriate location for a drill hole on the femur and tibia where anchors for the ACL graft are to be placed.
  • the drill hole should be in an appropriate anatomical location within the native footprint, and sufficiently distant from edges of the bone, especially on the posterior femoral condyle, to avoid blowout failure of the bone.
  • computer vision methods are used to detect and track instruments used to mark the drill hole location (such as a microfracture awl, electrocautery wand, guide wire etc.).
  • instruments used to mark the drill hole location such as a microfracture awl, electrocautery wand, guide wire etc.
  • the tip 1610 of a micro-fracture awl 1600 is tracked using the image processing steps 1 - 4 of the invention.
  • a virtual circle 1630 corresponding to the drill diameter (or graft size) is then shown near the tip of the instrument in the arthroscopy video. Coordinate transformations based on 6DOF instrument pose prediction is used to appropriately display the size and shape of the virtual circle.
  • the virtual circle 1630 helps the surgeon visualize where the edges of planned drill hole would be relative to the bony anatomy, prior to the
  • the instrument tip can be tracked and used as a virtual pen/brush by the surgeon to trace out areas of interest, such as a virtual “ACL footprint” 1640 traced by using awl as a pen/pointer.
  • areas of interest such as a virtual “ACL footprint” 1640 traced by using awl as a pen/pointer.
  • the arthroscope should not be moved excessively to maintain virtual markings at correct location relative to the underlying anatomy. Areas of interest that could be traced in this manner, include for example the ACL footprint 1640.
  • Centroids of footprint 1650 (or different quadrants) can then be automatically displayed to serve as potential targets for the drilling location (see Figure 17).
  • the generalizability of computer vision-based detection and tracking of instruments may be improved by adding specific features or making modifications to instruments such that: a) they are easier to detect, and b) making these features common across instruments so that the same computer vision method/s can be used to detect/track various instruments without explicitly retraining the computer vision model for each instrument.
  • this is achieved by adding a sleeve 1820 to a hook probe 1810 (see Figure 18, Panel A).
  • this is achieved by adding a sleeve 1840 to a hook probe 1830 (see Figure 18, Panel B).
  • this is achieved by adding a sleeve 1860 to an awl 1850 (see Figure 18, Panel C).
  • this is achieved by adding a sleeve 1880 to an awl 1870 (see Figure 18, Panel D).
  • the sleeve can be a separate component, or made integral part of the instrument.
  • the sleeve in turn contains features/and or textures such as parallel lines, vertical bands, a central spherical shape etc., to aid in computer vision-based detection of the sleeve (and by extension the instrument).
  • a computer vision model is then trained to detect/track the sleeve. Thereafter, the sleeve can be added to any instrument and used in concert with the pre-trained computer vision model for instrument detection and tracking, without retraining the computer vision model for the new instrument.
  • the relationship between the sleeve location (such as sleeve center) as detected by computer vision, and another point on the instrument (such as instrument tip) can be calculated.
  • the use of orientation sensors can aid in more accurate coordinate transformation from sleeve to another location on the instrument.
  • smoothening/curve fitting operations may be used to refine the predicted segmentation.
  • FIG. 19 is a schematic diagram illustrating an example computer system 1900, which can be used to implement the processes and systems described above.
  • the computer system 1900 can include a computing device 1902, a camera 1904, and a communication network 1906.
  • the camera 1904 can be a camera sized and configured to capture an image of an internal area of a subject.
  • a "subject" is a mammal, preferably a human.
  • the camera 1904 can be one of an arthroscope, an endoscope, a laparoscope, a cardioscope, or any other known camera for capturing internal images of a subject.
  • the camera 1904 is directly connected to the computing device 1902 (e.g., via a wired connection).
  • the camera can communicate images to the computing device 1902 through the communications network 1906.
  • the computing device 1902 can include a processor 1910, memory 1912, communications systems 1914, a display 1916, and input(s) 1918.
  • processor 1910 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a combination thereof, etc.
  • CPU central processing unit
  • GPU graphics processing unit
  • APU accelerated processing unit
  • ASIC application specific integrated circuit
  • FPGA field-programmable gate array
  • memory 1912 can include any suitable storage device or devices that can be used to store instructions, values, configurations, etc., that can be used, for example, by processor 1910 to process image data received from the camera 1904.
  • memory 1912 can include software which can be used by operators to implement processes for providing visualization for real-time image data received from camera 1904, machine learning models for processing images obtained by camera 1904 and calculating parameters therefrom, 2D projections of medical instruments (e.g., an awl, or a hooked probe, as described above) to be detected in the images from the camera 1904, 3D models of the medical instruments to be detected in images received from the camera 1904, pre-saved calibration file specific to the camera for use in distortion correction, known dimensions of the medical instrument, etc.
  • medical instruments e.g., an awl, or a hooked probe, as described above
  • Memory 1912 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof.
  • the memory 1912 can comprise a non-transitory computer readable medium including instructions for implementing the systems and processes described herein.
  • memory 1912 can include random access memory (RAM), read-only memory (ROM), electronically-erasable programmable readonly memory (EEPROM), one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc.
  • RAM random access memory
  • ROM read-only memory
  • EEPROM electronically-erasable programmable readonly memory
  • flash drives one or more hard disks
  • solid state drives one or more optical drives, etc.
  • optical drives etc.
  • memory 1912 can have encoded thereon a computer program for implementing the processes described herein.
  • communications systems 1914 can include any suitable hardware, firmware, and/or software for performing wired or wireless communications (e.g., with a user device, with camera 1904, etc.) over communication network 1906 and/or any other suitable communication networks.
  • communications systems 1914 can include one or more transceivers, one or more communication chips and/or chip sets, etc.
  • communications systems 1914 can include hardware, firmware and/or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.
  • image data can be streamed to the computing device 1902 from the camera 1904 through communications system 1914 (e.g., via the communication network 1906 or through a direct wired connection between the camera 1904 and the computing device 1902.
  • communications system 1914 can be in communication with the communications network 1906 and can communicate information thereto, receive instructions or configurations, receive firmware or software updates, etc.
  • the display 1916 can include any suitable display device, such as a computer monitor, a touchscreen, a television, a projector, a smart phone, a virtual reality headset, augmented reality goggles, etc.
  • a display can be remote relative to the computing device.
  • a processor and memory of a computing device can be hosted remotely (e.g., on a cloud server) and a display can receive visualizations and images from the computing device over a network connection.
  • inputs 1918 can include any suitable input devices and/or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a camera, a foot pedal, etc.
  • the present invention provides systems and methods for enabling realtime, digital on screen measurements and targeting during a medical procedure such as arthroscopic surgery.
  • any embodiment referenced herein is freely combinable with any one or more of the other embodiments referenced herein, and any number of features of different embodiments are combinable with one another.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Theoretical Computer Science (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • General Physics & Mathematics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Geometry (AREA)
  • Pathology (AREA)
  • Robotics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Image Processing (AREA)

Abstract

Un système permettant de réaliser des mesures numériques pendant une procédure médicale comprend : une caméra pour capturer des données d'image correspondant à un site de traitement ; un dispositif d'affichage ; un instrument placé au niveau du site de traitement, l'instrument comprenant une première section ayant une première dimension entre une première limite et une seconde limite ; et un processeur configuré pour exécuter des instructions afin de : (i) recevoir des données d'image en temps réel de la caméra, les données d'image en temps réel comprenant une partie de l'instrument, (ii) détecter, dans les données d'image en temps réel, un profil de l'instrument, (iii) identifier, sur l'instrument, la première limite et la seconde limite, (iv) déterminer une mesure en temps réel entre un premier ensemble de points des données d'image en temps réel, la mesure en temps réel étant déterminée sur la base en partie de la première dimension, (v) générer une visualisation sur la base de la mesure en temps réel, et (vi) afficher, au niveau du dispositif d'affichage, la visualisation et les données d'image en temps réel.
PCT/US2023/078944 2022-11-07 2023-11-07 Application de vision informatique pour permettre des mesures numériques en temps réel et un ciblage pendant l'arthroscopie Ceased WO2024102728A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202263382714P 2022-11-07 2022-11-07
US63/382,714 2022-11-07

Publications (1)

Publication Number Publication Date
WO2024102728A1 true WO2024102728A1 (fr) 2024-05-16

Family

ID=91033434

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2023/078944 Ceased WO2024102728A1 (fr) 2022-11-07 2023-11-07 Application de vision informatique pour permettre des mesures numériques en temps réel et un ciblage pendant l'arthroscopie

Country Status (1)

Country Link
WO (1) WO2024102728A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119090965A (zh) * 2024-11-07 2024-12-06 合肥工业大学 面向腹腔镜手术中的感兴趣区域监测方法和系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180049622A1 (en) * 2016-08-16 2018-02-22 Insight Medical Systems, Inc. Systems and methods for sensory augmentation in medical procedures
US20200261297A1 (en) * 2019-02-14 2020-08-20 Stryker Australia Pty Ltd Systems and methods for assisting surgery
US20200390452A1 (en) * 2001-05-25 2020-12-17 Conformis, Inc. Patient Selectable Joint Arthroplasty Devices and Surgical Tools

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200390452A1 (en) * 2001-05-25 2020-12-17 Conformis, Inc. Patient Selectable Joint Arthroplasty Devices and Surgical Tools
US20180049622A1 (en) * 2016-08-16 2018-02-22 Insight Medical Systems, Inc. Systems and methods for sensory augmentation in medical procedures
US20200261297A1 (en) * 2019-02-14 2020-08-20 Stryker Australia Pty Ltd Systems and methods for assisting surgery

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119090965A (zh) * 2024-11-07 2024-12-06 合肥工业大学 面向腹腔镜手术中的感兴趣区域监测方法和系统
CN119090965B (zh) * 2024-11-07 2025-03-07 合肥工业大学 面向腹腔镜手术中的感兴趣区域监测方法和系统

Similar Documents

Publication Publication Date Title
EP3273854B1 (fr) Systèmes pour chirurgie assistée par ordinateur au moyen d'une vidéo intra-opératoire acquise par une caméra à mouvement libre
JP7203148B2 (ja) 術中画像分析のためのシステム及び方法
US11275249B2 (en) Augmented visualization during surgery
EP3820396B1 (fr) Système de chirurgie assistée par ordinateur
US12582330B2 (en) Systems and methods for computer-assisted shape measurements in video
US20260076749A1 (en) Systems and methods for treating a joint
US20230233257A1 (en) Augmented reality headset systems and methods for surgical planning and guidance
JP2014533964A5 (fr)
JP7191539B2 (ja) 骨組織への視覚的近接性に基づく、解剖学的画像の位置追跡座標系との位置合わせの改善
US20230149092A1 (en) Systems and methods for compensating for obstructions in medical images
JP7012302B2 (ja) 手術支援端末及びプログラム
US20230263498A1 (en) System and methods for calibration of x-ray images
US20260026896A1 (en) System and method for lidar-based anatomical mapping
US20240197410A1 (en) Systems and methods for guiding drilled hole placement in endoscopic procedures
EP4714388A1 (fr) Systèmes de guidage d'élimination d'os
US20260123998A1 (en) Image-based surgical guidance
US20250095194A1 (en) Image processing apparatus, storage medium, and image processing method
US20240050045A1 (en) Computer-implemented method for ascertaining an item of torsion information of a bone, x-ray facility, computer program and electronically readable data carrier
WO2025250376A1 (fr) Lumière structurée pour enregistrement 3d sans contact dans une navigation chirurgicale à base de vidéo
JP2026000896A (ja) プログラム、情報処理方法、及び情報処理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23889592

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23889592

Country of ref document: EP

Kind code of ref document: A1