TWI856893B - Operation image alignment method and system thereof - Google Patents

Abstract

An operation image alignment method includes: inputting an actual three-dimensional image and an actual two-dimensional image of an actual desired part into a computer system; converting the actual three-dimensional image into multiple sets of two-dimensional projection images according to image parameters of the actual three-dimensional image through a built-in image alignment prediction model of the computer system; comparing each set of two-dimensional projection images and the actual two-dimensional image to obtain an image parameter difference value, and selecting one set of two-dimensional projection images which image parameter difference value match a preset difference value to obtain a predicted rotation angle and a predicted offset amount. Among them, the image alignment prediction model is an artificial intelligence model trained by a model algorithm, in which multiple sets of historical images containing at least one historical three-dimensional image and at least one historical two-dimensional image are used as a data set of the image alignment prediction model.

Description

Surgical image alignment method and system

The present invention relates to a surgical image alignment method and system thereof, and in particular to a surgical image alignment method and system thereof for aligning three-dimensional images and two-dimensional images.

In medical surgery, surgical navigation systems play an extremely important role, helping doctors to accurately locate the lesion during surgery. Before surgery, a CT scanner or MRI scanner is used to obtain a three-dimensional image of the surgical site, allowing doctors to accurately grasp the image content of the lesion. During surgery, surgical image registration is required to align the two-dimensional image obtained during surgery with the three-dimensional image before surgery to align the lesion position, so that the subsequent navigation system can perform dynamic image tracking. Therefore, how to reduce the alignment error and alignment processing time of surgical images is a concern of technical personnel in this field.

The present invention proposes a surgical image alignment method, which is executed by a computer system, and includes: inputting an actual three-dimensional image and an actual two-dimensional image of the actual part to be aligned into the computer system; using a built-in image alignment prediction model of the computer system to convert the actual three-dimensional image into multiple sets of two-dimensional projection images according to several image parameters of the actual three-dimensional image; and comparing each set of two-dimensional projection images with the actual two-dimensional image to calculate the image parameter difference value, and selecting the set of two-dimensional projection images whose image parameter difference value meets the preset difference value to obtain the predicted rotation angle and the predicted translation amount; wherein, the image alignment prediction model uses multiple sets of historical images including at least one historical three-dimensional image and at least one historical two-dimensional image as a data set, and is an artificial intelligence model trained based on the data set using a model algorithm.

In some embodiments, the surgical image alignment method further includes a model building step to establish the image alignment prediction model, including: defining multiple historical alignment regions of the historical alignment site from at least one historical two-dimensional image and at least one historical three-dimensional image in each set of historical images, wherein each of these historical alignment regions includes historical position information.

In some embodiments, the model building step further includes: converting at least one historical three-dimensional image in each set of historical images into at least one historical two-dimensional projection image with a first viewing angle or a second viewing angle through an image projection conversion technique; and using the historical position information of the historical alignment area as the initial position, obtaining the historical rotation angle and historical translation amount between at least one historical three-dimensional image and at least one historical two-dimensional image in each set of historical images at the first viewing angle or the second viewing angle through at least one historical two-dimensional projection image.

In some embodiments, the first viewing angle is a sideways viewing angle, and the second viewing angle is a top-down viewing angle.

In some embodiments, these image parameters include image contour and image gradient value.

In some embodiments, the model algorithm is one of a Generative Adversarial Networks algorithm and a Deep Iterative 2D/3D Registration algorithm.

In some embodiments, the surgical image alignment method further includes: using an imaging device to take an actual two-dimensional image of the actual part to be aligned, wherein the actual two-dimensional image includes actual position information.

The present invention proposes a surgical image alignment system, which uses a computer system to execute a surgical image alignment method as described in any of the above items.

In some embodiments, the surgical image alignment system further includes an imaging device, which is a C-arm X-ray machine having a transmitting end and a receiving end.

In some embodiments, the surgical image alignment system also includes a computer tomography device for photographing the actual part to be aligned to obtain an actual three-dimensional image.

100:Surgical image alignment system

120: Three-dimensional imaging equipment

130: Two-dimensional imaging equipment

131: C-type rack

132: Transmitter

133: Receiving end

140: Computer system

150: Spine

151: Vertebrae to be aligned

200: Surgical image alignment method

210: Model building steps

220: Online matchup steps

211,212,213,213a,213b: Steps

221,222,223: Steps

301,302,303: Area

I _H1 :Historical 3D images

I _H2 : Historical 2D images

I _PH1 : Historical 2D projection images

I _T1 : Actual 3D image

I _T2 : Actual 2D image

I _E1 : Experimental 3D imaging

_IE2 : Experimental 2D imaging

P _E : Experimental prediction of projected images

S:Radiation source

The following detailed description in conjunction with the accompanying drawings will provide a better understanding of the present invention. It should be noted that, in accordance with standard industry practice, the features are not drawn to scale. In fact, the size of each feature may be increased or decreased arbitrarily to facilitate discussion.

FIG. 1 is a schematic diagram of a surgical image alignment system according to an embodiment of the present invention; FIG. 2 is a flow chart of a surgical image alignment method according to an embodiment of the present invention; FIG. 3A and FIG. 3B are schematic diagrams of the alignment area of the historical two-dimensional images of the side view angle image and the top view angle image according to the embodiment of the present invention; and FIG. 4 is a schematic diagram of the historical two-dimensional projection image according to the embodiment of the present invention.

Various embodiments of the present invention are discussed in detail below. However, it is understood that the embodiments provide many applicable concepts that can be implemented in a variety of specific contexts. The embodiments discussed and disclosed are for illustration only and are not intended to limit the scope of the present invention.

FIG. 1 is a schematic diagram of a surgical image alignment system 100 according to an embodiment of the present invention. The surgical image alignment system 100 includes a three-dimensional imaging device 120, a two-dimensional imaging device 130, and a computer system 140. The surgical image alignment system 100 uses the computer system 140 to train an artificial intelligence model to align the three-dimensional image taken by the three-dimensional imaging device 120 before the operation with the two-dimensional image taken by the two-dimensional imaging device 130 during the operation, thereby reducing the surgical image alignment error and improving the situation that the surgical process requires a lot of time to complete the surgical image registration. In the following embodiment, the spine 150 is used as the desired alignment part, and the vertebra 151 in the spine 150 is used as the desired alignment area. It should be understood that other areas that require regional alignment and surgical navigation are within the scope of the present invention.

The three-dimensional imaging device 120 is used to take pictures before surgery and generate three-dimensional images. The three-dimensional imaging device 120 can be a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, a positron emission tomography (PET) scanner, a single photon emission CT (SPECT) scanner, or any device that can obtain a three-dimensional image of the photographed target. For example, a patient can take a computed tomography image of the spine 150 before surgery, and convert the spine 150 into a three-dimensional simulated object in the computer system 140. In this way, the computer system 140 can obtain a three-dimensional image of a three-dimensional simulated object and segment at least one vertebral segment 151 to be aligned therefrom.

The two-dimensional imaging device 130 is used to take pictures during surgery and generate two-dimensional images. The two-dimensional imaging device 130 is a C-arm X-ray machine, which has a C-frame 131, a transmitting end 132, and a receiving end 133. The C-frame 131 can drive the transmitting end 132 and the receiving end 133 to rotate around the target, so that the C-arm X-ray machine can take two-dimensional images of the target at different angles. For example, the transmitting end 132 and the receiving end 133 are arranged opposite to each other, and the transmitting end 132 transmits X-rays to the vertebrae 151 to be aligned, and then the X-rays passing through these vertebrae 151 to be aligned are received by the receiving end 133, and finally converted into a two-dimensional image. The two-dimensional image includes the P-A view image (i.e. the top view angle) and the lateral view image (i.e. the side view angle).

The computer system 140 is communicatively connected to the three-dimensional imaging device 120 and the two-dimensional imaging device 130, and data can be transmitted between them by any wired or wireless means. The computer system 140 includes a memory and a processor, which can be used to store multiple sets of historical images of the spine 150 (where each set of historical images includes a historical three-dimensional image I _H1 and a historical two-dimensional image I _H2 corresponding thereto), and perform image processing on these historical images to define at least one vertebra 151 to be aligned from these historical images, and finally use these historical three-dimensional images I _H1 and historical two-dimensional images I _H2 as data sets to train an artificial intelligence model, thereby establishing the image alignment prediction model of the present invention. The training of the image alignment prediction model is realized by image projection conversion technology, which converts the historical three-dimensional image I _H1 into the historical two-dimensional projection image I _PH1 , and then obtains the relative position information between the historical two-dimensional projection image I _PH1 and the historical two-dimensional image I _H2 of different viewing angles. In this way, when the computer system 140 receives the actual three-dimensional image I _T1 taken by the three-dimensional imaging device 120 before the operation and the actual two-dimensional image I _T2 taken by the two-dimensional imaging device 130 during the operation, the vertebrae 151 to be aligned in the two images can be aligned together according to the image alignment prediction model, so as to provide the doctor with subsequent surgical image tracking. The computer system 140 may be a smart phone, a tablet computer, a personal computer, a laptop computer, a server, an industrial computer, or various electronic devices with computing capabilities, but the present invention is not limited thereto.

FIG. 2 is a schematic diagram of a surgical image alignment method 200 according to an embodiment of the present invention. The surgical image alignment method 200 includes a model building step 210 and an online alignment step 220, wherein the model building step 210 includes steps 211 to 213, and step 213 further includes steps 213a and 213b, and the online alignment step 220 includes steps 221 to 223. The surgical image alignment method 200 can be implemented through the surgical image alignment system 100 shown in FIG. 1, or through a structure with similar functions. The following is a description of the surgical image alignment method 200 in conjunction with the surgical image alignment system 100 shown in FIG. 1.

In the model building step 210, step 211 is first performed to obtain multiple groups of historical images of the historical vertebra 150, wherein each group of historical images includes at least one historical three-dimensional image I _H1 previously taken by the three-dimensional imaging device 120 of the historical vertebra 150, and at least one historical two-dimensional image I _H2 previously taken by the two-dimensional imaging device 130 of the historical vertebra 150. The source of the historical vertebra 150 can be taken from different objects, and the historical two-dimensional images I _H2 and the historical three-dimensional images I _H1 previously taken by the objects are used as data sets for training the artificial intelligence model. In some embodiments, one historical three-dimensional image I _H1 can correspond to one or more historical two-dimensional images I _H2 with different viewing angles. For example but not limited thereto, the historical two-dimensional image I _H2 is an image taken at a sideways viewing angle (first viewing angle) or an image taken at a top-down viewing angle (second viewing angle).

Then, in step 212, a plurality of historical regions to be aligned (corresponding to a plurality of historical vertebrae 151 to be aligned) are defined from the historical two-dimensional image I _H2 of the historical spine 150 in each set of historical images. Please refer to FIG. 3A and FIG. 3B. In the lateral angle image (FIG. 3A) and the top view angle image (FIG. 3B) of the historical two-dimensional image I _H2 , a plurality of historical regions to be aligned include regions 301, 302, and 303, respectively covering each historical vertebrae 151 to be aligned, and each defined historical region to be aligned includes historical position information. The historical position information is the position information of the historical region to be aligned in three-dimensional space.

Please return to FIG. 2 . In step 213 , the historical three-dimensional image I _H1 and the historical two-dimensional image I _H2 in each set of historical images are used as data sets, and the model algorithm is used to train the artificial intelligence model based on the data sets to establish the image alignment prediction model. In this way, the image alignment prediction model can predict the correct rotation angle and the correct translation amount between the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 of each vertebral segment 151 to be aligned in the actual spine 150, thereby aligning the actual three-dimensional images I _T1 of multiple actual vertebral segments 151 to be aligned with the actual two-dimensional image I _T2 one by one. In some embodiments, the model algorithm for training the image registration prediction model includes but is not limited to a Generative Adversarial Networks algorithm and a Deep Iterative 2D/3D Registration algorithm.

Please continue to refer to FIG. 2 , step 213 of the embodiment of the present invention further includes step 213a and step 213b. First, in step 213a, the computer system 140 converts the historical three-dimensional image I _H1 in each set of historical images into a historical two-dimensional projection image I _PH1 with a first viewing angle or a second viewing angle through an image projection conversion technology. The image projection conversion technology includes but is not limited to perspective conversion, or any technology that can convert a three-dimensional image into a two-dimensional image. As shown in FIG4 , the first viewing angle and the second viewing angle are the historical two-dimensional projection image I _PH1 of the side viewing angle and the top viewing angle respectively obtained by projecting the historical three-dimensional image I H1 by the radiation source S, or the historical two-dimensional projection image I _PH1 of any viewing angle obtained by projecting the historical three-dimensional image I _H1 by the radiation source S. The historical three-dimensional image I _H1 includes a plurality of vertebrae 151 to be aligned in the historical spine 150, and after being converted into _the historical two-dimensional projection image I _PH1 with the first viewing angle or the second viewing angle by the image projection conversion technology, the plurality of historical vertebrae 151 to be aligned in the historical two-dimensional projection image I _PH1 are cut out one by one. Alternatively, the computer system 140 may first cut out a plurality of vertebrae 151 to be aligned from the historical 3D image I _H1 as a plurality of historical 3D images I _H1 , and then convert them one by one into a plurality of historical 2D projection images I _PH1 with the first viewing angle or the second viewing angle through image projection conversion technology, but the present invention is not limited thereto.

Please return to FIG. 2 . Then, in step 213b, the computer system 140 uses the historical position information of each historical region to be aligned as the initial position, and obtains the historical correct rotation angle and historical correct translation amount between at least one historical three-dimensional image I _H1 and at least one historical two-dimensional image I _H2 in each set of historical images at the first viewing angle or the second viewing angle through the historical two-dimensional projection image I _PH1 . Specifically, the historical position information of the historical region to be aligned (e.g., the vertebral segment 151 to be aligned) in the historical two-dimensional image I _H2 can be used as the initial position in the coordinate system for the computer system 140 to obtain the image parameter difference value between the historical two-dimensional projection image I _PH1 and the historical two-dimensional image I _H2 . In this way, the historically correct rotation angle and historically correct translation amount between the historical three-dimensional image I _H1 and the historical two-dimensional image I _H2 at the top view angle or the side view angle can be obtained according to the image parameter difference value, so that the historical vertebrae 151 to be aligned in the historical two-dimensional image I _H2 can be aligned one by one with the historical vertebrae 151 to be aligned in the historical three-dimensional image I _H1 . In other words, the historically correct rotation angle and historically correct translation amount can enable the image alignment prediction model to rotate/translate the historical three-dimensional image I _H1 of the historical vertebrae 151 to be aligned so that it is aligned (aligned) to the historical vertebrae 151 to be aligned in the historical two-dimensional image I _H2 . If there are multiple historical vertebrae 151 to be aligned, multiple historical 3D images I _H1 corresponding to these historical vertebrae 151 to be aligned can be rotated/translated so that these historical 3D images I _H1 can be aligned (aligned) one by one to the historical vertebrae 151 to be aligned in the historical 2D image I _H2 .

Please continue to refer to FIG. 2 . After the model building step 210 is performed, the online alignment step 220 is performed. First, step 221 is performed to input the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 of the actual spine 150 into the computer system 140. Then in step 222, the computer system 140 generates a plurality of sets of two-dimensional projection images I _PT1 from the actual three-dimensional image I _T1 according to various image parameters through the aforementioned image projection conversion technology by using the image alignment prediction model established in the model building step 210. Among them, each set of two-dimensional projection images I _PT1 is a two-dimensional projection image of any viewing angle obtained by projecting the simulated radiation source S onto the actual three-dimensional image I _T1 in the image alignment prediction model. In the embodiment of the present invention, the image parameters include but are not limited to image contour and image brightness change value (ie, image gradient value, used to indicate the degree of brightness change of the image).

Then, in step 223, the image alignment prediction model compares each set of two-dimensional projection images I _PT1 with the actual two-dimensional image I _T2 and calculates the image parameter difference value between each set of two-dimensional projection images I _PT1 and the actual two-dimensional image I _T2 . Afterwards, the image alignment prediction model selects a set of two-dimensional projection images I _PT1 whose image parameter difference value meets the preset difference value as the prediction result, so as to obtain the predicted rotation angle and predicted translation amount between the actual vertebra 151 to be aligned in the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 . In this way, the surgical navigation system can complete the surgical image registration according to the alignment between the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 .

In some embodiments, the model building step 210 also includes an identification step of the image alignment prediction model to train the accuracy of the image alignment prediction model. Specifically, the computer system 140 inputs the experimental three-dimensional image _IE1 and the experimental two-dimensional image _IE2 of the historical vertebra 150 into the image alignment prediction model to obtain the experimental predicted rotation angle and the experimental predicted translation amount between the experimental three-dimensional image _IE1 and the experimental two-dimensional image _IE2 . Then, the image alignment prediction model generates an experimental predicted projection image _PE according to the experimental predicted rotation angle and the experimental predicted translation amount. Finally, the generated experimental predicted projection image _PE is used to establish an image alignment identification model to determine whether the accuracy of the image alignment prediction model is qualified. Specifically, the experimental three-dimensional image _IE1 and the experimental two-dimensional image _IE2 are input into the image alignment prediction model as the experimental set in the data set, and the experimental predicted projection image _PE predicted by the image alignment prediction model (including the experimental predicted rotation angle and experimental predicted translation between the experimental three-dimensional image _IE1 and the experimental two-dimensional image _IE2 ) and the correct experimental two-dimensional image _IE2 are used to train the artificial intelligence model to establish the image alignment identification model. Therefore, when the image alignment identification model cannot determine the authenticity between the experimental predicted projection image _PE and the correct experimental two-dimensional image _IE2 , it is determined that the prediction of the image alignment prediction model is sufficiently accurate. Otherwise, the image alignment prediction model is re-predicted until the accuracy of the prediction result is qualified.

During the entire alignment process, the image alignment model will use the actual position information of the actual vertebra 151 to be aligned in the actual two-dimensional image I _T2 (top view) as the initial position, and continuously judge whether the prediction result of the image alignment model is accurate enough through the image alignment identification model, until the correct rotation angle and correct translation amount between the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 are predicted, so that the actual three-dimensional image I _T1 and the actual two-dimensional image I _T2 of the actual vertebra 151 to be aligned can finally be aligned with each other. It should be understood that the purpose of establishing an image alignment identification model is only to improve the prediction accuracy of the image alignment model. In fact, in some embodiments of the present invention, even if the establishment of the image alignment identification model is omitted, it will not affect the image alignment between the three-dimensional image and the two-dimensional image of the present invention.

In addition, the image alignment model can be a series of alignment processes, through continuous prediction and adjustment, so that the actual three-dimensional image I _T1 of the vertebra 151 to be aligned can eventually be aligned with the actual two-dimensional image I _T2 of the posterior-anterior photo (top view) and the lateral photo (lateral view).

The above summarizes the features of several embodiments, so that those skilled in the art can better understand the state of the present invention. Those skilled in the art should understand that they can easily use the present invention as a basis to design or modify other processes and structures, thereby achieving the same goals and/or achieving the same advantages as the embodiments introduced herein. Those skilled in the art should also understand that these equivalent constructions do not deviate from the spirit and scope of the present invention, and they can make various changes, substitutions and modifications without departing from the spirit and scope of the present invention.

200: Surgical image alignment method

210: Model building steps

220: Online matchup steps

211,212,213,213a,213b: Steps

221,222,223: Steps

Claims (9)

A method for alignment of surgical images is implemented by a computer system, comprising: inputting an actual three-dimensional image and an actual two-dimensional image of an actual part to be aligned into the computer system; using an image alignment prediction model built into the computer system to convert the actual three-dimensional image into a plurality of sets of two-dimensional projection images according to a plurality of image parameters of the actual three-dimensional image, wherein the image parameters include an image contour and an image gradient value; and comparing the two-dimensional projection images of the two-dimensional projection images. The projected image and the actual two-dimensional image are compared to calculate an image parameter difference value, and the set of two-dimensional projection images whose image parameter difference value meets a preset difference value is selected to obtain a predicted rotation angle and a predicted translation amount; wherein the image alignment prediction model is an artificial intelligence model trained based on a model algorithm using multiple sets of historical images including at least one historical three-dimensional image and at least one historical two-dimensional image as a data set. The surgical image alignment method as described in claim 1 further includes a model building step to establish the image alignment prediction model, including: defining a plurality of historical alignment regions of a historical alignment site from each of the at least one historical two-dimensional image and the at least one historical three-dimensional image in each set of historical images, wherein each of the historical alignment regions includes a historical position information. The surgical image alignment method as described in claim 2, wherein the model building step further includes: converting the at least one historical three-dimensional image in each set of historical images into at least one historical two-dimensional projection image with a first viewing angle or a second viewing angle through an image projection conversion technology; and using the historical position information of the historical alignment areas as an initial position, obtaining a historical rotation angle and a historical translation amount between the at least one historical three-dimensional image in each set of historical images and the at least one historical two-dimensional image at the first viewing angle or the second viewing angle through the at least one historical two-dimensional projection image. The method for positioning surgical images as described in claim 3, wherein the first viewing angle is a sideways viewing angle and the second viewing angle is a top-down viewing angle. A surgical image registration method as described in claim 1, wherein the model algorithm is one of a Generative Adversarial Networks algorithm and a Deep Iterative 2D/3D Registration algorithm. The surgical image alignment method as described in claim 1 further comprises: Using an imaging device to take the actual two-dimensional image of the actual part to be aligned, wherein the actual two-dimensional image includes actual position information. A surgical image alignment system uses a computer system to execute a surgical image alignment method as described in any one of claims 1 to 6. The surgical image alignment system as described in claim 7 also includes an imaging device, which is a C-arm X-ray machine with a transmitting end and a receiving end. The surgical image alignment system as described in claim 7 further includes: a computer tomography scanning device for photographing the actual part to be aligned to obtain the actual three-dimensional image.