WO2024259613A1

WO2024259613A1 - Methods for detecting and displaying calcification pixels in x-ray images using artificial neural networks

Info

Publication number: WO2024259613A1
Application number: PCT/CN2023/101579
Authority: WO
Inventors: Xuran ZHAO; Peng Chen
Original assignee: Shenzhen Xpectvision Technology Co Ltd
Current assignee: Shenzhen Xpectvision Technology Co Ltd
Priority date: 2023-06-21
Filing date: 2023-06-21
Publication date: 2024-12-26
Anticipated expiration: 2025-12-21
Also published as: CN119137620A

Abstract

Disclose herein is a method including obtaining a training data set which comprises multiple actual normal tissue images and multiple calcification scattered tissue images, and configuring a computer system that implements a generative neural network using the training data set and a discriminator neural network. An output of the generative neural network is a processed normal tissue image generated from an image input into the generative neural network. An output of the discriminator neural network is a probability of an image input into the discriminator neural network being an actual normal tissue image.

Description

METHODS FOR DETECTING AND DISPLAYING CALCIFICATION PIXELS IN X-RAY IMAGES USING ARTIFICIAL NEURAL NETWORKS

Background

Breast cancer ranks first in the mortality of female cancers worldwide. In order to reduce the impact of breast cancer on women's health, early detection and effective treatment are the primary methods to improve the survival rate of patients.

Breast calcification is one of the main signs of breast cancer, and mammography is the most commonly used imaging method in breast cancer screening and diagnosis.

Breast calcifications are highly absorptive to X-rays and therefore often appear as bright spots on X-ray images. The diameter of small calcifications is as small as a few pixels, and the contrast is low, so it is easy to miss diagnosis in the high-intensity reading environment of radiologists.

Summary

Disclose herein is a method comprising: obtaining a training data set which comprises multiple actual normal tissue images and multiple calcification scattered tissue images; and configuring a computer system that implements a generative neural network, using the training data set and a discriminator neural network. An output of the generative neural network is a processed normal tissue image generated from an image input into the generative neural network. An output of the discriminator neural network is a probability of an image input into the discriminator neural network being an actual normal tissue image.

In an aspect, in each of the multiple calcification scattered tissue images, a region of interest is outlined which comprises multiple calcification pixels and multiple normal tissue pixels.

In an aspect, for each image of the multiple calcification scattered tissue images, a number of all pixels of the region of interest of said each image is at least 10 times a number of all calcification pixels of the region of interest of said each image.

In an aspect, for each image of the multiple calcification scattered tissue images, a number of all pixels of the region of interest of said each image is at least 50%a number of all pixels of said each image.

In an aspect, said configuring the computer system comprises: computing a first loss function of parameters of the computer system, the parameters representing associations among nodes of the generative neural network, the first loss function representing (A) deviation between the image input into the generative neural network and a generated image output by the generative neural network when the image input into the generative neural network is an actual normal tissue image, and (B) the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image; and upon determination that a first termination condition is not satisfied, adjusting the values of the parameters.

In an aspect, said configuring the computer system comprises: computing a second loss function of associations among nodes of the discriminator neural network, the second loss function representing the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image; and upon determination that a second termination condition is not satisfied, adjusting the associations among nodes of the discriminator neural network.

In an aspect, each of the multiple actual normal tissue images and the multiple calcification scattered tissue images is an X-ray image.

Also disclosed herein is a method, comprising: generating with a neural network a processed normal tissue image based on a calcification scattered tissue image; and generating a calcification image based on (A) the calcification scattered tissue image and (B) the processed normal tissue image.

In an aspect, said generating the calcification image is based on a difference between (A) the calcification scattered tissue image and (B) the processed normal tissue image.

In an aspect, said generating the calcification image comprises subtracting the processed normal tissue image from the calcification scattered tissue image resulting in a residual image.

In an aspect, said generating the calcification image further comprises, for each pixel of the residual image whose value is less than 0, setting a pixel value of said each pixel to 0.

In an aspect, the method further comprises generating a calcification enhanced image based on (A) the calcification scattered tissue image and (B) the calcification image.

In an aspect, said generating the calcification enhanced image comprises: multiplying the calcification image by a coefficient resulting in a multiply image; and adding the multiply image to the calcification scattered tissue image.

In an aspect, the method further comprises filtering out single pixel bright spots in the calcification image.

In an aspect, the method further comprises generating the calcification scattered tissue image, which comprises: creating a foreground mask for an input image; and cropping the input image based on the foreground mask, resulting in the calcification scattered tissue image.

In an aspect, the processed normal tissue image and the calcification scattered tissue image are of the same size.

In an aspect, for each calcification pixel of the calcification scattered tissue image, the corresponding pixel of the processed normal tissue image is at a different degree of brightness; and for each normal tissue pixel of the calcification scattered tissue image, the corresponding pixel of the processed normal tissue image is at the same degree of brightness.

In an aspect, the calcification scattered tissue image is an X-ray image.

Also disclosed herein is a computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method above.

Brief Description of Figures

Fig. 1A –Fig. 1D show X-ray images of human breast regions.

Fig. 2 shows a flowchart generalizing a method for training artificial neural networks, according to an embodiment.

Fig. 3A –Fig. 3D show X-ray images of human breast regions.

Fig. 4 shows a flowchart generalizing a method for detecting and displaying calcification pixels in X-ray images, according to an embodiment.

Detailed Description

TRAINING OF ARTIFICIAL NEURAL NETWORKS FOR GENERATING PROCESSED NORMAL TISSUE IMAGES

DEFINITIONS

A normal tissue pixel in an X-ray image of a specimen (e.g., a region of a human female breast) is a pixel caused by X-rays that have interacted with normal tissues in the specimen. In an embodiment, based on empirical data, a threshold value may be specified, and a pixel may be considered a normal tissue pixel if its degree of brightness is at most the specified threshold value.

A calcification pixel in an X-ray image of a specimen is a pixel caused by X-rays that have interacted with calcification in the specimen. In an embodiment, a pixel may be considered a calcification pixel if its degree of brightness exceeds the specified threshold value.

Note that calcifications are more absorptive to X-rays than normal tissues and therefore appear as brighter spots on X-ray images. In other words, on X-ray images, calcification pixels have higher degrees of brightness than normal tissue pixels.

An actual normal tissue image is an X-ray image that has normal tissue pixels but no calcification pixels. The actual normal tissue image is captured by an X-ray detector.

A calcification scattered tissue image is an X-ray image that has both normal tissue pixels and calcification pixels. The calcification scattered tissue image is captured by an X-ray detector.

A processed normal tissue image is an image that has normal tissue pixels but no calcification pixels. The processed normal tissue image is generated by an artificial neural network based on a calcification scattered tissue image.

A calcification image is an image that has (A) calcification pixels and (B) the remaining pixels having degrees of brightness way below the specified threshold value and near zero. As a result, the calcification image displays bright calcification pixels on a dark background.

TRAINING DATA SET

In an embodiment, a training data set may be collected from hospitals; the training data set may include (A) multiple actual normal tissue images of human breast regions and (B) multiple calcification scattered tissue images of human breast regions.

For illustration, Fig. 1A shows an actual normal tissue image 110 of the training data set. Note that there are only normal tissue pixels and no calcification pixels in the actual normal tissue image 110.

For illustration, Fig. 1B shows a calcification scattered tissue image 120 of the training data set. Note that there are both normal tissue pixels and calcification pixels (e.g., calcification pixels 123) in the calcification scattered tissue image 120.

In an embodiment, the training data set may be obtained by (i.e., sent to or loaded to) a computer system (not shown) .

ARTIFICIAL NEURAL NETWORKS

In an embodiment, a generative neural network and a discriminator neural network may be constructed and trained by the computer system using the training data set.

In other words, the computer system is configured to implement the generative neural network, using the training data set and the discriminator neural network.

GENERATIVE NEURAL NETWORK

In an embodiment, after being trained by the computer system (i.e., once implemented by the computer system) , the generative neural network may be used to (A) receive as input a calcification scattered tissue image of a human breast region and (B) generate as output a processed normal tissue image based on the received calcification scattered tissue image.

For illustration, with reference to Fig. 1C and Fig. 1D, after being trained, the generative neural network may receive as input a calcification scattered tissue image 130 (Fig. 1C) and generate as output a processed normal tissue image 140 (Fig. 1D) based on the calcification scattered tissue image 130. By appearance, it almost looks as if all the calcification pixels of the calcification scattered tissue image 130 (e.g., calcification pixels 133) were replaced by pixels similar to the neighboring normal tissue pixels, resulting in the processed normal tissue image 140 (Fig. 1D) .

DISCRIMINATOR NEURAL NETWORK

In an embodiment, after being trained by the computer system, the discriminator neural network may (A) receive as input an input image, and (B) generate as output a probability of that input image being an actual normal tissue image.

FLOWCHART FOR NEURAL NETWORK TRAINING

Fig. 2 shows a flowchart 200 generalizing the method for training the generative neural network and the discriminator neural network, using the training data set, according to an embodiment.

In step 210, the method may include obtaining a training data set which comprises multiple actual normal tissue images and multiple calcification scattered tissue images. For example, in the embodiments described above, the computer system obtains the training data set which includes (A) multiple actual normal tissue images and (B) multiple calcification scattered tissue images.

In step 220, the method may include configuring a computer system that implements a generative neural network, using the training data set and a discriminator neural network. For example, in the embodiments described above, the computer system constructs and trains the generative neural network and the discriminator neural network, using the training data set.

In addition, in step 220, an output of the generative neural network is a processed normal tissue image generated from an image input into the generative neural network. For example, in the embodiments described above, the generative neural network (A) receives as input a calcification scattered tissue image of a human breast region and (B) generates as output a processed normal tissue image based on the received calcification scattered tissue image.

In addition, in step 220, an output of the discriminator neural network is a probability of an image input into the discriminator neural network being an actual normal tissue image. For example, in the embodiments described above, the discriminator neural network (A) receives as input an input image, and (B) generate as output a probability of that input image being an actual normal tissue image.

OTHER EMBODIMENTS REGARDING NETWORK TRAINING

ANNOTATION BY REGIONAL OUTLINING

In an embodiment, in each of the multiple calcification scattered tissue images of the training data set, a region of interest may be outlined which includes multiple calcification pixels and multiple normal tissue pixels. For example, with reference to Fig. 1B, in the calcification scattered tissue image 120, a region of interest 127 may be outlined using a polygon 125. The region of interest 127 includes multiple calcification pixels (e.g., the calcification pixels 123) and multiple normal tissue pixels.

In an embodiment, for each image of the multiple calcification scattered tissue images of the training data set, the region of interest of said each image does not have to include all the calcification pixels of said each image. For example, with reference to Fig. 1B, the region of interest 127 of the calcification scattered tissue image 120 does not include calcification pixels 129.

In an embodiment, the regional outlining (e.g., drawing the polygon 125 of Fig. 1B) for each of the multiple calcification scattered tissue images of the training data set may be performed manually by radiologists or other professionals before the training data set is sent to the computer system for training the generative neural network and the discriminator neural network.

MORE NORMAL TISSUE PIXELS THAN CALCIFICATION PIXELS IN REGION OF INTEREST

In an embodiment, for each image of the multiple calcification scattered tissue images of the training data set, the number of all pixels of the region of interest of said each image may be at least 10 times the number of all calcification pixels of the region of interest of said each image.

For example, with reference to Fig. 1B, for the calcification scattered tissue image 120 of the training data set, the number of all pixels of the region of interest 127 may be at least 10 times the number of all calcification pixels (including the calcification pixels 123) of the region of interest 127. In other words, the number of all normal tissue pixels of the region of interest 127 is at least 9 times the number of all calcification pixels of the region of interest 127.

REGION OF INTEREST IS AT LEAST 50%THE IMAGE

In an embodiment, for each image of the multiple calcification scattered tissue images of the training data set, the number of all pixels of the region of interest of said each image may be at least 50%the number of all pixels of said each image.

For example, with reference to Fig. 1B, for the calcification scattered tissue image 120 of the training data set, the number of all pixels of the region of interest 127 may be at least 50%the number of all pixels of the calcification scattered tissue image 120.

FIRST LOSS FUNCTION FOR TRAINING GENERATIVE NEURAL NETWORK

In an embodiment, a first loss function for training the generative neural network may be computed based on 2 components: (A) deviation between the image input into the generative neural network and a generated image output by the generative neural network based on the input image, when the image input into the generative neural network is an actual normal tissue image, and (B) the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image.

Note that during the training of the generative neural network, the image output by the generative neural network may include calcification pixels, and therefore is called “generated image” instead of “processed normal tissue image” . After training, the output of the generative neural network no longer includes calcification pixels and therefore can be called “processed normal tissue image” . Note that, by definition (stated above) , a processed normal tissue image includes no calcification pixels.

In an embodiment, component (A) above may be computed using the following formula:

In an embodiment, component (B) above may be computed using the following formula:

As a result, in an embodiment, the first loss function F₁ may be computed using the following formula:
F₁=λ*L₁ (G) +L_cGAN (G, D)

wherein, in the formulas above, λ is a model hyperparameter defined empirically, and δ is a judgment function.

In an embodiment, the training of the generative neural network may include, upon determination that a first termination condition is not satisfied, adjusting the values of the parameters representing associations among nodes of the generative neural network.

In an embodiment, the first termination condition may include one or more scenarios selected from the following: minimization of the first loss function; maximization of the first loss function; reaching a certain number of iterations; reaching a value of the first loss function equal to or beyond a certain threshold value; reaching a certain computation time; and/or reaching a value of the first loss function within an acceptable error limit.

SECOND LOSS FUNCTION FOR TRAINING DISCRIMINATOR NEURAL NETWORK

In an embodiment, a second loss function for training the discriminator neural network may be computed based on the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image.

In an embodiment, this probability may be computed using the following formula:

As a result, in an embodiment, the second loss function F₂ may be computed using the following formula:
F₂=L_cGAN (G, D)

In an embodiment, the training of the discriminator neural network may include, upon determination that a second termination condition is not satisfied, adjusting the associations among nodes of the discriminator neural network.

In an embodiment, the second termination condition may include one or more scenarios selected from the following: minimization of the second loss function; maximization of the second loss function; reaching a certain number of iterations; reaching a value of the second loss function equal to or beyond a certain threshold value; reaching a certain computation time; and/or reaching a value of the second loss function within an acceptable error limit.

ALTERNATIVE EMBODIMENTS FOR NEURAL NETWORK TRAINING

In the embodiments described above, the computer system both (A) obtains the training data set and (B) trains the generative neural network and the discriminator neural network. Alternatively, a first computer system may obtain the training data set; and a second computer system may train the generative neural network and the discriminator neural network, using the training data set which the second computer system may receive from the first computer system.

DETECTING AND DISPLAYING OF CALCIFICATION PIXELS IN CALCIFICATION SCATTERED TISSUE IMAGE

In an embodiment, with reference to Fig. 3A and Fig. 3B, after being trained as described above, the generative neural network may be used to generate a processed normal tissue image 320 (Fig. 3B) based on a calcification scattered tissue image 310 (Fig. 3A) .

Note that the pixels of the processed normal tissue image 320 (Fig. 3B) corresponding to the calcification pixels (bright spots) of the calcification scattered tissue image 310 (Fig. 3A) are normal tissue pixels. By appearance, it almost looks as if the calcification pixels of the calcification scattered tissue image 310 (Fig. 3A) were replaced by pixels similar to the neighboring normal tissue pixels, resulting in the processed normal tissue image 320 (Fig. 3B) .

Next, in an embodiment, with reference to Fig. 3C, a calcification image 330 may be generated based on (A) the calcification scattered tissue image 310 (Fig. 3A) and (B) the processed normal tissue image 320 (Fig. 3B) . By appearance, it almost looks as if the calcification pixels of the calcification scattered tissue image 310 (Fig. 3A) were kept unchanged while the remaining pixels were blackened, resulting in the calcification image 330 (Fig. 3C) .

FLOWCHART FOR DETECTING AND DISPLAYING CALCIFICATION PIXELS

Fig. 4 shows a flowchart 400 generalizing the method for detecting and displaying the calcification pixels in a calcification scattered tissue image.

In step 410, the method may include generating with a neural network a processed normal tissue image based on a calcification scattered tissue image. For example, in the embodiments described above, with reference to Fig. 3A and Fig. 3B, the generative neural network generates the processed normal tissue image 320 (Fig. 3B) based on the calcification scattered tissue image 310 (Fig. 3A) .

In step 420, the method may include generating a calcification image based on (A) the calcification scattered tissue image and (B) the processed normal tissue image. For example, in the embodiments described above, with reference to Fig. 3A -Fig. 3C, the calcification image 330 (Fig. 3C) is generated based on (A) the calcification scattered tissue image 310 (Fig. 3A) and (B) the processed normal tissue image 320 (Fig. 3B) .

OTHER EMBODIMENTS FOR CALCIFICATION DISPLAY

DETAILS ON HOW TO GENERATE CALCIFICATION IMAGE

In an embodiment, with reference to Fig. 3A -Fig. 3C, the calcification image 330 (Fig. 3C) may be generated based on the difference between (A) the calcification scattered tissue image 310 (Fig. 3A) and (B) the processed normal tissue image 320 (Fig. 3B) .

In an embodiment, with reference to Fig. 3A -Fig. 3C, the generation of the calcification image 330 (Fig. 3C) may include subtracting the processed normal tissue image 320 (Fig. 3B) from the calcification scattered tissue image 310 (Fig. 3A) , resulting in a residual image (not shown) .

Then, in an embodiment, for each pixel of the residual image whose value is less than 0, the pixel value of said each pixel may be set to 0.

In an embodiment, with reference to Fig. 3C, single pixel bright spots in the calcification image 330 may be filtered out.

CALCIFICATION ENHANCED IMAGE

In an embodiment, with reference to Fig. 3A, Fig. 3C, and Fig. 3D, a calcification enhanced image 340 (Fig. 3D) may be generated based on (A) the calcification scattered tissue image 310 (Fig. 3A) and (B) the calcification image 330 (Fig. 3C) .

In an embodiment, the generation of the calcification enhanced image 340 (Fig. 3D) may include multiplying the calcification image 330 (Fig. 3C) by a coefficient resulting in a multiply image (not shown) ; and then adding (pixel by pixel) the multiply image to the calcification scattered tissue image 310 (Fig. 3A) .

GENERATION OF CALCIFICATION SCATTERED TISSUE IMAGE

In an embodiment, with reference to Fig. 3A, the calcification scattered tissue image 310 may be generated by (A) creating a foreground mask for an input image; and (B) cropping the input image based on the foreground mask, resulting in the calcification scattered tissue image 310.

CALCIFICATION SCATTERED TISSUE IMAGE VS. PROCESSED NORMAL TISSUE IMAGE

In an embodiment, the calcification scattered tissue image 310 (Fig. 3A) and the processed normal tissue image 320 (Fig. 3B) may be of the same size.

In an embodiment, for each calcification pixel of the calcification scattered tissue image 310 (Fig. 3A) , the corresponding pixel of the processed normal tissue image 320 (Fig. 3B) may be at a different (e.g., lower) degree of brightness. Alternatively, each pixel in the processed normal tissue image 320 (Fig. 3B) corresponding to a calcification pixel in the calcification scattered tissue image 310 (Fig. 3A) may be inferred from the normal tissue pixels of the processed normal tissue image 320 (Fig. 3B) in the vicinity of said each pixel.

In an embodiment, for each normal tissue pixel of the calcification scattered tissue image 310 (Fig. 3A) , the corresponding pixel of the processed normal tissue image 320 (Fig. 3B) may be at the same degree of brightness. Alternatively, all pixels in the processed normal tissue image 320 (Fig. 3B) corresponding to normal tissue pixels in the calcification scattered tissue image 310 (Fig. 3A) may be very similar (e.g., within 5%of the full dynamic range) .

In an embodiment, the calcification scattered tissue image 310 (Fig. 3A) may be an X-ray image.

While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

A method, comprising:

obtaining a training data set which comprises multiple actual normal tissue images and multiple calcification scattered tissue images; and

configuring a computer system that implements a generative neural network, using the training data set and a discriminator neural network,

wherein an output of the generative neural network is a processed normal tissue image generated from an image input into the generative neural network, and

wherein an output of the discriminator neural network is a probability of an image input into the discriminator neural network being an actual normal tissue image.
The method of claim 1, wherein in each of the multiple calcification scattered tissue images, a region of interest is outlined which comprises multiple calcification pixels and multiple normal tissue pixels.
The method of claim 2, wherein for each image of the multiple calcification scattered tissue images, a number of all pixels of the region of interest of said each image is at least 10 times a number of all calcification pixels of the region of interest of said each image.
The method of claim 2, wherein for each image of the multiple calcification scattered tissue images, a number of all pixels of the region of interest of said each image is at least 50%a number of all pixels of said each image.
The method of claim 1, wherein said configuring the computer system comprises:

computing a first loss function of parameters of the computer system, the parameters representing associations among nodes of the generative neural network, the first loss function representing (A) deviation between the image input into the generative neural network and a generated image output by the generative neural network when the image input into the generative neural network is an actual normal tissue image, and (B) the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image; and

upon determination that a first termination condition is not satisfied, adjusting values of the parameters.
The method of claim 5, wherein said configuring the computer system comprises:

computing a second loss function of associations among nodes of the discriminator neural network, the second loss function representing the probability, determined by the discriminator neural network, of the generated image being an actual normal tissue image, when the image input into the generative neural network is a calcification scattered tissue image; and

upon determination that a second termination condition is not satisfied, adjusting the associations among nodes of the discriminator neural network.
The method of claim 1, wherein each of the multiple actual normal tissue images and the multiple calcification scattered tissue images is an X-ray image.
A computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method of any one of claims 1-7.
A method, comprising:

generating with a neural network a processed normal tissue image based on a calcification scattered tissue image; and

generating a calcification image based on (A) the calcification scattered tissue image and (B) the processed normal tissue image.
The method of claim 9, wherein said generating the calcification image is based on a difference between (A) the calcification scattered tissue image and (B) the processed normal tissue image.
The method of claim 10, wherein said generating the calcification image comprises subtracting the processed normal tissue image from the calcification scattered tissue image resulting in a residual image.
The method of claim 11, wherein said generating the calcification image further comprises, for each pixel of the residual image whose value is less than 0, setting a pixel value of said each pixel to 0.
The method of claim 9, further comprising generating a calcification enhanced image based on (A) the calcification scattered tissue image and (B) the calcification image.
The method of claim 13, wherein said generating the calcification enhanced image comprises:

multiplying the calcification image by a coefficient resulting in a multiply image; and

adding the multiply image to the calcification scattered tissue image.
The method of claim 9, further comprising filtering out single pixel bright spots in the calcification image.
The method of claim 9, further comprising generating the calcification scattered tissue image, wherein said generating the calcification scattered tissue image comprises:

creating a foreground mask for an input image; and

cropping the input image based on the foreground mask, resulting in the calcification scattered tissue image.
The method of claim 9, wherein the processed normal tissue image and the calcification scattered tissue image are of the same size.
The method of claim 9,

wherein for each calcification pixel of the calcification scattered tissue image, the corresponding pixel of the processed normal tissue image is at a different degree of brightness, and

wherein for each normal tissue pixel of the calcification scattered tissue image, the corresponding pixel of the processed normal tissue image is at the same degree of brightness.
The method of claim 9, wherein the calcification scattered tissue image is an X-ray image.
A computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method of any one of claims 9-19.