CN121481902A

CN121481902A - A virtual staining method, apparatus, and storage medium based on broadband autofluorescence.

Info

Publication number: CN121481902A
Application number: CN202610013237.6A
Authority: CN
Inventors: 陈永台; 祁绩; 毕一鸣; 周长江
Original assignee: Zhejiang Lab
Current assignee: Zhejiang Lab
Priority date: 2026-01-07
Filing date: 2026-01-07
Publication date: 2026-02-06

Abstract

This invention discloses a virtual staining method, apparatus, and storage medium based on broadband autofluorescence. The method utilizes a fluorescence microscopy imaging system configured with multiple excitation wavelengths and broadband autofluorescence channels. First, it acquires broadband autofluorescence signals from various endogenous fluorophores in the sample at different excitation wavelengths. Then, the broadband autofluorescence images are sequentially assigned to the blue, green, and red color channels and synthesized into a virtual RGB image. Further image contrast enhancement processing improves the morphological contrast of different tissue structures and lesion areas, achieving label-free acquisition of histopathological information. Compared to narrow-spectrum autofluorescence imaging, this invention utilizes the different wavelengths of autofluorescence emitted by various endogenous fluorophores after excitation, coupling them into a broadband autofluorescence signal, thereby enhancing the information content and signal-to-noise ratio of the spectral channels in a single image, eliminating the need for chemical staining.

Description

Virtual dyeing method and device based on broad spectrum autofluorescence and storage medium

Technical Field

The invention relates to the technical field of pathological fluorescence staining, in particular to a virtual staining method, device and storage medium based on broad-spectrum autofluorescence.

Background

Cancer has become the most serious challenge for human life health, and rapid and accurate pathological diagnosis is critical to improve prognosis of cancer patients. Currently, stained sections such as hematoxylin-eosin (H & E) are used as gold standard for histopathological diagnosis, and have a number of drawbacks. On the one hand, the preparation process of the H & E and other staining slices is complicated, a pathologist with special and experienced needs to operate, and consumes a great deal of time, especially, the improvement of diagnosis efficiency is severely limited in the time-invaluable intraoperative histopathological diagnosis scene, which is unfavorable for the prognosis of patients, and on the other hand, the H & E and other staining or marking can generate irreversible chemical influence on tissue characteristics, which prevents the reuse of tissues in other subsequent diagnosis projects (such as molecular detection), and is extremely wasteful for precious tissue samples. How to reduce or even eliminate the histochemical staining process has become an urgent clinical issue to be addressed.

In recent years, in response to the above-mentioned problems, the technology of label-free imaging characterization has been rapidly developed. The imaging technology based on autofluorescence generates image contrast by virtue of autofluorescence characteristics of endogenous fluorophores of biological tissues, can acquire tissue function and structure information, and has great potential in the field of tissue label-free imaging characterization. However, the existing tissue label-free imaging characterization workflow based on autofluorescence mostly adopts tissue narrow-spectrum autofluorescence signals, and single acquisition of the narrow-spectrum autofluorescence signals only can utilize autofluorescence of a small amount of endogenous fluorophores, so that the information amount and the signal to noise ratio obtained by a single spectrum imaging channel are limited, the image visualization enhancement is realized by virtual dyeing based on a deep learning method, but the technical route of the deep learning is complex, the generalization capability of a model is limited, and an incorrect virtual dyeing result can be generated.

Therefore, a virtual staining method, device and storage medium based on broad spectrum autofluorescence are needed to solve the above problems.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a virtual dyeing method, a virtual dyeing device and a storage medium based on broad-spectrum autofluorescence.

The invention aims at realizing the following technical scheme that the first aspect of the embodiment of the invention provides a virtual dyeing method based on wide-spectrum autofluorescence, which comprises the following steps:

(1) Placing a tissue sample to be detected on an objective table of a fluorescence microscopic imaging system, controlling to close an excitation light source channel through a shutter in an illumination light path, starting a camera and setting gain and exposure values;

(2) Switching a fluorescence cube in a fluorescence microscopic imaging system, and configuring a required excitation wavelength channel and a broad spectrum autofluorescence channel;

(3) Turning on an excitation light source of the fluorescence microscopic imaging system, adjusting the power of the excitation light source, switching an imaging channel to an eyepiece end, observing a tissue sample to be detected through the eyepiece, and adjusting the position of an objective table to focus so as to move a target area to the center of a visual field;

(4) The imaging channel is switched to the camera end, the focal length of the objective table and the power of the excitation light source are finely adjusted to the optimal image contrast while the display interface of the camera is observed, and the wide-spectrum autofluorescence image at the moment is saved;

(5) And mapping the wide-spectrum autofluorescence images to different color channels of a color space respectively, and synthesizing the wide-spectrum autofluorescence images of the color channels to form a virtual RGB image so as to realize virtual dyeing.

Further, the tissue sample to be measured is a tissue slice or a thick tissue.

Further, in the step (1), the fluorescence microscopic imaging system is a fluorescence microscopic imaging system in the form of an upright fluorescence microscope, an inverted fluorescence microscope or any other optical path layout;

The gain of the camera is 0-20dB, and the exposure value of the camera is 1ms-1s.

Further, the fluorescence cube comprises a narrow-spectrum excitation filter, a long-pass dichroic mirror and a long-pass fluorescence filter, and is used for coupling into a wide-spectrum autofluorescence signal by utilizing autofluorescence of different wavelengths of radiation after various endogenous fluorophores are excited.

Further, the step (2) specifically includes:

The method comprises the steps of switching a fluorescence cube in a fluorescence microscopic imaging system, firstly configuring a corresponding narrow-spectrum excitation filter based on a required excitation wavelength, then configuring a cut-off wavelength of a long-pass dichroic mirror to be larger than the selected excitation wavelength, and then configuring the cut-off wavelength of the long-pass fluorescence filter to be slightly larger than the cut-off wavelength of the long-pass dichroic mirror to complete the configuration of a required excitation wavelength channel and a required wide-spectrum autofluorescence channel.

Further, the excitation light source is an LED or laser lighting system, the wavelength range of the excitation light source covers deep ultraviolet to infrared, and the excitation light source is matched with a fluorescent cube for use, so that excitation of multiple wavelengths is realized;

The camera is sCMOS, CMOS, CCD scientific research camera or industrial camera;

The at least three different excitation wavelengths include a lowest excitation wavelength, a middle excitation wavelength, and a highest excitation wavelength.

Further, the mapping the broad-spectrum autofluorescence images to different color channels of the color space respectively specifically includes:

and mapping the broad-spectrum autofluorescence image corresponding to the lowest excitation wavelength to a blue channel, mapping the broad-spectrum autofluorescence image corresponding to the middle excitation wavelength to a green channel, and mapping the broad-spectrum autofluorescence image corresponding to the highest excitation wavelength to a red channel according to the order of the excitation wavelengths from low to high.

Further, the method further comprises the following steps:

image contrast enhancement processing is performed on the virtual RGB image, wherein the image contrast enhancement processing method comprises high-pass filtering, gray stretching, nonlinear gray transformation and contrast-limiting adaptive histogram equalization.

The second aspect of the embodiment of the invention provides a virtual dyeing device based on broad spectrum autofluorescence, which comprises one or more processors and a memory, wherein the memory is coupled with the processors, and is used for storing program data, and the processor is used for executing the program data to realize the virtual dyeing method based on broad spectrum autofluorescence.

A third aspect of the embodiments of the present invention provides a computer-readable storage medium having stored thereon a program for implementing the above-described virtual staining method based on broad spectrum autofluorescence when executed by a processor.

Compared with the prior art, the invention has the beneficial effects that:

(1) According to the invention, the autofluorescence of various endogenous fluorophores with different wavelengths emitted after being excited is coupled into a broad-spectrum autofluorescence signal, so that the morphological information quantity and the signal-to-noise ratio of a spectrum channel under single photographing are enhanced;

(2) The invention can synthesize virtual RGB images based on autofluorescence images of different wide spectrum channels obtained under a plurality of excitation wavelengths, realize quick virtual dyeing, enhance morphological contrast of different tissue structures and lesion areas under the condition of not depending on deep learning, simplify the image data processing flow and improve the efficiency.

(3) The invention can realize the marking-free imaging characterization of tissues, avoid the use of chemical reagents, obtain the tissue morphology identification effect similar to H & E dyeing without chemical dyeing, and shorten the acquisition flow of tissue morphology information.

(4) The invention proves that the broad spectrum autofluorescence imaging method has clinical application potential in rapid pathological diagnosis, is beneficial to improving the examination efficiency, promotes the clinical transformation of a label-free pathological analysis technology based on autofluorescence broad spectrum, assists in pathological diagnosis and provides diagnostic reference for pathologists.

Drawings

FIG. 1 is a flow chart of a virtual staining method based on broad spectrum autofluorescence of the present invention;

FIG. 2 is a graph showing the imaging effect and contrast of a broad spectrum autofluorescence image versus a narrow spectrum autofluorescence image of a colon specimen in accordance with one embodiment of the present invention;

FIG. 3 is a comparison of the morphological information of the tissue structures of the virtual RGB image and the golden standard image of the frozen and dewaxed sections according to one embodiment of the present invention;

FIG. 4 is a time-varying plot of the structural similarity index of the broad spectrum autofluorescence images of the frozen and dewaxed sections at different excitation wavelengths according to one embodiment of the present invention, wherein (a) in FIG. 4 is a time-varying plot of the structural similarity index of the broad spectrum autofluorescence images of the frozen section at different excitation wavelengths, and (b) in FIG. 4 is a time-varying plot of the structural similarity index of the broad spectrum autofluorescence images of the dewaxed section at different excitation wavelengths;

FIG. 5 is a comparison of enhanced virtual RGB images and gold standard images of dewaxed sections of neoplastic liver tissue in accordance with one embodiment of the present invention at different tissue regions;

FIG. 6 is another comparison of enhanced virtual RGB images and gold standard images of dewaxed sections of neoplastic liver tissue in accordance with one embodiment of the present invention at different tissue regions;

FIG. 7 is a comparison of virtual RGB images, virtual RGB to gray scale images, and gold standard images of frozen sections of neoplastic liver tissue in accordance with one embodiment of the present invention;

FIG. 8 is a comparison of virtual RGB gray scale images and gold standard images of frozen sections of neoplastic liver tissue in accordance with one embodiment of the present invention;

Fig. 9 is a schematic structural diagram of a virtual staining apparatus based on broad spectrum autofluorescence according to the present invention.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. It is obvious that the drawings used in the following description are only some embodiments of the present invention, and that other drawings may be obtained from them without inventive faculty for a person skilled in the art. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the accompanying claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the invention. The term "if" as used herein may be interpreted as "at..once" or "when..once" or "in response to a determination", depending on the context.

The present invention will be described in detail with reference to the accompanying drawings. The features of the examples and embodiments described below may be combined with each other without conflict.

Referring to fig. 1, the virtual staining method based on broad spectrum autofluorescence of the invention specifically comprises the following steps:

(1) And placing the tissue sample to be detected on an objective table of a fluorescence microscopic imaging system, closing an excitation light source channel through shutter control in an illumination light path, starting a camera, and setting gain and exposure values.

Further, the tissue Sample to be measured is a tissue slice or a thick tissue, which can be expressed as Sample e { tissue slice, thick tissue }, wherein Sample represents the tissue Sample to be measured. When the tissue sample to be measured is a tissue slice, the tissue sample to be measured can be prepared for the same tissue slice, or a pair of adjacent tissue slices can be selected to prepare the tissue sample to be measured. When a tissue sample to be measured is selected to be prepared on the same tissue slice, a broad spectrum autofluorescence image of the slice is firstly obtained when the slice is not dyed, then the slice is dyed, and then a color image of the dyed slice is obtained. When a pair of adjacent tissue slices are selected to prepare a tissue sample to be measured, directly selecting adjacent undyed white slices and dyed slices, and respectively obtaining a broad spectrum autofluorescence image of the undyed white slices and a color image of the dyed slices. When the tissue sample to be measured is thick tissue, a broad spectrum autofluorescence image of the surface of the undyed tissue block is acquired, then the undyed tissue block is dyed, and then a color image of the dyed tissue block surface is acquired.

Wherein the result after the staining process is used to provide a gold standard, i.e. the color image of the stained tissue sample is capable of providing a gold standard for the unstained tissue sample.

Further, the staining treatment includes hematoxylin-eosin (H & E), immunohistochemistry, immunofluorescence staining and other conventional staining treatment methods.

Further, the fluorescence microscopic imaging system is a fluorescence microscopic imaging system of an upright fluorescence microscope or an inverted fluorescence microscope or any other optical path layout form (such as a cage system built by optical mechanical elements).

Specifically, the tissue sample to be measured is placed on a fluorescent microscopic imaging system objective table, and the objective table can be a manual objective table or an electric objective table. The automatic focusing and scanning imaging function is realized by a control program of a host computer written in Labview or C++. The fluorescence microscopic imaging system is provided with an illumination light path, so that the excitation light source channel is closed through the shutter control in the illumination light path, and the sample photobleaching caused by unexpected strong light excitation can be effectively prevented. The camera is started and appropriate gain and exposure values are set for the camera. Wherein the gain of the camera is 0-20dB, and the exposure value of the camera is 1ms-1s.

(2) The fluorescence cube in the fluorescence microscopic imaging system is switched, and a required excitation wavelength channel and a broad-spectrum autofluorescence channel are configured.

Further, the fluorescence cube comprises a narrow-spectrum excitation filter, a long-pass dichroic mirror and a long-pass fluorescence filter, and is used for coupling into a broad-spectrum autofluorescence signal by utilizing autofluorescence of different wavelengths of radiation after various endogenous fluorophores are excited.

The method comprises the steps of switching a fluorescent cube in a fluorescent microscopic imaging system, firstly configuring a corresponding narrow-spectrum excitation filter based on a required excitation wavelength, then configuring a cut-off wavelength of a long-pass dichroic mirror to be larger than the selected excitation wavelength, and then configuring the cut-off wavelength of the long-pass fluorescent filter to be slightly larger than the cut-off wavelength of the long-pass dichroic mirror.

In this example, a Nikon microscope model ECLIPSE CI-L plus was used to construct a fluorescence microscopy imaging system with a spectral range of 315-900nm. The microscope may be equipped with a plurality of fluorescence cubes, three of which are employed in this embodiment, each containing a narrow-spectrum excitation filter, a long-pass dichroic mirror, and a long-pass fluorescence filter. The fluorescent cube 1 contains a 355+ -25 nm narrow spectrum excitation filter, 400-700nm long-pass dichroic mirror and 410-700nm long-pass fluorescent filter. The fluorescent cube No. 2 contains 470+/-20 nm of narrow-spectrum excitation filter, 505-700nm of long-pass dichroic mirror and 510-700nm of long-pass fluorescent filter. The fluorescent cube No. 3 contains 535+/-25 nm narrow-spectrum excitation filter, 575-700nm long-pass dichroic mirror and 580-700nm long-pass fluorescent filter.

(3) And (3) turning on an excitation light source of the fluorescence microscopic imaging system, adjusting the power of the excitation light source, switching an imaging channel to an eyepiece end, observing a tissue sample to be detected through the eyepiece, adjusting the position of an objective table to focus, and moving a target area to the center of a visual field.

Furthermore, the excitation light source is a high-power LED or a laser illumination system, the wavelength range of the excitation light source covers deep ultraviolet to infrared, and the excitation of multiple wavelengths can be realized by matching with a proper fluorescent cube. Accordingly, the wavelength range of the broad spectrum fluorescence imaging channel in the present invention also covers deep ultraviolet to infrared.

Specifically, an excitation light source of the fluorescence microscopic imaging system is turned on, the power of the excitation light source is adjusted, and the excitation light source is adjusted to a lower level (namely low power), wherein a high-power LED or a laser illumination system is used as the excitation light source to effectively excite the tissue sample to be detected. In this embodiment, a Nikon D-LEDI fluorescent LED lighting system is used, which can emit excitation light with multiple wavelengths, and covers excitation light with center wavelengths of 385nm, 475nm and 550nm, which correspond to the fluorescent cubes 1, 2 and 3 in sequence. Then, the imaging channel is switched to the end of the ocular, the tissue sample to be detected is observed through the ocular, the position of the objective table is adjusted to focus, the focusing can be realized by adjusting the focusing spiral, the objective table is moved after the ocular can clearly see the texture of the tissue sample to be detected, and the target area of the tissue sample to be detected is moved to the center of the visual field.

(4) The imaging channel is switched to the camera end, the focal length of the objective table and the power of the excitation light source are finely adjusted to the optimal image contrast while the display interface of the camera is observed, the wide-spectrum autofluorescence image at the moment is saved, and the wide-spectrum autofluorescence image of the tissue sample to be detected under at least three different excitation wavelengths is obtained by switching the fluorescence cube. Wherein the broad spectrum autofluorescence image at different excitation wavelengths can be expressed asWhereinIndicating an excitation wavelength ofA broad spectrum autofluorescence image of the lower part,Representing the acquisition of a single-channel broad-spectrum autofluorescence image,Indicating an excitation wavelength ofThe corresponding fluorescent cubes are used in the process,Indicating an excitation wavelength ofThe power of the light source is then excited,,AndRepresenting the minimum power and the maximum power of the excitation light source, respectively.

Furthermore, the camera is sCMOS, CMOS, CCD scientific research camera or industrial camera, and black-and-white or color camera can be flexibly selected according to the observation requirement.

Further, the at least three different excitation wavelengths include a lowest excitation wavelength, a middle excitation wavelength, and a highest excitation wavelength.

The imaging channel is switched to the camera end, the camera display interface is observed, the focal length of the objective table is further finely adjusted through fine adjustment Jiao Luoxuan until the image is clear, the power of the excitation light source is adjusted to a proper value (10% -100%) from low to high until the characteristic contrast of the tissue sample under the observation of the camera is optimal, and the wide-spectrum autofluorescence image at the moment is stored. And (3) switching the fluorescent cube, and finely adjusting the power of the excitation light source to obtain wide-spectrum autofluorescence images of the tissue sample to be detected under different excitation wavelengths. The tissue sample is irradiated after 385nm excitation light passes through a 355+/-25 nm narrow-spectrum excitation filter, a camera obtains a wide-spectrum autofluorescence image in a spectrum range of 410-700nm, which passes through a long-pass dichroic mirror and a long-pass fluorescence filter, 475nm excitation light passes through a 470+/-20 nm narrow-spectrum excitation filter and irradiates the tissue sample, a camera obtains a wide-spectrum autofluorescence image in a spectrum range of 510-700nm, which passes through a long-pass dichroic mirror and a long-pass fluorescence filter, 550nm excitation light passes through a 535+/-25 nm narrow-spectrum excitation filter and irradiates the tissue sample, and a camera obtains a wide-spectrum autofluorescence image in a spectrum range of 580-700nm, which passes through the long-pass dichroic mirror and the long-pass fluorescence filter.

Further, the mapping of the broad-spectrum autofluorescence images to different color channels of the color space comprises the specific steps of mapping the broad-spectrum autofluorescence image corresponding to the lowest excitation wavelength to a Blue (Blue) channel, mapping the broad-spectrum autofluorescence image corresponding to the middle excitation wavelength to a Green (Green) channel, and mapping the broad-spectrum autofluorescence image corresponding to the highest excitation wavelength to a Red (Red) channel according to the order of the excitation wavelengths from low to high.

Specifically, a broad-spectrum autofluorescence image obtained by configuration 1 (excitation light center wavelength 385nm, no. 1 fluorescence cube) was mapped to the Blue channel, a broad-spectrum autofluorescence image obtained by configuration 2 (excitation light center wavelength 475nm, no.2 fluorescence cube) was mapped to the Green channel, and a broad-spectrum autofluorescence image obtained by configuration 3 (excitation light center wavelength 550nm, no. 3 fluorescence cube) was mapped to the Red channel. Broad-spectrum autofluorescence images of the Blue channel, the Green channel and the Red channel are synthesized, a virtual RGB image is formed, and virtual dyeing is realized. Wherein the virtual RGB image may be represented as,The generation of a virtual RGB image is indicated,Representing the lowest excitation wavelengthThe corresponding broad-spectrum autofluorescence image is mapped to the Blue channel,Representing the intermediate excitation wavelengthThe corresponding broad-spectrum autofluorescence image is mapped to the Green channel,Indicating the highest excitation wavelengthThe corresponding broad-spectrum autofluorescence image is mapped to the Red channel.

When the broad-spectrum autofluorescence images are sequentially allocated to the Blue, green and Red channels, the allocation order of the broad-spectrum autofluorescence images of different channels among the Blue, green and Red channels can be flexibly configured according to the actual display effect, and the method is not limited to the implementation in the order of the excitation wavelength from low to high.

In still other embodiments, after step (5), further comprising (6) performing an image contrast enhancement process on the virtual RGB image, the image contrast enhancement process may be represented by:

In the formula, Representing the enhanced virtual RGB image,Representing the image contrast enhancement process. Methods of image contrast enhancement processing include, but are not limited to, high-pass filtering, gray stretching, nonlinear gray-scale transformation, limiting contrast adaptive histogram equalization, and the like.

The image contrast enhancement processing is performed on the virtual RGB image to improve the contrast of the image, where the contrast of the image refers to the difference degree of gray values (or colors) between different areas (or pixels) in the image, and the difference degree directly reflects the brightness, details and level of sharpness of the image. Therefore, in order to enhance the contrast of the image, a method of limiting the image contrast enhancement process such as the contrast adaptive histogram equalization may be adopted, for example, the contrast adaptive histogram equalization limiting method may be used to prevent overexposure and detail loss while enhancing the local contrast of the image by dividing the image into sub-blocks and separately performing the histogram equalization process with contrast limitation on each block.

It should be understood that by performing image contrast enhancement processing on the virtual RGB image, the contrast of the virtual RGB image can be enhanced, image overexposure and detail loss are avoided, a lesion region of tissue is highlighted, morphological characteristics of tissue are obtained, and the display effect of the morphological characteristics of tissue structure and the lesion region is improved.

In conclusion, the invention does not need chemical dyeing, utilizes autofluorescence of different wavelengths of radiation after various endogenous fluorophores are excited to couple into a broad-spectrum autofluorescence signal, thereby enhancing morphological information quantity and signal to noise ratio of a spectrum channel under single photographing, and can synthesize virtual RGB images based on autofluorescence images of different broad spectrum channels obtained under a plurality of excitation wavelengths to realize quick virtual dyeing, thereby enhancing morphological contrast of different tissue structures and lesion areas, realizing tissue label-free imaging characterization and providing diagnostic reference for pathologists.

For the same colon sample, the center wavelength of the excitation light source is set to 385nm, and a fluorescent cube No. 1 is matched, and an RGB image of broad-spectrum autofluorescence (i.e., a broad-spectrum autofluorescence image) is obtained by the sCMOS color scientific camera, as shown in fig. 2. Then, the long-wave-pass fluorescence filter in the fluorescent cube 1 is replaced by a narrow-spectrum fluorescence filter (the wavelength of which is 447+/-60 nm), and RGB images of the narrow-spectrum autofluorescence are obtained through the same sCMOS color scientific camera. Based on RGB images of broad spectrum or narrow spectrum autofluorescence, respectively extracting gray images of three channels of Red, green and Blue, and evaluating the information content level of the three channels through information entropy. The method is characterized in that color and tissue structure information of RGB images with wide spectrum autofluorescence acquired in a range of 410-700nm are better than RGB images with narrow spectrum autofluorescence acquired in a range of 447+/-60 nm, information entropy of Red channel, green channel and Blue channel images in the wide spectrum autofluorescence images acquired in the range of 410-700nm are 3.7587, 7.2042 and 7.5959 respectively, tissue morphological characteristics exist, information entropy of Red channel, green channel and Blue channel images in the narrow spectrum autofluorescence images acquired in the range of 447+/-60 nm are 3.4575, 3.1077 and 6.7174 respectively, images of Red channel and Green channel do not show tissue morphological characteristics, and information quantity of wide spectrum autofluorescence images acquired in the range of 410-700nm is better than those of narrow spectrum autofluorescence images acquired in the range of 447+/-60 nm. Therefore, the method obtains the wide-spectrum autofluorescence images under three different excitation lights, synthesizes the virtual RGB images, can realize virtual dyeing, and enhances the display effect of the morphological characteristics of tissues.

For example, fig. 3 shows the recognition result of the tissue structure morphology information by the broad spectrum autofluorescence imaging, as shown in fig. 3, the kidney and liver of the normal mouse are used for representing the parenchymal viscera, the colon of the normal mouse is used for representing the hollow viscera, the broad spectrum autofluorescence images of three color channels of the frozen section and the dewaxed section of the sample and the color images of the adjacent H & E staining sections are respectively obtained, and the virtual RGB images are synthesized by the broad spectrum autofluorescence images of the three color channels, so as to realize the virtual staining. And identifying and analyzing the histomorphology information of the kidney, the liver and the colon by the virtual RGB image, and performing effect evaluation according to the adjacent H & E staining sections, wherein the color image of the H & E staining sections is used as a gold standard to perform effect evaluation on the virtual staining results. Overall, the morphological features of tissue structures were identified from virtual RGB images of frozen and dewaxed sections of kidney, liver and colon of mice, and the specific results are shown in Table 1.

TABLE 1 morphological characterization of tissue structures

Illustratively, frozen and dewaxed sections of normal liver of a mouse are taken as samples, the samples are excited for a long time (60 seconds), and broad-spectrum autofluorescence images of the samples are acquired every 1 second. Based on the image of the starting time of each color channel, calculating the Structural Similarity Index (SSIM) of the images of other times and the starting time, and obtaining the time stability result of the stability of the wide-spectrum autofluorescence image, as shown in fig. 4. Wherein, the graph of the structural similarity index of the wide-spectrum autofluorescence image of the frozen section at different excitation wavelengths with time is shown in (a) of fig. 4, and the graph of the structural similarity index of the wide-spectrum autofluorescence image of the dewaxed section at different excitation wavelengths with time is shown in (b) of fig. 4. From the results, the SSIM values of the broad-spectrum autofluorescence images of frozen and dewaxed slice samples are above 0.7, which shows that the broad-spectrum autofluorescence images have better time stability and can ignore the effect of photobleaching.

By way of example, the method of the invention is adopted to obtain a broad spectrum autofluorescence image of a dewaxed slice (facing postoperative pathological examination) of mouse neoplastic lesion liver tissue, and a virtual RGB image is synthesized to realize virtual staining, and the virtual RGB image is subjected to image contrast enhancement processing to obtain an enhanced virtual RGB image, as shown in fig. 5 and 6, and then, a color image of an adjacent H & E stained slice is taken as a gold standard image to carry out morphological information comparison. Figures 5 and 6 show the results of broad-spectrum autofluorescence imaging of dewaxed sections of neoplastic lesion liver tissue. The white arrows in fig. 5 refer to stained blood cells, the black arrows refer to cell nuclei, wherein the golden blood cells indicated by the white arrows are virtually stained, the stained blood cells indicated by the other white arrows are H & E stained, and as can be seen from fig. 5, the virtual RGB image of the dewaxed section can effectively distinguish between tissue areas of low-differentiation tumor, high-differentiation tumor, necrosis, normal, etc., and the boundary contrast effect of different tissue areas is superior to that of the H & E stained image. And the recognition results of other tumor regions are shown in fig. 6.

As shown in fig. 7 and 8, the method of the present invention is used to obtain a broad-spectrum autofluorescence image of a frozen section (facing to pathological examination in surgery) of mouse neoplastic lesion liver tissue, and to synthesize a virtual RGB image, to implement virtual staining, and to compare morphological information with a color image of an adjacent H & E stained section as a gold standard image. Figures 7 and 8 show the results of broad-spectrum autofluorescence imaging of frozen sections of neoplastic diseased liver tissue. The virtual RGB image, the virtual RGB-to-gray image, and the gold standard image in fig. 7 can identify the tissue areas such as tumor, normal, and necrosis, wherein the virtual RGB image has a distinct band texture between the normal and lesion tissue areas, and has a strong contrast ratio, and the feature has a weak display effect in the gold standard image. As shown in fig. 8, in both the virtual RGB-to-gray image and the gold standard image, the boundary of the range between immune cells and tumor cells in the tumor tissue region can be seen, and in particular, abnormal nuclear division images exist in the tumor tissue region of both images. In addition, the accumulation of immune cells in the vicinity of blood vessels was seen in the normal tissue region of both the virtual RGB-to-gray image and the gold standard image.

It should be noted that, histopathological diagnosis mainly focuses on morphological structural features of tissues, including structural arrangement of tissues, morphological characteristics of size of cells (especially nuclei), and objective morphological information such as deposition of special substances and reaction of matrix, and can focus on distinction between different endogenous fluorophore components. Therefore, the method performs virtual dyeing on the tissue and performs image contrast enhancement treatment, so that the pathological change region of the tissue can be highlighted, the morphological characteristics of the tissue can be obtained, and the display effect of the morphological characteristics of the tissue structure and the pathological change region can be improved. And then, comparing the enhanced virtual RGB image with the corresponding dyed slice image serving as the gold standard, namely, referring to the dyed slice image serving as the gold standard, and constructing the association relationship between the virtual RGB image and pathology, so that pathological diagnosis can be realized.

From the results of the examples, the method can achieve the tissue morphology identification effect equivalent to the H & E staining result, further proves that the method has clinical application potential in rapid pathological diagnosis, establishes the connection between the intra-operative decision (applicable to frozen section) and the post-operative verification (applicable to paraffin section), and promotes the transformation of the label-free tissue pathology technology based on broad-spectrum autofluorescence to clinic.

Corresponding to the embodiment of the virtual dyeing method based on the broad spectrum autofluorescence, the invention also provides an embodiment of the virtual dyeing device based on the broad spectrum autofluorescence.

Referring to fig. 9, the virtual dyeing apparatus based on broad spectrum autofluorescence provided by the embodiment of the invention includes one or more processors and a memory, wherein the memory is coupled to the processors, and the memory is used for storing program data, and the processor is used for executing the program data to implement the virtual dyeing method based on broad spectrum autofluorescence in the above embodiment.

The embodiment of the virtual dyeing apparatus based on broad spectrum autofluorescence can be applied to any device with data processing capability, such as a computer or the like. The apparatus embodiments may be implemented by software, or may be implemented by hardware or a combination of hardware and software. Taking software implementation as an example, the device in a logic sense is formed by reading corresponding computer program instructions in a nonvolatile memory into a memory by a processor of any device with data processing capability. In terms of hardware, as shown in fig. 9, a hardware structure diagram of an apparatus with any data processing capability where the virtual dyeing apparatus based on broad spectrum autofluorescence of the present invention is located is shown in fig. 9, and in addition to a processor, a memory, a network interface, and a nonvolatile memory shown in fig. 9, the apparatus with any data processing capability where the apparatus is located in the embodiment generally includes other hardware according to an actual function of the apparatus with any data processing capability, which is not described herein again.

The implementation process of the functions and roles of each unit in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.

For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present invention. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

The embodiment of the invention also provides a computer readable storage medium, on which a program is stored, which when executed by a processor, implements the virtual dyeing method based on broad spectrum autofluorescence in the above embodiment.

The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any of the data processing enabled devices described in any of the previous embodiments. The computer readable storage medium may also be any device having data processing capabilities, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), an SD card, a flash memory card (FLASH CARD), or the like, provided on the device. Further, the computer readable storage medium may include both internal storage units and external storage devices of any data processing device. The computer readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing apparatus, and may also be used for temporarily storing data that has been output or is to be output.

The foregoing embodiments are merely for illustrating the technical solution of the present invention, but not for limiting the same, and although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that modifications may be made to the technical solution described in the foregoing embodiments or equivalents may be substituted for parts of the technical features thereof, and that such modifications or substitutions do not depart from the spirit and scope of the technical solution of the embodiments of the present invention in essence.

Claims

1. A virtual staining method based on broadband autofluorescence, characterized by comprising the following steps:

(1) Place the tissue sample to be tested on the stage of the fluorescence microscopy system, close the excitation source channel by controlling the shutter in the illumination optical path, start the camera and set the gain and exposure value;

(2) Switch the fluorescence cube in the fluorescence microscopy system and configure the required excitation wavelength channel and broadband autofluorescence channel;

(3) Turn on the excitation source of the fluorescence microscopy system and adjust its power. Switch the imaging channel to the eyepiece end, observe the tissue sample to be tested through the eyepiece, and adjust the stage position to focus so as to move the target area to the center of the field of view.

(4) Switch the imaging channel to the camera end, observe the camera display interface, and fine-tune the focal length of the stage and the power of the excitation light source to the optimal image contrast, and save the broadband autofluorescence image at this time; by switching the fluorescence cube, obtain the broadband autofluorescence images of the tissue sample under test at at least three different excitation wavelengths.

(5) Map the broadband autofluorescence image to different color channels of the color space respectively, and synthesize the broadband autofluorescence image of the color channel to form a virtual RGB image to realize virtual staining.

2. The virtual staining method based on broadband autofluorescence according to claim 1, wherein the tissue sample to be tested is a tissue section or thick tissue.

3. The virtual staining method based on broadband autofluorescence according to claim 1, characterized in that, in step (1), the fluorescence microscopy system is an upright fluorescence microscope, an inverted fluorescence microscope, or a fluorescence microscopy system with any other optical path layout;

The camera gain is 0-20dB, and the camera exposure value is 1ms-1s.

4. The virtual staining method based on broadband autofluorescence according to claim 1, characterized in that the fluorescence cube includes a narrow-spectrum excitation filter, a long-pass dichroic mirror and a long-pass fluorescence filter, used to couple the autofluorescence of different wavelengths emitted by various endogenous fluorophores after excitation into a broadband autofluorescence signal.

5. The virtual staining method based on broadband autofluorescence according to claim 4, characterized in that step (2) specifically includes:

Switch the fluorescence cube in the fluorescence microscopy system. First, configure the corresponding narrow-spectrum excitation filter based on the required excitation wavelength. Then, configure the cutoff wavelength of the long-pass dichroic mirror to be greater than the selected excitation wavelength. Next, configure the cutoff wavelength of the long-pass fluorescence filter to be slightly greater than the cutoff wavelength of the long-pass dichroic mirror, thus completing the configuration of the required excitation wavelength channel and the broadband autofluorescence channel.

6. The virtual staining method based on broadband autofluorescence according to claim 1, characterized in that the excitation light source is an LED or laser illumination system with a wavelength range covering deep ultraviolet to infrared, used in conjunction with a fluorescence cube to achieve excitation at multiple wavelengths;

The camera is an sCMOS, CMOS, CCD scientific research camera or industrial camera.

The at least three different excitation wavelengths include the lowest excitation wavelength, the intermediate excitation wavelength, and the highest excitation wavelength.

7. The virtual staining method based on broadband autofluorescence according to claim 6, characterized in that, mapping the broadband autofluorescence image to different color channels of the color space specifically includes:

According to the order of excitation wavelength from low to high, the broadband autofluorescence image corresponding to the lowest excitation wavelength is mapped to the blue channel, the broadband autofluorescence image corresponding to the middle excitation wavelength is mapped to the green channel, and the broadband autofluorescence image corresponding to the highest excitation wavelength is mapped to the red channel.

8. The virtual staining method based on broadband autofluorescence according to claim 1, characterized in that it further comprises:

Image contrast enhancement processing is performed on virtual RGB images. The methods for image contrast enhancement processing include high-pass filtering, gray-level stretching, nonlinear gray-level transformation, and contrast-limited adaptive histogram equalization.

9. A virtual staining device based on broadband autofluorescence, comprising one or more processors and a memory, characterized in that the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the virtual staining method based on broadband autofluorescence as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that it stores a program thereon, which, when executed by a processor, is used to implement the virtual staining method based on broadband autofluorescence as described in any one of claims 1-8.