WO2017152702A1 - 一种结构光照明显微镜的成像方法及装置 - Google Patents

一种结构光照明显微镜的成像方法及装置 Download PDF

Info

Publication number
WO2017152702A1
WO2017152702A1 PCT/CN2016/112637 CN2016112637W WO2017152702A1 WO 2017152702 A1 WO2017152702 A1 WO 2017152702A1 CN 2016112637 W CN2016112637 W CN 2016112637W WO 2017152702 A1 WO2017152702 A1 WO 2017152702A1
Authority
WO
WIPO (PCT)
Prior art keywords
original image
spatial
light illumination
structured light
original
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/112637
Other languages
English (en)
French (fr)
Inventor
杨怀栋
刘国漩
张四纯
张新荣
金国藩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Publication of WO2017152702A1 publication Critical patent/WO2017152702A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/06—Means for illuminating specimens
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/364—Projection microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison

Definitions

  • the present invention relates to the field of optical microscope technology, and in particular, to an imaging method and apparatus for a structured light illumination microscope.
  • Microscopy is an indispensable research tool in modern life science research.
  • the conventional optical microscope has a resolution limit.
  • SIM Structured Illumination Microscope
  • STORM Stochastic Optical Reconstruction Microscopy
  • PAM Photoactivated Localization Microscopy
  • STED Stimulated Emission Depletion Microscopy
  • the minimum number of original images required for SIM image reconstruction is the shortest.
  • SIM is the most suitable method for observing fast imaging in living cells or wide field of view in various super-resolution imaging methods.
  • Fig. 1 The basic structure of the existing structured light illumination microscope is shown in Fig. 1.
  • a coherent or incoherent collimated wide beam is used as the light source 1.
  • the incident light passes through the light modulating device 2, and is modulated by the lens 3 and the dichroic color.
  • An imaging system consisting of a beam 8 and an objective lens 9 is then projected onto the illumination sample 10 and shaped on the plane in which the illumination sample 10 is located A periodic light intensity distribution.
  • a spatial filtering system consisting of the lens 4, the spatial filter 5 and the lens 6 can also be selectively added to filter out the zero-order diffraction component, and the filtered light continues to pass through the dichroic color division.
  • An imaging system consisting of a beamer 8 and an objective lens 9 interferes with the formation of a structured light illumination pattern at the plane of the illumination sample 10.
  • the light of the illuminated sample is collected by the detector 13 through a microscope system consisting of the objective lens 9 and the lens barrel 11.
  • SIM-series time-lapse SIM
  • the imaging process is as follows:
  • each structured light illumination pattern captures an original image, and the original image is reconstructed to obtain a SIM super-resolution image.
  • the next SIM super-resolution image is reconstructed using the original image under the next set of structured light illumination. The above process is repeated to obtain a plurality of SIM super-resolution images, which constitute a time series of SIM super-resolution images.
  • SIMs the device used to modulate the structured light distribution was a grating. Modulation and switching of different structured light illumination patterns is achieved by translating or rotating the grating. Due to the presence of mechanical motion in the system, the imaging speed is relatively slow. Subsequent SIMs generally use a spatial light modulator (SLM) and a digital micro-mirror device (DMD) to modulate the structured light illumination pattern. Thanks to the fast response of SLM and DMD, coupled with high-sensitivity detectors such as Electron Multiplying CCD (EMCCD) and Scientific CMOS (sCMOS), the exposure time can be greatly reduced, which provides favorable conditions for SIM high-speed imaging.
  • SLM spatial light modulator
  • DMD digital micro-mirror device
  • the present invention provides an imaging method for a structured light illumination microscope, comprising:
  • the preset N pieces of structured light illumination patterns are cyclically switched in a preset order, where N is a preset constant;
  • the original image of the original image sequence and the N-1 original images after the image are reconstructed to obtain a super-resolution image sequence of the sample to be imaged.
  • the image is reconstructed by using the original image of the original image sequence and the N-1 original image after the original image to obtain a super-resolution image sequence of the sample to be imaged, which specifically includes:
  • Each spatial spectral component of the spatial frequency reduction in each spatial spectral component group is weighted and superimposed with each spatial spectral component whose spatial frequency is not changed, to obtain a super-resolution image sequence of the sample to be imaged.
  • the spatial frequency of each spatial spectral component whose spatial frequency is changed in each spatial spectral component group according to the spatial frequency of the preset N-structure light illumination pattern is restored, and the spatial spectral components of the spatial frequency reduction are obtained, which specifically include:
  • each spatial spectral component of the spatial frequency component change in each spatial spectral component group is multiplied by an exponential function corresponding to its spatial frequency change amount to obtain spatial spectral components of the spatial frequency reduction.
  • the photographing the original image of the sample to be imaged under each structured light illumination pattern comprises:
  • the original image of the sample to be imaged under each structured light illumination pattern is taken at an interval preset time.
  • the method further includes:
  • the spatial frequency of each spatial spectral component whose spatial frequency is changed in each spatial spectral component group is restored according to the spatial frequency of the preset N-structure light illumination pattern, and specifically includes:
  • the spatial frequency of each spatial frequency spectral component of the spatial frequency component of each of the deconvolved spatial spectral components is reduced according to the spatial frequency of the predetermined N-structured light illumination pattern.
  • the method before the periodically circulating the preset N pieces of structured light illumination patterns, the method further includes:
  • the method before the image reconstruction is performed on each of the original image sequence and the N-1 original images in the original image sequence, the method further includes:
  • the reconstructing each of the original image in the original image sequence and the N-1 original images after the image sequence comprises:
  • the method further includes:
  • the method before the image reconstruction is performed on each of the original image sequence and the N-1 original images in the original image sequence, the method further includes:
  • Performing image enlargement processing on each original image in the original image sequence may be generally performed by an image interpolation method or by performing zero-padding on the image spectrum and then transforming back into the spatial domain, thereby obtaining image enlargement.
  • the processing may be generally performed by an image interpolation method or by performing zero-padding on the image spectrum and then transforming back into the spatial domain, thereby obtaining image enlargement.
  • the reconstructing each of the original image in the original image sequence and the N-1 original images after the image sequence comprises:
  • the method further comprises: performing image enlargement processing on each super-resolution image in the super-resolution image sequence to obtain super-resolution after image enlargement Image sequence.
  • the method further includes:
  • the structured light illumination pattern is obtained by modulating incident light by a light modulating device disposed in an illumination light path of the structured light illumination microscope, and is obtained by projecting an image of the light modulation device disposed in the illumination light path of the microscope
  • the incident light is modulated, and then subjected to high-pass spatial filtering and projection imaging.
  • the structured light illumination pattern comprises one or more non-zero spatial frequencies.
  • the present invention provides an imaging device for a structured light illumination microscope, comprising:
  • a pattern switching unit configured to cyclically switch the preset N pieces of structured light illumination patterns in a preset order, where N is a preset constant;
  • An original image acquiring unit configured to acquire an original image of the sample to be imaged under each structured light illumination pattern, to obtain an original image sequence of the sample to be imaged;
  • an image reconstruction unit configured to reconstruct an image of each of the original image sequences and the N-1 original images after the original image sequence to obtain a super-resolution image sequence of the sample to be imaged.
  • a super-resolution image sequence of the sample to be imaged as a function of time is obtained, and the time interval of each of the two super-resolution images is equal to the time interval of capturing each of the two original images, compared with the prior art structured light illumination microscope imaging method.
  • the temporal resolution of the imaging method and apparatus of the present invention is greatly improved.
  • FIG. 1 is a schematic structural view of a conventional structured light illumination microscope
  • FIG. 2 is a schematic flow chart of an imaging method of a structured light illumination microscope according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of comparison between an imaging method of a structured light illumination microscope and an existing imaging method according to an embodiment of the present invention
  • FIG. 4 is a schematic light illumination of a non-zero spatial frequency according to an embodiment of the invention. Schematic diagram of the intensity distribution of the pattern;
  • FIG. 5 is a schematic diagram of light intensity distribution of a plurality of non-zero spatial frequency structured light illumination patterns according to an embodiment of the present invention
  • FIG. 6 is a schematic structural diagram of an image forming apparatus for a structured light illumination microscope according to an embodiment of the present invention.
  • FIG. 2 is a schematic flow chart showing an imaging method of a structured light illumination microscope according to an embodiment of the present invention. As shown in FIG. 2, the imaging method of the structured light illumination microscope of the present embodiment includes steps S21 to S23.
  • N is a preset constant
  • the preset N pieces of structured light illumination patterns are grouped, and the set of structured light illumination patterns are cyclically switched in a fixed order.
  • a set of structured light illumination patterns includes five structured light illumination patterns, respectively a b c d e.
  • the cyclic switching of the structured light illumination pattern is performed in the above a b c d e, and the order of the five sets of structured light illumination patterns of the first group can be arbitrarily set, but each structured light illumination pattern is required to be included, that is, the first
  • the order of the set of structured light illumination patterns may be a b c d e, d a c e b, e c a b d , and the like.
  • the order of the structured light illumination patterns of the first group is e d c b a
  • the order of the subsequent structured light illumination patterns is the same as the order of the first group of structured light illumination patterns, ie, during the user observation process, the structure
  • the switching order of the light illumination patterns is e d c b a e d c b a e d c b a...
  • each structured light illumination pattern illuminates the sample to be imaged
  • an original image of the sample to be imaged is taken, and the original image of the sample to be imaged is obtained within the observation time required by the user. sequence.
  • the order of the structured light illumination pattern is e d c b a e d c b a e d c b a... then the original image sequence of the sample to be imaged should be E D C B A E D C B A E D C B A...
  • the original image of the sample to be imaged described above constitutes a sequence of original images in chronological order of shooting.
  • S23 Perform image reconstruction on each original image in the original image sequence and the N-1 original images after the image to obtain a super-resolution image sequence of the sample to be imaged.
  • the first original image in the original image sequence of the obtained sample to be imaged and the N-1 original image after the image are taken, and a SIM super-resolution image is obtained through image reconstruction.
  • Each subsequent original image is reconstructed from the original N-1 original image to obtain a SIM super-resolution image.
  • All SIM super-resolution images are arranged in time to form a super-resolution image sequence.
  • the first original image E of the sample to be imaged and the subsequent 4 original images D C B A are reconstructed to obtain the first SIM super-resolution image.
  • the second original image D is grouped with the following four original images, and reconstructed to obtain a second SIM super-resolution image. That is, the original image E D C B A is reconstructed to obtain a first SIM super-resolution image S(1), and the original image D C B A E is reconstructed to obtain a second SIM super-resolution image S(2), the original image C B A E D reconstructs the third SIM super-resolution image S(3)...
  • the reconstructed SIM super-resolution image is arranged in order to form a super-resolution time series S(1) S(2) S(3) S(4 ) «
  • the reconstruction process of the original image may be performed after the original image sequence of the sample to be imaged is taken, or simultaneously with the original image of the sample to be imaged.
  • the imaging method of the structured light illumination microscope of the present embodiment can be used not only in the structured light illumination microscope SIM, but also in the following technologies: nonlinear structure light illumination microscope NL-SIM, total internal reflection fluorescence structure Light illumination microscope TIRF-SIM, 3D super-resolution imaging of a structured light illumination microscope 3D-SIM, a combination of a light microscope and a structured light illumination microscope Lattice-lightsheet-SIM.
  • nonlinear structure light illumination microscope NL-SIM total internal reflection fluorescence structure Light illumination microscope TIRF-SIM
  • 3D super-resolution imaging of a structured light illumination microscope 3D-SIM a combination of a light microscope and a structured light illumination microscope Lattice-lightsheet-SIM.
  • SIM structured light illumination microscope
  • the imaging method of the structured light illumination microscope of the embodiment can obtain the super-resolution image sequence of the sample to be imaged with time, and the time resolution is greatly improved.
  • step S23 specifically comprises sub-steps S231 to S233 which are not shown in FIG.
  • the N original images are composed of each original image in the original image sequence and N-1 original images thereafter.
  • a set of spatial spectral components can be calculated for every N original images. As described in the above embodiment, a plurality of N original images can be obtained in the original image sequence, and thus, a plurality of spatial spectral component groups can be obtained.
  • Each spatial spectral component group contains some spatial spectral components whose spatial frequencies have changed, and also some spatial spectral components whose frequencies have not changed.
  • each spatial spectral component whose spatial frequency is changed in each spatial spectral component group is restored to its original spatial frequency according to the spatial frequency of the preset N structured light illumination patterns to correctly reproduce the sample to be imaged.
  • the imaging method of the structured light illumination microscope of the embodiment can accurately obtain the super-resolution image sequence of the sample to be imaged, obtain a finer variation process of the sample to be tested, and improve The time resolution of the structured light illumination microscope SIM.
  • step S232 specifically includes sub-steps S2321 to S2323 not shown in FIG. 2:
  • S2321 Determine a spatial frequency change amount of each spatial spectral component of the spatial frequency component change in each spatial spectral component group according to the spatial frequency of the preset N-structure light illumination patterns.
  • the spatial frequency change of the spatial spectral component is determined prior to restoring the spatial frequency of the spatial spectral component of the spatial frequency change.
  • the amount of spatial frequency change is determined according to the spatial frequency of the preset N structured light illumination patterns.
  • S2322 Determine an exponential function corresponding to each spatial frequency change amount in each spatial spectral component group in the spatial domain according to the spatial frequency change amount.
  • Different spatial frequency changes correspond to different exponential functions, and the exponential function corresponding to each spatial spectral component is determined according to the spatial frequency change amount, so that the spatial spectral components of each spatial frequency change can be correctly restored.
  • each spatial spectral component of the spatial frequency component change in each spatial spectral component group by an exponential function corresponding to its spatial frequency change amount, the spatial frequency reduction of the frequency-changed spatial spectral component can be achieved.
  • the super-resolution image corresponding to each original image can be accurately obtained by restoring the spatial frequency of each spatial spectral component.
  • step S22 specifically includes the following steps not shown in FIG. 2:
  • the original time of the sample to be imaged under each structured light illumination pattern is taken at intervals of preset time image.
  • the time interval of two consecutive images taken continuously for reconstructing each super-resolution image should be basically the same.
  • time interval is the interval of the time at which the two original images are integrated.
  • the minimum time interval is greater than or equal to 0.5t, and the time interval is at least 2t.
  • FIG. 3 The specific imaging process of this embodiment is shown in FIG. 3 compared to the imaging method of the existing structured light illumination microscope.
  • the original image sequence of the captured sample to be imaged includes: original image 1, original image 2, ... original image n.
  • the imaging method of the existing structured light illumination microscope adopts 1-5 original images as one group, 6-10 original images as another group, and each set of original images is reconstructed to obtain a SIM super-resolution image.
  • the time interval between each of the two SIM super-resolution images obtained by continuous reconstruction is 5t.
  • the imaging method of the structured light illumination microscope of the embodiment adopts 1-5 original images as one group, 2-6 original images as another group, and 3-7 original images as a next group, each group original
  • the image is reconstructed to obtain a SIM super-resolution image.
  • the time interval between each of the two SIM super-resolution images obtained by continuous reconstruction is t. Equal to the time interval between the original image was taken.
  • the imaging method of the structured light illumination microscope of the embodiment can effectively shorten the super-resolution imaging time interval of the structured light illumination microscope, and effectively improve the time resolution of the structured light illumination microscope imaging.
  • step S231 the above method further The following steps are not shown in Figure 2:
  • Each spatial spectral component in each spatial spectral component group is deconvolved to obtain a deconvolved spatial spectral component group.
  • the deconvolution process can be added before the image reconstruction of the original image or during the image reconstruction process to compensate for the high frequency information in the spatial frequency domain during the imaging process. A decrease in the contrast of the original image caused by the attenuation.
  • the deconvolution process commonly used in SIM imaging methods is Wiener filtering: OTF * F(f) / (
  • the optical transfer function OTF can be obtained by theoretical calculation, which can be obtained by actual measurement, or it can be obtained by iterative calculation using relevant deconvolution software.
  • c is an empirical constant and can be adjusted according to the actual filtering effect.
  • Deconvolution algorithms such as iterative deconvolution and blind deconvolution can be applied to SIM imaging methods.
  • step S232 specifically includes:
  • the spatial frequency of each spatial frequency spectral component of the spatial frequency component of each of the deconvolved spatial spectral components is reduced according to the spatial frequency of the predetermined N-structured light illumination pattern.
  • the imaging method of the structured light illumination microscope of the embodiment can effectively improve the contrast of the sample to be imaged.
  • the method before the step S23, the method further includes the following steps not shown in FIG. 2:
  • the original image under each structured light illumination pattern of the sample to be imaged is subjected to noise reduction processing to obtain a sequence of the original image after noise reduction.
  • Denoising the original image can effectively improve the signal-to-noise ratio of the original image.
  • step S23 specifically includes:
  • the method further includes:
  • the imaging method of the structured light illumination microscope of the embodiment effectively reduces the signal-to-noise ratio of the original image or the super-resolution image by denoising the original image or the reconstructed super-resolution image.
  • the method before the step S23, the method further includes the following steps not shown in FIG. 2:
  • An image enlargement process is performed on each original image in the original image sequence to obtain each original image in which the image is enlarged.
  • the method further includes:
  • the image enlargement in this embodiment refers to expanding the number of pixels used to represent the entire image by using an interpolation method or an image spectrum zero-padding extension, for example, interpolating an image of N ⁇ N pixels to obtain (2N-1) ⁇ (2N). -1) An image of a pixel.
  • step S13 specifically includes:
  • An original image magnified by each of the original image sequences and an original image magnified by N-1 images subsequent thereto are subjected to image reconstruction.
  • the imaging method of the structured light illumination microscope of the present embodiment can make the actual size represented by the adjacent pixels smaller by the image enlargement processing, so that the fine reconstruction result can be expressed.
  • the method further includes the step S20 not shown in FIG. 2:
  • the structured light illumination pattern is obtained by modulating incident light by a light modulating device disposed in an illumination light path of the structured light illumination microscope, and then obtaining a light modulation in an illumination light path of the structured light illumination microscope;
  • the device modulates the incident light and performs high-pass spatial filtering and projection imaging.
  • any one of the spatial light modulator SLM, the digital micromirror device DMD, and the grating may be used to modulate the spatial distribution of the incident light, and the image is obtained by direct projection imaging or high-pass spatial filtering processing and then imaging.
  • a structured light illumination pattern is provided.
  • the structured light illumination pattern can be obtained in various ways, and the universality of the imaging method is improved.
  • the structured light illumination pattern has a periodic distribution of light intensity in a plane in which the sample to be imaged is located.
  • Each structured light illumination pattern may contain one or more non-zero spatial frequencies, wherein the structured light illumination pattern of a single non-zero spatial frequency is typically a series of parallel sinusoids, as shown in FIG.
  • a plurality of non-zero spatial frequency structured light illumination patterns are typically the result of superposition or multiplication of sinusoidal stripes in different directions, as shown in FIG.
  • the structured light illumination microscope shown in FIG. 1 is taken as an example.
  • the fiber coupled output laser, the collimator and the beam expander constitute the light source module 1 to obtain a collimated wide beam, and Use it as incident light.
  • the incident light is incident on the surface of the light modulating device 2 according to actual requirements, and is modulated by the light modulating device 2 to obtain a structured light illumination pattern.
  • the outgoing light transmitted through the light modulating device 2 is reflected by the lens 3 at the dichroic beam splitter 8, and the sample 10 to be imaged is illuminated by the objective lens 9.
  • a DMD is employed as a modulation device for incident light.
  • the image focal plane of the lens 3 coincides with the back focal plane of the objective lens 9, and the lens 3
  • the focal plane of the object coincides with the plane of the DMD.
  • the image to be imaged 10 is placed on the front focal plane of the objective lens, and the plane of the DMD and the sample 10 to be imaged is conjugated.
  • the image loaded on the DMD can be imaged by the projection image to be imaged. On sample 10, illumination of the structured light is achieved.
  • the imaging optical path of the sample 10 to be imaged is an existing microscope optical path, which is not described herein.
  • EMCCD is used as the detector 13.
  • the computer is used to simultaneously control the switching of the DMD loaded image and the detection of the EMCCD, and implement subsequent super-resolution image reconstruction.
  • the shooting integration time and the shooting interval time of the EMCCD are selected according to information such as the specific illumination light intensity and the moving speed of the sample to be imaged.
  • each time an image is captured the DMD-loaded image is translated in the x and y directions to change the phase of the structured light, and each structured light illumination pattern is in x and
  • the phase ( ⁇ x, ⁇ y) in the y direction is (0, 0), ( ⁇ /5, ⁇ /5), (2 ⁇ /5, 2 ⁇ /5), (3 ⁇ /5, 3 ⁇ /5), (4 ⁇ /). 5,4 ⁇ /5).
  • the main time spent on imaging is the time interval between the integration time taken by the EMCCD and the original image. Set the time for integrating each original image to 20ms. After the end of the integration, the interval is 10ms. Switch the DMD loading pattern (structured light illumination pattern) and start collecting the next original image. It can be seen from the calculation that with the existing imaging method, a SIM image can be obtained every 150 ms, that is, every five original images are acquired, and the motion of the sample within the 150 ms is not known.
  • image reconstruction can be performed every 30 ms, that is, every time an original image is acquired, and a SIM super-resolution image is obtained, and motion information of the sample to be tested in a shorter time can be obtained.
  • the time interval for obtaining the SIM image is significantly shortened compared with the existing imaging method, and the time resolution of the SIM imaging is remarkably improved.
  • FIG. 6 is a schematic structural diagram of an image forming apparatus of a structured light illumination microscope according to an embodiment of the present invention.
  • the imaging apparatus of the structured light illumination microscope of the present embodiment includes a pattern switching unit 601, an original image acquisition unit 602, and an image reconstruction unit 603.
  • the pattern switching unit 601 is configured to cyclically switch the preset N pieces of structured light illumination patterns according to a preset sequence, where N is a preset constant;
  • the original image obtaining unit 602 is configured to obtain an original image of the sample to be imaged under each structured light illumination pattern, to obtain an original image sequence of the sample to be imaged;
  • the image reconstruction unit 603 is configured to perform image reconstruction on each of the original image sequences and the N-1 original images after the original image sequence to obtain a super-resolution image sequence of the sample to be imaged.
  • the imaging device of the structured light illumination microscope of the present embodiment can effectively improve the temporal resolution of the super-resolution image of the structured light illumination microscope.
  • the invention provides an imaging method for a structured light illumination microscope, comprising: cyclically switching a preset N-structure light illumination pattern according to a preset sequence, N is a preset constant; and acquiring a sample to be imaged under each structured light illumination pattern Raw image, obtaining an original image sequence of the sample to be imaged; reconstructing each original image in the original image sequence and N-1 original images after the image to obtain a super-resolution image of the sample to be imaged sequence.
  • the super-resolution image sequence of the sample to be imaged changes with time, and the time resolution is greatly improved.
  • nonlinear structured light illumination microscope NL-SIM nonlinear structured light illumination microscope NL-SIM, total internal reflection fluorescence structure light illumination microscope TIRF-SIM, three-dimensional super-resolution imaging structured light illumination microscope 3D - Lattice-lightsheet-SIM combined with SIM, light microscopy and structured light illumination microscope.
  • TIRF-SIM total internal reflection fluorescence structure light illumination microscope
  • 3D three-dimensional super-resolution imaging structured light illumination microscope
  • 3D - Lattice-lightsheet-SIM combined with SIM
  • light microscopy and structured light illumination microscope light microscopy and structured light illumination microscope.
  • the super-resolution image sequence of the sample to be imaged can be accurately obtained, and the process of changing the sample to be tested can be obtained, and the time resolution of the structured light illumination microscope SIM can be improved.
  • An imaging device for a structured light illumination microscope comprising: a pattern switching unit, configured to cyclically switch a preset N-structure light illumination pattern according to a preset sequence, where N is a preset constant; and an original image acquisition unit configured to acquire a sample to be imaged An original image sequence of the sample to be imaged is obtained from an original image under each structured light illumination pattern; an image reconstruction unit for using each of the original image sequences and N-1 originals The image is subjected to image reconstruction to obtain a super-resolution image sequence of the sample to be imaged.
  • a super-resolution image sequence of the sample to be imaged as a function of time is obtained, and the time interval of each of the two super-resolution images is equal to the time interval of capturing each of the two original images, compared with the prior art structured light illumination microscope imaging method.
  • the temporal resolution of the imaging method and apparatus of the present invention is greatly improved.

Landscapes

  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Microscoopes, Condenser (AREA)

Abstract

一种结构光照明显微镜的成像方法及装置。成像方法包括：按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数（S21）；获取待成像样品在每张结构光照明图案下的原始图像，得到待成像样品的原始图像序列（S22）；将原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到待成像样品的超分辨图像（S23）。这种方法及装置可得到待成像样品随时间变化的超分辨图像序列，且每两张超分辨图像的时间间隔与拍摄每两张原始图像的时间间隔相等，提高了时间分辨率。

Description

一种结构光照明显微镜的成像方法及装置
本申请要求2016年3月10日提交、申请号为201610136330.2的中国专利申请的优先权，其所公开的内容作为参考全文并入本申请。
技术领域
本发明涉及光学显微镜技术领域，尤其涉及一种结构光照明显微镜的成像方法及装置。
背景技术
现代的生命科学研究中，显微镜是必不可少的研究工具。然而由于光的衍射，传统的光学显微镜存在分辨率的极限，这个分辨率的极限可以由瑞利判据(Rayleigh criterion)给出：R＝0.61λ/NA，其中λ是光的波长，NA是显微物镜的数值孔径。近年来，出现了各种用于提高光学显微镜分辨率的方法，结构光照明显微镜(Structured Illumination Microscope，SIM)就是其中之一。与其他的方法相比，如：随机光学重构显微镜(Stochastic Optical Reconstruction Microscopy，STORM)，光激活定位显微镜(Photo Activated Localization Microscopy，PALM)，受激发射损耗显微镜(Stimulated Emission Depletion Microscopy，STED)，SIM的图像重构过程中所需的原始图像数量最少，合成每帧超分辨图像所需采集数据的时间最短，此外SIM的原始图像采集是宽场成像，成像速度受视场大小影响不大。故SIM是各种超分辨成像方法中最适合观察活细胞或宽视场中快速成像的方法。
现有的结构光照明显微镜的基本结构如图1所示，采用相干或非相干的准直宽光束作为光源1，入射光经过光线调制器件2，被调制后经过由透镜3、二向色分束器8及物镜9组成的成像系统，然后调制出的图案投影在照明样品10上，并在照明样品10所在的平面上形 成周期性的光强分布。在使用相干光源的情况下，也可以选择性的加入由透镜4、空间滤波器5及透镜6组成的空间滤波系统，以滤除零级衍射分量，滤波后的光继续通过由二向色分束器8及物镜9组成的成像系统，在照明样品10所在的平面处干涉生成结构光照明图案。被照明的样品的光通过由物镜9、镜筒透镜11组成的显微系统后被探测器13采集。
现有技术中，通过结构光照明显微镜随时间推移生成SIM序列(time-lapse SIM)，其成像过程如下：
在一组结构光照明图案下，每张结构光照明图案都拍摄一张原始图像，对这一组原始图像进行图像重构得到一张SIM超分辨图像。下一张SIM超分辨图像使用下一组结构光照明下的原始图像重构得到。重复上述过程，获得多个SIM超分辨图像，构成SIM超分辨图像的时间序列。
最早的SIM中，用于调制结构光分布的器件为光栅。不同结构光照明图案的调制与切换是通过平移或旋转光栅实现的。由于系统中存在机械运动的部分，其成像速度相对较慢。之后的SIM普遍采用空间光调制器(Spatial Light Modulator，SLM)以及数字微镜器件(Digital Micro-mirror Device，DMD)这两种光电器件调制结构光照明图案。由于SLM和DMD响应速度快，再加上高灵敏度的探测器如电子倍增CCD(EMCCD)、科学级CMOS(sCMOS)能够大幅的缩短曝光时间，这些都为SIM的高速成像提供了有利条件。
若要再进一步提升SIM的成像速度，需要通过增大照明光强缩短曝光时间，从而提高成像的时间分辨率。然而在荧光成像中，照明光强的增强会加速荧光分子的光漂白效应，缩短总体的观察时间，这在活细胞成像中是很不利的，需要折中考虑。故在现有光电器件响应速度和探测器灵敏度的条件下，SIM的时间分辨率很难有大幅的提升。
发明内容
(一)要解决的技术问题
本发明的目的是提供一种结构光照明显微镜的成像方法及装置，以解决现有技术中结构光照明显微镜成像的时间分辨率难以进一步提升的技术问题。
(二)技术方案
为了解决上述技术问题，第一方面，本发明提供一种结构光照明显微镜的成像方法，包括：
按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；
获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；
将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
可选地，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列，具体包括：
计算所述原始图像序列中每N张原始图像中混叠的各空间频谱分量，得到多个空间频谱分量组，所述N张原始图像由所述原始图像序列中的每张原始图像与其之后的N-1张原始图像组成；
根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，得到空间频率还原的各空间频谱分量；
将每个空间频谱分量组中空间频率还原的各空间频谱分量与空间频率未改变的各空间频谱分量进行加权叠加，得到所述待成像样品的超分辨图像序列。
可选地，所述根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频 率还原，得到空间频率还原的各空间频谱分量，具体包括：
根据所述预设的N张结构光照明图案的空间频率，确定每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率改变量；
根据所述空间频率改变量，确定各空间频谱分量组中每个空间频率改变量在空间域内对应的指数函数；
在空间域内，将每个空间频谱分量组中空间频率改变的各空间频谱分量与自身的空间频率改变量对应的指数函数相乘，得到空间频率还原的各空间频谱分量。
可选地，所述拍摄待成像样品在每张结构光照明图案下的原始图像，具体包括：
间隔预设时间拍摄待成像样品在每张结构光照明图案下的原始图像。
可选地，在计算所述原始图像序列中N张原始图像中混叠的各空间频谱分量，得到多个空间频谱分量组之后，所述方法还包括：
对每个空间频谱分量组中的各空间频谱分量进行反卷积，得到反卷积后的各空间频谱分量组；
相应地，所述根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，具体包括：
根据所述预设的N张结构光照明图案的空间频率，将每个反卷积后的空间频谱分量组中空间频率改变的各空间频率谱分量的空间频率还原。
可选地，在所述周期性循环预设的N张结构光照明图案之前，所述方法还包括：
采用光线调制器件调制所述结构光照明显微镜的入射光；
对调制后的入射光进行投影，得到所述预设的N张结构光照明图案。
可选地，在所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构之前，所述方法还包括：
对所述待成像样品在每张结构光照明图案下的原始图像进行降噪处理，得到降噪后的原始图像序列；
相应地，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，具体包括：
将所述降噪后的原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构；或
在所述得到所述待成像样品的超分辨图像序列之后，所述方法还包括：
对所述超分辨图像序列中的每张超分辨图像进行降噪，得到降噪后的超分辨图像序列。
可选地，在所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构之前，所述方法还包括：
对所述原始图像序列中的每张原始图像进行图像放大处理，比如：这一处理一般可以通过图像内插的方法或者通过对图像频谱进行补零再变换回空域的方式实现，得到图像放大的各原始图像；
相应地，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，具体包括：
将所述原始图像序列中的每张图像放大的原始图像与其之后的N-1张图像放大的原始图像进行图像重构；
或，在所述得到所述待成像样品的超分辨图像序列之后，所述方法还包括：对所述超分辨图像序列中的每张超分辨图像进行图像放大处理，得到图像放大后的超分辨图像序列。
可选地，按照预设顺序循环切换预设的N张结构光照明图案之前，所述方法还包括：
获取结构光照明显微镜成像的所述预设的N张结构光照明图案；
所述结构光照明图案由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后投影成像得到；或由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后进行高通空间滤波并投影成像得到。
可选地，所述结构光照明图案包含一个或多个非零空间频率。
第二方面，本发明提供一种结构光照明显微镜的成像装置，其特征在于，包括：
图案切换单元，用于按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；
原始图像获取单元，用于获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；
图像重构单元，用于将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
(三)有益效果
本发明提供的结构光照明显微镜的成像方法及装置，具有以下优点：
可得到待成像样品随时间变化的超分辨图像序列，且每两张超分辨图像的时间间隔与拍摄每两张原始图像的时间间隔相等，与现有技术的结构光照明显微镜成像方法相比，本发明的成像方法及装置的时间分辨率得到了极大的提高。
附图说明
图1为现有的结构光照明显微镜的结构示意图；
图2为本发明一实施例提供的结构光照明显微镜的成像方法的流程示意图；
图3为本发明一实施例提供的结构光照明显微镜的成像方法与现有成像方法的对比示意图；
图4为本发明一实施例提供的一个非零空间频率的结构光照明 图案光强分布的示意图；
图5为本发明一实施例提供的多个非零空间频率的结构光照明图案光强分布的示意图；
图6为本发明一实施例提供的结构光照明显微镜的成像装置的结构示意图。
具体实施方式
下面结合附图和实施例，对本发明的具体实施方式作进一步详细描述。以下实例用于说明本发明，但不用来限制本发明的范围。
图2示出了本发明一实施例提供的结构光照明显微镜的成像方法的流程示意图。如图2所示，本实施例的结构光照明显微镜的成像方法包括步骤S21至S23。
S21、按照预设顺序循环切换预设的N张结构光照明图案。
其中，N为预设常数。
将预设的N张结构光照明图案设为一组，按固定的顺序循环切换这一组结构光照明图案。
举例来说，假如N＝5，则一组结构光照明图案中包括5张结构光照明图案，分别为a b c d e。结构光照明图案的循环切换就在上述a b c d e中进行，第一组的5张结构光照明图案的顺序可任意设定，但要求每个结构光照明图案都包括在内，即第一组的结构光照明图案的顺序可以为a b c d e、d a c e b、e c a b d等。如果第一组的结构光照明图案的顺序为e d c b a，那么接下来的结构光照明图案的顺序都与第一组的结构光照明图案的顺序相同，即在用户观察过程中，结构光照明图案的切换顺序为e d c b a e d c b a e d c b a……
S22、获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列。
每张结构光照明图案照明待成像样品时，都拍摄一张待成像样品的原始图像，在用户需要的观察时间内，得到待成像样品的原始图像 序列。
举例来说，结构光照明图案的顺序为e d c b a e d c b a e d c b a……那么得到的待成像样品的原始图像序列应为E D C B A E D C B A E D C B A……
上述待成像样品的原始图像，按照拍摄的时间顺序组成原始图像序列。
S23、将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
拍摄得到的待成像样品的原始图像序列中的第一幅原始图像与其之后的N-1幅原始图像，经过图像重构得到一张SIM超分辨图像。之后的每一幅原始图像，都与其之后的N-1幅原始图像经过图像重构得到一张SIM超分辨图像。所有的SIM超分辨图像按时间排列组成超分辨图像序列。
举例来说，当N＝5时，待成像样品的第一张原始图像E与之后的4张原始图像D C B A重构得到第一张SIM超分辨图像。然后，第二张原始图像D与其之后的4张原始图像组成一组，进行重构得到第二张SIM超分辨图像。即，原始图像E D C B A重构得到第一张SIM超分辨图像S(1)，原始图像D C B A E重构得到第二张SIM超分辨图像S(2)，原始图像C B A E D重构得到第三张SIM超分辨图像S(3)……重构得到的SIM超分辨图像按顺序排列成超分辨时间序列S(1) S(2) S(3) S(4)……
可以理解的是，对原始图像的重构过程可以是在拍摄完待成像样品的原始图像序列之后进行，也可与拍摄待成像样品的原始图像同时进行。
可以理解的是，本实施例的结构光照明显微镜的成像方法，不仅能用于结构光照明显微镜SIM中，也能用于以下技术：非线性结构光照明显微镜NL-SIM、全内反射荧光结构光照明显微镜TIRF-SIM、 三维超分辨成像的结构光照明显微镜3D-SIM、光片显微镜与结构光照明显微镜的结合Lattice-lightsheet-SIM。此外，在其他SIM衍生出的显微镜技术中都适用。
本实施例的结构光照明显微镜的成像方法，可得到待成像样品随时间变化的超分辨图像序列，时间分辨率得到了极大的提高。
在本发明一个优选的实施例中，步骤S23具体包括图2中未示出的子步骤S231至S233。
S231、计算所述原始图像序列中每N张原始图像中混叠的各空间频谱分量，得到多个空间频谱分量组。
其中，所述N张原始图像由所述原始图像序列中的每张原始图像与其之后的N-1张原始图像组成。
每N张原始图像可计算得到一组空间频谱分量，如上一实施例所述，原始图像序列中可得到多个N张原始图像，因此，可得到多个空间频谱分量组。
每个空间频谱分量组中包含一些空间频率发生改变的空间频谱分量，也包含一些频率未改变的空间频谱分量。
S232、根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，得到空间频率还原的各空间频谱分量。
根据预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原到其本来的空间频率处，以正确的再现待成像样品。
S233、将每个空间频谱分量组中空间频率还原的各空间频谱分量与空间频率未改变的各空间频谱分量进行加权叠加，得到所述待成像样品的超分辨图像序列。
本实施例的结构光照明显微镜的成像方法，可以准确的得到待成像样品的超分辨图像序列，获取更精细的待测样品的变化过程，提高 了结构光照明显微镜SIM的时间分辨率。
在本发明一个优选的实施例中，步骤S232具体包括图2中未示出的子步骤S2321至S2323：
S2321、根据所述预设的N张结构光照明图案的空间频率，确定每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率改变量。
在将空间频率改变的空间频谱分量的空间频率还原之前，要先确定空间频谱分量的空间频率改变量。
根据预设的N张结构光照明图案的空间频率，确定各空间频率改变量。
S2322、根据所述空间频率改变量，确定各空间频谱分量组中每个空间频率改变量在空间域内对应的指数函数。
不同的空间频率改变量对应不同的指数函数，根据空间频率改变量确定每个空间频谱分量对应的指数函数，才能正确的还原各空间频率改变的空间频谱分量。
S2323、在空间域内，将每个空间频谱分量组中空间频率改变的各空间频谱分量与自身的空间频率改变量对应的指数函数相乘，得到空间频率还原的各空间频谱分量。
将每个空间频谱分量组中空间频率改变的各空间频谱分量与自身的空间频率改变量对应的指数函数相乘，能够实现频率改变的空间频谱分量的空间频率的还原。
本实施例的结构光照明显微镜的成像方法，通过对各空间频谱分量的空间频率的还原，能够准确地得到各原始图像对应的超分辨图像。
在本发明一个优选的实施例中，步骤S22具体包括图2中未示出的以下步骤：
间隔预设时间拍摄待成像样品在每张结构光照明图案下的原始 图像。
为尽可能保证重构结果在时间轴上是均匀的采样过程，重构每张超分辨图像所用的一组原始图像中，连续拍摄的两张原始图像的时间间隔，应基本一致。
可以理解的是上述时间间隔，为两张原始图像积分开始的时刻的间隔。
连续拍摄的两张原始图像的时间间隔相等时，可最大程度的接近待成像样品的真实情况，提高重构得到的超分辨图像的精确度。
如果连续拍摄的两张原始图像的时间间隔平均值为t，则其时间间隔最小要大于等于0.5t，其时间间隔最大要小于等于2t。
与现有的结构光照明显微镜的成像方法相比，本实施例的具体成像过程如图3所示。
拍摄得到的待成像样品的原始图像序列包括：原始图像1、原始图像2……原始图像n。
现有的结构光照明显微镜的成像方法，采用1-5张原始图像作为一组，6-10张原始图像作为另一组，每一组原始图像进行重构得到一张SIM超分辨图像。连续重构得到的每两张SIM超分辨图像之间的时间间隔是5t。
而本实施例的结构光照明显微镜的成像方法，采用1-5张原始图像作为一组，2-6张原始图像作为另一组，3-7张原始图像作为下一组，每一组原始图像进行重构得到一张SIM超分辨图像。连续重构得到的每两张SIM超分辨图像之间的时间间隔是t。与拍摄原始图像的时间间隔相等。
本实施例的结构光照明显微镜的成像方法，可有效缩短结构光照明显微镜的超分辨成像时间间隔，有效的提高了结构光照明显微镜成像的时间分辨率。
在本发明一个优选的实施例中，在步骤S231之后，上述方法还 包括图2中未示出的以下步骤：
对每个空间频谱分量组中的各空间频谱分量进行反卷积，得到反卷积后的各空间频谱分量组。
为提升带成像样品的原始图像的对比度，可以在对原始图像进行图像重构之前、或是在图像重构的过程中加入反卷积的处理，补偿由成像过程中空间频率域内的高频信息衰减造成的原始图像对比度的下降。
SIM成像方法中常用的反卷积处理是维纳滤波：OTF*F(f)/(|OTF|2+c2)。其中光学传递函数OTF可以通过理论计算得到，可以通过实际测量得到，也可以使用相关反卷积软件迭代计算获得。其中c为一个经验常数，根据实际滤波效果可以调整。
迭代反卷积、盲解卷积等反卷积算法都可以应用于SIM成像方法。
相应地，步骤S232具体包括：
根据所述预设的N张结构光照明图案的空间频率，将每个反卷积后的空间频谱分量组中空间频率改变的各空间频率谱分量的空间频率还原。
本实施例的结构光照明显微镜的成像方法，可有效提高待成像样品的对比度。
在本发明一个优选的实施例中，在步骤S23之前，上述方法还包括图2中未示出的以下步骤：
对所述待成像样品在每张结构光照明图案下的原始图像进行降噪处理，得到降噪后的原始图像序列。
对原始图像进行降噪处理，可有效的提高原始图像的信噪比。
相应地，步骤S23具体包括：
将所述降噪后的原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构。
或在步骤S23之后，上述方法还包括：
对所述超分辨图像序列中的每张超分辨图像进行降噪，得到降噪后的超分辨图像序列。
本实施例的结构光照明显微镜的成像方法，通过对原始图像或重构得到的超分辨图像进行降噪，从而有效提高原始图像或超分辨图像的信噪比。
在本发明一个优选的实施例中，在步骤S23之前，上述方法还包括图2中未示出的以下步骤：
对所述原始图像序列中的每张原始图像进行图像放大处理，得到图像放大的各原始图像。
有的时候图像的像素代表的实际尺寸过大，不足以表示超分辨重构之后的结果，因此采用图像放大处理。
或在步骤S23之后，上述方法还包括：
对所述超分辨图像序列中的每张超分辨图像进行图像放大处理，得到图像放大处理后的超分辨图像序列。
本实施例的图像放大，是指采用插值法或图像频谱补零扩展等其他算法，扩展表示整幅图像所用的像素数，例如将N×N像素的图像插值得到(2N-1)×(2N-1)像素的图像。
相应地，步骤S13具体包括：
将所述原始图像序列中的每张图像放大的原始图像与其之后的N-1张图像放大的原始图像进行图像重构。
本实施例的结构光照明显微镜的成像方法，经过图像放大的处理，可以使得相邻像素所表示的实际尺寸更小，这样能够表现精细的重构结果。
在本发明一个优选的实施例中，在步骤S21之前，上述方法还包括图2中未示出的步骤S20：
S20、获取结构光照明显微镜成像的所述预设的N张结构光照明图案。
其中，所述结构光照明图案由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后投影成像得到；或由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后进行高通空间滤波并投影成像得到。
本实施例中，可采用空间光调制器SLM、数字微镜器件DMD和光栅中的任意一种器件，调制入射光的空间分布，通过直接投影成像或高通空间滤波处理之后再成像的方式得到预设的结构光照明图案。
本实施例的结构光照明显微镜的成像方法，可通过多种方式获得结构光照明图案，提高了此成像方法的普适性。
在本发明一个优选的实施例中，所述结构光照明图案在所述待成像样品所在的平面内，光强成周期性分布。
每张结构光照明图案可以含有一个或多个非零空间频率，其中单一非零空间频率的结构光照明图案通常为一系列平行的正弦曲线，具体如图4所示。多个非零空间频率的结构光照明图案通常为不同方向正弦条纹叠加或相乘的结果，具体如图5所示。
可以理解的是，在待成像样品的实际加载的结构光照明图案，其空间频谱中，每个空间频率处是有一定宽度的窄峰。
在本发明一个优选的实施例中，采用图1所示的结构光照明显微镜为例来说，光纤耦合输出激光器、准直器与扩束器组成光源模块1，得到准直的宽光束，并将其作为入射光。入射光按实际需求入射到光线调制器件2的表面，经光线调制器件2调制后得到结构光照明图案。透过光线调制器件2的出射光经过透镜3在二向色分束镜处8发生反射，并通过物镜9照明待成像样品10。
本实施例中，采用DMD作为入射光的调制器件。
本实施例中，透镜3的像方焦面与物镜9的后焦面重合，透镜3 的物方焦面与DMD所在的平面重合，待成像样品10置于物镜前焦面上，DMD与待成像样品10所在的平面是共轭的，DMD上加载的图像可以经过投影成像在待成像样品10上，实现结构光的照明。
本实施例中，待成像样品10的成像光路为现有的显微镜光路，在此不赘述。
本实施例中，为保证成像速度，采用EMCCD作为探测器13。使用计算机同时控制DMD加载图像的切换以及EMCCD的探测，并实现后续的超分辨图像重构。
在本实施例的实施过程中，根据具体照明光强以及待成像样品的运动速度等信息选取EMCCD的拍摄积分时间以及拍摄间隔时间。
使用五张图4或图5所示的二维结构光照明图案，每拍摄一张图像，在x和y方向平移DMD加载的图像，以改变结构光的相位，各结构光照明图案在x和y方向的相位(φx，φy)分别为(0，0)，(π/5，π/5)，(2π/5，2π/5)，(3π/5，3π/5)，(4π/5，4π/5)。
由于DMD切换时间在毫秒以下，其所消耗的时间可以忽略。成像的主要耗时为EMCCD拍摄的积分时间和原始图像之间拍摄的时间间隔。设每张原始图像积分的时间为20ms，积分结束以后，间隔10ms，切换DMD加载图案(结构光照明图案)，开始采集下一张原始图像。经计算可知，采用现有的成像方法，每隔150ms，即每采集五张原始图像，才可获得一张SIM图像，而样品在这150ms内的运动过程是无法获知的。
采用本实施例的成像方法，每隔30ms，即每采集一张原始图像都可以进行一次图像重构，获得一张SIM超分辨图像，可以获取更短时间内待测样品的运动信息。
因此，本实施例的结构光照明显微镜的成像方法，与现有的成像方法相比，获得SIM图像的时间间隔明显缩短，显著提升了SIM成像的时间分辨率。
图6示出了本发明一实施例提供的结构光照明显微镜的成像装置的结构示意图。如图6所示，本实施例的结构光照明显微镜的成像装置包括：图案切换单元601、原始图像获取单元602和图像重构单元603。
图案切换单元601，用于按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；
原始图像获取单元602，用于获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；
图像重构单元603，用于将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
本实施例的结构光照明显微镜的成像装置，能够有效提高结构光照明显微镜的超分辨图像的时间分辨率。
工业实用性
本发明提供了结构光照明显微镜的成像方法，其包括：按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。可得到待成像样品随时间变化的超分辨图像序列，时间分辨率得到了极大的提高。不仅能用于结构光照明显微镜SIM中，也能用于以下技术：非线性结构光照明显微镜NL-SIM、全内反射荧光结构光照明显微镜TIRF-SIM、三维超分辨成像的结构光照明显微镜3D-SIM、光片显微镜与结构光照明显微镜的结合Lattice-lightsheet-SIM。此外，在其他SIM衍生出的显微镜技术中都适用。可以准确的得到待成像样品的超分辨图像序列，获取更精细的待测样品的变化过程，提高了结构光照明显微镜SIM的时间分辨率。 结构光照明显微镜的成像装置，其包括：图案切换单元，用于按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；原始图像获取单元，用于获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；图像重构单元，用于将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。可得到待成像样品随时间变化的超分辨图像序列，且每两张超分辨图像的时间间隔与拍摄每两张原始图像的时间间隔相等，与现有技术的结构光照明显微镜成像方法相比，本发明的成像方法及装置的时间分辨率得到了极大的提高。

Claims (10)

  1. 一种结构光照明显微镜的成像方法，其特征在于，包括：
    按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；
    获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；
    将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
  2. 根据权利要求1所述的方法，其特征在于，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列，具体包括：
    计算所述原始图像序列中每N张原始图像中混叠的各空间频谱分量，得到多个空间频谱分量组，所述N张原始图像由所述原始图像序列中的每张原始图像与其之后的N-1张原始图像组成；
    根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，得到空间频率还原的各空间频谱分量；
    将每个空间频谱分量组中空间频率还原的各空间频谱分量与空间频率未改变的各空间频谱分量进行加权叠加，得到所述待成像样品的超分辨图像序列。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，得到空间频率还原的各空间频谱分量，具体包括：
    根据所述预设的N张结构光照明图案的空间频率，确定每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率改变量；
    根据所述空间频率改变量，确定各空间频谱分量组中每个空间频 率改变量在空间域内对应的指数函数；
    在空间域内，将每个空间频谱分量组中空间频率改变的各空间频谱分量与自身的空间频率改变量对应的指数函数相乘，得到空间频率还原的各空间频谱分量。
  4. 根据权利要求1所述的方法，其特征在于，所述拍摄待成像样品在每张结构光照明图案下的原始图像，具体包括：
    间隔预设时间拍摄待成像样品在每张结构光照明图案下的原始图像。
  5. 根据权利要求2所述的方法，其特征在于，在计算所述原始图像序列中N张原始图像中混叠的各空间频谱分量，得到多个空间频谱分量组之后，所述方法还包括：
    对每个空间频谱分量组中的各空间频谱分量进行反卷积，得到反卷积后的各空间频谱分量组；
    相应地，所述根据所述预设的N张结构光照明图案的空间频率，将每个空间频谱分量组中空间频率改变的各空间频谱分量的空间频率还原，具体包括：
    根据所述预设的N张结构光照明图案的空间频率，将每个反卷积后的空间频谱分量组中空间频率改变的各空间频率谱分量的空间频率还原。
  6. 根据权利要求1所述的方法，其特征在于，在所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构之前，所述方法还包括：
    对所述待成像样品在每张结构光照明图案下的原始图像进行降噪处理，得到降噪后的原始图像序列；
    相应地，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，具体包括：
    将所述降噪后的原始图像序列中的每张原始图像与其之后的N-1 张原始图像进行图像重构；或
    在所述得到所述待成像样品的超分辨图像序列之后，所述方法还包括：
    对所述超分辨图像序列中的每张超分辨图像进行降噪，得到降噪后的超分辨图像序列。
  7. 根据权利要求1所述的方法，其特征在于，在所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构之前，所述方法还包括：
    对所述原始图像序列中的每张原始图像进行图像放大处理，得到图像放大的各原始图像；
    相应地，所述将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，具体包括：
    将所述原始图像序列中的每张图像放大的原始图像与其之后的N-1张图像放大的原始图像进行图像重构；或，
    在所述得到所述待成像样品的超分辨图像序列之后，所述方法还包括：
    对所述超分辨图像序列中的每张超分辨图像进行图像放大处理，得到图像放大后的超分辨图像序列。
  8. 根据权利要求1所述的方法，其特征在于，按照预设顺序循环切换预设的N张结构光照明图案之前，所述方法还包括：
    获取结构光照明显微镜成像的所述预设的N张结构光照明图案；
    所述结构光照明图案由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后投影成像得到；或由所述结构光照明显微镜的照明光路中设置的光线调制器件对入射光进行调制后进行高通空间滤波并投影成像得到。
  9. 根据权利要求1至8任一项所述的方法，其特征在于，所述结构光照明图案包含一个或多个非零空间频率。
  10. 一种结构光照明显微镜的成像装置，其特征在于，包括：
    图案切换单元，用于按照预设顺序循环切换预设的N张结构光照明图案，N为预设常数；
    原始图像获取单元，用于获取待成像样品在每张结构光照明图案下的原始图像，得到所述待成像样品的原始图像序列；
    图像重构单元，用于将所述原始图像序列中的每张原始图像与其之后的N-1张原始图像进行图像重构，得到所述待成像样品的超分辨图像序列。
PCT/CN2016/112637 2016-03-10 2016-12-28 一种结构光照明显微镜的成像方法及装置 Ceased WO2017152702A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610136330.2 2016-03-10
CN201610136330.2A CN105589188B (zh) 2016-03-10 2016-03-10 一种结构光照明显微镜的成像方法及装置

Publications (1)

Publication Number Publication Date
WO2017152702A1 true WO2017152702A1 (zh) 2017-09-14

Family

ID=55928890

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/112637 Ceased WO2017152702A1 (zh) 2016-03-10 2016-12-28 一种结构光照明显微镜的成像方法及装置

Country Status (2)

Country Link
CN (1) CN105589188B (zh)
WO (1) WO2017152702A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109724951A (zh) * 2017-10-27 2019-05-07 黄晓淳 一种动态超分辨荧光成像技术
CN110223238A (zh) * 2019-04-30 2019-09-10 北京理工大学 编码光照成像重构方法及装置
CN110308125A (zh) * 2019-07-11 2019-10-08 清华大学 三维显微层析计算摄像方法及装置
CN111308682A (zh) * 2019-11-18 2020-06-19 天津大学 基于结构光照明的超分辨重构方法
CN112508791A (zh) * 2020-12-18 2021-03-16 中国工程物理研究院激光聚变研究中心 一种初始相位提取方法、装置、电子设备及存储介质
CN115541550A (zh) * 2022-10-21 2022-12-30 南京理工大学 基于主成分分析的结构光照明显微成像方法
CN116402678A (zh) * 2022-12-19 2023-07-07 中国科学院苏州生物医学工程技术研究所 超分辨结构光照明显微镜的频谱优化直接重建方法
CN118982461A (zh) * 2024-10-22 2024-11-19 中国科学院苏州生物医学工程技术研究所 一种空频域联合降噪的结构光照明超分辨图像重建方法

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105589188B (zh) * 2016-03-10 2018-01-16 清华大学 一种结构光照明显微镜的成像方法及装置
CN106770095A (zh) * 2016-11-30 2017-05-31 浙江大学 一种基于非线性光斑调制的超分辨显微成像方法和装置
CN108305213B (zh) * 2017-01-13 2020-12-25 中国科学技术大学 图像重建方法、系统及结构照明显微镜
CN106770147B (zh) * 2017-03-15 2019-07-19 北京大学 一种结构光照明超分辨显微成像方法
CN107180411B (zh) * 2017-05-19 2021-05-18 中国科学院苏州生物医学工程技术研究所 一种图像重构方法及系统
CN108038824B (zh) * 2017-12-11 2020-07-03 北京大学 一种基于结构光照明的荧光偶极子定向方法
CN109146790B (zh) * 2018-08-27 2022-10-11 深圳大学 一种图像重构方法、装置、电子设备和存储介质
EP3734345B1 (en) * 2019-04-30 2023-08-02 Leica Microsystems CMS GmbH Image processing device, system, and method for improving signal-to-noise of microscopy images
WO2021179127A1 (zh) * 2020-03-09 2021-09-16 深圳华大生命科学研究院 超分辨成像系统与方法、生物样品识别系统与方法、核酸测序成像系统与方法及核酸识别系统与方法
CN111458318B (zh) * 2020-05-12 2021-06-22 西安交通大学 利用正方晶格结构光照明的超分辨成像方法及系统
CN119784631B (zh) * 2024-12-23 2025-10-31 深圳大学 提高sted成像质量的数字图像处理方法、装置及相关组件

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080069467A1 (en) * 2006-09-20 2008-03-20 Carl Zeiss Imaging Solutions Gmbh A Corporation Of Germany Methods and devices for miages processing with higher harmonics of an illumination grating
CN101661159A (zh) * 2008-08-25 2010-03-03 麦克奥迪实业集团有限公司 一种基于二维调制技术的切层图像获取方法
CN102540446A (zh) * 2011-12-28 2012-07-04 中国科学院西安光学精密机械研究所 一种基于数字微镜器件的高速结构照明光学显微系统及方法
CN105589188A (zh) * 2016-03-10 2016-05-18 清华大学 一种结构光照明显微镜的成像方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080069467A1 (en) * 2006-09-20 2008-03-20 Carl Zeiss Imaging Solutions Gmbh A Corporation Of Germany Methods and devices for miages processing with higher harmonics of an illumination grating
CN101661159A (zh) * 2008-08-25 2010-03-03 麦克奥迪实业集团有限公司 一种基于二维调制技术的切层图像获取方法
CN102540446A (zh) * 2011-12-28 2012-07-04 中国科学院西安光学精密机械研究所 一种基于数字微镜器件的高速结构照明光学显微系统及方法
CN105589188A (zh) * 2016-03-10 2016-05-18 清华大学 一种结构光照明显微镜的成像方法及装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
WU, MEIRUI: "Study on Structured Illumination Fluorescence Microscopy Imaging System", UNIVERSITY OF CHINESE ACADEMY OF SCIENCES MASTER'S DISSERTATION, 15 May 2015 (2015-05-15) *
YANG, HUAIDONG ET AL.: "Algorithm for Resorting Spectrogram with Sub-pixel Resolution", SPECTROSCOPY AND SPECTRAL ANALYSIS, vol. 29, no. 12, 30 December 2009 (2009-12-30), pages 1000 - 0593 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109724951A (zh) * 2017-10-27 2019-05-07 黄晓淳 一种动态超分辨荧光成像技术
CN110223238A (zh) * 2019-04-30 2019-09-10 北京理工大学 编码光照成像重构方法及装置
CN110308125A (zh) * 2019-07-11 2019-10-08 清华大学 三维显微层析计算摄像方法及装置
CN111308682A (zh) * 2019-11-18 2020-06-19 天津大学 基于结构光照明的超分辨重构方法
CN112508791A (zh) * 2020-12-18 2021-03-16 中国工程物理研究院激光聚变研究中心 一种初始相位提取方法、装置、电子设备及存储介质
CN112508791B (zh) * 2020-12-18 2022-11-04 中国工程物理研究院激光聚变研究中心 一种初始相位提取方法、装置、电子设备及存储介质
CN115541550A (zh) * 2022-10-21 2022-12-30 南京理工大学 基于主成分分析的结构光照明显微成像方法
CN116402678A (zh) * 2022-12-19 2023-07-07 中国科学院苏州生物医学工程技术研究所 超分辨结构光照明显微镜的频谱优化直接重建方法
CN116402678B (zh) * 2022-12-19 2023-10-20 中国科学院苏州生物医学工程技术研究所 超分辨结构光照明显微镜的频谱优化直接重建方法
CN118982461A (zh) * 2024-10-22 2024-11-19 中国科学院苏州生物医学工程技术研究所 一种空频域联合降噪的结构光照明超分辨图像重建方法
CN118982461B (zh) * 2024-10-22 2024-12-31 中国科学院苏州生物医学工程技术研究所 一种空频域联合降噪的结构光照明超分辨图像重建方法

Also Published As

Publication number Publication date
CN105589188A (zh) 2016-05-18
CN105589188B (zh) 2018-01-16

Similar Documents

Publication Publication Date Title
CN105589188B (zh) 一种结构光照明显微镜的成像方法及装置
Chen et al. Superresolution structured illumination microscopy reconstruction algorithms: a review
JP5412394B2 (ja) 標本観察装置
JP5738765B2 (ja) 構造化照明を備えた顕微鏡法のための改良された方法および装置
US11022731B2 (en) Optical phase retrieval systems using color-multiplexed illumination
CN114113019B (zh) 一种基于多重信号分类算法的阵列扫描超分辨显微成像装置、方法、设备及存储介质
US20100135547A1 (en) Optical sectioning microscopy
WO2019024491A1 (zh) 基于环状可编程led照明的高效率定量相位显微成像方法
CN107389631A (zh) 高速多色多模态结构光照明超分辨显微成像系统及其方法
AU2016233588A1 (en) Correcting for aberrations in incoherent imaging system using fourier ptychographic techniques
CN111175259A (zh) 具有结构化照明的三维显微术的加速的方法和设备
JP2016534389A (ja) 開口走査フーリエタイコグラフィ撮像
JP6544789B2 (ja) パターン投影装置、パターン投影方法、及び、位相変調量設定方法
RU2747129C1 (ru) Способ и устройство для реконструкции изображения, а также устройство для формирования изображения в микроскопе
CN107014795B (zh) 一种基于均匀结构光照明的共聚焦并行显微成像系统
EP1466204A1 (en) Microscopy imaging apparatus and mehod for generating an image
JP2011107140A (ja) 物体表面の高さマップを求める方法及びその装置
CN117369106B (zh) 一种多点共聚焦图像扫描显微镜及成像方法
CN108121059A (zh) 一种基于结构光照明的sted并行显微成像系统
CN111123496B (zh) 基于希尔伯特变换的结构照明快速三维彩色显微成像方法
CN110954520B (zh) 一种扫描结构光显微成像方法及装置
Fixler et al. Pattern projection for subpixel resolved imaging in microscopy
EP3729161A1 (en) Method and apparatus for optical confocal imaging, using a programmable array microscope
CN115839935B (zh) 基于共振扫描稀疏结构光照明的层析显微成像装置及方法
CN115272504B (zh) 一种用于宽场照明共焦显微成像系统的信号提取和重建图像方法

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16893344

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 16893344

Country of ref document: EP

Kind code of ref document: A1