WO2005086890A2 - Systeme et procede pour l'analyse hyper-spectrale - Google Patents

Systeme et procede pour l'analyse hyper-spectrale Download PDF

Info

Publication number
WO2005086890A2
WO2005086890A2 PCT/US2005/007893 US2005007893W WO2005086890A2 WO 2005086890 A2 WO2005086890 A2 WO 2005086890A2 US 2005007893 W US2005007893 W US 2005007893W WO 2005086890 A2 WO2005086890 A2 WO 2005086890A2
Authority
WO
WIPO (PCT)
Prior art keywords
spectral
hyper
elements
patch
classifier
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2005/007893
Other languages
English (en)
Other versions
WO2005086890A3 (fr
Inventor
Frank Geshwind
Ronald R. Coifman
Andreas C. Coppi
Gustave L. Davis
Richard A. Deverse
William G. Fateleyy
Mauro Maggioni
Frederick J. Warner
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Plain Sight Systems Inc
Original Assignee
Plain Sight Systems Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US10/832,684 external-priority patent/US20050024640A1/en
Application filed by Plain Sight Systems Inc filed Critical Plain Sight Systems Inc
Publication of WO2005086890A2 publication Critical patent/WO2005086890A2/fr
Anticipated expiration legal-status Critical
Publication of WO2005086890A3 publication Critical patent/WO2005086890A3/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/0205—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows
    • G01J3/0208—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows using focussing or collimating elements, e.g. lenses or mirrors; performing aberration correction
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/0205—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows
    • G01J3/021—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows using plane or convex mirrors, parallel phase plates, or particular reflectors
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/0205—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows
    • G01J3/0218—Optical elements not provided otherwise, e.g. optical manifolds, diffusers, windows using optical fibers
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/02—Details
    • G01J3/027—Control of working procedures of a spectrometer; Failure detection; Bandwidth calculation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/28—Investigating the spectrum
    • G01J3/2823—Imaging spectrometer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/693—Acquisition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/695—Preprocessing, e.g. image segmentation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/28—Investigating the spectrum
    • G01J2003/2866—Markers; Calibrating of scan
    • G01J2003/2869—Background correcting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction

Definitions

  • the present invention relates generally to methods for hyper-spectral data processing and more particularly to a method and system for characterizing diverse elements within hyper-spectral images.
  • Imagers employ either a two-dimensional (2D) multichannel detector array or a single element detector.
  • Imagers using a 2D detector array measure the intensity distribution of all spatial resolution elements simultaneously during the entire period of data acquisition.
  • Imagers using a single detector require that the individual spatial resolution elements be measured consecutively via a raster scan so that each one is observed for a small fraction of the period of data acquisition.
  • Prior art imagers using a plurality of detectors at the image plane can exhibit serious signal-to-noise ratio problems.
  • Prior art imagers using a single element detector can exhibit more serious signal-to-hoise ratio problems.
  • Spectrometers are commonly used to analyze the chemical composition of samples by determimng the abso ⁇ tion or attenuation of certain wavelengths of electromagnetic radiation by the sample or samples. Because it is typically necessary to analyze the abso ⁇ tion characteristics of more than one wavelength of radiation to identify a compound, and because each wavelength must be separately detected to distinguish the wavelengths, prior art spectrometers utilize a plurality of detectors, have a moving grating, or use a set of filter elements. However, the use of a plurality of detectors or the use of a macro moving grating has signal-to-noise limitations.
  • variable band pass filter spectrometers typically employ a multitude of filters that require macro moving parts or other physical manipulation in order to switch between individual filter elements or sets of filter elements for each measurement.
  • Each filter element employed can be very expensive, difficult to manufacture and all are permanently set at the time of manufacture in the wavelengths (bands) of radiation that they pass or reject. Physical human handling of the filter elements can damage them and it is time consuming to change filter elements. There is, therefore, a need for variable band pass filter spectrometers, variable band reject filter spectrometers, variable multiple band pass filter spectrometers or variable multiple band reject filter spectrometers without a requirement for discrete (individual) filter elements that have permanently set band pass or band reject properties.
  • variable band pass filter spectrometers variable band reject filter spectrometers, variable multiple band pass filter spectrometers or variable multiple band reject filter spectrometers to be able to change the filters corresponding to the bands of radiation that are passed or rejected rapidly, without macro moving parts and without human interaction.
  • an object be irradiated with radiation having particularly shaped spectrum.
  • this approach does not work in a more general case, and therefore it is desirable to have a controllable radiation source capable of providing arbitrary spectrum shapes and intensities.
  • a controllable radiation source where virtually arbitrary spectrum shape and characteristics can be designed, and where disadvantages associated with the prior art are obviated. Further, it is desirable not only to shape the spectrum of the radiation source, but also encode its components differently, which feature can be used to readily perform several signal processing functions useful in a number of practical applications.
  • the phrase "a spectrum shape" in this disclosure refers not to a mathematical abstraction but rather to configurable spectrum shapes having range(s) and resolution necessarily limited by practical considerations.
  • a single sensor system can cost less to produce, but will take longer to fully measure an object under study.
  • prior art multi-sensor systems one often encounters a problem in which the different sensor elements have different response characteristics, and it is necessary to add components to the system to calibrate for this. It is desirable to have a system with which one gains the lower-cost, better signal-to-noise, and automatic inter-pixel calibration advantages of a single-sensor system while not suffering all of the time loss usually associated with using single sensors.
  • the spectral analysis of tissue sections has eyolved from two wavelength image subtraction techniques to Raman near infra-red micro-spectroscopic mapping permitting discrimination of cell types & tissue patterns.
  • spectral vectors in a given image patch will exhibit variability from a variety of sources. Some of these sources are biological in nature, such as the local density of cytoplasm; others are non-biological in nature and can include such things as non-uniformities in the light source used to collect the data, drifts in instrumental parameters during the time of data collection, orientation of cells in the tissue and the like. Hence, it is desirable to eliminate variabilities due to non-biological factors, and to characterize tissue elements by spectral variability which is due only to the intrinsic biology.
  • An object of the present invention is to provide a method and system for hyper-spectral analysis which overcomes the above-noted shortcomings.
  • An object of the present invention is to provide a method and system for hyper-spectral analysis as aforesaid, which characterizes or distinguishes diverse elements within hyper-spectral images.
  • An object of the present invention is to provide a method and system for hyper-spectral analysis of normal, abnormal and malignant micro-array tissue sections.
  • the hyper- spectral analysis method for characterizing or distinguishing diverse elements within hyper-spectral images comprises the steps of extracting a plurality of patches of pixels from within the hyper-spectral images as being patches around pixels of the elements to be characterized or distinguished; computing the statistics of selected spectral features for each patch of pixels, a first classifier from feature- wise standard deviation of the selected spectral features in each patch, a set of second classifiers from principal components of the spectral in each patch, and a classifier based on the output of the first classifier and at least one of the second classifiers; and characterizing or distinguishing the elements based on the output of at least one of the classifiers, preferably the combined classifier.
  • a computer readable medium comprises code for characterizing diverse elements within hyper- spectral images, the code comprises instructions for extracting a plurality of patches of pixels from within the hyper-spectral images as being patches around pixels of the elements to be characterized or distinguished; computing the statistics of selected spectral features for each patch of pixels, a first classifier from feature- wise standard deviation of the selected spectral features in each patch, a set of second classifiers from principal components of the spectral in each patch, and a classifier based on the output of the first classifier and at least one of the second classifiers; and characterizing or distinguishing the elements based on the output of at least one of the classifiers, preferably the combined classifier.
  • a hyper- spectral analysis system for characterizing or distinguishing diverse elements within hyper-spectral images, comprises an extracting module for extracting a plurality of patches of pixels from within the hyper-spectral images as being patches around pixels of the elements to be characterized or distinguished; a computing module for computing the statistics of spectra for each patch of pixels, a first classifier from frequency-wise standard deviation of the spectra in each patch, a set of second classifiers from principal components of the spectra in each patch, and a combined classifier based on the output of the first classifier and at least one of the second classifiers; and a characterization module for characterizing or distinguishing the elements based on the output of at least one of the classifiers.
  • the hyper-spectral analysis system and method characterizes sub-elements of a tissue image, collects hyper- spectral tissue signatures, and analyzes local variability of such hyper-spectral signatures to characterize the tissue elements.
  • Such spectral signatures generally possess both biological and non-biological variability, and hyper-spectral analysis system and method of the present invention characterizes and removes such non-biological variability.
  • the hyper- spectral analysis system and method analyzes the local variability of spectra in image patches, thereby enabling spectral and spatio-spectral characterization of local tissue elements.
  • the hyper-spectral system and method can be applied to the analysis of any biological tissues, including but not limited to prepared microscopic slides, in vivo dermatologic tissues, tissues accessed via endoscopy and the like.
  • the hyper- spectral analysis system and method characterizes sub-elements of hyper-spectral datasets, analyzes the local spectral variability of image patches, and the discriminates between variabilities due to different factors, such as biological and non-biological factors.
  • the hyper- spectral analysis system and method are widely applicable to hyper-spectral data analysis, including the analysis of biological tissue samples, such as the analysis of normal, abnormal and malignant micro-array tissue sections.
  • the hyper- spectral analysis system and method characterizes sub-elements of a tissue image in which each pixel of the image is represented by a spectral vector of responses to various wavelengths or combinations of wavelength.
  • the nature of the tissue in the vicinity of the pixel can be characterized by analyzing the variability of the spectral signatures in small image patches overlapping that vicinity.
  • the hyper- spectral analysis system and method removes the non-biological variability from the pixel spectra by considering the local principal components calculated from all of the spectra in a small image patch containing the pixel.
  • such components can measure large-scale effects due to normalization deficiencies in the data collection process. How many of the local top principal component vectors are related to normalization effects can be determined based on the optimization of a cross-validated measure of success for a given biologically-relevant task. Such tasks might include separation of nuclei from other tissue elements, such as cytoplasm, distinguishing normal from abnormal tissues, or quantifying the density of a material (e.g. keratin). Deleterious normalization effects can be removed by projecting the data onto only the most biologically-relevant principal vectors. It is appreciated that these vectors are distinct from the principal component vectors of the full tissue scene. These vectors carry only local information on the nature of the variability for a given patch.
  • the hyper- spectral analysis system and method characterizes the tissue elements according to the adaptive descriptions of their intrinsic biological variability.
  • the variability of spectra in a given image patch can be summarized by a variety of methods, such as a vector of variances for each spectral response over the entire patch.
  • each patch is associated with a particular vector and features of these vectors can be extracted which represent successful performance of some biologically-relevant task as described herein. That is, the local variability space can be transformed into a new coordinate system which has a much smaller dimensionality and optimized for solving some relevant biological problem.
  • the present invention can determine such coordinates from the variability vectors in either a linear or a non-linear manner.
  • the hyper- spectral analysis system and method utilizes the spectral data collected from a pathology slide using a tuned light source spectral imaging system.
  • the local variability analysis in accordance with an embodiment of the present invention characterizes nuclei and other tissue types, and distinguishes normal from abnormal tissue in a mixed (cancer and non- cancer) patient population. It is appreciated that the present invention can be applied in a similar manner to different spectral image data sets and to any locally quantifiable biological task.
  • the hyper-spectral analysis system and method of the present invention is not limited to biological tissues.
  • the present invention is applicable to any hyper-spectral image in which diverse elements exist.
  • the present invention can be easily applied to a hyper-spectral satellite image to distinguish man-made objects from natural terrain.
  • the devices and methods in this application in general are capable of operating in various ranges of electromagnetic radiation, including the ultraviolet, visible, infrared, and microwave spectrum portions.
  • signal processing be it acoustic, electric, magnetic, etc., that the devices and techniques disclosed herein for optical signal processing can be applied in a straightforward way to those other signals as well.
  • FIGs. IA and IB are schematic diagrams illustrating a spectrometer constructed in accordance with two embodiments of the invention.
  • FIG. 2 is a plan view of a micro-mirror array used in the present invention.
  • FIG. 3 is a schematic diagram of two micro-mirrors illustrating the modulations of the mirrors of the micro-mirror device of FIG. 2;
  • FIG. 4 is a graph illustrating an output signal of the spectrometer when used to analyze the composition of a sample; [0029] FIG.
  • FIG. 5 is a graph illustrating an output signal of the imager when used for imaging pu ⁇ oses;
  • FIG. 6 is a schematic diagram illustrating an imager constructed in accordance with a preferred embodiment of the invention;
  • FIG. 6A illustrates spatio- spectral distribution of a DMA, where individual elements can be modulated;
  • FIG. 7 is an illustration of the input to the DMA Filter Spectrometer and its use to pass or reject wavelength of radiation specific to constituents in a sample;
  • FIG. 8 illustrates the design of a band pass filter in accordance with the present invention (top portion) and the profile of the radiation passing through the filter (bottom portion); [0033] FIG.
  • FIG. 9 illustrates the design of multi-modal band-pass or band-reject filters with corresponding intensity plots, in accordance with the present invention
  • FIG. 10 illustrates the means for the intensity variation of a spectral filter built in accordance with this invention
  • FIGs. 11-14 illustrate alternative embodiments of a modulating spectrometer in accordance with this invention
  • FIGs. 11A and 1 IB show embodiments in which the DMA is replaced with concave mirrors
  • FIG. 12 illustrates an embodiment of a complete modulating spectrometer in which the DMA element is replaced by the concave mirrors of FIG. 11.
  • FIG. 13 illustrates a modulating lens spectrometer using lenses instead of DMA, and a "barber pole” arrangement of mirrors to implement variable modulation.
  • FIG. 14. illustrates a "barber pole” modulator arrangement;
  • FIGs. 15 and 16 illustrate an embodiment of this invention in which one or more light sources provide several modulated spectral bands using a fiber optic bundle;
  • FIG. 17 illustrates in diagram form an apparatus using controllable radiation source;
  • FIGs. 18A and 18B illustrate in a diagram form an optical synapse processing unit (OSPU) used as a processing element in accordance with the present invention;
  • FIG. 19 illustrates in a diagram form the design of a spectrograph using OSPU; [0040] FIG.
  • OSPU optical synapse processing unit
  • FIG. 20 illustrates in a diagram form an embodiment of a tunable light source
  • FIG. 21 illustrates in a diagram form an embodiment of the spectral imaging device, which is built using two OSPUs
  • FIGs. 22 and 23 illustrate different devices built using OSPUs
  • FIGs. 24-26 are flow charts of various scans used in accordance with the present invention. Specifically, FIG. 24 is a flow chart of a raster-scan used in one embodiment of the present invention; FIG. 25 is a flowchart of a Walsh-Hadamard scan used in accordance with another embodiment of the invention. FIG. 26 is a flowchart of a multi-scale scan, used in a different embodiment; FIG.
  • FIG. 26A illustrates a multi-scale tracking algorithm in a preferred embodiment of the present invention
  • FIG. 27 is a block diagram of a spectrometer with two detectors
  • FIG. 29 is a generalized block diagram of hyper-spectral processing in accordance with the invention
  • FIG. 30 illustrates the difference in two spectral components (red and green) of a data cube produced by imaging the same object in different spectral bands
  • FIG. 31 illustrates hyper-spectral imaging from airborne camera
  • FIG. 32 is an illustration of a hyper-spectral image of human skin
  • FIG. 34 shows an axial and a cross-sectional views of a fiber optic assembly
  • FIG. 35 shows a physical arrangement of the fiber optic cable, detector and the slit
  • FIG. 36 illustrates a fiber optic surface contact probe head abutting tissue to be examined
  • FIGs. 37A and 37 B illustrate a fiber optic c-Probe for pierced ears that can be used for medical monitoring applications in accordance with the present invention
  • FIGs. 37A and 37 B illustrate a fiber optic c-Probe for pierced ears that can be used for medical monitoring applications in accordance with the present invention
  • FIG. 38 A, 38B and 38C illustrate different configurations of a hyper- spectral adaptive wavelength advanced illuminating imaging spectrograph (HAWAIIS) in accordance with this invention
  • FIG. 39 illustrates a DMA search by splitting the scene
  • FIG. 40 illustrates wheat spectra data (training) and wavelet spectrum in an example of determining protein content in wheat
  • FIG. 41 illustrates the top 10 wavelet packets in local regression basis selected using 50 training samples in the example of FIG. 40
  • FIG 42 is a scatter plot of protein content (test data) vs. correlation with top wavelet packet
  • Fig 43 illustrates PLS regression of protein content of test data
  • FIG. 40 illustrates wheat spectra data (training) and wavelet spectrum in an example of determining protein content in wheat
  • FIG. 41 illustrates the top 10 wavelet packets in local regression basis selected using 50 training samples in the example of FIG. 40
  • FIG 42 is a scatter plot of protein content (test data) vs. correlation with top wavelet packet
  • Fig 43 illustrate
  • FIG. 44 illustrates the advantage of DNA-based Hadamard Spectroscopy used in accordance with the present invention over the regular raster scan;
  • FIGs. 45-49(A-D) illustrate hyperspectral processing in accordance with the present invention;
  • FIG. 50 is a flow chart describing the steps of the hyper-spectral analysis method in accordance with an embodiment of the present invention.
  • a spectrometer assembly 10 constructed in accordance with one embodiment of the invention is illustrated.
  • the device broadly includes a source 12 of electromagnetic radiation, a mirror and slit assembly 14, a wavelength dispersing device 16, a spatial light modulator 18, a detector 20, and an analyzing device 22.
  • the electromagnetic radiation source 12 is operable to project rays of radiation onto or through a sample 24 that is to be analyzed, such as a sample of body tissue or blood.
  • the radiation source can be any device that generates electromagnetic radiation in a known wavelength spectrum such as a globar, hot wire, or light bulb that produces radiation in the infrared spectrum.
  • a parabolic reflector 26 can be inte ⁇ osed between the source 12 and the sample 24.
  • the source of electromagnetic radiation is selected as to yield a continuous band of spectral energies, and is referred to as the source radiation. It should be apparent that the energies of the radiation source are selected to cover the spectral region of interest for the particular application.
  • the mirror and slit assembly 14 is positioned to receive the radiation rays from the source 12 after they have passed through the sample 24 and is operable to focus the radiation onto and through an entrance slit 30.
  • the collection mirror 28 focuses the radiation rays through slit 30 and illuminates the wavelength dispersing device 16.
  • radiation rays from the slit can also be collected through a lens 15, before illuminating a wavelength dispersion device 16.
  • the wavelength dispersing device 16 receives the beams of radiation from the mirror and slit assembly 14 and disperses the radiation into a series of lines of radiation each corresponding to a particular wavelength of the radiation spectrum.
  • the preferred wavelength dispersing device is a concave diffraction grating; however, other wavelength dispersing devices, such as a prism, can be utilized.
  • the wavelengths from the dispersing device 16 are in the near infrared portion of the spectrum and can cover, for example, the range of 1650-1850 nanometers (nm). It should be emphasized, however, that in general this device is not limited to just this or to any spectral region. It is intended that the dispersion device in general is capable of operating in other ranges of electromagnetic radiation, including the ultraviolet, visible, infrared, and microwave spectrum portions, as well as acoustic, electric, magnetic, and other signals, where applicable.
  • the spatial light modulator (SLM) 18 receives radiation from the wavelength dispersing device 16, individually modulates each spectral line, and reflects the modulated lines of radiation onto the detector 20.
  • the SLM is implemented in a first preferred embodiment as a micro-mirror array that includes a semiconductor chip or piezo-electric device 32 having an array of small reflecting surfaces 34 thereon that act as minors.
  • a micro-mirror array is manufactured by Texas Instruments and is described in more detail in U.S. Pat. No. 5,061,049, hereby inco ⁇ orated into the present application by reference.
  • the semi-conductor 32 of the micro-mirror array 18 is operable to individually tilt each mirror along its diagonal between a first position depicted by the letter A and a second position depicted by the letter B in Fig. 3. In preferred forms, the semi-conductor tilts each mirror 10 degrees in each direction from the horizontal.
  • the tilting of the mirrors 34 is preferably controlled by the analyzing device 22, which can communicate with the micro-mirror array 18 through an interface 37.
  • the micro-mirror array 18 is positioned so that the wavelength dispersing device 16 reflects each of the lines of radiation upon a separate column or row of the array. Each column or row of mirrors is then tilted or wobbled at a specific and separate modulation frequency. For example, the first row of mirrors can be wobbled at a modulation frequency of 100 Hz, the second row at 200 Hz, the third row at 300 Hz, etc. [0068] In a specific embodiment, the mirrors are calibrated and positioned so that they reflect all of the modulated lines of radiation onto a detector 20. Thus, even though each column or row of mirrors modulates its corresponding line of radiation at a different modulation frequency, all of the lines of radiation are focused onto a single detector.
  • the detector 20 which can be any conventional radiation transducer or similar device, is oriented to receive the combined modulated lines of radiation from the micro-mirror array 18.
  • the detector is operable for converting the radiation signals into a digital output signal that is representative of the combined radiation lines that are reflected from the micro-mirror array.
  • a reflector 36 can be inte ⁇ osed between the micro-mirror array 18 and the detector 20 to receive the combined modulated lines . of radiation from the array and to focus the reflected lines onto the detector.
  • the analyzing device -22 is operably coupled with the detector 20 and is operable to receive and analyze the digital output signal from the detector.
  • the analyzing device uses digital processing techniques to demodulate the signal into separate signals each representative of a separate line of radiation reflected from the micro-mirror array.
  • the analyzing device can use discrete Fourier transform processing to demodulate the signal to determine, in real time, the intensity of each line of radiation reflected onto the detector.
  • the analyzing device can separately analyze the characteristics of each line of radiation for use in analyzing the composition of the sample.
  • the analyzing device is preferably a computer that includes spectral analysis software.
  • Fig. 4 illustrates an output signal generated by the analyzing device in accordance with one embodiment. The output signal illustrated in Fig.
  • FIG. 4 is a plot of the abso ⁇ tion characteristics of five wavelengths of radiation from a radiation source that has passed through a sample.
  • Fig. 6 A it is used for digital imaging pu ⁇ oses.
  • an image of a sample 38 is focused onto a -micro-mirror array 40 and each micro-mirror in the array is modulated at a different modulation rate.
  • the micro-mirror array geometry is such that some or all of the reflected radiation impinges upon a single detector element 42 and is subsequently demodulated to reconstruct the original image improving the signal- to-noise ratio of the imager.
  • an analyzing device 44 digitally processes the combined signal to analyze the magnitude of each individual pixel.
  • Fig. 6B illustrates spatio-spectral distribution of the DMA, where individual elements can be modulated.
  • Fig. 5 is a plot of a three dimensional image showing the magnitude of each individual pixel.
  • Fig. 7 illustrates the output of a digital micro-mirror array (DMA) filter spectrometer used as a variable band pass filter spectrometer, variable band reject filter spectrometer, variable multiple band pass filter spectrometer or variable multiple band reject filter spectrometer.
  • DMA digital micro-mirror array
  • the wavelengths of electromagnetic radiation selected to pass to the detector correspond to the abso ⁇ tion band for compound A and abso ⁇ tion band for compound C in a sample consisting of compounds A, B, and C.
  • the spectral region corresponding to the abso ⁇ tion band of compound B and all other wavelengths of electromagnetic radiation are rejected.
  • the DMA filter spectrometer is not limited to the above example and can be used to pass or reject any combination of spectral resolution elements available to the DMA. Various examples and modifications are considered in detail below.
  • the spatial resolution elements (pixels) of an image can b selectively passed or rejected (filtered) according to the requirements of the image measurement.
  • the advantages of both the DMA filter spectrometer and DMA filter imager are: (1) All spectral resolution elements or spatial resolution elements corresponding to the compounds of interest in a particular sample can be directed simultaneously to the detector for measurement. This has the effect of increasing the signal-to-noise ratio of the measurement. (2) The amount of data requiring processing is reduced. This reduces storage requirements and processing times.
  • using a DMA one can provide one or more spectral band pass or band-reject filter(s) with a chosen relative intensity.
  • the radiation wavelengths that are reflected in the direction of the detector are selected by specific columns of micro-mirrors of the DMA, as illustrated in Fig. 8.
  • the relative intensity of the above spectral band is controlled by the selection of specific area of micro-mirrors on the DMA, represented by the dark area designated "A" in Fig. 8.
  • the dark area shown in Fig. 8 is the mirrors that direct specific wavelength radiation, i.e., spectral band, to the detector.
  • the "on" minors in the dark area create a band-pass filter, the characteristics of which are determined by the position of the "on” area in the DMA.
  • the bottom portion. of the figure illustrates the profile of the radiation reaching the detector.
  • Fig. 8 also demonstrates the selection of specific rows and columns of mirrors in the DMA used to create one 'spectral band filter with a single spectral mode. It should be apparent, however, that using the same technique of blocking areas in the DMA one can obtain a plurality of different specific spectral band filters, which can have multi- modal characteristics. The design of such filters is illustrated in Fig. 9. [0077] As shown in Fig. 9, a multitude of different specific filters can be designed on one DMA using simple stacking. Fig. 9 illustrates the creation of several filters by selective reflection from specific micro-mirrors. In particular, the left side of the figure illustrates the creation of three different filters, designated 1, 2, and 3.
  • Figure 9 left side Figure 9, right side Reflected radiation from micro-mirrors Filter created area A 1 area B 2 area C 3 areas a + b + c 1 + 2 + 3
  • Figure 10 illustrates the means for the intensity variation of a spectral filter built in accordance with this invention, and is summarized in the table below.
  • Figures 9 and 10 illustrate the ability to design spectral filters with different characteristics using a DMA.
  • a point to keep in mind is that different spectral components of the radiation from the sample have been separated in space and can be filtered individually. The ability to process individual spectral components separately should be retained. To this end, in accordance with the present invention, spectral components are modulated.
  • the basic idea is to simply modulate the output from different filters differently, so one can identify and process them separately.
  • different modulation is implemented by means of different modulation rates.
  • a system built in accordance with the present invention is capable of providing: a) Spectral bandwidth by selection of specific columns of micro- mirrors in an array; b) Spectral intensity by selection of rows of the array; and c) Spectral band identification by modulation.
  • Figures 11-14 illustrate alternative embodiments of a modulating spectrometer in accordance with this invention, where the DMA is replaced with different components.
  • Fig. 11A and B show an embodiment in which the DMA is replaced with fixed elements, in this case concave mirrors.
  • the idea is to use fixed spectral grating, which masks out spectrum block components that are not needed and passes those which are.
  • the idea here is that the broadly illuminated dispersive element distributes spectral resolution elements in one dimension so that in the orthogonal dimension one can collect light of the same wavelengths.
  • the focal plane one can see that at a particular plane, herein called the focal plane, one has a wavelength axis(x or columns) and a spatial axis(y or rows). If one were to increase the number of spatial resolution elements (y) that are allowed to pass energy through the system and out of the exit aperture for any given wavelength (x), or spectral resolution element (x), this would have the effect of increasing the intensity of the particular spectral resolution elements' intensity at the detector. [0085] If the array of spatio/spectral resolution elements at the focal plane as shown in Fig. 6A is replaced with fixed elements, such as the concave mirrors in Fig.
  • Fig. 11A shows the spatio/spectral resolution elements at the focal plane to be used.
  • the fixed optical elements are placed to interact with predetermined spatio/spectral resolution elements provided by the grating and entrance aperture geometry and to direct the specific assortment of spatio/spectral elements to specific spatial locations for modulation encoding (possibly using the barber pole arrangement, shown next).
  • Fig. 12 illustrates an embodiment of a complete modulating spectrometer in which the DMA element is replaced by the concave mirrors of Fig. 11.
  • Figure 13 illustrates a modulating lens spectrometer using lenses instead of DMA, and a "barber pole” arrangement of mirrors to implement variable modulation.
  • the "barber pole” modulation arrangement is illustrated in Fig. 14. [0087] With reference to Fig. 14, modulation is accomplished by rotating this "barber pole" that has different number of mirrors mounted for reflecting light from the spatially separated spectral wavelengths. Thus, irradiating each vertical section will give the reflector its own distinguishable frequency. In accordance with this embodiment, light from the pole is collected and simultaneously sent to the detector.
  • concave mirror 1 impinges upon the four-mirror modulator; concave mirror 2 radiation is modulated by the five-mirror modulator, and concave mirror 3 directs radiation to the six- mirror modulator.
  • the modulator rate is four, five, or six times per revolution of the "barber pole.”
  • this arrangement yields a modulation rate of 4/360° for the radiation from Area A, Figure 12.
  • the mirrors of Areas B and C are modulated at the rate of 5/360° and 6/360°, respectively.
  • all radiation from mirrors A, B, and C is simultaneously directed to the detector.
  • This radiation is collected by either a simple mirror lens or a toroidal mirror, which focuses the radiation onto a single detector.
  • the signal from the detector now goes to electronic processing and mathematical analyses for spectroscopic results.
  • modulating spectrometers a single light source of electromagnetic radiation was described. There exist yet another possibility for a unique optical design - a modulating multi-light source spectrometer.
  • Figs. 15 and 16 illustrate an embodiment of this invention in which a light source 12 provides several modulated spectral bands, e.g., light emitting diodes (LED), or lasers (shown here in three different light sources). The radiation from these light sources impinges upon the sample 24.
  • a light source 12 provides several modulated spectral bands, e.g., light emitting diodes (LED), or lasers (shown here in three different light sources).
  • the radiation from these light sources impinges upon the sample 24.
  • One possible illumination design is one in which light from a source, e.g. LED, passes through a multitude of filters, impinging upon the sample 24.
  • the radiation from the sample is transmitted to a detector 20, illustrated as a black fiber.
  • the signal from the detector is electronically processed to a quantitative and qualitative signal describing the sample chemical composition.
  • a plurality of light sources is used at differed modulating rates.
  • Fig. 15 and 16 illustrate the combination of several light sources in the
  • Spectral band Modulation Source Wavelength 1500-1700 mi 2 1600-1800 m 2 3 1700-1900 m 3
  • the radiation path consists of optical fibers.
  • mirrors and lenses could also constitute the optical path for a similar modulating multi-light source spectrometer.
  • the spectrometer described herein records spectral information about one unique area on a single detector. In a similar manner, the spectral characteristic of a multitude of areas in a sample can be recorded with a multitude of detectors in accordance with different embodiments of the invention.
  • Array detectors arc known in the art and include, for example [0096] Charge coupled devices (CCD), in the ultraviolet, and visible portions of the spectrum; InSb - array in near infrared; InGaAs - array in near infrared; Hg-Cd-Te - array in mid-infrared and other array detectors. [0097] Array detectors can operate in the focal plane of the optics. Here each detector of the array detects and records the signal from a specific area, xy. Practical Example B described herein on the gray-level camera provides a further illustration. Different aspects of the embodiments discussed herein are considered in more detail.
  • the postsample processing i.e., signal processing performed after a sample had been irradiated, describes an aspect of the present invention.
  • significant benefits can result from irradiating a sample with pre-processed radiation, in what is referred to as pre-sample processing.
  • one or more light sources capable of providing modulated temporal and/or spatial patterns of input radiation, should be used.
  • controllable source(s) of radiation which in general are capable of generating arbitrary combinations of spectral radiation components within a predetermined spectrum range.
  • Several types of prior art devices are known that are capable of providing controllable radiation. Earlier prior art devices primarily relied upon various "masking" techniques, such as electronically alterable masks inte ⁇ osed in the optical pathway between a light source and a detector. More recent prior art devices use a combination of two or more light-emitting diodes (LEDs) as radiation sources. Examples are provided in U.S. Pat. Nos. 5,257,086 and 5,488,474, the content of which is hereby inco ⁇ orated by reference for all pu ⁇ oses.
  • an array of LEDs or light- emitting lasers is configured for activation using a particular encoding pattern, and can be used as a controllable light source.
  • a disadvantage of this system is that it relies on an array of different LED elements, each operating in a different, relatively narrow spectrum band.
  • Fig. 17 illustrates a schematic representation of an apparatus in accordance with the present invention using a controllable radiation source.
  • the system includes a broadband radiation source 12, DMA 18, wavelength dispersion device 16, slit assembly 30, detector 20 and control assembly 22.
  • control assembly 22 can include a conventional personal computer 104, interface 106, pattern generator 108, DMA driver 110, and analog to digital (A/D) converter 114.
  • Pattern generator 108 can include an EPROM memory device (not shown) which stores the various encoding patterns for array 18, such as the Hadamard encoding pattern discussed below.
  • generator 108 delivers signals representative of successive patterns to driver 110. More particularly, generator 108 produces output signals to driver 110 indicating the activation pattern of the mirrors in the DMA 18.
  • A/D converter 114 is conventional in nature and receives the voltage signals from detector 20, amplifies these signals as analog input to the converter in order to produce a digital output representative of the voltage signals.
  • Radiation . source 12, grating 16, DMA 18 slit assembly 30 and detector 20 cooperatively define an optical pathway.
  • Radiation from source 12 is passed through a wavelength dispersion device, which separates in space different spectrum bands.
  • the desired radiation spectrum can them be shaped by DMA 18 using the filter arrangement outlined herein.
  • radiation falling on a particular micro-mirror element can also be encoded with a modulation pattern applied to it.
  • DMA 18 is activated to reflect radiation in a successive set of encoding patterns, such as Hadamard, Fourier, wavelet or others.
  • the resultant set of spectral components is detected by detector 20, which provides corresponding output signals.
  • Computer 22 then processes these signals. [00105]
  • Computer 22 initiates an analysis by prompting pattern generator
  • the active light source in accordance with the present invention consists of one or more light sources, from which various spectral bands are selected for transmission, while being modulated with a temporal and/or spatial patterns. The resulting radiation is then directed at a region (or material) of interest to achieve a variety of desired tasks.
  • a brief listing of these tasks include: (a) Nery precise spectral coloring of a scene, for pu ⁇ oses of enhancement of display and photography; (b) Precise illumination spectrum to correspond to specific abso ⁇ tion lines of a compound that needs to be detected, (see figures 40-44 on protein in wheat as an illustration) or for which it is desirable to have energy abso ⁇ tion and heating, without affecting neighboring compounds (This is the principle of the microwave oven for which the radiation is tuned to be absorbed by water molecules allowing for heating of moist food only); (c) The procedure in (b) could be used to imprint a specific spectral tag on ink or paint, for watermarking, tracking and forgery prevention, acting as a spectral bar code encryption; (d) The process of light curing to achieve selected chemical reactions is enabled by the tunable light source.
  • Duality allows one to reverse or "turn inside out” any of the post-sample processing configurations described previously, to yield a pre-sample processing configuration. Essentially, in the former case one takes post sample light, separates wavelengths, encodes or modulates each, and detects the result. The dualized version for the latter case is to take source light, separates wavelengths, encode or modulate each, interact with a sample, and detect the result. [00108] Various embodiments of systems for performing post- and pre- sample processing were discussed herein.
  • the central component of the system is a digital micro-mirror array (DMA), in which individual elements (micro-mirrors) can be controlled separately to either pass along or reject certain radiation components.
  • DMA array can perform various signal processing tasks.
  • the functionality of the DMAs discussed above can be generalized using the concept of Spatial Light Modulators (SLMs), devices that broadly perform spatio-spectral encoding of individual radiation components, and of optical synapse processing units (OSPUs), basic processing blocks. This generalization is considered herein as well as the Hadamard processing, spatio-spectral tagging, data compression, feature extraction and other signal processing tasks.
  • SLMs Spatial Light Modulators
  • OSPUs optical synapse processing units
  • an SLM in accordance with this invention is any device capable of controlling the magnitude, power, intensity or phase of radiation or which is otherwise capable of changing the direction of propagation of such radiation. This radiation can either have passed through, or be reflected or refracted from a material sample of interest.
  • an SLM is an array of elements, each one capable of controlling radiation impinging upon it.
  • an SLM placed in appropriate position along the radiation path can control either spatial or spectral components of the impinging radiation, or both.
  • "light” is used here in a broad sense to encompass any portion of the electromagnetic spectrum and not just the visible spectrum.
  • SLM's in accordance with different embodiments of the invention include liquid crystal devices, actuated micro-mirrors, actuated mirror membranes, di-electric light modulators, switchable filters and optical routing devices, as used by the optical communication and computing environments and optical switches.
  • a DMA as an example of spatial light modulating element is discussed herein.
  • a ID, 2D, or 3D SLM is configured to receive any set of radiation components and functions to selectively pass these components to any number of receivers or image planes or collection optics, as the application can require, or to reject, reflect or absorb any input radiation component, so that either it is or is not received by one or more receivers, image planes or collection optics devices.
  • the SLM is implemented as a DMA, virtually any array of switched elements can be used in accordance with the present invention.
  • an SLM in accordance with the invention is capable of receiving any number of radiation components, which are then encoded, tagged, identified, modulated or otherwise changed in terms of direction and/or magnitude to provide a unique encodement, tag, identifier or modulation sequence for each radiation component in the set of radiation components, so that subsequent optical receiver(s) or measuring device(s) have the ability to uniquely identify each of the input radiation components and its properties.
  • properties include, but are not limited to, irradiance, wavelength, band of frequencies, intensity, power, phase and/or polarization.
  • the tagging of individual radiation components can be accomplished using rate modulation.
  • spectral components of the input radiation that have been separated in space using a wavelength dispersion device are then individually encoded by modulating the micro-mirrors of the DMA array at different rates.
  • the encoded radiation components are directed to a single detector, . but nevertheless can be analyzed individually using Fourier analysis of the signal from the detector. Other examples for the use of "tagging" are discussed below.
  • various processing modalities can be realized with an array of digitally controlled switches (an optical synapse), which function to process and transmit signals between different components of the system.
  • the basic OSPU can be thought of as a data acquisition unit capable of scanning an array of data, such as an image, in various modes, including raster, Hadamard, multiscale wavelets, and others, and transmitting the scanned data for further processing.
  • a synapse is a digitally controlled array of switches used to redirect image (or generally data) components or combinations of light streams, from one location to one or more other locations.
  • it can perform Hadamard processing, as defined below, on a plurality of radiation elements by combining subsets of the elements (i.e., binning) before conversion to digital data.
  • a synapse can be used to modulate light streams by modulating temporally the switches to impose a temporal bar code (by varying in time the binning operation). This can be . built in a preferred embodiment from a DMA, or any of a number of optical switching or routing components, used for example in optical communications applications.
  • An OSPU unit in accordance with the present invention is shown in diagram form in Fig. 18A and 18B, as three-port device taking input from a radiation source S, and distributing it along any of two other paths, designated C (short for camera) and D (for detector). Different scanning modes of the OSPU are considered in more detail herein.
  • an OSPU is implemented using a DMA, where individual elements of the array are controlled digitally to achieve a variety of processing tasks while collecting data.
  • information bearing radiation sources could be, for example, a stream of photons, a photonic wavefront, a sound wave signal, an electrical signal, a signal propagating via an electric field or a magnetic field, a stream of particles, or a digital signal.
  • Examples of devices that can act as a synapse include spatial light modulators, such as LCDs, MEMS mirror arrays, or MEMS shutter arrays; optical switches; optical add-drop multiplexers; optical routers; and similar devices configured to modulate, switch or route signals.
  • DMAs and other optical routing devices can be used to this end.
  • LCD liquid crystal displays
  • CCD charge coupled devices
  • CMOS logic arrays of microphones, acoustic transducers, or antenna elements for electromagnetic radiation and other elements with similar functionality that will be developed in the future, can also be driven by similar Methods.
  • Applicants' contribution in this regard is in the novel process of performing pretransduction digital computing on analog data via adaptive binning means. Such novelty can be performed in a large number of ways. For example, one can implement adaptive current addition using a parallel/serial switch and wire networks in CMOS circuits.
  • one or more microphones can be used in combination with an array of adjustable tilting sound reflectors (like a DMD for sound).
  • an array of adjustable tilting sound reflectors like a DMD for sound.
  • the shift of paradigm is clear: in the prior art traditionally analog signals are captured by a sensor, digitized, stored in a computer as a "data cube", and then processed. Considerable storage space and computational requirements are extended to do this processing.
  • data from one or more sensors is processed directly in the analogue domain, the processed result is digitized and sent to a computer, where the desired processing result can be available directly, or following reduced set of processing operations.
  • the digitally controlled array is used as a hybrid computer, which through the digital control of the array elements performs (analog) computation of inner products or more generally of various correlations between data points reaching the elements of the array and prescribed patterns.
  • the digital control at a given point (i.e., element) of the array can be achieved through a variety of different mechanisms, such as applying voltage differences between the row and column intersecting at the element; the modulation is achieved by addressing each row and column of the array by an appropriately modulated voltage pattern.
  • the mirrors are fluctuating between two tilted positions, and modulation is achieved through the mirror controls, as known in the art.
  • the specifics of providing to the array element of signal(s) following a predetermined pattern will depend on the design implementation of the array and are not considered in further detail.
  • the OSPU array is processing raw data to extract desired information.
  • FIG. 19 illustrates in block diagram form the design of a spectrograph using OSPU.
  • the basic design brings reflected or transmitted radiation from a line in the sample or source onto a dispersing device 16, such as a grating or prism, onto the imaging fiber into the OSPU to encode and then forward to a detector 20.
  • Fig. 20 illustrates in a diagram form an embodiment of a tunable light source, which operates as the spectrograph in Fig. 19, but uses a broadband source.
  • Fig. 21 illustrates in a diagram form an embodiment of the spectral imaging device discussed herein, which is built with two OSPUs. Different configurations of generalized processing devices are illustrated in Fig. 22, in which each side is imaging in a different spectral band, and Fig. 23, which illustrates the main components of a system for processing input radiation using an OSPU.
  • Fig. 24 is a flow chart of a raster-scan using in one embodiment of the present invention. This algorithm scans a rectangle, the "Region Of Interest (ROI)," using ordinary raster scanning. It is intended for use in configurations in this disclosure that involve a spatial light modulator (SLM).
  • SLM spatial light modulator
  • Fig. 25 is a flowchart of a Walsh-Hadamard scan used in accordance with another embodiment of the invention. This algorithm scans a rectangle, the "Region Of Interest (ROI)", using Walsh-Hadamard multiplexing.
  • Walsh( dx, m, i, dy, n, j) is the Walsh-Hadamard pattern with origin (dx, dy), of width 2 m and height 2 n , horizontal Walsh index i, and vertical Walsh index j.
  • Fig. 26 is a flowchart of a multi-scale scan.
  • Fig. 26A illustrates a multi-scale tracking algorithm in a preferred embodiment of the present invention.
  • the algorithm scans the region of interest, (using multi-scan search), to find an object of interest and then tracks the object's movement across the scene. It is intended for use in a setting where multi-scale search can be used, and where the "interest factor" is such that a frackable object can be found.
  • F(L;) distance between the spectrum read by the sensor, and the spectrum of a compound of interest, (distance could be, e.g., Euclidean distance of some other standard distance). This will cause the algorithm to zoom in on a substance of interest.
  • F(Li) distance between the spectrum read by the sensor, and the spectrum already read for L 0 . This will cause the algorithm to zoom in on substances that are anomalous compared to the background.
  • F(Li) can depend on a priori data from spectral or spatiospectral libraries.
  • the interest factor definitions can be pre-stored so a user can analyze a set of data using different interest factors.
  • Hadamard processing refers generally to analysis tools in which a signal is processed by correlating it with strings of 0 and 1 (or +/- 1). Such processing does not require the signal to be converted from analogue to digital, but permits direct processing on the analogue data by means of an array of switches (synapse).
  • an array of switches such as a DMA, is used to provide spatio-spectral tags to different radiation components. In alternative embodiments it can also be used to impinge spatio/spectral signatures, which directly correlate to desired features.
  • a simple way to explain Hadamard spectroscopy is to consider the example of the weighing schemes for a chemical scale. Assume that we need to weigh eight objects, x 1?
  • x 2; ..., x 8 on a scale.
  • Wi (1,-1,1,1,-1-1,1,-1)
  • This processing method has certain advantages to the raster scan in which the detector measures one band at a time.
  • a Walsh packet is a localized Walsh string of ⁇ 1.
  • a correlation of a vector x with a Walsh packet measures a variability of x at the location where the packet oscillates.
  • the Walsh packet library is a simple and computationally efficient analytic tool allowing sophisticated discrimination with simple binary operations. It can be noted that in fact, it is precisely the analog of the windowed Fourier transform for binary arithmetic.
  • a correlation of a vector x with a Walsh packet measures a variability of x at the location where the packet oscillates.
  • the Walsh packet library is a simple and computationally efficient analytic tool allowing sophisticated discrimination with simple binary operations. It can be noted that in fact, it is precisely the analog of the windowed Fourier transform for binary arithmetic.
  • [00144] As an illustration, imagine two compounds A and B with subtle differences in their spectrum. The task is to discriminate among them in a noisy environment and design efficient mirror configurations for DMA spectroscope. In accordance with a preferred embodiment, the following procedure can be used: (1) Collect samples for both A and B, the
  • a sample in this context is a full set x of the spectrum of the compound.
  • the objective of this imaging is to obtain a sequence of images, one for each spectral band.
  • true hyper-spectral imaging devices having the ability to collect and process the full combination of spectral and spatial data are not really practical as they require significant storage space and computational power.
  • significant improvement over the prior art can be achieved using hyper-spectral processing that focuses of predefined characteristics of the data. For example, in many cases only a few particular spectral lines or bands out of the whole data space are required to discriminate one substance over another. It is also often the case that target samples do not possess very strong or sha ⁇ spectral lines, so it can not be necessary to use strong or sha ⁇ bands in the detection process.
  • FIG. 29 A generalized block diagram of hyper-spectral processing in accordance with the invention is shown in Fig. 29.
  • Fig. 30 illustrates two spectral components (red and green) of a data cube produced by imaging the same object in different spectral bands. It is quite clear that different images contain completely different kinds of information about the object. The same idea is illustrated in Figs. 31 and 32, where Fig.
  • .31 illustrates hyper-spectral imaging from airborne camera and shows how one can identify different crops in a scene, based on the predominant spectral characteristic of the crop.
  • Fig. 32 is an illustration of a hyper-spectral image of human skin with spectrum progressing from left to right and top to bottom, with increasing wavelength.
  • Figs. 33A-E illustrate different embodiments of an imaging spectrograph in dedispersive mode, that can be used in accordance with this invention for hyper-spectral imaging in the UN, visual, near infrared and infrared portions of the spectrum.
  • the figures show a fiber optic probe head with a fixed number of optical fibers. As shown, the fiber optic is placed at an exit slit.
  • FIG. 34 shows an axial and cross-sectional view of the fiber optic assembly illustrated in Figs. 33A-E .
  • FIG. 35 shows a physical arrangement of the fiber optic cable, detector and the slit.
  • FIG. 36 illustrates a fiber optic surface contact probe head abutting tissue to be examined.
  • Fig. 37A and 37B illustrate a fiber optic e-Probe for pierced ears that can be used for medical monitoring applications in accordance with the present invention.
  • Fig. 38A, 38B and 38C illustrate different configurations of a hyper-spectral adaptive wavelength advanced illuminating imaging spectrograph (HAWAIIS).
  • HWAIIS hyper-spectral adaptive wavelength advanced illuminating imaging spectrograph
  • DMD shown illuminating the -1 order
  • the illumination is fully programmable and can be modulated by any contiguous or non-contiguous combination at up to 50KHz,.
  • the corresponding spatial resolution element located at the Object/sample is thus illuminated and is simultaneously spectrally imaged by the CCD (located in order +1 with efficiency at 80%) as in typical CCD imaging spectrographs used for Raman spectral imaging.
  • the output of a broadband light source such as a TQH light bulb(l ⁇ l) is collected by a collection optic (lens 1002) and directed to a spatial light modulator such as the DMA used in this example(1003).
  • a spatial light modulator such as the DMA used in this example(1003).
  • Specific spatial resolution elements are selected by computer controlled DMA driver to propogate to the transmission diffraction grating(1005) via optic (lens 1004).
  • the DMA(1003) shown illuminating the -1 order of the transmission diffraction grating(1005) is a programmable spatial ligh modulator that is used to select spatio/spectral resolution elements projecting through the entrance/exit slit(#1007) collected and focused upon the sample(1009) by optic (lens 1008).
  • the spatio/spectral resolution elements illuminating the sample are fully programmable.
  • the sample is thus illuminated with specific and known spectral resolution elements.
  • the reflected spectral resolution elements from specific spatial coordinates at the sample plane are then collected and focused back through the entrance/exit slit by optic (lens 1008).
  • Optic (lens 1006) coUimates the returned energy and presents it to the transmission diffraction grating(1005).
  • the light is then diffracted preferentially into the +1 order and is subsequently collected and focused by the optic (lens 1010) onto a 2D dector aray(1011).
  • This conjugate spectral imaging device has the advantage of rejecting out of focus photons from the sample.
  • Spectral resolution elements absorbed or reflected are measured with spatial specificity by the device.
  • Figs. 45-49(A-D) illustrate hyperspectral processing in accordance with the present invention, including data maps, encodement mask, DMA programmable resolution using different numbers of mirrors and several encodegrams.
  • One aspect of the present invention is the use of modulation of single array elements or groups of array elements to "tag" radiation impinging on these elements with its own pattern of modulation. In essence, this aspect of the invention allows to combine data from a large number of array elements into a few processing channels, possibly a single channel, without losing the identity of the source and/or the spatial or spectral distribution of the data.
  • the combination and binning of a plurality of radiation sources is manipulated in accordance with this invention to perform calculations on the analog data, which is traditionally done in the digital data analysis process.
  • a whole processing step can be eliminated by preselecting the switching modulation to perform the processing before the A/D conversion, thereby only converting data quantities of interest.
  • This aspect of the present invention enables realtime representation of the final processed data, which in processing intense applications can be critical. [00161]
  • the raw data can be converted directly on the sensor to provide the data in transform coordinates, such as Fourier transform, Wavelet transform, Hadamard, and others.
  • this can be achieved by use of the best basis algorithm. See, for example, Coifman, R. R. and Wickerhauser, M. N., "Entropy-based Algorithms for Best Basis Selection", IEEE Trans. Info. Theory 38 (1992), 713-718, and U.S. Pat ⁇ os. 5,526,299 and 5,384,725 to one of the inventors of this application.
  • the referenced patents and publications are inco ⁇ orated herein by reference. [00163]
  • the reduction of dimensionality of a set of data vectors can be accomplished using the projection of such a set of vectors onto a orthogonal set of functions, which are localized in time and frequency.
  • the projections are defined as correlation of the data vectors with the set of discretized re-scaled Walsh functions, but any set of appropriate functions can be used instead, if necessary.
  • the best basis algorithm to one of the co-inventors of this application provides a fast selection of an adapted representation for a signal chosen from a large library of orthonormal bases. Examples of such libraries are the local trigonometric bases and wavelet packet bases, both of which consist of waveforms localized in time and frequency.
  • An orthonormal basis in this setting corresponds to a tiling of the timer frequency plane by rectangles of area one, but an arbitrary such tiling in general does not correspond to an orthonormal basis.
  • Walsh packet analysis is a robust, fast, adaptable, and accurate alternative to traditional chemometric practice. Selection of features for regression via this method reduces the problems of instability inherent in standard methods, and provides a means for simultaneously optimizing and automating model calibration.
  • the function w p and w q are orthogonal if and only if the tiles p and q are disjoint.
  • any disjoint tiling will give rise to an orthonormal basis of L 2 (0,1) consisting of rescaled Walsh functions.
  • N the number of sample spectra in the given data set (N is 50 for the calibration set). Retaining the same notation as for the calibration set, one can compute the feature X; for each validation spectrum Si and use the above model to predict Yj for the validation spectra. The average percentage regression error on the validation set is .62 %, and this serves as the measure of success for the model.
  • This model is known to be state-of-the-art in terms of both concept and performance for this data, and will be used as point of comparison. [00172] .
  • the wavelength-by- wavelength data of each spectrum is a presentation of the data in a particular coordinate system. Walsh packet analysis provides a wealth of alternative coordinate systems in which to view the data.
  • the coordinates of an individual spectrum would be the correlation of the spectrum with a given Walsh packet.
  • the Walsh packets themselves are functions taking on the values 1, -1, and 0 in particular patterns, providing a square- wave analogue of local sine and cosine expansions. Examples of Walsh packets are shown in Fig. 28. [00173]
  • such functions can be grouped together to form independent coordinate systems in different ways.
  • Fig. 41A shows a typical wheat spectrum together with one of the top 4 Walsh packets used in this model.
  • the feature that is input to the regression model is the correlation of the Walsh packet with the wheat spectrum.
  • the Walsh feature computes a second derivative, which suppresses the background and detects the curvature of the hidden protein spectrum in this region).
  • the feature being measured is a sum from many wavelengths, naturally reducing the effect of the noise.
  • the Walsh packet method described here has other advantages, such as automation.
  • the fast search algorithm automatically selects the best Walsh packets for performing the regression. If the data set were changed to, say, blood samples and concentrations of various analytes, the same algorithm would apply off the shelf in determining optimal features. The standard model would need to start from scratch in determining via lengthy experiment which wavelengths were most relevant.
  • Adaptability is also a benefit.
  • the optimality of the features chosen is based on a numeric cost function, in this case a linear regression error. However, many cost functions can be used and in each case a representation adapted to an associated task will be chosen.
  • Optimal coordinates can be chosen for classification, compression, clustering, non-linear regression, and other tasks. In each case, automated feature selection chooses a robust set of new coordinates adapted to the job in question.
  • a system in which a video camera is synchronized to the tunable light source modulation allowing analysis of the encoded spectral bands from a plurality of video images, thereby providing a multispectral image. Since the ambient light is not modulated it can be separated from the desired spectral information.
  • This system is the functional equivalent of imaging the scene a number of times with a multiplicity of color filters. It allows the formation of any virtual photographic color filter with any abso ⁇ tion spectrum desired.
  • a composite image combining any of these spectral bands can be formed to achieve a variety of image analysis, filtering and enhancing effects.
  • an object with characteristic spectral signature can be highlighted by building a virtual filter transparent to this signature and not to others (which should be suppressed).
  • an ordinary video camera used in accordance with this invention is equipped with a synchronized tunable light source so that odd fields are illuminated with a spectral signature which is modulated from odd field to odd field while the even fields are modulated with the complementary spectral signature so that the combined even odd light is white.
  • a synchronized tunable light source so that odd fields are illuminated with a spectral signature which is modulated from odd field to odd field while the even fields are modulated with the complementary spectral signature so that the combined even odd light is white.
  • This illumination processing system can be used for machine vision for tracking objects and anywhere that specific real time spectral information is useful
  • a gray level camera can measure several preselected light bands using, for example, 16 bands by illuminating the scene consecutively by the 16 bands and measuring one band at a time.
  • a better result in accordance with this invention can be obtained by selecting 16 modulations, one for each band, and illuminating simultaneously the scene with all 16 colors.
  • the sequence of 16 frames can be used to demultiplex the images.
  • the advantages of multiplexing will be appreciated by those of skill in the art, and include: better signal to noise ratio, elimination of ambient light interference, tunability to sensor dynamic range constraints, etc.
  • a straightforward extension of this idea is the use of this approach for multiplexing a low resolution sensor array to obtain better image quality.
  • a 4X4 array of mirrors with Hadamard coding could distribute a scene of 400x400 pixels on a CCD array of 100X100 pixels resulting in an effective array with 16 times the number of CCD. Further, the error could be reduced by a factor of four over a raster scan of 16 scenes.
  • the present invention by irradiating a sample of material with well-chosen bands of radiation that are separately identifiable using modulation, one can directly measure constituents in the material of interest. This measurement, for example, could be of the protein quantity in a wheat pile, different chemical compounds in human blood, or others.
  • Fig. 39 shows a DMA search by splitting the scene. The detection is achieved by combining all photons from the scene into a single detector, then splitting the scene in parts to achieve good localization. In this example, one is looking for a signal with energy in the red and blue bands. Spectrometer with two detectors, as shown in Fig.
  • Fig. 40 illustrates the sum of wheat spectra training data (top) Sum of
  • Fig. 41 illustrates the top 10 wavelet packets in local regression basis selected using 50 training samples. Each Walsh packet provides a measurement useful for estimation. For example, the top line indicates that by combining the two narrow bands at the ends and then subtracting the middle band we get a quantity which is linearly related to the protein concentration.
  • Fig. 42 is a scatter plot of protein content (test data) vs. correlation with top wavelet packet. This illustrates a simple mechanism to directly measure relative concentration of desired ingredients of a mixture.
  • Fig 43 illustrates PLS regression of protein content of test data: using top 10 wavelet packets (in green - 1.87% error, from 6 LVs) and top 100 (in red - 1.54% error from 2 LVs) - compare with error of 1.62% from 14 LVs using all original data. This graph compares the performance of the simple method described above to the true concentration values.
  • Fig 44 illustrates the advantage of DNA-based Hadamard
  • a simple detection mechanism for compounds with known abso ⁇ tion is to use an active illumination system that transmits light (radiation) only in areas of the abso ⁇ tion spectrum of the compound. The resulting reflected light will be weakest where the compound is present, resulting in dark shadows in the image (after processing away ambient light by, for example, subtracting the image before illumination).
  • this approach can be used to dynamically track objects in a video scene.
  • a red ball could be tracked in a video sequence having many other red objects, simply by characterizing the red signature of the ball, and tuning the illumination to it, or by processing the refined color discrimination.
  • this capability is useful for interactive TV or video-gaming, machine vision, medical diagnostics, or other related applications.
  • similar processing can be applied in the infrared range (or UV) to be combined with infrared cameras to obtain a broad variety of color night vision or (heat vision), tuned to specific imaging tasks.
  • PCM pulse code modulation
  • PWM pulse width modulation
  • TDM time division multiplexing
  • any other modulation technique that has the property of identifying specific elements of a complex signal or image.
  • the tunable light source of this invention can be tuned to the abso ⁇ tion profile of a compound that is activated by absorbing energy, to achieve curing, drying, heating, cooking of specific compounds in a mixture.
  • Apps further include photodynamic therapy, such as used in jaundice treatment, chemotherapy, and others.
  • Yet another application is a method for conducting spectroscopy with determining the contribution of individual radiation components from multiplexed measurements of encoded spatio-spectral components.
  • a multiplicity of coded light in the UV band could be used to cause fluorescence of biological materials, the fluorescent effect can be analyzed to relate to the specific coded UV frequency allowing a multiplicity of measurements to occur in a multiplexed form.
  • An illumination spectrum can be designed to dynamically stimulate the material to produce a detectable characteristic signature, including fluorescence effects and multiple fluorescent effects, as well as Raman and polarization effects.
  • Shining UV light in various selected wavelengths is known to provoke characteristic fluorescence, which when spectrally analyzed can be used to discriminate between various categories of living or dead cells.
  • Another application of the system and method of this invention is the use of the OSPU as a correlator or mask in an optical computation device.
  • an SLM such as DMA can act as a spatial filter or mask placed at the focal length of a lens or set of lenses.
  • the SLM can be configured to reject specific spatial resolution elements, so that the subsequent image has properties that are consistent with the spatial filtering in Fourier space. It will be apparent that the transform of the image by optical means is spatially effected, and that the spatial resolution of images produced in this manner can be altered in any desired way.
  • Yet another area of use is performing certain signal processing functions in analog domain.
  • spatial processing with a DMA can be achieved directly in order to acquire various combinations of spatial patterns.
  • an array of mirrors can be arranged to have all mirrors of the center of the image point to one detector, while all the periphery goes to the other.
  • Another useful arrangement designed to detect vertical, edges will raster scan a group of, for example, 2x2 mirrors pointing left combined with an adjacent group of 2x2 mirrors pointing right. This corresponds to a convolution of the image with an edge detector.
  • the ability to design filters made out of patterns of 0,1,-1 i.e., mirror configurations, will enable the imaging device to only measure those features which are most useful for display, discrimination or identification of spatial patterns.
  • the hyper-spectral analysis system and method based on a tuned light MOEMS system transmits any combination of light wavelengths, e.g., in the range 450 nm - 850 nm trans- illuminating H & E stained micro-array tissue sections of normal and malignant colon with a microscope, such as a Nikon Biophot microscope.
  • Hyper-spectral pictures of tissues obtained with a charge coupled device (CCD) camera such as a Sensovation Samba QS-34 (see http://www.sensovation.com), are captured by a computer and analyzed mathematically to discriminate between normal and malignant cells and tissues.
  • the method includes a training phase and a testing phase.
  • An illustrative example of a training phase can consist of a data collection in which 61 hyper-spectral pictures are collected at 400X magnification: 15 pictures of normal colon tissue from 10 different patients and 46 pictures of malignant colon tissue from 42 different patients.
  • the hyper-spectral analysis system and method of the present invention normalizes, compresses and analyzes the spectra of each pixel to discriminate between gland nuclei, gland cytoplasm and lamina intestinal/lumens as described herein.
  • the hyper-spectral analysis system and method automatically extracts the pixel spectra and classifies the extracted pixel spectra as nuclei.
  • the hyper-spectral analysis method and system finds spectral features separating normal nuclei from abnormal nuclei. Once the spectral features are found, a testing phase can be conducted in which samples of unknown status are imaged, and the status (i.e. normal vs.
  • each hyper-spectral image is a 3-D data cube.
  • each datacube has spatial coordinates x (491 pixels) and y (653 pixels), and spectral coordinate z (128 pixels) (for a total of 41 million pixels) representing transmitted spectra.
  • the present invention calculates the logarithm of spectra so that Beer's law can be applied.
  • the data is de-noised and reduced from the original 128 spectra to 64 spectral samples in the range 480 - 600 nm.
  • the hyper- spectral analysis method and system normalizes, compresses and analyzes as follows: 1) the spectra are classified or labeled as belonging to one of three classes: gland nuclei, gland cytoplasm, or lamina intestinal/lumens; 2) discriminating spectral signatures are found using a local discriminant basis algorithm or other comparable algorithm; 3) nuclei spectra are extracted using a nearest neighbor algorithm (e.g.
  • the hyper-specfral analysis method and system extracts sets of spectra (patches) belonging to the same nucleus, or neighboring nuclei groups and generates the following support vector machine classifiers: a frequency- wise standard deviation classifier, two principal component classifiers, and a final classifier based on these three classifiers.
  • the hyper- spectral analysis system and method In discriminating normal and abnormal data cubes, the hyper- spectral analysis system and method randomly collects a training set of 2440 patches from 61 datacubes (excluding adenomas) using the following criteria: any patch with "no abnormal" nuclei is classified as "normal” and any patch with "any abnormal” nuclei is classified as "abnormal”. [00209] In evaluate mean classifying of data cube spectra patches, the hyper-spectral analysis system and method builds test sets of 1800 patches by randomly selecting 30 patches from 67 data cubes (15 normal, 45 malignant, 7 adenomas) and evaluating the classifier.
  • the hyper-spectral analysis method and system sets a threshold of 0.5 to classify the biopsies.
  • the term frequency as used herein means the speed of light divided by wavelength. In measuring the spectrum, each pixel is assumed as having 128 dimensional vector, thereby resulting in different sets of coordinates in this 128 dimensional space.
  • ⁇ ⁇ I ⁇ 5(V 0 + & v)]
  • A 1 128 ⁇
  • These measurement correspond naturally to a raster scan of the sample: for each k, light at frequency v ⁇ +kA v [ s illuminated through the sample and the CCD registers the number of photons at that frequency transmitted through the sample. However, since the total amount of light of the source is constant, the amount of .
  • the hyper-spectral system and method utilizes a basis of Walsh packets.
  • An orthogonal basis different from “R , the system denotes it as "w , consisting of vectors w ⁇ , each vector (besides one measuring the mean of the signal) having half entries equal to 1 and half entries equal to — 1 .
  • the present system hierarchically organizes the vectors by scale, such that for each J between 0 and 1°S 2 (N) (N being the length of the signal), the J dimensional space, spanned by signals constant on dyadic intervals at scale 2 ? is spanned by exactly J packets.
  • N the length of the signal
  • the measurement of is physically impossible since this would involve illuminating light having spectral shape w t where certain w t are partly negative.
  • the present system and method recharacterizes or writes wi as w i —L i —H i where the functions -" / are positive.
  • This technique is well-known as Hadamard spectroscopy, an example of the more general idea of multiplexing. It is appreciated that there is an orthogonal transformation (or, dually, a change of orthogonal bases) mapping the Hadamard coefficient bijective to the raster scan coefficients. These are just two representations of the same signal on two different orthogonal bases.
  • the hyper-spectral analysis system and method performs a randomized Hadamard scan which essentially randomly shuffles the frequency axis by a bijective map P : the shuffled spectra f P ) are no longer smooth, and the size and signal to noise ratio of all the coefficients W,- (P )) are almost uniform.
  • the new patterns w, ⁇ P ) look like noise, but are treated as another orthogonal basis by the present invention. There is a simple orthogonal transformation mapping this basis into the old w t basis and into the raster basis ° ⁇ , thus allowing transformation of the coefficients from one basis onto another basis. [00217]
  • the hyper-specfral analysis system and method acquires data by performing a randomized Hadamard scan with a fixed random permutation of the frequencies for all measurements.
  • the first column v ⁇ of ⁇ is the axis of maximum variance of the data
  • the second column v ⁇ of N is the axis of maximum variance for the projection of the data onto the subspace orthogonal to v ⁇
  • v j is the axis of maximum variance for the projection of the data onto the subspace orthogonal to
  • the principal components form an orthogonal basis of axes of maximum variances for the data, and the singular value associated to each is proportional to the variance of the data in that direction.
  • the orthogonal projection of the data onto the subspace spanned by the first k principal components is optimal in the sense that it minimizes
  • the first k principal components are an optimal choice for a k-dimensional projection of the data, and can be used for data representation and de-noising.
  • nonlinearities in the data can cause principal components to be a bad choice for representation in the data, as in fact could be any linear projection, since a nonlinear structure can be intrinsically low-dimensional without having to lie in any low-dimensional linear subspace. This is why nonlinear transformation of the data into low-dimensional spaces are often considered.
  • the hyper-specfral analysis method and system employs Local Discriminant Bases (LDB) which apply naturally to a family of labeled vectors that represents smoothly varying functions, for example spectra and sounds. If these labeled correspond to more or less well-defined clusters in the data, the vectors can be very high dimensional and clustering or non-linear separation methods between classes can be very expensive if not unfeasible.
  • LDB Local Discriminant Bases
  • the goal of LDB is to find directions in these high dimensional spaces such that the data projected onto these directions are still well-discriminated (i.e., readily distinguishable).
  • discriminating the low-dimensional projections of the data should be almost as good as discriminating in the high-dimensional space with all the advantages and tools available in the lower dimensional space. While discriminating features are preserved by the LDB, the non-discriminating features are removed, thereby de-noising the data, at least with respect to the discrimination task.
  • the features search in the high dimensional space is notoriously difficult.
  • One way some local discriminant bases alleviate some aspects of the "curse of dimensionality" is to search the sub-optimal projections among hierarchically well- organized dictionaries of wavelet or Fourier packets. It is appreciated that there are fast algorithms to search through them and compute the projections onto ensembles of these patterns.
  • the hyper-specfral analysis method and system utilizes a version of local discriminant bases that uses arbitrary Haar packet decompositions of the phase-space, but other, even less flexible, wavelet dictionaries can be used as well. In all cases the discriminating features have properties of smoothness and locality.
  • the hyper-specfral analysis method and system employs support vector machine (SNM) techniques to solve discrimination problems by finding a function that generally fits the prescribed labels and stays as simple as possible, thereby guaranteeing good generalization error and preventing overfitting problems.
  • SNM support vector machine
  • the hyper-spectral analysis method for characterizing or distinguishing diverse elements within hyper-specfral images comprising the steps of extracting a plurality of patches of pixels from within the hyper-specfral images as being patches around pixels of the elements to be characterized or distinguished; computing the statistics of spectra for each patch of pixels, a first classifier from frequency- wise standard deviation of the spectra in each patch, a set of second classifiers from principal components of the spectral in each patch, and a classifier based on the output of the first classifier and at least one of the second classifiers; and characterizing or distinguishing the elements based on the output of at least one of the classifiers, preferably the combined classifier.
  • a hyper-spectral analysis system for characterizing or distinguishing diverse elements within hyper-spectral images, comprising an extracting module for extracting a plurality of patches of pixels from within the hyper-spectral images as being patches around pixels of the elements to be characterized or distinguished; a computing module for computing the statistics of spectra for each patch of pixels, a first classifier from frequency-wise standard deviation of the spectra in each patch, a set of second classifiers from principal components of the spectra in each patch, and a combined classifier based on the output of the first classifier and at least one of the second classifiers; and a characterization module for characterizing or distinguishing the elements based on the output of at least one of the classifiers.
  • step 2000 the hyper-spectral analysis system extracts the nuclei aggregates from each data cube. That is, the system extracts from each datacube groups of neighboring nuclei, collecting about 40 (in general overlapping) square patches
  • the hyper-spectral analysis system computes the statistics of the spectra for each nuclei aggregate.
  • the present system For each set of nuclei spectra l-** a * in the patch °> , the present system computes the mean spectrum and, for each spectral band, the standard deviation of the band, as well as the first 10 principal components. While the mean and the first few principal components of normal and abnormal nuclei were similar, the "frequency-wise" standard deviations and some of the higher order principal components showed some differences. [00227] In steps 2020 and 2030, the hyper-spectral system builds or constructs three classifiers: ⁇ , C 2 . ⁇ h e first classifier takes advantage of the differences in the standard deviations, whereas the second and third classifier use the principal components. The classifiers are similarly constructed and combined in a nonlinear voting manner.
  • the hyper-specfral analysis system computes the first classifier C ⁇ from the frequency-wise standard deviation of the spectra in each aggregate. Since the standard deviations are smooth functions of the frequency index, the present system employs LDB as described herein to find the features that "best" discriminate between the frequency-wise standard deviations of groups of normal nuclei and the frequency-wise standard deviations of groups of abnormal nuclei. The present system keeps the first four such features and projects orthogonally all the standard deviations onto these four features. In the four dimensional space onto which the standard deviations were projected, the present system employs a non-linear support vector machine (SVM) to separate the family of standard deviations corresponding to groups of normal nuclei from those corresponding to groups of abnormal nuclei.
  • SVM non-linear support vector machine
  • the present system optimizes over the parameters of the SVM by 10-fold cross validation which is in attempt to guarantee that the present system is not overfitting the data.
  • the present system weights the classifier by penalizing misclassifications of normal tissue more than misclassifications of abnormal tissue.
  • the hyper-spectral analysis system finds the "best" classifier Ci under these constraints. [00228]
  • the hyper-spectral analysis system computes a second set of classifiers from the principal component of the spectra in each aggregate. This is analogous to the construction of ? except that the present system applies it to each principal component. Let y-1, ..., 10 be the index for the first ten principal components.
  • the present system For each k, the present system considers the y ' -th principal component of each group i ⁇ ikik , to obtain 812 k-th principal components, some relative to normal nuclei spectra and some relative to abnormal nuclei spectra. Since the principal component is a smooth function of the frequency, the present system employs the LDB to find features which discriminate between the principal components of normal and abnormal groups. The present system keeps the first four features and projects the principal components onto the four dimensional space spanned by these first four features. In this 4-dimensional subspace, the present system employs SVMs, optimizing the parameters under cross- validation. For each k, the present system obtains a classifier, and a posteriori, the present system selects the k that gives the "best" result.
  • the 4 th and 6 th principal components provided the best result and was respectively denoted as classifiers C 2 and C 2 .
  • the hyper-spectral analysis system computes a classifier from the output of the two sets of classifiers to combine the classifiers i , and C 2 . ft is appreciated that these classifiers are "soft", in the sense that each of them returns a real number (mostly in [-1,1]) whose sign is determinative of the classification, i.e., normal or abnormal.
  • each classifier Before taking the "signum" of the classifier, the present system view each classifier as a map of a patch °,- to C ⁇ ( ⁇ -) ( C 2 (P,) and C 2 (J J ) respectively) with the real numbers (concentrated around the values -1 and +1).
  • the present system can view both sets of the classifiers as mapping each patch ?i onto the 3 -dimensional vector C 2 [P i ), C 2 (P.)) ⁇ h e p rese nt system then looks for a classifier in this space of outputs of the two sets of classifiers and utilizes SVMs to optimize the parameters under cross validation.
  • the hyper-spectral system characterizes the elements within the input image based on the results of the classifiers, preferably the combined classifier.
  • the hyper-spectral analysis method for characterizing or distinguishing diverse elements within hyper-spectral images comprises the steps of extracting a plurality of patches of pixels from within the hyper-spectral images as being patches around pixels of the elements to be characterized or distinguished; computing the statistics of selected spectral features for each patch of pixels, a first classifier from feature- wise standard deviation of the selected spectral features in each patch, a set of second classifiers from principal components of the spectral in each patch, and a classifier based on the output of the first classifier and at least one of the second classifiers; and characterizing or distinguishing the elements based on the output of at least one of the classifiers, preferably the combined classifier.
  • a computer readable medium comprises code for characterizing diverse elements within hyper-spectral images, the code comprises instructions for extracting a plurality of patches of pixels from within the hyper-specfral images as being patches around pixels of the elements to be characterized or distinguished; computing the statistics of selected spectral features for each patch of pixels, a first classifier from feature-wise standard deviation of the selected spectral features in each patch, a set of second classifiers from principal components of the spectral in each patch, and a classifier based on the output of the first classifier and at least one of the second classifiers; and characterizing or distinguishing the elements based on the output of at least one of the classifiers, preferably the combined classifier.
  • a hyper-spectral analysis system for characterizing or distinguishing diverse elements within hyper-specfral images, comprises an extracting module for extracting a plurality of patches of pixels from within the hyper-specfral images as being patches around pixels of the elements to be characterized or distinguished; a computing module for computing the statistics of spectra for each patch of pixels, a first classifier from frequency-wise standard deviation of the spectra in each patch, a set of second classifiers from principal components of the spectra in each patch, and a combined classifier based on the output of the first classifier and at least one of the second classifiers; and a characterization module for characterizing or distinguishing the elements based on the output of at least one of the classifiers.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Spectrometry And Color Measurement (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)

Abstract

La présente invention a trait à un procédé d'analyse hyper-spectrale pour la caractérisation ou la différenciation d'éléments hétérogènes dans des images hyper-spectrales. Une pluralité de carreaux de pixels dérivés à partir des images hyper-spectrales sont extraits comme étant des carreaux autour des pixels des éléments à caractériser ou différencier. Les statistiques de spectres pour chaque carreau de pixels sont calculées. Un premier classificateur est calculé à partir de la déviation standard en fréquence des spectres dans chaque carreau et un ensemble de deuxièmes classificateurs est calculé à partir des principales composantes des spectres dans chaque carreau. Un classificateur combiné est calculé à partir de la sortie du premier classificateur et d'au moins un parmi les deuxièmes classificateurs. Les éléments sont caractérisés ou différenciés selon la sortie d'au moins un parmi les classificateurs, de préférence le classificateur combiné.
PCT/US2005/007893 2004-03-06 2005-03-07 Systeme et procede pour l'analyse hyper-spectrale Ceased WO2005086890A2 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US55061504P 2004-03-06 2004-03-06
US60/550,615 2004-03-06
US10/832,684 US20050024640A1 (en) 1999-04-09 2004-04-26 System and method for encoded spatio-spectral information processing
US10/832,684 2004-04-26

Publications (2)

Publication Number Publication Date
WO2005086890A2 true WO2005086890A2 (fr) 2005-09-22
WO2005086890A3 WO2005086890A3 (fr) 2006-11-23

Family

ID=34976215

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2005/007893 Ceased WO2005086890A2 (fr) 2004-03-06 2005-03-07 Systeme et procede pour l'analyse hyper-spectrale

Country Status (1)

Country Link
WO (1) WO2005086890A2 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220414837A1 (en) * 2021-06-25 2022-12-29 Fujifilm Healthcare Corporation Medical image processing device, medical imaging apparatus, and noise reduction method for medical image
CN115918060A (zh) * 2020-05-28 2023-04-04 卡勒格锐研究有限公司 重建图像、特别是精确彩色图像的方法以及相关联的计算机程序、设备和系统
CN117495809A (zh) * 2023-11-06 2024-02-02 西安电子科技大学 基于背景纯化和光谱差异引导的高光谱异常检测方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5966460A (en) * 1997-03-03 1999-10-12 Xerox Corporation On-line learning for neural net-based character recognition systems
AU2002243602A1 (en) * 2001-01-19 2002-07-30 U.S. Army Medical Research And Materiel Command A method and apparatus for generating two-dimensional images of cervical tissue from three-dimensional hyperspectral cubes

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115918060A (zh) * 2020-05-28 2023-04-04 卡勒格锐研究有限公司 重建图像、特别是精确彩色图像的方法以及相关联的计算机程序、设备和系统
US20220414837A1 (en) * 2021-06-25 2022-12-29 Fujifilm Healthcare Corporation Medical image processing device, medical imaging apparatus, and noise reduction method for medical image
US12148127B2 (en) * 2021-06-25 2024-11-19 Fujifilm Corporation Medical image processing device, medical imaging apparatus, and noise reduction method for medical image
CN117495809A (zh) * 2023-11-06 2024-02-02 西安电子科技大学 基于背景纯化和光谱差异引导的高光谱异常检测方法

Also Published As

Publication number Publication date
WO2005086890A3 (fr) 2006-11-23

Similar Documents

Publication Publication Date Title
US7219086B2 (en) System and method for hyper-spectral analysis
US6859275B2 (en) System and method for encoded spatio-spectral information processing
US7652765B1 (en) Hyper-spectral imaging methods and devices
US7180588B2 (en) Devices and method for spectral measurements
US20050270528A1 (en) Hyper-spectral imaging methods and devices
US20040218172A1 (en) Application of spatial light modulators for new modalities in spectrometry and imaging
US7562057B2 (en) System and method for hyper-spectral analysis
US20030062422A1 (en) System and method for encoded spatio-spectral information processing
WO2005088264A1 (fr) Procedes et dispositifs d'imagerie hyperspectrale
Cao et al. A prism-mask system for multispectral video acquisition
US5483335A (en) Multiplex spectroscopy
US6392748B1 (en) Radiation filter, spectrometer and imager using a micro-mirror array
US5479258A (en) Image multispectral sensing
WO2023026742A1 (fr) Procédé d'acquisition d'image de colorant, dispositif d'acquisition d'image de colorant et programme d'acquisition d'image de colorant
WO2022018527A1 (fr) Dispositif multi-spectral
Balas et al. Multi/hyper-spectral imaging
Saragadam et al. Programmable spectrometry: per-pixel material classification using learned spectral filters
WO2005086818A2 (fr) Dispositifs et procede de mesures spectrales
WO2005086890A2 (fr) Systeme et procede pour l'analyse hyper-spectrale
WO2006034223A2 (fr) Systeme et methode d'analyse hyper-spectrale
Graff et al. Real-time matched-filter imaging for chemical detection using a DMD-based programmable filter
Graff et al. Toward real-time spectral imaging for chemical detection with a digital micromirror device-based programmable spectral filter
Majumder et al. High-definition (HD) snapshot diffractive computational spectral imaging and inferencing
Lu et al. A programmable optical filter with arbitrary transmittance for fast spectroscopic imaging and spectral data post-processing
Luo et al. Optical unmixing using programmable spectral source based on DMD

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): BW GH GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Country of ref document: DE

122 Ep: pct application non-entry in european phase