Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/73—Deblurring; Sharpening
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10056—Microscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10064—Fluorescence image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30096—Tumor; Lesion
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
  • G16B25/00—ICT specially adapted for hybridisation; ICT specially adapted for gene or protein expression
  • G16B25/10—Gene or protein expression profiling; Expression-ratio estimation or normalisation

Definitions

• This present invention also provides a new approach utilizing spatial density generated from the readout density of the spatial transcriptomics assays as the morphology images for cell soma and performing cell segmentation on the generated readout density map.
• a software module configured to retrieve a 3D scan image of a biological sample in High Dynamic Range (HDR) mode, wherein the biological sample is labeled with one of a plurality of morphology markers;
• a software module configured to reduce the 3D scan image to a 2D image, the optimal focus region in z-slice is obtained; or utilize the entire 3D stack volume
• a software module configured to retrieve a readout density map from transcriptomic assays of the biological sample; and
• a software module configured to perform subcellular segmentation on the biological sample based on the plurality of morphology markers or the readout density map
• the morphology markers comprise fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes.
• the images are microscope-derived images, comprising images from optical microscopes, electron microscopes, or scanning probe microscopes.
• the images can comprise images from spatial molecular imagers.
• the images are applied to a bleaching correction.
• the transcriptomic assays comprise gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA-seq).
• RNA-seq RNA sequencing
• RT-PCR reverse transcription polymerase chain reaction
• scRNA-seq single-cell RNA sequencing
• the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than at least 200 images of the biological sample.
• the biological sample is obtained at least in part of one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, and implanted tissue sampling.
• the biological WSGR Docket No. 31353-709.601 sample comprises cells or tissues.
• the cells comprise primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells.
• subcellular segmentation comprises nucleic segmentation, cytoplasm segmentation, or extra-cellular segmentation.
• subcellular segmentation comprises training a machine learning algorithm and/or applying a machine learning algorithm.
• a software module configured to retrieve a 3D scan image of a biological sample in High Dynamic Range (HDR) mode, wherein the biological sample is labeled with one of a plurality of morphology markers;
• b.1 optionally a software module configured to reduce the 3D scan image to a 2D image, retaining the optimal focus regions from various z slices, using techniques such as maximum intensity projection, focus stacking and extended depth of field;
• (b.2) alternatively a software module configured to deconvolve the 2D images to improve the sharpness and clarity of the images while maintaining the z position of those images in their 3D scan image stack
• b.3 alternatively a software module configured to use each deconvolved 2D slice forming 3D volume stack
• the morphology markers comprise fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes.
• the images are microscope-derived images, comprising images from optical microscopes, electron microscopes, or scanning probe microscopes.
• the images can comprise images from spatial molecular imagers.
• the images are applied to a bleaching correction.
• the transcriptomic assays comprise gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single- cell RNA sequencing (scRNA-seq).
• the software module configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least WSGR Docket No.
• the biological sample is obtained at least in part of one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, and implanted tissue sampling.
• the biological sample comprises cells or tissues.
• the cells comprise primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells.
• the morphology markers comprise fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes.
• the images are microscope-derived images, comprising images from optical microscopes, electron microscopes, or scanning probe microscopes.
• the images can comprise images from spatial molecular imagers.
• the images are applied to a bleaching correction.
• the transcriptomic assays comprise gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA-seq).
• the software module is configured to retrieve at least 1 image, at least 3 WSGR Docket No.
• the biological sample is obtained at least in part of one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, and implanted tissue sampling.
• the biological sample comprises cells or tissues.
• the cells comprise primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells.
• subcellular segmentation comprises nucleic segmentation, cytoplasm segmentation, or extra-cellular segmentation.
• subcellular segmentation comprises training a machine learning algorithm and/or applying a machine learning algorithm.
• the systems, media, and methods disclosed herein comprise combining protein images expressing similar structures and/or cell types into multi-channel input morphology images.
• the systems, media, and methods disclosed herein comprise segmentation of extra-cellular processes.
• Various properties and statistics can be computed on per cell bases to enable researchers to study specif ic structures or regions of interest. Those properties and statistics allow the establishment of downstream analysis that use cell locations and per-cell fluorescence and shape properties as generated during segmentation. It also serves as a baseline performance across multiple sample types and serves as metrics for ground-truth-free segmentation evaluation of new datasets. In some embodiments, the evaluation of cell segmentation performance is based on direct comparison.
• the evaluation of cell segmentation performance is achieved by projecting the statistics of new datasets to the embedding space created by the statistics of reference datasets of similar sample types.
• image preprocessing is performed on raw images before feeding into at least one cell segmentation module.
• image preprocessing is based on conventional image processing techniques such as denoising, deblurring and image enhancements.
• various denoising algorithms can be performed, including filter based (Wiener, Gaussian, Median), using wavelet transforms, total variation (TV) WSGR Docket No. 31353-709.601 denoising, and deep learning-based approaches.
• various deblurring algorithms can be performed, including the nearest-neighbor deconvolution, Richardson-Lucy deconvolution and total variation (TV) regularization.
• various image enhancement algorithms can be performed, including histogram equalization and contrast limited adaptive histogram equalization (CLAHE), contrast stretching, gamma correction and unsharp masking.
• CLAHE contrast limited adaptive histogram equalization
• cell segmentation is based on algorithm comprising Cellpose Deep Learning algorithm, adaptive thresholding, Watershed algorithm, thresholding, region- based segmentation, active contours, Convolutional Neural Networks (CNN), Level Set Methods, Graph- based algorithms, fuzzy c-means clustering, Deep watershed, image morphology, or Markov Random Fields.
• nuclei segmentation, cytoplasm segmentation, protrusion segmentation and extra-cellular segmentation are based on the same algorithm. In some embodiments, nuclei segmentation, cytoplasm segmentation, protrusion segmentation and extra-cellular segmentation are based on different algorithms.
• image segmentation software based on machine learning (ML) algorithms may be applied to create cell boundaries from fluorescent images of protein assays.
• the protein assays may comprise protein antibodies binding to membrane proteins.
• ML algorithms applied to image segmentation may comprise semantic segmentation, instance segmentation, or generative networks for segmentation.
• image segmentation software may comprise ImageJ, CellProfiller, Cellpose, Ilastik, or QuPath, or any combination thereof .
• applications and use cases include, by way of non -limiting examples, discovering and mapping cell types and cell states, phenotypes of tissue microenvironment, differential expression of cell type based on spatial context, quantify ing subcellular expression, and spatially resolved biomarker identification. BRIEF DESCRIPTION OF THE DRAWINGS [0015]
• the patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
• FIG.1 shows a non-limiting example of a computing device; in this case, a device with WSGR Docket No.
• FIG.2 shows a non-limiting example of a web/mobile application provision system; in this case, a system providing browser-based and/or native mobile user interfaces;
• FIG.3 shows a non-limiting example of a cloud-based web/mobile application provision system; in this case, a system comprising an elastically load balanced, auto-scaling web server and application server resources as well as synchronously replicated databases;
• FIGS.4A-4C show a non-limiting example of Multi-Modal Cell Acquisition and Segmentation System, FIG.4A illustrates exemplary protein acquisition and 3D morphology sample collection, FIG.4B illustrates exemplary RNA acquisition and detection, and FIG.4C illustrates exemplary Sub-Cellular cell segmentation;
• FIG.5 shows a non-limiting example of High Dynamic Range (HDR) imaging method;
• FIG.6 shows a non-limiting example of two-
• FIG.17B shows a non-limiting example of various 5-Channel model outputs
• FIGS.18A-18D show a non-limiting example of dense neuron cells in human brain, FIG.18A shows neuron cells with nuclei markers, FIG.18B shows neuron cells with membrane markers, FIG.18C shows astrocytes with GFAP markers, and FIG.18D shows nuclei with DAPI markers;
• FIG.19 shows a non-limiting example of extra-cellular process segmentation;
• FIGS.20A-20C show a non-limiting example of results from cell segmentation, FIG.
• FIG. 20A shows a non-limiting example of a segmentation results overlay of a cell and extracellular processes in a human brain
• FIG.20B shows a non-limiting example of corresponding cell segmentation masks
• FIG.20C shows a non-limiting example of detection of extracellular cell segmentation masks
• FIG.21 shows a non-limiting example of cell segmentation output providing sub- cellular and extra-cellular masks and statistics
• FIG.22 illustrates a non-limiting example of an objective lens cone angle for a blurring function
• FIG.23 illustrates a non-limiting example of a 3D segmentation pipeline
• FIG.24 illustrates image preprocessing techniques such as image sharpening and enhancement
• FIG.25 illustrates a non-limiting example of an image acquired using High Definition Range (HDR) settings
• FIG.26 illustrates a non-limiting example of output produced by the nearest neighbor (NN) deconvolution method
• FIG.27 illustrates a non-limiting example of
• Described herein, in certain embodiments, are systems comprising at least one processor and instructions executable by the at least one processor to provide a multi-modal segmentation application comprising: (a) a software module configured to retrieve a 3D scan image of a biological sample in High Dynamic Range (HDR) mode, wherein the biological sample is labeled with one of a plurality of morphology markers; (b) a software module configured to reduce the 3D scan image to a 2D image and obtain an optimal focus region in a z slice, or utilize the entire 3D stack where each 2D z-slice has been deconvolved to enhance features and reduce blur; (c) a software module configured to retrieve a readout density map from transcriptomic assays of the biological sample; and (d) a software module configured to perform subcellular segmentation on the biological sample based on the plurality of morphology markers or the readout density map.
• HDR High Dynamic Range
• the morphology markers can comprise one or more fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes, or any combination thereof .
• the images can comprise microscope-derived images.
• the microscope-derived images can comprise images from optical microscopes, electron microscopes, or scanning probe microscopes.
• the images can comprise images from spatial molecular imagers.
• a bleaching correction can be applied to the images.
• the transcriptomic assays can comprise one or more of gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof.
• RNA-seq RNA sequencing
• RT-PCR reverse transcription polymerase chain reaction
• scRNA-seq single-cell RNA sequencing
• the software module can be configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than at least 200 images of the biological sample.
• the biological sample can be obtained at least in part of from one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, or implanted tissue sampling, WSGR Docket No. 31353-709.601 or any combination thereof.
• the biological sample can comprise cells or tissues.
• the cells can comprise one or more of primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells, or any combination thereof .
• subcellular segmentation can comprise one or more of nucleic segmentation, cytoplasm segmentation, or extra-cellular segmentation, or any combination thereof .
• subcellular segmentation can comprise training a machine learning algorithm, or applying a machine learning algorithm, or both.
• the morphology markers can comprise one or more of fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes, or any combination thereof.
• the images can be microscope-derived images.
• microscope-derived images can comprise images from optical microscopes, electron microscopes, or scanning probe microscopes, or any combination thereof.
• the images can comprise images from spatial molecular imagers.
• a bleaching correction can be applied to the images.
• the transcriptomic assays can comprise one or more of gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof .
• the software module can be configured to retrieve at least 1 image, at least 3 images, at least 5 images, at WSGR Docket No.
• 31353-709.601 least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than at least 200 images of the biological sample.
• the biological sample can be obtained at least in part from one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, and implanted tissue sampling, or any combination thereof.
• the biological sample can comprise cells or tissues.
• the cells can comprise primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells, or any combination thereof .
• the morphology markers can comprise one or more of fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo- cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes, or any combination thereof.
• the images can be microscope-derived images.
• the microscope-derived images can comprise images from optical microscopes, electron microscopes, or scanning probe microscopes, or any combination thereof .
• the images can comprise images from spatial molecular imagers.
• a bleaching WSGR Docket No. 31353-709.601 correction can be applied to the images.
• the transcriptomic assays can comprise gene expression assays comprising one or more of fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA- seq), or any combination thereof .
• RNA-seq RNA sequencing
• RT-PCR reverse transcription polymerase chain reaction
• scRNA- seq single-cell RNA sequencing
• the software module can be configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than at least 200 images of the biological sample.
• the biological sample can be obtained at least in part from one or more of biopsy collection, surgical resection, xenograft, animal model, fine needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, neoplastic tissue sampling, malignant tissue sampling, diseased tissue sampling, and implanted tissue sampling, or any combination thereof.
• the biological sample can comprise cells or tissues.
• the cells can comprise one or more of primary cells, stem cells, immune cells, carcinoma cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or neoplastic cells, or any combination thereof .
• subcellular segmentation can comprise one or more of nucleic segmentation, cytoplasm segmentation, or extra-cellular segmentation, or any combination thereof .
• subcellular segmentation can comprise training a machine learning algorithm, applying a machine learning algorithm, or both.
• Multi-Modal Cell Acquisition and Segmentation System the systems, media, and methods disclosed herein can comprise combining morphology staining with both the proteomic and transcriptomic data.
• the systems, media, and methods disclosed herein can further comprise receiving information regarding cell structures.
• cell structures can include nuclei, cytoplasm, or membrane regions, or any combination thereof .
• the systems, media, and methods disclosed herein can comprise combining protein images expressing similar structures into multi-channel input morphology images.
• the systems, media, and methods disclosed herein can comprise combining protein images expressing similar cell types into multi-channel input morphology images.
• the systems, media, and methods disclosed herein can comprise combining protein images expressing similar cell types and similar structures into multi-channel input morphology images.
• the systems, media, and methods disclosed herein can comprise a multi-step process of segmentation.
• the multi-step process of segmentation can comprise segmenting regions of nuclei.
• the multi-step process of segmentation can comprise segmenting regions of cytoplasm.
• the multi-step process of segmentation can comprise segmenting regions of membrane.
• the systems, media, and methods disclosed herein can comprise segmenting of extra-cellular objects to segment regions.
• the WSGR Docket No. 31353-709.601 segment regions can be segment regions of an organ or tissue.
• the segment regions can be segment regions of the brain.
• the segment regions can be distal or disconnected to the soma, or both.
• the systems, media, and methods disclosed herein can comprise combining images expressing similar structures into one or more input morphology images.
• the systems, media, and methods disclosed herein can comprise combining images expressing similar cell types into one or more input morphology images.
• the systems, media, and methods disclosed herein can comprise combining images expressing similar structures into one or more input morphology images and cell types into one or more input morphology images.
• the input morphology images can be multi-channel input morphology images.
• the systems, media, and methods disclosed herein can comprise combining images expressing similar structures into 5-channel input morphology images. In some embodiments, the systems, media, and methods disclosed herein can comprise combining images expressing similar cell types into 5-channel input morphology images. In some embodiments, the systems, media, and methods disclosed herein can comprise combining images expressing similar structures into 5-channel input morphology images and cell types into 5-channel input morphology images. [0056] In some embodiments, the systems, media, and methods disclosed herein can comprise combining one or more protein images.
• the number of protein images combined can be about 1, about 2, about 3, about 4, about 5, about 10, about 20, about 30, about 40, about 50, about 60, about 70, about 80, about 90, about 100, about 120, about 140, about 160, about 180, about 200, or more than, a about 200 protein images.
• the systems, media, and methods disclosed herein can comprise processing an additional input channel of raw RNA spot readout density to supplement missing areas of signal from the morphology markers. [0057]
• the systems, media, and methods disclosed herein can comprise segmentation of extra-cellular processes.
• the systems, media, and methods disclosed herein can further comprise labeling one or more images with a unique id for each cell.
• the systems, media, and methods disclosed herein can further comprise labeling one or more images with a unique id for each one of the connected processes. In some embodiments, the systems, media, and methods disclosed herein can further comprise labeling one or more images with a unique id for each cell and each one of the connected processes. The systems, media, and methods disclosed herein can further comprise identifying a "compartment label" for a different compartment of the area for each WSGR Docket No. 31353-709.601 cell. [0058] As a non-limiting example, a multi-modal cell acquisition and segmentation system is illustrated in FIGS.4A-4C. For example, protein acquisition and 3D morphology samples were collected as illustrated in FIG.4A.
• Protein samples and a morphology marker for example, a photo-cleavable marker, were incubated and scanned into 3D images in High Dynamic Range (HDR) mode.
• markers were cleared using for example UV illumination and chemical wash in the tissue.
• UV illumination and chemical wash in the tissue For example, repeat re-staining and clearing of markers using UV illumination and chemical wash in the tissue were performed.
• RNA acquisition and detection were performed as illustrated in FIG.4B.
• RNA spots were detected.
• markers were cleared using UV illumination and wash in the samples.
• the systems, media, and methods disclosed herein can comprise retrieving images from one or more tissue scans.
• the one or more tissue scans can comprise one or more scans of one or more tissues stained with morphology markers. In some embodiments, one or more tissue scans can comprise one or more scans of one or more tissues incubated with protein reporters. In some embodiments, one or more tissue scans can comprise one or more scans of one or more tissues incubated with RNA reporters. In some embodiments, the systems, media, and methods disclosed herein can comprise removing one or more reporters from the one or more tissue samples. In some embodiments, one or more reporters can be reduced in one or more tissue samples. In some embodiments, one or more reporters can be bleached, cleaved, or both in one or more tissue samples.
• reporters can be reduced in one or more tissue samples by applying one or more of UV light, heat, proteases, endonuclease, nucleases, esterases, ribonucleases, RNase A, RNase Tl, RNase H, disulfate bond reducing agents (e.g., dithiothreitol, tris(2- carboxyethyl)phosphine), salt buffer, alkali, hydrogen bond destabilization solvents (e.g., formamide, DMSO), or any combination thereof.
• UV light heat
• proteases endonuclease
• nucleases nucleases
• esterases ribonucleases
• RNase A RNase Tl
• RNase H disulfate bond reducing agents
• salt buffer e.g., dithiothreitol, tris(2- carboxyethyl)phosphine
• salt buffer e.g., alkali
• hydrogen bond destabilization solvents e.
• cell segmentation processes included performing, for example, background subtraction, normalization, segmentation for regions to nuclei, regions to cytoplasm or regions to membrane , or any combination thereof.
• different compartments of the area of each cell can be identified by a unique "compartment label.”
• various properties WSGR Docket No. 31353-709.601 and statistics can be computed for each cell.
• various properties and statistics computed for each cell can provide information concerning specific structures or regions of interest, or both.
• the systems, media, and methods disclosed herein can comprise retrieving images of biological samples.
• the retrieved images of biological examples can be labeled with morphology markers.
• the morphology markers can comprise one or more of fluorescent dyes, nuclear stains, fluorescently labeled antibodies, immunohistochemistry (IHC) stains, photo-cleavable morphology markers, genetically encoded tags, magnetic resonance image (MRI) contrast agents, or nucleic acid probes, or any combination thereof .
• the systems, media, and methods disclosed herein can comprise generating a readout density map using transcriptomic assays of biological samples.
• the density map can be a heatmap, a choropleth map, a kernel density map, a dot density map, a contour map, or a proportional symbol map.
• the heatmap can further comprise a graphical representation of data where the values are depicted using colors.
• the heatmap can be used to visualize the density or intensity of a particular phenomenon across a two-dimensional space.
• the two-dimensional space can be geographical areas, images, or grids, or any combination thereof .
• the density map can show the density or concentration of a particular attribute or event within a given area.
• the density map can provide visual information about the distribution and intensity of data points or events across a spatial domain.
• the transcriptomic assays can comprise one or more of gene expression assays with fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), cap analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof .
• RNA-seq RNA sequencing
• RT-PCR reverse transcription polymerase chain reaction
• scRNA-seq single-cell RNA sequencing
• the systems, media, and methods disclosed herein can comprise retrieving images of biological samples.
• the systems, media, and methods disclosed herein can comprise retrieving images of spatially resolved high-plex gene expression data from tissue.
• the systems, media, and methods disclosed herein can comprise retrieving images of spatially resolved high-plex protein data from tissue.
• the systems, media, and methods disclosed herein can comprise retrieving images of RNA assays.
• the systems, media, and methods disclosed herein can comprise retrieving images of RNA assays to profile an entire transcriptome.
• the systems, media, and methods disclosed herein can comprise retrieving images of RNA assays to profile an entire transcriptome from tissues on a single formalin-fixed paraffin-embedded (FFPE) sample.
• FFPE formalin-fixed paraffin-embedded
• the systems, media, and methods disclosed herein can comprise retrieving images of RNA assays to profile an entire transcriptome from tissues on a fresh frozen (FF) sample slide.
• the systems, media, and methods disclosed herein can comprise retrieving images of protein assays to generate quantitative analysis, spatial analysis, or both, of multiple proteins.
• the quantitative analysis, spatial analysis, or both, of multiple proteins can be generated from a single FFPE or FF sample slide.
• FFPE or FF tissue sections may be stained with barcoded in-situ hybridization probes that bind to endogenous mRNA transcripts.
• a user may select regions of the interest (ROI) to profile.
• ROI regions of the interest
• FIG.1 a block diagram is shown depicting an exemplary machine that includes a computer system 100 (e.g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and/or methodologies for static code scheduling of the present disclosure.
• the components herein are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments.
• WSGR Docket No. 31353-709.601 Computer system 100 may include one or more processors 101, a memory 103, and a storage 108 that communicate with each other, and with other components, via a bus 140.
• the bus 140 may also link a display 132, one or more input devices 133 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 134, one or more storage devices 135, and various tangible storage media 136. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 140. For instance, the various tangible storage media 136 can interface with the bus 140 via storage medium interface 126.
• Computer system 100 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.
• ICs integrated circuits
• PCBs printed circuit boards
• mobile handheld devices such as mobile telephones or PDAs
• laptop or notebook computers distributed computer systems, computing grids, or servers.
• Computer system 100 includes one or more processor(s) 101 (e.g., central processing units (CPUs), general purpose graphics processing units (GPGPUs), or quantum processing units (QPUs)) that carry out functions.
• processor(s) 101 optionally contains a cache memory unit 102 for temporary local storage of instructions, data, or computer addresses.
• Processor(s) 101 are configured to assist in execution of computer readable instructions.
• Computer system 100 may provide functionality for the components depicted in FIG.1 as a result of the processor(s) 101 executing non-transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 103, storage 108, storage devices 135, and/or storage medium 136.
• the computer-readable media may store software that implements particular embodiments, and processor(s) 101 may execute the software.
• Memory 103 may read the software from one or more other computer-readable media (such as mass storage device(s) 135, 136) or from one or more other sources through a suitable interface, such as network interface 120.
• the software may cause processor(s) 101 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 103 and modifying the data structures as directed by the software.
• the memory 103 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 104) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase-change random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 105), and any combinations thereof.
• ROM 105 may act to communicate data and instructions unidirectionally to processor(s) 101
• RAM 104 may act to communicate data and instructions bidirectionally with processor(s) 101.
• ROM 105 and RAM 104 may include WSGR Docket No. 31353-709.601 any suitable tangible computer-readable media described below.
• a basic input/output system 106 including basic routines that help to transfer information between elements within computer system 100, such as during start-up, may be stored in the memory 103.
• Fixed storage 108 is connected bidirectionally to processor(s) 101, optionally through storage control unit 107. Fixed storage 108 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 108 may be used to store operating system 109, executable(s) 110, data 111, applications 112 (application programs), and the like. Storage 108 can also include an optical disk drive, a solid- state memory device (e.g., flash- based systems), or a combination of any of the above.
• a solid- state memory device e.g., flash- based systems
• storage device(s) 135 may be removably interfaced with computer system 100 (e.g., via an external port connector (not shown)) via a storage device interface 125.
• storage device(s) 135 and an associated machine-readable medium may provide non-volatile and/or volatile storage of machine-readable instructions, data structures, program modules, and/or other data for the computer system 100.
• software may reside, completely or partially, within a machine-readable medium on storage device(s) 135.
• software may reside, completely or partially, within processor(s) 101.
• Bus 140 connects a wide variety of subsystems.
• Bus 140 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.
• Computer system 100 may also include an input device 133.
• a user of computer system 100 may enter commands and/or other information into computer system 100 via input device(s) 133.
• Examples of an input device(s) 133 include, but are not limited to, an alpha- numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input WSGR Docket No. 31353-709.601 device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof.
• the input device is a Kinect, Leap Motion, or the like.
• Input device(s) 133 may be interfaced to bus 140 via any of a variety of input interfaces 123 (e.g., input interface 123) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.
• input interfaces 123 e.g., input interface 123
• computer system 100 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 130. Communications to and from computer system 100 may be sent through network interface 120.
• network interface 120 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 130, and computer system 100 may store the incoming communications in memory 103 for processing.
• Computer system 100 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 103 and communicated to network 130 from network interface 120.
• Processor(s) 101 may access these communication packets stored in memory 103 for processing.
• Examples of the network interface 120 include, but are not limited to, a network interface card, a modem, and any combination thereof.
• Examples of a network 130 or network segment 130 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof.
• a network, such as network 130 may employ a wired and/or a wireless mode of communication. In general, any network topology may be used.
• Information and data can be displayed through a display 132.
• the display is a video projector.
• the display is a head-mounted display (HMD) such as a VR headset.
• HMD head-mounted display
• suitable VR headsets include, by way of non-limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOYE VR, Zeiss VR One, Avegant Glyph, Freefly VR headset, and the like.
• the display is a combination of devices such as those disclosed herein.
• computer system 100 may include one or more other peripheral output devices 134 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 140 via an output interface 124. Examples of an output interface 124 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof. [0079] In addition or as an alternative, computer system 100 may provide functionality as a result of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein.
• references to software in this disclosure may encompass logic, and reference to logic may encompass software.
• reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate.
• the present disclosure encompasses any suitable combination of hardware, software, or both.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, WSGR Docket No. 31353-709.601 microcontroller, or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• the steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two.
• a software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
• An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an ASIC.
• the ASIC may reside in a user terminal.
• the processor and the storage medium may reside as discrete components in a user terminal.
• suitable computing devices include, by way of non-limiting examples, distributed computing systems, cloud computing platforms, server clusters, server computers, desktop computers, laptop computers, notebook computers, sub- notebook computers, netbook computers, netpad computers, handheld computers, Internet appliances, mobile smartphones, and tablet computers.
• suitable server operating systems include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD ® , Linux, Apple ® Mac OS X Server ® , Oracle ® Solaris ® , Windows Server ® , and Novell ® NetWare ® .
• suitable personal computer operating systems include, by way of non-limiting examples, Microsoft ® Windows ® , Apple ® Mac OS X ® , UNIX ® , and UNIX-like operating systems such as GNU/Linux ® .
• the operating system is provided by cloud computing.
• Non-transitory computer readable storage medium the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of an optionally networked computing device.
• a computer readable storage medium is a tangible component of a computing device. In still further embodiments, a computer readable storage medium is optionally removable from a computing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media. Computer program [0086] In some embodiments, the platforms, systems, media, and methods disclosed herein include at least one computer program, or use of the same.
• a computer program includes a sequence of instructions, executable by one or more processor(s) of the computing device's CPU, written to perform a specified task.
• Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those of skill in the art will recognize that a computer program may be written in various versions of various languages. [0087] The functionality of the computer readable instructions may be combined or distributed as desired in various environments.
• a computer program comprises one sequence of instructions.
• a computer program comprises a plurality of sequences of instructions.
• a computer program is provided from one location.
• a computer program is provided from a plurality of locations.
• a computer program includes one or more software modules.
• a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or WSGR Docket No. 31353-709.601 more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.
• Web application [0088]
• a computer program can include a web application.
• a web application in various embodiments, utilizes one or more software frameworks and one or more database systems.
• a web application is created upon a software framework such as Microsoft ® .NET or Ruby on Rails (RoR).
• a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non- relational, object oriented, associative, XML, and document-oriented database systems.
• suitable relational database systems include, by way of non-limiting examples, Microsoft ® SQL Server, mySQLTM, and Oracle ® .
• a web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof.
• a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or eXtensible Markup Language (XML).
• a web application can be written to some extent in a presentation definition language such as Cascading Style Sheets (CSS).
• a web application can be written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash ® ActionScript, JavaScript, or Silverlight ® .
• AJAX Asynchronous JavaScript and XML
• Flash ® ActionScript JavaScript
• JavaScript JavaScript
• Silverlight ® Silverlight ®
• a web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion ® , Perl, JavaTM, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), PythonTM, Ruby, Tel, Smalltalk, WebDNA ® , or Groovy.
• a web application can be written to some extent in a database query language such as Structured Query Language (SQL).
• SQL Structured Query Language
• a web application integrates enterprise server products such as IBM ® Lotus Domino ® .
• a web application includes a media player element.
• a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe ® Flash ® , HTML 5, Apple ® QuickTime ® , Microsoft ® Silverlight ® , JavaTM, and Unity ® .
• an application provision system comprises one or more databases 200 accessed by a relational database management system (RDBMS) 210.
• RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle WSGR Docket No. 31353-709.601 Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, and the like.
• the application provision system further comprises one or more application servers 220 (such as Java servers, .NET servers, PHP servers, and the like) and one or more web servers 230 (such as Apache, IIS, GWS and the like).
• the web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 240.
• APIs app application programming interfaces
• an application provision system alternatively has a distributed, cloud-based architecture 300 and comprises elastically load balanced, auto-scaling web server resources 310 and application server resources 320 as well synchronously replicated databases 330.
• a computer program can include a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in.
• a compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C, COBOL, Delphi, Eiffel, JavaTM, Lisp, PythonTM, Visual Basic, and VB .NET, or combinations thereof. Compilation is often performed, at least in part, to create an executable program.
• a computer program can include one or more executable complied applications.
• Software modules [0092]
• the platforms, systems, media, and methods disclosed herein can include software, server, and/or database modules, or use of the same.
• software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art.
• the software modules disclosed herein are implemented in a multitude of ways.
• a software module can comprise a file, a section of code, a programming object, a programming structure, a distributed computing resource, a cloud computing resource, or combinations thereof.
• a software module can comprise a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, a plurality of distributed computing resources, a plurality of cloud computing resources, or WSGR Docket No. 31353-709.601 combinations thereof.
• the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, a standalone application, and a distributed or cloud computing application.
• software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine.
• software modules are hosted on a distributed computing platform such as a cloud computing platform.
• software modules are hosted on one or more machines in one location.
• software modules are hosted on one or more machines in more than one location.
• Databases [0093]
• the systems, media, and methods disclosed herein can include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of user information, study information, slide information, field of view (FoV) information, flow cell information, image information, genomic information, transcriptomic information, and proteomic information.
• suitable databases include, by way of non - limiting examples, relational databases, non-relational databases, object-oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, document-oriented databases, and graph databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, and MongoDB.
• a database can be Internet-based.
• a database is web-based.
• a database can be cloud computing-based.
• a database can be a distributed database.
• a database can be based on one or more local computer storage devices.
• data stored on databases can include, biological image data.
• biological image data can include, in some non-limiting examples, microscopy images (e.g., micrographs) of formalin fixed paraffin embedded (FFPE) samples of cells, , microscopy images (e.g., micrographs) of formalin fixed paraffin embedded (FFPE) samples of tissues, microscopy images (e.g., micrographs) of fresh frozen (FF) samples of cells, or microscopy images (e.g., micrographs) of fresh frozen (FF) samples of tissues, or any combination thereof.
• data from a single slide can be split into two datasets.
• the two datasets can comprise an RNA Assays dataset and a Protein Assays dataset.
• data from a single slide can be combined into one WSGR Docket No. 31353-709.601 dataset.
• the one dataset can include both RNA assays data and Protein Assays data.
• the image data can be two-dimensional image data.
• the image data can be three-dimensional image data.
• the data includes, by way of non-limiting examples, “-omics” data such as genomic data, proteomic data, metabolomic data, metagenomic data, phenomic data, or transcriptomic data, or any combination thereof.
• the tissue can be incubated with RNA reporters.
• the systems, media, and methods disclosed herein can comprise removing reporters from the tissue.
• reporters can be bleached.
• reporters can be cleaved.
• reporters can be bleached, cleaved, or both, by applying one or more of UV light, heat, proteases, endonuclease, nucleases, esterases, ribonucleases, RNase A, RNase Tl, RNase H, disulfate bond reducing agents (e.g., dithiothreitol, tris(2- carboxyethyl)phosphine), salt buffer, alkali, hydrogen bond destabilization solvents (e.g., formamide, DMSO) or any combination thereof.
• the systems, media, and methods disclosed herein can comprise retrieving images.
• the images can comprise microscope-derived images.
• the microscope-derived images can comprise images from optical microscopes, electron microscopes, or scanning probe microscopes, or any combination thereof. In some embodiments, the images can comprise images from spatial molecular imagers.
• morphology signals are acquired.
• RNA protein signals are acquired. In some embodiments, both morphology signals and RNA protein signals are acquired. In some embodiments, the morphology and RNA protein signals are WSGR Docket No. 31353-709.601 acquired using the high dynamic range imaging (HDR) mode. In some embodiments, the signals are acquired using HDR to improve the dynamic range of the signal.
• HDR high dynamic range imaging
• Image I was, in one non-limiting example, obtained with exposure of 0.0125 second with details in the bright area, while image 2 was obtained with exposure of 0.1 second with details in the dark area.
• the result image can be generated by dividing an accumulator by a counter.
• the final intensity scale was equivalent to exposure with 0.0125 second, 1/8 of the nominal exposure time.
• the systems, media, and methods disclosed herein can comprise combining protein images expressing similar structures or cell types, or both, into multi- channel input morphology images.
• the systems, media, and metho ds disclosed herein can comprise adding an additional input channel.
• the additional input channel can comprise raw RNA spot density input to supplement missing signal from the morphology markers.
• a spatial molecular imager can be used to image nuclei and DAPI for staining.
• the spatial molecular imager can be a CosMxTM instrument.
• the systems, media, and methods disclosed herein can comprise biological samples labeled with photocleavable morphology markers.
• the DAPI channel acquisition for one FOV can cause cleaving or bleaching of reporters, or both, in adjacent FOVs. In some embodiments, order of image or channel acquisition, or both, is determined.
• order of image or channel acquisition can be determined to prevent visible and unrecoverable signal loss along the edges of the FOVs, particularly in contiguous scans.
• the systems, media, and methods disclosed herein can comprise combining multi-channel input morphology images.
• the morphology images can be used for different markers expressing nuclei, cytoplasm, or cell membrane, or any combination thereof .
• the 5-channel morphology images used for segmentation can comprise Blue (B), Green (G), Yellow (Y), Red (R) and UV (U).
• the systems, media, and methods disclosed herein further comprise a multi-step image acquisition process.
• the multi-step image acquisition process can be performed to avoid signal loss.
• the systems, media, and methods disclosed herein can comprise a software module configured to reduce the 3D scan image to a 2D image. In some cases, reducing the 3D scan image to a 2D image can obtain the optimal focus region in a z-slice.
• the systems, media, WSGR Docket No. 31353-709.601 and methods disclosed herein can comprise a software module configured to deconvolve the 2D images.
• the deconvolution of the 2D images can provide the optimal focus area for the images while retaining the z position of the images in their 3D scan image stack In some embodiments, the deconvolution of the 2D images can provide the optimal clarity for the images while retaining the z position of the images in their 3D scan image stack.
• two step image acquisition P01 and P02 were performed to avoid signal loss from DAPI acquisition using UV light.
• Step P01 the system acquired the four channels of B, G, Y, R of z-stacks of 3D High Dynamic Range (HDR) across the tissue for FOVs of choice.
• HDR High Dynamic Range
• Step P02 of FIG.6 for example, the system separated DAPI from the other two morphology channels, for registration with the previous step P01 to a 3D coordinate space.
• the FOV locations were revisited to acquire a new Z-stack with DAPI and two additional morphology channels, for example B and Y.
• the B and Y morphology channels in one non-limiting example, were required to register the P01 and P02 z-stacks to the same 3D coordinate space, as shown in Step Register of FIG.6.
• the stacks were aligned, they were combined to generate a complete 5-channel Z-stack as shown in Step P99 of FIG.6.
• protein images can be acquired for all FOVs followed by the RNA acquisition.
• images can be registered to the same 3D coordinate space using fiducial markers that can be placed on the tissue.
• the systems, media and methods disclosed herein can comprise acquisition of timelapse images for the sample of interest using the multi-step image acquisition process.
• Image Enhancement and Bleaching Correction [0101]
• photo-cleavable markers are used to obtain an unlimited set of source input channels by the process of repeat re-staining and clearing of markers using UV illumination and chemical wash in the tissue. In some cases, repeat scanning of tissue areas an introduce some photo bleaching and/or dimming at the edge of the FOV.
• FIG.7 shows darkening bands on the right and bottom sides of the image that degrade the signal and detectability of cells in those regions.
• the systems, media, and methods disclosed herein can further comprise normalizing the intensity of dimming regions using the nearest un-bleached portion of the image to correct edge artifacts.
• measurements can be taken WSGR Docket No. 31353-709.601 where the bleaching occurred by down sampling the image and dividing it into smaller blocks, as shown in, for example, FIG.8A.
• a transition smoothing function can be applied during the scaling calculation as shown in, for example, FIG.8B.
• the scheme of Normalization and scaling factor computation is illustrated in FIG.8C.
• the lower 25th percentile and top 75th percentile values can be computed, as shown in FIG.8C.
• the ratios of the lower 25th percentile and top 75th percentile can be computed, as shown in FIG.8C.
• the lower intensity quantile ratio can be used to normalize the intensity drop at that location by applying the scaling factor shown in, for example FIG.8B.
• the upper quantile can be used to determine the high intensity value of each block.
• the sudden drop of this high intensity quantile between neighboring blocks can be used to determine the position where the bleaching starts.
• the systems, media, and methods disclosed herein can further comprise applying bleaching correction to morphology and protein image acquisition.
• images can be corrected by conventional image enhancement techniques including denoising, deblurring, or image sharpening, or any combination thereof .
• one or more denoising algorithms can be applied to the one or more images.
• the one or more denoising algorithms include filter based (Wiener, Gaussian, Median), using wavelet transforms, total variation (TV) denoising and deep learning-based approaches.
• Deblurring algorithms include the nearest-neighbor deconvolution, Richardson- Lucy deconvolution and total variation (TV) regularization, or any combination thereof .
• image enhancement algorithms can include histogram equalization, contrast limited adaptive histogram equalization (CLAHE), contrast stretching, gamma correction, unsharp masking, or any combination thereof .
• Nearest Neighbor Deconvolution can be used. In some cases, Nearest Neighbor Deconvolution can comprise eliminating blur signals by utilizing adjacent z-planes. In some embodiments, Nearest Neighbor Deconvolution does not require point spread function (PSF) data.
• PSF point spread function
• Nearest Neighbor Deconvolution can be performed for the correction on an image at location z, I, using images from the previous and next z locations ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ and ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ as described below.
• A is the multiplier factor 0 – 1.
• G Gaussian kernel, and ⁇ is determined from optical numerical aperture (NA) and sampling rate in z ( ⁇ ⁇ ).
• NA optical numerical aperture
• the systems, media, and methods disclosed herein can further comprise applying the location and density of raw spots to infer additional cell morphology. In some embodiments, the systems, media, and methods disclosed herein can further comprise applying the location and density of raw spots to augment additional cell morphology.
• each RNA spot can be detected from image slices acquired across the tissue section. In some embodiments, the detection can use the Laplacian of Gaussian (LoG) bandpass filter. In some embodiments, the LoG bandpass filter can be designed to match the morphology of reporter signal in each channel. In some embodiments, the precise location of the spots can be estimated using a 2D paraboloid surface fit.
• LoG Laplacian of Gaussian
• chromatic aberration of the optics can be accounted for ensuring spatial alignment between each channel.
• the lens does not focus all wavelengths (channels) to the same point.
• FIG.9 illustrates the aberration generated in the axial direction (middle graphic) and the lateral direction (right graphic), comparing to the perfect lens on the left.
• the systems, media, and methods disclosed herein can further comprise applying unique calibrations that determine the z-offsets for each wavelength during acquisition to correct for axial chromatic aberrations. [0107]
• the systems, media, and methods disclosed herein can further comprise performing the lateral chromatic aberration correction on each multi-channel image.
• performing the lateral chromatic aberration correction can be performed using a precalibrated transform.
• the spot WSGR Docket No. 31353-709.601 locations can be binned into a lower-resolution pixel space. As shown in, for example, FIG.
• the systems, media, and methods disclosed herein can further comprise applying binned 2D histograms of spot locations as an additional input segmentation channel.
• raw spots were detected with a resolution of 0.1 of a pixel, to form a detailed morphology density image.
• spots for each FOV were resolved to a tenth of a pixel, as illustrated in Table 1.
• X and Y were in decipixels with a range [0, 42560]. Table 1.
• a grid 1330 x 1330 was overlaid on the range [0, 42560]. For example, the total number of spots falling into each box of the grid were tallied to determine the grayscale value in the resulting spots density map image. For example, the density map image was resized to the FOV dimensions of 4256 x 4256, reducing noise in the density map image. For example, the image was then used in the cell segmentation workflow as a standard morphology channel obtained with fluorescence staining. Multi-Modal Cell Segmentation [0109]
• the systems, media, and methods disclosed herein can comprise retrieving five morphological channels.
• retrieving five morphological channels can express more descriptions of cell boundaries than using the nuclear channel alone.
• the systems, media, and methods disclosed herein can comprise retrieving 64 or more different protein channels. In some embodiments, about 1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, or more than 100 different protein channels can be retrieved.
• FIG.13A shows several protein channels including a channel for Histone stain, DAPI stain, and Soma rRNA stain.
• FIG.13B shows several segmentation processed channels, including a readout density heatmap, segmentation based on all 3 morphology-stained images, and segmentation based on density heatmap and histone stain.
• the number of different protein channels retrieved can be dictated by the number of protein markers, scanning cycling parameters, additional transcript- based information using a spot density map, or any combination thereof, as illustrated in FIG. 14.
• the systems, media, and methods disclosed herein can comprise combining the morphology and protein images with the same morphology type into smaller sets of input channels.
• each protein can be annotated to describe its morphology.
• the annotations can include cytoplasm or nuclear expression, or both.
• the annotation can describe different cell type expressions.
• the cell type expression can comprise Glia, Astrocytes and other neuronal cell types, or any combination thereof.
• these annotations can be used to combine each protein image together based on cells having the same morphology or same cell type, or both.
• a projection method can be used to collapse multiple protein images with the same morphology into the 5- channel images.
• the projection method can be one or more of maximum intensity projection, principal component analysis, singular value decomposition, multi-reference alignment, density map averaging, or any combination thereof.
• the readout or spot density image, or both can be added as an additional channel.
• all input channels can be background subtracted and normalized prior to a segmentation step.
• cell segmentation can be one or more of nuclei segmentation, cytoplasm segmentation or extra-cellular segmentation, or any combination thereof .
• outputs from each cell segmentation can be combined as the final segmentation outputs.
• per-cell and per-compartment statistics can be performed.
• image preprocessing to minimize imaging artifacts and improve the quality of images can be applied to raw images before processing by cell segmentation modules.
• transcriptional spot density heatmaps can be treated processed using image preprocessing techniques.
• the preprocessing can be used to achieve better signal to noise ratio between intracellular and extracellular regions.
• spot density heatmaps can be used without further preprocessing.
• the image preprocessing step can employ a combination of conventional image processing techniques, such as one or more of deconvolution, gaussian blur, normalization, feature extraction, Fourier transforming, linear filtering, contrast enhancement, binarization, or any combination thereof.
• the image preprocessing step can utilize machine learning methods, for example Noise2Self and cellpose 3.0 image restoration models.
• WSGR Docket No. 31353-709.601 [0112]
• to provide sub-cellular segmentation a nuclear channel can be segmented.
• the nuclear channel can be segmented in addition to the other channels containing membrane and cytoplasmic content.
• multi-step segmentation produces sub-cellular results defining the separation of nuclei and cytoplasmic regions.
• overlap between cells can be measured using the Intersection over Union (IoU) score.
• IoU Intersection over Union
• nuclei channels may have a stronger signal.
• low intersection pixels can be assigned as nuclear segments.
• the nuclear segments are not cytoplasm. In some embodiments, when the intersection is high, the two segmentation results can be merged to combine both nuclei and cytoplasm.
• performing the IoU-based combination in a step-wise manner can combine multiple modalities of morphology in achieving the final outcome.
• the IoU-based combination approach can combine 2D cell segmentation outcomes of individual 2D images into the 3D cell segmentation volumetric outcomes of the corresponding 3D scan image stack.
• the correlation can comprise labelling each cell in all z-locations using maximum IoU scores for mapping consistently.
• other measures such as cell focus profile across z-slices can be used to determine cell label continuity through the z-slices.
• analyzing centroid and shape changes of cells across the z slices can be used to label cells in 3D stack.
• 3D stack can be directly segmented using a machine learning model that takes a 3D stack input.
• extending 2D to 3D in this arrangement can comprise performing multi-modal segmentation in 2D for each z-slice followed by 3D cell label stitching to correlate cellids in each z slice, forming 3D cell labels.
• FIG.23 illustrates a non-limiting example of a 3D segmentation pipeline.
• WSGR Docket No. 31353-709.601 [0114]
• localization of objects inside cells can be mapped to the cell body contour using maximum IoU scores.
• localization of cells in different source images can be mapped to each other using maximum IoU scores.
• the localization points of each of the different source images taken at different z coordinates or time points that, when matched, obtain the maximum IoU scores can be mapped together to form a stacked image.
• the spot images can be registered to the same reference as the morphology stained images.
• timelapse images can be taken for the sample of interest.
• the IoU-based combination approach can be applied to combine cell segmentation outcomes of same sample at different time points to track the behavior of same cells over time. [0115] In some embodiments, extra-cellular regions can be detected.
• extra-cellular regions can be included as part of segmentation results.
• extra- cellular regions can comprise one or more structures of the extracellular space of, for example, the brain.
• extracellular structures can comprise one or more vesicles.
• each connected component can be detected and masked.
• masking can be performed using auto-thresholding techniques.
• auto-thresholding techniques can comprise finding high intensity regions in the input morphology channel or using neural network models that have been trained to detect cell membrane or specific extra-cellular regions. [0116] In one non-limiting example, as illustrated in FIG.15, all protein expressing microglia cell type were combined with the microglia morphology channel using an Ibal marker.
• the input image channels included morphology channels and a channel of spot density map for the same biological sample.
• the maximum intensity pixel values can be projected for all input morphology channels.
• the readout or spot density image, or both can be added as a separate channel.
• the nuclei, cytoplasm, or extra-cellular segmentation can be performed for each feeding channel.
• a cell segmentation algorithm can be selected based on one or more feeding channels. In some embodiments, cell segmentation can be based on one or more algorithms.
• the one or more algorithms can comprise Cellpose Deep Learning algorithm, adaptive thresholding, Watershed algorithm, thresholding, region-based segmentation, active contours, Convolutional Neural Networks (CNN), Level Set Methods, Graph-based algorithms, fuzzy c-means clustering, Deep watershed, image morphology, or Markov Random Fields, or any combination thereof .
• nuclei WSGR Docket No. 31353-709.601 segmentation, cytoplasm segmentation and extra-cellular segmentation can be based on the same algorithm.
• nuclei segmentation, cytoplasm segmentation, and extra-cellular segmentation can be based on different algorithms.
• a model can be selected to fit a specific input morphology by comparing the image and model style vectors.
• outputs can be generated by measuring image property within each cell or cell compartment mask area.
• outputs from all models can be combined as one final output.
• per-cell or per-compartment statistics, or both, is performed.
• the systems, media, and methods disclosed herein can comprise multiple input channels in the model.
• FIGS.18A-18D nuclei expressed images were selected as input for nuclei segmentation , and all morphology input channels including nuclei were fed for cytoplasm segmentation.
• results from nuclei and cytoplasm segmentations were combined by analyzing overlaps and intersections (IoU) between segmentation results.
• Various markers were applied to the cell WSGR Docket No. 31353-709.601 segmentation results, for example, FIG.18A shows neuron cells with nuclei marker HistoneH3 applied, FIG.18B shows neuron cells with rRNA membrane markers applied, FIG.18C shows astrocytes with GFAP markers applied, and FIG.18D shows nuclei with DAPI markers applied.
• the systems, media, and methods disclosed herein can further comprise segmentation of extra-cellular processes as a third segmentation step.
• extra-cellular regions can be structures that exist in the image but do not contain nucleic information.
• the systems, media, and methods disclosed herein can further comprise tracing each of one or more neuronal processes using a filter.
• the filter can comprise Laplacian of Gaussian (LoG) filter.
• the filter can be followed by a connected component algorithm to separate detected processes into individual components, as illustrated in FIG.19.
• the LoG filter can extract thin neuron processes from normalized brain-specific markers.
• processes that fully encapsulate cell soma can be merged with the nuclei segments and marked as cells.
• processes that have no connections to the cell body can be marked as objects, as shown in FIGS. 20A-20C.
• FIG.20A shows an overlay of cells and extracellular processes that were masked as shown in FIG.20B to isolate the extra-cellular segments as shown in, for example, FIG.20C.
• the merging between cell protrusions and nuclei can be performed using a machine learning algorithm trained to perform merging operations.
• each neuron process can be individually identifiable as one or more objects
• transcriptomic, cell typing, and other spatial data analysis can be performed in the same manner as single cell analysis.
• all three segmentation steps can produce single cell labels using cell measurements and statistics, as WSGR Docket No. 31353-709.601 well as per-object measurements.
• cell labels can be further split into nuclei and cytoplasm compartments. In one non-limiting example for brain tissue, each individual cell type was marked in different channels and were identified as shown in FIG. 21.
• Per-Cell Statistics [0122]
• the segmentation output can provide statistics for each detected cell.
• the statistics can include statistics describing fluorescence properties, spatial locations, or features, or any combination thereof .
• these properties can comprise average, median, and maximum fluorescent intensities.
• these properties can comprise various shape descriptors.
• height can be assigned to be the longer value of the bounding box size.
• Circularity wherein A is the area of the cell and P conv is the convex perimeter to avoid concave irregularities. Wherein A is the area of the cell and P is regular cell perimeter.
• Convexity Eccentricity ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . ⁇ ⁇ ⁇ ⁇ ⁇ Wherein L minor is the length of the minor axis and L major is the length of the major axis.
• Solidity ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ _ ⁇ ⁇ ⁇ ⁇ ⁇ .
• A is the area of the cell and A conv is the convex area.
• the systems, media, and methods disclosed herein can comprise subcellular segmentation further comprising one or more deep learning models.
• one or more deep learning models can be trained using images of cells or tissues.
• images of cells or tissues can comprise microscope-derived images, morphology images, or fluorescence images, or any combination thereof .
• a deep learning model can be trained to segment a particular tissue type.
• a deep learning model can be trained to restore image quality for better cell segmentation outcomes.
• the systems, media, and methods disclosed herein can comprise training a machine learning model or applying a machine learning model, or both.
• the machine learning model may perform dimension reduction.
• the dimension reduction can be performed through a nonlinear dimensionality reduction algorithm.
• the nonlinear dimensionality reduction algorithm may comprise Sammon’s mapping, Principal curves and manifolds, Laplacian eigenmaps, Isomap, Locally-linear embedding, Local tangent space alignment, Maximum variance unfolding, Gaussian process latent variable models, t-distributed stochastic neighbor embedding, Relation perspective map, Contagion maps, Curvilinear component analysis, Curvilinear distance analysis, Diffeomorphic dimensionality reduction, Manifold alignment, Diffusion maps, Local multidimensional scaling, Nonlinear PCA, Data-driven high- dimensional scaling, Manifold sculpting, RankVisu, Topologically constrained isometric embedding, Uniform manifold approximation or projection (UMAP), or any combination thereof.
• UMAP Uniform manifold approximation or projection
• the systems, media, and methods disclosed herein can comprise training a machine learning model or applying a machine learning model, or both.
• the machine learning model may comprise an unsupervised machine learning model, a supervised machine learning model, semi-supervised machine learning model, or a self-supervised machine learning, or any combination thereof .
• the supervised machine learning model may comprise, for example, a Random Forest, a support vector machine (SVM), a neural network, or a deep learning algorithm, or any combination thereof.
• the platforms, systems, media, and methods disclosed herein can comprise a machine learning model utilizing one or more neural networks.
• a neural network can learn the relationships between an input dataset and a target dataset.
• a neural network may be a software representation of a human neural system.
• the human neural system may be the cognitive system.
• the neural network capture "learning" and "generalization” abilities as used by a human.
• the machine learning algorithm can comprise a neural network comprising a CNN.
• machine learning algorithm structural components can comprise one or more of: CNNs, recurrent neural networks, dilated CNNs, fully-connected neural networks, deep generative models, transformers, or Boltzmann machines, or any combination thereof .
• a neural network can comprise a series of layers termed "neurons.”
• a neural network can comprise an input layer, to which data is presented; one or more internal, and/or "hidden,” layers; and an output layer.
• a neuron may be connected to neurons in other layers via connections that have weights, which are parameters that control the strength of the connection.
• the number of neurons in each layer may be related to the complexity of the problem to be solved.
• the minimum number of neurons required in a layer may be determined by the problem complexity, and the maximum number may be limited WSGR Docket No. 31353-709.601 by the ability of the neural network to generalize.
• the input neurons may receive data being presented and then transmit that data to the first hidden layer through connections' weights, which are modified during training.
• the first hidden layer may process the data and transmit its result to the next layer through a second set of weighted connections.
• each subsequent layer may "pool" the results from the previous layers into more complex relationships.
• neural networks are programmed by training them with a known sample set and allowing them to modify themselves during (and after) training so as to provide a desired output such as an output value.
• the neural network can comprise artificial neural networks (ANNs).
• ANNs may be machine learning algorithms that may be trained to map an input dataset to an output dataset, where the ANN comprises an interconnected group of nodes organized into multiple layers of nodes.
• the ANN architecture may comprise at least an input layer, one or more hidden layers, and an output layer.
• the ANN may comprise any total number of layers, and any number of hidden layers, where the hidden layers function as trainable feature extractors that allow mapping of a set of input data to an output value or set of output values.
• a deep learning algorithm such as a deep neural network (DNN)
• DNN deep neural network
• Each layer of the neural network may comprise a number of nodes (or "neurons").
• a node can receive input that comes either directly from the input data or the output of nodes in previous layers, and performs a specific operation, e.g., a summation operation.
• a connection from an input to a node can be associated with a weight or weighting factor.
• the node may sum up the products of all pairs of inputs and their associated weights.
• the weighted sum may be offset with a bias.
• the output of a node or neuron may be gated using a threshold or activation function.
• the activation function may be a linear or non-linear function.
• the activation function may be, for example, a rectified linear unit (ReLU) activation function, a Leaky ReLU activation function, or other function such as a saturating hyperbolic tangent, identity, binary step, logistic, arctan, softsign, parametric WSGR Docket No. 31353-709.601 rectified linear unit, exponential linear unit, softplus, bent identity, softexponential, sinusoid, sine, Gaussian, or sigmoid function, or any combination thereof.
• the weighting factors, bias values, and threshold values, or other computational parameters of the neural network may be "taught" or "learned” in a training phase using one or more sets of training data.
• the parameters may be trained using the input data from a training dataset and a gradient descent or backward propagation method so that the output value(s) that the ANN computes are consistent with the examples included in the training dataset.
• the number of nodes used in the input layer of the ANN or DNN may be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000, or greater than 100,000, including increments therein.
• the number of nodes used in the input layer may be at most about 100,000, 90,000, 80,000, 70,000, 60,000, 50,000, 40,000, 30,000, 20,000, 10,000, 9000, 8000, 7000, 6000, 5000,4000, 3000, 2000, 1000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10,or less than 10, including increments therein.
• the total number of layers used in the ANN or DNN may be at least about 3, 4, 5, 10, 15, 20, or greater, including increments therein. In other instances, the total number of layers may be at most about 20, 15, 10, 5, 4, 3, or less, including increments therein.
• the total number of learnable or trainable parameters, e.g., weighting factors, biases, or threshold values, used in the ANN or DNN may be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000, or greater than 100,000, including increments therein.
• the number of convolutional layers can be between 1-10. In some embodiments, the number of dilated layers can be between 0-10. In some embodiments, the total number of convolutional WSGR Docket No. 31353-709.601 layers (including input and output layers) may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or greater, and the total number of dilated layers may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or greater. In some embodiments, the total number of convolutional layers may be at most about 20, 15, 10, 5, 4, 3, or less, and the total number of dilated layers may be at most about 20, 15, 10, 5, 4, 3, or less. In some embodiments, the number of convolutional layers can be between 1-10 and the fully-connected layers between 0-10.
• a machine learning model can comprise one or more CNNs.
• the CNN may be deep and feedforward ANNs.
• the CNN may be applicable to analyzing visual imagery.
• the CNN may comprise an input, an output layer, and multiple hidden layers.
• the hidden layers of a CNN may comprise convolutional layers, pooling layers, fully -connected layers and normalization layers.
• the layers may be organized in 3 dimensions: width, height, and depth.
• the convolutional layers may apply a convolution operation to the input, and pass results of the convolution operation to the next layer.
• the convolution operation may reduce the number of free parameters, allowing the network to be deeper with fewer parameters.
• each neuron may receive input from some number of locations in the previous layer.
• neurons may receive input from only a restricted subarea of the previous layer.
• the convolutional layer's parameters may comprise a set of learnable filters (or kernels). In some cases, the learnable filters may have a small receptive field and extend through the full depth of the input volume.
• each filter may be convolved across the width and height of the input volume, compute the dot product between the entries of the filter and the input, and produce a two-dimensional activation map of that filter.
• the network may learn filters that activate when it detects some specific type of feature at some spatial position in the input.
• a machine learning model can comprise an RNN.
• RNNs are neural networks with cyclical connections that can encode and process sequential data.
• an RNN can include an input layer that is configured to receive a sequence of inputs.
• an RNN may additionally include one or more hidden recurrent layers that maintain a state.
• each hidden recurrent layer can compute an output and a next state for the layer.
• the next sate may depend on the previous state and the current input.
• the state may be maintained across steps and may capture dependencies in the input sequence.
• an RNN can be a long short-term memory (LSTM) network.
• LSTM network may be made of LSTM units.
• an LSTM unit may include of a cell, an input gate, an output gate, and a forget gate.
• the cell may be responsible for keeping track of the dependencies between the elements in the input sequence.
• the input gate can control the extent to which a new value flows into the cell
• the forget gate can control the extent to which a value remains in the cell
• the output gate can control the extent to which the value in the cell is used to compute the output activation of the LSTM unit.
• the neural network can comprise an attention mechanism (e.g., a transformer).
• attention mechanisms may focus on, or "attend to,” certain input regions while ignoring others. In some embodiments, this may increase model performance because certain input regions may be less relevant.
• an attention unit can compute a dot product of a context vector and the input at the step, among other operations.
• the output of the attention unit may define where the most relevant information in the input sequence is located.
• the attention mechanism can comprise a vision transformer.
• the vision transformer can comprise a natural language model.
• the vision transformer can comprise a masked autoencoder, a Swin transformer, a vector quantized variational autoencoder, or another type of vision transformer.
• the vision transformer can be combined with a generative adversarial network.
• the pooling layers can comprise global pooling layers.
• the global pooling layers may combine the outputs of neuron clusters at one layer into a single neuron in the next layer.
• max pooling layers may use the maximum value from each of a cluster of neurons in the prior layer
• average pooling layers WSGR Docket No. 31353-709.601 may use the average value from each of a cluster of neurons at the prior layer.
• the fully-connected layers can connect every neuron in one layer to every neuron in another layer. In some cases in neural networks, each neuron may receive input from some number locations in the previous layer. In some cases, in a fully-connected layer, each neuron may receive input from every element of the previous layer.
• the normalization layer can be a batch normalization layer.
• the batch normalization layer may improve the performance and stability of neural networks.
• the batch normalization layer may provide any layer in a neural network with inputs that are zero mean/unit variance.
• the advantages of using batch normalization layer may include faster trained networks, higher learning rates, easier to initialize weights, more activation functions viable, and simpler process of creating deep networks.
• the trained algorithm may be configured to accept a plurality of input variables and to produce one or more output values based on the plurality of input variables.
• the trained algorithm may comprise a classifier, such that each of the one or more output values may comprise one of a fixed number of possible values.
• the possible values can comprise a linear classifier, a logistic regression classifier.
• the algorithm can indicate a classification of the biological sample or the subject by the classifier, or both.
• the trained algorithm may comprise a binary classifier.
• each of the one or more output values can comprise one of two values.
• the values can comprise ⁇ 0, 1 ⁇ , ⁇ positive, negative ⁇ , or ⁇ high-risk, low-risk ⁇ , indicating a classification of the biological sample or subject, or both, by the classifier.
• the trained algorithm may be another type of classifier, such that each of the one or more output values can comprise one of more than two values.
• the values can comprise ⁇ 0, 1, 2 ⁇ , ⁇ positive, negative, or indeterminate ⁇ , or ⁇ high-risk, intermediate-risk, or low-risk ⁇ , indicating a classification of the biological sample or subject, or both, by the classifier.
• the output values may comprise descriptive labels, numerical values, or a combination thereof.
• some of the output values may comprise descriptive labels.
• some of the output values may comprise numerical values, such as binary, integer, or continuous values.
• Such binary output values may comprise, for example, ⁇ 0, 1 ⁇ , ⁇ positive, negative ⁇ , or ⁇ high-risk, low-risk ⁇ .
• Such integer output values may comprise, for example, ⁇ 0, 1, 2 ⁇ .
• Such continuous output values may comprise, for example, a probability value of at least 0 and no more than 1.
• Such continuous output values may comprise, for WSGR Docket No. 31353-709.601 example, an un- normalized probability value of at least 0.
• Such continuous output values may indicate a prognosis of the cancer-related category of the subject.
• Some numerical values may be mapped to descriptive labels, for example, by mapping 1 to "positive” and 0 to "negative.”
• the classification of samples may assign an output value of "indeterminate” or 2 if the sample is not classified as "positive,” “negative,” 1, or 0.
• a set of two cutoff values can be used to classify samples into one of the three possible output values.
• sets of cutoff values may include ⁇ 1%, 99% ⁇ , ⁇ 2%, 98% ⁇ , ⁇ 5%, 95% ⁇ , ⁇ 10%, 90% ⁇ , ⁇ 15%, 85% ⁇ , ⁇ 20%, 80% ⁇ , ⁇ 25%, 75% ⁇ , ⁇ 30%, 70% ⁇ , ⁇ 35%, 65% ⁇ , ⁇ 40%, 60% ⁇ , and ⁇ 45%, 55% ⁇ .
• sets of cutoff values may be used to classify samples into one of n+1 possible output values, where n is any positive integer.
• the systems, media, and methods disclosed herein can further comprise generating a data summary, result, or visualization, or any combination thereof .
• visualization displays may comprise graphs, plots, and overlay points representing transcripts or cell annotations onto tissue images.
• visualization may query that Seurat/tileDB object to retrieve the relevant data from the object for display.
• the relevant data may comprise transcript locations and/or cell annotations.
• visualization may be displayed as transcript locations as points, cell annotations (e.g., cell type) as points, boxplots, violin plots, dot plots , or any combination thereof.
• visualization on cell segmentation statistics may be displayed as points for raw values or projection coordinates in embedding space for evaluation of the segmentation outcomes of interest with respect to either of the alternative segmentation outcomes of same sample.
• the visualization may form overlays between segmentation boundaries, transcripts and downstream data analysis output.
• downstream data analysis output comprises cell types.
• downstream data analysis output comprises one or more of cell states, phenotypes of tissue microenvironment, differential expression of cell type based on spatial context, qu antifying subcellular expression, and spatially resolved biomarker identification, or any combination thereof.
• the visualization may be from different segmentation configurations or methods, or the segmentation outcomes of reference samples.
• the visualization may be performed using a vision transformer.
• the vision transformer can process an input image into a series of patches.
• the vision transformer can serialize each patch into a vector.
• the vision transformer can match each vector to a smaller dimension with matrix WSGR Docket No. 31353-709.601 multiplication.
• a transformer encoder can process the vision transformer data.
• transformers can measure relationships between one or more input tokens through the transformer encoder.
• the visualization may be trained in a masked encoder, a Swin transformer, a vector quantized variational autoencoder, or any combination thereof.
• Example 1 Chromatic Corrections
• the lateral chromatic aberration correction was performed on each multi-channel image using a pre-calibrated transform. As illustrated in FIG.10, Raw Image on the left was applied to chromatic aberration and was corrected up. In correction , the fiduciary in all channels lining up was visibly observed in the Corrected Image on the right. The parameters of chromatic aberration correction are listed in Correction Transform table.
• Example 2 Spot density heatmap [0154] As illustrated in FIGS.13A and 13B, reasonable segmentation results were achieved and revealed cell structure that was not shown in original segmentation using the morphology markers alone.
• the 3 columns in FIG.13A show morphology images of brain tissue with image channels including Histone stain, DAPI stain, and Soma rRNA stain.
• the 3 columns in FIG.13B show readout density heatmap, segmentation based on all 3 morphology- stained images, and segmentation based on density heatmap & histone stain.
• the readout density heatmap in column I of FIG.13B displays the contours of cell soma at higher signal-to-noise ratio with fine details as compared to the rRNA soma stain morphology image in the third column of FIG.13A.
• FIG.24 shows the raw image of readout density heatmap compared to the image restoration output after processing by cellpose 3.0 denoising model.
• Example 3 Comparison of 2-Channel and 5-Channel model outputs WSGR Docket No. 31353-709.601
• FIG.17A illustrates the initial network output using Cellpose algorithm.
• the gradient flow output was generated using the 2-channel model.
• Ground truth annotations were generated from segmentation results produced by existing custom cell segmentation algorithm, a process known as bootstrapping, based on the 2-channel model as initial ground truth. Subsequently, the annotations were reviewed and corrected to ensure quality of training data.
• the new 5-channel network model was initialized with the existing 2-channel model parameters, which helped reduce the training burden and yield good initial segmentation performance.
• the network was trained and fine-tuned to optimize the parameters for all channels based on the annotated ground truth. Parameter sets were finalized once they met a specified learning rate and minimize specific cost functions.
• FIG.17B illustrates the output gradient flow result from the trained 5-channel model initialized from transfer learning.
• FIG.25 illustrates a non-limiting example of an image acquired using High Definition Range (HDR) settings.
• HDR High Definition Range
• a nuclei stained image was acquired with two non-HDR mode, one with a higher exposure time setting and one with a lower exposure time.
• DAPI nuclear staining was used and excited with UV at 385nm and detected in the blue channel with emission at 512nm.
• the low exposure scan was acquired at 3ms at 3% power (A) followed by 24ms at 4% power (B). As shown in FIG.25, some cells in B are saturated. As illustrated in FIG.25, (C) shows the image acquired with HDR using both exposure times combined to provide a larger dynamic ranges and at the same time reduced intensity saturation. Illustrated in FIG.25, (A) was an image acquired at 3ms (low); (B) was an image acquired at 24 ms (high); and (C) was an image acquired with HDR. [0158] For example, the morphology channels describing cell membrane used the following acquisition settings below in Table 2. In Table 2, the channel/dye used is listed, as well as the Excite/Emission wavelengths and the exposure times.
• FIG.26 illustrates a non-limiting example of output produced by the nearest neighbor (NN) deconvolution method.
• NN nearest neighbor
• a multiplier A 0.75 was used to ensure that dim cells were still retained while removing majority of the blur signal.
• areas are circled to mark regions where cells previously blurred by an out of focus signal (left panel) were recovered with the nearest neighbor deconvolution, uncovering clear boundaries between cells (right panel) that were previously difficult to see.
• FIG.27 illustrates a non-limiting example of cell protrusion merging.
• FIG.28 illustrates a non-limiting example of IoU merging results between a nuclei segmentation output and a membrane plus nuclei segmentation.
• Areas encircled in FIG.28 mark cases of cell merging.
• One merging example at the top right location of FIG.28 shows that the cell mask was only detected in the nuclei segmentation step (A) and not at the WSGR Docket No. 31353-709.601 cytoplasm/membrane segmentation step (B).
• the second merging example at the bottom left of FIG.28 shows that when the two masks in (A) (B) overlap, the masks were merged, retaining the shapes from both (A) and (B) [0162]
• this intersection analysis can be extended to correlate cells in different z-slices to perform 3D cell segmentation.
• FIG.29A illustrates a non- limiting example of IoU merging results to correlate cellIds between multiple 2D image slices.
• FIG.29B illustrates a non-limiting example of IoU merging results within a 3D volume.
• FIG. 29C shows a non-limiting example of one cell marked in all z-slices to show that this cell is segmented throughout the entire Z-stack. This data was generated using the NanoString whole transcriptome (Wtx) Pancreas public dataset.
• Example 7 – Segmentation Results Across Different Tissue Types [0163] The multi-modal segmentation system has been tested in multiple tissue samples including those typically used in immune oncology research such as human breast, colon, lung, kidney, liver, and tonsils both in normal and diseased states.
• Table 5 illustrates an example of the split cell measurements. In Table 5, values close to 1 indicate that those cells at the boundary may be retained since their size is similar to the average and only a small portion of the split cells are located at the neighboring FOVs. Table 5. Split Ratio of Cells.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Physics & Mathematics (AREA)
• Public Health (AREA)
• Theoretical Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Biomedical Technology (AREA)
• Life Sciences & Earth Sciences (AREA)
• Genetics & Genomics (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Radiology & Medical Imaging (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Pathology (AREA)
• Molecular Biology (AREA)
• Biophysics (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Biotechnology (AREA)
• Evolutionary Biology (AREA)
• Spectroscopy & Molecular Physics (AREA)
• Investigating Or Analysing Biological Materials (AREA)
• Measuring Or Testing Involving Enzymes Or Micro-Organisms (AREA)
• Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=93852876

Family Applications (1)

Country Status (5)

Cited By (2)

Family Cites Families (3)

• 2024
  • 2024-06-12 EP EP24824082.2A patent/EP4728471A2/de active Pending
  • 2024-06-12 CN CN202480039614.2A patent/CN121569321A/zh active Pending
  • 2024-06-12 KR KR1020267000485A patent/KR20260022393A/ko active Pending
  • 2024-06-12 WO PCT/US2024/033630 patent/WO2024258979A2/en not_active Ceased
  • 2024-06-12 AU AU2024304188A patent/AU2024304188A1/en active Pending

Also Published As

Similar Documents

Legal Events