EP3685355B1 - Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs - Google Patents

Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs Download PDF

Info

Publication number
EP3685355B1
EP3685355B1 EP18780265.7A EP18780265A EP3685355B1 EP 3685355 B1 EP3685355 B1 EP 3685355B1 EP 18780265 A EP18780265 A EP 18780265A EP 3685355 B1 EP3685355 B1 EP 3685355B1
Authority
EP
European Patent Office
Prior art keywords
dhm
images
cells
image
stained
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18780265.7A
Other languages
German (de)
English (en)
Other versions
EP3685355C0 (fr
EP3685355A1 (fr
Inventor
Noha Youssry El-Zehiry
Saikiran Rapaka
Ali Kamen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Healthcare Diagnostics Inc
Original Assignee
Siemens Healthcare Diagnostics Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Healthcare Diagnostics Inc filed Critical Siemens Healthcare Diagnostics Inc
Publication of EP3685355A1 publication Critical patent/EP3685355A1/fr
Application granted granted Critical
Publication of EP3685355B1 publication Critical patent/EP3685355B1/fr
Publication of EP3685355C0 publication Critical patent/EP3685355C0/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/10—Texturing; Colouring; Generation of textures or colours
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Optical investigation techniques, e.g. flow cytometry
    • G01N15/1429—Signal processing
    • G01N15/1433—Signal processing using image recognition
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Optical investigation techniques, e.g. flow cytometry
    • G01N15/1434—Optical arrangements
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
    • G01N33/50—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing
    • G01N33/5005—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing involving human or animal cells
    • G01N33/5094—Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing involving human or animal cells for blood cell populations
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/0005—Adaptation of holography to specific applications
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G03H1/0443—Digital holography, i.e. recording holograms with digital recording means
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G03H1/08—Synthesising holograms, i.e. holograms synthesized from objects or objects from holograms
    • G03H1/0866—Digital holographic imaging, i.e. synthesizing holobjects from holograms
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24143—Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/698—Matching; Classification
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N2015/1006—Investigating individual particles for cytology
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Optical investigation techniques, e.g. flow cytometry
    • G01N15/1434—Optical arrangements
    • G01N2015/1454—Optical arrangements using phase shift or interference, e.g. for improving contrast
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
    • G01N33/483—Physical analysis of biological material
    • G01N33/487—Physical analysis of biological material of liquid biological material
    • G01N33/49—Blood
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/0005—Adaptation of holography to specific applications
    • G03H2001/005—Adaptation of holography to specific applications in microscopy, e.g. digital holographic microscope [DHM]
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/04—Processes or apparatus for producing holograms
    • G03H1/0443—Digital holography, i.e. recording holograms with digital recording means
    • G03H2001/0445—Off-axis recording arrangement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Definitions

  • Digital holographic microscopy also known as interference phase microscopy
  • interference phase microscopy is an imaging technology that provides the ability to quantitatively track sub-nanometric optical thickness changes in transparent specimens.
  • DHM captures both phase and intensity.
  • the phase information, captured as a hologram can be used to reconstruct extended morphological information (such as depth and surface characteristics) about the specimen using a computer algorithm.
  • Modem DHM implementations offer several additional benefits, such as fast scanning/data acquisition speed, low noise, high resolution and the potential for label-free sample acquisition
  • Off-axis DHM systems create holograms where there is a modulating sinusoidal fringe pattern over the entire field of view due to a small angle between the object and reference beams.
  • plane waves impinging a sensor surface interfere destructively and constructively at the location of the sensor and thus form the sinusoidal pattern.
  • a reference beam 20 is created from an object beam 10 using a pin hole at the location of the optical Fourier-plane (or conjugated plane to the Fourier-plane) to erase the object spatial signature and to provide a uniform plane waves for creating an interference or hologram image which may be stored and displayed to a user.
  • DHM is used in hematology to image different cells within a blood sample.
  • the beam passes through the cell to be imaged, it gets refracted based on the cell characteristics.
  • These refraction variations can be captured through DHM as changes in the optical thickness within the nucleus.
  • WBCs white blood cells
  • the nuclei and the membranes have different properties and the structure of the cell constituents differ based on the cell type.
  • the appearance of the DHM image changes depending on the cell type.
  • counts of different WBC types may be obtained by providing a blood sample to an automated machine, such as a laser flow cytometer, which performs an automated count and analysis.
  • an automated machine such as a laser flow cytometer
  • blood cells are also manually examined under a microscope and the different types of WBCs are counted and reviewed by a clinician.
  • the blood sample is stained with a dye before examination, such as through a peripheral blood smear.
  • a blood film may be made by placing a drop of blood on one end of a glass slide and using a spreader to disperse the blood sample into a monolayer over the slide's length.
  • the nucleus of each different type of WBC absorbs the stain differently, for example, as shown in FIG. 2 , allowing the clinician to count and examine the different WBCs in a blood smear sample.
  • DHM images have become an alternative to conventional microscopy and can be used to examine a blood sample and count the different types of WBCs in the sample.
  • DHM images such as those shown in the example DHM images of FIG. 3 , may not include sufficient detail or resolution in order to allow each cell type to be easily identified and categorized by the clinician.
  • WO 2015195609 A1 discloses a method for analyzing digital holographic microscopy (DHM) for hematology applications.
  • the method includes receiving DHM images depicting cells, which are acquired using a digital holographic microscopy system as a DHM device. Further, a digital staining or pseudo-coloring of the classified DHM images of the cells is performed with a learnt regression function in order to map the DHM image pixels. Thus, it can be assumed that the DHM image is virtually stained by processing, using an image conversion algorithm.
  • embodiments of the present disclosure are directed to a computer-implemented method for producing a virtually stained digital holographic microscopy (DHM) image, according to claim 1.
  • DHM digital holographic microscopy
  • embodiments of the present disclosure are directed to a cell visualization system according to claim 8, an apparatus according to claim 7, a computer program product according to claim 11 and a computer-readable medium according to claim 12.
  • the cell visualization system is trained to classify white blood cells using deep learning algorithms which utilize unpaired data samples.
  • Generative adversarial networks with cycle consistency are used to train the cell visualization system to learn how to identify different types of white blood cells in DHM images and how to colorize the images in order for the white blood cells to appear as if they had been stained in a comparable manual process.
  • FIG. 4A is a diagram of a cell visualization system 100, according to a disclosed embodiment.
  • the cell visualization system 100 includes a DHM device 110, a training device 120, and a virtual staining device 130.
  • the cell visualization system 100 is configured to produce DHM images which are virtually stained to imitate the appearance of cells which have been manually stained and viewed under a conventional microscope (e.g., the images in FIG. 2).
  • FIG. 2 further depicts an exemplary flow of data through the cell visualization system 100 in order to produce the virtually stained DHM images.
  • the DHM device 110 includes DHM components known in the art and configured to produce DHM images based on a received blood sample.
  • the training device 120 is configured to receive DHM images and training data, and execute a training process which results in a trained image conversion process.
  • the virtual staining device 130 is configured to receive and execute the trained image conversion process in order to convert DHM images received from the DHM device 110 into virtually stained images.
  • the virtual staining device 130 may be further configured to display the virtually stained images to a user (e.g., a clinician) for review.
  • Fig. 4B is a schematic diagram of an exemplary process flow through the cell visualization system 100.
  • the DHM device 110 is configured to produce DHM images of cells, such as those shown in FIG. 3 .
  • the DHM device 110 may receive a blood sample and capture periodic data using off-axis DHM while the sample is flowing.
  • the captured data may be converted into DHM images which encompass many cells, including red and white blood cells. These capture images which include many cells may be generally referred to as raw DHM images.
  • the DHM device 110 is configured to pre-process the raw DHM images in order to separate the raw data into separate images of one or more cells.
  • the DHM device 110 may look for bright spots within the raw DHM images and perform an analysis to identify which of the bright spots are above a size or other classifier threshold in order to identify which of the spots are cells.
  • Each of the classified cells may be transformed into a separate DHM image, resulting in separate cell images such as those shown in FIG. 3 .
  • the virtual staining device 130 is configured to communicate with the training device 120 and receives a trained process from the training device 120.
  • the trained process is an algorithm, generated by the training device 120 through generative adversarial networks, which converts DHM images into virtually stained DHM images. This conversion is sometimes referred to herein as "virtually staining" the DHM images and, in at least some embodiments, includes digital colorization of the one or more cells to imitate the appearance of a corresponding manually stained cell.
  • the virtual staining device 130 is configured to execute an associated process and produce the virtually stained DHM images for display to a clinician for review.
  • the virtual staining device 130 may be further configured to produce and store data associated with the virtually stained DHM images for analysis and/or use in further processes, such as image filtering, cell counting, cell analysis, etc.
  • the training device 120 is configured to use generative adversarial networks in order to train the virtual staining device 130 to convert DHM images into virtually stained DHM images.
  • the training device 120 uses generative adversarial networks as a training algorithm.
  • Generative adversarial networks generally represent a class of artificial intelligence algorithms that falls under the category of unsupervised learning.
  • generative adversarial networks are a combination of two neural networks: one network is learning how to generate examples (e.g., virtually stained images) from a training data set (e.g., images of physically stained samples) and another network attempts to distinguish between the generated examples and the training data set.
  • the training process is successful if the generative network produces examples which converge with the actual data such that the discrimination network cannot consistently distinguish between the two.
  • training examples consist of two data sets X and Y.
  • the data sets are unpaired, meaning that there is no one-to-one correspondence of the training images in X and Y.
  • the generator network trains the mapping G:X ⁇ Y such that y' is indistinguishable from y by a discriminator network trained to distinguish y' from y. In other words, the generator network continues producing examples until the discriminator network cannot reliably classify the example as being produced by the generator network (y') or supplied as an actual example (y).
  • mapping will be successful in fooling the discriminator network without producing images y' which meet the objective.
  • the generative network may produce examples y' which are similar to the images in y, but not in the aspects or characteristics that are important to the viewer.
  • the training device 120 applies an adversarial loss to each mapping, per the below equations.
  • G is a mapping which produces images G(x) which are similar to images in the domain Y.
  • D y is a discriminator which aims to distinguish between generated examples G(x) and actual examples y in the domain Y.
  • F is an inverse mapping which produces images F(y) which are similar to images in the domain X.
  • D x is a discriminator which aims to distinguish between generated examples F(y) and actual examples x in the domain X.
  • the training device 120 is configured to use cycle consistency losses for each mapping G, F.
  • the cycle consistency losses aim to keep the acceptable successful mappings in a realm in which the generated images are cycle consistent in that they can be input into the inverse mapping to reproduce the original image.
  • the cycle consistency losses can be expressed using the below equation.
  • L cyc F G E x ⁇ P data x ⁇ F G x ⁇ x ⁇ + E y ⁇ P data y ⁇ G F y ⁇ y ⁇
  • the training device 120 uses these X and Y domain images as input for the generative adversarial networks with cycle consistency process described herein. As a result, the training device produces algorithms F* and G*, where F* translates DHM images into virtually stained DHM images and G* translates stained cell images into virtual DHM images (images of cells which are not stained as they would appear in a DHM images).
  • the F* algorithm is transmitted or otherwise provided to virtual staining device 130.
  • the virtual staining device 130 is configured to execute a virtual staining process using the F* algorithm to convert DHM images into virtually stained DHM images.
  • FIGS. 5A, 5B, and 5C illustrate examples of stained cell images and FIGS. 5D, 5E, and 5F illustrate examples of respective corresponding virtually stained DHM images.
  • FIG. 5A illustrates an example of a manually stained Lymphocyte and FIG. 5D illustrates an example of a virtually stained Lymphocyte.
  • FIG. 5B illustrates an example of a manually stained Eosinophil and FIG. 5E illustrates an example of a virtually stained Eosinophil.
  • FIG. 5C illustrates an example of a manually stained Neutrophil and FIG. 5F illustrates an example of a virtually stained Neutrophil.
  • the borderline around each stained portion has been added to the images to emphasize the similarity. It should be understood that the stained cells in FIGS. 5A-5C do not correspond to the same cells in FIGS. 5D-5F.
  • FIGS. 5A-5F illustrate a visual comparison which shows the similarity between virtually stained DHM images and manually stained cells.
  • the cell visualization system 100 is configured to convert a blood sample into virtually stained DHM images for review by a clinician.
  • DHM images can be reviewed by the clinician in a similar manner to the way in which manually stained cells are reviewed.
  • the virtual staining process is much less time and labor intensive and provides a more efficient method in which stained cells can be obtained for the review by the clinician.
  • the cell visualization system 100 may include a cell classification unit which is configured to review the virtually stained DHM images and classify the cell type (e.g., basophil, eosinophil, lymphocyte, monocyte or neutrophil).
  • the cell classification unit may be trained to determine the cell type based on the presentation of the virtual stain in the virtually stained DHM image.
  • the cell type may be stored as data with the virtually stained DHM images.
  • the cell visualization system 100 can use the identification data to produce useful information, such as cell type count and proportions.
  • the cell visualization system 100 can use the identification data to identify abnormalities which can be prioritized and presented to clinician for review.
  • the cell visualization system 100 may use the identification data to filter the virtually stained DHM images, allowing a clinician to selectively review certain images.
  • Virtual staining can also be beneficial because of the ease in which data can be produced for other uses. Due to the time and labor intensiveness of manual staining and the capturing of images of these cells, it is difficult to obtain large data sets of stained cell images. Virtual staining of DHM images addresses this problem because stained cell images can be produced at a much greater scale in a shorter amount of time. The availability of larger data sets can help other processes.
  • DHM images can be used to improve classification of the different types of white blood cells.
  • Deep learning algorithms have been used to classify stained white blood cell images. Deep learning methods are known to provide better classification results when more data is used.
  • Virtual staining can be used to generate stained cell images that can be used to augment the training data sets and provide a larger data set to train classification algorithms.
  • the speed with which stained cell images can be produced is beneficial in many areas.
  • virtual staining can create a more efficient pipeline for detecting diseases such as Malaria.
  • the doctor suspects Malaria and one test comes out negative the patient is required to have repeated blood smear tests (thick and thin smears) every 8 hours for two days. This process is time and labor consuming due to the manual staining process.
  • virtual staining of DHM images can be used for a more efficient pipeline of identifying Malaria parasites.
  • the computer system 610 also includes a system memory 630 coupled to the bus 621 for storing information and instructions to be executed by processors 620.
  • the system memory 630 may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM) 631 and/or random access memory (RAM) 632.
  • the system memory RAM 632 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM).
  • the system memory ROM 631 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM).
  • the system memory 630 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 620.
  • Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the bus 621.
  • Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
  • the embodiments of the present disclosure may be implemented with any combination of hardware and software.
  • the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer-readable, non-transitory media.
  • the media has embodied therein, for instance, computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure.
  • the article of manufacture can be included as part of a computer system or sold separately.
  • a graphical user interface comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions.
  • the GUI also includes an executable procedure or executable application, such as a mobile application on a mobile device.
  • the executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user.
  • the processor under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.
  • An activity performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Chemical & Material Sciences (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Immunology (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Hematology (AREA)
  • Data Mining & Analysis (AREA)
  • Cell Biology (AREA)
  • Urology & Nephrology (AREA)
  • Dispersion Chemistry (AREA)
  • Tropical Medicine & Parasitology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Microbiology (AREA)
  • Ecology (AREA)
  • Food Science & Technology (AREA)
  • Medicinal Chemistry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biotechnology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)

Claims (12)

  1. Méthode informatisée de production d'une image numérique virtuellement tachée de microscopie holographique (DHM), comprenant :
    réception d'une image DHM acquise à l'aide d'un dispositif DHM (110), l'image DHM comprenant
    des représentations d'une ou plusieurs cellules ; et
    coloration virtuelle, par un dispositif de coloration virtuelle (130), de l'image DHM en traitant l'image DHM à l'aide d'un algorithme de conversion d'images,
    dans laquelle l'image DHM virtuellement colorée comprend la colorisation numérique d'un ou plusieurs éléments de l'image DHM
    pour imiter l'apparence d'une cellule correspondante réellement colorée,
    caractérisé en ce que,
    l'algorithme de conversion d'images est formé à l'aide de réseaux adversaires génératifs,
    recevoir, au niveau d'un dispositif d'apprentissage (120), un premier ensemble de données d'apprentissage comprenant des images DHM de globules blancs individuels et un second ensemble de données d'apprentissage comprenant des images de globules blancs réellement colorés, les images du premier ensemble de données d'apprentissage n'étant pas appariées avec les images du second ensemble de données d'apprentissage ; et
    appliquer, par le dispositif d'apprentissage (120), un processus d'apprentissage qui utilise les réseaux accusatoires génératifs au premier ensemble de données d'apprentissage et au second ensemble de données d'apprentissage pour générer l'algorithme de conversion d'image,
    les réseaux adversaires génératifs comprennent un premier réseau génératif configuré pour générer une pluralité d'images DHM virtuellement tachées et un premier réseau discriminant configuré pour faire la distinction entre la pluralité d'images DHM virtuellement tachées et les images du deuxième ensemble de données d'apprentissage,
    dans lequel les réseaux adversaires génératifs comprennent en outre un deuxième réseau génératif configuré pour générer une pluralité d'images DHM virtuelles et un deuxième réseau discriminant configuré pour faire la distinction entre la pluralité d'images DHM virtuelles et les images du premier ensemble de données d'apprentissage.
  2. La méthode mise en oeuvre par ordinateur de la revendication 1, dans laquelle la coloration virtuelle des
    images DHM comprennent la colorisation des globules blancs en fonction du type de chaque globule blanc.
  3. La méthode mise en oeuvre par ordinateur de la revendication 2, comprenant en outre la quantification d'une
    rapport des types de globules blancs dans un échantillon en fonction de la coloration des globules blancs.
  4. La méthode mise en oeuvre par ordinateur de la revendication 1, dans laquelle une ou plusieurs cellules
    comprennent des cellules bactériennes et la coloration des cellules bactériennes dépend du type de bactérie.
  5. La méthode mise en oeuvre par ordinateur de la revendication 1, dans laquelle le processus d'apprentissage utilise une perte contradictoire et une perte de cohérence de cycle pour générer l'algorithme de conversion d'image.
  6. La méthode mise en oeuvre par ordinateur de la revendication 1, comprenant en outre l'affichage de l'image DHM virtuellement colorée par le biais d'une interface utilisateur graphique.
  7. Appareil comprenant un système informatique (610) configuré pour exécuter la méthode de l'une des revendications 1 à 6.
  8. Système de visualisation cellulaire (100) configuré pour produire des images numériques virtuellement colorées
    de microscopie holographique (DHM), comprenant :
    un dispositif d'apprentissage (120) configuré pour générer un algorithme de conversion d'image à l'aide de données non appariées
    les ensembles de données non appariés comprennent un premier ensemble de données d'apprentissage comprenant des images DHM
    des cellules individuelles et un deuxième ensemble de données d'apprentissage comprenant des images de cellules effectivement colorées ; et
    un dispositif de coloration virtuelle (130) configuré pour recevoir une image DHM d'une ou plusieurs cellules et appliquer l'algorithme de conversion d'image
    à l'image DHM pour produire une image DHM virtuellement tachée,
    caractérisé en ce que,
    le dispositif d'apprentissage est configuré avec des réseaux adversaires génératifs pour générer l'algorithme de conversion d'image,
    comprenant un premier réseau génératif configuré pour générer des exemples d'images DHM virtuellement tachées et un premier réseau de discrimination configuré pour distinguer les exemples d'images DHM virtuellement tachées des images dans le deuxième ensemble de données d'apprentissage ; et un deuxième réseau génératif configuré pour générer des exemples d'images DHM virtuelles et un deuxième réseau de discrimination configuré pour distinguer les exemples d'images DHM virtuelles des images dans le premier ensemble de données d'apprentissage.
  9. Le système de visualisation cellulaire de la revendication 8, comprenant en outre un dispositif DHM (110)
    configuré pour fournir le premier ensemble de données d'apprentissage au dispositif d'apprentissage.
  10. Le système de visualisation cellulaire de la revendication 9, dans lequel le dispositif DHM (110) est en outre
    configuré pour fournir l'image DHM d'une ou plusieurs cellules au dispositif de coloration virtuel.
  11. Produit programme d'ordinateur comprenant des instructions qui mettent en oeuvre la méthode selon l'une des revendications 1 à 6.
  12. Support lisible par ordinateur comprenant le produit de programme informatique selon la revendication 11.
EP18780265.7A 2017-09-19 2018-09-06 Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs Active EP3685355B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762560272P 2017-09-19 2017-09-19
PCT/US2018/049702 WO2019060141A1 (fr) 2017-09-19 2018-09-06 Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs

Publications (3)

Publication Number Publication Date
EP3685355A1 EP3685355A1 (fr) 2020-07-29
EP3685355B1 true EP3685355B1 (fr) 2024-07-10
EP3685355C0 EP3685355C0 (fr) 2024-07-10

Family

ID=63720759

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18780265.7A Active EP3685355B1 (fr) 2017-09-19 2018-09-06 Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs

Country Status (5)

Country Link
US (1) US11169484B2 (fr)
EP (1) EP3685355B1 (fr)
JP (2) JP2020534627A (fr)
CN (1) CN111095360B (fr)
WO (1) WO2019060141A1 (fr)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111699510A (zh) * 2018-02-12 2020-09-22 豪夫迈·罗氏有限公司 数字病理学图像的变换
WO2020247409A1 (fr) * 2019-06-02 2020-12-10 Essenlix Corporation Coloration rapide de biomatériaux améliorée par un traitement d'image et l'intelligence artificielle
CN110648309B (zh) * 2019-08-12 2024-05-28 平安科技（深圳）有限公司 基于条件生成对抗网络合成红细胞图像的方法及相关设备
CN114365179B (zh) * 2019-09-19 2025-08-19 香港科技大学 无载玻片组织学成像方法和系统
KR102302333B1 (ko) 2019-11-05 2021-09-16 주식회사 토모큐브 3차원 굴절률 영상과 딥러닝을 활용한 비표지 방식의 3차원 분자 영상 생성 방법 및 장치
EP3839884A1 (fr) * 2019-12-20 2021-06-23 Fundación Tecnalia Research & Innovation Procédé, système et produit-programme d'ordinateur pour la transformation d'images
US20230394716A1 (en) * 2022-06-06 2023-12-07 Pictor Labs, Inc. Method of generating inference-based virtually stained image annotations
CN116165044A (zh) * 2022-12-16 2023-05-26 麦克奥迪实业集团有限公司 结合相衬技术和扫描仪对病理切片的数字染色方法和系统
CN117829259B (zh) * 2024-01-12 2025-06-10 北京航空航天大学 训练方法、虚拟染色方法和虚拟染色系统

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101197768A (zh) * 2006-12-04 2008-06-11 西门子公司 提高移动自组网安全性的方法和节点设备
CN103154732B (zh) * 2010-08-05 2015-11-25 艾博特健康公司 通过显微图像用于全血样品自动分析的方法和装置
WO2012078796A1 (fr) 2010-12-07 2012-06-14 Life Technologies Corporation Coloration cellulaire virtuelle
US10032064B2 (en) * 2012-08-21 2018-07-24 Cambridge Research & Instrumentation, Inc. Visualization and measurement of cell compartments
US9786050B2 (en) 2013-03-15 2017-10-10 The Board Of Trustees Of The University Of Illinois Stain-free histopathology by chemical imaging
US9135682B2 (en) * 2013-03-15 2015-09-15 Indian Institute Of Technology Delhi Image recovery from single shot digital hologram
JP6437184B2 (ja) * 2013-06-05 2018-12-12 学校法人関西医科大学 舌上皮幹細胞の単離方法
US10176363B2 (en) 2014-06-16 2019-01-08 Siemens Healthcare Diagnostics Inc. Analyzing digital holographic microscopy data for hematology applications
US10614287B2 (en) * 2014-06-16 2020-04-07 Siemens Healthcare Diagnostics Inc. Virtual staining of cells in digital holographic microscopy images using general adversarial networks
CN107111874B (zh) * 2014-12-30 2022-04-08 文塔纳医疗系统公司 用于共表达分析的系统和方法

Also Published As

Publication number Publication date
JP2022107671A (ja) 2022-07-22
US20200183327A1 (en) 2020-06-11
EP3685355C0 (fr) 2024-07-10
US11169484B2 (en) 2021-11-09
EP3685355A1 (fr) 2020-07-29
JP7439165B2 (ja) 2024-02-27
JP2020534627A (ja) 2020-11-26
CN111095360B (zh) 2023-09-05
WO2019060141A1 (fr) 2019-03-28
CN111095360A (zh) 2020-05-01

Similar Documents

Publication Publication Date Title
US10614287B2 (en) Virtual staining of cells in digital holographic microscopy images using general adversarial networks
EP3685355B1 (fr) Coloration virtuelle de cellules au sein d'images de microscopie holographique numérique à l'aide de réseaux antagonistes génératifs
Moen et al. Deep learning for cellular image analysis
Heebner et al. Deep learning-based segmentation of cryo-electron tomograms
US11227386B2 (en) Identifying the quality of the cell images acquired with digital holographic microscopy using convolutional neural networks
Wagner et al. Classification and segmentation of nanoparticle diffusion trajectories in cellular micro environments
Ong et al. Digitalized organoids: integrated pipeline for high-speed 3D analysis of organoid structures using multilevel segmentation and cellular topology
JP2017519985A (ja) 血液学用デジタルホログラフィ顕微鏡検査データ分析
JP2022506135A (ja) ヒトの寄与を組み込む反復的深層学習フローを使用した顕微鏡画像内の３ｄ細胞間構造のセグメント化
JP7538718B2 (ja) 粒子解析用データ生成方法、粒子解析用データ生成プログラム、及び粒子解析用データ生成装置
Agus et al. GLAM: Glycogen-derived Lactate Absorption Map for visual analysis of dense and sparse surface reconstructions of rodent brain structures on desktop systems and virtual environments
Mao et al. Deep learning in fluorescence imaging and analysis
RU2732895C1 (ru) Метод для выделения и классификации типов клеток крови с помощью глубоких сверточных нейронных сетей
Pan et al. Accurate detection and instance segmentation of unstained living adherent cells in differential interference contrast images
Chomean et al. AI-Powered Body Fluid Cell Classification: Development and Validation Using Roboflow and YOLOv11n Framework
Kiflie et al. Sputum smears quality inspection using an ensemble feature extraction approach
Kovacs et al. Single-cell classification based on label-free high-resolution optical data of cell adhesion kinetics
Bearer Overview of image analysis, image importing, and image processing using freeware
Zinchuk et al. Machine learning for analysis of microscopy images: a practical guide and latest trends
Kim et al. Analyzing 3D cell data of optical diffraction tomography through volume rendering
Hasija Artificial Intelligence and Digital Pathology Synergy: For detailed, accurate and predictive analysis of WSIs
Hardo et al. Quantitative Microbiology with Microscopy: Effects of Projection and Diffraction
Chen et al. Interpretable Deep Learning Enables Reliable and Label-Efficient Fluorescence Imaging
Poonam et al. Evaluating graphical perception capabilities of Vision Transformers
Hamberg Deep Learning for Bright Field-Based Cell Viability Prediction in 3D Spheroids: Reducing the cost of oncology research with generative AI

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200317

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220509

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G03H 1/00 20060101ALN20240223BHEP

Ipc: G01N 15/10 20060101ALN20240223BHEP

Ipc: G06V 10/776 20220101ALI20240223BHEP

Ipc: G06F 18/2413 20230101ALI20240223BHEP

Ipc: G06V 10/764 20220101ALI20240223BHEP

Ipc: G01N 15/1433 20240101ALI20240223BHEP

Ipc: G06V 20/69 20220101ALI20240223BHEP

Ipc: G06V 10/82 20220101ALI20240223BHEP

Ipc: G01N 33/49 20060101ALI20240223BHEP

Ipc: G01N 15/1434 20240101ALI20240223BHEP

Ipc: G03H 1/04 20060101ALI20240223BHEP

Ipc: G01N 33/50 20060101ALI20240223BHEP

Ipc: G06T 7/00 20170101ALI20240223BHEP

Ipc: G06T 11/00 20060101AFI20240223BHEP

INTG Intention to grant announced

Effective date: 20240319

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018071600

Country of ref document: DE

U01 Request for unitary effect filed

Effective date: 20240805

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT RO SE SI

Effective date: 20240902

U20 Renewal fee for the european patent with unitary effect paid

Year of fee payment: 7

Effective date: 20240919

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241011

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241010

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241010

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241110

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20241011

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240710

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20250411

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240906

U20 Renewal fee for the european patent with unitary effect paid

Year of fee payment: 8

Effective date: 20250919

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20251002

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180906

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180906