WO2023037101A1 - Procédé d'obtention d'un résultat de test à partir d'une bandelette réactive pour tester un échantillon liquide - Google Patents

Procédé d'obtention d'un résultat de test à partir d'une bandelette réactive pour tester un échantillon liquide Download PDF

Info

Publication number
WO2023037101A1
WO2023037101A1 PCT/GB2022/052268 GB2022052268W WO2023037101A1 WO 2023037101 A1 WO2023037101 A1 WO 2023037101A1 GB 2022052268 W GB2022052268 W GB 2022052268W WO 2023037101 A1 WO2023037101 A1 WO 2023037101A1
Authority
WO
WIPO (PCT)
Prior art keywords
test
input image
region
sample
estimate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/GB2022/052268
Other languages
English (en)
Inventor
Cecilia Margareta LINDGREN
Christoffer NELLAKER
Mitchell DAWSON
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Oxford University Innovation Ltd
Original Assignee
Oxford University Innovation Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oxford University Innovation Ltd filed Critical Oxford University Innovation Ltd
Publication of WO2023037101A1 publication Critical patent/WO2023037101A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/8483—Investigating reagent band
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/8483—Investigating reagent band
    • G01N2021/8488—Investigating reagent band the band presenting reference patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to methods of obtaining a test result from a test strip.
  • the disclosure is particularly applicable to test strips for testing urine.
  • Test strips for urine analysis are used to test for a range of conditions. These include urinary tract infections, chronic kidney disease, pregnancy, and a range of metabolic conditions. Such tests may be used in a clinical setting or by the wider public as a health product with direct data feedback to healthcare providers.
  • Test strips typically contain plural test regions that change colour after the test strip has been dipped in a sample.
  • the colour of each test region provides information about a concentration of a particular target substance in the sample.
  • test strips can be done manually by comparing test regions with corresponding colour charts. This approach requires no additional hardware or software but is time consuming and prone to human error. Systems for automatically interpreting test strips are also known but these tend to require sophisticated additional hardware and are expensive and/or complicated to operate.
  • a computer-implemented method of obtaining a test result from a test strip for testing a liquid sample by contact between the test strip and the liquid sample wherein: the test strip comprises at least one test region and at least one reference region; each test region is configured to undergo a visually perceptible change after contact between the test strip and a liquid sample, the visually perceptible change being dependent on a target characteristic of the sample associated with the test region; and the method comprises: receiving input data representing an input image or a sequence of input images of the test strip captured after contact between the test strip and the sample; processing the input data to extract an estimate of the target characteristic associated with each test region by analysing the test region and one or more reference regions for each input image; and outputting a test result comprising the estimate of the target characteristic for each test region.
  • test regions and reference regions reduces any negative influence on accuracy from variable lighting conditions.
  • Variations in the appearance of reference regions provide information about lighting conditions more effectively than variations in the appearance of test regions because the influence of lighting conditions contributes to the appearance of the reference regions to a greater extent than to the appearance of test regions (which also change appearance due to interaction with the sample).
  • the approach is less time-consuming and/or more reliable than manual interpretation of the test regions, for example based on manual comparison of test regions with a colour chart.
  • the inventors have found that the approach is effective using input images corresponding to a range of different exposure times. The approach thus provides greater robustness and/or ease of use in comparison to alternative approaches that rely on only using input images obtained at specified fixed times after an initial sample exposure.
  • Processing a sequence of input images of the test strip (e.g. in the form of a video) provides richer information about the test strip than is available from a single image.
  • the evolution with time of colours in test regions can be used, for example, to automatically detect when a stable colour has been reached.
  • the or each reference region is a region outside of any test region and configured not to undergo any significant visually perceptible change on contact with the sample.
  • Each reference region may, for example, be a non-functional portion of the test strip, not intended by the manufacturer to be used in analysis procedure.
  • One or more regions between test regions may be used as reference regions, for example. This approach means the method is applicable to a wide range of commercially available test strips. It is not necessary for a manufacturer to specially adapt a test strip for use in the method.
  • locations of the or each test region and of the or each reference region are detected automatically.
  • a user does not therefore need to take any special action in relation to defining and/or locating test regions or reference regions.
  • a user can simply record a video of the whole strip.
  • the test strip comprises a plurality of reference regions.
  • plural reference regions are provided in such a way that a reference region is present adjacent to each test region.
  • reference regions and test regions are provided in an alternating sequence along a longitudinal axis of the test strip. This approach ensures that a reference region is present near to each test region, thus allowing a suitable measure of lighting conditions to be obtained for each test region even in the case where lighting conditions vary spatially over the test strip.
  • the processing of the input (e.g. video) data uses a machine learning algorithm trained to estimate a target characteristic of a sample from an image of a single test region in a single input image.
  • the machine learning algorithm may be used to extract the following in each input image: a test region feature vector representing each test region in the input image; and a reference region feature vector representing each reference region in the input image.
  • the inventors have found that such use of feature vector provides a particularly robust and flexible approach to analysing test strips.
  • the intrinsically spatially aware nature of the approach provides distinct improvements relative to alternatives involving averaging of colours in test regions.
  • the disclosed approach works particularly well in comparison to spatial averaging based alternatives where test regions are used for testing for blood in urine, for example when determining whether a sample contains haemolysed vs non-haemolysed blood, which requires identifying speckling pattern on the test square. This necessitates a feature extraction method which can encode spatial information.
  • Using spatially aware image features also helps to interpret larger scale variations in lighting conditions, such as shadowing.
  • the localisation of the test square may be slightly inaccurate, in which case pixels from a background behind the test strip and/or reference regions may be found in the detected test region.
  • a spatially aware algorithm is better able to ignore this extraneous information.
  • the sample may not have completely contacted a test region and/or not reacted uniformly, in which case it is desirable for the algorithm to ignore parts of the test region (e.g., the areas that have been underexposed to the sample and/or where the reaction has progressed more weakly).
  • there is colour bleed from a test region into a neighbouring reference region This can change the colour of the reference region making it potentially less useful for determining lighting conditions. This is solved by having a model which can learn to ignore or interpret the lighting conditions even with colour bleed.
  • test region and reference region feature vectors may be extracted using a feature vector extraction portion of the same trained machine learning algorithm.
  • the inventors have found that using the same machine learning algorithm to extract feature vectors for the test regions and the reference regions is highly effective in providing a useful representation of information both for the test regions and the reference regions. This occurs despite the fact that the reference regions are not configured to provide any information about a target characteristic of a sample or, indeed, to change colour significantly on contact with the sample.
  • the approach is convenient to implement because it does not require separate trained machine learning algorithms to process the test regions compared to the reference regions.
  • the approach can be implemented using relatively modest hardware, such as a typical smart phone.
  • Using the same algorithm for getting the feature vectors from both the test regions and the reference regions is improved convenience and reduced algorithm memory footprint.
  • a first algorithm is used to extract feature vectors for the test regions and a second algorithm, different from the first algorithm, is used to extract feature vectors for the reference regions.
  • Using "hard-coded” features such as “brightness” or “hue” may be algorithmically simpler, but would be fundamentally limited in its ability to represent lighting conditions and colour.
  • the inventors’ approach has been demonstrated to be able to delineate the relationships between lighting conditions and colour over a large range of test strips.
  • the method further comprises estimating an exposure time for each input image, the exposure time being the time interval between a sample application time, defined as the time the biological sample is applied to the test strip, and a input image capture time for the input image, defined as the time when the input image is captured by an image capture device.
  • the visual appearance of test regions will change as a function of exposure time. Estimating the exposure time thus provides information that contributes to correct interpretation of the test regions.
  • the estimation of exposure time comprises inputting the extracted feature vectors and the input image capture time to a trained classifier that has been trained by supervised learning to estimate exposure time based on input extracted feature vectors and input image capture times.
  • a test such as Leukocytes takes a long time for a positive test result to appear. Therefore a white LEU square indicates either a negative result at any time or an early time positive result. Similarly, a dark LEU square indicates a longer exposure time, because it has needed time to show.
  • the algorithm is leveraging this information from multiple test regions which change colour at different rates.
  • the estimation of exposure time for each input image after a first input image comprises combining an estimate of the exposure time obtained for the input image and an estimate of exposure time obtained for a preceding input image.
  • the estimate of the exposure time for each input image implicitly provides an estimate of the sample application time (which is equal to the difference between the exposure time for a input image and the input image capture time for that input image).
  • Each estimate of an exposure time for an input image thus provides additional information about the sample application time and can be used to improve an overall estimate of the sample application time, which in turn improves the estimate of exposure time for each new input image that is processed. A confidence in the estimate of the exposure time thus increases from input image to input image.
  • the estimation of the target characteristic associated with each test region for each input image after a first input image comprises combining an estimate of the target characteristic obtained for the input image and an estimate of the target characteristic obtained for a preceding input image.
  • the continuous improvement in the estimate is quantified by calculating a confidence score for the estimate of the target characteristic associated with one or each of two or more test regions in each input image. The calculated confidence score can be used as a reference to automatically stop processing of the input (e.g. video) data and/or the image (e.g. video) capture process when the estimate of the target characteristic for one or each of two or more test regions is sufficiently good.
  • the method may comprise automatically stopping the processing of the input data and/or an image capture process for providing the input data when the confidence score exceeds a predetermined threshold value.
  • This approach avoids inaccuracies associated with prematurely stopping the processing and/or image capture and avoids continuing with either for a period that is unnecessarily long.
  • Figure 1 is a flow chart showing a method of obtaining a test result from a test strip
  • Figure 2 depicts a portion of an example test strip
  • Figure 3 is a flow chart showing example sub-steps in a step of identifying regions of interest in the method of Figure 1;
  • Figure 4 schematically depicts a sequence of input images of an image capture process of a test strip
  • Figure 5 a timeline showing events from a sample application time through to capture of multiple input images of input data
  • Figure 6 depicts a boundary box representing a detected position of a test strip
  • Figure 7 depicts detection of an orientation of a detected test strip
  • Figure 8 depicts the detection test strip of Figure 7 after processing to align the test strip with a reference (horizontal axis);
  • Figure 9 depicts bounding boxes around test regions
  • Figure 10 depicts the addition of bounding boxes around reference regions
  • Figure 11 depicts extracting of feature vectors representing the test regions and reference regions of Figure 10;
  • Figures 12-14 and 18-20 are example input images containing a test strip on different respective surfaces and/or subjected to different lighting conditions
  • Figures 15-17 and 21-23 respectively show outputs after processing of the input images of Figures 12-14 and 18-20 using methods of the present disclosure
  • Figure 24 is a graph depicting performance of methods of estimating exposure time of the present disclosure.
  • Figures 25 and 26 are bar charts showing the influence of taking into account the reference regions and/or exposure time in methods of the present disclosure, with Figure 25 showing data for the case where a test region is configured to detect nitrites (NIT) and Figure 26 showing data for a case where the test region is configured to detect leukocytes (LEU);
  • NIT nitrites
  • LEU leukocytes
  • Figure 27 is a bar chart showing the accuracy with which the method can predict the correct corresponding target characteristic (bottle label), within one bottle label error, when applied to a single image of a test strip having test regions corresponding to nine reactants typically found on urinalysis dipsticks (LEU, NIT, URO, PRO, pH, SG, KET, BIL and GLU);
  • Figures 28-30 depict confusion matrices showing performance of methods of the present disclosure.
  • Figures 31 and 32 are scatter plots showing the accuracy with which methods of the present disclosure can estimate a target characteristic (bottle label) at a given sample exposure time for the cases of specific gravity (SG) and bilirubin (BIL), respectively.
  • SG specific gravity
  • BIL bilirubin
  • the present disclosure describes methods that can be computer-implemented.
  • Each computer-implemented step may be performed by a computer in the most general sense of the term, meaning any device capable of performing the data processing steps of the method, including dedicated digital circuits.
  • the computer may comprise various combinations of computer hardware, including for example CPUs, RAM, SSDs, motherboards, network connections, firmware, software, and/or other elements known in the art that allow the computer hardware to perform the required computing operations.
  • the required computing operations may be defined by one or more computer programs.
  • the one or more computer programs may be provided in the form of media or data carriers, optionally non-transitory media, storing computer readable instructions. When the computer readable instructions are read by the computer, the computer performs the required method steps.
  • the computer may consist of a self-contained unit, such as a general-purpose desktop computer, laptop, tablet, mobile telephone, or other smart device.
  • the computer may consist of a distributed computing system having plural different computers connected to each other via a network such as the internet or an intranet.
  • Figure 1 depicts a framework of a method of analysing a test strip according to the disclosure.
  • the analysis may be performed to obtain a test result for a liquid (e.g. biological) sample that has been applied to the test strip (e.g., by making contact between the test strip and the sample, such as by dipping the test strip into the sample).
  • An example test strip 2 is shown schematically in Figure 2.
  • the test strip comprises at least one test region 11-14 and at least one reference region 21-24.
  • the test strip 2 will contain a plurality of the test regions 11-14 and a plurality of the reference regions 21-24, as in the example shown.
  • Each test region 11-14 is configured to undergo a visually perceptible after contact between the test strip and the sample.
  • the visually perceptible change is dependent on a target characteristic of the sample and therefore provides information about the target characteristic.
  • Arrangements of the present disclosure are particularly applicable to the case where the sample is a urine sample, but other types of sample could also be used, including non-biological samples.
  • the approach may be used for example in the context of dipsticks for swimming pools (e.g. for chlorine or pH measurements), detection of hardness/softness and/or ammonia levels in water, testing of frying oil, or testing of liquids in beer brewing (e.g. to test pH).
  • the reference region 21- 24 is a region outside of any test region 11-14 and configured not to undergo any significant visually perceptible change on contact with the sample.
  • the visually perceptible change associated with each test region may be a change in colour for example.
  • the target characteristic may be a concentration of a component of interest in the sample.
  • Example components of interest in the case of application to a urine sample include proteins, glucose, ketones, haemoglobin, bilirubin, urobilinogen, acetone, nitrite, leukocytes, microalbumin, creatinine, and pathogens.
  • Example target characteristics also include pH and specific gravity.
  • Test strips are typically designed to allow a human operator to determine a test result for each test region by comparing the appearance of the test region (e.g., its colour) against a calibration scale (e.g., a colour scale) provided by a manufacturer of the test strip.
  • a calibration scale e.g., a colour scale
  • This process can be time consuming because the change in visual appearance of the test regions takes time to stabilise and it can be difficult for a user to know when it is appropriate to compare the exposed sample to the calibration scale. This issue is compounded by the intrinsic variation in development rates of the different test regions. Manual interpretation may also be prone to error because the appearance of the test regions may vary significantly as a function of lighting conditions.
  • the method described with reference to Figure 1 aims to improve over the abovedescribed manual approach by providing an automated alternative that can be implemented with modest hardware resources (e.g., by a typical smart phone) and which provides a reliable result with minimal delay.
  • the method is performed by a computer, which can take a wide variety of forms as explained above.
  • the method can be performed in real time (e.g. with the processing of input data overlapping with capturing of the input data) or offline.
  • the processing can be performed by a single device (e.g. the same device that performs the image/video capture) or by more than one device (e.g. a local device may perform the image/video capture and a remote device may perform some or all of the data processing).
  • a cloud-based server may be used to perform some or all of the data processing for example.
  • a mobile phone or tablet device may be used to perform the image/video capture.
  • the method comprises receiving input data.
  • the input data may comprise a single input image or sequence of input images. Each input image may be referred to as a frame. Each input image in a sequence of input images may be associated with a time stamp indicating a relative time of capture of the respective input image. The sequence of input images may be regularly or irregularly spaced in time.
  • the input data represents an input image or a sequence of input images 30 of the test strip 2 captured (e.g. by a video recording) after contact between the test strip 2 and the sample, i.e. after the sample has been applied to the test strip 2, as depicted schematically in Figure 4.
  • a sample application time defined as the time point at which the sample is applied to the test strip 2 (e.g., when the test strip is dipped into the sample), is denoted t 0 .
  • the input images may be captured, for example, by the camera of a smart phone.
  • Each input image has an associated time stamp tj indicating the time the input image was captured (which may be referred to as a input image capture time).
  • t x a first input image captured by the camera after contacting the test strip 2 with a sample may have a time stamp denoted as t x .
  • Subsequent input images will have a series of time stamps, typically at regularly spaced intervals, up to a most recently obtained input image having a time stamp t n .
  • the input data is processed to extract an estimate of the target characteristic associated with each test region.
  • an estimate of each of two or more target characteristics may be extracted, each target characteristic being associated with a different one of the test regions.
  • the processing extracts the estimate of the respective target characteristic by analysing the test region and one or more reference regions for each of multiple input images.
  • the extracted estimate or estimates is or are output as a test result in step S6 (e.g., as a data stream or to a user via a display or other interface).
  • Step S2 comprises analysing an input image to identify regions of interest within the input image.
  • the regions of interest may include one or more test regions and one or more reference regions.
  • a framework for an example approach is shown schematically in Figure 3. Strip detection
  • step S2-1 of the example approach video data corresponding to the input image is analysed to detect the presence of a test strip 2.
  • a module used to perform this processing may be referred to as a strip detection module.
  • the strip detection module may determine whether the input image contains a test strip, what brand of test strip is found, output a bounding box defining the location of the detected test strip, and provide a confidence score that a strip has been detected.
  • a strip brand is manually input by a user.
  • the strip detection module uses a convolutional neural network (CNN)-based object detector.
  • CNN convolutional neural network
  • One of many off-the-shelf architectures may be used, such as SSD, YOLO and R-CNN.
  • the network is trained using a large number of images of strips with bounding boxes of the strips manually annotated by human annotators.
  • An output from step S2-1 for an input image may be a portion of the input image tightly cropped by a bounding box 31 around the detected test strip 2.
  • Step S2-2 receives as input the tightly cropped image of the detected test strip 2 from S2-1.
  • the test strip 2 may be in any orientation in the image at this stage.
  • Step S2-2 detects test regions on the test strip 2 and uses the locations of the test regions to determine an orientation of the strip.
  • test region detection module recognises test regions visible on the test strip and surrounds each one with a close-fitting bounding box 32. This is depicted schematically in Figure 7, where each bounding box 32 surrounds a different test region.
  • the bounding boxes 32 will be aligned roughly along a longitudinal axis of the test strip 2 and therefore serve to determine the angle and, optionally, the midpoint of the test strip.
  • step S2-3 geometrical transformations are applied to the image using the locations of the detected test regions to align the test strip along a reference axis of the image.
  • a result of this operation is depicted schematically in Figure 8 for the case where the reference axis is the horizontal axis.
  • the image may also be geometrically warped to ensure that a perspective of the test strip 2 in the image corresponds to a reference perspective.
  • test region detector module may be implemented using CNN-based object detector, chosen from the numerous variations of SSD, YOLO and R- CNN architectures for example.
  • the network may be trained from a large number of annotated training examples consisting of images of test strips with bounding boxes of their test regions manually annotated by human annotators.
  • shallow image features are used to register the image to a standard size and configuration.
  • the cropped strip image could be fed to a classifier which is able to regress the coordinates of a point on the centre of the strip and the line of best fit along which the strip lies. This would provide an alternative way to find the angle of the strip.
  • Shallow image features could then be used to determine the start and end points of the strip, and scale accordingly.
  • a further alternative approach is to use the coordinates of bounding boxes of detected test regions to deduce the orientation of the test strip using trigonometry.
  • the inventors have found that the bounding boxes are detected extremely accurately, which allows determination of the test strip to be obtained with correspondingly high accuracy.
  • An output of the above strip alignment processing is an image of the test strip that has been rotated to be horizontally aligned.
  • Whether the strip is upside-down or not can be determined based on a geometrical balance of features relative to a midpoint. For example, where more test regions are expected to be present on one side of the midpoint than on the other side of the midpoint, for example because one end of the strip, which may be a logo end, allows space for a user to hold the strip, this can be detected and used to orient the strip (e.g.
  • an expected ordering of the test regions relative to a reference end of the test strip can be used to determine which test region belongs to which reactant (e.g. for one known commercial device, the ordering would be, left to right, GLU, BIL, KET, ..., NIT, LEU).
  • step S2-4 the test region detection module is reapplied to obtain tighter bounding boxes for the test regions.
  • the bounding boxes are tighter because the principal axes of the bounding boxes can be better aligned with the principal axes of the test regions. Having bounding boxes that fit the test regions more tightly helps to remove as much extraneous information as possible before later processing steps are performed on the isolated test regions.
  • the test region detection module may also be configured to output a confidence score for each identification of a test region.
  • the rotation of the strip may be omitted, which may result in looser fitting bounding boxes.
  • the regions inside the bounding boxes would include pixels from neighbouring reference regions, background and potentially even other test regions, but a large portion of each region would still correspond to the test region of interest and may therefore be useable.
  • the test region detection module may be configured to apply rotated bounding boxes, but this would complicate the detection architecture.
  • Step S2-5 receives as input bounding box coordinates of test regions identified in S2-4.
  • Example bounding boxes 32 are depicted in Figure 9 for four example test regions
  • Bounding boxes 33 of reference regions 21-24 are determined such that the reference regions 21-24 lie outside of any test region 11-14. This can be achieved based on the locations of the test regions and the geometry of the test strip.
  • the reference regions 21-24 can be positioned, for example, so as to be in line with the test regions 11-14 along a longitudinal axis of the test strip and displaced relative thereto so that they do not overlap with the test regions 11-14.
  • the test regions 11-14 are spaced apart regularly along the longitudinal axis of the test strip and the reference regions are located at different positions along the longitudinal axis. At least a subset of the reference regions are positioned in between respective pairs of test regions 11-14.
  • a reference region is provided adjacent to each test region.
  • reference region 21 is adjacent to test region 11
  • reference region 22 is adjacent to test region 12
  • reference region 23 is adjacent to test region 13
  • reference region 24 is adjacent to test region 14.
  • the reference regions 21-14 and test regions 11-14 may thus be provided in an alternating sequence along the longitudinal axis of the test strip.
  • test region 11-14 and each reference region 21-24 may thus be detected automatically.
  • Figures 12-23 depict application of the above methodology to input images containing a test strip on various surfaces and under different lighting conditions.
  • test strip is identified (e.g., as in step S2-1 described above) and aligned horizontally (e.g., as in steps S2-2 and S2-3 described above). Locations of test regions and reference regions are determined (e.g., as in steps S2-4 and S2-5 described above). The orientation of the stick may then be determined to deduce a sequence of the test regions.
  • step S3 of the method comprises using a machine learning algorithm to extract, in each input image, a test region feature vector and a reference region feature vector.
  • a test region feature vector may be extracted for each of the test regions (FV2, FV4, FV6 and FV8 in the example shown) and a reference region feature vector may be extracted for each of the reference regions (FV1, FV3, FV5 and FV7 in the example shown).
  • Each feature vector is an information-rich representation of a test region or reference region.
  • the feature vector may be a reduced-dimension representation of the respective region.
  • the feature vector may represent the lighting, colour and/or texture properties of the respective region.
  • Each feature vector may be obtained using a trained machine learning algorithm.
  • the training machine learning algorithm may comprise a feature extractor (i.e., be configured to perform feature extraction).
  • the feature vectors are extracted using a feature vector extraction portion of a neural network that has been trained end-to-end to estimate the target characteristic of the sample from the image of a single test region in a single input image.
  • the feature vector extraction portion may, for example, comprise all but the final layer of a deep convolutional neural network which has been trained end-to-end to estimate the target characteristic from the image of a single test region in a single input image.
  • the processing of the video data uses a machine learning algorithm trained to estimate a target characteristic of a sample from an image including at least 90%, optionally the whole, of one side of the test strip in a single input image.
  • the trained machine learning algorithm may comprise a neural network trained end-to-end to estimate the target characteristic of the sample from the image including at least 90%, optionally the whole, of one side of the test strip in the single input image.
  • the test region and reference region feature vectors are extracted using a feature vector extraction portion of the same trained machine learning algorithm.
  • Deep neural networks can be trained to extract expressive and discriminative image features for a particular classification task. Treating reference regions (which predominantly represent information about lighting conditions) and test regions the same is simpler and faster than trying to extract out an explicit representation of lighting conditions and still provides good performance.
  • shallow image statistics such as mean pixel hue, median pixel brightness and/or mode green-channel intensity are obtained instead. This method requires no training and involves lower computational cost but is less flexible and/or robust than the above-described approach using machine learning.
  • Step S4 comprises estimating an exposure time for each input image.
  • the exposure time t exposure is the time interval between the sample application time t 0 and the input image capture time t, of the input image being considered.
  • the estimation of exposure time may comprise inputting the feature vectors FV 1 - FV8 extracted in step S3 and the input image capture time t, to a trained classifier that has been trained by supervised learning to estimate exposure time.
  • the extracted feature vectors and input image capture time are input as a single vector to the trained classifier (i.e., a concatenation of the feature vectors is input to the trained classifier).
  • Using feature vectors representing multiple different test regions is desirable because it increases the chances of the trained classifier receiving sufficient information to make a reliable estimate.
  • this approach takes account of the fact that the colour of some of the tests (such as pH) will stabilise too quickly to be useful for estimating exposure time. Other tests will take much longer to stabilise and therefore be more useful, such as those for urobilinogen or leukocytes.
  • the estimation of the exposure time is performed for plural input images, optionally for all of the input images.
  • Each estimation of the exposure time for an input image implicitly provides an estimate for the sample application time t 0 .
  • this information is used to improve the estimations of the exposure time for each input image after the first input image.
  • the estimation of the exposure time may include information from one or more (optionally all) preceding input images.
  • the estimation of the exposure time in a given input image may comprise, for example, updating an estimate of the sample application time and estimating the exposure time for an input image using the updated sample application time.
  • the estimation of exposure time for each input image after a first input image may thus comprise combining an estimate of the exposure time obtained for the input image (by processing video data corresponding to that input image) and an estimate of exposure time obtained for a preceding input image.
  • the estimation of the exposure time is thus expected to improve as more input images are analysed.
  • the method may output an exposure time confidence score representing a confidence in the estimation of the exposure time.
  • a user is encouraged to wait a set waiting period (e.g. 1 min) between when the test strip is first contacted with the sample and the start of video capture.
  • a set waiting period e.g. 1 min
  • An on-device timer may be provided to facilitate accurate timing. Although the overall test procedure may take longer, this approach may allow the user to film for a much shorter period, as test colours will take less long to stabilise after the waiting period.
  • step S4 For each input image, step S4 outputs the corresponding estimate of the exposure time.
  • Figure 24 is a graph depicting performance of the exposure time estimation methodology described above.
  • the graph shows estimations based on a single input image using glucose (GLU) test regions (“test squares”).
  • GLU glucose
  • test squares For a given image of a reactant test region, the model predicts the time which has elapsed since the test region was exposed to the sample. It can be seen that the model is able to estimate sample exposure time within a reasonable error margin.
  • This exposure time prediction accuracy can be further improved by using multi-frame multi-reactant exposure time prediction, where the elapsed time predictions are aggregated from multiple reactant test regions across multiple frames (images).
  • one or more of the extracted test region feature vectors, one or more of the extracted reference region feature vectors, and an estimated exposure time are used to obtain an estimate of a target characteristic corresponding to one or more test regions. This may be achieved for example by inputting this information to a trained machine learning algorithm (e.g. a neural network classifier).
  • a trained machine learning algorithm e.g. a neural network classifier
  • This approach is based on the expectation that the three most important factors which change a visual appearance (e.g. colour) of a test region are 1) the underlying reactant concentration (defined by the target characteristic of interest), 2) the exposure time, and 3) lighting conditions.
  • the exposure time can be estimated using multiple pieces of information from the test strip (as explained above).
  • the lighting conditions are encoded in the features extracted from the reference regions.
  • step S5 a multi-input image analysis approach is provided based on this insight.
  • this approach comprises processing the input data to improve an estimate of the target characteristic associated with a test region for each input image after a first input image by using the estimation of the target characteristic from a preceding input image.
  • the estimation of the target characteristic associated with each test region for each input image after a first input image may comprise combining an estimate of the target characteristic obtained for the input image (e.g., by processing video data corresponding to that input image) and an estimate of the target characteristic obtained for a preceding input image.
  • the approach may further output a confidence score associated with each improved estimate.
  • the processing of the input (e.g. video) data may comprise calculating a confidence score for the estimate of the target characteristic for each test region in each input image. As more input images are analysed the confidence scores will increase.
  • the processing of the input data is performed in real time and the method may comprise automatically stopping the image capture process (and/or the processing of the input data) based on the confidence scores.
  • the processing of the input data may be performed post hoc and automatically stopped based on the confidence scores.
  • Various approaches may be taken depending on the number of test regions being considered and the natures of the tests performed by the test regions.
  • the processing of the input data and/or capture process may be stopped when the confidence score associated with each test region exceeds a respective predetermined threshold (with the predetermined thresholds being potentially different for different test regions). In other arrangements, the processing of the input data and/or capture process is stopped when the confidence score for all of the test regions exceeds a single predetermined threshold level. In other arrangements, further facts may be taken into account when deciding when to stop the processing of the input data and/or capture process. For example, the system may require both that a predetermined number of input images is processed (e.g. greater than N input images) and that the confidence score exceeds a threshold level (e.g. above 90% confidence).
  • a threshold level e.g. above 90% confidence
  • a Bayesian inference update model is used.
  • the Bayesian model is answering the question "how often is the model correct, when it predicts this class at the current time point"?
  • the model will be updated with the estimated exposure time and estimated class (e.g. set of estimated target characteristics for the test regions on the test strip) obtained from the single input image analysis described above.
  • the prior distribution may be uniform or move to the prevalence of results in a population.
  • the likelihood distribution will be learned from the trained network by running a large amount of test data through the network and building this Bayesian head on top. Doing it this way allows the algorithm to improve its predictions as more samples (input images) are introduced. It is expected that the network will not make any high-confidence predictions with early timestamp data in some cases, as later timestamps will be needed for full exposure and proper separation and stabilisation of colours.
  • the predicted class may comprise an estimation of the bottle class, concentration of reactant, or test result.
  • Bottle class corresponds to the labels on the accompanying bottle that the test strips are stored in. These labels may look like (-, +, ++, +++) or (Neg, Trace, Pos) or (4.0, 4.5, ..., 7.5).
  • the scale unit is different for each reactant.
  • Predicting bottle class may comprise picking one class from 3-7 classes. This may also be referred to as "manufacturer standard colour reference".
  • Concentration of reactant refers to the underlying concentration of the reactant in the sample.
  • the scale unit is typically the same for each test and is often measured in mg/dL. Predicting concentration is regressing an actual concentration value of the reactant (and so is typically more precise than bottle label).
  • Test results refers to whether a sample is diagnostically positive or negative with regards to a sample. Positive refers to a value outside of the diagnostically healthy range. Predicting test result is picking from the classes 'positive' or 'negative'. This may also be referred to as "reactant presence positivity". Different users may want different information from the algorithm, so it is useful to be able to have metrics representing different prediction types.
  • Neural networks have a property of performing very confident predictions, and being confidently wrong.
  • the above-described Bayesian attachment for handling multiple network samples will help stop the case of a network confidently misidentifying test results early in exposure time, and instead will mean the network continues to run until a more reasonable exposure time has been reached.
  • this approach allows early stopping, as having a Bayesian estimate of belief of a test result means that a threshold can be chosen above which there is a high confidence that the result is correct.
  • the estimates of the target characteristics for test regions obtained from the most recent input image are output as a test result in step S6.
  • Figures 25-32 are graphs demonstrating various aspects of performance of embodiments of the present disclosure.
  • Figures 25 and 26 are bar charts showing the influence of taking into account the reference regions and/or exposure time.
  • Figure 25 shows data for the case where the test region is configured to detect nitrites (NIT).
  • Figure 26 shows data for the case where the test region is configured to detect leukocytes (LEU).
  • the bar charts show how using additional input features improves the accuracy with which the model can predict the corresponding target characteristic (bottle label) of titrated samples using a single input image (frame).
  • the leftmost bar in each plot (labelled “Test Squares Only”) shows the performance of a version of the model that only uses test regions to predict the bottle label. The prediction accuracy is shown to be improved by providing the model with more information.
  • the next bar to the right in each plot shows performance when the model is allowed to use the time for which the sample has been exposed as input.
  • the next bar to the right in each plot shows performance when the model uses the reference regions. A large increase is accuracy is observed, both relative to the case where test regions are used on their own and where test regions are used in combination with exposure time.
  • the rightmost bar shows how a further performance gain is achieved when the model uses both reference regions and exposure time.
  • Figure 27 is a bar chart showing the accuracy with which the method can predict the correct corresponding bottle label, within one bottle label error, when applied to test strips having test regions corresponding to nine reactants typically found on urinalysis dipsticks (LEU, NIT, URO, PRO, pH, SG, KET, BIL and GLU). Results were obtained from a titrated dataset from test strips exposed for between zero and three minutes. The model is shown to be able to achieve over 98% accuracy, within +/1 bottle label, for eight of the nine reactants.
  • LEU urinalysis dipsticks
  • Figures 28-30 depict confusion matrices showing performance of the method.
  • the confusion matrices show the distribution of bottle label predictions (estimates of a target characteristics) vs actual bottle label on titrated datasets using a single input image. It can be seen that the disclosed methodology achieves a top-1 bottle label prediction accuracy of 80.7% for specific gravity (SG), 91.4% for ketones (KET), and 93.9% accuracy for proteins (PRO). Furthermore, it can be seen that prediction errors are typically within one bottle label of the true bottle label. Accuracy as a function of exposure time
  • Figures 31 and 32 are scatter plots showing the accuracy with which the method can estimate a target characteristic (bottle label) for the cases of specific gravity (SG) and bilirubin (BIL), respectively, using a titrated dataset corresponding to different exposure times within three minutes of an initial sample exposure.
  • Each dot in the figures shows the model accuracy averaged over 30 sticks (test strips) at a given time after sample exposure.
  • model accuracy varies over time due to data noise, the moving average of the prediction accuracy over time stays relatively stable throughout the three-minute exposure time.
  • the dashed vertical line indicates the exposure time at which the reactant test is supposed to be read if following the dipstick (test strip) manufacturer’s instructions. This plot indicates that the test result can be predicted accurately for a wider range of times within the first three minutes after sample exposure.
  • Optional auxiliary modules are discussed below.
  • the modules may assist with making the approach practical for use with a real -world, time-pressured operator. These include methods for ensuring the algorithm is receiving good data, and is producing accurate predictions, as well as alerting the user that a particular behaviour would allow them to achieve results sooner.
  • the method may be configured to detect multiple different test strip brands and/or models. This makes it possible to assess whether a given test strip is supported. This could be used to prevent people trying to use the algorithm with test trips that are unsupported.
  • the method may be configured to constrain a user by providing only a subsection of the normal camera video screen on which they can film the stick. This may help to standardise the rotation of the data received, which makes training the algorithms easier.
  • test regions (squares) detected in each test strip image should align along a single principal (longitudinal) axis. This can be used to determine the quality of a detection of a test region, interpolate missing inlier locations, and decide whether to reperform a detection if results are unsatisfactory (optionally using standard deep learning augmentations such to improve performance).
  • performance may be improved by leveraging the idea of performing multiple tests at deployment.
  • a small set of augmentations such as left-right flipping or spatial and colour jitter
  • outlier detection is applied to these features to determine whether lighting conditions are very unusual, or a test region has been found to have a previously unseen colour. This could be done with a simple binary classifier for each test region and/or reference region, which could predict if a test square or reference region is an inlier or not.
  • test strip detection it is desirable to ensure that the entire test strip is within the image input image. It would therefore be useful to alert a user if the test strip moves out of input image. This can be achieved if we find no bounding boxes for test sticks in a particular detection, or if the bounding boxes start drifting too close to the edge of the screen. We would also stop further analysis for test regions that are unseen. Overexposed stick
  • test squares are occluded during filming by an object passing between the camera and the test stick then the user will need to alert to this, and information from the test square not passed further along into the algorithm. This can be achieved through noting bad results on test square detection, or through outlier detection once test squares have been extracted, and the colour does not match a colour we have seen before for a test square or reference region.
  • colour extraction of reference regions we would like to know if colour bleed has occurred from a test region into a reference region. This can be performed by a classifier on the reference region feature extracted images. There are alternative options for dealing with colour bleed, including ignoring it and training the classifier to deal with potentially compromised regions, or we could use the next nearest reference region to act as a reference region for the test square.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Theoretical Computer Science (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)

Abstract

Des procédés d'obtention d'un résultat de test à partir d'une bandelette réactive sont divulgués. La bandelette réactive comprend au moins une région de test et au moins une région de référence. Chaque région de test subit un changement visuellement perceptible après contact entre la bandelette réactive et un échantillon liquide. Le changement visuellement perceptible dépend d'une caractéristique cible de l'échantillon associé à la région de test. Des données d'entrée reçues représentent une image d'entrée ou une séquence d'images d'entrée de la bandelette réactive capturées après contact entre la bandelette réactive et l'échantillon. Les données d'entrée sont traitées pour extraire une estimation de la caractéristique cible associée à chaque région de test par analyse de la région de test et d'une ou de plusieurs régions de référence pour chaque image d'entrée.
PCT/GB2022/052268 2021-09-08 2022-09-07 Procédé d'obtention d'un résultat de test à partir d'une bandelette réactive pour tester un échantillon liquide Ceased WO2023037101A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB2112798.0 2021-09-08
GBGB2112798.0A GB202112798D0 (en) 2021-09-08 2021-09-08 Method of obtaining a test result from a test strip for testing a liquid sample

Publications (1)

Publication Number Publication Date
WO2023037101A1 true WO2023037101A1 (fr) 2023-03-16

Family

ID=78076861

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2022/052268 Ceased WO2023037101A1 (fr) 2021-09-08 2022-09-07 Procédé d'obtention d'un résultat de test à partir d'une bandelette réactive pour tester un échantillon liquide

Country Status (2)

Country Link
GB (1) GB202112798D0 (fr)
WO (1) WO2023037101A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4485361A1 (fr) * 2023-06-28 2025-01-01 Calpro AS Procédé de détermination d'un résultat de test à l'aide d'un dispositif mobile

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015134820A1 (fr) * 2014-03-05 2015-09-11 Scanadu Incorporated Concentration en analytes par quantification et interprétation de couleur
US20160080548A1 (en) * 2013-01-21 2016-03-17 Cornell University Smartphone-Based Apparatus and Method
US20170184506A1 (en) * 2015-12-29 2017-06-29 Pritesh Arjunbhai Patel Reagent test strips comprising reference regions for measurement with colorimetric test platform
EP3612963B1 (fr) * 2017-04-18 2021-05-12 Yeditepe Universitesi Analyseur biochimique basé sur un algorithme d'apprentissage automatique utilisant des bandelettes d'essai et un dispositif intelligent
US20210264604A1 (en) * 2018-06-22 2021-08-26 Oova, Inc. Methods, devices, and systems for detecting analyte levels

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160080548A1 (en) * 2013-01-21 2016-03-17 Cornell University Smartphone-Based Apparatus and Method
WO2015134820A1 (fr) * 2014-03-05 2015-09-11 Scanadu Incorporated Concentration en analytes par quantification et interprétation de couleur
US20170184506A1 (en) * 2015-12-29 2017-06-29 Pritesh Arjunbhai Patel Reagent test strips comprising reference regions for measurement with colorimetric test platform
EP3612963B1 (fr) * 2017-04-18 2021-05-12 Yeditepe Universitesi Analyseur biochimique basé sur un algorithme d'apprentissage automatique utilisant des bandelettes d'essai et un dispositif intelligent
US20210264604A1 (en) * 2018-06-22 2021-08-26 Oova, Inc. Methods, devices, and systems for detecting analyte levels

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HOQUE TANIA MARZIA ET AL: "Intelligent image-based colourimetric tests using machine learning framework for lateral flow assays", EXPERT SYSTEMS WITH APPLICATIONS, ELSEVIER, AMSTERDAM, NL, vol. 139, 26 July 2019 (2019-07-26), XP085851946, ISSN: 0957-4174, [retrieved on 20190726], DOI: 10.1016/J.ESWA.2019.112843 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4485361A1 (fr) * 2023-06-28 2025-01-01 Calpro AS Procédé de détermination d'un résultat de test à l'aide d'un dispositif mobile
WO2025003304A1 (fr) * 2023-06-28 2025-01-02 Calpro As Procédé de détermination d'un résultat de test à l'aide d'un dispositif mobile

Also Published As

Publication number Publication date
GB202112798D0 (en) 2021-10-20

Similar Documents

Publication Publication Date Title
Gehrig et al. E-raft: Dense optical flow from event cameras
Taylor et al. Causal video object segmentation from persistence of occlusions
US9536321B2 (en) Apparatus and method for foreground object segmentation
Milan et al. Challenges of ground truth evaluation of multi-target tracking
US8873852B2 (en) Method and apparatus for foreground object detection
US11915430B2 (en) Image analysis apparatus, image analysis method, and storage medium to display information representing flow quantity
CN108230352B (zh) 目标对象的检测方法、装置和电子设备
CN113411571B (zh) 一种基于滑窗梯度熵的视频帧清晰度检测方法
US20250131762A1 (en) Two-stage body pose estimation
CN116309781B (zh) 一种基于跨模态融合的水下视觉目标测距方法及装置
CN118379297B (zh) 聚四氟乙烯蚀刻管缺陷检测方法、装置、设备及存储介质
CN108230354A (zh) 目标跟踪、网络训练方法、装置、电子设备和存储介质
Chan et al. Real-time detection of aquarium fish species using YOLOv4-tiny on Raspberry Pi 4
EP3012779A1 (fr) Procédé et appareil permettant d'extraire des correspondances de caractéristique à partir d'images multiples
WO2024003900A1 (fr) Procédés de détection, de classification et d'évaluation de la motilité des spermatozoïdes dans des images ou des vidéos
CN110827269A (zh) 农作物生长变化状况检测方法、装置、设备及介质
US12073937B2 (en) Medical information processing apparatus
WO2023037101A1 (fr) Procédé d'obtention d'un résultat de test à partir d'une bandelette réactive pour tester un échantillon liquide
US20150254850A1 (en) System for detecting blood vessel structures in medical images
Shao et al. Hyper RPCA: joint maximum correntropy criterion and Laplacian scale mixture modeling on-the-fly for moving object detection
US20170214935A1 (en) Method and device for processing a video sequence
EP4220573A1 (fr) Segmentation multi-résolution pour images gigapixel
Liu et al. Geometric and physical constraints for head plane crowd density estimation in videos
CN113052019B (zh) 目标跟踪方法及装置、智能设备和计算机存储介质
EP4614442A1 (fr) Programme de traitement d'informations, dispositif de traitement d'informations et procédé de traitement d'informations

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22769352

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22769352

Country of ref document: EP

Kind code of ref document: A1