ES3045182A1 - Systems and methods for data-driven analysis of radiological images - Google Patents

Abstract

Systems and methods for enhanced machine learning for accessing and processing radiomic features are described. A system can receive data from a plurality of participants in one or more first clinical trials and generate, using the data from the plurality of participants, a training dataset comprising a plurality of examples. Each example includes a medical image or set of medical images of a respective participant from the first or more clinical trials, and a label corresponding to an outcome of the respective participant from the first or more clinical trials. The system can train a machine learning model using the training dataset to receive a second medical image of a second participant as input and generate a predicted outcome of the second participant in a second clinical trial as output. (Machine-translation by Google Translate, not legally binding)

Description

[0001] DESCRIPTION

[0003] Systems and methods for data-driven analysis of radiological images

[0004] Background of the invention

[0006] Clinical trials are subject to strict inclusion and exclusion criteria. Even patients who meet the selection criteria for a clinical trial may have latent physiological factors that affect the outcome of that trial. Conventional approaches to selecting patients for clinical trials fail to reliably select optimal patients because such approaches rely on manually specified inclusion criteria and tend to overlook individual variability and patient heterogeneity, which can significantly affect treatment outcomes.

[0008] Compendium

[0010] The systems and methods described in this dissertation enable improved techniques for training and running machine learning models to generate indications of additional patient attributes, including predicted patient outcomes in clinical trials. Unlike conventional approaches to subject identification, the techniques described herein use image-based classifiers with clinical trial data and features from previous clinical trials. Such image-based classifiers can utilize any suitable patient image, particularly radiological images and/or radiomic features, to generate predicted clinical trial outcomes for eligible patients.

[0012] Because the techniques described in this disclosure use outcome information from previous clinical trials, the systems and methods in this disclosure can be used to automatically identify patient features not considered at the start of the trial that impact patient outcomes. These additional features can be learned using the machine learning techniques described in this disclosure to more efficiently and accurately generate predicted outcomes for clinical trials. In addition, the techniques described in this disclosure use enhanced techniques to train and run machine learning models to extract radiomic features that are not accessible or generated using conventional technology. These and other enhancements are detailed in this disclosure.

[0014] At least one aspect of this disclosure is directed to a system. The system includes one or more processors coupled to non-transient memory. The system can receive data from a plurality of participants in one or more first clinical trials. The system can generate, using the data from the plurality of participants, a training dataset comprising a plurality of examples. Each example can include a medical image or set of medical images of a respective participant from the one or more first clinical trials, and a label corresponding to an outcome of the respective participant from the one or more first clinical trials. The system can train a machine learning model using the training dataset to receive a second medical image of a second participant as input and generate a predicted outcome of the second participant in a second clinical trial as output.

[0016] In some implementations, the system may generate the training dataset to also include, for each example in the plurality of examples, one or more clinical attributes of the respective participant from the first one or more clinical trials. In some implementations, the system may generate, using an image harmonization function, a set of harmonized medical images based on the data from the plurality of participants. In some implementations, the system may generate the training dataset to include the set of harmonized medical images. In some implementations, the medical image for each example comprises a computed tomography (CT) scan image, a positron emission tomography (PET) scan image, a bone scan image, an ultrasound (US) image, an X-ray image, or a magnetic resonance imaging (MRI) scan image.

[0018] In some implementations, the label comprises one or more of the following: an indication of a response to the first one or more clinical trials; an indication of additional injury during or after the first one or more clinical trials; an indication of a histology type; an indication of a germline mutation state; an indication of a somatic mutation state; or an indication of a change in tumor stage resulting from the first one or more clinical trials. In some implementations, each instance of the plurality of instances further comprises a second label that identifies a time-to-event metric for the respective participant in the first one or more clinical trials. In some implementations, the system can train a second machine-learning model using the training dataset to receive the second medical image of the second participant as input and generate a predicted time-to-event metric for the second participant during the second clinical trial.

[0019] In some implementations, the system may receive a candidate participant's medical image from a computer system associated with the second clinical trial. In some implementations, the system may provide the candidate participant's medical image as input to the machine learning model to generate the candidate participant's predicted outcome during the second clinical trial. In some implementations, the system may provide the computer system with an indication of whether the candidate participant's predicted outcome meets a selection criterion for the second clinical trial.

[0021] Another aspect of this disclosure relates to a system. The system may include one or more processors and non-transient memory. The system may identify a medical image or set of medical images of a candidate participant in one or more clinical trials. The system may provide, as input to a machine learning model, the medical image of the candidate participant to generate a predicted outcome for the candidate participant in a first clinical trial. The machine learning model is trained using a training dataset comprising a plurality of examples. Each example comprises a respective radiological image of a respective participant in one or more second clinical trials, and a label corresponding to an outcome for the respective participant in the one or more second clinical trials. The system may determine, in response to providing the medical image as input to the machine learning model, that the predicted outcome for the candidate participant in the first clinical trial satisfies a selection criterion for the first clinical trial. The system can store, in one or more data structures, an association between a candidate participant identifier and the first clinical trial to indicate that the candidate participant meets the selection criteria of the first clinical trial in response to determining that the predicted outcome of the candidate participant for the first clinical trial meets the selection criteria of the first clinical trial.

[0023] In some implementations, the machine learning model is trained to also receive clinical attributes as input. In some implementations, the system can identify one or more clinical attributes of the candidate participant. In some implementations, the system can provide the machine learning model with one or more clinical attributes along with the medical image or set of medical images of the candidate participant to generate the predicted outcome for the candidate participant in the first clinical trial. In some implementations, the system can provide the candidate participant's medical image as input to a segmentation model to identify one or more regions of interest in the medical image or set of medical images.

[0024] In some implementations, the machine learning model is trained to also receive image features, including radiomic features, as input. In some implementations, the system can extract one or more radiomic features from the medical image of the candidate participant, based on one or more regions of interest in the medical image. In some implementations, the system can provide the medical image or set of medical images of the candidate participant as input to a second machine learning model to generate a time-to-event metric for the predicted event for the candidate participant during the first clinical trial.

[0025] In some implementations, the system may perform an image harmonization function using the medical image or set of medical images of the candidate participant before providing the medical image or set of medical images as input to the machine learning model. In some implementations, the system may generate a report indicating a set of candidate participants suitable for the first clinical trial; the set of candidate participants comprises the candidate participant.

[0027] Yet another aspect of this disclosure addresses a method. The method may be performed, for example, by one or more processors coupled to nontransient memory. The method may include receiving data from a plurality of participants in one or more first clinical trials. The method may include generating, using the data from the plurality of participants, a training dataset comprising a plurality of examples. Each example may include a medical image or set of medical images of a respective participant from the one or more first clinical trials, and a label corresponding to an outcome of the respective participant from the one or more first clinical trials. The method may include training a machine learning model using the training dataset to receive a second medical image of a second participant as input and generate a predicted outcome of the second participant in a second clinical trial as output.

[0029] In some implementations, the method may include generating the training dataset to also include, for each example in the plurality of examples, one or more clinical attributes of the respective participant from the first one or more clinical trials. The method may include generating, using an image harmonization function, a set of harmonized medical images based on the data from the plurality of participants. The method may include generating the training dataset to include the set of harmonized medical images.

[0030] Yet another aspect of this disclosure addresses a method. The method may include identifying a medical image or set of medical images of a candidate participant in one or more clinical trials. The method may include providing, to a machine learning model, as input, the medical image or set of medical images of the candidate participant to generate a predicted outcome for the candidate participant in a first clinical trial, the machine learning model being trained using a training dataset comprising a plurality of examples. Each example may include a respective medical image or set of medical images of a respective participant in one or more second clinical trials, and a label corresponding to an outcome for the respective participant in the one or more second clinical trials. The method may include determining, in response to providing the medical image or set of medical images as input to the machine learning model, that the predicted outcome for the candidate participant in the first clinical trial satisfies a selection criterion for the first clinical trial. The method may include storing, in one or more data structures, an association between a candidate participant identifier and the first clinical trial to indicate that the candidate participant meets the selection criteria of the first clinical trial in response to determining that the predicted outcome of the candidate participant for the first clinical trial meets the selection criteria of the first clinical trial.

[0032] In some implementations, the method may include identifying one or more clinical attributes of the candidate participant. In some implementations, the method may include providing the machine learning model, as input, with the one or more clinical attributes along with the medical image or set of medical images of the candidate participant to generate the predicted outcome for the candidate participant for the first clinical trial. In some implementations, the method may include generating a report indicating a suitable set of candidate participants for the first clinical trial; the set of candidate participants comprises the candidate participant.

[0034] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide a general overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide further illustration and understanding of the various aspects and implementations and are incorporated herein and form a part of this disclosure. Aspects may be combined, and it will be readily apparent that features described in the context of one aspect of this disclosure may be combined with other aspects. Aspects may be implemented in any convenient form. For example, by means of appropriate computer programs, which may be carried on appropriate carrier media (computer-readable media), which may be tangible carrier media (e.g., disks) or intangible carrier media (e.g., communications signals). Aspects may also be implemented using suitable apparatus, which may take the form of programmable computers running computer programs arranged to implement the aspect. As used in the descriptive report and in the claims, the singular form of “a”, “an” and “the”, “the” includes plural referents unless the context clearly dictates otherwise.

[0036] Brief description of the drawings

[0038] The accompanying drawings are not intended to be drawn to scale. In the various drawings, reference numbers and similar designations indicate similar elements. For the sake of clarity, not every component may be labelled in every drawing. In the drawings:

[0040] FIG. 1 illustrates an example system for generating radiomic features of participants using machine learning techniques, according to one or more implementations;

[0042] FIG. 2 illustrates an example of a data flow diagram of a process for training and updating machine learning models that can be used in connection with the techniques described in FIG. 1, according to one or more implementations;

[0044] FIG. 3 illustrates a flowchart of an example method for training machine learning models using participant radiomic features, according to one or more implementations;

[0046] FIG. 4 illustrates a flowchart of an example method for running machine learning models to predict participant outcomes using machine learning techniques, according to one or more implementations; and

[0048] FIG. 5 illustrates a block diagram of an example computer system suitable for use in the various arrangements described in this memorandum, according to one or more implementations.

[0049] Detailed description

[0051] The following are detailed descriptions of various concepts related to techniques, approaches, methods, devices, and systems, and their implementations, for generating radiomic features of participants using machine learning techniques. The various concepts introduced above and discussed in detail below can be implemented in any number of ways, as the concepts described are not limited to any particular implementation method. Examples of specific implementations and applications are provided primarily for illustrative purposes.

[0053] Accurate and efficient selection of participants in clinical trials is critical for optimal therapeutic outcomes. Traditional patient selection methodologies often rely on manual processes or simplistic criteria, which may not fully capture the complexities of individual patient characteristics and the nuances of disease manifestations. Systems and methods improve upon these shortcomings by using machine learning techniques to process radiomics data derived from medical images and information from other participants in previous clinical trials, enabling a more accurate and efficient participant selection process for clinical trials.

[0055] To accomplish this, the systems and methods described in this dissertation provide techniques for training machine learning models using data from previous clinical trials. The machine learning models can be generated and trained for different classes of treatments, including drugs or other therapies, or types of clinical trials, for the outcomes of future trials that implement similar treatment processes. Previous data used to train the machine learning models described in this dissertation may include the use of radiomics features from previous participants.

[0057] Radiomic features can include any type of feature extracted from radiological imaging data, which can be generated from diagnostic imaging techniques such as MRI, CT, X-ray, PET, bone scans, or ultrasound. Radiomic features can be extracted or identified using any suitable technique, including the use of segmentation models or other machine learning models. Segmentation, for example, can be used to delineate regions of interest within radiological images, which may include various anatomical structures or pathological findings as identified radiological images. By extracting a comprehensive set of quantitative features, the machine learning models described in this dissertation are trained to predict features that may be indicative of disease progression, treatment response, or patient prognosis that are not possible using conventional techniques. Furthermore, radiomic features generated according to the techniques described in this dissertation include those that are not discernible by humans.

[0059] Inputs to machine learning models can include radiomic features, images, and additional medical information related to the participant. These techniques improve overall accuracy and allow the machine learning model to automatically derive additional correlations between participant characteristics and clinical trial outcomes that are not possible using conventional approaches. Unlike conventional approaches, which rely solely on rule-based techniques to automatically compare existing patient information with trial inclusion criteria and do not consider or use radiomic features, the systems and methods described in this dissertation implement machine learning models trained using radiomic features that are practically undiscernible using the human eye or conventional approaches to processing radiological images. The use of improved machine learning models in participant selection enhances the accuracy and efficiency of trial enrollment. Furthermore, the machine learning models described in this dissertation improve the efficiency and performance of computer technology by providing techniques for securely processing and efficiently predicting outcomes for clinical trials and by providing a more effective approach to clinical trial participant selection.

[0061] Referring to FIG. 1, a block diagram of an example system 100 for generating radiomic features of participants using machine learning techniques is illustrated, according to one or more implementations. The system 100 may include at least one data processing system 105, at least one network 110, and one or more remote computing systems 160A-160N (sometimes referred to as “remote computing systems 160”). The data processing system 105 may include a trial data manager 130, a dataset generator 135, a model updater 140, one or more machine learning models 150, and storage 115. Storage 115 may include participant data 170 and training datasets 175. The training dataset 175 may include one or more training examples that have radiological images 180 and labels 185. Although shown here as internal to the data processing system 105, storage 115 may be external to the data processing system 105, for example, as part of a local storage repository.

[0063] Each of the components (e.g., the data processing system 105, the network 110, the remote computing systems 160, the test data manager 130, the dataset generator 135, the model updater 140, the one or more machine learning models 150, the storage 115, etc.) of system 100 can be implemented using the hardware components or a combination of software and hardware components of a computer system, such as the computer system 500 described in relation to FIG. 5, or any other computer system described herein. Each of the components of the data processing system 105 can perform the functionalities detailed herein.

[0065] The data processing system 105 may include at least a processor and a memory (e.g., a processing circuit). The memory may store processor-executable instructions which, when executed by a processor, cause the processor to perform one or more of the operations described in that memory. The processor may include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), etc., or combinations thereof. The memory (which may be or include storage 115) may include, but is not limited to, electronic storage or optical, magnetic, or any other transmission device capable of providing program instructions to the processor. The memory may also include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ASIC, FPGA, read-only memory (ROM), random-access memory (RAM), electrically erasable programmable ROM (EEPROM), erasable programmable ROM (EPROM), flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may include code in any suitable computer programming language. The data processing system 105 may include one or more computer devices or servers that can perform various functions as described herein. The data processing system 105 may include any or all of the components and perform any or all of the functions of the computer system 500 described in connection with Figure 5.

[0067] In some implementations, the data processing system 105 can communicate with one or more remote computer systems 160, for example, to retrieve or receive information related to identified clinical trial participants, or to provide results from various machine learning techniques described herein. In one example, the data processing system 105 (or components thereof) can transmit data indicative of predicted outcomes for candidate participants in a particular trial, which can be generated using one or more of the machine learning models 150, as described herein. In another example, the data processing system 105 can be, or include, an application server or web server, which can include software modules for accessing or transmitting data stored by the data processing system 105 (e.g., in storage 115). For example, the data processing system 105 may include a web server that enables the remote computer system 160 to request or receive clinical trial information, participant data 170 (including radiological images 180, clinical trial results, or any other participant information), or any other information that can be processed by the data processing system 105 (or its components), among other things. In response to a corresponding request, the data processing system 105 may perform the techniques described herein to train or run one or more machine learning models 150. In some implementations, the data processing system 105 may transmit data to one or more remote computer systems 160 in response to user input by an operator of the data processing system 105.

[0069] Network 110 may include computer networks such as the internet, local area networks, wide area networks, intranets, and satellite networks, other computer networks such as mobile data or voice telephony communication networks, or combinations thereof. The data processing system 105 of system 100 may communicate via network 110 with one or more computing devices, such as one or more remote computer systems 160. Network 110 may be any form of computer network that can forward information between the data processing system 105 and the remote computer system 160. In some implementations, network 110 may include the internet and/or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. The 110 network can also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) that are configured to receive or transmit data within the 110 network.

[0071] Network 110 may also include any number of wired or wireless connections. Any or all of the computing devices described in this documentation (e.g., data processing system 105, one or more remote computing systems 160, computer system 500, etc.) may communicate wirelessly (e.g., via Wi-Fi, cellular communication, radio, etc.) with a transceiver that is wired (e.g., via fiber optic cable, CAT5 cable, etc.) to other computing devices in Network 110. Any or all of the computing devices described in this documentation (e.g., data processing system 105, one or more remote computing systems 160, computer system 500, etc.) may also communicate wirelessly with computing devices in Network 110 by means of a proxy device (e.g., a router, network switch, or gateway).

[0073] Each remote computing system 160 may include at least one processor and memory (e.g., a processing circuit). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more of the operations described in that memory. The processor may include a microprocessor, an ASIC, an FPGA, a GPU, etc., or combinations thereof. The memory may include, but is not limited to, electronic storage or transmission devices, optical, magnetic, or any other medium capable of providing program instructions to the processor. The memory may further include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ASIC, FPGA, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may include code in any suitable computer programming language. Each remote computing system 160 may include one or more computing appliances, servers, personal computing devices, or data repositories. The remote computer system 160 may include any or all of the components and perform any or all of the functions of the computer system 500 described in relation to FIG. 5.

[0075] Storage 115 may be a computer-readable memory that can store or hold any of the information described in this memory. Storage 115 may store or hold one or more data structures, which may contain, index, or otherwise store each of the values, pluralities, sets, variables, vectors, numbers, or thresholds described in this memory. Storage 115 may be accessed using one or more memory addresses, index values, or identifiers of any item, structure, or region held in storage 115. In implementations where storage 115 is external to the data processing system 105, storage 115 may be accessed by components of the data processing system 105 via the network 110 or via a local communications interface. Storage 115 may be distributed across many different computer storage systems or elements. The data processing system 105 can store, in one or more regions of the memory of the data processing system 105, or in storage 115, the results of any or all computations, determinations, selections, identifications, generations, constructions, or calculations in one or more data structures indexed or identified with appropriate values.

[0077] Any or all values stored in storage 115 may be accessed by components of the data processing system 105 to perform any of the functionalities or functions described herein. In some implementations, the data processing system 105 may use authentication information (e.g., username, password, email address, etc.) to indicate that an operator of the data processing system 105 is authorized to access requested information in storage 115. Storage 115 may include permission settings that indicate which users, devices, or profiles are authorized to access certain information stored in storage 115. In some implementations, one or more remote computer systems 160 may access the functionality of the data processing system 105 using corresponding authentication information, which may be provided through one or more web-based interfaces.

[0079] Storage 115 may store various participant data 170, for example, in one or more data structures. Participant data 170 may be a dataset that can include clinical or non-clinical participant information from any number of participants in any number of ongoing or historical clinical trials. Participant data 170 may be used by the data processing system 105 to generate one or more training datasets 175 for one or more of the machine learning models 150 using the techniques described in this document. Participant data 170 may be stored or otherwise indexed using identifiers from one or more clinical trials. Participant data 170 may include demographic information, including but not limited to participant age, gender, ethnicity, weight, height, and other relevant demographic statistics that may influence disease presentation and response to treatment. Participant 170 data may include genetic information, biomolecular markers, or other biological data that may influence disease progression and treatment effectiveness. Demographic data may include patient history, including previous medical conditions, surgeries, treatments, and medication use.

[0081] Participant data 170 may include detailed clinical records for each participant, such as symptoms, diagnosis, comorbidities, concurrent medications, laboratory test results (e.g., blood counts, biomarker levels), and vital signs (e.g., blood pressure, heart rate). This clinical record information may be indexed by time, treatment phase, or progress, or by any other metric associated with a corresponding ongoing or historical clinical trial. Participant data 170 may include one or more radiological images 180, which may be identified as having been used to generate one or more training datasets 175 according to the techniques described in this document. Radiological images 180 may include any type of image, including but not limited to CT images, MRI images, PET images, bone scan images, X-ray images, or ultrasound images.

[0082] Participant data 170 may include information related to the treatment protocols used during at least one ongoing or historical clinical trial administered to participants, including dose, route of administration, frequency, and duration of therapy. Participant data 170 may also include various information related to clinical trial outcomes for one or more participants, which can be used to generate corresponding labels 185 for one or more training datasets 175 according to the techniques described in this document. Outcome data may include participants' responses to treatment, which may include efficacy (e.g., tumor size reduction, symptom improvement), adverse events, or any other occurrences during clinical trials.

[0084] Storage 115 may store one or more training datasets 175, each of which may be generated using participant data 170 according to the techniques described in this document. The training datasets 175 may be used to train or update one or more of the machine learning models 150, for example, to generate predicted outcomes for patient candidates for a given clinical trial. Each training dataset 175 may include one or more training examples. Each training example may correspond to a respective participant and may include any of the participant data 170 for that participant. Each training example in the training dataset 175 may include a dataset to be provided as input to one or more machine learning models 150, and corresponding fundamental truth data to be predicted for the given input.

[0086] In one example, the input data for a training example may include participant data 170, as well as any additional information derived from participant data 170. For example, the input data may include one or more radiological images 180 of the participant, as well as segmentation information, radiomic features, and/or other information derived from such radiological images 180 generated according to the techniques described in this specification. Additional details relating to various features that can be generated from participant data 170 and radiological images 180 are described in relation to FIG. 2. In some implementations, a training dataset 175 may be generated for a particular clinical trial or set of clinical trials corresponding to a particular drug, treatment, or series of treatments.

[0087] Radiological images 180 can be any type of radiological image, including but not limited to CT images, MRI images, X-ray images, or ultrasound images, among others. As described in further detail in this document, radiological images 180 can be preprocessed using various preprocessing techniques, such as image harmonization. Image harmonization can be implemented to align and standardize the appearance of radiological images 180 acquired on different scanners, protocols, and from different patient populations, thereby enabling more accurate comparisons and analyses. In some implementations, each of the remote computer systems 160 can correspond to a computer system at a respective treatment location and can receive and provide radiological images 180 from different radiological equipment for different clinical trial participants.

[0089] The labels 185 of a training dataset 175 can be any fundamental truth data for a given output to be predicted using a machine learning model 150. Each training example can include one or more labels 185 corresponding to an actual outcome for a corresponding participant. For example, labels 185 can include indications of overall response, whether a new lesion or tumor has been detected during the trial, a participant's histology type, a tumor mutation status, a protein expression forecast, a gene expression signature forecast, or a change in the participant's tumor stage. Labels 185 can include various time-to-event outcomes, including, but not limited to, an indication of whether the participant has survived, indications of time to adverse events, or indications of times corresponding to disease progression, indications of relapse following treatment in the corresponding clinical trials, among others. Additional training examples for a training dataset 175 may be generated to include information from additional participants as such information is provided or otherwise received from one or more remote computer systems 160.

[0091] The data processing system 105 can generate and/or store one or more machine learning models 150, for example, in one or more data structures. Although shown as external to storage 115, it should be understood that machine learning models 150 can be maintained in storage 115 and executed by one or more processors of the data processing system 105 to generate forecasts for candidate participants in clinical trials, as described herein. Each machine learning model 150 can be trained or updated to generate any type of predicted clinical trial outcome described herein using one or more training datasets 175.

[0093] Each machine learning model 150 can be generated for a particular type of clinical trial and/or treatment plan and trained to generate one or more corresponding predicted outcomes for that type of clinical trial and/or treatment plan. For example, a machine learning model 150 can be generated and trained to generate predicted outcomes for a particular drug, class of drugs, drug combination, or other treatment process for a particular disease or condition. Machine learning models 150 can be stored or otherwise indexed with an identifier of the corresponding clinical trial or treatment process for which they are trained.

[0095] Machine learning models 150 may include any type of machine learning model 150 capable of being trained and/or updated to generate predicted clinical trial outcomes. Non-limiting examples of machine learning models include deep neural networks (DNNs), which may include fully connected networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), or transformer neural networks. Machine learning models 150 may include, but are not limited to, decision tree models, regression models, support vector machines (SVMs), random forest models, gradient boosting machine model assemblies, or hybrid models. Machine learning models 150 may be trained using any suitable training technique, including, but not limited to, supervised learning, semi-supervised learning, self-supervised learning, unsupervised learning, or combinations thereof.

[0097] In implementations where a machine learning model 150 is a variant of a neural network, the machine learning model 150 may include any number or type of neural network layers. Non-limiting examples of neural network layers include fully connected layers, convolutional layers, activation layers, accumulation layers, softmax layers, recurrent layers, normalization layers, embedding layers, transformer layers, adapter layers, and dropout layers, among others. The number and type of layers in the machine learning model 150 may be provided by means of machine learning model 150 hyperparameters, which may be specified in a request to generate or train a machine learning model 150. In some implementations, the data processing system 105 may initialize a machine learning model by generating one or more data structures representing each layer of the machine learning model 150, which may be initialized with random or default values. In some implementations, one or more of the 150 machine learning models may be pre-trained models that are fine-tuned or refined using the training techniques described in this dissertation.

[0099] Referring now to the operations of the components of the data processing system 105, the trial data manager 130 can receive participant data 170 from one or more participants in any number of clinical trials. The trial data manager 130 can receive participant data from one or more remote computer systems 160, each of which can correspond to one or more trial sites, hospitals, or treatment centers that maintain participant data 170 for one or more clinical trials. The information can be provided by means of one or more web-based APIs, which can be secure APIs accessed using appropriate encryption keys or other secure data transmission techniques.

[0101] In some implementations, the trial data manager 130 may provide a web-based interface through which treatment sites can manage or otherwise maintain participant data 170 for one or more clinical trials in storage 115. In such implementations, the trial data manager 130 may implement authentication and authorization to manage access to and modifications of corresponding participant datasets 170, provide graphical user interface elements for data annotation and validation, enable navigation, retrieval, modification, and updating of participant data 170 corresponding to given treatment sites, and other functionalities. In some implementations, the trial data manager 130 may receive updated participant data 170 for ongoing or historical clinical trials and store such updated information in storage 115.

[0103] The dataset generator 135 can generate a training dataset 175 to predict outcomes for a particular drug, drug class/combination, or treatment plan using participant data 170 for one or more ongoing or historical clinical trials. The dataset generator 135 can receive a prompt to begin generating a training dataset 175 for corresponding clinical trials in response to a message from a remote computer system 160, or by means of user input to a data processing system input device 105 in some implementations. The prompt can provide an indication of one or more historical or ongoing clinical trials to be accessed when generating the training dataset 175.

[0105] In some implementations, the dataset generator 135 may receive a suggestion for one or more drugs, treatment plans, or treatment protocols for particular diseases or conditions and automatically identify historical and/or ongoing clinical trials for such drugs, treatment plans, or treatment protocols for the particular diseases or conditions. The dataset generator 135 may access participant data 170 from participants in the identified or suggested clinical trials to generate the training dataset 175. In some implementations, the training dataset 175 may include data for training multiple machine learning models 150. Generating the training dataset 175 may include generating multiple training examples, each of which may correspond to a respective participant. Each training example may include an input dataset for at least one machine learning model 150 and corresponding ground truth data (e.g., one or more labels 185) representing different outcomes for the corresponding participant in the relevant clinical trial.

[0107] Training examples of the training dataset 175 can be generated for a radiological image 180 or set of radiological images of a respective participant in one or more clinical trials. In some implementations, the radiological images 180 may be a sequence of radiological images captured at different times during the clinical trials. In some implementations, training examples may include various radiomic features, which can be extracted from the participant's radiological images 180 using one or more segmentation models or other feature extraction techniques, as described in further detail in relation to FIG. 2. For example, the dataset generator 135 may use an image harmonization technique to generate harmonized radiological images 180 from the images stored in the participant data 170. The radiological images stored in the training dataset 175 may include the harmonized radiological images 180.

[0109] The input data included in each training example of the generated training dataset 175 may also include one or more clinical attributes of the respective participant, which may include any of the information from participant data 170. Clinical attributes may include, but are not limited to, demographic information of the participant, treatment plan or progress information, laboratory results from different tests performed during the clinical trials, or molecular information of the drug or treatment plan used to treat the participant during the trial, among any other clinical attributes described in this document.

[0110] The dataset generator 135 can generate the training dataset such that each training example includes at least one label 185 corresponding to a respective participant outcome from one or more clinical trials. As described herein, the labels 185 can include corresponding outcomes that are to be predicted by one or more machine learning models 150 once trained. For example, the labels 185 can include one or more indications of a response to corresponding clinical trials, an indication of additional injury during or after corresponding clinical trials, an indication of a histology type, an indication of a germline mutation status, an indication of a somatic mutation status, or an indication of a change in tumor stage resulting from corresponding clinical trials. The labels 185 can also include a time-to-event metric for the respective participant, which can be a survival time, a time to disease progression, or a time to relapse, among others.

[0111] The dataset generator 135 can generate multiple training datasets 175 for multiple machine learning models 150. For example, the dataset generator 135 can generate a first training dataset 175 with input data and labels 185 to train a classification model and generate a second training dataset 175 with input data and labels 185 to train a machine learning model to forecast time-to-event metrics. Each training dataset 175 can be generated from the same participant dataset 170, but include different input data (e.g., radiological images 180, radiomic features, clinical data, etc.) for the machine learning model 150 and different labels 185 corresponding to the forecasting objective for the corresponding machine learning model 150. Each training dataset 175 can be stored in association with respective identifiers of the machine learning model 150 to which it corresponds, as well as identifiers of the clinical trials that have the participant data 170 used to generate the training dataset 175.

[0113] Once the training datasets 175 have been generated, the model updater 140 can train at least one machine learning model 150 using the training datasets 175. In one example, a machine learning model 150 can be trained to receive at least one radiological image 180 of a candidate participant as input and generate a predicted outcome for the candidate clinical trial participant that has attributes similar to those corresponding to the training dataset 175. The model updater 140 can train the machine learning models 150 using any suitable training technique, including supervised learning.

[0115] In a non-limiting example of using supervised learning to train a machine learning model 150, the model updater 140 can first enumerate a first subset of the training examples in the training dataset 175 for the machine learning model and enumerate a second subset of the training examples in the training dataset 175 as an evaluation set. The training example from the first subset can be used by the model updater 140 to train the machine learning model 150 by feeding the input data from the training example (e.g., radiological images 180, extracted features, clinical data) into the model to a predicted output. The model updater 140 can compare the predicted outputs of the machine learning model 150 against the actual results indicated by the labels 185.

[0116] The model updater 140 can use a loss function to quantify the error between the predicted values generated by the machine learning model 150 and the actual fundamental truth value stored in the corresponding labels 185 of the training example. In some implementations, the model updater 140 can calculate an error for one or more batches of training examples, for example, when a batch training/update approach is implemented. The loss function used to calculate the error can be any suitable loss function, including, but not limited to, mean square error loss, cross-entropy loss, hinge loss, L1 loss, or L2 loss.

[0118] The model updater 140 can use the loss to optimize the trainable parameters of the machine learning model 150. For example, the model updater 140 can implement a backpropagation algorithm to propagate the loss backward through each layer of the machine learning model, modifying the parameters to optimize the loss according to a suitable optimization function, such as gradient descent or an Adam optimization process, in some implementations. The model updater 140 can modify the trainable parameters of the machine learning model 150 until a training completion condition is reached. The amount by which the trainable parameters are modified to train the machine learning model 150 can be a function of a learning rate, which can be a dynamic or scheduled learning rate, in some implementations.

[0120] An example of a training condition might include a predetermined accuracy threshold, which can be compared to a predetermined accuracy of the machine learning model 150 using a validation set (e.g., the second subset of the training dataset 175 for the machine learning model 150). Other training termination conditions might include a predetermined number of training epochs or indications that the machine learning model 150 begins to overfit for the training dataset 175. The parameters of the machine learning model 150 can be stored in the data processing system memory 105 and/or storage 115 after training is complete. In some implementations, multiple snapshots of the machine learning model 150 can be captured and stored during the training process.

[0122] In some implementations, the model updater 140 can multiply machine learning models 150, each of which forecasts corresponding clinical trial outcomes or corresponding given treatment plan input data. For example, a first set of machine learning models 150 can be trained as classification models, while a second set of machine learning models 150 can be trained as time-to-event models. Multiple training datasets 175 can be generated using the techniques described in this dissertation to accommodate the training of multiple machine learning models in some implementations. Further details of the training process for the machine learning models 150 are described in relation to FIG. 2.

[0124] Referring to FIG. 2 in the context of the components described in relation to FIG. 1, an example data flow diagram 200 of a process for training and updating machine learning models (e.g., the machine learning models 150) is illustrated. These models may be used in connection with the techniques described in FIG. 1, according to one or more implementations. Any of the techniques described in relation to FIG. 2 may be performed by the data processing system 105 or components thereof, or by any other computer system described herein.

[0126] As shown, input data for a training dataset (e.g., a training dataset 175) can be generated using an image harmonization process 206. The image harmonization process 206 aligns and standardizes input images 202 acquired from different sources, such as scanners, sites, or equipment. The image harmonization process 206 may involve adjusting intensity values, pixel distributions, spatial relationships, and other characteristics within multiple input images 202 (e.g., radiological images 180) to ensure consistency across all images in the dataset. Image harmonization can be implemented because input radiological images 292 from different sources may exhibit varying levels of brightness, contrast, resolution, or noise due to differences in imaging protocols, scanner settings, patient populations, or acquisition techniques.

[0128] The image harmonization process 206 can receive a set of input images 202 as input and generate a corresponding set of harmonized images 204 as output. The input images 202 selected for an iteration of the image harmonization process 206 can correspond to a similar image type, capture orientation, or other common image attribute. Examples of image harmonization processes 206 may include, but are not limited to, performing an intensity standardization process, in which techniques such as histogram matching, gamma correction, or wavelet-based processing are used to normalize the pixel intensity in each set of input images 202. The corrected images are stored as the harmonized images 204, which can be used to generate a training dataset or for feature extraction techniques.

[0130] As shown, the input data for training examples from a training dataset 175 can be generated using additional input variables 208 in some implementations. The input variables 208 can include any participant attributes (e.g., retrieved from participant data 170), including, but not limited to, clinical data of the participants, laboratory results of tests performed on the participant, information about treatment dosage or other treatment attributes, or any other information described herein. The input variables 208 for a participant can be stored in association with one or more input images 202 and/or one or more harmonized images 204 of that participant. This information can be used as input for the deep learning modeling process 216, as shown.

[0132] Harmonized image sets 204 may also be provided as input to one or more segmentation models 210. Segmentation models 210 may be image-based machine learning models that are trained/updated to process input radiological images (e.g., input images 202 or harmonized images 204) to identify, locate, or otherwise quantify anatomical structures or abnormalities. Such a segmentation model 210 may include tumor segmentation models 210, which may be trained to identify pixels in an input harmonized image 204 or input image 202 that correspond to a tumor or other cancerous tissue. Segmentation models 210 may be trained on a dataset comprising annotated radiological images, where each image has been labeled with specific regions or features of interest (e.g., regions corresponding to organs, tumors, or other anatomical structures, etc.).

[0134] In some implementations, segmentation models can implement pixel-based classification of input radiological images, in which each pixel within an input image 202 or a harmonized image 204 is assigned a label indicating a classification of a corresponding anatomical structure or feature. The output of such segmentation models can be segmentation masks, which in some implementations can be used as input data to train the various machine learning models described in this dissertation (e.g., the machine learning models 150, etc.). In some implementations, and as shown, input images 202, harmonized images 204, and/or corresponding segmentation masks for various regions of interest can be provided as input to a feature extraction process 212.

[0136] Feature extraction process 212 may include any process suitable for identifying and extracting features from input images 202 and/or harmonized images 204 of participants in one or more clinical trials. Radiomic features extracted using feature extraction process 212 may include any attribute that can be derived from input images 202, harmonized images 204, and/or the corresponding segmentation masks. By way of example, such features may include tumor size, tumor location, tumor type, number of tumors detected, changes in tumor size over time, changes in the anatomical structure of different organs, bones, or other tissues, tumor homogeneity, lesion morphology and growth patterns, bone density or bone shape, among other radiomic features.

[0138] Additional examples of radiomic features may include image-specific metrics, such as first-order statistics, that relate to the distribution of pixel or voxel intensities within a harmonised image.204 Examples of such first-order features include, but are not limited to, energy, total energy, entropy, interquartile range, absolute deviation from the mean, root mean square, skewness, kurtosis, variance, or uniformity. Other radiomic features may include shape-based features, which may include, but are not limited to, mesh volume or image area, voxel volume, surface area of a 3D image, surface area-to-volume ratio, sphericity, maximum 3D diameter, maximum 2D diameter (e.g., of a 2D image or a portion of a 3D image), major axis length, minor axis length, elongation, or flatness. In some implementations, radiomic features may include features derived from a gray-level co-occurrence matrix (GLCM), a gray-level dependence matrix (GLDM), a gray-level length matrix (GLRLM), a gray-level size zone matrix (GLSZM), or a neighboring gray-tone difference matrix (NGTDM).

[0140] In some implementations, radiomic features extracted using feature extraction process 212 may include deep features or imaging biomarkers. Deep features may include input image features 202 and/or harmonized image features 204 that are extracted from one or more intermediate/hidden layers of a deep convolutional neural network, which may be generated by providing the input images 202 and/or harmonized image features 204 as input to a convolutional neural network trained to process radiological images. Imaging biomarkers extracted using feature extraction process 212 may include any type of indicator of biological/pathogenic processes or treatment responses present in a radiological image, or related measurement standards.

[0142] In some implementations, the features extracted using the feature extraction process 212 can be determined based on the type of treatment or condition related to the clinical trials from which the input images 202 are retrieved. For example, the features to be extracted can be determined based on the imaging modality, diagnosed disease or condition, type of treatment, or other clinical trial-specific attributes. Radiomic features can be extracted using any suitable analysis technique and may involve the use of specially trained machine learning models in some implementations. Radiomic features extracted by the feature extraction process 212 can be used in a feature engineering process 214 and/or used in connection with the deep learning modeling process 216, as shown. In some implementations, the radiomic features extracted using the feature extraction process 212 can be stored as part of input data in training examples of a training dataset 175, as described in this memory.

[0144] The feature engineering process 214 and the deep learning modeling process 216 can be run to train and/or evaluate one or more machine learning models (e.g., machine learning models 150, etc.). Machine learning models trained using the deep learning modeling process 216 can be updated according to the techniques described in relation to FIG. 1, in some implementations. The feature engineering process 214 can be used to generate or select optimal features for training machine learning models to most accurately predict a particular outcome for a candidate clinical trial participant. The feature engineering process 214 may include selecting a subset of relevant features from a set of radiomic features generated using the feature extraction process 212. The subset may be selected, in some implementations, based on a certain correlation with a given variable target or outcome, or based on other criteria for predicting an outcome (e.g., a classification outcome, a time-to-event outcome, etc.) of a clinical trial participant.

[0145] The feature engineering process 214 may include feature processing, for example, to normalize or combine features to improve the performance or accuracy of the machine learning models described herein. Such processing techniques may include, but are not limited to, selecting specific features, reducing feature dimensionality, standardizing features, detecting outliers, or deriving additional data from related features (e.g., calculating ratios of different inputs, etc.). The feature engineering process 214 may be performed to generate input data for one or more training datasets 175 in some implementations. As shown, the output of the feature engineering process 214 can be provided as input to the deep learning modeling process 216.

[0147] The deep learning modeling process 216 may include training one or more machine learning models 218 to predict outcomes for candidates for clinical trials or particular treatment plans. As shown, and as described herein, the machine learning models 218 may include one or more classification models 220 and one or more time-to-event models 222. The deep learning modeling process 216 may include performing various techniques to update the parameters of the machine learning models 218, including supervised learning using training datasets generated using input images 202 and/or harmonized images 204, input variables 208, segmentation masks, and extracted/engineered features, in some implementations. As shown, the machine learning models may train one or more classification models 220 and/or one or more time-to-event models 222.

[0149] It should be understood that, although multiple inputs are shown as provided to the deep learning modeling process 216, this is merely an example representation. In some implementations, the deep learning modeling process 216 may have fewer inputs, for example, training only on input images 202 and/or the harmonized images 204 as input, or any of the additional variables 208, segmentation masks generated by means of the segmentation models 210, features generated by means of the feature extraction process 212 and/or the feature engineering process 214, either in isolation or in any combination. Furthermore, it should be understood that in some implementations, different machine learning models 218 may be trained/updated using different input datasets to predict different outcomes for candidate participants, as described herein.

[0151] Classification models 220 can be trained to classify participants to determine the best overall response, whether new lesions are predicted to occur during a clinical trial, a histology type, a mutation status, a degree of protein expression or lack thereof, a degree of gene expression signature or lack thereof, or a tumor stage or tumor stage progression during a clinical trial, among others. Time-to-event models 222 can be trained to predict overall survival time, event-free survival time, progression-free survival time, relapse-free survival time, or disease-free survival time, among others. The trained machine learning models 218 can be used to predict outcomes for clinical trial candidates, as described herein.

[0153] Referring again to FIG. 1, once the data processing system 105 has trained one or more machine learning models 150, the data processing system 105 can predict outcomes for candidate clinical trial participants in order to select candidates for potential inclusion in clinical trials. To do this, the trial data manager 130 can identify a radiological image 180 of a candidate participant for a first clinical trial. The radiological image 180 can be received from a remote computer system, for example, as part of participant data 170 of the candidate participant. In some implementations, one or more remote computer systems 160 can transmit participant data 170 (which may include one or more radiological images 180) from a pool of candidate participants to the data processing system 105 to predict outcomes for an upcoming or potential clinical trial. The set of candidate participants can be included or identified in a corresponding request from the remote computer system 160, or in a corresponding request identified by means of input to the data processing system 105, in some implementations.

[0155] The request may identify one or more characteristics of the upcoming or potential clinical trial, including treatment types (e.g., drugs, treatment protocol, etc.), or the disease being treated by the upcoming or potential clinical trial, among other characteristics. These characteristics may be used by the trial data manager 130 to select a trained machine learning model 150 that corresponds to the respective request.

[0156] For example, each machine learning model 150 can be stored in association with identifiers of treatments, diseases/conditions, or past clinical trials for which the machine learning model 150 was trained. To accurately predict outcomes for candidate participants, the trial data manager 130 can select one or more trained machine learning models 150 suitable for predicting outcomes for the upcoming or potential clinical trial. In some implementations, the machine learning models 150 can be identified in a request, message, or input received by the data processing system 105.

[0158] Once the trained machine learning models 150 have been identified, the trial data manager 130 can provide at least the radiological images 180 of the candidate participants as input to the machine learning models 150 to generate predicted outcomes for the candidate participants in the upcoming or potential clinical trial. In addition to, or as an alternative to, providing the radiological images 180 of the candidate participants as input to the machine learning models 150, the trial data manager 130 can provide clinical attributes of the candidate participants as input to the machine learning models 150. The clinical attributes can be identified from the participant data 170 of the candidate participants, for example, or received in response to a request transmitted to one or more of the remote computer systems 160.

[0160] Additional input data provided to the machine learning models 150 may include segmentation masks generated by at least one segmentation model (e.g., a segmentation model 210), as well as any radiomic features extracted using the techniques described herein. The trial data manager 130 may generate input data for each machine learning model 150 trained to produce predicted results for a particular trial. In some implementations, preprocessing, such as image harmonization, may be performed on the input radiological images 180 before providing the radiological images 180 as input to the machine learning model 150.

[0162] In some implementations, and as described herein, different machine learning models 150 can be trained to receive different input data, each of which can be generated and provided by the trial data manager 130 using corresponding participant data 170 from the candidate participants. For example, the trial data manager 130 can provide a first set of input data for one or more candidate participants as input to a first machine learning model (e.g., a classification model 220), and can provide a second set of input data for the one or more candidate participants as input to a second machine learning model (e.g., a time-to-event model 222).

[0164] The trial data manager 130 can generate the predicted results by running the machine learning models 150 using the provided input data (which may include one or more radiological images 180, radiomic features, clinical data of the candidate participants, etc.). In an example where a machine learning model is a neural network model, the trial data manager 130 can run the machine learning model 150 by providing the input data through each layer of the neural network until the machine learning model 150 generates a predicted result value. In some implementations, for example, where the machine learning model 150 is a classification model, the output may be generated by means of a softmax layer of the machine learning model.

[0166] Predicted outcomes include forecasts of how prospective participants in ongoing or potential clinical trials will respond, as described herein. For example, machine learning models 150 can generate forecasts of whether the disease or condition related to the clinical trials will progress over the course of the clinical trial, or specific changes in tumor stage over the course of the clinical trial, among others.

[0168] Using the output forecasts from the machine learning models 150, the trial data manager 130 can determine whether any of the pool of candidate participants corresponds to predicted outcomes that satisfy one or more selection criteria for the ongoing or potential clinical trial. In some implementations, the selection criteria for the clinical trial may be specified in the request to process the candidate participants, may be received or retrieved from one or more of the remote computer systems 160, or may be provided by means of input to the data processing system 105. In some implementations, the selection criteria may include thresholds or conditions for the generated predicted outcomes for the candidate participants.

[0170] Identifiers of candidate participants who meet (e.g., fall within ranges, thresholds, or predetermined values) the selection criteria of the ongoing or potential clinical trial may be stored in association with the clinical trial. Multiple candidate participants may be associated with the clinical trial by modifying one or more databases, files, or other data structures associated with the ongoing or potential clinical trial. In some implementations, the association may indicate various participant data of the corresponding candidate participant selected for potential inclusion in the clinical trial, including any attributes, traits, or qualities that indicate the candidate participant is suitable for the clinical trial.

[0172] In some implementations, the trial data manager 130 can generate a report indicating a pool of candidate participants identified as suitable for the ongoing or potential clinical trial. The report may include various participant data 170, clinical trial-related information, and the various attributes, traits, or qualities that indicate why the pool of candidate participants is suitable for the clinical trial. In some implementations, the pool of candidate participants can be categorized in the report according to their likelihood of success in the clinical trial.

[0173] In some implementations, the overall probability of success for a candidate participant indicated in the report may be generated as the output of one or more machine learning models 150. In some implementations, the probability of success in the clinical trial may be calculated as a weighted sum of the various predicted outputs of multiple machine learning models 150. The weight values may be specific to the ongoing or potential clinical trial for which the report is generated. In some implementations, the weight values may be provided by one or more remote computing systems 160 or by means of input to the data processing system 105. Indications of which candidate participants are suitable for the clinical trial (and generated reports, if any) may be provided for display in the data processing system 105 or transmitted to one or more remote computing systems 160. In some implementations, the candidate participants or reports may be provided by means of a web-based interface.

[0175] Referring to FIG. 3, a flowchart of an example method 300 for training machine learning models using radiomic features of participants is illustrated, according to one or more implementations. Method 300 can be performed, for example, by a data processing system (e.g., data processing system 105), or any computing devices described in this document (e.g., one or more remote computing systems 160, computing system 500 in FIG. 5). It should be understood that the method 300 shown in FIG. 3 is an example, and that additional steps may be performed, steps may be omitted, or steps may be performed in a different order than shown, to achieve desired results.

[0177] In Act 305, a data processing system (e.g., data processing system 105) may receive data from one or more participants (e.g., participant data 170) from one or more historical or ongoing clinical trials. The data processing system may receive the participant data from one or more remote computing systems (e.g., remote computing systems 160), or by means of input to a web-based interface provided by the data processing system 105. The participant data may be received in a request to generate, train, or update one or more machine learning models (e.g., machine learning models 150) to predict outcomes for one or more clinical trials or treatments. Receiving the participant data may include performing any of the operations described in this specification in relation to the trial data manager 130 of FIG. 1, in some implementations.

[0179] In Act 310, the data processing system may generate one or more training datasets (e.g., one or more training datasets 175) using the participant data. The training dataset may include multiple training examples. Each training example may include a radiological image 180 or set of radiological images 180 from a respective participant, and one or more labels 185 corresponding to an outcome of that participant, as described herein. Generating the training dataset may include generating input data for the machine learning models, which may include one or more radiological images of a corresponding participant, clinical attributes of a corresponding participant, or radiomic features of the corresponding participant. In some implementations, radiological images received by the data processing system may be preprocessed using an image harmonization function or technique, as described herein, with the harmonized radiological images being included in the training dataset. In some implementations, the data processing system can perform either the functionality of the dataset generator 135 or the process shown in FIG. 2.

[0181] In Act 315, the data processing system may train one or more machine learning models (e.g., machine learning models 150) using the training dataset to receive one or more radiological images of a candidate participant as input to generate one or more predicted outcomes of the candidate participant for an ongoing or potential clinical trial as output. Any number of machine learning models may be trained using the techniques described in this document, including, but not limited to, classification models (e.g., classification models 220), time-to-event models (e.g., time-to-event models 222), or machine learning models that predict any other attribute or characteristic of a candidate participant with respect to a treatment, drug, or process in a clinical trial. The machine learning models may be trained or updated using any suitable training technique, including the use of supervised learning in relation to the training dataset generated in Act 310.

[0183] Machine learning models can be trained until a training termination condition is reached. The training termination condition can be a predetermined accuracy threshold, a predetermined number of training epochs, or training examples provided to the models to train them or detect that the machine learning models are beginning to overfit the training dataset. Once trained, the machine learning models can be stored in association with an identifier related to the ongoing or potential clinical trial, treatment, or process for which the machine learning models were trained. The trained machine learning models can be accessed by the data processing system to identify one or more candidate participants who are suitable for the clinical trial, for example, using the process described in relation to FIG. 4.

[0185] Referring to FIG. 4, a flowchart of an example method 400 for running machine learning models to predict participant outcomes using machine learning techniques is illustrated, according to one or more implementations. Method 400 can be performed, for example, by a data processing system (e.g., the data processing system 105), or any computing devices described in this document (e.g., one or more remote computing systems 160, the computing system 500 in FIG. 5). It should be understood that the method 400 shown in FIG. 4 is an example, and that additional steps may be performed, steps may be omitted, or steps may be performed in a different order than shown, to achieve desired results.

[0187] In Act 405, a data processing system (e.g., Data Processing System 105) may identify a radiological image of a candidate participant for a potential or ongoing clinical trial. The radiological image may be identified in a request to identify suitable candidate participants for the potential or ongoing clinical trial using the machine learning models (e.g., Machine Learning Models 150) described in this specification. The request may identify a pool of candidate participants, each of whom is to be evaluated using the machine learning models described in this specification. The request may identify one or more selection criteria for the potential or ongoing clinical trial, against which the predicted outcomes of the candidate participants are to be evaluated.

[0189] In some implementations, radiological images received (e.g., from one or more remote computer systems 160) or otherwise identified by the data processing system may be preprocessed by the data processing system. For example, the data processing system may perform an image/technique harmonization function using the radiological images received from candidate participants, which may include harmonizing the radiological images with other radiological images from historical or ongoing clinical trials. Radiological images may include CT images, PET images, MRI images, X-ray images, or ultrasound images, among others. In some implementations, the data processing system may receive or otherwise identify clinical attributes of candidate participants to be assessed, as described herein.

[0191] In Act 410, the data processing system may provide the radiological images of the pool of candidate participants to one or more machine learning models (e.g., machine learning models 150) as input. As described herein, the machine learning models may each be trained to generate one or more predicted outcomes for the candidate participants based on their respective radiological images. In some implementations, the machine learning models may be trained to receive one or more clinical attributes of the candidate participants, radiomic features of the candidate participants, or attributes related to the ongoing or potential clinical trial (or the drugs/treatments therein). The data processing system may provide such information as input to the machine learning models to generate the predicted outcomes for the participants, as described herein in relation to Figures 1 and 2.

[0193] In one example, radiomic features of candidate participants can be generated by providing the candidate participants' radiological images as input to one or more segmentation models, which can be trained to identify one or more regions of interest in the radiological image. Data from the regions of interest, including indications of shape, size, or other characteristics, can be extracted as part of the radiomic features for the candidate participants. Such radiomic features can be provided as part of the input data to the machine learning models described in this dissertation. In some implementations, instead of receiving the patient's radiological image, the machine learning models described in this dissertation can be trained to receive radiomic features as input to generate predicted outcomes for the candidate participants.

[0195] Predicted outcomes may include classifications, including global response classifications, whether a new lesion or tumor is predicted to occur, a predicted histology type of the candidate participant, a predicted mutation status of a candidate participant's tumor, a predicted degree of protein expression or lack thereof, a predicted degree of gene expression signature or lack thereof, or a predicted change in the candidate participant's tumor stage. In some implementations, one or more machine learning models may be trained/updated to generate one or more predicted time-to-event outcomes. Predicted time-to-event outcomes may include, but are not limited to, a predicted survival time of the candidate participant, predicted time to any adverse events, predicted time to disease progress or remission, and predicted relapse incidents following relevant ongoing or potential clinical trials, among others.

[0197] In Act 415, the data processing system can determine whether the predicted outcomes for each candidate participant in the ongoing or potential clinical trial meet one or more selection criteria for the clinical trial. The selection criteria may include thresholds or conditions for the predicted outcomes generated for the candidate participants. The data processing system can compare the predicted outcomes for each candidate participant with the thresholds, conditions, or values in the selection criteria to identify a subset of the candidate participants who meet (e.g., fall within predetermined ranges, thresholds, or values) the selection criteria for the ongoing or potential clinical trial.

[0198] In some implementations, the data processing system can calculate a score for each candidate participant in the subset that indicates the candidate participant's overall probability of success. The score indicating the overall probability of success in the clinical trial can be calculated as a weighted sum of the various predicted outputs from multiple machine learning models or can be predicted directly from an output of a machine learning model trained to generate such a value. In an implementation where a weighted sum is calculated, the weight values can be specific to the ongoing or potential clinical trial and can each correspond to a respective predicted outcome or predicted attribute for the ongoing or potential clinical trial.

[0200] In Act 420, the data processing system may store an association between the ongoing or potential clinical trial and an identifier for each candidate participant identified as meeting one or more inclusion criteria. The association may indicate that each candidate participant meets a selection criterion for the first clinical trial in response to determining that the candidate participant's predicted outcome for the first clinical trial meets a selection criterion for that trial. In some implementations, the data processing system may generate a report indicating the subset of candidate participants identified as suitable for the ongoing or potential clinical trial.

[0202] The report or indications of whether candidate participants are suitable for the clinical trial can be provided as output by the data processing system. In some implementations, the report/indications can be provided via a screen on the data processing system. In some implementations, the report/indications can be provided via a web-based interface hosted by or otherwise associated with the data processing system. In some implementations, the report/indications can be transmitted to one or more remote computer systems. In some implementations, the subset of candidate participants can be categorized according to a predicted overall probability of success in the ongoing or potential clinical trial.

[0204] FIG. 5 is a component diagram of an example computer system suitable for use in the various implementations described in this disclosure, according to an example implementation. For example, computer system 500 may implement data processing system 105 or remote computer system 160 of FIG. 1, or various other example systems and devices described in this disclosure.

[0206] The computer system 500 includes a bus 502 or other communication component for communicating information and a processor 504 coupled to the bus 502 for processing information. The computer system 500 also includes main memory 506, such as RAM or another dynamic storage device, coupled to the bus 502 for storing information and instructions to be executed by the processor 504. The main memory 506 can also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 504. The computer system 500 may further include a ROM 508 or other static storage device coupled to the bus 502 for storing static information and instructions for the processor 504. A storage device 510, such as a solid-state drive, magnetic disk, or optical disk, is coupled to the bus 502 for persistently storing information and instructions.

[0208] The computer system 500 can be coupled via bus 502 to a display 514, such as a liquid crystal display or active-matrix display, to present information to a user. An input device 512, such as a keyboard including alphanumeric and other keys, can be coupled to bus 502 to communicate information and command selections to the processor 504. In another implementation, the input device 512 has a touchscreen. The input device 512 can include any type of biometric sensor or cursor control, such as a mouse, trackball, or cursor direction keys, to communicate direction information and command selections to the processor 504 and to control cursor movement on the display 514.

[0210] In some implementations, the 500 computer system may include a 516 communications adapter, such as a network adapter. The 516 communications adapter can be coupled to the 502 bus and can be configured to allow communication with a computer or communications network or other computer systems. In various illustrative implementations, any type of network configuration can be achieved using the 516 communications adapter, such as wired (e.g., via Ethernet), wireless (e.g., via Wi-Fi, Bluetooth), satellite (e.g., via GPS), preconfigured, ad-hoc, LAN, WAN, and the like.

[0212] According to various implementations, the processes of the illustrative implementations described herein can be achieved by the computer system 500 in response to the processor 504 executing an implementation of instructions contained in main memory 506. Such instructions can be read into main memory 506 from another computer-readable medium, such as the storage device 510. The execution of the implementation of instructions contained in main memory 506 causes the computer system 500 to perform the illustrative processes described herein. One or more processors in a multiprocessing implementation can also be employed to execute the instructions contained in main memory 506. In alternative implementations, hardwired circuitry can be used instead of, or in combination with, software instructions to implement the illustrative implementations. Thus, implementations are not limited to any specific combination of hardware and software circuitry.

[0214] The implementations described in this document have been described with reference to drawings. The drawings illustrate certain details of specific implementations that implement the systems, methods, and programs described herein. However, describing the implementations with drawings should not be interpreted as imposing on the disclosure any limitations that may be present in the drawings.

[0216] It should be understood that no claim in this memorandum is to be construed under the provisions of 35 U.S.C. § 112(f), unless the claim is expressly mentioned using the phrase “means to”.

[0218] As used herein, the term “circuit” may include hardware structured to perform the functions described herein. In some implementations, each respective “circuit” may include machine-readable means for configuring the hardware to perform the functions described herein. The circuit may materialize as one or more circuitry components, including, but not limited to, processing circuitry, network interfaces, peripheral devices, input devices, output devices, sensors, etc. In some implementations, a circuit may take the form of one or more analog circuits, electronic circuits (e.g., integrated circuits (ICs), discrete circuits, system-on-a-chip (SoC) circuits), telecommunications circuits, hybrid circuits, and any other type of “circuit.” In this respect, the “circuit” may include any type of component for achieving or facilitating the achievement of the operations described herein. For example, a circuit as described in this document may include one or more transistors, logic gates (e.g., NAND, AND, NOR, OR, XOR, NOT, XNOR), resistors, multiplexers, registers, capacitors, inductors, diodes, wiring, etc.

[0220] The “circuit” may also include one or more processors communicatively coupled to one or more memories or memory devices. In this respect, the one or more processors may execute instructions stored in memory or may execute instructions otherwise accessible to the one or more processors. In some implementations, the one or more processors may materialize in various ways. The one or more processors may be constructed in a manner sufficient to perform at least the operations described in this memory. In some implementations, the one or more processors may be shared by multiple circuits (e.g., circuit A and circuit B may comprise or otherwise share the same processor, which, in some example implementations, may execute instructions stored in, or otherwise accessed through, different memory areas). Alternatively or additionally, the one or more processors may be structured to perform or otherwise execute certain operations independent of one or more coprocessors.

[0222] In other example implementations, two or more processors may be coupled via a bus to enable independent, parallel, developing, or multithreaded instruction execution. Each processor may be implemented as one or more general-purpose processors, ASICs, FPGAs, GPUs, TPUs, digital signal processors (DSPs), or other suitable structured data processing electronic components to execute instructions provided by memory. The one or more processors may take the form of a single-core processor, a multi-core processor (e.g., a dual-core, triple-core, or quad-core processor), a microprocessor, etc. In some implementations, the one or more processors may be external to the device; for example, the one or more processors may be a remote processor (e.g., a cloud-based processor). Alternatively or additionally, the one or more processors may be internal or local to the device. In this respect, a given circuit or its components can be located locally (e.g., as part of a local server or computer system) or remotely (e.g., as part of a remote server such as a cloud-based server). To this end, a “circuit” as described herein may include components distributed across one or more locations.

[0224] An exemplary system for implementing the overall system or parts of the implementations could include a general-purpose computing device in the form of a computer, comprising a processing unit, system memory, and a system bus that connects the various system components, including the system memory, to the processing unit. Each memory device may include non-volatile storage media, non-volatile storage media, non-volatile storage media (e.g., one or more volatile or non-volatile memories), etc. In some implementations, the non-volatile media may take the form of ROM, flash memory (e.g., flash memory such as NAND, 3D NAND, NOR, 3D NOR), EEPROM, MRAM, magnetic storage, hard disks, optical disks, etc. In other implementations, the volatile storage media may take the form of RAM, TRAM, ZRAM, etc. Combinations of the above are also included within the scope of machine-readable media. In this respect, machine-executable instructions comprise, for example, instructions and data that cause a general-purpose computer, special-purpose computer, or special-purpose processing machines to perform a certain function or group of functions. Each respective memory device can be operated to maintain or otherwise store information related to the operations performed by one or more associated circuits, including processor instructions and related data (e.g., database components, object code components, script components), as described in the implementation examples provided herein.

[0226] It should also be noted that the term “input devices,” as described herein, may include any type of input device, including but not limited to a keyboard, keypad, mouse, joystick, or other input devices that perform a similar function. Comparatively, the term “output device,” as described herein, may include any type of output device, including but not limited to a computer monitor, printer, fax machine, or other output devices that perform a similar function.

[0228] It should be noted that although the diagrams in this specification may show a specific order and composition of method steps, it is understood that the order of these steps may differ from that depicted. For example, two or more steps may be performed concurrently or with partial concurrency. Also, some method steps that are performed as discrete steps may be combined, steps that are performed as a combined step may be separated into discrete steps, the sequence of certain processes may be reversed or otherwise varied, and the nature or number of discrete processes may be altered or varied. The order or sequence of any element or apparatus may be varied or substituted according to alternative implementations. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the appended claims. Such variations will depend on the machine-readable media systems and hardware systems chosen and the designer's choice. It is understood that all such variations are within the scope of the disclosure. Similarly, software and web implementations of this disclosure could be achieved using standard programming techniques with rule-based logic and other logic to achieve the various stages of database search, correlation stages, comparison stages, and decision stages.

[0229] Although this specification contains many implementation-specific details, these should not be construed as limitations on the scope of the inventions or on what may be claimed, but rather as descriptions of specific features for particular implementations of the systems and methods described herein. Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple separate implementations or in any suitable subcombination. Furthermore, although features acting in certain combinations may have been previously described and even initially claimed as such, one or more features of a claimed combination may in some cases be split off from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0231] In certain circumstances, multitasking and parallel processing can be advantageous. Furthermore, it should not be understood that the separation of various system components in the implementations described above requires such separation in all implementations, and it should be understood that the program components and systems described can generally be integrated together into a single software product or packaged into multiple software products.

[0233] Having described some illustrative implementations, it is evident that the foregoing is illustrative and not limiting, and has been presented by way of example. In particular, although many of the examples presented in this dissertation involve specific combinations of method acts or system elements, those acts and elements can be combined in other ways to achieve the same objectives. Acts, elements, and features discussed only in relation to one implementation are not intended to be excluded from a similar role in other implementations.

[0235] The phraseology and terminology used in this document are for descriptive purposes and should not be considered exhaustive. The use of "including," "comprising," "having," "containing," "implying," "characterized by," "characterized in that," and variations thereof in this document is understood to encompass the elements listed after them, their equivalents, and additional elements, as well as alternative implementations consisting solely of the elements listed after them. In one implementation, the systems and methods described in this document consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0237] References to implementations or elements or acts of the systems and methods in this document referred to in the singular may also encompass implementations that include a plurality of these elements, and references in the plural to any implementation or element or act in this document may also encompass implementations that include only a single element. References in singular or plural forms are not intended to limit the systems or methods described herein, their components, acts, or elements to a single or plural configuration. References to any act or element that is based on any information, act, or element may include implementations where the act or element is based at least in part on any information, act, or element.

[0239] Any implementation described in this document may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternative implementation,” “various implementations,” “an implementation,” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in relation to the implementation may be included in at least one implementation. Such terms as employed in this document do not all necessarily refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations described in this document.

[0241] References to “or” may be interpreted as inclusive so that terms described using “or” may indicate any of a single, more than one, and all of the terms described.

[0242] Where technical features in the drawings, the detailed description, or any claim are followed by reference signs, the reference signs have been included for the sole purpose of enhancing the intelligibility of the drawings, the detailed description, and the claims. Accordingly, neither the reference signs nor their absence have a limiting effect on the scope of the elements of the claim.

[0244] The foregoing description of the implementations has been presented for illustrative and descriptive purposes. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and modifications and variations are possible in light of prior learning or that may be acquired from this disclosure. The implementations have been chosen and described to explain the principles of the disclosure and their practical application, to enable a person skilled in the art to use the various implementations and with various modifications as suitable for the particular use contemplated. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and implementation of the implementations without leaving the scope of this disclosure as expressed in the appended claims.

Claims (20)

1. CLAIMS 1. A system comprising: one or more processors coupled to non-transient memory, the one or more processors configured to: receive data from a plurality of participants in one or more first clinical trials; Generate, using the data from the plurality of participants, a training dataset comprising a plurality of examples, each example comprising: a medical image or set of medical images of a respective participant in one or more of the first clinical trials, and a label corresponding to a result of the respective participant from one or more initial clinical trials; and Train a machine learning model using the training dataset to receive a second medical image of a second participant as input and generate a predicted outcome of the second participant in a second clinical trial as output. 2. The system of claim 1, wherein the one or more processors are further configured to: generate the training dataset to also include, for each example of the plurality of examples, one or more clinical attributes of the respective participant from the first one or more clinical trials. 3. The system of claim 1, wherein the one or more processors are further configured to: to generate, using an image harmonization function, a set of harmonized medical images based on data from multiple participants; and Generate the training dataset to include the harmonized medical image set. 4. The system of claim 1, wherein the medical image in each example comprises a computed tomography (CT) scan image, scan image positron emission tomography (PET), bone scanning imaging, ultrasound (US), X-rays, or a magnetic resonance imaging (MRI) scan. 5. The system of claim 1, wherein the label comprises one or more of an indication of a response to the first one or more clinical trials, an indication of additional injury during or after the first one or more clinical trials, an indication of a histology type, an indication of a germline mutation status, an indication of a somatic mutation status, an indication of a certain degree of protein expression or lack thereof, an indication of a certain degree of a gene expression signature or lack thereof, or an indication of a change in a tumor stage resulting from the first one or more clinical trials. 6. The system of claim 1, wherein each instance of the plurality of instances further comprises a second label identifying a time-to-event metric for the respective participant in the first one or more clinical trials, wherein the one or more processors are further configured to: Train a second machine learning model using the training dataset to receive the second medical image of the second participant as input and generate a time-to-the-predicted event metric for the second participant during the second clinical trial. 7. The system of claim 1, wherein the one or more processors are further configured to: receive the medical image of a candidate participant from a computer system associated with the second clinical trial; to provide, as input, the medical image of the candidate participant to the machine learning model to generate the predicted outcome of the candidate participant during the second clinical trial; and to provide the computer system with an indication of whether the predicted outcome of the candidate participant meets a selection criterion for the second clinical trial. 8. A system comprising: one or more processors and non-transient memory, the one or more processors configured to: to identify a medical image or set of medical images of a candidate participant in one or more clinical trials; providing, to a machine learning model, as input, the medical image of the candidate participant to generate a predicted outcome of the candidate participant for a first clinical trial, the machine learning model trained using a training dataset comprising a plurality of examples, each example comprising a respective radiological image of a respective participant from one or more second clinical trials, and a label corresponding to an outcome of the respective participant from one or more second clinical trials; to determine, in response to providing the medical image as input to the machine learning model, that the predicted outcome of the candidate participant for the first clinical trial satisfies a selection criterion of the first clinical trial; and to store, in one or more data structures, an association between a candidate participant identifier, the first clinical trial to indicate that the candidate participant meets the selection criteria of the first clinical trial in response to determining that the predicted outcome of the candidate participant for the first clinical trial meets the selection criteria of the first clinical trial. 9. The system of claim 8, wherein the machine learning model is trained to further receive clinical attributes as input, and wherein the one or more processors are further configured to: identify one or more clinical attributes of the candidate participant; and Provide the machine learning model, as input, with one or more clinical attributes along with the medical image or set of medical images of the candidate participant to generate the predicted outcome for the candidate participant for the first clinical trial. 10. The system of claim 8, wherein the one or more processors are further configured to: provide the candidate participant's medical image as input to a segmentation model to identify one or more regions of interest in the medical image or set of medical images. 11. The system of claim 10, wherein the machine learning model is further trained to receive image features, including radiomic features, as input, and wherein the one or more processors are further configured to: Extract, based on one or more regions of interest in the medical image, one or more radiomic features from the medical image of the candidate participant. 12. The system of claim 8, wherein the one or more processors are further configured to: Provide the medical image or set of medical images of the candidate participant as input to a second machine learning model to generate a time-to-the-predicted event metric for the candidate participant during the first clinical trial. 13. The system of claim 8, wherein the one or more processors are further configured to: Perform an image harmonization function using the candidate participant's medical image or set of medical images before providing the medical image or set of medical images as input to the machine learning model. 14. The system of claim 8, wherein the one or more processors are further configured to: generate a report that indicates a set of suitable candidate participants for the first clinical trial, the set of candidate participants comprises the candidate participant. 15. A method, comprising: to receive, by one or more processors coupled to non-transient memory, data from a plurality of participants in one or more first clinical trials; Generate, by one or more processors, using data from the plurality of participants, a training dataset comprising a plurality of examples, each example comprising: a medical image or set of medical images of a respective participant in one or more of the first clinical trials, and a label corresponding to a result of the respective participant from one or more initial clinical trials; and train, by one or more processors, a machine learning model using the training dataset to receive a second medical image of a second participant as input and generate a predicted outcome of the second participant in a second clinical trial as output. 16. The method of claim 15, further comprising: generate, by one or more processors, the training dataset to also include, for each example of the plurality of examples, one or more clinical attributes of the respective participant from the first one or more clinical trials. 17. The method of claim 15, further comprising: to generate, by one or more processors, using an image harmonization function, a set of harmonized medical images based on data from multiple participants; and generate, by one or more processors, the training dataset to include the harmonized medical image set. 18. A method, comprising: to identify, by one or more processors coupled to non-transient memory, a medical image or set of medical images of a candidate participant in one or more clinical trials; providing, by one or more processors, to a machine learning model, as input, the medical image or set of medical images of the candidate participant to generate a predicted outcome of the candidate participant for a first clinical trial, the machine learning model trained using a training dataset comprising a plurality of examples, each example comprising a respective medical image or set of medical images of a respective participant of one or more second clinical trials, and a label corresponding to an outcome of the respective participant of the one or more second clinical trials; to determine, by one or more processors, in response to providing the medical image or set of medical images as input to the learning model of machine, that the predicted outcome of the candidate participant for the first clinical trial meets a selection criterion of the first clinical trial; and to store, by one or more processors, in one or more data structures, an association between a candidate participant identifier, the first clinical trial to indicate that the candidate participant meets the selection criteria of the first clinical trial in response to determining that the predicted outcome of the candidate participant for the first clinical trial meets the selection criteria of the first clinical trial. 19. The method of claim 18, further comprising: to identify, by one or more processors, one or more clinical attributes of the candidate participant; and Provide, by one or more processors, to the machine learning model as input, the one or more clinical attributes with the medical image or set of medical images of the candidate participant to generate the predicted outcome for the candidate participant for the first clinical trial. 20. The method of claim 18, further comprising: generate, by one or more processors, a report indicating a set of candidate participants suitable for the first clinical trial, the set of candidate participants comprising the candidate participant.