Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/09—Supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/047—Probabilistic or stochastic networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/088—Non-supervised learning, e.g. competitive learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N5/00—Computing arrangements using knowledge-based models
  • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound

Definitions

• the present invention relates to a method for evaluating the performance of a prediction algorithm.
• the present invention also relates to a computer program product and an associated readable information medium.
• the present invention is in the field of the development of prediction algorithms that have been learned by using a machine learning technique.
• Machine learning is referred to by many different terms such as the term “machine learning”, the term “automatic learning”, the term “artificial learning” or the term “statistical learning”. Machine learning involves using data to learn a prediction algorithm.
• the description describes a method for evaluating the performance of a prediction algorithm for a predefined use case, the prediction algorithm predicting for given inputs the value of one or more outputs, the prediction algorithm having been learned by using a machine learning technique and a learning data set, the method comprising a step of obtaining data sets, each datum of a data set corresponding to the output values that the prediction algorithm should give in the presence of the input values of the data set, a step of receiving the probability for each data set that a data set is observed during the use case of the prediction algorithm, a step for collecting the outputs predicted by the prediction algorithm for each input value of the data of the datasets, a step for determining the distribution of the prediction accuracy of the predicted output for each data set to obtain determined distributions, a step of aggregating the determined distributions by employing an aggregation function using the received probabilities to obtain an aggregated prediction accuracy distribution , and a step of applying at least one risk metric to the aggregated prediction accuracy distribution to obtain at least one indicator of the performance of the prediction
• the evaluation method has one or more of the following characteristics, taken in isolation or in all technically possible combinations:
• a risk metric is a quantile metric and an indicator of the performance of the prediction algorithm is the value of a quantile of a predetermined level.
• a risk metric is a conditional expectation and an indicator of the performance of the prediction algorithm is a value of the conditional expectation.
• the prediction accuracy is calculated using an evaluation metric, the evaluation metric being an average of the absolute prediction error, a quantile metric, or an empirical moment of the accuracy distribution of prediction.
• the prediction accuracy is calculated using a reference prediction algorithm.
• the process includes the establishment of a report giving all the information that made it possible to obtain the performance indicator.
• the obtaining step is implemented by generating each data set from a reference data set according to a given probability distribution.
• the obtaining step is implemented, for each data set, by generation by a generative model of initial data and by selection according to a given probability law of the initial data to form the data set.
• the obtaining step involves modifying the data sets by introducing imperfections in the environment of the system that the prediction algorithm models.
• the obtaining step involves modifying the data sets by introducing adverse disturbances aimed at manipulating the outputs of the prediction algorithm.
• the present description also describes a computer program product comprising a readable information medium, on which is stored a program computer comprising program instructions, the computer program being loadable on a data processing unit and implementing an evaluation method as previously described when the computer program is implemented on the unit of data processing.
• the present description also describes a readable information medium comprising program instructions forming a computer program, the computer program being loadable on a data processing unit and implementing an evaluation method as previously described. when the computer program is implemented on the data processing unit.
• Figure 1 is a schematic representation of a computer program system and product
• Figure 2 is a flowchart of an example implementation of a method for evaluating a prediction algorithm.
• FIG. 1 A system 10 and a computer program product 12 are shown in Figure 1.
• the interaction between the system 10 and the computer program product 12 allows the implementation of a method for evaluating a prediction algorithm.
• the evaluation process is thus a computer-implemented process.
• System 10 is a desktop computer.
• system 10 is a rack-mounted computer, laptop, tablet, personal digital assistant (PDA), or smartphone.
• PDA personal digital assistant
• the computer is adapted to operate in real time and/or is in an on-board system, in particular in a vehicle such as an airplane.
• the system 10 comprises a calculation unit 14, a user interface 16 and a communication device 18.
• the calculation unit 14 is an electronic circuit designed to manipulate and/or transform data represented by electronic or physical quantities in registers of the system 10 and/or memories into other similar data corresponding to physical data in the register memories or other types of display devices, transmission devices or storage devices.
• the computing unit 14 includes a single-core or multi-core processor (such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller and a digital signal processor (DSP)), a programmable logic circuit (such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a logic device (PLD) and programmable logic arrays (PLA)), a state machine, a logic gate and discrete hardware components.
• a single-core or multi-core processor such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller and a digital signal processor (DSP)
• DSP digital signal processor
• ASIC application-specific integrated circuit
• FPGA field-programmable gate array
• PLD logic device
• PLA programmable logic arrays
• the calculation unit 14 comprises a data processing unit 20 adapted to process data, in particular by performing calculations, memories 22 adapted to store data and a reader 24 adapted to read a computer-readable medium.
• the user interface 16 includes an input device 26 and an output device 28.
• the input device 26 is a device allowing the user of the system 10 to enter information or commands on the system 10.
• the input device 26 is a keyboard.
• the input device 26 is a pointing device (such as a mouse, touchpad, and graphics tablet), a voice recognition device, an eye tracker, or a haptic (motion analysis) device.
• the output device 28 is a graphical user interface, i.e. a display unit designed to provide information to the user of the system 10.
• the output device 28 is a display screen allowing visual presentation of the output.
• the output device is a printer, an augmented and/or virtual display unit, a speaker or other sound generating device for presenting the output in sound form, a unit producing vibrations and/or odors or a unit adapted to produce an electrical signal.
• the input device 26 and the output device 28 are the same component forming human-machine interfaces, such as an interactive screen.
• the communication device 18 allows unidirectional or bidirectional communication between the components of the system 10.
• the communication device 18 is a bus communication system or an input/output interface.
• the presence of the communication device 18 allows that, in certain embodiments, the components of the system 10 are remote from each other.
• Computer program product 12 includes computer readable medium 32.
• the computer-readable medium 32 is a tangible device readable by the reader 24 of the calculation unit 14.
• the computer-readable medium 32 is not a transient signal per se, such as radio waves or other freely propagating electromagnetic waves, such as light pulses or electronic signals.
• Such a computer-readable storage medium 32 is, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. these.
• computer-readable storage medium 32 is a mechanically encoded device, such as punched cards or grooved relief structures, floppy disk, hard disk, ROM (ROM), Random Access Memory (RAM), Erasable Programmable Read Only Memory (EROM), Electrically Erasable Readable Memory (EEPROM), Magneto-Optical Disk, Static Random Access Memory (SRAM), Compact Disc ( CD-ROM), digital versatile disk (DVD), USB flash drive, floppy disk, flash memory, solid state disk (SSD) or PC card such as a PCMCIA memory card.
• ROM Read Only Memory
• EROM Erasable Programmable Read Only Memory
• EEPROM Electrically Erasable Readable Memory
• SRAM Static Random Access Memory
• CD-ROM Compact Disc
• DVD digital versatile disk
• USB flash drive floppy disk
• SSD solid state disk
• PCMCIA memory card such as a PCMCIA memory card.
• a computer program is stored on computer-readable storage medium 32.
• the computer program includes one or more stored program instruction sequences.
• Such program instructions when executed by data processing unit 20, cause steps of the evaluation method to be executed.
• the form of program instructions is source code form, computer executable form, or any intermediate form between source code and computer executable form, such as the form resulting from source code conversion through an interpreter , assembler, compiler, linker, or locator.
• the program instructions are microcode, firmware instructions, state definition data, integrated circuit configuration data (eg VHDL) or object code.
• Program instructions are written in any combination of one or more languages, for example an object-oriented programming language (FORTRAN, C++, JAVA, HTML), a procedural programming language (C language for example).
• object-oriented programming language for example
• C++ JAVA
• HTML JAVA
• C language for example
• program instructions are downloaded from an external source via a network, as is notably the case for applications.
• the computer program product comprises a data medium readable by computer on which the program instructions are stored or a data carrier signal on which the program instructions are encoded.
• the computer program product 12 comprises instructions which can be loaded into the data processing unit 20 and adapted to cause the execution of the evaluation method when they are executed by the processing unit. of data 20.
• the execution is entirely or partially carried out either on the system 10, that is to say a single computer, or in a system distributed between several computers (in particular via the use of cloud computing).
• FIG. 2 is a flowchart illustrating an example of implementation of the evaluation method.
• the evaluation method is a method of evaluating the performance of a prediction algorithm for several distinct use cases.
• the prediction algorithm is able to predict for given inputs the value of one or more outputs.
• the algorithm was learned by using a machine learning technique and a training dataset.
• the algorithm is a supervised statistical learning algorithm.
• the prediction algorithm is, for example, a support vector machine (better known under the English name of “Support Vectors Machine”), a neural network (better known under the English name of “Neural Network”) or a tree random (better known as “Random Forest”). More generally, any type of supervised prediction algorithm is possible for the present context.
• Such a prediction algorithm can be used for very diverse contexts such as image classification, recognition of three-dimensional shapes or decision support in the context of piloting autonomous drones.
• the prediction algorithm takes as input and/or gives as output physical quantities corresponding to measurements from one or more sensors.
• the prediction algorithm is a digit recognition algorithm.
• the recognition algorithm takes an image as input and determines the number contained in the image.
• the method comprises the following steps: a step for obtaining E50, a step for receiving E52, a step for collecting E54, a step for determining E56, a step for aggregating E58, a step for applying E60 and a step for establishment E62.
• n data sets are obtained.
• n p and N are defined more precisely in the remainder of the description.
• Each data set comprises a respective number n T of data.
• the number n T of data of a game is at least greater than 2, preferably greater than or equal to 100, and depends in practice on a time horizon T characteristic of the operational use of the prediction algorithm / considered.
• Each datum of a dataset corresponds to the output values that the prediction algorithm should give in the presence of the input values of the dataset.
• Such a set of references is a set comprising an image of each digit and comprising the associated digit.
• the obtaining step E50 is then implemented by generating each data set from the reference data set according to a given probability law.
• the generation is implemented by a random drawing according to a law of probability.
• the law of probability is a uniform law.
• the number 1 is more frequent than the others.
• the law of probability may favor the generation of data sets with the number 1 .
• the step of obtaining E50 is implemented, for each set of data, by generation by a generative model of initial data and by selection according to a given law of probability of the initial data to form the set of data .
• a generative model is an automatic learning algorithm that seeks to describe the data, subsequently allowing new samples to be generated according to the description (i.e. probability law) determined during the learning phase.
• a classic example is a generative adversarial network (more often referred to by the acronym of GAN in reference to the English name of "Generative Adversary Network”, allowing the synthesis of very realistic (fictitious) images from real images.
• GAN generative adversarial network
• VAE Variational Auto-encoder
• the obtaining step E50 includes the modification of the data sets (generated data sets or reference data sets) by introducing imperfections in the environment of the system that the prediction algorithm models.
• the datasets can be modified taking into account the imperfections of the scanner used.
• geometric transformations taking into account noise or external disturbances are considered for the modification.
• the step of obtaining E50 is the result of the implementation of a generation of input/output pairs making it possible to obtain various realizations of random variables (x,y) under the measure of probability P 3 ⁇ 4y .
• the probability is received for each set of data that a set of data is observed during a case of use of the prediction algorithm.
• the datasets correspond to black and white images whereas, in the use case, the images are in color
• the case where the color images will be comparable to the case of black images and white have a certain probability.
• the distribution of digits in use is not evenly distributed, so that the probability of having certain digits is higher than the probability of having other digits.
• the outputs predicted by the prediction algorithm for each input value of the data of the data sets are collected.
• the prediction algorithm is applied to the input values and the result is collected by the system 10.
• the value predicted by the prediction algorithm and the value that the prediction algorithm should have predicted (true value) are thus known.
• the distribution of the prediction accuracy of the prediction algorithm is determined.
• the accuracy of the prediction is obtained by applying an evaluation metric to a prediction error.
• Such an evaluation metric will be denoted ⁇ below.
• a prediction error corresponds to the following quantity: Where: ⁇ x and y are realizations of the random variables x and y of joint distribution P ⁇ , ⁇ and ⁇ is a precision metric, also called a loss function.
• the prediction accuracy is calculated by a metric by means of the absolute prediction error.
• the prediction accuracy is calculated by using a cross-entropy function.
• the prediction accuracy is evaluated using a quantile metric. More generally, the evaluation metric is constructed by applying a metric ⁇ to the empirical distribution of the random variable Otherwise formulated, the evaluation metric is an empirical moment of the prediction accuracy distribution.
• the prediction accuracy is calculated by a metric using a reference prediction algorithm noted g.
• the evaluation metric is a function ⁇ of the distribution of the relative precision between the algorithm f to be evaluated and the reference prediction algorithm g.
• the metric is the mean of evaluation of the relative l-differences, which is written mathematically as follows: It is also possible to refine the preceding metrics by conditioning them with respect to the precision of the reference prediction algorithm g.
• the conditional mean of relative l-differences is a particular example of such a metric, which is mathematically written as:
• the evaluation metric is calculated on each dataset independently.
• evaluation metric can be seen as a risk metric.
• n p distributions of the prediction error of the predicted output are thus obtained.
• Each distribution is thus a determined distribution specific to a respective data set sampled according to a distribution H s x i >y , with i an integer between 1 and where N represents the number of datasets used to obtain a realization of the prediction error calculation.
• Each of these determined distributions is denoted with i an integer between 1 and n p .
• the determined distributions are aggregated to obtain an aggregate prediction error distribution or aggregate prediction distribution noted
• an aggregation function using the received probabilities is used.
• the aggregation function is a weighted sum whose weights m; depend on the received probabilities.
• a risk metric MR is applied to the aggregated prediction error distribution.
• the value of the MR risk metric provides an indicator of the performance of the prediction algorithm.
• the method can include a subsequent step of aggregating the performance indicators obtained.
• the risk metric MR is the quantile of the aggregate prediction error distribution.
• the indicator of the performance of the prediction algorithm is the value of a quantile of a predetermined level (here level a).
• Level a is determined according to the criticality of the application envisaged for the prediction algorithm.
• the level a of the order of 5/100 could be acceptable whereas for a prediction algorithm in the field of transport, an acceptable level of confidence a could be order of 10 7 .
• VaR refers to the English name of “Value-at-Risk” literally meaning “value at risk”, or “value at stake”.
• the VaR risk metric is defined for a given time horizon T, a set S of scenarios and a confidence level a.
• the risk metric VaR thus corresponds to the amount of losses that should only be exceeded with a given probability over the time horizon T.
• the time horizon depends in particular on the intended use case.
• the time horizon takes into account the various constraints that may prevent a new training (re-calibration) of the algorithm.
• an algorithm embedded in a satellite will have a greater risk horizon than a spam classifier.
• the set S of scenarios corresponds to a probability measure specified jointly on the inputs and outputs considered. The probability measurement depends in particular on the operational application and the various risks to be measured.
• this probability distribution represents a distribution associated with the use of the algorithm which is decorrelated from the distribution of inputs/outputs used during the training of the algorithm of prediction.
• an estimator called the empirical mean risk
• an estimator is defined by:
• the present method also rules out the use of the notion of average risk by introducing an evaluation metric f operating on the training and use distributions
• the training error is thus defined as follows:
• the confidence level a corresponds to the level a of the risk metric MR1.
• the evaluation measure f is calculated by constructing a distribution representative of the evaluation of the algorithm, over N realizations with j an integer between 1 and N, obtained from the distribution P 3 ⁇ 4jy , for the considered time horizon T.
• the number n T of samples to be considered per realization depends on the fixed time horizon and frequency of use of the algorithm.
• the risk metric is a conditional expectation and an indicator of the performance of the prediction algorithm is a value of the conditional expectation.
• MR2 designates the risk metric obtained as the value of a conditional expectation and EC2 the limit value used for conditioning.
• Such a performance indicator is as effective as a metric known by the acronym of C-VaR referring to the English name of "Conditional Value-at- Risk” literally meaning “conditional value at risk”. This metric is also known as “Expected-Shortfall”, literally meaning “expected losses”.
• the method described with reference to FIG. 2 also comprises a step E62 of establishing a report giving all the information that made it possible to obtain the performance indicator.
• such a report is generated in pdf format.
• the report is generated from latex code generated on the fly.
• the content of the report is, according to the proposed example, a set of three types of information.
• the first type of information groups together the information provided at the start of the process, highlighting in particular the number of algorithms to be evaluated, the number of scenarios (datasets) considered and the various risk metrics to be considered.
• the second type of information relates to the risks for each algorithm for a dataset and a risk metric.
• a histogram of the risk metric values is presented in the form of a histogram or a graph.
• the third type of information relates to the aggregated risks for each algorithm for a risk metric.
• the algorithm, the risk metric, the different priors used and possibly an aggregation of the different values found for the different performance indicators will be indicated.
• Such a report is directly usable for a user.
• the present process is thus a tooled process for the measurement and management of risks associated with the use of supervised statistical learning algorithms in a specific framework.
• the process gives a realistic measurement of the performance of the algorithm on all the data that can be submitted to it, unlike a certification process which would only validate the performance of the algorithm under these specific conditions of use.
• the method makes it possible to obtain good precision for all the cases that may occur in practice.
• This method has the advantage of being generic in the sense that the method can be applied to any type of prediction algorithm in a supervised learning context, any type of input and any use case envisaged.
• the present method can be used at all stages of the life of the algorithm, and in particular in the development phase, the validation phase and the monitoring phase (after deployment of the algorithm).
• the process makes it possible to obtain the performance and therefore facilitates the comparison of two candidate versions of the algorithm.
• the development is therefore accelerated.
• the process can also be used to calibrate the values of hyper-parameters or to associate confidence indicators with the outputs of the selected algorithms.
• the validation phase aims to ensure that the algorithms developed are suitable for the intended operational use, focusing in particular on the impacts linked to the various sources of potential errors associated with their use.
• the range of inputs/outputs for which the algorithm behaves in line with the intended operational application is specified in particular during this phase.
• the follow-up phase consists of periodically evaluating the prediction algorithm to ensure that the validation carried out remains valid for the conditions of use observed.
• the method was implemented by the applicant according to several modules, namely a module for obtaining data sets, a module for collecting predicted values, a determination module, an aggregation module and an application module.
• modules namely a module for obtaining data sets, a module for collecting predicted values, a determination module, an aggregation module and an application module.
• Each of these modules has been successfully implemented in the Python programming language.
• Such a modular implementation makes the method easily adaptable to all types of algorithm since each module is relatively independent. Furthermore, the method can be easily parallelized, which makes it possible to limit the computational load, in particular by the use of a distributed computing structure.
• the method further comprises a display of all of said information on the output device 28 which then serves as a graphical interface with a user.
• Such a display would replace or supplement the established report by allowing the user to view the various results and performance indicators and to conduct detailed analyzes by allowing navigation in the various results files and modularity in the choice of risk metrics.
• the user will be able, if he wishes, to convert a given algorithmic risk tolerance level into validity ranges for the algorithms considered via the analysis of the contributions of the different scenarios.
• the graphical interface makes it possible to constitute a configuration file giving all the information useful for the implementation of the method, in particular the information of the steps of obtaining E50 and receiving E52.
• the output device 28 allows the user to enter the associated data or to select them via the use of drop-down menus.
• the configuration file includes the parameters needed to define the risk metrics to be calculated, the algorithms to be evaluated, the precision metrics to be considered, the different data sets to be considered, the methods for simulating the sets of datasets, reference algorithms and priors.
• Such an evaluation method is thus a method for measuring the performance of an algorithm.
• the evaluation method measures the precision of the predictions of the algorithm. If the algorithm is a temperature prediction method, the evaluation method thus measures the temperature difference between the actual temperature and the predicted temperature.
• the performance evaluated is thus objective data since it is a measurement.
• the way in which the performance is evaluated, that is to say the index or indices chosen to evaluate it, is irrelevant here.
• the evaluation process thus becomes a technical process that solves a problem of precision of a measurement.
• the evaluation process is thus a technical solution to a technical problem.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Software Systems (AREA)
• Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• General Engineering & Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Evolutionary Computation (AREA)
• Mathematical Physics (AREA)
• General Physics & Mathematics (AREA)
• Computing Systems (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Medical Informatics (AREA)
• Biomedical Technology (AREA)
• Molecular Biology (AREA)
• General Health & Medical Sciences (AREA)
• Computational Linguistics (AREA)
• Biophysics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Health & Medical Sciences (AREA)
• Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=75438859

Family Applications (1)

Country Status (4)

Families Citing this family (4)

Family Cites Families (9)

• 2020
  • 2020-11-30 FR FR2012387A patent/FR3116926A1/fr active Pending
• 2021
  • 2021-11-30 US US18/254,749 patent/US20240028960A1/en active Pending
  • 2021-11-30 EP EP21820592.0A patent/EP4252162A1/de active Pending
  • 2021-11-30 WO PCT/EP2021/083472 patent/WO2022112583A1/fr not_active Ceased

Also Published As

Similar Documents

Legal Events