WO1998022904A2

WO1998022904A2 - Methods for objectification of subjective classifications

Info

Publication number: WO1998022904A2
Application number: PCT/DK1997/000523
Authority: WO
Inventors: Jan Meyrowitsch
Original assignee: Individual
Current assignee: Individual
Priority date: 1996-11-14
Filing date: 1997-11-14
Publication date: 1998-05-28
Anticipated expiration: 1999-05-14
Also published as: CA2274701C; DE69719091D1; DE69719091T2; WO1998022904A3; IL129956A0; DK128796A; CA2274701A1; EP1012777A2; ATE232622T1; US6494720B1; EP1012777B1; AU5048298A

Abstract

The method is based on consensus analyses of classifications at the object level carried out by learning systems. To determine persons or groups of persons ability to analyse details, e.g. in visual objects, digitised images of the objects are feed into a computer, where each image is assigned a number of characteristics, N out of Q possible. Some of the characteristics in the images are being selected, after which a person is set to identify the images by determination of the characteristics. The same image in a series of images is presented for the person several times, and if the person is capable to distinguish the images from each other, and furthermore reach the same results with the same images, then the person possess good abilities for type determination of reference signals, e.g. in form of images. The invention is suitable to identify persons, who are gifted for analysis of e.g. biological samples, such as images samples of semen samples, cancer cells, etc.

Description

Methods for objectification of subjective classifications.

D E S C R I P T I O N

The invention relates to a method for determination of the perceptual recognition ability of persons. In addition, the invention includes utilisation of this method. 5 The method is applicable to determine all type of human, recognition abilities. For exemplification, the visual recognition is used here.

Within particular the healthcare sectors, a large number of laboratory analyses are carried out every day, in particular for diagnostically and scientifically purposes. Many of these visual analyses are biological samples, in which the result of the sample are 0 analysed by image analysis, for instance an image from a microscopy or a scanner. In many cases, the image analysis can be carried out by digital image analysis, but a quite large part is carried by visual inspections, which are carried out by technicians, medical doctors, or other trained personnel. Obviously, it is mandatory, that a visual inspection is carried out carefully, and the 5 person, who is carrying out the analysis, has sufficient experience to determine details in an image from an analysis. These images may be quite complex, such as images of sperm cells, cancer cells, or the like.

So far, personnel which carry out the analyses, has been trained by more experienced personnel, and after a certain period of training, they then had to carry out the analyses 0 on their own.

As mentioned, it is extremely important, that the analyses are carried out very carefully. However, while it is humans, who are carrying out the analyses, partly by subjective assessments, it is obvious, that differences exist between the individual persons ability to carry out the analyses totally accurate, and in particular during a longer period to 5 carry out the analyses with the same care.

Until now, there has not existed any particular method to identify the ability of persons to interpret images, and in particular if any persons or groups of persons are special gifted for analysing special types of images, which might be different from person to person or from one groups of persons to other groups of persons. 0 Therefore, it is a purpose of the invention to specify a method, which makes it possible to identify persons or groups of persons, who are particular, qualified to analyse given types of images. In other words, it is desirable to be able to identify persons or groups of persons, who may find details in an image, and that is to find the same details when the same image is being shown to the person or the group a number of times. The purpose of the invention is being fulfilled by a method given by that in the introduction to Claim 1 stated description, which is characteristic by this in Claim 1 characterised stated part.

The method includes in this way 3 main parts, which are: • recording and preliminary image segmentation of for instance a visual object in a computer, providing so-called visual preconditions.

• providing of machinery preconditions.

• providing of intellectual preconditions, which are the person or the group of persons execution of identifications in a number of images. Additionally, to make the identification of the ability of persons or groups of persons more accurate, it is useful as stated in Claim 2, that the persons in addition to identify characteristics in the images also has to determine the location of the characteristics in the separate images.

By correlation of the distribution of the persons or groups of persons observations for the same images as stated in Claim 3, thereby can persons or the group of persons reproducible ability and thereby their ability precisely to identify characteristics in visual objects been further improved.

Additional appropriate types of utilisations of the methods arise moreover of the independent Claims 4 - 7. As mentioned, the invention include applications of the method.

These applications are further described in the Claims 8 and 9.

The invention are now going to be explained in more details with reference to the figure of the shown executed example, by which:

Figure 1 shows the principle system configuration of the invention. Figure 2 and Figure 3 show a functional description of parts of the system configuration according to Figure 1.

Figure 4 shows an example of classification of differently appearing objects.

Figure 5 shows two tables how to calculate the type-specific, visual recognition ability.

Figure 6 shows an example of a statistical analysis of persons results. Figure 7 shows an example of a possible number of characteristics in an image.

The invention concerns in this way methods as indicated in Figure 1 and Figure 2 to objectification of subjective classifications based on statistical consensus analyses of repeating classifications of reference-signals, which represent objects stored as digitised reference signals, by carrying out as described in Claim 1 and Figure 1 and Figure 2 for learning systems like humans or computer-based, artificially intelligent systems, specific and unambiguous correlation between classification ability and classification type as shown in Figure 3, whereby it is possible, partly to remove disturbing noise signals in the form of systematic misjudged classifications, and partly continuously to improve the reproducibility for the specific classifications.

By the known methods for determination of the reproducibility in object recognition, the objects are been classified at the sample level concerning e.g. microscopy of spermatozoa, whereby it is not possible to demonstrate mistakings between different types of objects. Furthermore, the mentioned unambiguous correlation cannot be established, whereby qualification analyses of the classification ability of systems cannot be carried out.

By performing as described in Claim 2 qualitatively estimations of system-independent, type-specific classifications, a ranked, qualitatively evaluation of the executed type classifications can be established. By qualitative evaluations of system-dependent, type-specific classifications as described in Claim 3, it is further achievable to establish a ranking of qualitatively evaluation of the classification ability of the learning systems, relatively evaluated in comparison with the other systems in a group. Complex, discrete objects, which can be divided into minor elements can constitute a basis to define new classification criteria by classifying the respective elements of the objects, and thereafter calculate the most frequent combinations of the classifications of the elements, as described in Claim 4.

By selecting expert classifications as described in Claim 5 and Claim 6, these classifications can function as filters to construct compound, virtual classification sets with a high reproducibility.

By help of so-called two-dimensional scatter-plot as described in Claim 7 and Figure 4, it is possible to visualise and precisely localise the sort and the size of the specific agreements and disagreements between different classification systems. This visualisation will further contribute to select classifications in pairs, which can be visualised together with the given objects, as described in Claim 8.

By use of these visualisations, it is possible through education and training of learning systems to achieve type-specific improvements in classifications by adequate knowledge transfer, as described in Claim 9. Finally, it is possible, as described in Claim 10, to carry out by an expert qualitatively evaluations of changes in signals.

Claims

C LAI M S

1^ A method for objectification of subjective classifications of discrete and digital represented objects in form of reference signals, characterised by the reproducibility in the compound and combined abilities for different learning systems, physiological to Registrate, memory dependent to Recognise, and intellectual to Understand (RRU-abilities) to classify space- and time-varying, type- specific reference signals, which are correlated unequivocally as illustrated in Figure 3 on basis of consensus analyses of repeated classifications of the same reference signals into each of the respective classification types, as stated by the defined criteria for classification of the corresponding reference signals.

2. The method of claim 1 , by which it is characterised, that in agreement with the same established classification criteria and on basis of repeating classifications of the same reference signals carried out by a given group of different learning systems, ranking lists of the RRU-abilities to carry out classifications are established on basis of the reproducibility of the own classifications of systems multiplied with the systems agreement with the group for each of the type-specific object classifications.

3. The method of claim 1 , by which it is characterised, that in agreement with the same established classification criteria and on basis of repeating classifications of the same reference signals carried out by groups of different learning systems, ranking lists of the RRU-abilities to carry out classifications are established on basis of the reproducibility in the systems own classifications multiplied with the systems agreement with the group for all of the type-specific object classifications. The method of claim 2 and 3, by which it is characterised, that for given complex types of objects and thereby signals, separate classifications are carried out of those elements, which the objects and thereby the signals can be separated into, whereby the most frequently combinations of the different classifications of elements can be defined as new types of classifications.

5. The method of claim 2, by which it is characterised, that for a given group of different learning systems, which follow the same classification criteria, a type- specific classification is selected as an expert-classification, which from a qualitatively evaluation and ranking list is calculated on basis of the percentage scoring, which is calculated as the systems own reproducibility in the consensus analyses of the repeating classifications of the reference signals multiplied with the agreements of the respective systems with the other systems in the group for classification of the same reference signals in consistence with the same defined classification criteria. 6. The method of claim 5, by which it is characterised, that there through filtrations by use of expert classifications for each type of classification, virtual and system- independent sets of expert-classifications are being constructed for several types of classifications for all defined and practised classification types. ~ The method of claim 1 , by which it is characterised, that for a given group of different learning systems, which are using the same classification criteria, comparison and analysis in pairs are carried out of the sort and the size of agreements and disagreements between a pattern-specific expert and system classifications or mutual and likewise in pairs comparisons between classifications carried out by the learning systems of the group comprising classifications of the same reference signals based on specifically defined classification types and tally with the same defined classification criteria.

8. The method of claim 1 , by which as illustrated in Figure 5, that the classified objects and the in pairs classifications in a symbolic form are shown those physiological stimuli, which represent the respectively two-sided performed (claim

2 and 3) or constructed (claim 5 or 6) classifications, which two systems have been carried out on basis of the same objects and the same defined classification criteria.

9_;. The method of claim 1 , by which different learning systems, which are classifying the same reference signals, and which are using the same classification criteria through recursive education and training sessions in type-specific signal classifications at the same time are being presented on a symbolic form for physiological stimuli, which represent agreements and disagreements between either in pairs learning systems, which have classified the same reference signals based on the same defined classification criteria, or between virtual expert classifications of several classification types and classifications carried out by learning systems.

10. The method of claim 3, by which in conformity with the defined classification criteria expert consensus analyses are being carried out of signal changes caused by physical or chemical changes in time or space for those objects, which the signals represent.