OA18802A - Image based bilirubin determination. - Google Patents

Abstract

The invention relates to diagnosis in général and more specifically a system and a method for determining the presence of jaundice in newborn babies, also known as neonatal jaundice.A main objective of the present invention is to provide a simple system and method for determining the presence of jaundice. Particularly since most deaths due to jaundice occur in low-income countries, there is a large unmet need of simple, reliable and affordable technologies able to identify at-risk newborn.The objective is accomplished through receiving a depiction of skin from an RGB sensor, and then using either an optical diffusion model of the skin or Monte Carlo simulations to calculate the bilirubin concentration. A meta model of the optical diffusion model or Monte Carlo simulations can also be used. Colour calibration is also performed by e.g. thin-plate spline interpolation.

Description

Title: Image based bilirubin détermination

Background ofthe Invention

Technical Field

The invention relates to diagnosis in general and more specifically a system and a method for determining the presence of jaundice in newborn babies, also known as néonatal jaundice.

Background Art

From prior art one should refer to traditional methods for analysis such as analysis of blood samples and skin colour analysis. The main problem is cost and complexities that ruie these means out from much of the world and also restricts these to centralised sites.

One should also refer to more recent development in colour analysis such as provided by ClikJaundice. This analysis is performed using a cell phone caméra capturing a picture of a skin area as well as a colour calibration chart placed within the area captured by the caméra. The problem is that it relies on a colour calibration chart that must not be discoloured and also a central server that processes the pictures in view of how the colour calibration chart has been reproduced by the caméra. It is also important that the colour calibration chart is correctly positioned with respect to angle and illumination in order to provide information about cell phone caméra colour reproduction.

Also Bilicam uses a similar system.

Disclosure ofthe Invention

Problems to be Solved by the Invention

A main objective ofthe présent invention is to provide a simple system and method for determining the presence of jaundice. Most deaths due to jaundice occur in lowincome countries. There is therefore a large unmet need of simple, reliable and affordable technologies able to identify at-risk newborn.

Means for Solving the Problems

The objective is achieved according to the invention by a method for determining the presence of jaundice as defined in the preamble of claim 1, having the features of the characterising portion of claim 1.

Thus, in a first aspect the présent invention relates to a method for diagnosing a level of bilirubin comprised in blood causing jaundiced skin in a subject which comprises receiving a depiction using an RGB sensor of skin with a colour calibration chart, and calculating a level of bilirubin based on an optical diffusion model of a skin or Monte Carlo simulations of skin optics in the received depiction.

The calculation of the level of bilirubin can be performed using a meta model of the diffusion model of skin or Monte Carlo simulations of skin optics.

Furthermore, the objective of the présent invention is achieved by the other independent claims.

Thus, in a second aspect the présent invention relates to a method for determining a rate of change in level of bilirubin in blood causing jaundiced skin and sciera, comprising receiving a depiction using an RGB sensor of jaundiced skin and sciera with a colour calibration chart, calculating a level of bilirubin using the method of claim 1 for skin and sciera, and comparing the indicated level of bilirubin from the skin with the indicated level of bilirubin from the sciera, determining a rate of change in level of bilirubin in blood by the différence in the indicated level of bilirubin from the skin from the indicated level of bilirubin from the sciera

In a third aspect the présent invention relates to an apparatus for diagnosing a level of bilirubin comprised in blood causing jaundiced skin in a subject according to claims 1 or 2, which comprises means for receiving a depiction using an RGB sensor of skin with a colour calibration chart, and means for calculating a level of bilirubin based on an optical diffusion model of a skin or Monte Carlo simulations of skin optics in the received depiction.

In a fourth aspect the présent invention relates to a method for creating a customised treatment plan of a subject with jaundice by sunlight exposure by wherein the data obtained according to claims 1 or 2, in combination with sunlight exposure data of a location and/or weather data.

Yet another aspect of the présent invention relates to a use of a method according to claim 1 or 2 for creating a customised treatment plan of a subject with jaundice through sunlight exposure wherein the data obtained according to claim 1 or 2 are combined with sunlight exposure data of a location and/or weather data.

[n a fifth aspect the présent invention relates to a calibration card comprising a plurality of colour patches wherein the calibration card further comprises a plurality of grey patches for detecting the variations of illumination of the calibration card.

Thus, in a sixth aspect the présent invention relates to a calibration card according to claim 7, wherein the calibration card comprises further an opening where the skin or sciera is visible.

In a seventh aspect the présent invention relates to a calibration card according to claim 7 or 8, wherein the colour patches on the calibration card are printed using spectral printing.

In an eighth aspect the présent invention relates to a calibration card according to claim 7 to 9, wherein the grey patches are evenly distributed over the calibration card.

In a ninth aspect the présent invention relates to a calibration card according to claim 7 to 10, wherein at least one grey patch is arranged at a corner of the calibration card.

A number of non-exhaustive embodiments, variants or alternatives of the invention are defined by the dépendent claim(s).

The présent invention attains the above-described objective by depicting skin using RGB sensor, and calculating a level of bilirubin based on an optical diffusion model of skin, or Monte Carlo simulations of skin optics.

Preferably the sensor is calibrated using a calibration chart placed within view of the RGB sensor during the depiction ofthe skin. The calibration can be performed under varying light conditions and using a colour calibration chart that is known to be good.

Effects of the Invention

The technical différences over prior art as represented by ClickJaundice and Bîlicam is that the calculation of bilirubin level is based on an optical diffusion model of the skin, or Monte Carlo skin optics simulations. In addition, a meta model, a direct lookup table, or supervised machine learning techniques are used on this model or simulations. The colour calibration card is preferably printed using spectral printing making sure that the colours on the card changes similarly when subject to differing light sources or illumination conditions.

These effects in turn provide several further advantageous effects: it makes it possible to use simple and low cost mass market caméras as found in many cell phones, and it makes it possible to use the model to accommodate different types of skin, such as different skin colours. In addition, it makes it possible to use the model without a reliable internet connection.

Brief Description of the Drawings

The above and further features of the invention are set forth with particularity in the appended daims and together with advantages thereof will become clearerfrom considération of the following detailed description of an exemplary embodiment of the invention given with reference to the accompanying drawings.

The invention will be further described below in connection with exemplary embodiments which are schematically shown in the drawings, wherein:

Fig. 1 shows a cross section of human skin,

Fig. 2 shows extinction coefficients for haemogiobin, methaemoglobin and 10 bilirubin,

Fig. 3 shows the quantum efficiency of a CMOS sensor through a Bayer colour filter array,

Fig. 4 shows positions of the red, green and blue colour filters in a Bayer filter array,

Fig. 5 shows standard déviation of the caméras plotted with skin simulation results,

Fig. 6 shows predicted error of the calibration plotted with skin simulation results,

Fig. 7 shows light intensity spectra of fiashlights of three different phones,

Fig. 8 shows colour measurements of skin plotted with skin simulation results, Fig. 9 shows a first embodiment of a colour calibration card, and Fig, 10 shows a second embodiment of a colour calibration card.

Description of the Reference Signs

The following reference numbers and signs refer to the drawings:

100	Skin model
110	Epidermis
112	Mélanocytes
120	Dermîs
122	Blood vessels
124	Sweat gland
130	Haïr follicle
132	Follicle
134	Oil gland / sebaceous gland
136	Sébum

138	Hair
140	Fatty tissue
221	Extinction coefficients for oxygenated haemogiobin
222	Extinction coefficients for unoxygenated haemogiobin
223	Extinction coefficients for methaemoglobin
224	Extinction coefficients for bilirubin
231	Transmission of red wavelength through a Bayer colour filter array
232	Transmission of green wavelength through a Bayer colour filter array
233	Transmission of blue wavelength through a Bayer colour filter array
234	Transmission of caméra infrared filter
235	Quantum efficiency of CMOS sensor
241	Red filter in a Bayer colour filter array
242	Green filter in a Bayer colour filter array
243	Blue filter in a Bayer colour filter array
441	Simulated skin colour
442	HTC: White balance auto
443	HTC: White balance daylight
444	S3: White balance auto
445	S3: white balance daylight
461	HTC One V
462	iPhone 5
463	Samsung Galaxy S3
471	Skin simulations for increasing levels of bilirubin
472 - 475	Colour différence measured of bruised and non-bruised skin
500	Colour calibration card
502	Opening
504	Grey field
506	Colour patch

Detailed Description

Various aspects of the disclosure are described more fully hereinafter with reference to the accompanying drawîngs. This disclosure may, however, be embodied in many different forms and should not be construed as limited to any spécifie structure or function presented throughoutthis disclosure. Rather, these aspects are provided so that this disclosure wiil be thorough and complété, and will fuliy convey the scope of the disclosure to those skilled in the art. Based on the teachings herein one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspectofthe disclosure disclosed herein, whetherimplemented independently of or combined with any other aspect of the disclosure. For exampîe, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope ofthe disclosure is intended to cover such an apparatus or method which is practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more éléments of a claim.

Principles formîng the basis ofthe invention

Jaundice is a condition characterized by the skin of the afflicted turning yellow. This is due to elevated levels of the waste product bilirubin in the blood leaking into skin tissue. The condition is therefore often called hyperbilirubinemia. It is a condition affecting approximately half of ail newborn, but is in most cases harmless. The condition is still potentially dangerous because the bilirubin can accumulate in the basal ganglia of the brain, where it can cause permanent brain damage. Such brain damage, better known as kemicterus, can manifest itself as cérébral palsy, deafness, language difficulty, or in the worst cases death [1].

To prevent the usually harmless condition of jaundice from developing into kernicterus, it is highly important to identify the children at risk at an early stage. Treatment of hyperbilirubinemia is in most cases done by phototherapy, and in some extreme cases by blood transfusion. Sunlight is believed to be a cheap alternative to specialized phototherapy light-boxes, and studies are now underway investigating this [3]. It is therefore essential to be able to discover at-risk children at an early stage, so that effective treatment can be given.

The growing use of smartphones also make it possible to create customised treatment plans for néonatal jaundice using sunlight based on geolocation services on the smartphones. Using the GPS and e.g, weather data from the smartphone, one can give recommendations on when, for how long, and how often patients need to be in the sun to get effective treatment. This can be calculated by looking at when the sunlight has maximum amount of light in the wavelengths (~450nm wavelength) that provide treatment for jaundice, while at the same time has the minimum amount of damaging UV-radiation. This can be done by combining information from publicly available UV-index forecasts and spectral sunlight irradiance calculators for solar panels [42], This can be combined with a low-cost diagnostic tool to provide customised low-cost treatment as well.

Bilirubin colours the skin yellow. Jaundice can therefore often be seen visually even by people with no medical training. But mere visual judgment ofthe severity of jaundice has proven to be unreliable, even when perfomned by experienced doctors

[4], The measurement of bilirubin is therefore traditionally done by blood samples. To reduce the need of drawing blood from the newborn, devices hâve been developed that measure the bilirubin concentration by shining light through the skin, so-called transcutaneous bilirubinometers [5j. Both the lab equipment needed for blood sample measurements and the devices used to measure bilirubin transcutaneously are expensive, costing more than 10,000 US dollars, thus making them practically unavailable in low-income countries.

Skin 100 is the human organism's barder to the environment. It is a structure composed of different layers. The top layer, called the epidermis 110, is typically 100 micromètres thick and contains among other things the pigment melanin, which is the pigment responsible for the different skin colours of the world. Below the epidermis lies the dermis 120, which has a typical thickness of 1-4mm. In the demis, blood vessels 122, connective tissue, sweat glands 124, hair follicles 130 and sensory nerve Systems are found. The hair follicle 130 comprises a follicle 132 that surrounds hair 138 and further comprises an oil gland 134 also known as sebaceous gland that produces sébum 136. The subcutaneous fatty layer 140 lies below the dermis, and provides insulation and protection from mechanical stress. Figure 1 illustrâtes the different skin layers along with other components found in the skin.

Light hitting biological tissue, such as skin, is either scattered or absorbed. The intensity of light able to penetrate into the tissue is given by Beer-Lamberts law, l(x) = Ι(0)Θ^-μ,ΓΧ (2.1) where l(0) is the incident light intensity, x is the distance travelled in the tissue and p_tr is the transport coefficient, or the total atténuation coefficient. The transport coefficient can be written as the sum of the reduced scattering coefficient, μθ s, and the absorption coefficient, μ₃.

The scattering coefficient, μ₃ describes the amount of light that is scattered by the tissue. Some of this light is scattered in a forward direction, not decreasing the penetrating lights intensity. The reduced scattering coefficient incorporâtes this by being expressed as p’_s = p_s (1 - g), where g représente the amount of light scattered in a forward direction, g is called the anisotropy factor, and is calculated as the average ofthe cosine ofthe scattering angle distribution, g = avg(cos(9)). (2.2) in skin, the anisotropy factor is approximately equal to 0.8, indicating highly forward directed scattering.

Skin contains several different molécules responsable for the absorption and scattering of incident light. The properties of these molécules and the surrounding tissue are presented in the following sections.

Skin, as just mentioned, contains many absorbing and scattering molécules. The main absorber in the epidermis is melanin [7], Skin types based on varying amounts of the pigment melanin can be classified by the Fitzpatrick skin type scale IVI [8], On this scale, type I refers to very fair skin that sunburns and does not tan, while type VI is at the opposite end of the scale, referring to very dark skin.

Melanin absorbs light of wavelengths ranging from ultraviolet to near-infrared. The wavelength dependence of the absorption is reported as λ^-3·⁴⁶ [9]. The absorption of melanin across the whole spectrum can therefore be defined by the absorption at a single wavelength. Absorption can be measured at 694nm, and absorption values in adults hâve been found to vary from 300m”¹ for fair Caucasian skin to 2500m”¹ for dark African skin [10]. Newborn skin is reported to hâve lower concentrations of melanin than adult skin [11]. Here it is therefore assumed that the melanin absorption of newborn skin at 694 nm does not exceed 2000m”¹, although exact numbers hâve not been found in the literature.

The main absorbers in blood are oxygenated and deoxygenated haemoglobin. Methaemoglobin can also be formed if haemoglobin is exposed to oxîdative stress, but is generally found in low concentrations. Exceptions are e.g. drug use which can lead to methaemoglobinemia [12]. The absorption spectra of haemoglobin 221, deoxyhaemoglobin 222 and methaemoglobin 223 can be seen in Figure 2. The spectra of haemoglobin and deoxyhaemoglobin can be seen intersecting at several points. Such points are called isosbestic points. Measuring haemoglobin concentrations is often done at isosbestic wavelengths because the total measured concentration will not dépend on the oxygénation level of the blood [13],

Bilirubin is the breakdown product of heme catabolism [14], Heme is found in haemoglobin and myoglobin. Bilirubin 224 causes skin to turn yellow if it is allowed to accumulate in the dermis, due to its high absorption of the shorter wavelengths of the visible spectrum (see Figure 2). This is also the reason for the yellow colour seen in old bruises [15], as macrophages are recruited to the area of the bruise where it phagocytizes érythrocytes and haemoglobin molécules, catabolîzing the haemoglobin to bilirubin [16].

The yellow colour from bilirubin can also be seen across the whole body, and is then caused by either a high turnover rate of haemoglobin, or liver failure, or both. Newborns acquire jaundice due to a high turnover rate of haemoglobin after birth. An elevated concentration of bilirubin in combination with a not fully developed blood brain barrïercan lead to permanent brain damage ordeath [17]. For this reason, 510% of ail newborn receive either phototherapy, or in extreme cases blood transfusion to hd the body ofthe excess bilirubin [14].

Bilirubin in blood is bound to albumin. In this form, the combined molécules are too big to pass the blood vessels. When bilirubin concentrations exceed 400-500 micromolar, there is not enough albumin to bind ail the bilirubin molécules [17], The free bilirubin can then diffuse through the blood vessels and into the surrounding tissue. The skin concentration of bilirubin is therefore markedly lower than the blood sérum concentration. Good corrélation has been found between the skin concentration of bilirubin measured by transcutaneous bilirubinometers and the total blood sérum concentration [18]. This makes it possible to estimate the blood sérum concentration through transcutaneous bilirubin measurements.

Transcutaneous bilirubinometers measure the bilirubin concentration by shining light of certain wavelengths and wavelength ranges into the skin. The reflected light of each wavelength is measured and used to calculate the concentration. Full reflection spectroscopy of newborn can similarly be used to measure bilirubin concentration. In addition, the reflected spectrum allows the calculation of several other parameters such as melanin concentration and the gestational âge of the newborn. For details of how such measurements are performed, one should refer to a patent for a transcutaneous bilirubinometers [19] and a paper by Randeberg et al. [20].

As for other absorbers one should refer to carotenoîds which are organic pigments found in plants. These pigments cannot be produced by animais, so they are obtained through diets. They ail absorb light in the wavelength range 400550nm. A common carotenoid abundant in carrots, betacarotene, has a double peak in its absorption spectrum at 450 and 480nm, giving it a yellow/orange colour. This colour, which is similarto the colour of bilirubin could potentially be an error source in bilirubin measurements. But the skin concentration of ail carotenoîds is generaily too low to hâve an impact, especially in newborns that don't eat carotenoîds themselves [7, p. 10]. They only get carotenoîds through milk.

Water should also be mentioned because it is found in abundance in skin. But water has low absorption in the visible spectrum with a minimum at 418 nm and increasing absorption for wavelengths above 600 nm [21], The water content of skin is therefore not explicitly accounted for.

Large molécules such as collagen fibres are a major source of scattering in the dermis. These molécules and changes in refraction index between them and the surrounding tissues are responsible for the fact that scattering is the dominating process in this tissue. The epidermis has similar scattering properties, but absorption due to melanin can in some cases be the dominating process in this layer.

Bashkatov [6] showed that the reduced scattering coefficient of skin in the wavelength range of 400 to 2000 nm can be expressed as p'_s = 73.7λ^-0·²² + 1.1 10¹²λ“⁴. (2.3)

Background tissue absorption is absorption caused by other molécules than the ones mentioned in the above sections. This value is set to μ_η = 25m ¹ [22] for both the epidermis and the dermis. This value is similar to what is found in ocular (eye) tissue.

One mathematical model used for numerical simulations in this invention is based on optical diffusion theory. Optical diffusion theory can be applied when scattering dominâtes over absorption [23]. This theory has limited validity in thin layers, and finding appropriate boundary conditions is problematic. Optical diffusion theory does not apply to air, but Haskell et al. [24] discovered boundary conditions that can be used for interfaces such as those between air and tissue, giving good results of simulations of diffuse skin réflectance [22]. Monte Carlo methods are also known to be accurate for simulations of skin optics, but they are also much more computationally expensive [25]. Given enough computational time, Monte Carlo simulations couid be performed to be used to model the skin optics necessary to estimate bilirubin levels from skin colour.

Forthe simulations performed in this implémentation ofthe invention, the skin is modelled as consisting of three fiat layers. The top layer represents the epidermis. To account for the papillary structure between the dermis and epidermis, as seen in Figure 1, blood is included in the epidermis. The epidermis therefore contains both blood and melanin in the model. The middle layer represents the top partofthe dermis, and the bottom layer is a layer extending infinitely downwards. Ail molécules are modelled as uniformly distributed within each layer. The total transport coefficients of each layer can thus be calculated based on the background tissue scattering and absorption described earlier, and the concentrations ofthe different light absorbing molécules. These transport coefficients can then be used in the diffusion model of skin developed by Svaasand et al. [22], A summary of which will be presented below.

Svaasand et al. [22] starts by assuming an almost isotropie light distribution and by expressing the radiance L by a sériés expansion,

[_ = φ/4π + 3/4π] 1 + ...(2.4) where φ and j are the fluence rate and the diffuse photon flux vector respectively. I is the direction of the déviation from isotropy in the light distribution. The irradiance on a surface normal to the flux then becomes

Ε = φ/4±]/2. (2.5) where the sign is plus for surfaces against the flux and minus for surfaces along.

The diffuse photon flux vector is given by, j = -D V φ (2.6) with the diffusion constant,

D = 1 / 3p_tr. (2.7)

The continuity équation can then be expressed as,

V · j = -μ₃φ + q (2.8) where q is the source density of diffuse photons. The combination of équations 2.6 and 2.8 yields,

- φ/δ²= -q/D (2.9) where S=sqrt(1/(3p_trpa) is the optical pénétration depth.

The boundary conditions between two scattering media is then expressed by the continuity of irradiance in the forward and backward directions, . .71 02 . .72 01 . jl _ 02 . J2 4 2 ~ 4 2 (2.10)

Haskell et al. found that a very useful boundary condition at the skin-air interface is obtained by relating the reflected part of the irradiation at the inside of the interface to the irradiation propagating back into the skin[24]

(2.11) where R_eff is the effective reflection coefficient. The value of R_eff can be found by integrating the Fresnel reflection coefficient for unpolarised light over ail angles of incidence.

For an isotropie light distribution, the source density functions of Equation 2.9 are expressed as functions of the light intensity, P_o, transmitted through the skin-air interface as ¢1 = ^ρομ'_β,ιβ ?2 = Po P_8i2^e ?3 = ft)X,,3^e “Mtr.l

-Ptr.l

Mtr,2 (® ^— 41 )

C “Mtr.l — Mtr,2 42 „ — Mtr,3 (®~41 — d₂) C c (2.12) where the indices 1,2 and 3 represents each layer, d represents the thickness of a layer, and x the distance from the skin surface.

The solutions to équation 2.9 using these source équations can then be written i __j χ

Φ1 = 777:----2' X2>^{e δ}1 +7126^{01 D}ld-Kr,i^SV <t>₂ = n + Â₃e^-^ + À₄e^

E>2(1 - rtr,₂Î2)

_e — ptr,ldi_e — p_trt2d2_e — ptr,3(.X — dl—d2) _|_ _β S₃ (2.13)

The values of the constants Ai - A₅ can then be found by applying the boundary conditions of équation 2.10 and 2.11. After this, the diffuse reflection coefficient can be calculated by j|x = 0 ⁼ ---Po (2.14)

For the complété expression for 7, the reader is referred to the appendix of Svaasand et al.[22].

These calculations and simulations of skin optics can be used to calculate the total reflected spectrum from skin with different amounts of blood, melanin, bilirubin, and ail other aforementioned parameters. These simulated reflection spectra can then be used to calculate the skin colours these spectra represent by employing the standards defined by the International Commission on Illumination [26].

Estimating skin parameters from measured colour

Searching through the space of input skin parameters by simulating skin colour for each set of input parameters to find a colour that matches with the measured colour is extremely time consuming using the simulation methods described above. An alternative method of estimating the skin parameters from a measured colour that is both fast and requires a reasonably small amount of disk space is therefore needed.

Estimating the input skin parameters from a measured colour can be performed efficiently through inverse meta modelling ofthe skin simulations or through supervised machine learning techniques such as régression methods and neural networks. The inverse meta model is a surrogate model ofthe skin simulations that map the output of the skin simulations to its input skin parameters. This meta model can be created by employing several different régression methods, or in some cases through a direct lookup table, or a combination of the two. Partial least squares-based régression models (PLSR) are recommended for inverse meta modelling of this kind [40]. For a detailed description ofthe PLSR meta modelling technique, the reader is referred to appendix A of [40]. An advantage with using supervised machine learning methods, e.g. neural networks or standard régression methods, a system can be created that is able to use colours from an deviceindependent colour space, such as the calibrated colours from pre-recorded images, and map these colours to their most likely corresponding bilirubin values. This alternative to creating a meta-model has been described earlier in the application.

Several input skin parameters can resuit in approximately the same output colours from the simulations (called sloppiness). This makes it difficult to use regression-based methods such as PLSR. This sloppiness can be handled by employing the method of hierarchical cluster-based partial least squares régression (HC-PLSR). HC-PSLR Works by first creating a global PLSR model [42], This model is then separated into clusters either by prior knowledge of the data or by algorithme such as fuzzy clustering [41], A local PLSR model is then created for each cluster. The input skin parameters can then be estimated from a measured colour by using the global PLSR model to find the cluster the colour most likely belongs to, and then using the PLSR model created for that model to predict the input skin parameters. If the number of clusters in the HC-PLSR model equals the number of skin simulations performed to train the model, the method becomes a direct lookup method.

As the direct lookup method requires both the largest amount of disk space, and usually the longest computational time to perform, it should only be used when it is known that the hardware that runs the algorithm is capable of handling both the disk storage and the computational demand. If disk storage space and computational demand becomes an issue, the HC-PLSR method should be employed. For lowerend hardware, a small number of clusters in the HC-PLSR method should be used, and vice versa.

Another way of reducing the sloppiness of the simulations is to search for the colours that hâve the highest probability of occurring around a measured colour. As many ofthe input parameters to the numerical skin model are normally distributed, one can use the mean and the standard déviation of these parameters found in the literature to find which input parameters were the most likely to produce the measured colour. The colour calibration errer could be used as a measure of how far away from the measured colour one should search for probable input parameters.

Colour Calibration

The eye has three types of cône cells which are sensitive to light of varying wavelengths. These cône cells provide the sensory input needed for colour perception. One type of cône cells primarily absorbs light of shorter, blue, wavelengths, and the other two absorb mainly green and mainly red, respectively. Nevertheless there is significant overlap between the sensitivity spectra of the cells. Three parameters corresponding to the stimulus values provided by the cône cells can therefore be used to describe any perceivable colour.

Almost ail smartphone caméras today include a CMOS (Complimentary Metal-Oxide Semiconductor) sensor. These sensors hâve arrays of photodiodes that generate current when photons are absorbed. The efficiency with which the photodiodes generate current dépends on the wavelength and is called the quantum efficiency. A typical quantum efficiency of a CMOS sensor can be seen as the line 235 in Figure 3.

On top of the CMOS sensor is a colour filter array. A single cell in a colour filter array covers a single photodiode and transmits only wavelengths of a certain colour to this diode. A commonly used filter of this type is the Bayer filter array. The Bayer filter array contains 50% green, 25% red and 25% blue filters. This is to resemble the human eye’s increased sensitivity to the intensity of green light. The filters are placed in a spécifie pattern as can be seen in Figure 4, and their transmission frequencies can be seen in Figure 3.

The figure shows transmissions of red 231, green 232 and blue 233 wavelengths through a Bayer colour filter array. Caméras are typically provided with an infrared filter having a transmission 234 shown in the same figure. Also shown is the quantum efficiency 235 of a CMOS sensor.

This type of filter arrangement créâtes in effect three different images, one red, one green and one blue. Ail three of these images are collectively called a raw format image. High end digital caméras hâve the option to output images in this format. Cheaper caméras, including smartphone cameras, do not hâve this option. In these cameras the three original images are combined using a demosaicing algorithm, interpolating the missing red, green or blue pixel values from the surrounding pixels. This process can be done by both hardware and software, and the spécifie algorithms used differfor the different cameras. The resuit of this process in smartphones is a JPEG image with RGB values for each pixel in the sRGB colour space.

White balance adjustments of the image are also performed in addition to the demosaicing algorithm. These colour adjustments are performed to attempt to recreate the colours of the scene more accurately. The process is called white balance because photographers often use images of known white or grey objects as référencés when performing these adjustments. The images need such colour adjustments because the light source illuminating the scene will create different colour responses in the caméra depending on the light source used. Caméras therefore often corne preset with white balance settings such as daylight, cloudy, incandescent and fluorescent to accommodate for common light sources. The caméras also hâve an auto white balance setting which adjusts the colours of the images automatically depending on the distribution of colours in the image.

The white balance mode can be set using the smartphone caméra app on almost ali smartphones. Other settings, such as the shutter speed, which is the amount of time the caméra allows light to reach the sensor, can only be set on a few smartphones. This lack of control over caméra settings could pose a challenge because small changes in e.g. the light intensity of a scene could potentially alter the image substantially.

White balance adjustments can create images that look good, but that does not mean that colours are reproduced accurately. To achieve good colour reproduction, images of objects with known colours can be captured. The colours of the captured images can then be adjusted according to the known colours of this colour target. A common target used in photography settings is the Macbeth ColorChecker [27]. The ColorChecker contains 24 squares of different colours. The upper half contains colours often found in nature, while the bottom half contains a grey scale and colours close to the primary colours ofthe RGB and CMY colour spaces.

Two different calibration methods hâve tested both of which are presented in the following.

Ilie et al. [28] tested three different methods for the colour calibration of images. The first was a linear least squares matching, the second was a linear RGB to RGB matrix transformation, and the third was a general polynomial transform. Of these, the general polynomial transform was found to be the most accurate because it was the only method that could account for both linear and non-linear error sources. It was therefore chosen to be implemented here.

The équation for the general polynomial transform for colour channel c e {r, g, b} of sample colour s is

D + tgc_k^9g + t-bc_kIb^k _s) + t_c„ — Tc_s k=i (2.22) where D is the degree of the polynomial approximation. ^an<^ Ib_s are the red, green and blue sample colour values ofthe captured image raised to the power of k, while Tc_s j_s the true value of the target colour sample s. js the polynomial coefficient of order k, specifying the influence of the input colour channel ^x G {r, g, 6} _on the output colour channel c. For D = 2 with 24 colour samples, such as when using the ColorChecker, équation 2.22 can be written in matrix form as /rj Jfj

I”2 Irl /r₂₄ lr², hi >91

I924 ^921

Ib_t Ib²,

Ib₂ Ibj

Ib_M Ib²,

	TC j
il	trcz
1	f lgci
x	Lgci
1]	t-bcy
	tbc2
	. tC0 .

— Tc_s (2.23)

This équation can be solved for the polynomial coefficients, t_Ck, by calculating the pseudo-inverse of the matrix, B, containing the input sample colour values. The équation to be solved is therefore

B x t_Ck Tc_s t_Ck ~ Pinu(B) x Tc_s (2.24) resulting in a vector t_Ck that can be used to convert any input colour from the input colour space to the calibrated colour space. The équations outlined here assumed usage of an RGB colour space, but the method can be used for any threedimensional vector space, including the XYZ colour space.

Menesatti et al.[29] compared a commonly used commercial colour profiling tool called ProfileMaker to a novel calibration procedure using thin-plate spline interpolation. The thin-plate spline method was found to give significantly better calibration results.

The thin-plate spline interpolation method îs named after the physical analogy of bending thin métal plates to fit to certain fixed coordinates [30]. It is used in the field of medical imaging, as a means of transforming and analysing images from e.g. magnetic résonance imaging (MRI) scans[31]. In three dimensions the method works by finding a function f(^xi > ^2· ^3) that minimizes ¹ 52(1/. /(^1(1),^2(1)^3(1)))² -h AJ(/) ⁿ 1 = 1 (2.25) where n is the number of known reference sample points, y, is a value at such a sample point, x1(i), x2(i) and x3(i) are the input coordînate values of sample i, and λ is a smoothing parameter determining the effect the penalty function J(f) will hâve on the final interpolation. J(f) represents the bending energy of the thin plates. In three dimensions this function is given by [38, p. 89].

^)²+(Ʋ ₊(/^m<w

OXlX2 0X1X3 0X^X3

Duchon[32] showed that the interpolation function f minimizing 2.25 is of the form f(x) = ai + 0.2X1 + «3-T2 + «4-T3 + biU(- x(i)|) ⁱ⁼¹ (2.27) where U(r) = r log(r )

By defining a matrix K with éléments Kij = U (|x(i) - x(j)|) and a matrix M = K + ηλΙ, where I is the identity matrix, Wahba[33] shows that the équations above can be written as

M b + T a = y

T'b = 0 (2.28) where a and b are vectors containing the coefficients in équation 2.27, y is a vector with values of the n reference samples, and T is a matrix of the form

	'1	riU)	τ2(1)	•Ml)'
T =	1	n(2)	æ2(2)	M2)
	1	Ti(n)	τ2(η)		(2.29)

Equation 2.28 can thus be expressed in matrix form as

M

_o(⁴, l)j (2.30) where O(r, c) is a zéro matrix. The values for the coefficients in vectors a and b are

Μ T

T 0(4.4) then calculated by inversing the matrix. To include the calibration of ail three dimensions in the above équation, vectors a, b and y can be extended to matrices with three columns of the form . [^ai ^û2 «3] For a more rigorous mathematical development of the above relations, the reader is referred to chapter

2.4 of the book by Wahba[33].

Other calibration methods can also be used to calibrate the measured skin colour to within an acceptable level, such as Kriging methods, or Partal LeastSquares Régression methods.

Test methodology

A Samsung Galaxy S3 (GT-19300, Samsung, South-Korea) and an HTC One V(One V, HTC, Taiwan) were used for this project. Compared to other smartphones, the S3 has a high quality caméra [34], while the One V has a decent caméra [35].

Several methods were tested using these two phones. They were tested on skin of a light colour, type II on the Fitzpatrick skin type scale. For control purposes, numerical skin simulations and measurements of this skin with bruises were performed. The methods tested and the Controls used are described below.

There exist a couple of methods to either simulate increased bilirubin concentration in the skin or to obtain an actual higher skin bilirubin concentration. One of these methods is to perform measurements on bruised skin. The yellow colour seen in or around bruises is, as mentioned earlier, caused by increased bilirubin concentration. Measurements on both yellow bruised skin and skin in close proximity to the bruise, but with no distinct yellow colour, hâve been performed. This gives a qualitative resuit of whether the measurement method can separate skin with low bilirubin concentration from skin with higher concentrations.

The use of bruises allowed for a coarse test of whether the different measurement methods would work or not. But quantitative results were also needed to test the accuracy of the promising methods. Therefore, numerical simulations of skin were performed. This allowed for the création of both reflection spectra and simulated colours of skin with varying concentrations of e.g. bilirubin and melanin. Simulations were performed using the three-layer diffusion model described in earlier. Wavelengths were sampled at 5nm intervals. The resulting simulated reflection spectra were then combined with a light source spectrum similar to one used during measurements, i.e. CIE standard illuminant D50 for daylight [44, p. 93].

The simulations were performed with a blood oxygénation level of the top and second layers of 0.5 and 0.8 respectively. The blood volume fractions of the top and second layer were set to 1 %. The thickness of the top layer was set to 100 microns, while the thickness ofthe second layerwas set to 250 microns. The water, fat, betacarotene and methaemoglobin levels were set to zéro. The scattering coefficient was calculated with values as described in équation 2.3. Melanin concentrations were varied from an absorption at 694 nm of 250 m^-1 to 2000 m^-1. Bilirubin concentrations were varied from 0 micromolar to 200 micromolar.

An alternative to taking pictures and blood samples of newborn and finding colour corrélations afterwards, is to attempt to predict the colourofthe newborns skin with varying levels of bilirubin. These prédictions can then be compared to a measurements performed using a caméra. This requires good numerical simulations to predict the skin colours, as well as caméras that are calibrated to reproduce the true colours of the captured scene. High-end digital caméras are today calibrated using images of e.g. a Macbeth ColorChecker in RAW format. Free tools are available that can perform such a calibration, but no smartphones on the market today support RAW file output. For this reason, a different calibration procedure was needed.

To calibrate the smartphone caméras, pictures of a Macbeth ColorChecker (MacBeth ColorChecker, Munsell Color, Baltimore, USA) were taken. Most of these images were captured using diffuse daylight through a window as light source. For these images, the colours given in a paper by Pascale [37] were used as the reference ColorChecker colours. Images were also captured using only the built-in flashlîght of the smartphones as the light source. For those images, the reference ColorChecker colours were calculated from the réfaction spectra ofthe ColorChecker and the light spectrum of the flashlights. The reflection spectra of the ColorChecker were gathered from the Munsell Color Science Laboratory website [37],

The pixel coordinates ofthe ColorChecker squares in the images were found manually. The colour of the squares were then calculated as the average of ail the pixels in a square box centered on the squares pixel coordinate. The side lengths of these square boxes were set to 21 pixels, making the colour of one of the ColorChecker squares the average of 441 pixels within that square. These colours were then converted to the xyY colour space, as they could then be compared with the reference ColorChecker values.

Before calibration was attempted, the standard déviation of the colour reproduction ofthe caméras was measured. This was done by taking 10 pictures of the ColorChecker from slightly different angles using diffuse daylight through a window as light source. Sériés often pictures were taken using both daylight and auto white balance mode for both caméras. The colours of the squares were then found and converted to xyY. An estimate of the standard déviation of the xy chromaticity was then found by averaging the standard déviation of the x and y values of the 24 individual ColorChecker squares. The final standard déviation was then calculated as the vector sum of the x and y standard déviations, Δ xy = sqrt ( Δ x² + Δ y²’.

Two calibration procedures were developed. The first is a generalized polynomial transform, which was reported as précisé by researchers in the field of computer vision [28]. The other method is an implémentation of the Thin-Plate Spline interpolation algorithm, which has been reported as a highly efficient calibration technique [29]. The theory behind the techniques is described earlier. Both methods were developed to be used for three colour channels, i.e. the RGB and XYZ colour spaces. The Thin-Plate Spline interpolation algorithm was later modified to also work using only two colour channels, so that it could be used for calibration of xy chromaticity. The general polynomial transform was not modified in the similar way because the Thin-Plate Spline method had proven to be superior.

The two methods efficiencies were tested using ordinary cross-validation. Ordinary cross-validation works by leaving one test sample out when creating a prédiction model such as a general polynomial transform. The prédiction model is then used to predict the value of the test sample that was left out during the models création. The error of this prédiction therefore becomes an estimate of the accuracy of the prédiction model. This procedure is repeated leaving out a different test sample each répétition until ail test samples hâve been left out once. The prédiction errors of these répétitions are then averaged to give the estimate of the final models prédiction error. In the case where the test samples are the 24 colours of the ColorChecker, 24 prédiction models were created leaving one different colour out for each model. These 24 models were then used to predict the value of the colour that was left out during the création of the model. The averages of the errors of these prédictions were then used as an estimate of the error of the final prédiction model created using ail 24 colours.

Ordinary cross-validation was also used to optimize the calibration methods. For the general polynomial transform, the polynomial order could be changed to test for example whether a higher order would yield a more précisé calibration. Finding the optimal order was quick and easy because increasing the order above three more often than not decreased the quality of the calibration. Thus, finding the optimum polynomial order involved running ordinary cross-validation for order one, two and three and finding the polynomial order with the smallest error. The Thin18802

Plate Spline method on the other hand, needed fine-tuning of the smoothing parameter λ which can be any number above or equal to zéro (see équation 2.25). An itérative procedure was developed that first tested a range of values of λ using ordinary cross validation. After this, a new range of values were tested centered on the value λ that showed the least error in the previous itération. Using this method, a highly accurate estimation of the optimal value of λ was found after only 6 or 7 such itérations.

The calibration methods were tested on images taken using diffuse daylight through a window as light source. Pictures of the ColorChecker were taken using ail the different white balance settings on the caméras to find the best option. At the same time, pictures were taken of a bruised arm so that the calibration method could be compared to a real case of increased bilirubin as well as simulations. The colours of the pictures of the arm were then corrected using the calibrations calculated from the images of the ColorChecker. Then the pixel coordinates of two points on the arm were found manually. First, a point where there was a clear yellow colour from the bruise, and second, a point on the arm with no bruise or other clearly visible pigments such as moles. The colour of these two points were then calculated as the average colour of the colours inside a square with side lengths of 21 pixels centered around the points. These colours were then converted to xy chromaticities in order to compare them to numerical simulations and the calibration quality.

Calibration was, as mentioned earlier, also performed on pictures of the ColorChecker captured using the built-in flashlight on the smartphones as the only light source. This was done because the reference colours of the ColorChecker given in the paper by Pascale [36] were calculated using D50 as light source. D50 is a good daylight simulator, but it is highly likely that there is some déviation from the true colours of the ColorChecker. To calculate the true colours of the ColorChecker using the flashlights, their light spectra were measured using the SD2000 spectrometer with the SpectraSuite computer application. A 10 millisecond intégration time was used along with smoothing by using the average of ten spectra as the resulting spectrum. In addition, the colour spectrum of the flashlight on an Iphone 5 was also measured to see if there is a large variance in different smartphone flashlights. These spectra could then be combined with the reflection spectra of the ColorChecker [37] to get the true colours of the ColorChecker.

Results

The standard déviation of the colour reproduction of the HTC was estimated to be 0.0036 for the auto white balance setting in xy colour coordinates. For the daylight white balance setting, it was estimated to 0.0034. Both white balance settings resulted in a standard déviation of 0.0016 using the Samsung. These standard déviations are illustrated in Figure 5, where the standard déviation is seen as the radius of the circles 442-445 in the figure. The lines 441 in the figure represent sériés of simulations with increasing concentrations of bilirubin, from 0 micromolar at the bottom left of the line to 200 micromolar at the top right. Each line to the right of another is another sériés of simulations of increasing bilirubin, but for a higher concentration of melanin. The absorption of melanin at 694nm is 250 m^-1 for the leftmost line, and 2000 m^-1 for the rightmost line.

The size of the circles 442-445 in Figure 5 can be compared to the length of the lines 441 as an indicator of whether the caméras colour variance is low enough to be able to separate skin with high concentrations of bilirubin from skin with lower concentrations. This appears to be true for both caméras except for the HTC when compared to simulations with very high melanin concentrations.

Before colour calibration was performed on the images, the average error of the measured colours of the ColorChecker was usually between 50 and 60, measured as the vectorial colour distance in RGB colour space. Using the general polynomial transform to colour calibrate the images, this error was reduced to approximately 30. The Thin-Plate Spline interpolation on the other hand, managed to decrease the error to approximately 25. Using the Thin-Plate Spline interpolation on the images in xy coordinates yields prédiction errors of 0.0126 and 0.0275 for the HTC with white balance set to auto and daylight respectively. For the Samsung the prédiction errors were 0.0157 and 0.0163 with the same white balance settings. These values are plotted in Figure 6 similarly to the standard déviations pîotted in Fig. 5. The reason for the large value of the calibration error of the HTC with white balance set to daylight is not known. Prédiction errors were also calculated for the other white balance settings, but auto and daylight were the ones with the best results for the images captured with diffuse daylight through a window as light source.

The diameter of the circles 452-455 in Figure 6 are of approximately the same length as the length of the lines 451. This indicates that this calibration procedure is not précisé enough for bilirubin concentrations to be estimated from comparing a colour measurement to numerical simulations. A calibration improvement was seen when the images taken using the flashlight of the phones as light source were used. When the reference colours from the paper by Pascale [36] were used to calibrate the images taken using the flashlights, the results were prédiction errors of 0.015 and 0.023 for the HTC and Samsung respectively. Using the colours of the

ColorChecker calculated by combining the flashlight spectra and the reflection spectra of the ColorChecker, calibration errors were reduced to 0.012 and 0.018,an improvement of approximately 20%. These errors hâve been further decreased using Kriging methods and by correcting for light intensity variations as described in the following paragraphs concerning the colour calibration card.

Gaussian process régression is a régression method used to generate both good prédictions of values at unknown inputs, but also to get an estimate of the error at that point. Other régression methods can create uncertainty estimâtes through e.g. cross validation, which créâtes an error measurement based on the predicted values at certain known points. Gaussian process régression is able to create error estimâtes for ail points in the input space, including the exact point we are measuring.

Gaussian process régression has also provided the best colour calibration of the methods that has been tested. This, in addition to providing an error estimate at the point that is measured, makes it the preferred calibration method. For a general introduction of Gaussian process régression the reader is referred to Ebden[43], For a detailed guide to Gaussian process régression with uncertain inputs, as used hère, the reader is referred to chapter 3 of Girard[44].

The measured flashlight spectra can be seen in Figure 7. Ail spectra are very similar and share the same characteristic shape. This makes it likeîy that most smartphones use the same type of diodes in their flashlights, sharing a similar spectrum.

Colour measurements of skin with and without a bruise were also performed using the same calibrations as were used for the prédiction errors shown in Fig. 6. These measurements can be seen in Figure 8. Each line with two connected circles 472-475 represents two colour measurements. The bottom left circles of 472-475 indicate a colour measurement done on skin with no bruise. The upper right circles of 472-475 indicate measurements done on a bruise. The lines 471 represent simulations as before. Ail the lines 472-475 going from a measurement of the skin to a measurement of the skin with a bruise point up to the right similarly to the numerical simulations of increasing bilirubin concentrations. This indicates that the calibrated colour measurements are indeed capable of measuring increased bilirubin concentrations.

The purpose of the colour calibration is to make the caméras able to measure the true colours of the scene it is capturing. These captured colours can then be compared to numerical simulations of skin colour to give an estimate of the bilirubin concentration of the skin. Figure 5 shows that the standard déviation of the caméras îs small enough that this is possible. If this had not been the case, several images of the target could hâve been captured and the means of the captured colour values could hâve been used instead. This technique can of course be used when the standard déviation is small as well, as a means of reducing the errors caused by the caméra itself. The results of the calibration procedures in Figure 6 on the other hand, show a far larger errer than can be accepted if bilirubin concentrations are to be measured. A remedy for this is suggested by Menesatti et al.

Menesattî et al. [35, p. 12] reports that when using the thin-plate spline interpolation method, the use of ColorCheckers with more colour patches results in calibrations of higher accuracy than calibrations performed with fewer colour patches. In addition, they showed that the farther away the measured colour was from the closest reference patch colour, the larger the calibration errer. For this reason, they suggest the use of ColorCheckers with several colours that closely match the colours of interest to reduce calibration errer. ColorCheckers could even be created and printed ad hoc, and the colours could then be measured a posteriori using a spectrometer.

Numerical skin simulations hâve been used to calculate the whole range of skin colours expected to exist in newborns. Skin colours to be used as reference colour points on the calibration card hâve been chosen by choosing colours that maximise the distance between them. This ensures that when the skin colour of newborns is measured the distance between the measured colour and the colours on the calibration card will be minimal, increasing the accuracy of the proceeding calibration algorithms.

The colours on the calibration card may be printed using a process known as spectral printing [36]. Spectral printing tries to replicate the entire reflection spectrum of light, instead of just the RGB colour. An advantage of using spectral printing is that the measured colours of the calibration card will change in a similar fashion to the measured colour of the skin when the light source illuminating the card and skin changes. Without using spectral printing, there is no guarantee that the printed colour remains similar to a skin colour when the light source or the illumination changes. Using spectral printing is therefore an advantage when creating a system that can be used in different light conditions or using different illumination sources. Typically 7 colour components are used in spectral printing. In other embodiments the there may be fewer than seven colour components to save cost of the calibration card. In other embodiments there may be more than seven colour components to allow control of the changes of the colours of the calibration card under changing illuminations conditions. An advantage with spectral printing of the calibration card is that it is possible to manufacture a calibration card that behaves similar to skin, sciera or area of the body under the same illumination conditions.

Typically the calibration card is made of cardboard or paper. The calibration card may hâve an opening or transparent section where the skin, sciera or an area of the body is visible. Preferably the calibration card is not transparent. The calibration card may be made of plastic or other material that provides sufficient stiffness in such a way that the calibration card does not easily bend or deform.

Fig. 9 shows a first embodiment of a colour calibration card 500 with grey patches 504 for light source strength correction. Rectangles with the number 504 indicate a grey rectangle used for light source strength correction. They are evenly distributed to ensure that ail coloured patches and skin is adjacent to at least one grey colour patch. Rectangles with the numbers 506 indicate coloured rectangles which are used for colour calibration. The colour of the coloured rectangles may be chosen among suitable skin colours, simulated skin colours or other colours suitable for calibration of images. The coloured rectangles 506 may hâve different colours, or some of the coloured rectangles may hâve similar colour. In an embodiment of the calibration card the coloured patches 506 comprises of 24 different colours. The rectangle with the number 502 indicates the opening in the calibration card where the skin will be visible. If there, for some reason, is no need for light source strength correction for a certain application, the grey patches can be replaced by more coloured patches.

Fig. 10 shows a second embodiment of a colour calibration card 500. Rectangles with the numbers 506 indicate coloured rectangles which are used for colour calibration. The colour of the coloured rectangles may be chosen among suitable skin colours, simulated skin colours or other colours suitable for calibration of images. The coloured rectangles 506 may hâve different colours, or some of the coloured rectangles may hâve similar colour. In an embodiment of the calibration card the coloured patches 506 comprises of 24 different colours. The rectangle with the number 502 indicates the opening in the calibration card where the skin will be visible. If there, for some reason, is no need for light source strength correction for a certain application, the grey patches can be replaced by more coloured patches.

In other embodiments of the calibration card, there may be no opening or the opening may be located at the periphery of the calibration card.

Preferably when photographing the skin, sciera or other area of the body the calibration card is part of the image.

The grey colours on the calibration card are used to correct any variation in light source strength across the imaged calibration card. When using a flash or ambient lighting as light source there is no guarantee that very part of the calibration is illuminated with the same intensity. By having grey calibration patches with the same colour across the card we can model the light source strength across the card and correct for any variation. This technique may be used both with calibration cards printed using spectral printing and with calibration cards printed using traditional printing techniques.

In addition to skin colours on the calibration card chosen from the numerical skin simulations described above, the calibration card may include some grey patches. These grey patches should be as similar as possible as they are used to correct for any variation in the illumination across the calibration card. Using the fact that the grey patches should reflect the same colour regardless to where they are situated on the calibration card, the variation in the reflected colour from theses patches can be used to correct for light intensity variation on to the colour patches and on to the skin. Several different algorithms could be used to perform this correction, such as bilinear interpolation, linear régression, Gaussian process régression methods, etc.

This method could possibly also be used as a diagnostic tool for other conditions. As mentioned earlier, the reflection spectrum of newborn skin can be used to calculate more than the bilirubin concentration. The colour of the skin is a resuit ofthe reflection spectrum and could therefore be used to calculate some of the same parameters, even though most ofthe spectral information is lost when the reflection spectrum is collapsed to three colour values. Takiwaki and Serup [39] found such colour indicators on psoriatic plaque, opening the possibility of using colour analysis to diagnose psoriasis, or at least to measure the extent of it. But the technique is not limited to the medical field. The colour measurement technique may be used to objectively measure the colour of e.g. paint, or custom colour calibration cards may be produced to asses the colour quality of food in factories and stores etc. The applications mentioned here should not be seen as an exhaustive list.

Best Modes of Carrying Out the Invention

Presently the best mode is to depict the skin using a smartphone and the smartphone’s flash with a custom made colour calibration card with colours selected from numerical skin simulations and printed using spectral printing placed on the skin. The skin colour is then calibrated using a Gaussian process régression method and compared with simulated skin colours and corresponding bilirubin concentrations in a direct lookup table. This gives a good estimation of the bilirubin concentration from the measurement of skin colour.

The calibration card needs to be as close possible to the area of the skin, sciera or other area that is to be analysed. Placing the calibration card close to the area to be analysed ensures that the same lighting conditions exist on the calibration card as the skîn/sclera being measured. A preferred way to get the card close to the 5 area that needs to be analysed is to hâve an opening in the calibration card. The calibration card is then placed on a child’s chest when measuring the skin, or held nearthe eye ofthe child when measuring sciera.

Alternative Embodiments

I0 A number of variations on the above can be envïsaged. For instance also sciera of the eyes can be used, using a similar model. Since jaundice affects the eyes at a delayed rate from the skin it is possible with one set of measurements to détermine if the level of bilirubin is increasing or decreasing. Thus if the level of bilirubin is increasing the analysis using skin gives a higher indication than for sciera. At steady

State both give the same indication.

The estimation of a bilirubin concentration from the calibrated skin colour measurement can also be performed by a meta model instead of a direct lookup table. The accuracy of this estimation may be improved by finding which input skin parameters to the model are most likely with regard to the measured colour. The 20 colour calibration can also be performed by other algorithme than the thin-plate spline, for example by partial least squares régression or Gaussian process régression. In addition, calibration can be improved by using both an image captured without flash and an image captured with flash. Calibration can also be performed on a single image captured without flash.

The calibration card may be manufactured of a stiff and transparent material like Plexiglas. The calibration colours are printed on the transparent material. The calibration card may be pressed towards the skin and thereby providing an even and fiat surface that may be imaged.

The calibration card may be equipped with light sources to provide calibrated 30 illumination of the skin, sciera or other part of the body. This would be advantageous in having a known illumination source.

In other embodiments the calibration card may be replaced with light projected on the skin, sciera or an area ofthe body. The light may be projected from a known light source like video projector, screen of a smartphone, flash or other suitable light 35 source. The screen of a smartphone may be used for illumination where a spécifie light pattern is shown on the display and when in proximity ofthe skin the light from the display is reflected from the skin and captured by the caméra in the smartphone.

The invention is not limited to the embodiment/s illustrated in the drawings. Accordingly it should be understood that where features mentioned in the appended daims are followed by reference signs, such signs are included solely for the purpose of enhancing the intelligibility ofthe daims and are in no way limiting on the scope of the daims.

Référencés

[1] Tina M. Slusher, Alvin Zipursky, and Vinod K. Bhutani. A global need for affordable néonatal jaundice technologies. Seminars in Perinatology, 35(3):185-191, June 2011.

Available from: http://linkinghub.elsevier.com/retrieve/pii/S0146000511000437, doi:10.1053/j.semperi.2011.02.014.

[2] Vinod K. Bhutani, Alvin Zipursky, Hannah Blencowe, Rajesh Khanna, Michael Sgro, Finn Ebbesen, Jennifer Bell, Rintaro Mon, Tina M. Slusher, Nahed Fahmy, Vinod K. Paul, Lizhong Du, Angela A. Okolo, Maria-Fernanda de Almeida, Bolajoko O. Olusanya, Praveen Kumar, Simon Cousens, and Joy E. Lawn. Néonatal hyperbilirubinemia and rhésus disease ofthe new-born: incidence and impairment estimâtes for 2010 at régional and global levels. Pédiatrie Research, 74:86-100, December 2013. Available from: http://www.nature.com/doifinder/10.1038/pr.2013.208, doi:10.1038/ pr.2013.208.

[3] Tina M Slusher, Bolajoko O Olusanya, Hendrik J Vreman, Ronald J Wong, Ann M Brearley, Yvonne E Vaucher, and David K Stevenson. Treatment of néonatal jaundice with filtered sunlight in nigérian neonates: study protocol of a non-inferiority, randomized controiled trial. Trials, 14:446, 2013. doi: 10.1186/1745-6215-14-446.

[4] V A Moyer, C Ahn, and S Sneed. Accuracy of clinical judgment in néonatal jaundice. Archives of pediatrics & adolescent medicine, 154(4):391-394, April 2000,

[5] A Carceller-Blanchard, J Cousineau, and E E Delvin. Point of care testing: transcutaneous bilirubinometry in neonates. Clinical biochemistry, 42(3):143-149, February 2009. doi: 10.1016/j.clinbiochem.2008.09.106.

[6] A N Bashkatov, E A Genina, V I Kochubey, and V V Tuchin. Optical properties of human skin, subeutaneous and mucous tissues in the wavelength range from 400 to 2000 nm. Journal of Physics D: Applied Physics, 38(15):2543 - 2555, August 2005. Available from: http://stacks.iop.org/0022-

3727/38/i=15/a=004?key=crossref.33282f52c8be33f7c28513be248e8c27, doi: 10.1088/0022-3727/38/15/004.

[7] R. Rox Anderson and John A. Parrish. The optics of human skin. Journal of Investigative Dermatology, 77(1):13-19, July 1981. Available from: http://www.nature.com/jid/journal/v77/n1/abs/5615637a.html, doi:10.1111/15231747.ep12479191.

[8] Fitzpatrick TB. The validity and practicality of sun-reactive skin types I through VI. Archives of Dermatology, 124(6):869-871, June 1988.

Available from: http://dx.doi.Org/10.1001/archderm.1988.01670060015008, doi:10.1001/archderm. 1988.01670060015008.

[9] M. L. Wolbarsht, A. W. Walsh, and G. George. Melanin, a unique biological absorber. Applied Optics, 20(13):2184-2186, 1981.

Available from: http://www.opticsinfobase.org/ao/fulltext.cfm?uri=ao-20-13-2184.

[10] L. T. Norvang, T. E. Milner, J. S. Nelson, M. W. Berns, and L. O. Svaasand. Skin pigmentation characterized by visible réflectance measurements. Lasers in Medical Science, 12(2):99-112, June 1997.

Available from: http://link.springer.eom/article/10.1007/BF02763978, doi:10.1007/BF02763978.

[11] Rashmi Sarkar, Srikanta Basu, R. K. Agrawal, and Piyush Gupta. Skin care for the newborn. Indian pediatrics, 47(7):593-598, 2010. Available from: http://link.springer.eom/article/10.1007/s13312-010-0132-0.

[12] Rachel Ash-Bernal, Robert Wise, and Scott M. Wright. Acquired methemoglobinemia: A rétrospective sériés of 138 cases at 2 teaching hospitals. Medicîne September 2004, 83(5):265-273, 2004.

Available from:

http://ovidsp.ovid. com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=ovft g&AN=00005792-200409000-00001.

[13] Janelle E. Phelps, Karthik Vishwanath, Vivide T. C. Chang, and Nirmala Ramanujam. Rapid ratiometric détermination of hemoglobîn concentration using L)VVIS diffuse réflectance at isosbestic wavelengths. Optics Express, 18(18):1877918792, August2010.

Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3093134/, doi:10.1364/OE.18.018779.

[14] A F McDonagh and D A Lightner. 'Like a shrivelled blood orange'-bilirubin, jaundice, and phototherapy. Pediatrics, 75(3):443-455, March 1985.

[15] N. E. I. Langlois and G. A. Gresham. The ageing of bruises: A re-view and study ofthe colour changes with time. Forensic Science International, 50(2):227-238, September 1991.

Available from: http://www.sciencedirect.com/science/article/pii/037907389190154B, doi: 10.1016/0379-0738(91 )90154-B.

[16] Neville R. Pimstone, Raimo Tenhunen, Paul T. Seitz, Harvey S. Marver, and Rudi Schmid. The enzymatic dégradation of hemoglobin to bile pigments by macrophages. The Journal of Experimental Medicine, 133(6):1264-1281, June 1971.

Available from: http://jem.rupress.Org/content/133/6/1264, doi:10.1084/jem. 133.6.1264.

[17] T. W. R. Hansen and D. Bratlid. Bilirubin and brain toxicity. Acta Pædiatrica, 75(4):513-522, July 1986.

Available from: http://onlinelibrary.wiley.eom/doi/10.1111/j.1651-

2227.1986.tb10242.x/abstract, doi:10.1111/j.1651-2227.1986.tb10242.x.

[18] Vinod K. Bhutani, Glenn R. Gourley, Saul Adler, Bill Kreamer, Chris Dalin, and Lois H. Johnson. Noninvasive measurement of total sérum bilirubin in a multiracial predischarge newborn population to assess the risk of severe hyperbilirubinemia. Pediatrics, 106(2):e17-e17, August 2000.

Available from:http://pediatrics.aappublications.org/content/106/2/e17.

[19] Steven L Jacques, David G Oelberg, and lyad Saidi. Method and apparatus for optical measurement of bilirubin in tissue, October 1994. Available from:http://www.google.com/patents/US5353790.

[20] L. Lyngsnes Randeberg, E. Bruzell Roll, L. T. Norvang Nilsen, T. Christensen, and L. O. Svaasand. In vivo spectroscopy of jaundiced newborn skin reveals more than a bilirubin index. Acta Pædiatrica, 94(1):65-71, 2005.

Available from: http://onlinelibrary.wiley.eom/doi/10.1111/j.16512227.2005.tb01790.x/abstract, doi:10.1111/j.1651-2227.2005.tb01790.x.

[21] Robin M. Pope and Edward S. Fry. Absorption spectrum (3807700 nm) of pure water. II. integrating cavity measurements. Applied Optics, 36(33):8710-8723, November 1997.

Available from: http://ao.osa.org/abstract.cfm?URI=ao-36-33-8710, doi:10.1364/AO.36.008710.

[22] L. O. Svaasand, L. T. Norvang, E. J. Fiskerstrand, E. K. S. Stopps, M. W. Berns, and J. S. Nelson. Tissue parameters determining the Visual appearance of normal skin and port-wine stains. Lasers in Medical Science, 10(1):55-65, March 1995. Available from: http://link.springer.eom/article/10.1007/BF02133165, doi:10.1007/BF02133165.

[23] Akîra Ishimaru. Diffusion of light in turbid material. Applied Optics, 28(12):2210, June 1989.

Available from:http://www.opticsinfobase.org/ao/fulltext.cfm?uri=ao-28-122210&id=32482,doi:10.1364/AO.28.002210.

[24] Richard C. Haskell, Lars O. Svaasand, Tsong-Tseh Tsay, Ti-Chen Feng, Matthew S. McAdams, and Bruce J. Tromberg. Boundary conditions for the diffusion équation in radiative transfer. JOSA A, 11(10):2727-2741, 1994.

Available from: http://www.opticsinfobase.org/abstract.cfm?id=841.

[25] Thorsten Spott and Lars O. Svaasand. Collimated light sources in the diffusion approximation. Applied Optics, 39(34):6453, December 2000. Available from: http://8.18.37.105/ao/abstract.cfm?uri=ao-39-34-6453, doi:10.1364/AO.39.006453.

[26] T. Smith and J. Guild. The C.I.E. colorimétrie standards and their use. Transactions ofthe Optical Society, 33(3):73, January 1931. Available from: http://iopscience.iop.Org/1475-4878/33/3/301, doi:10.1088/1475-4878/33/3/301.

[27] C.S. McCamy, H. Marcus, and J.G. Davidson. A color-rendition chart. Journal of Applied Photographie Engineering, 2(3):95-99, 1976.

[28] Adrian llie and Greg Welch. Ensuring color consistency across multiple caméras. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conférence on, volume 2, page 1268-1275. IEEE, 2005. Available from: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1544866.

[29] Paolo Menesatti, Claudio Angelini, Federico Pallottino, Francesca Antonucci, Jacopo Aguzzi, and Corrado Costa. RGB color calibration for quantitative image analysis: The “3D thin-plate spline warping approach. Sensors, 12(12):7063-7079, May 2012.

Available from: http://www.rndpi.com/1424-

8220/12/6/7063/,doi: 10.3390/s120607063.

[30] Fred L. Bookstein. Principal warps: thin-plate splines and the décomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6):567-585, June 1989. doi:10.1109/34.24792.

[31] Malcolm H. Davis, Alireza Khotanzad, Duane P. Flamig, and Steven E. Harms.

A physics-based coordinate transformation for 3-d image matching. Medical Imaging, IEEE Transactions on, 16(3):317-328, 1997.

Available from: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=585766.

[32] Jean Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Prof Dr Walter Schempp and Prof Dr Kari Zeller, éditons, Constructive Theory of Functions of Several Variables, number 571 in Lecture Notes in Mathematics, pages 85-100. Springer Berlin Heidelberg, January 1977. Available from: http://link.springer.com/chapter/10.1007/BFb0086566.

[33] Grâce Wahba. Spline Models for Observational Data. SIAM, September 1990.

[34] TEST: Samsung galaxy s III (s3) - multimedia.

Available from: http://www.amobil.nO/artikler/samsung-galaxy-s-iii-s3/109432.

[35] TEST: HTC one v - multimedia.

Available from: http://www.amobil.no/artikler/htc-one-v/110020.

[36] 'Spectral Printing’. Wikipedia, 16 June 2016.

https://en.wikipedia.org/w/index.php?title=Spectral_printing&oldid=725497925.

[37] Danny Pascale. RGB coordinates of the macbeth ColorChecker. 2006.

Available from: http://www.babelcolor.com/download/RGB%20Coordinates% 20of%20the%20Macbeth%20ColorChecker.pdf.

[38] Useful color data.

Available from: http://www.rit.edu/cos/colorscience/rc_useful_data.php.

[39] Hirotsugu Takiwaki and Jorgen Serup. Measurement of color parameters of psoriatic plaques by narrow-band réflectance spectrophotometry and tristimulus colorimetry. Skin Pharmacology and Physiology, 7(3):145-150, 1994.

Available from: http://www.karger.com/Article/Abstract/211289, doi:10.1159/000211289.

[40] Tondel, Kristin, and Harald Martens. Analyzing Complex Mathematical Model Behavior by Partial Least Squares Regression-Based Multivariate Metamodeling: Mathematical Model Behavior by Partial Least Squares Regression-Based Multivariate Metamodeling. Wiley Interdisciplinary Reviews: Computational Statistics 6, no. 6 (November 2014): 440-75. doi:10.1002/wics.1325.

[41] Berget, Ingunn, Bjorn-Helge Mevik, and Tormod Næs. New Modifications and Applications of Fuzzy -Means Methodoiogy. Computational Statistics & Data Analysis 52, no. 5 (January 20, 2008): 2403-18. doi:10.1016/j.csda.2007.10.020.

[42] Tondel, Kristin, Ulf G Indahl, Arne B Gjuvsland, Jon Olav Vik, Peter Hunter, Stig W Omholt, and Harald Martens. “Hierarchical Cluster-Based Partial Least Squares Régression (HC-PLSR) Is an Efficient Tool for Metamodelling of Nonlinear Dynamic Models. BMC Systems Biology 5 (June 1, 2011): 90. doi:10.1186/1752-0509-5-90.

[43] Solar spectrum calculator.

Available from:

https://www.pvlighthouse.com.au/calculators/solar%20spectrum%20calculator/solar %20spectrum%20calculator.aspx

[44] M. Ebden, arXiv: 1505.02965 [Math, Stat] (2015)

[45] A. Girard, Approximate Methods for Propagation of Uncertainty with Gaussian Process Models (Citeseer, 2004).

Industrial Applicability

The invention according to the application finds use in determining presence of jaundice in newborns using simple low cost means in the field.

Claims (9)

1. Claims

   1. A method for estimating a level of bilirubin in a subject which comprises receiving a depiction of skin from an RGB sensor with a colour calibration chart (500), calibrating a skin colour from the depiction using the colour calibration chart (500), calculating a level of bilirubin in skin of the subject by comparing simulated skin colours of known bilirubin concentration with the calibrated skin colour wherein the simulated skin colours are obtained by using an optical diffusion model of skin or Monte Carlo simulations of skin optics.
2. 2. A method for determining a rate of change in level of bilirubin skin and sciera, comprising receiving a depiction from an RGB sensor of skin and sciera with a colour calibration chart (500), calculating a level of bilirubin using the method of claim 1 for skin and sciera, and comparing the calculated level of bilirubin from the skin with the calculated level of bilirubin from the sciera, determining a rate of change in level of bilirubin by the différence in the indicated level of bilirubin from the skin from the indicated level of bilirubin from the sciera.
3. 3. An apparatus for estimating a level of bilirubin in a subject, which comprises means for receiving a depiction of skin using an RGB sensor with a colour calibration card, means for calibrating a skin colour from the depiction using the colour calibration chart (500), means for calculating simulated skin colours of known bilirubin concentration by using an optical diffusion model of skin or Monte Carlo simulations of skin optics, and means for calculating a level of bilirubin in skin of the subject by comparing the simulated skin colours with the calibrated skin colour.
4. 4. The apparatus according to claim 3 wherein the calibration chart (500) comprises a plurality of colour patches (506) wherein the colour patches (506) on the calibration chart (500) are printed using spectral printing.
5. 5. The apparatus according to claim 4 wherein the calibration chart (500) comprises further an opening (502) where the skin or sciera is visible.
6. 6. The apparatus according to claim 4 or 5, wherein the calibration chart (500) further comprises a plurality of grey patches (504) for detecting the variations of illumination ofthe calibration chart (500).
7. 7. The apparatus according to daims 4 to 6, wherein the grey patches (506) of the calibration chart (500) are evenly distributed over the calibration chart (500).
8. 8. The apparatus according to claim 4 to 7, wherein at least one grey patch (506) ofthe calibration chart (500) is arranged at a cornerof the calibration chart (500).
9. 9. The apparatus according to claim 2 or 5, wherein at least one patch (506) of the calibration chart is coloured with a simulated skin colour with a corresponding bilirubin concentration.