WO2020141969A2

WO2020141969A2 - System and method for providing advertisement contents based on facial analysis

Info

Publication number: WO2020141969A2
Application number: PCT/MY2019/050137
Authority: WO
Inventors: Farshid PIRAHANSIAH; Hock Woon Hon; Nik Ahmad Akram Bin NIK ZULKEPELI
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2018-12-31
Filing date: 2019-12-27
Publication date: 2020-07-09
Anticipated expiration: 2021-06-30
Also published as: WO2020141969A3

Abstract

The present invention relates to a system (100) for providing advertisement contents based on facial analysis comprising an image acquisition device (10) to acquire an image of a user, a face detection module (20) to detect face of the user in the image, an analysis module (40) to analyse the facial features statistically using classification models retrieved from a classification module (30), a database (60) to store matching rules, weighted advertisements and a plurality of advertisement contents; and a display device (80) to display the advertisement contents. The system (100) further comprises a computation module (50) to compute weighted image of the user and a matching module (70) to match the weighted image of the user with the weighted advertisement to select an advertisement content based on facial analysis of the user. A method of providing the advertisement contents based on facial analysis is also provided thereof.

Description

SYSTEM AND METHOD FOR PROVIDING ADVERTISEMENT CONTENTS BASED

ON FACIAL ANALYSIS

FIELD OF INVENTION

This invention relates to the field of digital advertising and more particularly to a system and method for providing advertisement contents in digital advertising based on facial analysis of user.

BACKGROUND OF THE INVENTION

Digital advertising are recently gained popularities among advertisement providers to accomplish various business goals, ranging from building brand awareness among potential customers to advertising available goods and services. The leverage of digital advertising lies on growing demand from the advertisement providers to implement targeted advertisement and its flexibility in modifying the advertisement contents in digital advertising. Digital standee is one of the digital advertising mediums to replace traditional static signage via installation of a self-standing display in public areas such as airports, hospitals, shopping malls and other high-traffic public areas.

Unfortunately, advertisement contents delivered via the digital standee are usually passive and do not display dynamic contents relevant to user’s possible interest. In most application, the digital standee displays all types of advertisement contents prepared by the advertisement provider regardless categories of user viewing the digital standee. The advertisement contents displayed are not selective based on user’s needs due to such standee does not have interaction with user and therefore make the advertisement contents less interesting and attract less attention. This makes digital advertising not purposeful and unfavourable to the advertisement providers.

Efforts have been made in the past to overcome the aforesaid limitations associated with the pertinent art particularly to display targeted advertisement without directly collecting personal information from the user. Face recognition technologies are utilized to support merchants to identify target user and implement active judgment for delivering accurate advertisement contents to improve marketing efficiency as well as to reduce marketing cost. A prior art of patent application CN107248091 (Ό91 ) discloses face recognition interactive marketing system, which comprises an image acquisition module for acquiring face data. According to patent application Ό91 , face features are captured and factors such as age range and gender of user are analysed. Therefore commodities are recommended automatically according to analysis result of the face features.

Another prior art patent US 7636456 B2 (‘456) discloses an improvement on previously digital signage arrangements to display images (i.e. advertisement contents) for providing a less disruptive playout of material on a display device. Rather than responding to an individual face being detected, the prior art‘456 depends on statistics gathered at different periods of times in a day. Thereby, a set of images are selected to be displayed on the display device at certain time of day according to frequency of faces categories detected at one or more different periods of time.

The prior arts however insufficiently address engagement of the digital advertising to potential customer since the prior arts only search for advertisement contents based on labels and tags of a user’s facial features or available user profile. Consequently, there exists a need in digital advertising to provide a system and method for selecting the advertisement contents based on user’s facial features for better engagement with user’s possible interest via the digital standee.

SUMMARY OF INVENTION

The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.

It is an objective of the present invention to provide advertisement contents via a digital standee by extracting salient demographic from a user to indirectly obtain user information and behavioural preference.

It is another objective of the present invention to provide advertisement contents via a digital standee by applying rules to match with types of business to the user information and behavioural preference for better engagement with user’s possible interest.

The present invention relates to a system for providing advertisement contents based on facial analysis comprising an image acquisition device to acquire an image of a user; a face detection module to detect face of the user in the image to obtain facial features of the user; an analysis module to analyse the facial features statistically using classification models retrieved from a classification module; a database to store matching rules; weighted advertisements and a plurality of advertisement contents and a display device to display the advertisement contents. The system is characterized by a computation module to compute weighted image of the user based on analysed facial features received from the analysis module and a matching module to match the weighted image of the user with the weighted advertisement based on the matching rules obtained from the database in order to select an advertisement content based on facial analysis of the user.

In a preferred embodiment of the present invention, the user may be a single user or a group of users.

In a preferred embodiment of the present invention, the classification module extracts facial features of the face and trains the classification models to group important facial features of the user.

In a preferred embodiment of the present invention, the weighted image of the user is an output of binary sequence.

In a preferred embodiment of the present invention, sequence of the facial features for the weighted image of the user is arranged in a same order of binary sequence of the weighted advertisements.

In a preferred embodiment of the present invention, the matching rules includes order of the features, most similarity features, important features and nearest similar features. The present invention also relates to a method of providing advertisement contents based on facial analysis, comprising the steps of acquiring an image of user by an image acquisition device; detecting face of the user in the image by a face detection module; and analysing facial features of the face statistically by an analysis module using classification models retrieved by a classification module, characterized by the steps of computing weighted image of the user by a computation module; obtaining matching rules and weighted advertisement from a database and matching the weighted image of the user with the weighted advertisement based on the matching rules, by a matching module, to select an advertisement content based on facial analysis of the user.

In a preferred embodiment of the present invention, the method further comprises a step of extracting facial features of the face and training the classification models by the classification module to group important facial features of the use.

In a preferred embodiment of the present invention, the method further comprises a step of providing display of the selected advertisement content by a display device.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the invention will be more readily understood and appreciated from the following detailed description when read in conjunction with the accompanying drawings of the preferred embodiment of the present invention.

Figure 1 is a diagram illustrating a system for providing advertisement contents based on facial analysis in accordance to the present invention.

Figure 2 is a flow chart representing a method of providing advertisement contents based on facial analysis in accordance to the present invention.

Figure 3 is a flow chart representing an exemplary embodiment of matching rules for selecting suitable advertisement contents based on weighted image of the user in accordance to the present invention.

Figure 4 is a flow chart representing an exemplary embodiment for matching weighted advertisement and weighted image of the user based on the matching rules in accordance to the present invention. DESCRIPTION OF THE PREFERRED EMBODIMENT

The above mentioned features and objectives of this invention will become more apparent and better understood by reference to the following detailed description. It should be understood that the detailed description made known below is not intended to be exhaustive or limit the invention to the precise disclosed form, as the invention may assume various alternative forms. On the contrary, the detailed description covers all the relevant modifications and alterations made to the present invention, unless the claims expressly state otherwise.

The present invention relates to a system and method for providing advertisement contents based on facial analysis on a digital standee. The system detects user presence within capturing range of the digital standee and extracts salient demographic from the user. Behavioural preference and information of the user is obtained by analysing facial features of the user before applying rules to match with types of business to provide suitable advertisement contents to the user, wherein the rules is based on advertisement provider’s request.

Figure 1 illustrates a preferred embodiment of a digital standee (5) in accordance to the present invention. Figure 1 further illustrates a system (100) preferably embedded in the digital standee (5) for providing advertisement contents based on facial analysis. The system (100) of the present invention comprises an image acquisition device (10), a face detection module (20), a classification module (30), a data analysis module (40), a computation module (50), a database (60) and a matching module (70). The system (100) further comprises a display device (80) to display the advertisement contents preferably not limited to image, video, animation, text, etc.

The image acquisition device (10) is configured to acquire an image of at least one user. In one embodiment, the image acquisition device (10) is a camera mounted on the digital standee (5).The image acquisition device (10) may be discreetly mounted on the digital standee (5) so as not to impede a user’s normal reaction when viewing the advertisement contents. The image acquisition device (10) may also be integrated in the digital standee (5) so as not to distract user’s attention from viewing the advertisement contents. The image acquisition device is preferably (10) has a plurality of sensors for detecting presence of the user within capturing range of the image acquisition device (10). One of the sensors is preferably an ultrasonic sensor to detect presence of the user by processing distance of the user from the image acquisition device (10). It should also be appreciated that distance of the user from the image acquisition device (10) may also be detected using multiple cameras (e.g. two cameras) or depth camera.

The system (100) further comprises the face detection module (20) to detect face of the user in the image. The face detection module (20) implements deep learning technology which is also known as machine learning methods to detect face of the user in the image. Examples of the deep learning technology implemented in the present invention may include Deep Reinforcement Learning (DRL), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Generative Adversarial Network (GAN).

In another embodiment, the face detection module (20) may also compute time spent by the user within capturing range of the image acquisition device (10) with implementation of a timer in the system (100). In an embodiment, time spent by the user is computed by instigating a timer at a time of taking an image of a first user and continuously matching image of current user who is viewing the advertisement contents with the image of the first user. If image of a new face is detected, the image of the new face becomes the image of current user and the image of the first user is no longer matched. Therefore, the timer for the first user is halted while simultaneously timer of the current user is automatically instigated. Thereon, time spent for the first user within capturing range of the image acquisition device (10) is computed.

It is also preferred in the present invention to recognize if face in the image belongs to a single user or a group of users by estimating number of faces in the image. The face detection module (20) also normalizes and crops the detected face before sending to the data analysis module (40).

The classification module (30) comprises classification models of facial features of user’s face. The facial features are extracted and trained using the deep learning technology by the classification module (30). The extracted facial features are then classified into a plurality of classification models to group important facial features such as gender, age range, emotion (e.g. happy, sad, angry, fear, disgusting), style (e.g. eyeglasses, goatee, moustache, beard, sideburns, hat, young) and attention (i.e. detecting gaze of the user). Training for the classification models are preferably configured prior to the use of the system (100).

The system (100) further comprises the data analysis module (40).The data analysis module (40) receives the face detected and recognized from the face detection module (20) and analyse the facial features statistically using the trained classification models obtained from the classification module (30).The data analysis module (40) calculating statistical parameters using an algorithm called face landmark estimation. The landmark estimation estimates 68 specific points known as landmarks that exist on every face i.e. top of the chin, outside edge of each eye, inner edge of each eyebrow, etc.

The system (100) further comprises the computation module (50) to compute weights of the face based on the analysed facial features of the user. In the computation module (50), computation components such as Cumulative Match Score (CMS) curve, Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are used to compute mean value, standard error, median, mode, standard deviation, sample variance, skewness, kurtosis, range, minimum and maximum value to calculate weight of the face. Output of the computation module (50) for the calculated weight of the face of each facial feature is in binary sequence, i.e. 0 and 1 . As an exemplary of embodiment, the output is mapped onto a table as shown in Table 1 to find a weighted image of the user with an advertisement number Ad1 . The weighted image of the user is a collection of the binary sequence arranged in an order of the analysed facial features.

The facial features such as age, gender, emotion, styles and attention of the user are computed by the computation module (50) and outputted in binary format, i.e. 0 or 1 . The emotions may include happy, sad, angry, fear and disgusting while styles may be represented by accessories wore by the user. Referring to the Table 1 , the output of the weighted image of the user is binary 1 1 1000010000101 arranged in an order of requirement set by the system (100).

Table 1

In the same order of requirement (i.e. the age, gender, emotion, styles and attention), the system (100) then matches the weighted image of the user with weighted advertisements using a matching module (70). A plurality of weighted advertisements are generated by the system (100) from the advertisement contents and based on matching rules provided by the advertisement providers. The system (100) further comprises a database (60) to store the advertisement contents and the matching rules.

An example of the weighted advertisements from a company selling sunglasses is provided, whereby in a particular promotion said company may wish to provide advertisement contents based on gender of the user, e.g. male potential customer. Thereon, a binary sequence governing the requirement (i.e. gender, male) is provided to be matched with the weighted image of any user presents in the capturing range of the image acquisition device (10). In the present example, a sample of the binary sequence may be set to 100000 to satisfy the requirement set by the advertisement providers, with 1 represents male gender of the user. Thereupon, the system is looking for the weighted image of the user in a range of I xxxxx, wherein 1 represents male gender of the user and x is any integer number representing other facial features, in order to display the suitable advertisement contents of men’s sunglasses in the particular promotion. For example, if a man approaches the digital standee (5), the weighted image of the user computed by the computation module (50) should be in a range of I xxxxx.

The present invention also includes a method (200) of providing the advertisement contents based on facial analysis with reference to Figure 2. The method (200) comprising steps of acquiring an image of user (201 ) by the image acquisition device (10), detecting face of the user in the image (202) by the face detection module (20) to obtain facial features of the user, analysing facial features of the face statistically (203) by the analysis module (40) using the trained classification models retrieved by the classification module (30), computing the weighted image of the user (204) by the computation module (50), obtaining the matching rules and the weighted advertisement (205) from the database (60), matching the weighted image of the user with the weighted advertisement (206) based on the matching rules by the matching module (70) and providing display of the advertisement contents (207) by the display device (80).

The step of acquiring an image of user (201 ) by the image acquisition device (10) includes detecting presence of user within the capturing range of the image acquisition device (10). If presence of user is detected, image of the user is captured and sent to the face detection module (20). In the preferred embodiment, the step of detecting presence of the user further includes determining distance of the user from the image acquisition device (10) using sensor, for instance the ultrasonic sensor. The distance may be determined based on a geometry calculation of two cameras such as stereo vision, depth map and distance by stereo vision or by a plurality of depth cameras.

The step of detecting face of the user in the image (202) by the face detection module (20) is performed to identify position of a valid face in the acquired image. The face of the user may not be perfectly captured by the image acquisition device (10) (i.e. only half or a portion of the face is captured) or the user may only present within the capturing range of the image acquisition device (10) without facing the digital standee. Therefore, the valid face is a face of the user which is perfectly captured to be analysed by the analysis module (40). The step of detecting face of the user (202) preferably includes estimating a number of faces in the image to recognize if the face in the image belongs to a single user or a group of users. In one embodiment, the number of faces in the image is estimated based on feature regression technique for estimating number of faces in an image. The step of detecting face of the user (202) further includes normalizing and cropping the face detected for facial analysis.

The step of analysing facial features of the face statistically (203) by the analysis module (40) includes calculating statistical parameters of the facial features using the face landmark estimation algorithm. The facial features such as gender, age range, emotion, style and eye gaze of the user is analysed with reference to the classification models retrieved from the classification module (30). In one embodiment, the classification models is obtained by a step of extracting facial features of the face to extract important facial features including gender, age range, emotion, style and eye gaze of the user using the deep learning technology. In the preferred embodiment, the eye gaze of the user is preferably determined through eye detection from the image acquisition device (10). The eye detection determines if the valid face of the user is facing straight at the image acquisition device (10) with eyes open. The step of extracting facial features of the face is preferably a continuous real-time process to update the classification models.

The step of computing the weighted image of the user (204) by the computation module (50) is performed based on the analysed facial features of the user. The weighted image of the user is preferably in a form of binary format comprises computed facial features of the user such as age, gender, emotion, styles and attention. The next step is the step of obtaining the matching rules and the weighted advertisement (205) from the database (60). The matching rules set out rules to choose a suitable advertisement contents to match with requirements provided by the advertisement providers. Sequence of the facial features for the weighted image of the user is arranged in the same order of the sequence of the weighted advertisements.

Next is the step of matching the weighted image of the user with the weighted advertisement (206) based on the matching rules by the matching module (70). The matching rules are established by the advertisement providers provided along with the relevant advertisement contents. In the embodiment of the present invention, the matching rules may include order of the features, most similar features, important features and nearest similar features. In the present invention, the comparison and matching is assessed from left to right of the binary sequence. Therefore, the weighted advertisement matched from left to right is chosen.

Figure 3 shows an exemplary embodiment of the matching rules for selecting the suitable advertisement contents based on the weighted image of the user computed by the computation module (50). In the exemplary embodiment of Figure 3, Q1 to Q4 are the advertisement contents selected upon undergo the matching rules mentioned previously. Q5 displays the advertisement contents chosen based on the matching rules obtained from the database (60). If there is no advertisement satisfies any matching rules provided by the advertisement providers, then a random advertisement contents are displayed accordingly. It should also be understood that more than one matching rules may be applied for one weighted image of the user. Therefore, in such case, the advertisement contents placed in Q1 to Q4 are displayed consecutively according to sequence, for example starting from Q1 , followed by Q2, Q3 and finally Q4.

Figure 4 shows an exemplary embodiment for matching the weighted image of the user with the weighted advertisement based on the matching rules as set out in Figure 3. Referring to 4(a) to 4(b), the weighted image of the user computed by the computation module (50) is 100101 and in the exemplary embodiment, two weighted advertisements provided by the advertisement providers are 000101 and 1001 10.

In 4(a) of Figure 4, the weighted image of the user is matched using the order of the features. If the advertisement provider set order of the features as the matching rules for specific advertisement content, then, the matching module (70) selects the advertisement content according to arrangement of the binary output of the weighted image of the user. For example, if the priority of the facial features set by the advertisement provider is‘sunglass’ and followed by‘young’, then the matching module (70) select the advertisement contents from the most important features to not very important features. In the example 4(a), 1001 10 is selected due to the first feature is matched with 100101 . In 4(b) of Figure 4, if the advertisement provider set the most similarity features as the matching rules for specific advertisement content, then, the matching module (70) selects the advertisement content based on the most similar features, wherein in the example 1001 10 is selected due to the similarity of the first three features with 100101 . In example of 4(c), the matching rules of important features is selected if the advertisement provider set a particular feature as important for selecting the advertisement content. In this example compare and match with the rule of important of the features which means select the advertisements based on the features which important to company. Finally, the matching rule of nearest similar is chosen if the advertisement provider does not specify any particular feature as shown in 4(d), the weighted advertisement having nearest similarity that matched with the weighted image of user is selected.

Upon matching the weighted advertisement with the weighted image of user, display of the corresponding advertisement contents is provided (207) by the display device (80).

The terms“a” and“an,” as used herein, are defined as one or more than one. The term“plurality,” as used herein, is defined as two or more than two. The term“another,” as used herein, is defined as at least a second or more. The terms“including” and/or “having,” as used herein, are defined as comprising (i.e. , open language).

While this invention has been particularly shown and described with reference to the exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention as defined by the appended claims.

Claims

1 . A system (100) for providing advertisement contents based on facial analysis comprising:

an image acquisition device (10) to acquire an image of a user;

a face detection module (20) to detect face of the user in the image to obtain facial features of the user;

an analysis module (40) to analyse the facial features statistically using classification models retrieved from a classification module (30);

a database (60) to store matching rules, weighted advertisements and a plurality of advertisement contents; and

a display device (80) to display the advertisement contents, characterized by

a computation module (50) to compute weighted image of the user based on analysed facial features received from the analysis module (40); and

a matching module (70) to match the weighted image of the user with the weighted advertisement based on the matching rules obtained from the database (60) in order to select an advertisement content based on facial analysis of the user.

2. The system (100) according to claim 1 , wherein the user may be a single user or a group of users.

3. The system (100) according to claim 1 , wherein the classification module (30) extracts facial features of the face and trains the classification models to group important facial features of the user.

4. The system (100) according to claim 1 , wherein the weighted image of the user is an output in a form of binary sequence.

5. The system (100) according to claim 4, wherein sequence of the facial features for the weighted image of the user is arranged in a same order of binary sequence of the weighted advertisements.

6. The system (100) according to claim 1 , wherein the matching rules includes order of the features, most similar features, important features and nearest similar features.

7. A method (200) of providing advertisement contents based on facial analysis, comprising the steps of:

acquiring an image of user (201 ) by an image acquisition device (10); detecting face of the user in the image (202) by a face detection module (20) to obtain facial features of the user; and

analysing facial features of the face statistically (203) by an analysis module (40) using classification models retrieved by a classification module (30), characterized by the steps of

computing weighted image of the user (204) by a computation module

(50);

obtaining matching rules and weighted advertisement (205) from a database (60); and

matching the weighted image of the user with the weighted advertisement (206) based on the matching rules, by a matching module (70), to select an advertisement content based on facial analysis of the user.

8. The method (200) according to claim 7, wherein the method (200) further comprises a step of extracting the facial features of the face and training the classification models by the classification module (30) to group important facial features of the use.

9. The method (200) according to claim 7, wherein the method (200) further comprises a step of providing display of the selected advertisement content (207) by a display device (80).