Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T17/00—Three-dimensional [3D] modelling for computer graphics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/20—Image enhancement or restoration using local operators
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/40—Image enhancement or restoration using histogram techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10024—Color image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20021—Dividing image into blocks, subimages or windows
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20221—Image fusion; Image merging
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20224—Image subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30036—Dental; Teeth
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30196—Human being; Person
  • G06T2207/30201—Face
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30244—Camera pose
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2210/00—Indexing scheme for image generation or computer graphics
  • G06T2210/41—Medical
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2210/00—Indexing scheme for image generation or computer graphics
  • G06T2210/44—Morphing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/20—Indexing scheme for editing of 3D models
  • G06T2219/2021—Shape modification

Definitions

• the present invention relates generally to three-dimensional facial reconstruction, and more particularly to a method and to a device for three-dimensional reconstruction of a face having a toothed part, from a single image, as well as to a computer program product implementing the method.
• the invention finds applications, in particular, in digital processing techniques in the field of dentistry.
• the planned dental treatment can be of an aesthetic, orthodontic or prosthetic nature.
• Three-dimensional (3D) facial reconstruction is a rapidly expanding field and finds very varied applications. Previously used mainly in the audiovisual industry, it now finds other applications, notably in the simulation of aesthetic treatments.
• Document US2018174367A discloses an augmented reality display system of a model making it possible to directly see the simulated result of a planned dental treatment, and also offering the possibility of interacting with this model to modify it in real time.
• the system operates by acquiring video data (therefore relating to a plurality of images), simulating dental treatments on this video data, and rendering the result in augmented reality. If a 3D scan of the toothed part is available, it can be registered on an image, with or without the simulation of the planned treatment. Alternatively, a simulation can be done on video data, with the double disadvantage of having two image sensors on the one hand, and a rough simulation result on the other.
• Document US2018110590A discloses a simulation method in which a dental arch is digitized in 3D on which it is envisaged to apply a dental treatment (fitting of rings, crowns, aligners, etc.), then, in a reality system augmented we align the 3D dental arch including the simulation of the dental treatment projected on the real image of the patient which is in 2D, with the aim of visualizing in this system no longer the real dental arch of the patient but this arch with the result of the planned dental treatment.
• the invention aims to aim to make possible the facial reconstruction, ie, 3D reconstruction, of the face of a human subject with a visible toothed portion, from any series of 2D images or possibly from any 'a single 2D image of the face with the toothed portion, the 3D reconstruction thus obtained lending itself well to the apposition in the 3D domain of the result of the simulation of a projected dental treatment which modifies the toothed portion.
• This object is achieved by means of a method comprising the separation of the 2D image of the face into a part corresponding to the toothed part alone and another part corresponding to the rest of the face, the first part being subjected to a digital processing of enhancement before merging with the second, either at 2D level or at 3D level.
• the 3D reconstruction, or 3D surface, thus obtained is suitable for the simulation of a projected dental treatment to be applied to the toothed portion of the face, by substitution for the zone of the 3D surface corresponding to said toothed portion of another 3D surface corresponding to said toothed portion as it would be after said planned treatment.
• a three-dimensional, 3D reconstruction method to obtain, from at least one two-dimensional, 2D, color image of a human face with a visible toothed portion, a single reconstructed 3D surface. of the toothed portion and of the facial portion outside the toothed portion of the face, said method comprising:
• the embodiments use the enhanced 2D image (or images) with respect to the toothed portion, in order to produce a 3D facial reconstruction, with toothed portion, of the subject's face. It is the enhancement of the toothed part of the image of the patient's face that makes possible the 3D reconstruction not only of the facial part (excluding the toothed part) but also of the toothed part itself, from a single 2D image of the face with this toothed part visible.
• This first mode of implementation provides that the facial reconstruction is decoupled from that of the toothed portion.
• the generation of the 3D surface of the face can comprise:
• the second deep learning algorithm can be based on a method of the pose generation method type 3D via a 3D Morphable Model or 3DMM (standing for “3D Morphable Model”) adapted to deform a generic 3D surface so as to be closer photometrically to the 2D image.
• the first deep learning algorithm can be adapted to predict a depth map for the toothed portion of the face from training data by masking a depth map associated with the 2D image with the same mask as a mask used on the 2D image to get the first part of the 2D image corresponding to the toothed part of the face, and the depth map for the toothed portion of the face can be converted to a 3D reconstruction which is merged with the 3D reconstruction of the facial portion outside the toothed portion of the face to produce the 3D surface of the face.
• the second algorithm can also be adapted to produce the relative 3D position of the camera having taken the face as presented on the 2D image as well as an estimate of the 2D area of said 2D image in which is locates the toothed portion of the face, so that a consolidated 3D surface of the face can be obtained from a plurality of 2D images of the face taken by a camera at different respective viewing angles and for each of which the steps of the process are repeated to obtain respective reconstructed 3D surfaces, said reconstructed 3D surfaces then being combined using the relative 3D position of the camera having taken the face as presented on each 2D image as well as the estimation of the 2D area of said image 2D in which the toothed portion of the face is located.
• a second embodiment provides that the 3D reconstruction of the facial part outside the toothed part and that of the toothed portion are carried out by one and the same algorithm.
• the generation of the 3D surface of the face can comprise the implementation of a third deep learning algorithm, adapted to produce an overall 3D reconstruction of the toothed portion and of the facial portion outside the toothed portion from of the second part of the 2D image to which is added the first part of said enhanced 2D image with mutual registration of said second part of the 2D image and of said and first part of said enhanced 2D image.
• the third deep learning algorithm can be based on a method of the 3D pose generation method type via a 3D morphable or 3DMM model adapted to deform a generic 3D surface so as to approximate on the photometric plane of the second part of the 2D image to which is added the first part of said enhanced 2D image;
• - modifying the photometric characteristics of the first 2D part of the image may include increasing the sharpness and / or increasing the contrast of said first part of the 2D image;
• the enhancement of the toothed portion of the 2D image can be achieved using a series of purely photometric filters
• the enhancement 2D processing comprises the extraction of the blue channel, a high-pass contrast enhancement filtering applied to the extracted blue channel, as well as a local histogram equalization filtering, for example of CLAHE type, applied to the blue filtered channel;
• the high-pass contrast enhancement filtering applied to the blue channel may include an algorithm for increasing the sharpness, for example consisting in partially subtracting from said blue channel a blurred version of itself;
• the first part of the enhanced 2D image can be produced from the original 2D image as an intermediate output of a semantic segmentation deep learning network, having a higher contrast than the 2D image original, and selected according to a determined quantitative criterion;
• a contrast metric can be associated with the output of the convolution kernel of each of the convolution layers of the semantic segmentation deep learning network, and the selected intermediate output of the semantic segmentation deep learning network can be the output showing maximum contrast with respect to the metrics associated with the respective intermediate outputs of said semantic segmentation deep learning network.
• the invention also relates to a device having means suitable for carrying out all the steps of the method according to the first aspect above.
• a third aspect of the invention relates to a computer program product comprising one or more sequences of instructions stored on a memory medium readable by a machine comprising a processor, said sequences of instructions being adapted to achieve all the steps of the method according to the first aspect of the invention when the program is read from the memory medium and executed by the processor.
• the invention also relates to a method for simulating the aesthetic result of a dental treatment planned for a human subject, for example an aesthetic, orthodontic or prosthetic treatment, from at least one two-dimensional, 2D, color image of the subject's face with a visible toothed portion, said method comprising:
• the method comprises the implementation of an algorithm applied to a 3D reconstruction of the total dental arch of the subject, said algorithm being adapted to register the dental arch on the toothed portion of the 3D surface of the face as obtained by the method according to the first aspect, and to replace the toothed portion within said 3D surface of the face by a corresponding part of said 3D reconstruction of the dental arch of the subject, that is to say by the part of the dental arch of the subject which is visible in the 2D image;
• the dental arch can undergo a digital treatment, either automatic or manual, before realignment on the toothed portion of the 3D surface of the face, in order to simulate within said 3D surface of the face the aesthetic result of the planned treatment;
• the planned treatment may include at least one from the list of the following aesthetic, orthodontic or prosthetic treatments: a change in the color of the teeth, a realignment of the teeth, an affixing of veneers on the teeth, an installation of orthodontic material (for example for example rings) or prosthetic (for example a crown, a "bridge”, an "inlay-core”, an “inlay-onlay”).
• FIG. 1 is a functional diagram illustrating the segmentation, according to the method of the first aspect of the invention, of a 2D color image of a human face with a visible toothed portion, into a first part corresponding to the portion toothed face only and a second part corresponding to the facial portion, outside said toothed portion, of the face;
• FIG. 2 is a diagram of steps of a first embodiment of the method making it possible to obtain a 3D reconstruction from the 2D image of FIG. 1, in which the 3D reconstruction is carried out separately for each of the first and second parts of the 2D image, after enhancement of the first part and before merging to 3D level of the 3D reconstructions thus obtained;
• FIG. 3 is a diagram of steps of a first embodiment of the method making it possible to obtain a 3D reconstruction from the 2D image of FIG. 1, in which the 3D reconstruction is carried out together for the first and second parts of the 2D image, after enhancement of the first part and merging of the two parts to the 2D level;
• FIG. 4 is a functional diagram illustrating a first method of enhancing the toothed portion of the face of the 2D image, using a processing which implements a series of photometric filters;
• Figure 5 is a functional diagram illustrating a second method of enhancing the toothed portion of the face of the 2D image, exploiting advances in artificial intelligence using an intermediate output of a deep learning network;
• FIG. 6 is a functional diagram illustrating an example of implementation of the simulation method according to the fourth aspect of the invention, in which the intended treatment is teeth whitening.
• the invention takes advantage of deep learning architectures such as deep neural networks and convolutional neural networks (or neural networks) or convolutional neural network or even CNN (standing for "Convolutional Neural Networks) »)
• deep learning architectures such as deep neural networks and convolutional neural networks (or neural networks) or convolutional neural network or even CNN (standing for "Convolutional Neural Networks) »)
• CNN standing for "Convolutional Neural Networks” »
• An “image”, or “view”, or even “scan”, consists of a set of points of the real three-dimensional scene.
• the points concerned are the points of the real scene projected in the focal plane of the 2D sensor used to acquire the 2D image, and are defined by the pixels of the 2D image.
• this term designates the product or result of the 3D reconstruction processing, the points concerned being a 3D point cloud obtained by a transformation of a “depth map” (see definition given below), or by triangulation in the case of stereoscopy, or by 3D deformation of a generic 3D model in the case of a 3DMM type method (see definition given below).
• a point cloud defines a skeleton of the three-dimensional scene.
• a 3D mesh of this cloud of points for example a mesh of triangulated 3D points, can define an envelope.
• a "monocular" image acquisition device is a device having only a single image sensor and capable of acquiring images of a three-dimensional scene only under a single viewing angle only at a given device position.
• the “registration” (in English “registration") consists in determining the spatial relationship between two representations (2D image or 3D surface) of the same object so as to overlap the representations of the same physical point.
• the “pose calculation” is the estimation of the position and the orientation of the scene imaged with respect to the imager (image sensor). This is one of the fundamental problems in computer vision, often referred to as “Perspective-n-Points” (PnP).
• This problem consists in estimating the pose (2-tuple [R j ,; tj ⁇ formed by the rotation matrix flj and the translation vector tj) of the camera with respect to an object in the scene, which amounts to finding the pose making it possible to reduce the reprojection error between a point in space and its 2D correspondent in the image.
• ePNP from the English “Efficient Perspective-n-Point”
• This approach adds to that the fact of setting the pose of the camera through 4 control points, ensuring that the estimated transformation is rigid. The fact of proceeding in this way makes it possible to make the computation times shorter.
• enhancement of the toothed portion is understood to mean a 2D level treatment specific to the toothed portion aimed at improving the photometric characteristics of said toothed portion.
• this specific processing to the toothed portion may include applying a sequence of image processing filters. In other embodiments, it includes taking advantage of an intermediate output of a learning network.
• a “sharpening” algorithm is an image processing algorithm for increasing the sharpness of the image.
• 3DMM denotes a method of generating a 3D pose via a 3D morphable (that is to say modifiable) model. This method is particularly suitable for processing information on the face of a human being (skin, wrinkles, illumination, relief, etc.).
• the 3DMM method involves affixing a 3D face (mask) to the 2D image, and modifying it to match a face on the 2D image. The information corresponding to the modified mask is then extracted which will make it possible to create the 3D representation of the face of the 2D image.
• a "depth map” associated with a 2D image is a form of 2D representation of reconstructed 3D information, corresponding to the portion of the 3D scene reprojected in the 2D image.
• this is a set of values, coded in the form of levels (or shades) of gray, respectively associated with each pixel p, of the 2D image: plus the distance between the point of the three-dimensional scene and the plane of the 2D image is large, and the darker the pixel.
• One CNN is made up of two types of artificial neurons, arranged in “strata” or “layers” successively processing information:
• processing neurons which process a limited portion of the image (called the “receptive field”) through a convolution function; and, - the (total or partial) pooling neurons of the outputs, known as “pooling” neurons (which means “regrouping” or “pooling”, in English), which make it possible to compress the information by reducing the size of the intermediate image (often by downsampling).
• All the outputs of a processing layer make it possible to reconstitute an intermediate image, which serves as a basis for the following layer.
• a non-linear and punctual corrective treatment can be applied between each layer to improve the relevance of the result.
• CNNs are currently experiencing wide applications in the field of image recognition.
• the embodiments of the method of the invention include the segmentation of the two-dimensional image (2D) 21 of the face of a human subject, here a young woman, in a first part 22, on the one hand, and a second part 22, on the other hand.
• the first part 22 corresponds only to the toothed portion 1 of the face, which is visible in image 21. It is obtained by masking and blacking, in image 21, the facial portion 4 outside the toothed portion 1 of the face.
• the second part 23 corresponds only to the facial portion 4, apart from the toothed portion 1, of the face. It is obtained by masking and blacking in the 2D image of said toothed portion 1 of the face.
• the toothed part 1 is shown in Figure 1 in detail 10 of image 21, which corresponds to the area of the subject's mouth, which area is also identified by the same reference 10 in part 22 and in part 23 of image 21.
• the toothed part excludes the lips and the gums, in order to really only understand the portion visible in image 21, where appropriate, of the upper arch and / or the lower arch of the subject's dentition.
• This toothed portion presents, compared to the rest of the face, a high specularity and a particular texture which make 3D reconstruction difficult with conventional 3D facial reconstruction techniques.
• This segmentation of the 2D image into two parts makes it possible to implement an image processing specific to the toothed portion 1 which is the sole object of the first part 22, in order to overcome the poor photometric properties of said portion toothed 1 in relation to the other portions of the face.
• the image processing is adapted to enhance these properties, in particular the contrast.
• Such a treatment is designated by the term “enhancement”. It is only applied to the toothed portion 1, ie, only to the part 22 of the image 22 of the face.
• the toothed part after raising and the facial part outside the toothed part are then merged, that is to say recombined to finally give the 3D reconstruction of the two-dimensional image 21 of the face with the toothed part.
• the method begins, in step 201, with the acquisition of at least one image (ie, of a 2D view) of the face of a subject that includes a visible toothed portion.
• a 2D view is the case, especially when the subject is smiling.
• a smile is the result of a natural expression of an emotion, which can also be controlled by the subject.
• smiling exposes all or part of the upper dental arch, and usually also the lower arch of the subject, due to the opening of the mouth and the stretching of the lips that smiling causes.
• step 201 includes taking a plurality of images of the patient's face, such as image 21, taken from different respective viewing angles. These embodiments, which will be returned to later, improve the precision of the 3D reconstruction of the subject's face.
• step 202 the segmentation of the image 21 is carried out into a first part 22 and a second part 24.
• the first part 22 corresponds to the toothed portion 1 from the face only.
• the second part 24 corresponds only to the facial portion 4, apart from said toothed portion 1, of the face.
• This segmentation step 202 can be carried out by a digital processing applied to the data of the image 21, via an algorithm 51 which implements the detection of external limits of the toothed portion 1 of the face thanks to a deep detection learning network. of characteristic points on a face. This makes it possible to generate a mask for each of said first and second parts 22 and 24, respectively, of the image. 21.
• the effect of these masks is as follows:
• - the first part 22 of image 21 is obtained from said image 21 by masking, that is to say by putting in black the facial portion 4 outside the toothed portion of the face; and, - the second part 24 of image 21 is obtained from said image 21 by putting the toothed portion 4 of the face in black.
• parts 22 and 24 of image 21 are 2D images each corresponding to said image 21 but in which part of the pixels are replaced by black pixels.
• This technique is known per se and its implementation is within the reach of those skilled in the art, which is why it will not be described in more detail in the present description.
• the deep learning network of algorithm 51 is, in particular, adapted to exclude the lips and the gums from the first part 22, so that the latter only includes the toothed part 1 proper, of which the specularity and texture are very different from that of organic tissues, whether they are soft or hard, such as the skin, lips or mucous membranes of the mouth.
• An example of such a deep learning network is described in the article Bulat et al. "How far are we from solving the 2D & 3D face aligned problem? (And a dataset of 230,0003D facial landmarks)", ICCV, 2017. The described algorithm finds characteristic points distributed along the lips.
• step 203 the implementation of a facial reconstruction algorithm is carried out which can also be implemented in the form of a deep learning network 42.
• This CNN is adapted to predict a 3D textured reconstruction 34 of the facial portion 4 outside the toothed portion 1 of the face. This reconstruction is obtained on the basis of the second part 24 of image 21.
• the algorithm 42 is for example based on the concept of 3DMM (standing for “3D Morphable Mode!”), According to which the 3D surface corresponding to the three-dimensional reconstruction of a Any face can be obtained by deformation of an average face, the deformation being parameterized by a vector comprising a face number K of real values.
• the deep learning network 42 has been trained for this purpose to be able to predict, given a 2D image supplied as input, the set of K face parameters which deforms the average 3D face model so that 'it resembles as much as possible, photometrically, to the face of the 2D image supplied as input.
• the algorithm implemented by the network deep learning 42 implements a 3DMM type method suitable for deforming a generic 3D surface so as to approach the 2D image photometrically.
• the algorithm can be based on a photometric proximity metric between the deformed 3D model and the starting 2D image, in connection with an optimization process based on this metric.
• This network 42 is learned from 2D images of faces, the 3D surface of which is also known by a spatially precise means (for example a facial scanner with structured light).
• the learning network 42 is also suitable for predicting also an illumination model (represented by 9 parameters) and a pose (represented by 6 parameters), which make it possible to estimate the relative 3D position of the camera having taken the face as presented on the 2D image supplied as input.
• This pose estimation can be advantageously used in the case of using the method with several 2D images as input, which will be explained later.
• step 204 comprises the application of digital processing 54 to the data of the first part 22 of the image 21, which corresponds to the toothed portion 1 of the subject's face.
• This processing 54 comprises an enhancement of the first part 22 of the image 21 in order to modify the photometric characteristics of this first part.
• this enhancement aims to improve the contrast of the image 22.
• the processing 54 therefore makes it possible to generate an enhanced version 23 of the image 22 corresponding to the toothed portion of the face. Two embodiments of the enhancement will be described below, with reference to FIG. 4 and to FIG. 5, respectively.
• step 205 one carries out the implementation of another deep learning algorithm 41, adapted to produce a depth map (in the 2D domain) of the toothed portion 1 of the face on the basis of the 'enhanced image 23 corresponding to the first part 22 of the two-dimensional image 21.
• the deep learning algorithm 41 is adapted to predict a depth map for the toothed portion of the face from training data, by masking a depth map associated with the image 21 with the same mask as a mask used in image 21 to obtain, in step 202, the first part 22 of image 21 corresponding to toothed part 1 of the face. This depth map for the toothed portion 1 of the face is then converted into a 3D reconstruction.
• the deep learning algorithm 41 can implement a particular example of CNN, which is in fact an FCN (standing for “Fully Convolutional Network”) inspired by the article by J. Long, E. Shelhamer and T. Darrell, "Fully convolutional networks for semantic segmentation", IEEE
• Such a deep learning network is specifically trained to produce a depth map of the toothed part 1. It takes as input 2D images, the toothed portion 1 of which is isolated as described above in connection with step 202 (the rest of the image being masked and put in black) then enhanced by processing 54 as explained above. in connection with step 204. At output, deep learning network predicts the expected depth map on the toothed portion 1, generated from the learning data of the network by masking the global depth map with the same mask used on the 2D image in the enhancement step 204.
• Step 206 then comprises the implementation of an algorithm 56 for merging the three-dimensional reconstruction 23 of the toothed portion 1 and the three-dimensional textured reconstruction 34 of the facial portion 4 of the face represented by the two-dimensional image 21 , to obtain the three-dimensional reconstruction 35 of the complete face, with its toothed portion 1.
• the three-dimensional reconstruction 33 corresponding to the depth map produced by the algorithm 41 for the toothed portion 1 of the face is merged with the three-dimensional reconstruction 34 obtained by algorithm 42 for the facial portion outside the toothed portion of the face, in order to produce the three-dimensional surface 35 of the complete face.
• the fusion algorithm 56 can again implement a deep learning network.
• this network it is necessary to constitute a database, with data tuples acquired for different people, and which associate, for each 2D image of a person, the surface 3D of his face as well as the toothed portion.
• the 2D image of each person can be acquired by any commercial device (camera, cell phone, digital tablet, etc.).
• any commercial device camera, cell phone, digital tablet, etc.
• a 3D reconstruction of the facial portion 4 of the face excluding the toothed part it is possible to use a 3D scan of facial reconstruction using structured light.
• a 3D reconstruction can be obtained by an intraoral scanner in real colors (for example a WoW TM scanner available from the company BIOTECH DENTAL), thus producing a complete, textured and precise 3D dental arch.
• such a scanner can restore the texture of the teeth by amalgamating the colors of the 2D images (encoded by an RGB coding, for example) used for the 3D reconstruction. It is then easy to re-texture the 3D model using not the raw 2D images, but images enhanced by algorithm 54 of step 204 of the process.
• the 3D model then presents a much more contrasted surface and better suited to subsequent image processing algorithms based on photometry, which can ultimately be implemented in the context of the use made of facial reconstructions. which are obtained by virtue of the method of the invention, for example for the simulation of the aesthetic effect of a planned dental treatment.
• the 3D reconstruction of the part of the image corresponding to the toothed portion 1 of the face, enhanced or textured in RGB depending on the use to be made of it is readjusted manually on the 3D reconstruction of the facial part 4 of the face, in order to produce a single 3D reconstruction comprising the facial portion 4 and the toothed portion 1 of the face.
• the relative pose of the 2D image with respect to the 3D reconstruction can be calculated semi-automatically, by choosing 3D points of interest on the 3D surface as well as their corresponding point on the 2D image. Thanks to these pairs, a relative pose algorithm, for example ePNP, makes it possible to find the pose.
• triplet training data ⁇ 2D image; 3D reconstruction; pose ⁇ .
• This training data can easily be converted into other triples ⁇ 2D image; depth map; pose ⁇ , the depth map possibly being preferred in certain embodiments. Thanks to the deep learning network 56 trained as it has just been explained, the 3D surface of the face generated in step 206 of the method from the enhanced version 23 of the first part 22 of the 2D image , on the one hand, and of the second part 24 of said 2D image, on the other hand, is a good quality 3D reconstruction including for the toothed portion 1 of the face.
• This 3D reconstruction is therefore well suited for the simulation of a projected treatment to be applied to the toothed portion of the face, by substitution for the zone of the 3D surface corresponding to said toothed portion of another 3D surface corresponding to said toothed portion such as that it would be after said planned treatment.
• the enhanced image 23 corresponding to the part 22 of the input image
• the deep learning algorithm 41 which corresponds to the toothed portion 1 of the face, is used by the deep learning algorithm 41 to produce a three-dimensional reconstruction 33 of the toothed portion 1 of the face in image 21.
• the deep learning algorithm 42 which is for example based on a 3DMM method, generates a three-dimensional reconstruction 34 of the facial portion 4 alone.
• Such an algorithm for example, is advantageously suitable for, moreover, producing the relative 3D position of the camera having taken the face as presented in the 2D image, as well as an estimate of the 2D area of said 2D image in which is the toothed portion of the face.
• step 206 a consolidated 3D surface of the face from a plurality of 2D images of the face such as image 21, taken by a camera according to respective different viewing angles.
• Each of these images is subjected to the 3D reconstruction method according to steps 202 to 205 of FIG. 2.
• the implementation of the method of FIG. 2 can be repeated to obtain respective reconstructed 3D surfaces.
• These reconstructed 3D surfaces can then be combined, in step 206, by using the relative 3D position of the camera having taken the face as presented on each 2D image as well as the estimation of the 2D area of said 2D image in which is the toothed portion of the face.
• the consolidated 3D surface of the face which is obtained by this type of implementation from a plurality of 2D images of the subject's face is a more precise 3D reconstruction of the face and teeth than that obtained from a single 2D image of said face.
• the generation of the 3D surface of the face comprises the implementation of another deep learning algorithm 43 capable of predicting a 3D reconstruction from a 2D image, which differs from the deep learning algorithms 41 and 42 from the embodiment of FIG. 2.
• This another algorithm is suitable for producing an overall 3D reconstruction of the toothed portion 1 and of the facial portion 4 outside the toothed portion, from the second part 24 of the 2D image to which is added the first part 22 of said enhanced 2D image , with mutual registration of said second part of the 2D image and of said first part of said enhanced 2D image.
• This third algorithm 43 can be derived from the algorithm 42 used in step 203 of the implementation mode illustrated by FIG. 2.
• the first step 301 and the second step 302 of the implementation mode according to Figure 3 are identical, the first step 201 and the second step 202, respectively, of the implementation mode according to Figure 2. Further , the third step 303 of the embodiment of FIG. 3 corresponds to step 304 of the embodiment of FIG. 2.
• the first step 301 corresponds to taking a 2D image of the face d 'a patient with a visible toothed portion 1.
• the second step 302 is the step of segmenting the acquired 2D image, into a first part 22 corresponding to the toothed portion alone, and a second part
• the third step 303 comprises the enhancement processing of the part 22 of the image corresponding to the toothed part 1, which makes it possible to produce an enhanced version 23 of said image 22
• step 304 in fact, the enhanced image 23 which corresponds to the image 22 of the toothed portion alone on which a specific treatment has been applied to enhance the photometric characteristics, is reinjected into the original 2D image 21. More particularly, this result can be obtained by merging the enhanced image 23 and the part 24 of the original 2D image 21 corresponding to the facial part 4 except the toothed part 1 of the face, by a fusion algorithm 52. The result of this fusion is a reflective two-dimensional image 25, in which the toothed part 1 is enhanced. In other words, the image 25 produced by the fusion algorithm 52 is still a 2D image, like the original image 21, but it differs in that the toothed part 1 of the face is enhanced there.
• step 305 the facial reconstruction and that of the toothed portion are carried out by the common implementation of a three-dimensional reconstruction algorithm 43, applied to the reflective two-dimensional image 25 in which the toothed part 1 is raised.
• This algorithm can be derived from the algorithm 42 used in step 303 of the implementation of the method according to FIG. 2, but on the condition of adding the image of a toothed portion with enhanced texture in the training data.
• the algorithm implemented by the deep learning network 43 implements a 3DMM type method applied to the reflowed 2D image 25, and which is adapted to deform a generic 3D surface so as to approach the 2D image photometrically.
• algorithm 43 can be based on a photometric proximity metric between the deformed 3D model and the starting 2D image, in connection with an optimization process based on this metric .
• the total reconstructions (of an image with a facial part and with a toothed part) showing an enhanced texture on the teeth, are registered with each other.
• a restricted parameterization is then set up on these readjusted data in order to best account for the interindividual deformations.
• the deformations are parameterized by a total number K of deformation parameters which is greater than the number K face of parameters of the algorithm 42 of FIG. 2, accounting for both the face and the teeth.
• the modification of the photometric characteristics of the first part 22 of the original 2D image 21 which is generated in the enhancement step 204 of FIG. 2 as in step 303 of FIG. 3 comprises increasing the sharpness and / or increasing the contrast of said first part 22 of the 2D image.
• the enhancement of the toothed portion of the 2D image can be achieved using a series of purely photometric filters.
• enhancement treatment 54 at the 2D level which is applied to the toothed portion 1 comprises:
• step 401 extracting the blue channel from the color image coded in the format RGB;
• a high-pass contrast enhancement filtering 402 applied to the blue channel extracted in step 401; as well as
• step 403 a CLAHE-type histogram local equalization filtering applied to the filtered blue channel which is obtained by step 402.
• step 401 one skilled in the art will appreciate that the blue channel is, spectrally, the one which contains the most contrast on dental tissue.
• the high-pass contrast enhancement filtering applied in step 402 to the blue channel may include an algorithm for increasing the sharpness such as a "sharpening" algorithm applied to the blue channel.
• a “sharpening” algorithm applied to the blue channel.
• One such algorithm involves partially subtracting a blurred version of itself from the blue channel, which has the effect of emphasizing high spatial frequency details.
• the local histogram equalization filtering of step 403 can for example be of CLAHE type, as described in the chapter by Karel Zuiderveld
• the enhanced version 23 of the first part 22 of the original two-dimensional image 21 corresponding to the toothed portion of the image 21 can be obtained from said original 2D image 21, as an intermediate output of a depth map prediction deep learning network 50, having a higher contrast than the original 2D image according to a determined quantitative criterion.
• a depth map prediction deep learning network 50 having a higher contrast than the original 2D image according to a determined quantitative criterion.
• the deep learning architecture 50 is for example a convolutional neural network (CNN) which can have a completely conventional structure.
• CNN convolutional neural network
• This type of CNN is available in libraries known to those skilled in the art which are in free access.
• the two-dimensional image 21 is provided in the form of a matrix of pixels.
• the color is coded by a third dimension, from depth equal to 3, to represent the fundamental colors [Red, Green, Blue]
• the CNN of FIG. 5 is in fact an FCN (standing for “Fully Convolutional Network”) inspired by the scientific article already mentioned above, by J. Long, et al., “Fully convolutional networks for semantic segmentation ", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 3431- 3440.
• This FCN has two very distinct parts, according to an encoding / decoding architecture.
• the first part of the encoding FCN is the convolutional part itself. It comprises the “convolutional processing layer” 51, which has a succession of filters, or “convolution cores”, applied in layers.
• the convolutional processing layer 51 functions as an extractor of the characteristics of the 2D images admitted as input to the CNN.
• the input image 21 is passed through the succession of convolution nuclei, each time creating a new image called a convolution map.
• Each convolutional kernel has two convolutional layers 511 and 512, and a layer 513 for reducing the resolution of the image by a pooling operation also called a local maximum operation ("maxpooling").
• the output of the convolutional part 51 is then supplied as the input of a final convolutional layer 520 capturing the entire visual field of action of the preceding layer, and thus mimicking a fully connected layer (“fully connected” layer in English).
• a final deconvolution layer 530 outputs a 22 ’depth map.
• this type of CNN is unfortunately not suitable for 3D reconstruction of toothed part 1 in picture 22, due to the high specularity and low texture of the teeth. This is why the 22 ′′ depth map generated by this network 50 is not usable for the intended application.
• each convolutional kernel (or kernel) of the convolutional processing layer 51 of the network 50 is suitable for extracting determined photometric characteristics from the 2D image admitted as input to the CNN.
• each kernel generates a convolution map in the form of a new image constituting a version of the input image which is enhanced from the point of view of said characteristics.
• the enhanced image 23 corresponding to the enhanced version of the image 21 at the input of the deep learning network 50 can be extracted as a determined intermediate output of said network 50, having a higher contrast than the original 2D image according to a determined quantitative criterion.
• This intermediate output can be selected from among the outputs of the convolution kernels by a selection engine 52, on the basis of the values of a contrast metric which are respectively associated with the output of each of the convolution kernels of each convolution layer of the network 50.
• the selected intermediate output of network 50 may be the output of the core of the convolutional processing layer 51 of said network which exhibits maximum contrast with respect to the metrics associated with the respective intermediate outputs of the network, that is, that is to say at the outputs of the respective cores of the layer 51.
• the image delivered by this intermediate output has a higher contrast than the original 2D image 21 supplied at the input of the CNN.
• the invention which has been described in the foregoing makes it possible to make possible facial reconstruction with a toothed portion on the basis of any single 2D image, or on any series of 2D images. In the latter case, multiple images are taken from different viewing angles, and a final multi-view stereoscopic reconstruction procedure is conducted to produce a more accurate 3D reconstruction of the face and teeth.
• the method finds very varied applications, in particular in the simulation of dental treatments having aesthetic implications.
• the functional diagram of FIG. 6 illustrates an example of a method of simulating the aesthetic result of an aesthetic, orthodontic or prosthetic dental treatment, which is designed for a human subject, ie, a patient from at least one two-dimensional, 2D, color image of the subject's face with a visible toothed portion.
• the envisaged treatment is a cosmetic treatment consisting of teeth whitening.
• the method comprises:
• the three-dimensional reconstruction 75 of the patient's dental arch 1 which is obtained in step 61 may be a 3D reconstruction of the patient's full arch.
• This 3D reconstruction can for example be reconstructed by an intraoral 3D scanner (IOS) 72.
• the three-dimensional reconstruction 75 of the dental arch 1 of the patient can be obtained by volumetric imaging by conical beam (or CBCT, put for " Cone Beam Computed Tomography ”).
• CBCT is a computed tomography technique used to produce a digital x-ray, located between the dental panoramic and the scanner.
• a dental practitioner (such as a dental surgeon or an orthodontist, for example) develops a dental treatment plan 74.
• the dental arch 1 undergoes automatic or manual digital processing which generates a simulation 2 of said dental arch after treatment.
• the treated dental arch 2 here we can speak of the bleached dental arch
• the treated dental arch 2 is registered on the toothed portion of the three-dimensional surface 73 of the patient's face, in order to simulate within said 3D surface the aesthetic result of the patient. planned treatment 74.
• step 63 the three-dimensional surface 77 of the toothed portion 2 as it would appear after the planned dental treatment, here teeth whitening, is readjusted on the toothed portion of the three-dimensional reconstruction 73 of the patient's face, thanks to a registration algorithm 76.
• a registration algorithm 76 which is applied to a three-dimensional reconstruction 77 of the dental arch is adapted to register the bleached dental arch 2 on the toothed portion of the three-dimensional surface 73 of the patient's face as obtained by the method according to the first aspect of the invention.
• the display of the three-dimensional surface 73 of the face with the toothed portion 2 as it would be after the projected treatment can be a 3D display, for example in 3D software of the Meshlab TM type (which is a free software for processing 3D meshes), in CAD software (put for ("Computer Aided Design"). It can also be the display of a 2D image, or a display on glasses virtual reality glasses, or on augmented reality glasses These examples are not limiting.
• the planned treatment can include at least one of the following aesthetic, orthodontic or prosthetic treatments: a change in the color of the teeth, a realignment of the teeth, an affixing of veneers on the teeth, an installation of orthodontic material (rings) or prosthetic (crown, "bridge”, “inlay core”, “inlay onlay”), etc.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Computer Graphics (AREA)
• Software Systems (AREA)
• Architecture (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Geometry (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)
• Processing Or Creating Images (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=73038070

Family Applications (1)

Country Status (4)

Families Citing this family (6)

Family Cites Families (9)

• 2020
  • 2020-06-06 FR FR2005928A patent/FR3111066B1/fr active Active
• 2021
  • 2021-06-04 EP EP21730227.2A patent/EP4162448A1/de active Pending
  • 2021-06-04 US US18/008,653 patent/US12354229B2/en active Active
  • 2021-06-04 WO PCT/EP2021/065067 patent/WO2021245273A1/fr not_active Ceased

Also Published As

Similar Documents

Legal Events