WO2022042670A1 - 基于眼部状态检测的图像处理方法、装置及存储介质 - Google Patents

基于眼部状态检测的图像处理方法、装置及存储介质 Download PDF

Info

Publication number
WO2022042670A1
WO2022042670A1 PCT/CN2021/114881 CN2021114881W WO2022042670A1 WO 2022042670 A1 WO2022042670 A1 WO 2022042670A1 CN 2021114881 W CN2021114881 W CN 2021114881W WO 2022042670 A1 WO2022042670 A1 WO 2022042670A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
face
target
human eye
eye
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/114881
Other languages
English (en)
French (fr)
Inventor
徐旺
陈光辉
许译天
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Priority to EP21860504.6A priority Critical patent/EP4206975B1/en
Priority to JP2023513999A priority patent/JP7822369B2/ja
Publication of WO2022042670A1 publication Critical patent/WO2022042670A1/zh
Priority to US18/087,660 priority patent/US11842569B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/60—Creating or editing images; Combining images with text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/197—Matching; Classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to the field of image data processing, and in particular, to an image processing method, device, device and storage medium based on eye state detection.
  • the eye state of the person in the captured photo may be unsatisfactory (for example, "someone's eyes are closed"), resulting in a situation where the user needs to retake, or even retake repeatedly.
  • problems with unsatisfactory eye conditions such as “someone closed their eyes” and “someone is not looking at the camera", which leads to repeated retakes and affects the user's shooting experience.
  • the user usually manually selects a photo with ideal eye conditions for most people as the final group photo based on multiple photos obtained repeatedly.
  • the selected group photo to a certain extent, there is still the problem that the condition of people's eyes is not ideal, and it is impossible to show the best eye condition of each person during the shooting process in the group photo. Therefore, the user's satisfaction with the final group photo is reduced to a certain extent.
  • the present disclosure provides an image processing method, device, device and storage medium based on eye state detection, which can improve the eye state effect of each person in a group photo, The quality of the group photo is guaranteed, and the user's satisfaction with the final group photo is improved.
  • the present disclosure provides an image processing method based on eye state detection, the method comprising:
  • the image set to be processed includes consecutive multi-frame images, in which the multi-frame images are Each frame of image includes at least one face;
  • the target effect image corresponding to the target face is synthesized onto the reference image in the to-be-processed image set to obtain a target image corresponding to the to-be-processed image set.
  • the preset condition includes that the eye opening and closing degree value is greater than a preset opening and closing threshold value.
  • the eye state of the target face in the image set to be processed is detected, and the target area image whose eye state meets a preset condition is obtained, including:
  • the performing eye state detection on the face image of the target face includes:
  • the eye state of the target face in the image set to be processed is detected, and the target area image whose eye state meets a preset condition is obtained, including:
  • the performing eye state detection on the human eye image of the target face includes:
  • the eye state corresponding to the human eye image is determined.
  • the determining the position information of the key points of the human eye in the human eye image of the target face includes:
  • the performing eye state detection on the human eye image of the target face includes:
  • the human eye state value includes an eye-open state value and a closed-eye state value
  • the eye state corresponding to the human eye image is determined.
  • the determining the human eye state value in the human eye image of the target face includes:
  • the eye state corresponding to the human eye image is determined based on the ratio of the vertical opening width of the human eye to the distance between the two corners of the eye in the horizontal direction; the vertical opening width of the human eye is determined.
  • the proportional value of the distance between the two corners of the eye in the horizontal direction is determined based on the position information of the key points of the human eye.
  • the method before the eye state of the target face in the image set to be processed is detected, the method further includes:
  • a continuous multi-frame preview image including the current image frame and the current image frame as the ending frame is acquired as the image set to be processed.
  • the method further includes: :
  • the current image frame corresponding to pressing the shutter key in the to-be-processed image is determined as the reference image.
  • determining the target effect image corresponding to the target face based on the target area image whose eye state meets a preset condition includes:
  • the target area image with the largest eye opening and closing degree value is determined as the target effect image corresponding to the target face.
  • the face image of the target face is determined from the set of images to be processed, including:
  • a face image corresponding to the position information of the target face in the individual faces in the images in the image set to be processed is determined as the face image of the target face.
  • the face image of the target face is determined from the set of images to be processed, including:
  • a face image whose similarity is greater than the preset similarity threshold is determined as the face image of the target face.
  • the present disclosure provides an image processing device based on eye state detection, the device comprising:
  • the first detection module is used to detect the eye state of the target face in the image set to be processed, and obtain the target area image whose eye state meets the preset condition; wherein, the image set to be processed includes continuous multi-frame images , each frame of image in the multi-frame image includes at least one face;
  • a first determining module configured to determine a target effect image corresponding to the target face based on the target area image whose eye state meets a preset condition
  • the synthesis module is used for synthesizing the target effect image corresponding to the target face to the reference image in the set of images to be processed to obtain the target image corresponding to the set of images to be processed.
  • the present disclosure provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the terminal device is made to implement the above method.
  • the present disclosure provides a device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the computer program, Implement the above method.
  • the present disclosure provides an image processing method based on eye state detection.
  • the eye state of a target face in a set of images to be processed is detected to obtain a target area image whose eye state meets preset conditions.
  • the target effect image corresponding to the target face is determined from the target area image whose state meets the preset conditions, and finally the target effect image is synthesized into the reference image in the to-be-processed image set to obtain the target image corresponding to the to-be-processed image set.
  • the present disclosure can improve the eye state of each person in the final target image by detecting the eye state, determining the target effect image of each face, and then synthesizing the target effect image of each face into the reference image. The quality of the target image is improved, and the user's satisfaction with the target image is improved to a certain extent.
  • FIG. 1 is a flowchart of an image processing method based on eye state detection provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic diagram of a human eye image extraction provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of a human eye key point in a human eye image according to an embodiment of the present disclosure
  • FIG. 4 is a flowchart of another image processing method based on eye state detection provided by an embodiment of the present disclosure
  • FIG. 5 is a structural block diagram of an image processing apparatus based on eye state detection according to an embodiment of the present disclosure
  • FIG. 6 is a structural block diagram of an image processing device based on eye state detection according to an embodiment of the present disclosure.
  • the eye state of a person in an image is a factor in evaluating the quality of an image.
  • a group photo as an example, in the actual shooting scene, in order to show the best eye condition of each person during the shooting process in the group photo, multiple group photos are taken repeatedly and then retaken.
  • the ideal group photo is manually selected from multiple group photos.
  • the present disclosure provides an image processing method based on eye state detection.
  • the eye state of a target face in a set of images to be processed is detected to obtain a target area image whose eye state meets preset conditions, and then The target effect image corresponding to the target face is determined from the target area image whose eye state meets the preset conditions, and finally the target effect image is synthesized into the reference image in the set of images to be processed to obtain the corresponding image of the set of images to be processed. target image.
  • the image processing method based on eye state detection provided by the embodiment of the present disclosure can determine the target of each face in the group photo by detecting the eye state of the person in the group photo after the group photo is taken effect image, and then synthesize the target effect image of each face into the original group photo, so that the eye state of each person on the final group photo is better, the quality of the group photo is improved, and the user's perception of the group photo is improved. satisfaction.
  • an embodiment of the present disclosure provides an image processing method based on eye state detection.
  • FIG. 1 a flowchart of an image processing method based on eye state detection provided by an embodiment of the present disclosure, the method includes: :
  • S101 Detect the eye state of the target face in the image set to be processed, and obtain a target area image whose eye state meets a preset condition.
  • the preset condition includes that the eye opening and closing program value is greater than the preset opening and closing threshold value
  • the image set to be processed includes consecutive multi-frame images, and each frame of the multi-frame images includes at least one human face.
  • a continuous multi-point frame including the current image frame and the ending frame with the current image frame is acquired.
  • the frame preview image as a continuous multi-frame image, obtains the image set to be processed in the embodiment of the present disclosure.
  • the preview images in the camera preview interface are stored in the form of preview preview streams.
  • the shutter button when the shutter button is detected, not only the current image frame, that is, the photo taken by the camera, but also the latest N frame previews need to be obtained from the preview images of the stored preview preview stream. image. Then, the latest N frames of preview images and the current image frame together form a to-be-processed image set.
  • the image set to be processed includes 8 frames of images or 16 frames of images, and the embodiment of the present disclosure does not limit the number of images in the image set to be processed. In other embodiments, the set of images to be processed may further include more frames of images.
  • the current mode is the continuous shooting mode
  • a trigger operation of pressing the shutter key is detected
  • multiple frames of images obtained by continuous shooting are acquired as continuous shooting. Multiple frames of images are obtained to obtain the image set to be processed in the embodiment of the present disclosure.
  • the eye state of the target face in the to-be-processed image set is detected.
  • the target face may be a face corresponding to the same person among the multiple faces in the images in the image set to be processed.
  • detecting the eye state of the target face in the set of images to be processed may include: determining the face image of the target face from the set of images to be processed, and then determining the face image of the target face from the set of images to be processed. Perform eye state detection, and obtain a face image whose eye state meets a preset condition in the face image of the target face as a target area image.
  • the embodiments of the present disclosure provide at least the following two methods for determining a face image of a target face from a set of images to be processed, which are respectively introduced below:
  • face detection is performed on the reference image in the image set to be processed, and the position information of each face on the reference image is determined. Then, according to the position information of each face, a face image corresponding to the position information of the target face in each face in the images in the image set to be processed is determined as the face image of the target face.
  • the current image frame corresponding to pressing the shutter button is usually an image in which most people's eyes are in good condition in this shooting. Therefore, in this embodiment of the present disclosure, the current image frame corresponding to pressing the shutter key in the image set to be processed may be determined as the reference image. In this way, on the basis of the reference image, the position information of each face is determined, and then the face image corresponding to the target face is further determined based on the position information of each face, which can improve the accuracy of the face image corresponding to the target face. sex.
  • face detection may be performed on the reference image based on the machine learning model to determine the position information of each face on the reference image. Because the position information of each face on the multi-frame images continuously shot in one shooting process is basically the same. Therefore, based on the position information of each face determined on the reference image, the face image corresponding to the target face on other images in the image set to be processed can be further determined. It can be understood that the face images at the same position on each image in the image set to be processed belong to the face images of the same person.
  • the face image of the target face may be the smallest rectangular area including the target face.
  • the minimum rectangular area including the target face can be determined based on the position information of the target face.
  • the face image of the target face can also be determined from the set of images to be processed by combining face detection and similarity calculation. Specifically, face detection is performed on each image in the image set to be processed to obtain a face image. Then, a face image with a similarity greater than a preset similarity threshold is determined as a face image of the target face.
  • the embodiment of the present disclosure may determine the target face based on the similarity of the face images after determining the face images on each image in the image set to be processed. face image. Specifically, a face image with a similarity greater than a preset similarity threshold is determined as the face image of the target face.
  • an embodiment of the present disclosure in the process of performing eye state detection on a face image, first extract a human eye image from the human face image, and then perform eye state detection on the human eye image to complete the eye state detection of the corresponding human face image. External status detection.
  • An embodiment of the present disclosure provides a method for detecting an eye state of a human eye image, which will be introduced later.
  • the human eye image of the target face in the process of detecting the eye state of the target face in the image set to be processed, can be determined from the image set to be processed, and then the human eye image of the target face can be determined.
  • the image is subjected to eye state detection, and a human eye image whose eye state meets a preset condition in the human eye image of the target face is obtained as the target area image.
  • a machine learning model can be used to perform human eye detection on the reference image in the atlas to be processed, so as to determine the position information of the human eye on the reference image. Then, based on the position information of the human eye determined on the reference image, the human eye image corresponding to the human eye on each image in the image set to be processed can be further determined. It should be noted that the human eye images at the same position on each image in the image set to be processed belong to the human eye images of the same person.
  • the human eye image may be the smallest rectangular area including the human eye.
  • the human eye image may be the smallest rectangular area including the left eye, the smallest rectangular area including the right eye, or the smallest rectangular area including both the left eye and the right eye.
  • the human eye image of the target face can also be determined from the set of images to be processed by combining human eye detection and similarity calculation. Specifically, human eye detection is performed on each image in the image set to be processed to obtain a human eye image. Then, a human eye image with a similarity greater than a preset similarity threshold is determined as a human eye image of the target face.
  • a target area image in which the eye state of each face meets a preset condition is obtained.
  • the target area image may be a face image or a human eye image.
  • the eye image corresponding to the human eye image may be determined based on the position information of the human eye key point or the human eye state value in the human eye image, or the combination of the position information of the human eye key point and the human eye state value. Ministry status.
  • the embodiment of the present disclosure provides a specific implementation manner of determining the eye state corresponding to the human eye image, which will be introduced later.
  • S102 Determine a target effect image corresponding to the target face based on the target area image whose eye state meets a preset condition.
  • the embodiment of the present disclosure first determines the target area image in which the eye state of the target face meets the preset condition, and then further determines the target effect image corresponding to each face based on the determined target area image.
  • that the eye state meets the preset condition may refer to that the value of the degree of eye opening and closing is greater than the preset opening and closing threshold.
  • the target area image with the largest eye opening and closing degree value in the target area image of the target face can be determined as the target effect image corresponding to the target face, so as to improve the performance of each person in the target image.
  • the degree of human eye openness of the face thereby improving the user's satisfaction with the target image.
  • any target area image in the target area images of the target face can be determined as the target effect image corresponding to the target face, so as to satisfy the user's expectation of the face in the target image.
  • the first face may not be synthesized, so as to Improve the efficiency of image processing.
  • S103 Synthesize the target effect image corresponding to the target face to the reference image in the set of images to be processed to obtain a target image corresponding to the set of images to be processed.
  • the target effect image is synthesized into the reference image in the to-be-processed image set, and then the target image corresponding to the to-be-processed image set is obtained.
  • the target image Since the target image is obtained based on the target effect image. Therefore, the target image can maximize the effect of each person's eye state on the image, thereby improving the quality of the target image and improving the user's satisfaction with the target image to a certain extent.
  • the target effect image corresponding to each face has position information, and the target effect image is synthesized to a corresponding position on the reference image based on the position information of the target effect image.
  • any image in the set of images to be processed may also be determined as the reference image.
  • the embodiments of the present disclosure do not specifically limit the manner of determining the reference image, and those skilled in the art can select according to actual needs.
  • the eye state of the target face in the image set to be processed is first detected, and the target area image whose eye state meets the preset condition is obtained, and then the eye state is obtained from the eye state.
  • the target effect image corresponding to the target face is determined from the target area image whose external state meets the preset conditions, and finally, the target effect image is synthesized into the reference image in the image set to be processed, and the target image corresponding to the image set to be processed is obtained.
  • the embodiments of the present disclosure detect the eye state, determine the target effect image of each face, and then synthesize the target effect image of each face into the reference image, which can improve the eye state effect of each person in the target image. The quality of the target image is improved, and the user's satisfaction with the target image is improved to a certain extent.
  • the eye state corresponding to the human eye image may be determined based on the position information of the key points of the human eye.
  • corresponding human eye images are extracted from 8 frames of human face images respectively. Then, for each human eye image, the position information of the human eye key points in the human eye image is determined, and then the eye state corresponding to the human eye image is determined based on the position information of the human eye key points.
  • the eye state corresponding to the human eye image of the target face is used as the eye state of the human face image corresponding to the target face, wherein the eye state can be represented by an eye opening and closing degree value.
  • the key points of the human eye may be the left corner key point 1 of the eye, the key points 2 and 3 on the upper eyelid, the right eye corner key point 4, and the key points 5 and 6 on the lower eyelid.
  • the value of the degree of eye opening and closing is determined based on the position information of each key point of the human eye.
  • the distance between the key point 1 and the key point 4 in FIG. 3 can be used as the distance between the two corners of the eye in the horizontal direction, and the distance between the key point 2 and the key point 6 and the distance between the key point 3 and the key point 3 The average value of the distance of 5, as the vertical opening width of the human eye. Then, the ratio of the vertical opening width of the human eye to the distance between the two corners of the eye in the horizontal direction is determined as the eye opening and closing degree value.
  • the location information of the key points of the human eye may be determined by using a machine learning model.
  • the first model is trained by using the human eye image samples marked with the position information of the key points of the human eye, the human eye image is input into the trained first model, and after processing by the first model, the human eye image is output The location information of the human eye key points in the image.
  • the eye state corresponding to the human eye image may also be determined based on the human eye state value.
  • the human eye state value includes an eye-opening state value and an eye-closing state value.
  • the human eye state value can be a value in the range of [0,1]. The larger the value of the human eye state value, the greater the value of the degree of eye opening and closing; The smaller the value of the degree of partial opening and closing.
  • the eye-closed state value may be a value in the range of [0, 0.5), and the eye-open state value may be a value in the range of [0.5, 1]; in other embodiments, the eye-closed state value may be [0 , 0.5], and the eye-opening state value can be a value in the range of (0.5, 1].
  • the state value of the human eye may be determined by using a machine learning model.
  • the second model is trained by using human eye image samples marked with human eye state values, the human eye image is input into the trained second model, and after processing by the second model, the human eye image is output. Human eye state value.
  • the eye state corresponding to the human eye image can be determined by the human eye state value, and the target area image with the largest human eye state value of the target face is determined as the target effect image.
  • the embodiment of the present disclosure can combine the position information of the key points of the human eye and the human eye state value to determine the eye opening and closing degree value corresponding to the face image, thereby improving the determination based on the eye opening and closing degree value.
  • the accuracy of the obtained target effect image thereby improving the quality of the target image.
  • An embodiment of the present disclosure provides an image processing method based on eye state detection.
  • FIG. 4 a flowchart of another image processing method based on eye state detection provided by an embodiment of the present disclosure, the method includes:
  • S401 Determine a face image belonging to the target face based on the image set to be processed.
  • the set of images to be processed includes continuous multiple frames of preview images with the current image frame corresponding to pressing the shutter key as the ending frame.
  • an eye image belonging to the target face is extracted.
  • the eyes are detected on the face image, the position information of the eyes in the face image is determined, and then based on the position information of the eyes, a rectangular frame area containing the eyes is determined, and the rectangle is The box area is extracted from the face image.
  • the image corresponding to the extracted rectangular area is taken as a human eye image. Among them, the method of eye detection will not be explained too much.
  • the human eye image extracted by the embodiment of the present disclosure may only include one of the eyes in the human face image, thereby improving the efficiency of image processing.
  • S403 Determine the state value of the human eye and the position information of the key point of the human eye in the human eye image.
  • the human eye state value and the position information of the human eye key point in the human eye image are determined.
  • a machine learning model may be used to determine the state value of the human eye and the position information of key points of the human eye.
  • the third model is trained by using the human eye image samples based on the position information of the marked human eye key points and the human eye state value, the human eye image is input into the trained third model, and processed by the third model After that, output the human eye state value of the human eye image and the position information of human eye key points.
  • S404 Determine, based on the human eye state value and the position information of the human eye key point, an eye opening and closing degree value corresponding to the human face image.
  • the human eye on the human face image may be determined based on the position information of the human eye key point The ratio of the vertical opening width to the horizontal distance between the two corners of the eyes. Then, a value of the degree of eye opening and closing corresponding to the face image is determined by combining the ratio value of the vertical opening width of the human eye to the distance between the two corners of the eye in the horizontal direction and the human eye state value.
  • formula (1) can be used to calculate the value of the degree of eye opening and closing corresponding to the face image, wherein formula (1) is as follows:
  • Open_Degree(OD) is used to represent the eye opening and closing degree value corresponding to the face image
  • H_d is used to represent the Euclidean distance between key point 1 and key point 4 in Figure 3
  • V_d_1 is used to represent the key point in Figure 3.
  • the Euclidean distance between point 2 and key point 6, V_d_2 is used to represent the Euclidean distance between key point 3 and key point 5 in Figure 3
  • round() is used to indicate that the parameters are rounded to the nearest integer
  • Eye_State is used to indicate Human eye state value, between [0,1].
  • S405 From the face images belonging to the target face, determine a face image whose eye opening and closing degree value is greater than a preset opening and closing threshold.
  • the face image in the closed eye state is eliminated. Then, based on the eye opening and closing degree value, the face images whose eye opening and closing degree value is lower than or equal to the preset opening and closing threshold are eliminated. It is also possible to sort the remaining face images according to the eye opening and closing degree values, and determine the face images whose opening and closing degree values are greater than the preset opening and closing threshold.
  • the corresponding face image on the reference image may not be displayed.
  • the face image is processed, and the effect of the face in the reference image is preserved.
  • S406 Determine a target effect image corresponding to the target face from the face images whose eye opening and closing degree value is greater than a preset opening and closing threshold.
  • the method can randomly select a face image whose eye opening and closing degree value is greater than the preset opening and closing threshold from the face images corresponding to the target face, and use the randomly selected face image as the target corresponding to the target face. Effect image to improve the effect of the eye state on the target face in the target image. Similar processing can be performed for face images corresponding to other faces, thereby obtaining target effect images corresponding to other faces.
  • the embodiment of the present disclosure selects the face image with the largest eye opening and closing degree value as the target effect image corresponding to the target face from the face images whose eye opening and closing degree value is greater than the preset opening and closing threshold value, so as to maximize the improvement of The effect of the eye state on the target face in the target image.
  • S407 Synthesize the target effect image corresponding to the target face onto the reference image in the to-be-processed image set to obtain a target image corresponding to the to-be-processed image set.
  • each target effect image is synthesized into the reference image, and finally the target image corresponding to the image set to be processed is obtained.
  • a group photo can be taken only once, and the eye state of as many people as possible in the group photo can be better, without the need for Repeated retakes improve the user's group photo shooting experience, and can also provide users with group photos with high satisfaction.
  • an image processing device based on eye state detection provided by an embodiment of the present disclosure , the device includes:
  • the first detection module 501 is used to detect the eye state of the target face in the image set to be processed, and obtain the target area image whose eye state meets the preset condition; wherein, the image set to be processed includes consecutive multiple frames an image, each of the multiple frames of images includes at least one human face;
  • a first determination module 502 configured to determine a target effect image corresponding to the target face based on the target area image whose eye state meets a preset condition
  • the synthesizing module 503 is configured to synthesize the target effect image corresponding to the target face onto the preset reference image in the set of images to be processed to obtain a target image corresponding to the set of images to be processed.
  • the preset condition includes that the eye opening and closing degree value is greater than a preset opening and closing threshold value.
  • the first detection module includes:
  • the first determination submodule is used to determine the face image of the target face from the set of images to be processed
  • the first detection submodule is used to detect the eye state of the face image of the target face, and obtain the face image whose eye state meets the preset condition in the face image of the target face, as the target area image.
  • the first detection sub-module includes:
  • an extraction submodule for extracting the human eye image of the target face from the face image of the target face
  • the second detection sub-module is used for performing eye state detection on the human eye image of the target face.
  • the first detection module includes:
  • the second determination submodule is used to determine the human eye image of the target face from the set of images to be processed
  • the third detection sub-module is configured to perform eye state detection on the human eye image of the target face, and obtain the human eye image whose eye state meets the preset condition in the human eye image of the target face, as the target area image.
  • the second detection module or the third detection sub-module includes:
  • the third determination submodule is used to determine the position information of the key points of the human eye in the human eye image of the target face;
  • the fourth determination sub-module is configured to determine the eye state corresponding to the human eye image based on the position information of the human eye key point.
  • the third determination sub-module is specifically used for:
  • the second detection module or the third detection sub-module includes:
  • a fifth determination submodule configured to determine the human eye state value in the human eye image of the target face; wherein the human eye state value includes an eye-open state value and a closed-eye state value;
  • the sixth determination sub-module is configured to determine the eye state corresponding to the human eye image based on the human eye state value.
  • the fifth determination sub-module is specifically used for:
  • the eye state corresponding to the human eye image is determined based on the ratio of the vertical opening width of the human eye to the distance between the two corners of the eye in the horizontal direction; the vertical opening width of the human eye is determined.
  • the proportional value of the distance from the two corners of the eye in the horizontal direction is determined based on the position information of the key points of the human eye.
  • the device further includes:
  • the acquisition module is configured to acquire, according to the triggering operation of the shutter key, a current image frame and a continuous multi-frame preview image with the current image frame as an end frame, as a set of images to be processed.
  • the device further includes:
  • the second determination module is configured to determine the current image frame corresponding to the pressing of the shutter key in the to-be-processed image as the reference image.
  • the first determining module is specifically used for:
  • the target area image with the largest eye opening and closing degree value is determined as the target effect image corresponding to the target face.
  • the first determination submodule includes:
  • the 7th determines submodule is used for performing face detection on the reference image in the image set to be processed, and determines the position information of each face on the reference image;
  • the eighth determination sub-module is used for determining, according to the position information of each face, the face image corresponding to the position information of the target face in the said face in the images in the set of images to be processed, as the A face image of a human face.
  • the first determination submodule includes:
  • the fourth detection submodule is used to perform face detection on each image in the to-be-processed image set to obtain a face image
  • the ninth determination sub-module is used for determining a face image whose similarity is greater than a preset similarity threshold as the face image of the target face.
  • the eye state of a target face in a set of images to be processed is detected, and an image of a target area in which the eye state of the target face meets a preset condition is obtained, Then, the target effect image corresponding to the face is determined from the target area image whose eye state meets the preset conditions, and finally the target effect image is synthesized into the reference image in the image set to be processed to obtain the target corresponding to the image set to be processed. image.
  • the eye state by detecting the eye state, determining the target effect image of each face, and then synthesizing the target effect image of each face into the reference image, it is possible to improve the eyesight of each person in the final target image. It can improve the quality of the target image and improve the user's satisfaction with the target image to a certain extent.
  • an embodiment of the present disclosure also provides an image processing device based on eye state detection, as shown in FIG. 6 , which may include:
  • Processor 601 , memory 602 , input device 603 and output device 604 The number of processors 601 in the image processing device based on eye state detection may be one or more, and one processor is taken as an example in FIG. 6 .
  • the processor 601 , the memory 602 , the input device 603 and the output device 604 may be connected by a bus or in other ways, wherein the connection by a bus is taken as an example in FIG. 6 .
  • the memory 602 can be used to store software programs and modules, and the processor 601 executes various functional applications and data processing of the image processing device based on eye state detection by running the software programs and modules stored in the memory 602 .
  • the memory 602 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, and the like. Additionally, memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
  • the input device 603 may be used to receive input numerical or character information, and generate signal input related to user settings and function control of the image processing apparatus based on eye state detection.
  • the processor 601 loads the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 executes the executable files stored in the memory 602
  • the application program can realize various functions of the above-mentioned image processing device based on eye state detection.
  • Embodiments of the present disclosure further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the terminal device is made to implement the foregoing method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Ophthalmology & Optometry (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

本公开提供了一种基于眼部状态检测的图像处理方法、装置、设备及存储介质，所述方法包括：对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，然后从中确定出该目标人脸对应的目标效果图像，最终将该目标效果图像合成到待处理图像集中的基准图像上，得到待处理图像集对应的目标图像。本公开基于眼部状态检测确定每张人脸的目标效果图像，然后将各个目标效果图像合成到基准图像上，能够提高目标图像中每张人脸眼部状态的效果，保证目标图像的质量，提高用户对目标图像的满意度。

Description

基于眼部状态检测的图像处理方法、装置及存储介质
本申请要求于2020年08月31日提交国家知识产权局、申请号为202010899317.9、申请名称为“基于眼部状态检测的图像处理方法、装置及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及图片数据处理领域，尤其涉及一种基于眼部状态检测的图像处理方法、装置、设备及存储介质。
背景技术
在拍摄照片时，拍摄得到的照片中人的眼部状态会出现不理想(例如“有人闭眼”)的问题，导致用户需要重拍，甚至反复重拍的情况。尤其是在多人拍摄合影时，更容易出现“有人闭眼”、“有人未注视镜头”等眼部状态不理想的问题，进而导致的反复重拍的情况，影响用户的拍摄体验。
目前，通常是用户基于反复拍摄得到的多张照片，人工选择出大多数人的眼部状态较为理想的照片作为最终的合影照片。而在选择出的合影照片中，一定程度上还会存在人的眼部状态不理想的问题，无法在合影照片中展现出本次拍摄过程中每个人最佳的眼部状态。因此，一定程度上降低了用户对最终的合影照片的满意度。
发明内容
为了解决上述技术问题或者至少部分地解决上述技术问题，本公开提供了一种基于眼部状态检测的图像处理方法、装置、设备及存储介质，能够改善合影照片中每个人的眼部状态效果，保证了合影照片的质量，提高了用户对最终的合影照片的满意度。
第一方面，本公开提供了一种基于眼部状态检测的图像处理方法，所述方法包括：
对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像；其中，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸；
基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像；
将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
一种可选的实施方式中，所述预设条件包括眼睛开合程度值大于预设开合阈值。
一种可选的实施方式中，所述对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，包括：
从待处理图像集中确定目标人脸的人脸图像；
对所述目标人脸的人脸图像进行眼部状态检测，得到所述目标人脸的人脸图像中眼部状态符合预设条件的人脸图像，作为目标区域图像。
一种可选的实施方式中，所述对所述目标人脸的人脸图像进行眼部状态检测，包括：
从所述目标人脸的人脸图像中提取所述目标人脸的人眼图像；
对所述目标人脸的人眼图像进行眼部状态检测。
一种可选的实施方式中，所述对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，包括：
从待处理图像集中确定目标人脸的人眼图像；
对所述目标人脸的人眼图像进行眼部状态检测，得到所述目标人脸的人眼图像中眼部状态符合预设条件的人眼图像，作为目标区域图像。
一种可选的实施方式中，所述对所述目标人脸的人眼图像进行眼部状态检测，包括：
确定所述目标人脸的人眼图像中的人眼关键点的位置信息；
基于所述人眼关键点的位置信息，确定所述人眼图像对应的眼部状态。
一种可选的实施方式中，所述确定所述目标人脸的人眼图像中的人眼关键点的位置信息，包括：
将所述目标人脸的人眼图像输入到第一模型中，得到所述人眼图像中的人眼关键点的位置信息；其中，所述第一模型是基于标记有人眼关键点的位置信息的人眼图像样本训练得到。
一种可选的实施方式中，所述对所述目标人脸的人眼图像进行眼部状态检测，包括：
确定所述目标人脸的人眼图像中的人眼状态值；其中，所述人眼状态值包括睁眼状态值和闭眼状态值；
基于所述人眼状态值，确定所述人眼图像对应的眼部状态。
一种可选的实施方式中，所述确定所述目标人脸的人眼图像中的人眼状态值，包括：
将所述目标人脸的人眼图像输入至第二模型中，得到所述人眼图像中的人眼状态值；其中，所述第二模型是基于标记有人眼状态值的人眼图像样本训练得到。
一种可选的实施方式中，所述人眼图像对应的眼部状态基于人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值确定；所述人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值是基于所述人眼关键点的位置信息确定。
一种可选的实施方式中，所述对待处理图像集中目标人脸的眼部状态进行检测之前，还包括：
根据对快门键的触发操作，获取包括当前图像帧和以所述当前图像帧为结束帧的连续多帧预览图像，作为待处理图像集。
一种可选的实施方式中，所述将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像之前，还包括：
将所述待处理图像集中按下快门键对应的当前图像帧，确定为基准图像。
一种可选的实施方式中，所述基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像，包括：
将所述目标人脸的眼部状态符合预设条件的目标区域图像中，眼睛开合程度值最大的目标区域图像，确定为所述目标人脸对应的目标效果图像。
一种可选的实施方式中，所述从待处理图像集中确定目标人脸的人脸图像，包括：
对待处理图像集中的基准图像进行人脸检测，确定所述基准图像上各个人脸的位置信息；
根据所述人脸的位置信息，确定所述待处理图像集中的图像上与所述各个人脸中目标人脸的位置信息对应的人脸图像，作为所述目标人脸的人脸图像。
一种可选的实施方式中，所述从待处理图像集中确定目标人脸的人脸图像，包括：
对待处理图像集中的每张图像进行人脸检测，得到人脸图像；
将相似度大于预设相似阈值的人脸图像，确定为目标人脸的人脸图像。
第二方面，本公开提供了一种基于眼部状态检测的图像处理装置，所述装置包括：
第一检测模块，用于对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像；其中，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸；
第一确定模块，用于基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像；
合成模块，用于将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
第三方面，本公开提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当所述指令在终端设备上运行时，使得所述终端设备实现上述的方法。
第四方面，本公开提供了一种设备，包括：存储器，处理器，及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时，实现上述的方法。
本公开提供的技术方案与现有技术相比具有如下优点：
本公开提供了一种基于眼部状态检测的图像处理方法，首先对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，然后从眼部状态符合预设条件的目标区域图像中确定出该目标人脸对应的目标效果图像，最终将该目标效果图像合成到待处理图像集中的基准图像上，得到待处理图像集对应的目标图像。本公开通过对眼部状态进行检测，确定每张人脸的目标效果图像，然后将每张人脸的目标效果图像合成到基准图像中，能够提高最终得到的目标图像中每个人的眼部状态效果，提高了目标图像的质量，一定程度上提高了用户对目标图像的满意度。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种基于眼部状态检测的图像处理方法的流程图；
图2为本公开实施例提供的一种人眼图像提取的示意图；
图3为本公开实施例提供的一种人眼图像中的人眼关键点的示意图；
图4为本公开实施例提供的另一种基于眼部状态检测的图像处理方法的流程图；
图5为本公开实施例提供的一种基于眼部状态检测的图像处理装置结构框图；
图6为本公开实施例提供的一种基于眼部状态检测的图像处理设备结构框图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
图像(例如合影照片)中人的眼部状态是评价一张图像质量的因素。以图像为合影照片为例，实际拍摄场景中，为了在合影照片中展示出本次拍摄过程中每个人最佳的眼部状态，通过反复多次重拍的方式拍摄多张合影照片，然后从多张合影照片中人工选择出理想的合影照片。
上述反复多次重拍的方式，不仅会降低人们的拍照体验，而且也不能保证重拍的合影照片中每个人的眼部状态较为理想，影响用户对合影照片的满意度。
为此，本公开提供了一种基于眼部状态检测的图像处理方法，首先对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，然后从该眼部状态符合预设条件的目标区域图像中确定出该目标人脸对应的目标效果图像，最终将该目标效果图像合成到待处理图像集中的基准图像上，得到待处理图像集对应的目标图像。
基于上述拍照场景，本公开实施例提供的基于眼部状态检测的图像处理方法能够在拍摄合影照片之后，通过对合影照片中人的眼部状态进行检测，确定合影照片中每张人脸的目标效果图像，然后将该每张人脸的目标效果图像合成到原合影照片中，使得最终的合影照片上每个人的眼部状态效果较好，提高了合影照片的质量，提高了用户对合影照片的满意度。
基于此，本公开实施例提供了一种基于眼部状态检测的图像处理方法，参考图1，为本公开实施例提供的一种基于眼部状态检测的图像处理方法的流程图，该方法包括：
S101：对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像。
其中，该预设条件包括眼睛开合程序值大于预设开合阈值，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸。
一种可选的实施方式中，在拍摄照片的场景下，当检测到对快门键的触发操作(例如按下快门键)时，获取包括当前图像帧和以当前图像帧为结束帧的连续多帧预览图像，作为连续的多帧图像，得到本公开实施例中的待处理图像集。
实际应用中，相机预览模式下，在相机预览界面中的预览图像是以preview预览流的形式进行存储的。本公开实施例在相机预览模式下，检测到按下快门键时，不仅获取当前图像帧，即相机拍摄的照片，还需要从已存储的preview预览流的预览图片中，获取最新的N帧预览图像。然后，将该最新的N帧预览图像与当前图像帧共同构成待处理图像集。通常待处理图像集中包括8帧图像或者16帧图像，本公开实施例不限制待处理图像集中的图像的个数。在另一些实施例中，待处理图像集中还可以包括更多帧图像。
另一种可选的实施方式中，在拍摄照片的场景下，如果当前模式为连拍模式，则在检测到按下快门键的触发操作时，获取连拍得到的多帧图像，作为连续的多帧图像，得到本公开实施例中的待处理图像集。
本公开实施例中，在获取到待处理图像集之后，对待处理图像集中目标人脸的眼部状态进行检测。目标人脸可以是待处理图像集中图像上多张人脸中的同一个人对应的人脸。
一种可选的实施方式中，对待处理图像集中目标人脸的眼部状态进行检测，可以包括：从待处理图像集中确定目标人脸的人脸图像，然后对该目标人脸的人脸图像进行眼部状态检测，得到该目标人脸的人脸图像中眼部状态符合预设条件的人脸图像，作为目标区域图像。
本公开实施例至少提供以下两种从待处理图像集中确定目标人脸的人脸图像的方法，以下分别介绍：
一种可选的实施方式中，对待处理图像集中的基准图像进行人脸检测，确定基准图像上各个人脸的位置信息。然后根据各个人脸的位置信息，确定待处理图像集中的图像上与该各个人脸中目标人脸的位置信息对应的人脸图像，作为该目标人脸的人脸图像。
实际应用中，由于在一次拍摄过程中，按下快门键对应的当前图像帧通常为本次拍摄中大多数人的眼部状态较佳的图像。因此，本公开实施例可以将待处理图像集中的按下快门键对应的当前图像帧，确定为基准图像。如此在该基准图像的基础上，确定各个人脸的位置信息，然后基于该各个人脸的位置信息进一步确定目标人脸对应的人脸图像，能够提高该目标人脸对应的人脸图像的准确性。
本公开实施例中，在确定待处理图像集中的基准图像之后，可以基于机器学习模型对基准图像进行人脸检测，以确定基准图像上每张人脸的位置信息。由于在一次拍摄过程中连拍的多帧图像上各个人脸的位置信息基本相同。因此，可以基于基准图像上确定的每张人脸的位置信息，进一步确定待处理图像集中其他图像上的目标人脸对应的人脸图像。可以理解的是，在待处理图像集中每张图像上的同一个位置的人脸图像属于同一个人的人脸图像。
其中，目标人脸的人脸图像可以为包括该目标人脸的最小矩形区域。该包括目标人脸的最小矩形区域可以基于目标人脸的位置信息确定。
另一种可选的实施方式中，还可以结合人脸检测和相似度计算的方式，从待处理 图像集中确定目标人脸的人脸图像。具体的，对待处理图像集中的各个图像进行人脸检测，得到人脸图像。然后将相似度大于预设相似阈值的人脸图像，确定为目标人脸的人脸图像。
实际应用中，由于目标人脸的人脸图像的相似度较高，因此本公开实施例在确定待处理图像集中各个图像上的人脸图像之后，可以基于人脸图像的相似度确定目标人脸的人脸图像。具体的，将相似度大于预设相似阈值的人脸图像，确定为目标人脸的人脸图像。
本公开实施例中，在对人脸图像进行眼部状态检测的过程中，首先从人脸图像中提取人眼图像，然后对人眼图像进行眼部状态检测，完成对应的人脸图像的眼部状态检测。本公开实施例提供了一种对人眼图像进行眼部状态检测的方法，后续进行介绍。
在另一种可选实施方式中，对待处理图像集中目标人脸的眼部状态进行检测的过程中，可以从待处理图像集中确定目标人脸的人眼图像，然后对目标人脸的人眼图像进行眼部状态检测，得到目标人脸的人眼图像中眼部状态符合预设条件的人眼图像，作为目标区域图像。
实际应用中，可以利用机器学习模型对待处理图集中的基准图像进行人眼检测，以确定基准图像上的人眼的位置信息。然后可以基于基准图像上确定的人眼的位置信息，进一步确定待处理图像集中各个图像上的人眼对应的人眼图像。需要说明的是，在待处理图像集中每张图像上的同一个位置的人眼图像属于同一人的人眼图像。
其中，人眼图像可以为包括人眼的最小矩形区域。具体的，人眼图像可以为包括左眼的最小矩形区域，也可以为包括右眼的最小矩形区域，还可以为同时包括左眼和右眼的最小矩形区域。
另一种可选的实施方式中，还可以结合人眼检测和相似度计算的方式，从待处理图像集中确定目标人脸的人眼图像。具体的，对待处理图像集中的各个图像进行人眼检测，得到人眼图像。然后，将相似度大于预设相似阈值的人眼图像，确定为目标人脸的人眼图像。
本公开实施例中，在对待处理图像集中目标人脸的眼部状态进行检测之后，得到每张人脸的眼部状态符合预设条件的目标区域图像。其中，目标区域图像可以是人脸图像，也可以是人眼图像。
本公开实施例中，可以基于人眼图像中的人眼关键点的位置信息或者人眼状态值，或者人眼关键点的位置信息和人眼状态值的结合，确定该人眼图像对应的眼部状态。本公开实施例提供了一种确定人眼图像对应的眼部状态的具体实现方式，后续进行介绍。
S102：基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像。
实际应用中，理想的照片中每个人的眼部状态通常是睁眼状态，同时眼睛睁开的程度应该符合一定标准。因此，本公开实施例首先确定目标人脸的眼部状态符合预设条件的目标区域图像，然后基于确定的目标区域图像，进一步确定每张人脸对应的目 标效果图像。其中，眼部状态符合预设条件可以指，眼睛开合程度值大于预设开合阈值。
一种可选的实施方式中，可以将目标人脸的目标区域图像中眼睛开合程度值最大的目标区域图像，确定为该目标人脸对应的目标效果图像，以提高目标图像中每张人脸的人眼睁开程度，进而提高用户对目标图像的满意度。
另一种可选的实施方式中，可以将目标人脸的目标区域图像中的任意一张目标区域图像，确定为该目标人脸对应的目标效果图像，以满足用户对目标图像中人脸的眼部状态效果的基本要求。
一种可选的实施方式中，如果确定基准图像上的某张人脸(例如第一张人脸)的眼部状态符合预设条件，则可以不针对该第一张人脸进行合成，以提高图像处理的效率。
S103：将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
本公开实施例中，在确定每张人脸对应的目标效果图像之后，将该目标效果图像合成到待处理图像集中基准图像上，进而得到待处理图像集对应的目标图像。
由于目标图像基于目标效果图像得到。因此，目标图像能够最大程度上提高图像上每个人的眼部状态的效果，从而提高了目标图像的质量，一定程度上提高了用户对目标图像的满意度。
一种可选的实施方式中，每张人脸对应的目标效果图像具有位置信息，基于该目标效果图像的位置信息将该目标效果图像合成到基准图像上的对应位置。
需要说明的是，本公开实施例中还可以将待处理图像集中的任一张图像确定为基准图像。本公开实施例不具体限定基准图像的确定方式，本领域技术人员可以根据实际需要进行选择。
本公开实施例提供的基于眼部状态检测的图像处理方法中，首先对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，然后从眼部状态符合预设条件的目标区域图像中确定出该目标人脸对应的目标效果图像，最终，将该目标效果图像合成到待处理图像集中的基准图像上，得到待处理图像集对应的目标图像。本公开实施例通过对眼部状态进行检测，确定每张人脸的目标效果图像，然后每张人脸的目标效果图像合成到基准图像中，能够提高目标图像中每个人的眼部状态效果，提高了目标图像的质量，一定程度上提高了用户对目标图像的满意度。
本公开实施例提供的基于眼部状态检测的图像处理方法中，可以基于人眼关键点的位置信息，确定人眼图像对应的眼部状态。
一种可选的实施方式中，如图2所示，分别从8帧的人脸图像中提取到对应的人眼图像。然后，针对每一张人眼图像，确定该人眼图像中的人眼关键点的位置信息，接着基于人眼关键点的位置信息，确定该人眼图像对应的眼部状态。在一种实现方式 中，将该目标人脸的人眼图像对应的眼部状态作为该目标人脸对应的人脸图像的眼部状态，其中，眼部状态可以利用眼睛开合程度值表示。
如图3所示，人眼关键点可以分别为眼部的左眼角关键点1、上眼睑上的关键点2和3、右眼角关键点4、下眼睑上的关键点5和6。在确定上述6个人眼关键点之后，基于各个人眼关键点的位置信息确定眼睛开合程度值。
一种可选的实施方式中，可以将图3中的关键点1到关键点4之间的距离作为水平方向两个眼角距离，以及将关键点2到关键6的距离与关键点3到关键5的距离的平均值，作为人眼竖直方向睁开宽度。然后将人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值，确定为眼睛开合程度值。
一种可选的实施方式中，可以利用机器学习模型确定人眼关键点的位置信息。具体的，利用标记有人眼关键点的位置信息的人眼图像样本对第一模型进行训练，将人眼图像输入至训练后的第一模型中，经过第一模型的处理后，输出该人眼图像中的人眼关键点的位置信息。
另外，本公开实施例提供的基于眼部状态检测的图像处理方法中，还可以基于人眼状态值，确定人眼图像对应的眼部状态。其中，人眼状态值包括睁眼状态值和闭眼状态值。具体的，人眼状态值可以为[0,1]范围内的数值，人眼状态值的数值越大，说明眼部开合程度值越大；相反的，人眼状态值越小，说明眼部开合程度值越小。具体的，闭眼状态值可以为[0，0.5)范围内的数值，而睁眼状态值可以为[0.5，1]范围内的数值；在另一些实施例，闭眼状态值可以为[0，0.5]范围内的数值，而睁眼状态值可以为(0.5，1]范围内的数值。
一种可选的实施方式中，可以利用机器学习模型确定人眼状态值。具体的，利用基于标记有人眼状态值的人眼图像样本对第二模型进行训练，将人眼图像输入至训练后的第二模型中，经过第二模型的处理后，输出该人眼图像的人眼状态值。
本公开实施例中，可以通过人眼状态值确定人眼图像对应的眼部状态，并将目标人脸的人眼状态值最大的目标区域图像确定为目标效果图像。
为了提高眼部状态检测的准确率，本公开实施例可以结合人眼关键点的位置信息和人眼状态值，确定人脸图像对应的眼睛开合程度值，从而提高基于眼睛开合程度值确定得到的目标效果图像的准确率，进而提高目标图像的质量。
本公开实施例提供了一种基于眼部状态检测的图像处理方法，参考图4，为本公开实施例提供的另一种基于眼部状态检测的图像处理方法的流程图，该方法包括：
S401：基于待处理图像集确定属于目标人脸的人脸图像。
其中，所述待处理图像集中包括以按下快门键对应的当前图像帧为结束帧的连续多帧预览图像。
本公开实施例中的S401可以参照上述实施例的描述进行理解，在此不再赘述。
S402：从所述目标人脸图像中提取人眼图像。
参考图2，在确定属于目标人脸的人脸图像之后，基于确定的目标人脸的人脸图像， 提取属于目标人脸的人眼图像。
一种可选的实施方式中，对人脸图像进行眼部检测，确定人脸图像中眼部的位置信息，然后基于眼部的位置信息，确定包含眼部的矩形框区域，并将该矩形框区域从人脸图像中提取出来。将该提取出的矩形区域对应的图像作为人眼图像。其中，眼部检测的方式不做过多说明。
实际应用中，考虑到人脸图像上的两只眼睛的眼部状态基本相同。因此，本公开实施例提取到的人眼图像可以仅包括人脸图像中的其中一只眼睛，从而提高图像处理的效率。
S403：确定所述人眼图像中人眼状态值和人眼关键点的位置信息。
本公开实施例中，在提取到人眼图像之后，确定人眼图像中人眼状态值和人眼关键点的位置信息。
一种可选的实施方式中，可以利用机器学习模型确定人眼状态值和人眼关键点的位置信息。具体的，利用基于标记有人眼关键点的位置信息和人眼状态值的人眼图像样本对第三模型进行训练，将人眼图像输入至训练后的第三模型中，经过第三模型的处理后，输出该人眼图像的人眼状态值和人眼关键点的位置信息。
S404：基于所述人眼状态值和所述人眼关键点的位置信息，确定所述人脸图像对应的眼睛开合程度值。
本公开实施例中，在确定所述人眼图像中人眼状态值和人眼关键点的位置信息之后，可以基于所述人眼关键点的位置信息，确定所述人脸图像上的人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值。然后，结合所述人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值和所述人眼状态值，确定所述人脸图像对应的眼睛开合程度值。
一种可选的实施方式中，参考图3，可以利用公式(1)计算人脸图像对应的眼睛开合程度值，其中，公式(1)如下：
Figure PCTCN2021114881-appb-000001
其中，Open_Degree(OD)用于表示人脸图像对应的眼睛开合程度值；H_d用于表示图3中的关键点1和关键点4之间的欧式距离，V_d_1用于表示图3中的关键点2和关键点6之间的欧式距离，V_d_2用于表示图3中的关键点3和关键点5之间的欧式距离；round()用于表示对参数进行四舍五入取整，Eye_State用于表示人眼状态值，在[0,1]之间。
S405：从属于目标人脸的人脸图像中，确定眼睛开合程度值大于预设开合阈值的人脸图像。
本公开实施例中，在确定属于目标人脸的人脸图像之后，针对任意一张人脸的人脸图像，首先基于眼睛开合程度值，剔除闭眼状态的人脸图像。然后基于眼睛开合程度值剔除眼睛开合程度值低于或等于预设开合阈值的人脸图像。也可以根据眼睛开合程度值对剩余的人脸图像进行排序，确定开合程度值大于预设开合阈值的人脸图像。
一种可选的实施方式中，如果在某张人脸的人脸图像中不存在眼睛开合程度值大于预设开合阈值的人脸图像，则可以不对基准图像上该张人脸对应的人脸图像进行处理，保留基准图像中该张人脸的效果。
S406：从所述眼睛开合程度值大于预设开合阈值的人脸图像中，确定所述目标人脸对应的目标效果图像。
本公开实施例中，在确定出每张人脸的眼睛开合程度值大于预设开合阈值的人脸图像之后。该方法可以从目标人脸对应的人脸图像中，随机选择一张眼睛开合程度值大于预设开合阈值的人脸图像，将随机选择出的人脸图像作为该目标人脸对应的目标效果图像，以提高目标图像中该目标人脸上眼部状态效果。对于其他人脸对应的人脸图像，可以进行类似的处理，进而得到其他人脸对应的目标效果图像。
一种可选的实施方式中，由于眼睛开合程度值越大，说明眼睛睁开的程度越大，越能够体现最佳的眼部状态。因此，本公开实施例从眼睛开合程度值大于预设开合阈值的人脸图像中，选择眼睛开合程度值最大的人脸图像作为该目标人脸对应的目标效果图像，以最大化提高目标图像中该目标人脸上眼部状态的效果。
S407：将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
本公开实施例中，在确定每张人脸对应的目标效果图像之后，将各个目标效果图像均合成至基准图像中，最终得到待处理图像集对应的目标图像。
在多人拍摄场景中，基于本公开提供的基于眼部状态检测的图像处理方法，能够仅拍摄一次合影照片，即可使合影照片中尽可能多的人的眼部状态的效果较佳，无需反复多次重拍，提高了用户的合照拍摄体验，同时也能够为用户提供满意度较高的合影照片。
与上述方法实施例基于同一个发明构思，本公开还提供了一种基于眼部状态检测的图像处理装置，参考图5，为本公开实施例提供的一种基于眼部状态检测的图像处理装置，所述装置包括：
第一检测模块501，用于对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像；其中，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸；
第一确定模块502，用于基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像；
合成模块503，用于将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的预设基准图像上，得到所述待处理图像集对应的目标图像。
一种可选的实施方式中，所述预设条件包括眼睛开合程度值大于预设开合阈值。
一种可选的实施方式中，所述第一检测模块，包括：
第一确定子模块，用于从待处理图像集中确定目标人脸的人脸图像；
第一检测子模块，用于对所述目标人脸的人脸图像进行眼部状态检测，得到所述 目标人脸的人脸图像中眼部状态符合预设条件的人脸图像，作为目标区域图像。
一种可选的实施方式中，所述第一检测子模块，包括：
提取子模块，用于从所述目标人脸的人脸图像中提取所述目标人脸的人眼图像；
第二检测子模块，用于对所述目标人脸的人眼图像进行眼部状态检测。
一种可选的实施方式中，所述第一检测模块，包括：
第二确定子模块，用于从待处理图像集中确定目标人脸的人眼图像；
第三检测子模块，用于对所述目标人脸的人眼图像进行眼部状态检测，得到所述目标人脸的人眼图像中眼部状态符合预设条件的人眼图像，作为目标区域图像。
一种可选的实施方式中，所述第二检测模块或所述第三检测子模块，包括：
第三确定子模块，用于确定所述目标人脸的人眼图像中的人眼关键点的位置信息；
第四确定子模块，用于基于所述人眼关键点的位置信息，确定所述人眼图像对应的眼部状态。
一种可选的实施方式中，所述第三确定子模块，具体用于：
将所述目标人脸的人眼图像输入到第一模型中，得到所述人眼图像中的人眼关键点的位置信息；其中，所述第一模型是基于标记有人眼关键点的位置信息的人眼图像样本训练得到。
一种可选的实施方式中，所述第二检测模块或所述第三检测子模块，包括：
第五确定子模块，用于确定所述目标人脸的人眼图像中的人眼状态值；其中，所述人眼状态值包括睁眼状态值和闭眼状态值；
第六确定子模块，用于基于所述人眼状态值，确定所述人眼图像对应的眼部状态。
一种可选的实施方式中，所述第五确定子模块，具体用于：
将所述目标人脸的人眼图像输入至第二模型中，得到所述人眼图像中的人眼状态值；其中，所述第二模型是基于标记有人眼状态值的人眼图像样本训练得到。
一种可选的实施方式中，所述人眼图像对应的眼部状态基于人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值确定；所述人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值基于所述人眼关键点的位置信息确定。
一种可选的实施方式中，所述装置还包括：
获取模块，用于根据对快门键的触发操作，获取包括当前图像帧和以所述当前图像帧为结束帧的连续多帧预览图像，作为待处理图像集。
一种可选的实施方式中，所述装置还包括：
第二确定模块，用于将所述待处理图像集中按下快门键对应的当前图像帧，确定为基准图像。
一种可选的实施方式中，所述第一确定模块，具体用于：
将所述目标人脸的眼部状态符合预设条件的目标区域图像中，眼睛开合程度值最大的目标区域图像，确定为所述目标人脸对应的目标效果图像。
一种可选的实施方式中，所述第一确定子模块，包括：
第七确定子模块，用于对待处理图像集中的基准图像进行人脸检测，确定所述基 准图像上各个人脸的位置信息；
第八确定子模块，用于根据所述各个人脸的位置信息，确定所述待处理图像集中的图像上与所述各个人脸中目标人脸的位置信息对应的人脸图像，作为所述人脸的人脸图像。
一种可选的实施方式中，所述第一确定子模块，包括：
第四检测子模块，用于分别对待处理图像集中的每张图像进行人脸检测，得到人脸图像；
第九确定子模块，用于将相似度大于预设相似阈值的人脸图像，确定为目标人脸的人脸图像。
本公开实施例提供的基于眼部状态检测的图像处理装置中，对待处理图像集中目标人脸的眼部状态进行检测，得到所述目标人脸的眼部状态符合预设条件的目标区域图像，然后从眼部状态符合预设条件的目标区域图像中确定出该人脸对应的目标效果图像，最终将该目标效果图像合成到待处理图像集中的基准图像上，得到待处理图像集对应的目标图像。本公开实施例通过对眼部状态进行检测，确定每张人脸的目标效果图像，然后将每张人脸的目标效果图像合成到基准图像中，能够提高最终得到的目标图像中每个人的眼部状态效果，提高了目标图像的质量，一定程度上提高了用户对目标图像的满意度。
另外，本公开实施例还提供了一种基于眼部状态检测的图像处理设备，参见图6所示，可以包括：
处理器601、存储器602、输入装置603和输出装置604。基于眼部状态检测的图像处理设备中的处理器601的数量可以一个或多个，图6中以一个处理器为例。在本公开的一些实施例中，处理器601、存储器602、输入装置603和输出装置604可通过总线或其它方式连接，其中，图6中以通过总线连接为例。
存储器602可用于存储软件程序以及模块，处理器601通过运行存储在存储器602的软件程序以及模块，从而执行基于眼部状态检测的图像处理设备的各种功能应用以及数据处理。存储器602可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序等。此外，存储器602可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。输入装置603可用于接收输入的数字或字符信息，以及产生与基于眼部状态检测的图像处理设备的用户设置以及功能控制有关的信号输入。
具体在本实施例中，处理器601会按照如下的指令，将一个或一个以上的应用程序的进程对应的可执行文件加载到存储器602中，并由处理器601来运行存储在存储器602中的应用程序，从而实现上述基于眼部状态检测的图像处理设备的各种功能。
本公开实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当所述指令在终端设备上运行时，使得所述终端设备实现上述的方法。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体 或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本公开的具体实施方式，使本领域技术人员能够理解或实现本公开。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本公开的精神或范围的情况下，在其它实施例中实现。因此，本公开将不会被限制于本文所述的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (18)

  1. 一种基于眼部状态检测的图像处理方法，其特征在于，所述方法包括：
    对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像；其中，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸；
    基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像；
    将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
  2. 根据权利要求1所述的方法，其特征在于，所述预设条件包括眼睛开合程度值大于预设开合阈值。
  3. 根据权利要求1所述的方法，其特征在于，所述对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，包括：
    从待处理图像集中确定目标人脸的人脸图像；
    对所述目标人脸的人脸图像进行眼部状态检测，得到所述目标人脸的人脸图像中眼部状态符合预设条件的人脸图像，作为目标区域图像。
  4. 根据权利要求3所述的方法，其特征在于，所述对所述目标人脸的人脸图像进行眼部状态检测，包括：
    从所述目标人脸的人脸图像中提取所述目标人脸的人眼图像；
    对所述目标人脸的人眼图像进行眼部状态检测。
  5. 根据权利要求1所述的方法，其特征在于，所述对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部状态符合预设条件的目标区域图像，包括：
    从待处理图像集中确定目标人脸的人眼图像；
    对所述目标人脸的人眼图像进行眼部状态检测，得到所述目标人脸的人眼图像中眼部状态符合预设条件的人眼图像，作为目标区域图像。
  6. 根据权利要求4或5所述的方法，其特征在于，所述对所述目标人脸的人眼图像进行眼部状态检测，包括：
    确定所述目标人脸的人眼图像中的人眼关键点的位置信息；
    基于所述人眼关键点的位置信息，确定所述人眼图像对应的眼部状态。
  7. 根据权利要求6所述的方法，其特征在于，所述确定所述目标人脸的人眼图像中的人眼关键点的位置信息，包括：
    将所述目标人脸的人眼图像输入到第一模型中，得到所述人眼图像中的人眼关键点的位置信息；其中，所述第一模型是基于标记有人眼关键点的位置信息的人眼图像样本训练得到。
  8. 根据权利要求4或5所述的方法，其特征在于，所述对所述目标人脸的人眼图像进 行眼部状态检测，包括：
    确定所述目标人脸的人眼图像中的人眼状态值；其中，所述人眼状态值包括睁眼状态值和闭眼状态值；
    基于所述人眼状态值，确定所述人眼图像对应的眼部状态。
  9. 根据权利要求8所述的方法，其特征在于，所述确定所述目标人脸的人眼图像中的人眼状态值，包括：
    将所述目标人脸的人眼图像输入至第二模型中，得到所述人眼图像中的人眼状态值；其中，所述第二模型是基于标记有人眼状态值的人眼图像样本训练得到。
  10. 根据权利要求7所述的方法，其特征在于，所述人眼图像对应的眼部状态基于人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值确定；所述人眼竖直方向睁开宽度与水平方向两个眼角距离的比例值基于所述人眼关键点的位置信息确定。
  11. 根据权利要求1所述的方法，其特征在于，所述对待处理图像集中目标人脸的眼部状态进行检测之前，还包括：
    根据对快门键的触发操作，获取包括当前图像帧和以所述当前图像帧为结束帧的连续多帧预览图像，作为待处理图像集。
  12. 根据权利要求1所述的方法，其特征在于，所述将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像之前，还包括：
    将所述待处理图像集中按下快门键对应的当前图像帧，确定为基准图像。
  13. 根据权利要求1所述的方法，其特征在于，所述基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像，包括：
    将所述目标人脸的眼部状态符合预设条件的目标区域图像中，眼睛开合程度值最大的目标区域图像，确定为所述目标人脸对应的目标效果图像。
  14. 根据权利要求3所述的方法，其特征在于，所述从待处理图像集中确定目标人脸的人脸图像，包括：
    对待处理图像集中的基准图像进行人脸检测，确定所述基准图像上各个人脸的位置信息；
    根据所述各个人脸的位置信息，确定所述待处理图像集中的图像上与所述各个人脸中目标人脸的位置信息对应的人脸图像，作为所述目标人脸的人脸图像。
  15. 根据权利要求3所述的方法，其特征在于，所述从待处理图像集中确定目标人脸的人脸图像，包括：
    对待处理图像集中的每张图像进行人脸检测，得到人脸图像；
    将相似度大于预设相似阈值的人脸图像，确定为目标人脸的人脸图像。
  16. 一种基于眼部状态检测的图像处理装置，其特征在于，所述装置包括：
    第一检测模块，用于对待处理图像集中目标人脸的眼部状态进行检测，得到所述眼部 状态符合预设条件的目标区域图像；其中，所述待处理图像集包括连续的多帧图像，所述多帧图像中的每一帧图像包括至少一张人脸；
    第一确定模块，用于基于所述眼部状态符合预设条件的目标区域图像，确定所述目标人脸对应的目标效果图像；
    合成模块，用于将所述目标人脸对应的目标效果图像合成到所述待处理图像集中的基准图像上，得到所述待处理图像集对应的目标图像。
  17. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有指令，当所述指令在终端设备上运行时，使得所述终端设备实现如权利要求1-15任一项所述的方法。
  18. 一种设备，其特征在于，包括：存储器，处理器，及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时，实现如权利要求1-15任一项所述的方法。
PCT/CN2021/114881 2020-08-31 2021-08-27 基于眼部状态检测的图像处理方法、装置及存储介质 Ceased WO2022042670A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21860504.6A EP4206975B1 (en) 2020-08-31 2021-08-27 Eye state detection-based image processing method and apparatus, and storage medium
JP2023513999A JP7822369B2 (ja) 2020-08-31 2021-08-27 眼部状態検出による画像処理方法、装置及び記憶媒体
US18/087,660 US11842569B2 (en) 2020-08-31 2022-12-22 Eye state detection-based image processing method and apparatus, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010899317.9 2020-08-31
CN202010899317.9A CN112036311A (zh) 2020-08-31 2020-08-31 基于眼部状态检测的图像处理方法、装置及存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/087,660 Continuation US11842569B2 (en) 2020-08-31 2022-12-22 Eye state detection-based image processing method and apparatus, and storage medium

Publications (1)

Publication Number Publication Date
WO2022042670A1 true WO2022042670A1 (zh) 2022-03-03

Family

ID=73587021

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/114881 Ceased WO2022042670A1 (zh) 2020-08-31 2021-08-27 基于眼部状态检测的图像处理方法、装置及存储介质

Country Status (5)

Country Link
US (1) US11842569B2 (zh)
EP (1) EP4206975B1 (zh)
JP (1) JP7822369B2 (zh)
CN (1) CN112036311A (zh)
WO (1) WO2022042670A1 (zh)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112036311A (zh) * 2020-08-31 2020-12-04 北京字节跳动网络技术有限公司 基于眼部状态检测的图像处理方法、装置及存储介质
CN113313790A (zh) * 2021-05-31 2021-08-27 北京字跳网络技术有限公司 视频生成方法、装置、设备及存储介质
CN113542598B (zh) * 2021-07-02 2023-11-21 维沃移动通信（杭州）有限公司 图像处理方法、装置、电子设备及可读存储介质
CN113747057B (zh) * 2021-07-26 2022-09-30 荣耀终端有限公司 一种图像处理方法、电子设备、芯片系统和存储介质
CN115914821A (zh) * 2022-12-07 2023-04-04 深圳创维－Rgb电子有限公司 多人拍照的图像处理方法、装置、设备以及介质
CN115953389B (zh) * 2023-02-24 2023-11-24 广州视景医疗软件有限公司 一种基于人脸关键点检测的斜视判别方法和装置
CN117911262A (zh) * 2024-01-05 2024-04-19 惠州Tcl移动通信有限公司 图像处理方法、装置、存储介质及电子设备
TWI908546B (zh) * 2024-12-17 2025-12-11 和碩聯合科技股份有限公司 眼睛狀態偵測裝置及其操作方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5878156A (en) * 1995-07-28 1999-03-02 Mitsubishi Denki Kabushiki Kaisha Detection of the open/closed state of eyes based on analysis of relation between eye and eyebrow images in input face images
CN107622483A (zh) * 2017-09-15 2018-01-23 深圳市金立通信设备有限公司 一种图像合成方法及终端
CN108520036A (zh) * 2018-03-30 2018-09-11 广东欧珀移动通信有限公司 图像的选取方法、装置、存储介质及电子设备
CN110163806A (zh) * 2018-08-06 2019-08-23 腾讯科技（深圳）有限公司 一种图像处理方法、装置以及存储介质
CN112036311A (zh) * 2020-08-31 2020-12-04 北京字节跳动网络技术有限公司 基于眼部状态检测的图像处理方法、装置及存储介质

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002199202A (ja) * 2000-12-26 2002-07-12 Seiko Epson Corp 画像処理装置
US7659923B1 (en) * 2005-06-24 2010-02-09 David Alan Johnson Elimination of blink-related closed eyes in portrait photography
JP2007323104A (ja) * 2006-05-30 2007-12-13 Fujifilm Corp 目の状態判別方法および装置ならびにプログラム
JP4853320B2 (ja) * 2007-02-15 2012-01-11 ソニー株式会社 画像処理装置、画像処理方法
JP4898532B2 (ja) * 2007-04-13 2012-03-14 富士フイルム株式会社 画像処理装置および撮影システム並びに瞬き状態検出方法、瞬き状態検出プログラムおよびそのプログラムが記録された記録媒体
JP4885084B2 (ja) * 2007-07-19 2012-02-29 富士フイルム株式会社 撮像装置、撮像方法、撮像プログラム
JP2009141811A (ja) * 2007-12-07 2009-06-25 Nec Corp 画像処理装置、方法、プログラム、記録媒体、撮像装置及び携帯端末装置
US20100302394A1 (en) * 2009-05-28 2010-12-02 Phanish Hanagal Srinivasa Rao Blinked eye artifact removal for a digital imaging device
KR101665130B1 (ko) * 2009-07-15 2016-10-25 삼성전자주식회사 복수의 인물에 대한 이미지 생성 장치 및 방법
JP2011128676A (ja) * 2009-12-15 2011-06-30 Seiko Epson Corp 画像処理装置、画像処理方法及び画像処理プログラム
KR101728042B1 (ko) * 2010-10-01 2017-04-18 삼성전자주식회사 디지털 촬영 장치 및 이의 제어 방법
AU2013200450B2 (en) * 2012-01-30 2014-10-02 Accenture Global Services Limited System and method for face capture and matching
JP2013207634A (ja) * 2012-03-29 2013-10-07 Nikon Corp 撮像装置
JP2014022922A (ja) * 2012-07-18 2014-02-03 Nikon Corp 撮像装置
US9336583B2 (en) * 2013-06-17 2016-05-10 Cyberlink Corp. Systems and methods for image editing
US9549118B2 (en) * 2014-03-10 2017-01-17 Qualcomm Incorporated Blink and averted gaze avoidance in photographic images
US20170032172A1 (en) * 2015-07-29 2017-02-02 Hon Hai Precision Industry Co., Ltd. Electronic device and method for splicing images of electronic device
JP6558388B2 (ja) * 2017-03-14 2019-08-14 オムロン株式会社 画像処理装置
US10475222B2 (en) * 2017-09-05 2019-11-12 Adobe Inc. Automatic creation of a group shot image from a short video clip using intelligent select and merge
CN107734253B (zh) * 2017-10-13 2020-01-10 Oppo广东移动通信有限公司 图像处理方法、装置、移动终端和计算机可读存储介质
CN108574803B (zh) * 2018-03-30 2020-01-14 Oppo广东移动通信有限公司 图像的选取方法、装置、存储介质及电子设备
CN109376624A (zh) * 2018-10-09 2019-02-22 三星电子（中国）研发中心 一种闭眼照片的修正方法和装置
CN110246305A (zh) * 2019-08-01 2019-09-17 肇庆学院 一种基于机器视觉的疲劳驾驶检测预警系统及方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5878156A (en) * 1995-07-28 1999-03-02 Mitsubishi Denki Kabushiki Kaisha Detection of the open/closed state of eyes based on analysis of relation between eye and eyebrow images in input face images
CN107622483A (zh) * 2017-09-15 2018-01-23 深圳市金立通信设备有限公司 一种图像合成方法及终端
CN108520036A (zh) * 2018-03-30 2018-09-11 广东欧珀移动通信有限公司 图像的选取方法、装置、存储介质及电子设备
CN110163806A (zh) * 2018-08-06 2019-08-23 腾讯科技（深圳）有限公司 一种图像处理方法、装置以及存储介质
CN112036311A (zh) * 2020-08-31 2020-12-04 北京字节跳动网络技术有限公司 基于眼部状态检测的图像处理方法、装置及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4206975A4

Also Published As

Publication number Publication date
EP4206975A4 (en) 2024-02-14
EP4206975B1 (en) 2025-12-24
EP4206975A1 (en) 2023-07-05
CN112036311A (zh) 2020-12-04
JP7822369B2 (ja) 2026-03-02
JP2023540273A (ja) 2023-09-22
US20230140313A1 (en) 2023-05-04
US11842569B2 (en) 2023-12-12

Similar Documents

Publication Publication Date Title
WO2022042670A1 (zh) 基于眼部状态检测的图像处理方法、装置及存储介质
CN111402135B (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
JP6961797B2 (ja) プレビュー写真をぼかすための方法および装置ならびにストレージ媒体
CN108108415B (zh) 图像处理方法、装置、存储介质和电子设备
WO2022028184A1 (zh) 一种拍摄控制方法、装置、电子设备及存储介质
WO2021169686A1 (zh) 一种拍摄控制方法、装置及计算机可读存储介质
TWI640199B (zh) 影像擷取裝置及其攝影構圖的方法
JP6907274B2 (ja) 撮像装置及び撮像方法
WO2018094648A1 (zh) 拍摄构图引导方法及装置
WO2017031901A1 (zh) 人脸识别方法、装置及终端
CN107820017A (zh) 图像拍摄方法、装置、计算机可读存储介质和电子设备
WO2016197657A1 (zh) 一种拍照处理方法和装置
CN108259770A (zh) 图像处理方法、装置、存储介质及电子设备
CN108419012A (zh) 拍照方法、装置、存储介质及电子设备
CN108259767B (zh) 图像处理方法、装置、存储介质及电子设备
CN108513074A (zh) 自拍控制方法及装置、电子设备
CN105430269B (zh) 一种应用于移动终端的拍照方法及装置
CN106303235A (zh) 拍照处理方法及装置
WO2022042669A1 (zh) 一种图片处理方法、装置、设备及存储介质
CN110381247A (zh) 摄像装置以及摄像装置的控制方法
CN107547789B (zh) 影像获取装置及其摄影构图的方法
CN110365897B (zh) 图像修正方法和装置、电子设备、计算机可读存储介质
CN108495038A (zh) 图像处理方法、装置、存储介质及电子设备
WO2025044439A1 (zh) 一种图像处理方法、装置、电子设备及存储介质
CN114387157B (zh) 一种图像处理方法、装置及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21860504

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023513999

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021860504

Country of ref document: EP

Effective date: 20230330

WWG Wipo information: grant in national office

Ref document number: 2021860504

Country of ref document: EP