EP3686845A1 - Bildverarbeitungsverfahren und -vorrichtung und -einrichtung - Google Patents

Bildverarbeitungsverfahren und -vorrichtung und -einrichtung Download PDF

Info

Publication number
EP3686845A1
EP3686845A1 EP18866515.2A EP18866515A EP3686845A1 EP 3686845 A1 EP3686845 A1 EP 3686845A1 EP 18866515 A EP18866515 A EP 18866515A EP 3686845 A1 EP3686845 A1 EP 3686845A1
Authority
EP
European Patent Office
Prior art keywords
image
exposure time
target image
ghost
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18866515.2A
Other languages
English (en)
French (fr)
Other versions
EP3686845B1 (de
EP3686845A4 (de
Inventor
Yinting WANG
Biying HU
Xi Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority claimed from PCT/CN2018/109951 external-priority patent/WO2019072222A1/zh
Publication of EP3686845A1 publication Critical patent/EP3686845A1/de
Publication of EP3686845A4 publication Critical patent/EP3686845A4/de
Application granted granted Critical
Publication of EP3686845B1 publication Critical patent/EP3686845B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/71—Circuitry for evaluating the brightness variation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/72—Combination of two or more compensation controls
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20216—Image averaging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20224—Image subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Definitions

  • the present invention relates to the field of terminal technologies, and in particular, to an image processing method and apparatus, and a device.
  • motion blur occurs more or less when a moving object is photographed, and streaking becomes even worse when a fast-moving target such as a playing child or a running pet is photographed.
  • a fast-moving target such as a playing child or a running pet is photographed.
  • unsteady hands may cause an image to be blurrier.
  • the present invention provides a set of photographing methods for snapping a moving scene.
  • Embodiments of the present invention provide an image processing method and apparatus, and a device, to provide a snapshot mechanism for a user.
  • an image with high sharpness can be snapped, thereby improving photographing experience of the user.
  • an embodiment of the present invention provides an image processing method.
  • the method includes: obtaining N frames of images; determining one of the N frames of images as a reference image, where the other N-1 frames of images are to-be-processed images; obtaining N-1 frames of ghost-free images based on the N-1 frames of to-be-processed images; and performing an averaging operation on the reference image and the N-1 frames of ghost-free images, to obtain a first target image, where the obtaining N-1 frames of ghost-free images based on the N-1 frames of to-be-processed images includes: performing step 1 to step 4 on an i th frame of image in the N-1 frames of to-be-processed images, where values of i are all positive integers that are not greater than N-1, and step 1: perform registration on the i th frame of image and the reference image, to obtain an i th registered image; step 2: obtain an i th difference image based on the i th registered image and the reference image; step
  • an embodiment of the present invention provides an image processing apparatus.
  • the apparatus includes: an obtaining module, configured to obtain N frames of images; a determining module, configured to determine one of the N frames of images as a reference image, where the other N-1 frames of images are to-be-processed images; a ghost-removal module, configured perform step 1 to step 4 on an i th frame of image in the N-1 frames of to-be-processed images, to obtain N-1 frames of ghost-free images, where values of i are all positive integers that are not greater than N-1, and step 1: perform registration on the i th frame of image and the reference image, to obtain an i th registered image; step 2: obtain an i th difference image based on the i th registered image and the reference image; step 3: obtain an i th ghost weight image based on the i th difference image; and step 4: perform fusion on the i th registered image and the reference image based on the i th ghost weight image,
  • a user in a moving scene, a user can still snap a moving image and obtain an image with high sharpness.
  • the method before the obtaining N frames of images, the method further includes: when it is detected that the following three cases exist at the same time, generating a control signal, where the control signal is used to instruct to obtain the N frames of images; and case 1: it is detected that a to-be-photographed image of a camera is a moving image; case 2: it is detected that a current exposure time of the camera exceeds a safe time; and case 3: it is detected that the camera is in an extreme high-light scene, and correspondingly, current ISO sensitivity of the camera is less than a first preset threshold and the current exposure time is less than a second preset threshold.
  • a corresponding control signal may be generated once at least one of the foregoing three cases is detected. For example, if the case 3 is detected, which indicates extreme high-light conditions, the camera intelligently switches to a first snapshot mode mentioned below, and generates the control signal to obtain the N frames of images in the first snapshot mode.
  • a method of detecting the foregoing case may be performed by a detection module.
  • the obtaining N frames of images includes: with a product of the current ISO sensitivity and the current exposure time of the camera unchanged, reducing the exposure time and increasing the ISO sensitivity at a preset ratio, to obtain a first exposure time and first ISO sensitivity; and setting the current exposure time and the current ISO sensitivity of the camera to the first exposure time and the first ISO sensitivity respectively, and photographing the N frames of images.
  • the method before the obtaining N frames of images, the method further includes: when it is detected that the following three cases exist at the same time, generating a control signal, where the control signal is used to instruct to obtain the N frames of images; and case 1: it is detected that a to-be-photographed image of a camera is a moving image; case 2: it is detected that a current exposure time of the camera exceeds a safe time; and case 3: it is detected that the camera is in a moderate high-light scene, and correspondingly, current ISO sensitivity of the camera is in a first preset threshold interval and the current exposure time is in a second preset threshold interval.
  • a corresponding control signal may be generated once at least one of the foregoing three cases is detected. For example, if the case 3 is detected, which indicates moderate high-light conditions, the camera intelligently switches to a second snapshot mode or a third snapshot mode mentioned below, and generates the control signal to obtain the N frames of images in the second snapshot mode or the third snapshot mode.
  • a method of detecting the foregoing case may be performed by a detection module.
  • the obtaining N frames of images includes: with a product of the current ISO sensitivity and the current exposure time of the camera unchanged, reducing the exposure time and increasing the ISO sensitivity at a preset ratio, to obtain a second exposure time and second ISO sensitivity; and setting the current exposure time and the current ISO sensitivity of the camera to the second exposure time and the second ISO sensitivity respectively, and photographing the N frames of images.
  • the method further includes: photographing one frame of first new image based on the current ISO sensitivity and the exposure time of the camera; and obtaining a second target image based on the first target image and the first new image.
  • the obtaining a second target image based on the first target image and the first new image includes: performing registration on the first new image and the reference image or the first target image, to obtain a first (second) registered image; obtaining a first (second) difference image based on the first (second) registered image and the first target image; obtaining a first (second) ghost weight image based on the first (second) difference image; performing fusion on the first (second) registered image and the first target image based on the first (second) ghost weight image, to obtain a first (second) ghost-free image; and performing pixel-value weighted fusion based on the first (second) ghost-free image and the first target image, to obtain the second target image.
  • the obtaining N frames of images includes: with the current ISO sensitivity of the camera unchanged, setting the current exposure time to a third exposure time that is shorter, and photographing the N frames of images; and the method further includes: photographing one frame of second new image based on the current ISO sensitivity and the current exposure time of the camera; and obtaining a third target image based on the first target image and the second new image.
  • This technical solution may be used for a pure snapshot mode, in which there is no need to decide on a current photographing environment.
  • the obtaining a third target image based on the first target image and the second new image includes: processing the first target image based on the second new image according to a preset luminance correction algorithm, to obtain a fourth target image; performing registration on the second new image and the reference image or the fourth target image, to obtain a third (fourth) registered image; obtaining a third (fourth) difference image based on the third (fourth) registered image and the fourth target image; obtaining a third (fourth) ghost weight image based on the third (fourth) difference image; performing fusion on the third (fourth) registered image and the fourth target image based on the third (fourth) ghost weight image, to obtain a third (fourth) ghost-free image; performing pixel-value weighted fusion based on the third (fourth) ghost-free image and the fourth target image, to obtain a fifth (sixth) target image; and performing pyramid fusion on the fifth (sixth) target image; and performing pyramid fusion on the fifth (sixth) target image; and performing
  • the foregoing possible technical implementations may be implemented by a processor invoking a program and instruction in a memory to perform corresponding computing processing.
  • the terminal directly enters a snapshot mode, for example, the first snapshot mode, the second snapshot mode, or the third snapshot mode that is mentioned above.
  • the terminal does not need to detect an environment of a to-be-photographed scene, because there is a preset parameter rule (prestored locally in the terminal or a cloud server) for each snapshot mode.
  • each snapshot mode has corresponding ISO sensitivity and a corresponding exposure time, and certainly, there may be other performance parameters or the like.
  • the camera Upon entering a particular snapshot mode, the camera automatically adjusts to corresponding ISO sensitivity and a corresponding exposure time to photograph. Therefore, if the user directly uses the snapshot mode, the obtaining N frames of images may be photographing the N images by using the corresponding ISO sensitivity and the corresponding exposure time, to perform subsequent image processing in a corresponding mode.
  • a photographing action may be triggered by the user pressing a shutter button.
  • the current exposure time and the current ISO sensitivity of the camera are already adjusted to the first exposure time and the first ISO sensitivity; and when the user presses the shutter button, the camera photographs the N images by using the first exposure time and the first ISO sensitivity, for subsequent processing.
  • the camera before the user presses the shutter button, the camera still maintains the current ISO sensitivity and the current exposure time; and when the user presses the shutter button, the current exposure time and the current ISO sensitivity of the camera are adjusted to the first exposure time and the first ISO sensitivity, and the camera photographs the N images by using the first exposure time and the first ISO sensitivity, for subsequent processing.
  • the image in a preview image data stream, the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the first exposure time and the first ISO sensitivity.
  • a photographing action may be triggered by the user pressing a shutter button.
  • the shutter button when the user presses the shutter button, one frame of first new image is obtained by using the current ISO sensitivity and the current exposure time; the current exposure time and the current ISO sensitivity are adjusted to the second exposure time and the second ISO sensitivity, and the N images are photographed in this condition; and N+1 images are obtained in total, for subsequent processing.
  • the current exposure time and the current ISO sensitivity are set to the second exposure time and the second ISO sensitivity, and the N images are photographed in this condition; then one frame of first new image is obtained still by using the current ISO sensitivity and the current exposure time; and N+1 images are obtained in total, for subsequent processing.
  • the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the second exposure time and the second ISO sensitivity.
  • a photographing action may be triggered by the user pressing a shutter button.
  • one frame of second new image is obtained by using the current ISO sensitivity and the current exposure time; with the current ISO sensitivity of the camera unchanged, the current exposure time is set to the third exposure time that is shorter, and the N images are photographed in this condition; and N+1 images are obtained in total, for subsequent processing.
  • the current exposure time is set to the third exposure time that is shorter, and the N images are photographed in this condition; then one frame of second new image is obtained still by using the current ISO sensitivity and the current exposure time; and N+1 images are obtained in total, for subsequent processing.
  • the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the third exposure time and the current ISO sensitivity.
  • an embodiment of the present invention provides a terminal device.
  • the terminal device includes a memory, a processor, a bus, a camera lens, and the memory, the camera lens, and the processor are connected by using the bus, where the camera lens is configured to collect an image signal under control of the processor; the memory is configured to store a computer program and instruction; and the processor is configured to invoke the computer program and instruction stored in the memory, so that the terminal device performs the method according to any one of the foregoing possible designs.
  • the terminal device further includes an antenna system, and under control of the processor, the antenna system sends and receives a radio communication signal to implement radio communication with a mobile communications network, where the mobile communications network includes one or more of the following: a GSM network, a CDMA network, a 3G network, a 4G network, FDMA, TDMA, PDC, TACS, AMPS, WCDMA, TDSCDMA, WIFI, and an LTE network.
  • Photographing includes normal photographing, selfie photographing, and other photographing modes such as a video call, a video conference, VR photography, and aerial photography.
  • the terminal in the embodiments of the present invention may include a plurality of photographing modes, for example, a pure snapshot mode, or a smart photographing mode that determines, depending on a result of scene detection, whether to snap an image.
  • a terminal is in a snapshot mode, for a moving scene or a scene with a relatively high signal-to-noise ratio in which a sharp image is not easy to photograph, a photo with high sharpness can be photographed by using the solutions, thereby improving a user's photographing experience.
  • a terminal may be a device that provides photographing and/or data connectivity for a user, a handheld device with a wireless connection function, or another processing device connected to a wireless modem, such as a digital camera, a single-lens reflex camera, a mobile phone (or referred to as a "cellular" phone), or may be a portable, pocket-sized, handheld, or wearable device (such as a smartwatch), a tablet computer, a personal computer (PC, Personal Computer), a PDA (Personal Digital Assistant, personal digital assistant), a POS (Point of Sale, point of sale), an in-vehicle computer, an unmanned aerial vehicle, a camera drone, or the like.
  • a wireless modem such as a digital camera, a single-lens reflex camera, a mobile phone (or referred to as a "cellular" phone), or may be a portable, pocket-sized, handheld, or wearable device (such as a smartwatch), a tablet computer, a personal computer (PC, Personal Computer), a
  • FIG. 1 is a schematic diagram of an optional hardware structure of a terminal 100.
  • the terminal 100 may include components such as a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera lens 150, an audio frequency circuit 160, a speaker 161, a microphone 162, a processor 170, an external interface 180, and a power supply 190.
  • a radio frequency unit 110 a radio frequency unit 110
  • a memory 120 a memory 120
  • an input unit 130 a display unit 140
  • a camera lens 150 an audio frequency circuit 160
  • speaker 161 a speaker 161
  • microphone 162 a processor 170
  • an external interface 180 a power supply 190.
  • the camera lens 150 is configured to collect an image or a video and may be enabled through a trigger of an application program instruction, to implement a photo or video photographing function.
  • the camera lens may include components such as an imaging lens, a light filter, an image sensor, and a motor for focusing and optical image stabilization. Light rays emitted or reflected by an object enter the imaging lens and finally converge on the image sensor through the light filter.
  • the imaging lens is mainly configured to converge light emitted or reflected by all objects in a photographing view (which may also be referred to as subjects to be photographed) for imaging.
  • the light filter is mainly configured to filter out redundant light waves (for example, a light wave other than visible light, such as infrared) from light rays.
  • the image sensor is mainly configured to: perform optical-to-electrical conversion on a received optical signal to obtain an electrical signal, and input the electrical signal to the processor 170 for subsequent processing.
  • FIG. 2 is merely an example of a portable multifunctional apparatus and constitutes no limitation on the portable multifunctional apparatus.
  • the terminal may include more or fewer components than shown in the figure, or combine some components, or have different components.
  • the input unit 130 may be configured to: receive entered digit or character information, and generate a key signal input related to user settings and function control of the portable multifunctional apparatus.
  • the input unit 130 may include a touchscreen 131 and another input device 132.
  • the touchscreen 131 may collect a touch operation (for example, an operation performed by a user on the touchscreen or near the touchscreen by using any proper object such as a finger, a joint, or a stylus) of the user on or near the touchscreen, and drive a corresponding connection apparatus according to a preset program.
  • the touchscreen may detect a touch action of the user on the touchscreen, convert the touch action into a touch signal, send the touch signal to the processor 170, and can receive and execute a command sent by the processor 170.
  • the touch signal includes at least touch point coordinate information.
  • the touchscreen 131 may provide an input interface and an output interface between the terminal 100 and the user.
  • the touchscreen may be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type.
  • the input unit 130 may further include the another input device.
  • the another input device 132 may include but is not limited to one or more of a physical keyboard, a function key (such as a volume control key or a power on/off key), a trackball, a mouse, a joystick, and the like.
  • the display unit 140 may be configured to display information entered by the user or information provided for the user, and various menus of the terminal 100.
  • the display unit is further configured to display an image obtained by the terminal by using the camera lens 150, including a preview image, an initial image that is photographed, and a target image that is obtained after being photographed and processed by using a specific algorithm.
  • the touchscreen 131 may cover a display panel 141. After detecting a touch operation on or near the touchscreen 131, the touchscreen 131 transfers the touch operation to the processor 170 to determine a type of a touch event. Then the processor 170 provides a corresponding visual output on the display panel 141 based on the type of the touch event.
  • the touchscreen and the display unit may be integrated into one component to implement input, output, and display functions of the terminal 100.
  • a touch display screen represents a function set of the touchscreen and the display unit. In some embodiments, the touchscreen and the display unit may alternatively be used as two independent components.
  • the memory 120 may be configured to store an instruction and data.
  • the memory 120 may mainly include an instruction storage area and a data storage area.
  • the data storage area may store an association relationship between a joint touch gesture and an application program function.
  • the instruction storage area may store software units such as an operating system, an application, and an instruction required by at least one function, or a subset and an extension set of the software units.
  • the memory 120 may further include a non-volatile random access memory, and provide the processor 170 with hardware, software, and data resources for managing a computing processing device, to support control of software and an application.
  • the memory 120 is further configured to store a multimedia file, and store an execution program and an application.
  • the processor 170 is a control center of the terminal 100, and is connected to various parts of an entire mobile phone through various interfaces and lines.
  • the processor 170 performs various functions of the terminal 100 and processes data by running or executing the instruction stored in the memory 120 and invoking the data stored in the memory 120, to perform overall monitoring on the mobile phone.
  • the processor 170 may include one or more processing units.
  • the processor 170 may be integrated with an application processor and a modem processor.
  • the application processor mainly processes an operating system, a user interface, an application program, and the like.
  • the modem processor mainly processes wireless communication. It may be understood that alternatively, the modem processor may not be integrated into the processor 170.
  • the processor and the memory may be implemented on a single chip.
  • the processor and the memory may be separately implemented on independent chips.
  • the processor 170 may be further configured to: generate a corresponding operation control signal; send the operation control signal to a corresponding component of the computing processing device; and read and process data in software, especially the data and program in the memory 120, so that each function module in the computing processing device performs a corresponding function, to control the corresponding component to perform an action as required by an instruction.
  • the radio frequency unit 110 may be configured to: receive and send information, or receive and send a signal during a call, and particularly, receive downlink information of a base station and then send the downlink information to the processor 170 for processing.
  • the radio frequency unit 110 is configured to send uplink data of the mobile phone to the base station.
  • the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, or the like.
  • the radio frequency unit 110 may further communicate with a network device and another device through wireless communication.
  • the wireless communication may use any communications standard or protocol, including but not limited to a global system for mobile communications (Global System for Mobile communication, GSM), a general packet radio service (General Packet Radio Service, GPRS), code division multiple access (Code Division Multiple Access, CDMA), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), long term evolution (Long Term Evolution, LTE), email, a short message service (Short Message Service, SMS), or the like.
  • GSM Global System for Mobile communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA wideband code division multiple access
  • LTE Long Term Evolution
  • SMS short message service
  • the audio frequency circuit 160, the speaker 161, and the microphone 162 may provide an audio interface between the user and the terminal 100.
  • the audio frequency circuit 160 may transmit, to the speaker 161, an electrical signal converted from received audio data, and the speaker 161 converts the electrical signal into a sound signal for outputting.
  • the microphone 162 is configured to collect a sound signal, and may further convert the collected sound signal into an electrical signal.
  • the audio frequency circuit 160 receives the electrical signal, converts the electrical signal into audio data, outputs the audio data to the processor 170 for processing, and then sends the audio data to, for example, another terminal through the radio frequency unit 110, or the audio data is output to the memory 120 for further processing.
  • the audio frequency circuit may also include a headphone jack 163, configured to provide a connection interface between the audio frequency circuit and a headset.
  • the terminal 100 further includes the power supply 190 (such as a battery) that supplies power to each component.
  • the power supply may be logically connected to the processor 170 by using a power supply management system, to implement functions such as charging and discharging management and power consumption management by using the power supply management system.
  • the terminal 100 further includes the external interface 180.
  • the external interface may be a standard micro-USB port, or may be a multi-pin connector.
  • the external interface may be configured to connect the terminal 100 to another apparatus for communication, or may be configured to connect to a charger to charge the terminal 100.
  • the terminal 100 may further include a flash, a wireless fidelity (wireless fidelity, Wi-Fi) module, a Bluetooth module, various sensors, and the like. Details are not described herein. All of the methods described below may be applied to the terminal shown in FIG. 1 .
  • an embodiment of the present invention provides an image processing method.
  • a specific processing method procedure includes the following steps.
  • Step 31 Obtain N frames of images, where N is a positive integer greater than 2.
  • Step 32 Determine one of the N frames of images as a reference image, where the other N-1 frames of images are to-be-processed images.
  • N is 20
  • the first frame of image is a reference image
  • the other 19 frames of images are to-be-processed images
  • i in step 33 may be any one of 1 to 19.
  • Step 33 Obtain N-1 frames of ghost-free images based on the N-1 frames of to-be-processed images. Specifically, steps s331 to s334 may be performed on an i th frame in the N-1 frames, where values of i may be all positive integers that are not greater than N-1. In some embodiments, alternatively, only M frames of to-be-processed images may be used to obtain M frames of ghost-free images, where M is a positive integer less than N-1. Still using N-1 for description, refer to FIG. 3 .
  • s331 Perform registration on the i th frame of image and the reference image, to obtain an i th registered image.
  • s332 Obtain an i th difference image based on the i th registered image and the reference image.
  • s334 Perform fusion on the i th registered image and the reference image based on the i th ghost weight image, to obtain an i th frame of ghost-free image.
  • Step 34 Obtain a first target image based on the reference image and the N-1 frames of ghost-free images. Specifically, an averaging operation is performed on the reference image and the N-1 frames of ghost-free images, to obtain the first target image, and the averaging operation may also include some modifications to an average value, or include an average value of absolute values, or the like.
  • N images are photographed consecutively in current parameter settings after a photographing instruction is received, and this may be used as a replacement for step 31 in the following first snapshot mode, second snapshot mode, and third snapshot mode.
  • the terminal directly enters a snapshot mode, for example, the first snapshot mode, the second snapshot mode, or the third snapshot mode that is to be mentioned below.
  • the terminal does not need to detect an environment of a to-be-photographed scene, because there is a preset parameter rule (prestored locally in the terminal or a cloud server) for each snapshot mode.
  • each snapshot mode has corresponding ISO sensitivity and a corresponding exposure time, and certainly, there may be other performance parameters or the like.
  • the camera Upon entering a particular snapshot mode, the camera automatically adjusts to corresponding ISO sensitivity and a corresponding exposure time to photograph. Therefore, if the user directly uses the snapshot mode, the obtaining N images may be photographing the N images by using the ISO sensitivity and the exposure time corresponding to the snapshot mode, to perform subsequent image processing of the corresponding mode.
  • the camera of the terminal is in an auto mode or a smart mode, the camera needs to detect an environment of a to-be-photographed scene. If it is detected that a to-be-photographed image of the camera is a moving image, and that a current exposure time of the camera exceeds a safe time, and that the environment of the to-be-photographed scene is an extreme high-light environment, the first snapshot mode provided in the present invention is used.
  • the second snapshot mode or the third snapshot mode provided in the present invention is used. If none of the foregoing cases are detected, any one of the photographing modes supported by the terminal may be used to photograph. For a specific photographing procedure, refer to FIG. 4 .
  • the "camera” in this specification generally refers to a system in a terminal device capable of implementing a photographing function, and the system includes a camera lens and a necessary processing module and storage module to obtain and transfer an image, and may further include some processing function modules.
  • the "current exposure time” and “current ISO sensitivity” are respectively a corresponding exposure time and corresponding ISO sensitivity when the camera previews a data stream of the to-be-photographed image in initial conditions, and usually are related to properties and initial settings of the camera.
  • a corresponding exposure time and corresponding ISO sensitivity when the camera previews a data stream of the to-be-photographed image are also considered as the "current exposure time” and "current ISO sensitivity”; and case 1: the to-be-photographed image of the camera is a moving image; case 2: it is detected that the current exposure time of the camera exceeds a safe time; and case 3: it is detected that the environment of the to-be-photographed scene is an extreme high-light environment or a moderate high-light environment.
  • the to-be-photographed image is a moving image.
  • motion detection is performed on a preview data stream to analyze a photographing preview stream
  • detection is performed every x frames (a quantity x of interval frames is variable, and x is a positive integer), and a difference between a current frame of image detected and a previous frame of image detected is analyzed during each time of detection.
  • the two frames of images each may be divided into areas in a same division manner, for example, 64 areas per frame of image. If a relatively big difference exists in one or more than one area, the image is considered as a moving image.
  • the current exposure time and a safe shutter speed may be obtained by obtaining camera parameters.
  • the safe shutter speed is a property of a terminal camera. Generally, a snapshot mode is considered only when the current exposure time is greater than the safe shutter speed.
  • An extreme high-light scene is defined as follows: ISO sensitivity (denoted as ISO) and an exposure time (denoted as expo) of a preview image are less than a threshold each, that is, ISO ⁇ iso_th1 and expo ⁇ expo_th1.
  • ISO and expo of the preview image may be obtained by obtaining camera parameters, and iso_th1 and expo thl may be determined based on the user's specific requirements.
  • a moderate high-light scene is defined as follows: iso_th1 ⁇ ISO ⁇ iso_th2 and expo_th1 ⁇ expo ⁇ expo_th2.
  • iso_th2 and expo_th2 may also be determined based on the user's specific requirements.
  • a low-light scene is defined as follows: iso_th2 ⁇ ISO and expo_th2 ⁇ expo. It should be understood that division of the intervals is determined based on a user requirement, and it is allowable that the value intervals are discontinuous or overlap.
  • the following describes the first snapshot mode, the second snapshot mode, and the third snapshot mode in detail.
  • Step 31 is specifically: obtaining parameters such as current ISO sensitivity and a current exposure time of a camera; with a product of the current ISO sensitivity and the current exposure time of the camera unchanged, reducing the exposure time and increasing the ISO sensitivity at a preset ratio, to obtain a first exposure time and first ISO sensitivity, for example, the first exposure time being 1/2 or 1/4 of the original exposure time and the first ISO sensitivity being two or four times of the original ISO sensitivity, where a specific ratio may be adjusted according to a requirement of a user or a specified rule; and setting the exposure time and the ISO sensitivity of the camera to the first exposure time and the first ISO sensitivity respectively, and photographing N frames of images.
  • the following step is performing noise reduction on the N frames of images.
  • a photographing action may be triggered by the user pressing a shutter button.
  • the current exposure time and the current ISO sensitivity of the camera are already adjusted to the first exposure time and the first ISO sensitivity; and when the user presses the shutter button, the camera photographs the N images by using the first exposure time and the first ISO sensitivity, for subsequent processing.
  • the camera still maintains the current ISO sensitivity and the current exposure time; and when the user presses the shutter button, the current exposure time and the current ISO sensitivity of the camera are adjusted to the first exposure time and the first ISO sensitivity, and the camera photographs the N images by using the first exposure time and the first ISO sensitivity, for subsequent processing.
  • the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the first exposure time and the first ISO sensitivity.
  • Step 32 is specifically: determining one of the N frames of images as a reference image, where the other N-1 frames of images are to-be-processed images.
  • the first frame of image or a frame of image in the middle of the N frames of images is used as the reference image.
  • the first frame of image is used as an example for description of the following steps.
  • Step 33 is specifically: obtaining N-1 frames of ghost-free images based on the N-1 frames of to-be-processed images. This step may be further divided into many sub-steps. Steps s331 to s334 may be performed on an i th frame in the other N-1 frames, where values of i may be all positive integers that are not greater than N-1. In a specific implementation process, alternatively, some of the N-1 frames may be selected to obtain some frames of ghost-free images. For ease of description, all of the N-1 frames are selected to obtain ghost-free images in this embodiment.
  • a specific registration method may be: (1) performing feature extraction on the i th frame of image and the reference image in a same manner, to obtain a series of feature points, and making a feature description for each feature point; (2) matching feature points of the i th frame of image and the reference image, to obtain a series of feature point pairs, and eliminating outliers by using a RANSAC algorithm (the prior art); and (3) solving for a transformation matrix (for example, a homography matrix or an affine matrix) of the two images from the matched feature point pairs, and performing registration and alignment, based on the transformation matrix, on the i th frame image and the reference image to obtain the i th frame of registered image.
  • a transformation matrix for example, a homography matrix or an affine matrix
  • s332 is specifically: obtaining an i th difference image based on the i th registered image and the reference image. Specifically, differences between the i th frame of registered image and the reference image are calculated pixel by pixel, and a difference image of the two images is obtained by taking an absolute value of each value.
  • s333 is specifically: obtaining an i th ghost weight image based on the i th difference image. Specifically, a pixel exceeding a preset threshold in the difference image is set to M (for example, 255), a pixel not exceeding the preset threshold is set to N (for example, 0), and Gaussian smoothing is performed on the difference image after reassignment, to obtain the i th ghost weight image.
  • M for example, 255
  • N for example, 0
  • Gaussian smoothing is performed on the difference image after reassignment, to obtain the i th ghost weight image.
  • s334 is specifically: performing fusion on the i th registered image and the reference image based on the i th ghost weight image, to obtain an i th frame of ghost-free image. Specifically, based on the ghost weight image (ghost_mask in the following formula), pixel-by-pixel fusion is performed on the i th frame of registered image (image_i in the following formula) and the reference image (image_1 in the following formula), to obtain the i th frame of ghost-free image (no_ghost_mask).
  • Step 34 is specifically: performing an averaging operation on the reference image and the N-1 frames of ghost-free images, to obtain a first target image.
  • a pixel averaging algorithm or the like in the industry may be used.
  • the first target image is a final image obtained when the terminal executes the photographing mode.
  • the second snapshot mode is more complex than the first snapshot mode. Some steps are the same as the method of the snapshot mode 1. For a flowchart of the second snapshot mode, refer to FIG. 5 .
  • Step 41 Photograph one frame of first new image based on current ISO sensitivity and a current exposure time of a camera; with a product of the current ISO sensitivity and the current exposure time, reduce the exposure time and increase the ISO sensitivity at a preset ratio, to obtain a second exposure time and second ISO sensitivity; and set the exposure time and the ISO sensitivity of the camera to the second exposure time and the second ISO sensitivity respectively, and photograph N frames of images.
  • a photographing action may be triggered by a user pressing a shutter button.
  • the shutter button when the user presses the shutter button, one frame of first new image is obtained by using the current ISO sensitivity and the current exposure time; the current exposure time and the current ISO sensitivity are adjusted to the second exposure time and the second ISO sensitivity, and the N images are photographed in this condition; and N+1 images are obtained in total, for subsequent processing.
  • the current exposure time and the current ISO sensitivity are set to the second exposure time and the second ISO sensitivity, and the N images are photographed in this condition; then one frame of first new image is obtained still by using the current ISO sensitivity and the current exposure time; and N+1 images are obtained in total, for subsequent processing.
  • the image in a preview image data stream, the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the second exposure time and the second ISO sensitivity.
  • Step 42 Obtain a first target image by using the solution of the first snapshot mode (step 31 to step 34) for the N frames of images obtained in the previous step.
  • the second ISO sensitivity, the second exposure time, and some of the foregoing adjustable thresholds may vary accordingly with a scene.
  • Step 43 Obtain a second target image based on the first target image and the first new image.
  • step 43 may include but is not limited to the following two implementations:
  • s4311 Perform registration on the first new image and a reference image (the same as the reference image selected when the first target image is obtained), to obtain a first registered image.
  • s4312 Obtain a first difference image based on the first registered image and the first target image.
  • s4313 Obtain a first ghost weight image based on the first difference image.
  • s4314 Perform fusion on the first registered image and the first target image based on the first ghost weight image, to obtain a first ghost-free image.
  • the s4315 Perform pixel-value weighted fusion based on the first ghost-free image and the first target image, to obtain the second target image.
  • the fusion may include four implementations: time domain fusion s4315(1), time domain fusion s4315(3), frequency domain fusion s4315(2), and frequency domain fusion s4315(4).
  • Time domain fusion s4315(1) Guided filtering is separately performed on the first target image and the first ghost-free image, to filter out short-frame information (an existing mature algorithm), where the first target image and the first ghost-free image after the guided filtering is performed are denoted as fusion_gf and noghost_gf. Pixel-value weighted fusion is performed on fusion_gf and noghost_gf.
  • v is magnitude of corresponding noise specified according to a current ISO stop, and is a constant
  • W is a weight
  • a value range of W is [0, 1).
  • Target details are added back to a fused image pixel by pixel.
  • a target detail is a larger value of details that are filtered out from the first target image and the first ghost-free image during guided filtering at the pixel, to add more image details, thereby obtaining the second target image.
  • Time domain fusion s4315(3) Downsampling is separately performed on the first target image (denoted as fusion) and the first ghost-free image (denoted as noghost), and both a width and a height are downsampled by two times, to separately obtain a downsampled image of the first target image and a downsampled image of the first ghost-free image, where the two downsampled images are denoted as fusionx4 and noghostx4 respectively. Then upsampling is performed on fusionx4 and noghostx4, and both a width and a height are upsampled by two times, to separately obtain two images that are consistent in size with those before sampling is performed, where the two images are denoted as fusion' and noghost'.
  • Differences between fusion and fusion' are calculated pixel by pixel, to obtain a sampling error image of the first target image, where the sampling error image is denoted as fusion_se.
  • Differences between noghost and noghost' are calculated pixel by pixel, to obtain a sampling error image of the first ghost-free image, where the sampling error image is denoted as noghost_se.
  • Guided filtering is separately performed on fusionx4 and noghostx4, to obtain two filtered images, where the two filtered images are denoted as fusion_gf and noghost_gf.
  • a pixel-value weighted fusion is performed on fusion_gf and noghost_gf to obtain a fused image, which is denoted as Fusion.
  • a specific fusion formula is the same as the formula in s4315(1). Details that are filtered out from the first target image during guided filtering are added back to the fused image pixel by pixel, upsampling is performed on the image, and both a width and a height are upsampled by two times, where the upsampled image is denoted as FusionUp. A larger value is selected pixel by pixel from the two sampling error images fusion _se and noghost se, and is added pixel by pixel to FusionUp, to add more image details, thereby obtaining the second target image.
  • Frequency domain fusion s4315(2) Guided filtering (an existing mature algorithm) is separately performed on the first target image and the first ghost-free image. Fourier transform is separately performed on the two filtered images, and corresponding amplitudes are calculated. A ratio of the amplitudes is used as a weight, and Fourier spectrums corresponding to the two images are fused. A specific fusion formula is similar to that for time domain fusion. Inverse Fourier transform is performed on the fused spectrum to obtain a fused image. Target details are added back to a fused image pixel by pixel. For any pixel, a target detail is a larger value of details that are filtered out from the first target image and the first ghost-free image during guided filtering at the pixel, to add more image details, thereby obtaining the second target image.
  • Frequency domain fusion s4315(4) Downsampling is separately performed on the first target image (denoted as fusion) and the first ghost-free image (denoted as noghost), and both a width and a height are downsampled by two times, to separately obtain a downsampled image of the first target image and a downsampled image of the first ghost-free image, where the two downsampled images are denoted as fusionx4 and noghostx4 respectively.
  • upsampling is performed on fusionx4 and noghostx4, and both a width and a height are upsampled by two times, to separately obtain two images that are consistent in size with those before sampling is performed, where the two images are denoted as fusion' and noghost'.
  • Differences between fusion and fusion' are calculated pixel by pixel, to obtain a sampling error image of the first target image, where the sampling error image is denoted as fusion_se.
  • Differences between noghost and noghost' are calculated pixel by pixel, to obtain a sampling error image of the first ghost-free image, where the sampling error image is denoted as noghost_se.
  • fusion_gf and noghost_gf Guided filtering (an existing mature algorithm) is separately performed on fusionx4 and noghostx4, and two filtered images are denoted as fusion_gf and noghost_gf.
  • Fourier transform is separately performed on the two filtered images, and corresponding amplitudes are calculated. A ratio of the amplitudes is used as a weight, and Fourier spectrums corresponding to the two images are fused.
  • a specific fusion formula is similar to that for time domain fusion.
  • Inverse Fourier transform is performed on the fused spectrum to obtain a fused image.
  • Details that are filtered out from the first target image during guided filtering are added back to the fused image pixel by pixel, and upsampling of two times in both a width and a height is performed on the image to which details are added back, where the upsampled image is denoted as FusionUp.
  • FusionUp A larger value is selected pixel by pixel from the two sampling error images fusion _se and noghost se, and is added pixel by pixel to FusionUp, to add more image details, thereby obtaining the second target image.
  • a specific algorithm used in s4311 to s4314 is the same as that used in s331 to s334, and a difference mainly lies in a replacement of an input image. Details are not described herein again.
  • s4321 Perform registration on the first new image and the first target image, to obtain a second registered image.
  • s4322 Obtain a second difference image based on the second registered image and the first target image.
  • s4323 Obtain a second ghost weight image based on the second difference image.
  • s4324 Perform fusion on the second registered image and the first target image based on the second ghost weight image, to obtain a second ghost-free image.
  • the s4325 Perform pixel-value weighted fusion based on the second ghost-free image and the first target image, to obtain the second target image.
  • the fusion may include two implementations: time domain fusion and frequency domain fusion, and refer to any one of the foregoing time domain fusion s4315(1) and s4315(3), and frequency domain fusion s4315(2) and s4315(4).
  • An algorithm is the same except for a replacement of an input image, and details are not described herein again.
  • the third snapshot mode is more complex than the first snapshot mode, and may be understood as a replacement for the second snapshot mode in a sense.
  • the second and the third snapshot modes are usually used in a moderate high-light scene.
  • Step 51 Photograph one frame of second new image based on current ISO sensitivity and current exposure time of a camera; and with the current ISO sensitivity of the camera unchanged, set the current exposure time to a third exposure time that is shorter, and photograph N frames of images.
  • a photographing action may be triggered by a user pressing a shutter button.
  • one frame of second new image is obtained by using the current ISO sensitivity and the current exposure time; with the current ISO sensitivity of the camera unchanged, the current exposure time is set to the third exposure time that is shorter, and the N images are photographed in this condition; and N+1 images are obtained in total, for subsequent processing.
  • the current exposure time is set to the third exposure time that is shorter, and the N images are photographed in this condition; then one frame of second new image is obtained still by using the current ISO sensitivity and the current exposure time; and N+1 images are obtained in total, for subsequent processing.
  • the image may be displayed with the current ISO sensitivity and the current exposure time, or may be displayed with the third exposure time and the current ISO sensitivity.
  • Step 52 Obtain a first target image by using the solution of the first snapshot mode (step 31 to step 34) for the N frames of images obtained in the previous step. It should be understood that the third exposure time and some adjustable thresholds may vary accordingly with a scene.
  • Step 53 Obtain a third target image based on the first target image and the second new image.
  • step 53 may include but is not limited to the following two implementations:
  • s5311 Perform luminance correction on the first target image based on the second new image, to obtain a fourth target image. Specifically, statistics about histograms of the second new image and the first target image are separately collected, and statistics about cumulative histograms are collected based on the histograms; the cumulative histogram of the first target image is mapped to the cumulative histogram of the second new image, to obtain a mapping relationship curve; smoothing is performed on the mapping relationship curve, to prevent the curve from being convex or concave with a relatively large slope; and luminance of the first target image is increased based on the mapping curve.
  • the luminance correction algorithm is an existing mature algorithm, and details are not described.
  • s5312 Perform registration on the second new image and a reference image (the same as the reference image selected when the first target image is obtained), to obtain a third registered image.
  • s5313 Obtain a third difference image based on the third registered image and the fourth target image.
  • s5314 Obtain a third ghost weight image based on the third difference image.
  • s5315 Perform fusion on the third registered image and the fourth target image based on the third ghost weight image, to obtain a third ghost-free image.
  • s5316 Perform pixel-value weighted fusion based on the third ghost-free image and the fourth target image, to obtain a fifth target image.
  • a fusion algorithm refer to any one of time domain fusion s4315(1), time domain fusion s4315(3), frequency domain fusion s4315(2), or frequency domain fusion s4315(4).
  • s5317 Perform pyramid fusion on the fifth target image and the first target image, to obtain the third target image. Specifically, Laplacian pyramids of the fifth target image and the first target image are separately built, a weight image of the image fusion is built, normalization and smoothing are performed on the weight image, and a Gaussian pyramid is built for the normalized and smoothed weight image; based on a weight setting of each level of pyramids, the pyramids of all the images are fused at the corresponding level to obtain a synthetic pyramid; starting from a top level of the Laplacian pyramid, the synthetic pyramid is rebuilt according to a reverse process of pyramid generation, and each level of information is added back one by one, to restore the fused image.
  • Pyramid fusion processing is an existing mature algorithm, and details are not described.
  • s5321 Perform luminance correction on the first target image based on the second new image, to obtain a fourth target image. Specifically, statistics about histograms of the second new image and the first target image are separately collected, and statistics about cumulative histograms are collected based on the histograms; the cumulative histogram of the first target image is mapped to the cumulative histogram of the second new image, to obtain a mapping relationship curve; smoothing is performed on the mapping relationship curve, to prevent the curve from being convex or concave with a relatively large slope; and luminance of the first target image is increased based on the mapping curve.
  • the luminance correction algorithm is an existing mature algorithm, and details are not described.
  • s5322 Perform registration on the second new image and a reference image (the same as the reference image selected when the first target image is obtained), to obtain a third registered image.
  • s5323 Obtain a third difference image based on the third registered image and the fourth target image.
  • s5324 Obtain a third ghost weight image based on the third difference image.
  • s5325 Perform fusion on the third registered image and the fourth target image based on the third ghost weight image, to obtain a third ghost-free image.
  • s5326 Perform pixel-value weighted fusion based on the third ghost-free image and the fourth target image, to obtain a fifth target image.
  • a fusion algorithm refer to one of time domain fusion s4315(1), time domain fusion s4315(3), frequency domain fusion s4315(2), and frequency domain fusion s4315(4).
  • s5327 Perform pyramid fusion on the fifth target image and the first target image, to obtain the third target image. Specifically, Laplacian pyramids of the fifth target image and the first target image are separately built, a weight image of the image fusion is built, normalization and smoothing are performed on the weight image, and a Gaussian pyramid is built for the normalized and smoothed weight image; based on a weight setting of each level of pyramids, the pyramids of all the images are fused at the corresponding level to obtain a synthetic pyramid; starting from a top level of the Laplacian pyramid, the synthetic pyramid is rebuilt according to a reverse process of pyramid generation, and each level of information is added back one by one, to restore the fused image.
  • Pyramid fusion processing is an existing mature algorithm, and details are not described.
  • the present invention provides an image processing method, and the method can provide a snapshot mode for the camera.
  • the user can snap a sharp image in different scenes, catering to the user's psychology of taking a snapshot; and the user can take a snapshot to record daily life anytime and anywhere, thereby greatly improving user experience.
  • an embodiment of the present invention provides an image processing apparatus 700.
  • the apparatus 700 may be applied to various photographing devices. As shown in FIG. 7 , the apparatus 700 includes an obtaining module 701, a determining module 702, a ghost removal module 703, and an averaging operation module 704.
  • the obtaining module 701 is configured to obtain N frames of images.
  • the obtaining module 701 may be implemented by a processor invoking a program instruction in a memory to control a camera lens to obtain the images.
  • the determining module 702 is configured to determine one of the N frames of images as a reference image, where the other N-1 frames of images are to-be-processed images.
  • the determining module 702 may be implemented by the processor invoking a program instruction in the memory or an externally entered program instruction.
  • the ghost removal module 703 is configured to perform the following step 1 to step 4 on an i th frame of image in the N-1 frames of to-be-processed images, to obtain N-1 frames of ghost-free images, where values of i are all positive integers that are not greater than N-1, and
  • the ghost removal module 703 may be implemented by the processor, and may perform corresponding computing by invoking data and an algorithm in a local memory or a cloud server.
  • the averaging operation module 704 is configured to perform an averaging operation on the reference image and the N-1 frames of ghost-free images, to obtain a first target image.
  • the averaging operation module 704 may be implemented by the processor, and may perform corresponding computing by invoking data and an algorithm in the local memory or the cloud server.
  • the obtaining module 701 is specifically configured to perform the method mentioned in step 31 and a method that can be equivalently replaced; the determining module 702 is specifically configured to perform the method mentioned in step 32 and a method that can be equivalently replaced; the ghost removal module 703 is specifically configured to perform the method mentioned in step 33 and a method that can be equivalently replaced; and the averaging operation module 704 is specifically configured to perform the method mentioned in step 34 and a method that can be equivalently replaced.
  • the apparatus 700 further includes a detection module 705.
  • the detection module 705 is configured to: when it is detected that the following three cases exist at the same time, control the obtaining module to obtain the N frames of images according to the following first obtaining manner, where
  • the first obtaining manner is: with a product of the current ISO sensitivity and the current exposure time of the camera unchanged, reducing the exposure time and increasing the ISO sensitivity at a preset ratio, to obtain a first exposure time and first ISO sensitivity; and setting the current exposure time and the current ISO sensitivity of the camera to the first exposure time and the first ISO sensitivity respectively, and photographing the N frames of images.
  • the detection module 705 is configured to: when it is detected that the following three cases exist at the same time, control the obtaining module to obtain the N frames of images according to the following second or third obtaining manner, where
  • the second obtaining manner is: with a product of the current ISO sensitivity and the current exposure time of the camera unchanged, reducing the exposure time and increasing the ISO sensitivity at a preset ratio, to obtain a second exposure time and second ISO sensitivity; and setting the current exposure time and the current ISO sensitivity of the camera to the second exposure time and the second ISO sensitivity respectively, and photographing the N frames of images.
  • the third obtaining manner is: photographing one frame of second new image based on the current ISO sensitivity and the current exposure time of the camera; and with the current ISO sensitivity of the camera unchanged, setting the current exposure time to a third exposure time that is shorter, and photographing the N frames of images.
  • the apparatus 700 may further include a fusion module 706, configured to obtain a second target image based on a first target image and the first new image; or configured to obtain a third target image based on the first target image and the second new image.
  • a fusion module 706 configured to obtain a second target image based on a first target image and the first new image; or configured to obtain a third target image based on the first target image and the second new image.
  • the fusion module 706 is specifically configured to: perform registration on the first new image and the reference image, to obtain a first registered image; obtain a first difference image based on the first registered image and the first target image; obtain a first ghost weight image based on the first difference image; perform fusion on the first registered image and the first target image based on the first ghost weight image, to obtain a first ghost-free image; and perform pixel-value weighted fusion based on the first ghost-free image and the first target image, to obtain the second target image.
  • the fusion module 706 is specifically configured to perform the method mentioned in manner (1) for step 43 and a method that can be equivalently replaced.
  • the fusion module 706 is specifically configured to: perform registration on the first new image and the first target image, to obtain a second registered image; obtain a second difference image based on the second registered image and the first target image; obtain a second ghost weight image based on the second difference image; perform fusion on the second registered image and the first target image based on the second ghost weight image, to obtain a second ghost-free image; and perform pixel-value weighted fusion based on the second ghost-free image and the first target image, to obtain the second target image.
  • the fusion module 706 is specifically configured to perform the method mentioned in manner (2) for step 43 and a method that can be equivalently replaced.
  • the fusion module 706 is specifically configured to: perform luminance correction on the first target image based on the second new image, to obtain a fourth target image; perform registration on the second new image and the reference image, to obtain a third registered image; obtain a third difference image based on the third registered image and the fourth target image; obtain a third ghost weight image based on the third difference image; perform fusion on the third registered image and the fourth target image based on the third ghost weight image, to obtain a third ghost-free image; perform pixel-value weighted fusion based on the third ghost-free image and the fourth target image, to obtain a fifth target image; and perform pyramid fusion on the fifth target image and the first target image, to obtain the third target image.
  • the fusion module 706 is specifically configured to perform the method mentioned in manner (1) for step 53 and a method that can be equivalently replaced.
  • the fusion module 706 is specifically configured to: perform luminance correction on the first target image based on the second new image, to obtain a fourth target image; perform registration on the second new image and the fourth target image, to obtain a fourth registered image; obtain a fourth difference image based on the fourth registered image and the fourth target image; obtain a fourth ghost weight image based on the fourth difference image; perform fusion on the fourth registered image and the fourth target image based on the fourth ghost weight image, to obtain a fourth ghost-free image; perform pixel-value weighted fusion based on the fourth ghost-free image and the fourth target image, to obtain a sixth target image; and perform pyramid fusion on the sixth target image and the first target image, to obtain the third target image.
  • the fusion module 706 is specifically configured to perform the method mentioned in manner (2) for step 53 and a method that can be equivalently replaced.
  • the terminal directly enters a snapshot mode, for example, the first snapshot mode, the second snapshot mode, or the third snapshot mode that is mentioned above.
  • the terminal does not need to detect an environment of a to-be-photographed scene, because there is a preset parameter rule (prestored locally in the terminal or a cloud server) for each snapshot mode.
  • each snapshot mode has corresponding ISO sensitivity and a corresponding exposure time, and certainly, there may be other performance parameters or the like.
  • the obtaining module automatically adjusts to corresponding ISO sensitivity and a corresponding exposure time to photograph. Therefore, if the user directly uses the snapshot mode, the obtaining module photographs the N images by using the corresponding ISO sensitivity and the corresponding exposure time, to perform subsequent image processing in a corresponding mode.
  • the foregoing detection module 705 and fusion module 706 may be implemented by the processor invoking a program instruction in the memory or a program instruction on a cloud.
  • the present invention provides the image processing apparatus 700.
  • the user can snap a sharp image in different scenes, catering to the user's psychology of taking a snapshot; and the user can take a snapshot to record daily life anytime and anywhere, thereby greatly improving user experience.
  • each of the foregoing modules may be a separate processing element, or may be integrated on a chip of a terminal, or may be stored in a storage element of a controller in a form of program code, and a processing element of the processor invokes and executes a function of each of the foregoing modules.
  • the modules may be integrated or may be implemented independently.
  • the processing element may be an integrated circuit chip and has a signal processing capability.
  • steps in the foregoing methods or the foregoing modules may be implemented by using a hardware integrated logical circuit in the processor element, or by using instructions in a form of software.
  • the processing element may be a general-purpose processor, such as a central processing unit (English: central processing unit, CPU for short), or may be one or more integrated circuits configured to implement the foregoing methods, for example, one or more application-specific integrated circuits (English: application-specific integrated circuit, ASIC for short), or one or more microprocessors (English: digital signal processor, DSP for short), or one or more field programmable gate arrays (English: field-programmable gate array, FPGA for short).
  • the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may use a form of hardware only embodiments, software only embodiments, or embodiments with a combination of software and hardware. Moreover, the present invention may use a form of a computer program product that is implemented on one or more computer-usable storage media (including but not limited to a disk memory, a CD-ROM, an optical memory, and the like) that include computer-usable program code.
  • computer-usable storage media including but not limited to a disk memory, a CD-ROM, an optical memory, and the like
  • These computer program instructions may be provided for a general-purpose computer, a dedicated computer, an embedded processor, or a processor of any other programmable data processing device to generate a machine, so that the instructions executed by a computer or a processor of any other programmable data processing device generate an apparatus for implementing a specific function in one or more processes in the flowcharts and/or in one or more blocks in the block diagrams.
  • These computer program instructions may be stored in a computer readable memory that can instruct the computer or any other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory generate an artifact that includes an instruction apparatus.
  • the instruction apparatus implements a specific function in one or more processes in the flowcharts and/or in one or more blocks in the block diagrams.
  • These computer program instructions may be loaded onto a computer or another programmable data processing device, so that a series of operations and steps are performed on the computer or the another programmable device, thereby generating computer-implemented processing. Therefore, the instructions executed on the computer or the another programmable device provide steps for implementing a specific function in one or more processes in the flowcharts and/or in one or more blocks in the block diagrams.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Studio Devices (AREA)
  • Geometry (AREA)
EP18866515.2A 2017-10-13 2018-10-12 Bildverarbeitungsverfahren und -vorrichtung und -einrichtung Active EP3686845B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
CN201710954301 2017-10-13
CN201710959936.0A CN109671106B (zh) 2017-10-13 2017-10-16 一种图像处理方法、装置与设备
PCT/CN2018/109951 WO2019072222A1 (zh) 2017-10-13 2018-10-12 一种图像处理方法、装置与设备

Publications (3)

Publication Number Publication Date
EP3686845A1 true EP3686845A1 (de) 2020-07-29
EP3686845A4 EP3686845A4 (de) 2020-12-02
EP3686845B1 EP3686845B1 (de) 2023-07-26

Family

ID=66139200

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18866515.2A Active EP3686845B1 (de) 2017-10-13 2018-10-12 Bildverarbeitungsverfahren und -vorrichtung und -einrichtung

Country Status (3)

Country Link
US (1) US11445122B2 (de)
EP (1) EP3686845B1 (de)
CN (1) CN109671106B (de)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112565595A (zh) * 2020-11-25 2021-03-26 西安闻泰电子科技有限公司 图像抖动消除方法、装置、电子设备和存储介质
CN116188343A (zh) * 2023-02-27 2023-05-30 上海玄戒技术有限公司 一种图像融合方法、装置、电子设备、芯片及介质
CN117710221A (zh) * 2022-09-05 2024-03-15 英特灵达信息技术(深圳)有限公司 一种图像处理方法和装置

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9955084B1 (en) * 2013-05-23 2018-04-24 Oliver Markus Haynold HDR video camera
WO2020082268A1 (en) * 2018-10-24 2020-04-30 Siemens Healthcare Gmbh Generating a medical result image
CN110428391B (zh) * 2019-08-02 2022-05-03 格兰菲智能科技有限公司 用以去除鬼影假象的图像融合方法及装置
CN110602467B (zh) * 2019-09-09 2021-10-08 Oppo广东移动通信有限公司 图像降噪方法、装置、存储介质及电子设备
CN112927144A (zh) * 2019-12-05 2021-06-08 北京迈格威科技有限公司 图像增强方法、图像增强装置、介质和电子设备
CN112288642B (zh) * 2020-09-21 2025-04-01 原力图新(重庆)科技有限公司 鬼影检测方法、图像融合方法及对应装置
CN114338956B (zh) * 2020-09-30 2024-07-23 北京小米移动软件有限公司 图像处理方法、图像处理装置及存储介质
CN112598716B (zh) * 2020-12-01 2024-04-05 浙江大华技术股份有限公司 一种多模态图像的配准方法、装置及系统
CN114650361B (zh) 2020-12-17 2023-06-06 北京字节跳动网络技术有限公司 拍摄模式确定方法、装置、电子设备和存储介质
WO2022141265A1 (zh) * 2020-12-30 2022-07-07 华为技术有限公司 图像处理方法和设备
CN112785537B (zh) * 2021-01-21 2024-07-30 北京小米松果电子有限公司 图像处理方法、装置以及存储介质
CN115706863B (zh) * 2021-08-12 2023-11-21 荣耀终端有限公司 视频处理方法、装置、电子设备和存储介质
CN113691761B (zh) * 2021-08-30 2023-01-06 联想(北京)有限公司 通信控制方法、装置及电子设备
CN113706421B (zh) * 2021-10-27 2022-02-22 深圳市慧鲤科技有限公司 一种图像处理方法及装置、电子设备和存储介质
CN116703790A (zh) * 2022-02-28 2023-09-05 北京小米移动软件有限公司 图像处理方法、装置、电子设备及存储介质
US20230342889A1 (en) * 2022-04-26 2023-10-26 Qualcomm Incorporated Frequency domain edge enhancement of image capture
CN117314802A (zh) * 2022-06-16 2023-12-29 普里露尼库斯新加坡私人有限公司 图像处理方法、计算机装置及存储介质
CN115035013B (zh) * 2022-06-22 2025-05-13 Oppo广东移动通信有限公司 图像处理方法、图像处理装置、终端及可读存储介质
CN117440253B (zh) * 2023-12-22 2024-07-09 荣耀终端有限公司 图像处理方法和相关装置
CN120751271A (zh) * 2024-08-30 2025-10-03 荣耀终端股份有限公司 图像处理方法及相关装置

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SG102627A1 (en) 1996-11-28 2004-03-26 Nikon Corp Lithographic device
US7509038B2 (en) * 2005-09-29 2009-03-24 Seiko Epson Corporation Determining maximum exposure time to limit motion blur during image capture
EP1934939A1 (de) * 2005-10-12 2008-06-25 Active Optics Pty Ltd. Verfahren zum erzeugen eines bildes auf der basis mehrerer einzelbilder, bildverarbeitungssystem und digitalkamera
KR101225482B1 (ko) 2012-02-15 2013-01-23 인텔 코오퍼레이션 디지털 영상 처리 방법, 장치, 및 컴퓨터로 판독 가능한 기록 매체
JP2013258444A (ja) * 2012-06-11 2013-12-26 Olympus Corp 画像処理装置、画像処理方法及びプログラム
TW201437965A (zh) * 2013-03-28 2014-10-01 Novatek Microelectronics Corp 影像模糊防止方法及其影像處理晶片
US9262684B2 (en) * 2013-06-06 2016-02-16 Apple Inc. Methods of image fusion for image stabilization
US9491360B2 (en) * 2013-06-06 2016-11-08 Apple Inc. Reference frame selection for still image stabilization
CN104349066B (zh) * 2013-07-31 2018-03-06 华为终端（东莞）有限公司 一种生成高动态范围图像的方法、装置
WO2017173585A1 (zh) * 2016-04-05 2017-10-12 华为技术有限公司 一种拍照方法及终端
CN105931213B (zh) 2016-05-31 2019-01-18 南京大学 基于边缘检测和帧差法的高动态范围视频去鬼影的方法
CN106506981B (zh) 2016-11-25 2019-04-19 阿依瓦（北京）技术有限公司 生成高动态范围图像的设备和方法

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112565595A (zh) * 2020-11-25 2021-03-26 西安闻泰电子科技有限公司 图像抖动消除方法、装置、电子设备和存储介质
CN112565595B (zh) * 2020-11-25 2022-07-01 西安闻泰电子科技有限公司 图像抖动消除方法、装置、电子设备和存储介质
CN117710221A (zh) * 2022-09-05 2024-03-15 英特灵达信息技术(深圳)有限公司 一种图像处理方法和装置
CN116188343A (zh) * 2023-02-27 2023-05-30 上海玄戒技术有限公司 一种图像融合方法、装置、电子设备、芯片及介质

Also Published As

Publication number Publication date
US20200244862A1 (en) 2020-07-30
CN109671106A (zh) 2019-04-23
CN109671106B (zh) 2023-09-05
EP3686845B1 (de) 2023-07-26
EP3686845A4 (de) 2020-12-02
US11445122B2 (en) 2022-09-13

Similar Documents

Publication Publication Date Title
US11445122B2 (en) Image processing method and apparatus, and device
US11206352B2 (en) Shooting method, apparatus, and device
EP3531689B1 (de) Verfahren und vorrichtung zur optischen bilderzeugung
CN112153301B (zh) 拍摄方法和电子设备
US20250240538A1 (en) Signal Processing Method, Apparatus, and Device, Storage Medium, and Computer Program
EP4123574B1 (de) Bildfusionsverfahren und -vorrichtung, speichermedium und mobiles endgerät
WO2018219274A1 (zh) 降噪处理方法、装置、存储介质及终端
US11363187B2 (en) Focusing method and apparatus applied to terminal device, and terminal device
CN112468722B (zh) 一种拍摄方法、装置、设备及存储介质
CN113132643B (zh) 图像处理方法及相关产品
WO2019072222A1 (zh) 一种图像处理方法、装置与设备

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200423

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

A4 Supplementary search report drawn up and despatched

Effective date: 20201029

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 7/30 20170101AFI20201023BHEP

Ipc: G06T 5/50 20060101ALI20201023BHEP

Ipc: G06T 5/00 20060101ALI20201023BHEP

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230317

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018054250

Country of ref document: DE

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230727

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230726

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1592821

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231126

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231127

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231026

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231126

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231027

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018054250

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20231031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231012

26N No opposition filed

Effective date: 20240429

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231031

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20181012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20181012

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20250904

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250908

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230726

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250902

Year of fee payment: 8