WO2020171450A1 - Dispositif électronique et procédé de génération carte de profondeur - Google Patents

Dispositif électronique et procédé de génération carte de profondeur Download PDF

Info

Publication number
WO2020171450A1
WO2020171450A1 PCT/KR2020/001822 KR2020001822W WO2020171450A1 WO 2020171450 A1 WO2020171450 A1 WO 2020171450A1 KR 2020001822 W KR2020001822 W KR 2020001822W WO 2020171450 A1 WO2020171450 A1 WO 2020171450A1
Authority
WO
WIPO (PCT)
Prior art keywords
incident light
pixel array
electronic device
boundary
light
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2020/001822
Other languages
English (en)
Korean (ko)
Inventor
볼로치뉴크앤드리
계용찬
리모노프알렉산더
송인선
최지환
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2020171450A1 publication Critical patent/WO2020171450A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single two-dimensional [2D] image sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects

Definitions

  • Embodiments disclosed in this document relate to an electronic device and a method for generating a depth map.
  • the image sensor may include at least one or more pixels.
  • a pixel of the image sensor may generate an image signal by sensing incident light incident on a lens and converting photons corresponding to a specific color into electrons.
  • the 3D image data may include depth map information including distance information of a subject.
  • the 3D image data can be used in various technical fields, such as motion detection based on images and user recognition based on images. In order to generate more accurate 3D image data, more accurate depth map generation methods are being studied.
  • the electronic device may use a plurality of sensors capable of generating stereoscopic information corresponding to binocular parallax.
  • the electronic device may generate a depth map based on the reflection time of the reflected light.
  • the electronic device may generate the depth map using both stereoscopic information and echo time.
  • different types of distance information eg, stereoscopic information and reverberation time information
  • sensors for sensing different types of distance information may have separate hardware configurations.
  • the electronic device may perform complex procedures to generate a depth map based on a combination of different types of distance information.
  • An electronic device includes: a lens assembly, a first pixel array aligned along a first direction, a second pixel array aligned along the first direction, the lens assembly and the first pixel A first microlens disposed between arrays and between the lens assembly and the second pixel array and covering the first pixel array and the second pixel array, a processor, and a memory operatively connected to the processor.
  • the memory includes, when executed, the processor causes the first incident light to be collected through the first microlens among incident light incident on the lens assembly by reflecting light emitted from at least one light emitting element from a subject.
  • Identifying the first boundary of the subject using a first image signal generated by sensing by the first pixel array and a second image signal generated by sensing the first incident light by the second pixel array Provides boundary information, and provides first distance information for identifying a first distance to a first point of the subject by comparing the light and a second incident light of the incident light, and the first distance information and the boundary information It stores instructions to create a depth map based on.
  • the method according to an exemplary embodiment disclosed in the present document includes light emitted from at least one light emitting device and the light reflected from a subject to a lens assembly by using an edge detection circuitry of an electronic device.
  • An operation of identifying the first boundary of the subject and providing boundary information based on the intensity of the first incident light among the incident light, using a distance measurement circuitry of the electronic device, Providing first distance information for identifying a first distance to a first point of the subject by comparing the second incident light with and based on the first distance information and the boundary information, the depth map It may include an operation of generating.
  • the electronic device includes, based on the intensity of the first incident light among incident light incident on the lens assembly by reflecting light emitted from at least one light emitting device from a subject, the A plenoptic circuitry that identifies a first boundary of a subject to provide boundary information, and a first to identify a first distance to a first point of the subject by comparing the light with a second incident light of the incident light
  • a time of flight (TOF) circuit for providing distance information
  • a processor and a memory operatively connected to the processor, the memory configured to cause the processor to, when executed, cause the first distance information and the boundary Instructions for generating a depth map using the information may be stored.
  • TOF time of flight
  • distance information for identifying a distance to a point of a subject and border information for identifying a boundary of a subject are separately provided by using incident light incident on the subject by reflecting light emitted from the light emitting device.
  • incident light incident on the subject By reflecting, an elaborate depth map capable of identifying the boundary of the subject can be implemented.
  • FIG. 1 is a block diagram of an electronic device in a network environment according to an embodiment disclosed in this document.
  • FIG. 2 is a diagram illustrating an electronic device according to an embodiment disclosed in this document.
  • 3A is a view for explaining the lens assembly and the image sensor of FIG. 2.
  • FIG. 3B is a diagram illustrating an arrangement of a lens assembly, a micro lens array, and an image sensor of FIG. 2.
  • FIG. 4 is a diagram for describing the lens assembly and the image sensor of FIG. 2.
  • 5A is a diagram illustrating the processor of FIG. 2.
  • FIG. 5B is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information on a subject according to an exemplary embodiment disclosed in the present document.
  • FIG. 5C is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information about a subject according to an embodiment disclosed in this document.
  • FIG. 6 is a flowchart illustrating a method of generating a depth map by an electronic device according to an embodiment disclosed in this document.
  • FIG. 7 is a diagram for explaining incident light according to an embodiment disclosed in this document.
  • FIG. 8 is a view for explaining the operation of FIG. 6.
  • FIG. 9 is a diagram for explaining the operation of FIG. 6.
  • 10A is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.
  • 10B is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.
  • 10C is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.
  • 10D is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.
  • FIG. 1 is a block diagram of an electronic device 101 in a network environment 100 according to an embodiment disclosed in this document.
  • the electronic device 101 communicates with the electronic device 102 through a first network 198 (for example, a short-range wireless communication network), or a second network 199 It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network).
  • the electronic device 101 may communicate with the electronic device 104 through the server 108.
  • the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) Can be included.
  • a sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 .
  • the sensor module 176 eg, a fingerprint sensor, an iris sensor, or an illuminance sensor
  • the display device 160 eg, a display
  • the processor 120 for example, executes software (eg, a program 140) to implement at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to an embodiment, as at least a part of data processing or operation, the processor 120 stores commands or data received from other components (eg, the sensor module 176 or the communication module 190) to the volatile memory 132 The command or data stored in the volatile memory 132 may be processed and result data may be stored in the nonvolatile memory 134.
  • software eg, a program 140
  • the processor 120 stores commands or data received from other components (eg, the sensor module 176 or the communication module 190) to the volatile memory 132
  • the command or data stored in the volatile memory 132 may be processed and result data may be stored in the nonvolatile memory 134.
  • the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and an auxiliary processor 123 (eg, a graphics processing unit, an image signal processor) that can be operated independently or together. , A sensor hub processor, or a communication processor). Additionally or alternatively, the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.
  • a main processor 121 eg, a central processing unit or an application processor
  • an auxiliary processor 123 eg, a graphics processing unit, an image signal processor
  • the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for a designated function.
  • the secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.
  • the coprocessor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, an application is executed). ) While in the state, together with the main processor 121, at least one of the components of the electronic device 101 (for example, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the functions or states related to. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.
  • an image signal processor or a communication processor may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.
  • the memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176).
  • the data may include, for example, software (eg, the program 140) and input data or output data for commands related thereto.
  • the memory 130 may include a volatile memory 132 or a nonvolatile memory 134.
  • the program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.
  • the input device 150 may receive a command or data to be used for a component of the electronic device 101 (eg, the processor 120) from an outside (eg, a user) of the electronic device 101.
  • the input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).
  • the sound output device 155 may output an sound signal to the outside of the electronic device 101.
  • the sound output device 155 may include, for example, a speaker or a receiver.
  • the speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls.
  • the receiver may be implemented separately from or as a part of the speaker.
  • the display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user).
  • the display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device.
  • the display device 160 may include a touch circuitry set to sense a touch, or a sensor circuit (eg, a pressure sensor) set to measure the strength of a force generated by the touch. have.
  • the audio module 170 may convert sound into an electric signal or, conversely, convert an electric signal into sound. According to an embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device directly or wirelessly connected to the electronic device 101 (for example, Sound may be output through the electronic device 102) (for example, a speaker or headphones).
  • the sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do.
  • the sensor module 176 is, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.
  • the interface 177 may support one or more designated protocols that may be used for the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ).
  • the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
  • HDMI high definition multimedia interface
  • USB universal serial bus
  • SD card interface Secure Digital Card
  • the connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ).
  • the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).
  • the haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that a user can perceive through a tactile or motor sense.
  • the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
  • the camera module 180 may capture a still image and a video.
  • the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.
  • the power management module 188 may manage power supplied to the electronic device 101.
  • the power management module 388 may be implemented as at least a part of a power management integrated circuit (PMIC), for example.
  • PMIC power management integrated circuit
  • the battery 189 may supply power to at least one component of the electronic device 101.
  • the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.
  • the communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, electronic device 102, electronic device 104, or server 108). It is possible to support establishment and communication through the established communication channel.
  • the communication module 190 operates independently of the processor 120 (eg, an application processor), and may include one or more communication processors that support direct (eg, wired) communication or wireless communication.
  • the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A LAN (local area network) communication module, or a power line communication module) may be included.
  • a corresponding communication module is a first network 198 (for example, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 199 (for example, a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (for example, a telecommunication network such as a LAN or WAN).
  • the wireless communication module 192 uses subscriber information stored in the subscriber identification module 196 (eg, International Mobile Subscriber Identifier (IMSI)) within a communication network such as the first network 198 or the second network 199.
  • IMSI International Mobile Subscriber Identifier
  • the antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive from the outside.
  • the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern.
  • the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, provided by the communication module 190 from the plurality of antennas. Can be chosen.
  • the signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna.
  • other components eg, RFIC
  • other than the radiator may be additionally formed as a part of the antenna module 197.
  • At least some of the components are connected to each other through a communication method (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices and signals ( E.g. commands or data) can be exchanged with each other.
  • a communication method e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)
  • GPIO general purpose input and output
  • SPI serial peripheral interface
  • MIPI mobile industry processor interface
  • commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199.
  • Each of the electronic devices 102 and 104 may be a device of the same or different type as the electronic device 101.
  • all or part of the operations executed by the electronic device 101 may be executed by one or more of the external electronic devices 102, 104, or 108.
  • the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 does not execute the function or service by itself.
  • One or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 101.
  • the electronic device 101 may process the result as it is or additionally and provide it as at least part of a response to the request.
  • cloud computing, distributed computing, or client-server computing technology Can be used.
  • FIG. 2 is a diagram illustrating an electronic device 201 according to an embodiment disclosed in this document.
  • an electronic device 201 (eg, the electronic device 101 of FIG. 1) includes a lens assembly 210, a light emitting device 220, an image sensor 230, and a memory 250 (eg: Buffer memory), a micro lens array 270, or a processor 260.
  • the electronic device 201 may be a device of various types.
  • the electronic device 201 may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance.
  • the electronic device according to the embodiment disclosed in this document is not limited to the above-described devices.
  • the lens assembly 210 may collect light emitted from a subject to be imaged.
  • the lens assembly 210 may include one or more lenses.
  • the electronic device 201 may include a plurality of lens assemblies 210.
  • the electronic device 201 when the electronic device 201 is a camera, the electronic device 201 may form, for example, a dual camera, a 360 degree camera, or a spherical camera.
  • Some of the plurality of lens assemblies 210 have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly It may have one or more lens properties different from the lens properties of.
  • the lens assembly 210 may include, for example, a wide-angle lens and/or a telephoto lens.
  • the micro lens array 270 may include a plurality of micro lenses. Each of the plurality of micro lenses included in the micro lens array 270 may correspond to at least two pixel arrays.
  • the micro lens array 270 may be disposed between the lens assembly 210 and the image sensor 230.
  • the micro lens array 270 may collect incident light incident on the lens assembly 210 by reflecting light emitted from the light emitting device 220 from a subject.
  • the light-emitting device 220 may emit light used to enhance light emitted or reflected from the subject.
  • the light-emitting device 220 may include at least one light-emitting device.
  • the light emitting device 220 may include one or more light emitting diodes (eg, red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.
  • RGB red-green-blue
  • the light emitting element 220 is shown to be disposed inside the electronic device 201, but is not limited thereto. For example, it goes without saying that the light emitting element 220 may be disposed outside the electronic device 201.
  • the image sensor 230 senses the light emitted or reflected from the subject and transmitted through the lens assembly 210 and the microlens array 270 and converts it into an electric signal, thereby obtaining an image signal corresponding to the subject.
  • the image sensor 230 is, for example, one image sensor selected from image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor. It may include a plurality of image sensors having attributes, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.
  • CCD charged coupled device
  • CMOS complementary metal oxide semiconductor
  • Memory 250 may be operatively connected to a processor.
  • the memory 250 may store instructions that enable the processor 260 to perform various operations according to embodiments disclosed in this document.
  • the memory 250 may temporarily store at least a part of the image signal acquired through the image sensor 230 for the next image processing operation. For example, when image acquisition is delayed according to the shutter, or when a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high resolution image) is stored in the memory 250 , A copy image corresponding thereto (eg, a low resolution image) may be previewed through a display device (display device 160 of FIG. 1 ). Thereafter, when a specified condition is satisfied (eg, user input or system command), at least a part of the original image stored in the memory 250 may be acquired and processed by, for example, the processor 260.
  • a specified condition eg, user input or system command
  • the processor 260 may perform one or more image processing on an image signal acquired through the image sensor 230 or an image signal stored in the memory 250.
  • One or more image processing can be, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g. noise reduction, resolution adjustment, brightness adjustment, blurring). ), sharpening, or softening
  • the processor 260 includes at least one of the components included in the electronic device 201 (eg, the image sensor 230). ) Can be controlled (eg, exposure time control, readout timing control, etc.)
  • the image processed by the processor 260 is stored again in the memory 250 for further processing or 201), such as the memory 130 of FIG. 1, the display device 160 of FIG. 1, the electronic device 102 of FIG. 1, the electronic device 104 of FIG. 1, or the server of FIG. 108)).
  • the lens assembly 210, the light emitting device 220, the image sensor 230, and the micro lens array 270 may be included in the camera module 180 of FIG. 1.
  • the memory 250 may be configured as at least a part of the memory 130 of FIG. 1 or as a separate memory that is independently operated.
  • the processor 260 may be configured as at least a part of the processor 120 of FIG. 1, or may be configured as a separate processor that is independently operated from the processor 120 of FIG. 1.
  • the processor 260 is configured as a separate processor from the processor 120 of FIG. 1, at least one image processed by the processor 260 is as is or undergoes additional image processing by the processor 120 of FIG. Then, it may be displayed through the display device 160 of FIG. 1.
  • the electronic device 201 may be the camera module 180 of FIG. 1.
  • the electronic device 101 of FIG. 1 includes a plurality of camera modules (eg, the camera module 180 of FIG. 1) each having different attributes or functions.
  • at least one of the plurality of camera modules eg, the camera module 180 of FIG. 1 may be a wide-angle camera, and at least the other may be a telephoto camera.
  • at least one of the plurality of camera modules (eg, the camera module 180 of FIG. 1) may be a front camera, and at least the other may be a rear camera.
  • 3A is a diagram illustrating the lens assembly 210 and the image sensor 230 of FIG. 2.
  • 3B is a view for explaining the arrangement of the lens assembly 210, the micro lens array 270, and the image sensor 230 of FIG. 2.
  • the image sensor 230 of the electronic device 201 includes a first pixel array PX1, a second pixel array PX2, and A third pixel array (PX3), a fourth pixel array (PX4), a fifth pixel array (PX5), a sixth pixel array (PX6), a seventh pixel array (PX7), and an eighth pixel array (PX8).
  • PX1 a first pixel array
  • PX2 a second pixel array PX2
  • PX3 A third pixel array
  • PX4 fourth pixel array
  • PX5 a fifth pixel array
  • PX6 a sixth pixel array
  • PX7 seventh pixel array
  • PX8 eighth pixel array
  • Each of the pixel array PX7 and the eighth pixel array PX8 may be aligned along the first direction X.
  • the first pixel array PX1 may include a first sub-pixel PXS1 and a second sub-pixel PXS2.
  • the first sub-pixel PXS1 and the second sub-pixel PXS2 may be aligned along the first direction X.
  • the first to eighth pixel arrays PX1, PX2, PX3, PX4, PX5, PX6, PX7, and PX8 may be arranged side by side along the second direction Y.
  • the first pixel array PX1 and the second pixel array PX2 may be disposed along the second direction Y.
  • the second pixel array PX2 and the third pixel array PX3 may be disposed along the second direction Y.
  • the first pixel array PX1, the second pixel array PX2, and the third pixel array PX3 may be arranged side by side along the second direction Y.
  • the micro lens array 270 may be disposed between the lens assembly 210 and the image sensor 230.
  • the lens assembly 210, the micro lens array 270, and the image sensor 230 may be aligned along the third direction Z.
  • the first direction (X), the second direction (Y), and the third direction (Z) may be directions that cross each other.
  • the micro lens array 270 may include a first micro lens ML1, a second micro lens ML2, a third micro lens ML3, and a fourth micro lens ML4.
  • the microlens array 270 is disposed on the image sensor 230 and includes at least one of the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) of the image sensor 230.
  • the lens assembly 210 includes a main lens, and the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) are light incident through one main lens. Can be sensed.
  • PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8 are light incident through one main lens. Can be sensed.
  • the micro lens array 270 may be disposed directly on the image sensor 230.
  • other components may not be interposed between the micro lens array 270 and the image sensor 230.
  • the micro lens array 270 may be disposed on the image sensor 230 and spaced apart from the image sensor 230.
  • other components may be interposed between the micro lens array 270 and the image sensor 230.
  • the first micro lens ML1 may be disposed between the lens assembly 210 and the first pixel array PX1, and between the lens assembly 210 and the second pixel array PX2.
  • the first micro lens ML1 may cover the first pixel array PX1 and the second pixel array PX2.
  • the first micro lens ML1 may overlap the first pixel array PX1 and the second pixel array PX2 in a third direction Z.
  • the second micro lens ML2 may be disposed between the lens assembly 210 and the third pixel array PX3, and between the lens assembly 210 and the fourth pixel array PX4.
  • the second micro lens ML2 may cover the third pixel array PX3 and the fourth pixel array PX4.
  • the second microlens ML2 may overlap the third pixel array PX3 and the fourth pixel array PX4 in the third direction Z.
  • the description of the first micro lens ML1 and the second micro lens ML2 may be applied to the third micro lens ML3 and the fourth micro lens ML4.
  • Each of the first to fourth microlenses ML1, ML2, ML3, and ML4 may extend along the first direction X.
  • Each of the first to fourth micro lenses ML1, ML2, ML3 and ML4 may be a cylindrical lens.
  • Each of the first to fourth microlenses ML1, ML2, ML3, and ML4 may be disposed along the second direction Y. Since each of the first to fourth microlenses ML1, ML2, ML3, and ML4 is a cylindrical lens, the first to fourth microlenses ML1 and ML1 are compared to a lens other than a cylindrical lens (for example, the lens assembly 210).
  • ML2, ML3, ML4) may have a relatively large curvature. Accordingly, the focal length may be shortened.
  • the first pixel array PX1 senses the incident light incident on the first microlens ML1 to provide a first image signal
  • the second pixel array PX2 receives incident light incident on the first microlens ML1.
  • a second image signal may be provided by sensing.
  • the image sensor 230 includes an array of eight pixels, but the present invention is not limited thereto.
  • the image sensor 230 may include at least two or more pixel arrays, if necessary.
  • each of the first to eighth pixel arrays PX1, PX2, PX3, PX4, PX5, PX6, PX7, and PX8 includes five sub-pixels, but is not limited thereto.
  • each of the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) may include an arbitrary number of subpixels as necessary.
  • FIG. 3A it is illustrated that the image sensor 230 includes an array of eight pixels, but the present invention is not limited thereto.
  • the image sensor 230 may include at least two or more pixel arrays, if necessary.
  • each of the first to eighth pixel arrays PX1, PX2, PX3, PX4, PX5, PX6, PX7, and PX8 includes five sub-pixels, but is not limited thereto
  • the micro lens array 270 is illustrated as including four micro lenses, but is not limited thereto. For example, if the micro lens array 270 covers two pixel arrays, an arbitrary number of micro lenses may be included according to the number of pixel arrays included in the image sensor 230.
  • FIG. 4 is a diagram illustrating the lens assembly 210 and the image sensor 230 of FIG. 2. For clarity of description, what has been described above may be simplified or omitted.
  • the image sensor 230 of the electronic device 201 may include a micro lens array 270 and a pixel array PX.
  • the micro lens array 270 may include a plurality of micro lenses.
  • Each of the plurality of micro lenses may include a pixel array corresponding to each of the plurality of micro lenses.
  • the first micro lens ML1 may include a pixel array PX corresponding to the first micro lens ML1.
  • the pixel array PX may be aligned along the first direction X.
  • the pixel array PX may include a first sub-pixel PXS1 and a second sub-pixel PXS2.
  • the first sub-pixel PXS1 and the second sub-pixel PXS2 may be aligned along the first direction X.
  • the pixel array PX may be disposed along the second direction Y.
  • Two readout circuits may be connected to the pixel array PX.
  • One of the two readout circuits may be connected to the first region PXR1 of the pixel array PX.
  • the other one of the two readout circuits may be connected to the second region PXR2 of the pixel array PX.
  • the first area PXR1 may be an area on one side of the pixel array PX
  • the second area PXR2 may be an area on the other side of the pixel array PX.
  • the two readout circuits sense different regions of the pixel array PX (e.g., the first region PXR1 and the second region PXR2), so that an image signal (e.g., photoelectric conversion signal) in one pixel array Can be obtained differentially.
  • the micro lens array 270 may cover one pixel array.
  • a readout circuit is connected to each of the first region PXR1 and the second region PXR2 of one pixel array PX, so that image signals may be differentially obtained.
  • the first area PXR1 of the pixel array PX senses the incident light incident on the first microlens ML1 to provide a first image signal
  • the second area PXR2 of the pixel array PX May provide a second image signal by sensing incident light incident on the first microlens ML1.
  • 5A is a diagram illustrating the processor 260 of FIG. 2. The operation described below with reference to the boundary detection circuit 261 and the distance measurement circuit 263 may be executed (or performed) by the processor 260.
  • the processor 260 of the electronic device 201 includes an edge detection circuitry 261 and a distance measurement circuitry. (263) may be included.
  • the boundary detection circuit 261 may be referred to as a plenoptic circuitry.
  • the boundary detection circuit 261 may identify the boundary of the subject and provide boundary information.
  • the boundary detection circuit 261 may identify the boundary of the subject based on the intensity of the first incident light among incident light incident on the lens assembly 210.
  • the boundary detection circuit 261 may generate a disparity map based on the intensity of the first incident light and identify the boundary of the subject using the disparity map.
  • the boundary detection circuit 261 may further refer to distance information of the distance measurement circuit 263 to generate a disparity map based on the intensity of the first incident light and identify the boundary of the subject. . In an embodiment, the boundary detection circuit 261 may further refer to face recognition information of the face recognition circuit, generate a disparity map based on the intensity of the first incident light, and identify the boundary of the subject. In an embodiment, the boundary detection circuit 261 may generate a disparity map based on the intensity of the first incident light by further referring to the autofocus information of the autofocus detection circuit and identify the boundary of the subject.
  • the incident light may be light incident on the lens assembly 210 by reflecting light emitted from the light emitting device 220 from a subject.
  • the first incident light may be, for example, light having a certain incident angle or within a certain range of incident light.
  • the first incident light may be light incident on the lens assembly 210 by reflecting light from the boundary of the subject.
  • the boundary of the subject may be, for example, a boundary between one subject and another subject in an image including a plurality of subjects.
  • the boundary of the subject may be, for example, a boundary between a foreground subject and a background subject in an image including a foreground subject and a background subject.
  • the first incident light may be incident on a first micro lens, which is one of the micro lens arrays 270.
  • the boundary detection circuit 261 includes a first image signal generated by sensing a first incident light by a first pixel array or a first region of a pixel array, and a first image signal generated by sensing a first pixel array or a first region of the pixel array.
  • the intensity of the first incident light may be detected using the second image signal generated by sensing the incident light.
  • the distance measurement circuit 263 may be referred to as a time of flight (TOF) circuit.
  • the distance measuring circuit 263 may provide distance information for identifying a distance to a point and/or a boundary of the subject.
  • the distance measuring circuit 263 may compare the light emitted from the light emitting element 220 and the incident light to identify a distance to a point of the subject. For example, the distance measuring circuit 263 may compare the light emitted from the light emitting element 220 with the second incident light to identify a distance to a point of the subject.
  • the second incident light may be, for example, light having a certain incident angle or within a certain range of incident light.
  • the second incident light may be, for example, light emitted from the light emitting device 220 is reflected from a point of the subject and incident on the lens assembly 210.
  • the distance to the point and/or boundary of the subject may be a value measured from an arbitrary point and/or boundary of the electronic device 201 in the third direction Z, which is a direction from the electronic device toward the subject.
  • the distance to the point and/or the boundary of the subject may be a distance between the lens assembly 210 of the electronic device 201 and the point and/or the boundary of the subject.
  • the distance to the point and/or the boundary of the subject may be a distance between the image sensor 230 of the electronic device 201 and the point and/or the boundary of the subject.
  • the point of the subject may be a point inside the subject.
  • the point of the subject may be, for example, a point at the boundary of the subject.
  • the distance measuring circuit 263 may determine a distance to a point and/or boundary of the subject by measuring a time when light is reflected from the subject and incident on the lens assembly 210. For example, the distance measuring circuit 263 may identify a distance to a boundary of the subject based on a phase difference between the light and the first incident light. For example, the distance measuring circuit 263 may identify a distance to a point of the subject based on a phase difference between the light and the second incident light.
  • the boundary detection circuit 261 may generate a disparity map based on the intensity of image signals sensed by a pixel array covered by one microlens. When generating the disparity map, the boundary detection circuit 261 may use information on an initial pixel group.
  • the boundary detection circuit 261 may identify an initial pixel group based on distance information for an object.
  • the electronic device 201 includes a group of pixels from 1 to n (however, n>1, n is a natural number), and each pixel group may include a pixel array covered by one microlens. have.
  • the boundary detection circuit 261 may identify that the x pixel group (where 1 ⁇ x ⁇ n, x is a natural number) is an initial pixel group corresponding to the distance to the subject based on distance information on the subject.
  • the boundary detection circuit 261 compares each of the n pixel groups from the x+1 pixel group and the initial pixel group (ie, x pixel group) to identify the disparity value of each pixel group. I can. At this time, the boundary detection circuit 261 compares each of the n pixel groups from the 2 pixel group and the 1 pixel group to detect the disparity value of each pixel group, and detects the disparity value from the initial pixel group, resulting in high speed. You can create a disparity map with. The identification of the disparity value of the pixel group may be based on the intensity of the first and second image signals sensed by the pixel group.
  • the boundary detection circuit 261 may identify a boundary of a subject by identifying a sudden change in the intensity difference (ie, disparity value) of an image signal between neighboring pixel groups. For example, an x pixel group (eg, a pixel group including a first pixel array PX1 and a second pixel array PX2, or a pixel array PX covered by the first micro lens ML1) and Display between the x+1 pixel group (e.g., the pixel group including the third pixel array PX3 and the fourth pixel array PX4, or the pixel array PX covered by the second microlens ML2).
  • an x pixel group eg, a pixel group including a first pixel array PX1 and a second pixel array PX2, or a pixel array PX covered by the first micro lens ML1
  • Display between the x+1 pixel group e.g., the pixel group including the third pixel array PX3 and the fourth pixel array PX4, or the pixel array
  • the difference between the parity value is smaller than the reference value, and the x+1 pixel group and the x+2 pixel group (e.g., a pixel group including the fifth pixel array PX5 and the sixth pixel array PX6, or the third microlens If the difference in disparity values between the pixel arrays PX covered by (ML3) is greater than the reference value, it may be identified that the x+1 pixel group corresponds to the boundary of the subject.
  • the x+1 pixel group and the x+2 pixel group e.g., a pixel group including the fifth pixel array PX5 and the sixth pixel array PX6, or the third microlens
  • the electronic device 201 may generate a depth map based on boundary information, disparity map, and distance information.
  • the boundary detection circuit 261 may identify an initial pixel group based on distance information for a subject.
  • Distance information on the subject may be distance information provided from the distance measuring circuit 263, for example.
  • FIG. 5B is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information on a subject according to an exemplary embodiment disclosed in the present document. For clarity of description, those overlapping with those described above may be simplified or omitted. The operation described below with reference to the boundary detection circuit 261 may be executed (or performed) by the processor 260.
  • a processor 260 (eg, the electronic device 201 and/or the boundary detection circuit 261 of FIG. 2) according to an embodiment disclosed in this document is configured to generate a disparity map.
  • an initial pixel group may be identified based on distance information for a subject.
  • Distance information on the subject may be, for example, face recognition information provided from the face recognition circuit 264.
  • the face recognition circuit 264 may be implemented as, for example, a circuit included in the processor 260. In an embodiment, the face recognition circuit 264 may be implemented in software. In an embodiment, the face recognition circuit 264 may be a component included in an external electronic device. The face recognition circuit 264 may provide, for example, face recognition information that is a result of recognizing a human face among subjects.
  • the boundary detection circuit 261 may obtain distance information for a face among subjects based on the face recognition information.
  • the boundary detection circuit 261 may identify an initial pixel group corresponding to a distance to a face based on the face recognition information.
  • FIG. 5C is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information about a subject according to an embodiment disclosed in this document. For clarity of description, those overlapping with those described above may be simplified or omitted. The operation described below with reference to the boundary detection circuit 261 may be executed (or performed) by the processor 260.
  • a processor 260 (eg, the electronic device 201 and/or the boundary detection circuit 261 of FIG. 2) according to an embodiment disclosed in the present document is configured to generate a disparity map.
  • an initial pixel group may be identified based on distance information for a subject.
  • Distance information about the subject may be, for example, auto focus information provided from the auto focus detection circuit 265.
  • the auto focus detection circuit 265 may be implemented as, for example, a circuit included in the processor 260. In one embodiment, the auto focus detection circuit 265 may be implemented in software. In an embodiment, the auto focus detection circuit 265 may be a component included in an external electronic device. The auto focus detection circuit 265 may provide, for example, auto focus information, which is focus information on one of the subjects.
  • the boundary detection circuit 261 may obtain distance information for any one of the subjects based on the auto focus information.
  • the boundary detection circuit 261 may identify an initial pixel group corresponding to a distance to any one of the subjects based on the auto focus information.
  • FIG. 6 is a flowchart illustrating a method of generating a depth map by the electronic device 201 according to an embodiment disclosed in this document.
  • the electronic device 201 of FIG. 2 performs the process of FIG. 6.
  • An operation described as being performed by the electronic device may be implemented as instructions that can be executed (or executed) by the processor 260 of the electronic device 201.
  • the instructions may be stored in, for example, a computer recording medium, the memory 130 of the electronic device 101 of FIG. 1, or the memory 250 of the electronic device 201 of FIG. 2.
  • the processor 260 may provide boundary information identifying a first boundary of a first subject by using at least two image signals. For example, in generating the disparity map to provide boundary information, the processor 260 is sensed and generated by the first pixel array PX1 or the first region PXR1 of the pixel array PX. The first image signal and a second image signal sensed and generated by the second pixel array PX2 or the second region PXR2 of the pixel array PX may be used. The first image signal and the second image signal may be signals generated from the first incident light among incident light reflected from the first subject and incident on the lens assembly (the lens assembly 210 of FIG. 2 ). The processor 260 may detect the intensity of the first incident light using the first image signal and the second image signal. The processor 260 may identify a first boundary of the first subject based on the intensity of the first incident light.
  • the processor 260 may generate a disparity map based on the intensity of the first incident light, and may identify a first boundary of the first subject.
  • the processor 260 may generate a disparity map based on the intensity of image signals sensed by the pixel array covered by the microlens by the first incident light incident on the microlens.
  • the processor 260 may identify an initial pixel group based on distance information for a subject.
  • the processor 260 may generate a disparity map using an initial pixel group, identify a first boundary from the disparity map, and provide boundary information.
  • the boundary information may be provided by the boundary detection circuit 261.
  • the processor 260 may provide distance information identifying a distance to the first point of the first subject.
  • the distance information may be provided by the distance measurement circuit 263.
  • the operation S101 and the operation S103 may be simultaneously performed (or executed) by the processor 260. In an exemplary embodiment, after any one of the operations S101 and S103 is performed (or executed) by the processor 260, the remaining operations may be performed (or executed).
  • the processor 260 may generate a depth map based on boundary information and distance information.
  • FIG. 7 is diagrams for explaining operations S101, S103, and S105 of FIG. 6.
  • the x-axis of FIG. 9 may represent time (unit: AU (arbitrary unit)), and the y-axis may represent light intensity (unit: AU).
  • light EL may emit light from at least one light emitting device 220 toward a first subject OB1 and a second subject OB2.
  • each of the first subject OB1 and the second subject OB2 may be one of a foreground subject and a background subject.
  • each of the first subject OB1 and the second subject OB2 may be a foreground subject.
  • the incident light IL may be light incident on the lens assembly 210 by reflecting the light EL on the first and second subjects OB1 and OB2.
  • the incident light IL may pass through the lens assembly 210 and may be collected through the micro lens array 270.
  • the incident light IL may include a first incident light IL1 and a second incident light IL2, a third incident light and a fourth incident light.
  • a first incident light IL1 and a second incident light IL2 may include a third incident light and a fourth incident light.
  • illustration of the third incident light and the fourth incident light has been omitted.
  • the first incident light IL1 may be incident light by reflecting the light EL from the first boundary 401 of the first subject OB1.
  • the first incident light IL1 may be collected through the first micro lens ML1.
  • the first boundary 401 of the first subject OB1 may be, for example, a part of the boundary between the first subject OB1 and the second subject OB2.
  • the second incident light IL2 may be light incident by reflecting the light EL from the first point 402 of the first object OB1.
  • the second incident light IL2 may be collected through the second micro lens ML2.
  • the first point 402 of the first subject OB1 may be, for example, a point inside the first subject OB1, not a boundary between the first subject OB1 and the second subject OB2. .
  • the third incident light may be light incident by reflecting the light EL from the second boundary 403 of the first subject OB1.
  • the third incident light may be collected through a third micro lens (eg, the third micro lens ML3 of FIG. 3A ).
  • the second boundary 403 of the first subject OB1 may be, for example, another part of the boundary between the first subject OB1 and the second subject OB2.
  • the first boundary 401 and the second boundary 403 may be located on the same X-Y plane, for example.
  • the electronic device 201 uses the microlens array 270 to divide the boundary between the first subject OB1 and the second subject OB2 into parts (for example, the first The boundary 401 and the second boundary 403, and incident light reflected from each portion of the boundary may be separately sensed.
  • the boundary of the subject of the depth map generated according to the embodiment disclosed in this document is clearly identified. Can be.
  • the fourth incident light may be light that is incident by reflecting the light EL from the second point 404 of the first subject OB1.
  • the fourth incident light IL4 may be collected through a fourth micro lens (eg, the fourth micro lens ML4 of FIG. 3A ).
  • the second point 404 of the first subject OB1 may be, for example, a point inside the first subject OB1, not a boundary between the first subject OB1 and the second subject OB2. .
  • the first point 402 and the second point 404 may have different distances from the lens assembly 210 in the third direction Z.
  • the first incident light IL1, the second incident light IL2, the third incident light, and the fourth incident light may have different incident angles incident on each of the first to fourth microlenses.
  • the first incident light IL1 may be light incident on the first microlens ML1 at a first incident angle ⁇ 1 among incident light IL.
  • the second incident light IL2 may be light incident on the second microlens ML2 at a second incident angle ⁇ 2 among incident light IL.
  • the first incident angle ⁇ 1 and the second incident angle ⁇ 2 may be different.
  • the description of the first incident light IL1 and the second incident light IL2 may also be applied to the third incident light IL3 and the fourth incident light IL4.
  • each of the incident angles at which the first incident light IL1, the second incident light IL2, the third incident light IL3, and the fourth incident light IL4 are incident to each of the first to fourth microlenses is constant. It can have a range.
  • the first incident angle ⁇ 1 may be within a certain angle range.
  • the incident light may be referred to as the first incident light IL1 if it is light within the range of the first incident angle ⁇ 1.
  • the image sensor 230 may sense light collected through the micro lens array 270 among incident light IL incident through the lens assembly 210.
  • the first pixel array PX1 of the image sensor 230 senses the first incident light IL1 collected through the first microlens ML1 among the incident light IL, and receives a first image signal. Can be generated.
  • the second pixel array PX2 of the image sensor 230 may generate a second image signal by sensing the first incident light IL1 collected through the first microlens ML1 among the incident light IL. .
  • the first image signal and the second image signal may include the same, similar, or different information.
  • the information on the amount of charge included in the first image signal may be different from the information on the amount of charge included in the second image signal.
  • the amount of charge sensed by the first pixel array PX1 and the amount of charge sensed by the second pixel array PX2 may be different.
  • the phase of the first image signal and the phase of the second image signal may be opposite to each other.
  • information on the intensity of the first incident light IL1 included in the first image signal and the information on the intensity of the first incident light IL1 included in the second image signal are substantially the same or It can be similar.
  • the first incident light IL1 is incident light reflected from the first boundary 401 of the first subject OB1, information on the first boundary 401 is included.
  • information on the intensity of the first incident light IL1 information on the intensity of the first incident light IL1
  • position information on the XYZ coordinates of the first boundary 401, and the first incident angle ⁇ 1 of the first incident light IL1 May contain information about.
  • the processor 260 may provide boundary information identifying the first boundary 401 of the first subject OB1 using the first image signal and the second image signal. For example, the processor 260 identifies an initial pixel group based on distance information on a subject, generates a disparity map based on the intensity of the first incident light IL1, and determines the first boundary 401 Can be identified.
  • the processor 260 may identify a distance to the first boundary 401 based on a phase difference between the light EL and the first incident light IL1 and include it in the distance information.
  • a first graph G1 may be a graph showing the intensity of light EL over time
  • a second graph G2 is a graph showing the intensity of first incident light IL1 over time.
  • the processor 260 may identify a distance to the first boundary 401 based on the phase difference ⁇ between the first graph G1 and the second graph G2.
  • the distance to the first boundary 401 may be a distance in the third direction Z from an arbitrary point related to the electronic device 201 to the first boundary 401.
  • the processor 260 emits light EL from the at least one light emitting element 220 and is reflected from the first boundary 401 of the first subject OB1 to the lens assembly 210. By measuring the incident time to, the distance to the first boundary 401 may be identified and included in the distance information.
  • the third pixel array PX3 of the image sensor 230 senses the second incident light IL2 collected through the second microlens ML2 among the incident light IL, It can generate an image signal.
  • the fourth pixel array PX4 of the image sensor 230 may generate a fourth image signal by sensing the second incident light IL2 collected through the second microlens ML2 among the incident light IL. .
  • the third image signal and the fourth image signal may include the same, similar, or different information.
  • the description of the first and second image signals may be applied to the third and fourth image signals.
  • the second incident light IL2 is incident light reflected from the first point 402 of the first object OB1
  • information on the first point 402 is included.
  • position information on the XYZ coordinates of the first point 402 and the second incident angle ⁇ 2 of the second incident light IL2 May contain information about.
  • the processor 260 may identify the first point 402 of the first subject OB1 using the third image signal and the fourth image signal. For example, the processor 260, based on the intensity of the second incident light IL2, if the intensity of the second incident light IL2 is less than the reference value, the second incident light IL2 is the first subject OB1 It may be identified as the light reflected from the inner point (ie, the first point 402) of the first subject OB1, not the boundary.
  • the processor 260 may identify a distance to the first point 402 based on a phase difference between the light EL and the second incident light IL2 and include it in the distance information.
  • a first graph G1 may be a graph showing the intensity of light EL over time
  • a second graph G2 is a graph showing the intensity of second incident light IL2 over time.
  • the processor 260 may identify the distance to the first point 402 based on the phase difference ⁇ between the first graph G1 and the second graph G2.
  • the distance to the first point 402 may be a distance in the third direction Z from an arbitrary point related to the electronic device 201 to the first point 402.
  • the processor 260 emits light EL from the at least one light emitting element 220 and is reflected from the first point 402 of the first subject OB1 to the lens assembly 210. By measuring the incident time to, the distance to the first point 402 may be identified and included in the distance information.
  • the fifth pixel array of the image sensor 230 (for example, the fifth pixel array PX5 of FIG. 3A) is a third incident light collected through a third microlens among the incident light IL ( IL3) may be sensed to generate a fifth image signal.
  • the sixth pixel array of the image sensor 230 (for example, the sixth pixel array PX6 in FIG. 3A) is a third incident light IL3 collected through a third microlens among the incident light IL. By sensing, a sixth image signal may be generated.
  • the fifth image signal and the sixth image signal may include the same, similar, or different information. Descriptions of the first and second image signals may also be applied to the fifth and sixth image signals.
  • the processor 260 may identify the second boundary 403 of the first subject OB1 using the fifth image signal and the sixth image signal.
  • a description of the processor 260 identifying the first boundary 401 using the first and second image signals is, for the processor 260 to identify the second boundary 403 using the fifth and sixth image signals. Can be applied for identifying.
  • the processor 260 may identify a distance to the second boundary 403. A description of the processor 260 identifying the distance to the first boundary 401 may be applied to the processor 260 identifying the distance to the second boundary 403.
  • the seventh pixel array of the image sensor 230 (for example, the seventh pixel array PX7 in FIG. 3A) senses the fourth incident light IL4 collected through the fourth microlens among the incident light IL, 7 Can generate image signals.
  • the eighth pixel array of the image sensor 230 (for example, the eighth pixel array PX8 in FIG. 3A) senses the fourth incident light IL4 collected through the fourth microlens among the incident light IL, 8 Image signals can be generated.
  • the seventh image signal and the eighth image signal may include the same, similar, or different information. Descriptions of the first and second image signals may also be applied to the seventh and eighth image signals.
  • the processor 260 may identify the second point 404 of the first subject OB1 using the seventh image signal and the eighth image signal.
  • a description of the processor 260 identifying the first point 402 using the first and second image signals is, for the processor 260 to use the seventh and eighth image signals, the second point 404 It can be applied for identifying.
  • the processor 260 may identify a distance to the second point 404.
  • the description of the processor 260 identifying the distance to the first point 402 may be applied to the processor 260 identifying the distance to the second point 404.
  • the processor 260 can clearly display the first boundary 401 and the second boundary 403 in the depth map by using boundary information for each of the first boundary 401 and the second boundary 403. .
  • the first boundary 401 may be clearly distinguished from the surrounding area of the first boundary 401.
  • the first boundary 401 can be clearly distinguished.
  • the processor 260 determines a first point 402 that is relatively farther from the first boundary 401 and the second boundary 403 than the first boundary 401 and the second boundary 403. It can be displayed darkly. In the depth map, the processor 260 may display the second point 404, which is relatively farther away than the first point 402, to be darker than the first point 402 in the depth map.
  • the depth map may include information on a boundary between the first subject OB1 and the second subject OB2, location information on XYZ coordinates, and the like.
  • An electronic device (eg, the electronic device 201 of FIG. 2) according to an embodiment disclosed in this document includes an image sensor (eg, the image sensor 230 of FIG. 2) and a lens assembly (eg, the lens assembly of FIG. 2 ).
  • a micro lens array eg, the micro lens array 270 of FIG. 2 between (210)
  • incident light incident at various incident angles may be collected, respectively.
  • incident light reflected from various points of the first subject for example, the first boundary 401, the first point 402, the second boundary 403, and the second point 404 of FIG. 8
  • Silver may be classified and collected through a micro lens array.
  • the processor of the electronic device eg, the processor 260 of FIG.
  • the electronic device may provide distance information and boundary information at various points of the first subject.
  • the processor of the electronic device can generate a depth map based on the boundary information at the boundary of the subject as well as distance information at the boundary and the internal point of the subject, a sophisticated depth map capable of identifying the boundary of the subject Can be implemented.
  • the electronic device may generate a disparity map at a high speed by identifying an initial pixel group based on distance information about the subject. For example, in an electronic device that provides a live view mode to a user, according to an embodiment disclosed in this document, by generating a disparity map based on an initial pixel group, a boundary of a subject may be identified in real time.
  • the electronic device may implement a sophisticated depth map capable of identifying the boundary of the subject based on distance information and boundary information about at least one point located at the boundary of the subject.
  • 10A, 10B, 10C, and 10D are diagrams for explaining a user interface provided by an electronic device (for example, the electronic device 201 and/or the processor 260) according to an embodiment disclosed in this document admit. For clarity of description, those overlapping with those described above may be simplified or omitted.
  • an electronic device may provide a live view mode and/or a video capture mode.
  • the first user interface 1001 may be a screen on which a third object OB3 and a fourth object OB4 are displayed through an electronic device in a live view mode and/or a video capture mode.
  • an electronic device may receive a user input related to selecting an object.
  • the user may select the third object OB3 in the live view mode.
  • the user may select the third object OB3 while capturing a video in the video capturing mode.
  • the electronic device may change properties of the third object OB3 and/or the fourth object OB4 based on a user input.
  • the second user interface 1002 may be a screen in which properties of the third object OB3 and/or the fourth object OB4 are changed based on a user input.
  • the electronic device identifies the boundary between the third object OB3 and the fourth object OB4, and identifies distance information for each point of the third object OB3 and the fourth object OB4. can do.
  • the electronic device determines the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4 based on the boundary information. And/or the attribute of the fourth object OB4 may be changed.
  • the electronic device provides the fourth object OB4 in a direction away from the third object OB3 with respect to a point inside the fourth object OB4 based on the distance information. You can increase or decrease the intensity of changing the properties of
  • the electronic device may apply a blur effect to the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.
  • the electronic device may change the color of the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.
  • the electronic device may apply a mosaic effect to the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.
  • the electronic device may be configured with a third object OB3 and a fourth object.
  • the photographing may be performed in a state in which properties of the third object OB3 and/or the fourth object OB4 are changed based on the boundary EG between the objects OB4.
  • the same image data as the second user interface 1002 may be stored in the electronic device.
  • the electronic device while the second user interface 1002 is displayed in the video recording mode of the electronic device, the electronic device is based on the boundary EG between the third object OB3 and the fourth object OB4, The photographing may be performed while the attributes of the third object OB3 and/or the fourth object OB4 are changed. In this case, the same video data as the second user interface 1002 may be stored in the electronic device.
  • the electronic device may receive a user input related to selection of the fourth object OB4 in the second user interface 1002.
  • the electronic device may change attributes of the third object OB3 and the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.
  • the electronic device applies a blur effect to the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4, and applies a blur effect to the fourth object OB4.
  • the electronic device may change the color of the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4.
  • the electronic device applies a mosaic effect to the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4, and the fourth object OB4 Can be sharpened.
  • phrases such as “at least one of, B, or C” may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof.
  • Terms such as “first”, “second”, or “first” or “second” may be used simply to distinguish the component from other corresponding components, and the components may be referred to in other aspects (eg, importance or Order) is not limited.
  • Some (eg, a first) component is referred to as “coupled” or “connected” with or without the terms “functionally” or “communicatively” to another (eg, second) component. When mentioned, it means that any of the above components can be connected to the other components directly (eg by wire), wirelessly, or via a third component.
  • module used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits.
  • the module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions.
  • the module may be implemented in the form of an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (for example, the program 140) including them.
  • the processor eg, the processor 120 of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This makes it possible for the device to be operated to perform at least one function according to the at least one command invoked.
  • the one or more instructions may include code generated by a compiler or code executable by an interpreter.
  • a storage medium that can be read by a device may be provided in the form of a non-transitory storage medium.
  • non-transient only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.
  • a signal e.g., electromagnetic wave
  • a method according to various embodiments disclosed in the present document may be included in a computer program product and provided.
  • Computer program products can be traded between sellers and buyers as commodities.
  • Computer program products are distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g., Play StoreTM) or two user devices ( It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones).
  • a device e.g. compact disc read only memory (CD-ROM)
  • an application store e.g., Play StoreTM
  • two user devices It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones).
  • at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.
  • each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities.
  • one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added.
  • a plurality of components eg, a module or a program
  • the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. .
  • operations performed by a module, program, or other component are sequentially, parallel, repeatedly, or heuristically executed, or one or more of the above operations are executed in a different order or omitted. Or one or more other actions may be added.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne un dispositif électronique. Le dispositif électronique comprend : un ensemble lentille ; un premier réseau de pixels aligné le long d'une première direction ; un second réseau de pixels aligné le long de la première direction ; une première microlentille disposée entre l'ensemble lentille et le premier réseau de pixels et entre l'ensemble lentille et le second réseau de pixels, et recouvrant le premier réseau de pixels et le second réseau de pixels ; un processeur ; et une mémoire connectée fonctionnellement au processeur, la mémoire stockant des instructions qui, lorsqu'elles sont exécutées, amènent le processeur à fournir des informations de limite identifiant une première limite d'un sujet, à l'aide d'un premier signal d'image et d'un second signal d'image, lorsque la lumière émise par au moins un élément électroluminescent est réfléchie par le sujet et rendue incidente sur l'ensemble lentille, le premier signal d'image est généré par détection, par le premier réseau de pixels, d'une première lumière incidente collectée à travers la première microlentille, de la lumière incidente, et le second signal d'image étant généré par détection, par le second réseau de pixels, de la première lumière incidente ; fournir des premières informations de distance identifiant une première distance à un premier point du sujet, par comparaison d'une seconde lumière incidente de la lumière incidente, à la lumière ; et générer une carte de profondeur sur la base des premières informations de distance et des informations de limite. De plus, divers autres modes de réalisation identifiés dans la description sont possibles.
PCT/KR2020/001822 2019-02-20 2020-02-10 Dispositif électronique et procédé de génération carte de profondeur Ceased WO2020171450A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020190020139A KR102606835B1 (ko) 2019-02-20 2019-02-20 뎁스 맵을 생성하는 전자 장치 및 방법
KR10-2019-0020139 2019-02-20

Publications (1)

Publication Number Publication Date
WO2020171450A1 true WO2020171450A1 (fr) 2020-08-27

Family

ID=72144323

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/001822 Ceased WO2020171450A1 (fr) 2019-02-20 2020-02-10 Dispositif électronique et procédé de génération carte de profondeur

Country Status (2)

Country Link
KR (1) KR102606835B1 (fr)
WO (1) WO2020171450A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114257766A (zh) * 2020-09-25 2022-03-29 爱思开海力士有限公司 图像感测装置

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102941945B1 (ko) * 2021-03-25 2026-03-20 삼성전자 주식회사 전자 장치 및 카메라의 원거리 성능 향상 방법
KR102910838B1 (ko) 2022-08-31 2026-01-12 성균관대학교산학협력단 라이트필드 비디오의 부복호화 방법 및 장치

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120100232A (ko) * 2011-03-03 2012-09-12 삼성전자주식회사 마이크로렌즈, 상기 마이크로렌즈를 포함하는 깊이 센서, 및 상기 마이크로렌즈 제조 방법
JP2014241595A (ja) * 2014-07-17 2014-12-25 オリンパス株式会社 撮像装置
KR20150015285A (ko) * 2013-07-31 2015-02-10 삼성전자주식회사 시프트된 마이크로 렌즈 어레이를 구비하는 라이트 필드 영상 획득 장치
WO2018072858A1 (fr) * 2016-10-18 2018-04-26 Photonic Sensors & Algorithms, S.L. Dispositif et procédé d'obtention d'informations de distance à partir de vues

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20120100232A (ko) * 2011-03-03 2012-09-12 삼성전자주식회사 마이크로렌즈, 상기 마이크로렌즈를 포함하는 깊이 센서, 및 상기 마이크로렌즈 제조 방법
KR20150015285A (ko) * 2013-07-31 2015-02-10 삼성전자주식회사 시프트된 마이크로 렌즈 어레이를 구비하는 라이트 필드 영상 획득 장치
JP2014241595A (ja) * 2014-07-17 2014-12-25 オリンパス株式会社 撮像装置
WO2018072858A1 (fr) * 2016-10-18 2018-04-26 Photonic Sensors & Algorithms, S.L. Dispositif et procédé d'obtention d'informations de distance à partir de vues

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KANG, YUN-SUK ET AL: "Foreground Segmentation and High-Resolution Depth Map Generation Using aTime-of~Flight Depth Camera.", THE JOURNAL OF KOREAN INSTITUTE OF COMMUNICATIONS AND INFORMALION SCIENCES., vol. 37, no. 9, 751 - 756, pages 201209, XP055734805 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114257766A (zh) * 2020-09-25 2022-03-29 爱思开海力士有限公司 图像感测装置
US12364045B2 (en) 2020-09-25 2025-07-15 SK Hynix Inc. Image sensing device

Also Published As

Publication number Publication date
KR20200101803A (ko) 2020-08-28
KR102606835B1 (ko) 2023-11-29

Similar Documents

Publication Publication Date Title
WO2020080845A1 (fr) Dispositif électronique et procédé pour obtenir des images
WO2020032497A1 (fr) Procédé et appareil permettant d'incorporer un motif de bruit dans une image sur laquelle un traitement par flou a été effectué
WO2020246710A1 (fr) Procédé de détermination de carte de profondeur et dispositif électronique auquel le même procédé est appliqué
WO2020071823A1 (fr) Dispositif électronique et son procédé de reconnaissance de geste
WO2021133025A1 (fr) Dispositif électronique comprenant un capteur d'image et son procédé de fonctionnement
WO2020116844A1 (fr) Dispositif électronique et procédé d'acquisition d'informations de profondeur à l'aide au moins de caméras ou d'un capteur de profondeur
WO2020159149A1 (fr) Dispositif électronique et procédé de traitement d'image
WO2021080307A1 (fr) Procédé de commande de caméra et dispositif électronique correspondant
WO2020231156A1 (fr) Dispositif électronique et procédé d'acquisition d'informations biométriques en utilisant une lumière d'affichage
WO2021033921A1 (fr) Module de caméra comprenant une carte de circuit imprimé, et dispositif électronique le comprenant
WO2019059635A1 (fr) Dispositif électronique pour fournir une fonction en utilisant une image rvb et une image ir acquises par l'intermédiaire d'un capteur d'image
WO2019088407A1 (fr) Module appareil photo comprenant une matrice de filtres colorés complémentaires et dispositif électronique le comprenant
WO2019143050A1 (fr) Dispositif électronique et procédé de commande de mise au point automatique de caméra
WO2021215795A1 (fr) Filtre couleur pour dispositif électronique, et dispositif électronique le comportant
WO2020190030A1 (fr) Dispositif électronique de génération d'image composite et procédé associé
WO2020190008A1 (fr) Dispositif électronique pour fonction de focalisation auto, et procédé de commande correspondant
WO2019168374A1 (fr) Procédé de génération d'informations multiples à l'aide d'une caméra pour détecter une largeur de bande d'onde multiple et appareil associé
WO2022203355A1 (fr) Dispositif électronique comprenant une pluralité de caméras
WO2021235884A1 (fr) Dispositif électronique et procédé de génération d'image par réalisation d'un awb
WO2020204428A1 (fr) Dispositif électronique et procédé de compensation d'erreur de profondeur en fonction de la fréquence de modulation
KR20200101803A (ko) 뎁스 맵을 생성하는 전자 장치 및 방법
WO2022250305A1 (fr) Procédé de stabilisation d'image pendant la capture, et dispositif électronique associé
WO2021158043A1 (fr) Dispositif électronique et procédé pour effectuer une mise au point automatique
WO2019190250A1 (fr) Procédé de synthèse d'image sur un objet réfléchissant en fonction d'un attribut d'objet réfléchissant inclus dans une image différente et dispositif électronique
WO2023140678A1 (fr) Dispositif électronique comprenant un capteur d'image et procédé de fonctionnement associé

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20759697

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20759697

Country of ref document: EP

Kind code of ref document: A1