EP2629503A2 - Vorrichtung und Verfahren zur Übertragung eines Kamerarahmenbildes - Google Patents

Vorrichtung und Verfahren zur Übertragung eines Kamerarahmenbildes Download PDF

Info

Publication number
EP2629503A2
EP2629503A2 EP20130155295 EP13155295A EP2629503A2 EP 2629503 A2 EP2629503 A2 EP 2629503A2 EP 20130155295 EP20130155295 EP 20130155295 EP 13155295 A EP13155295 A EP 13155295A EP 2629503 A2 EP2629503 A2 EP 2629503A2
Authority
EP
European Patent Office
Prior art keywords
image
compression
frame
camera
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP20130155295
Other languages
English (en)
French (fr)
Other versions
EP2629503A3 (de
EP2629503B1 (de
Inventor
Woohyun Baek
Sungwook An
Sungkoo Lee
Hyunho Park
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP2629503A2 publication Critical patent/EP2629503A2/de
Publication of EP2629503A3 publication Critical patent/EP2629503A3/de
Application granted granted Critical
Publication of EP2629503B1 publication Critical patent/EP2629503B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/21—Intermediate information storage
    • H04N1/2104—Intermediate information storage for one or a few pictures
    • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure

Definitions

  • the present invention relates to an apparatus and a method for processing data, and more particularly to an apparatus capable of variably transmitting frame images of a camera device according to a system bandwidth and a method thereof.
  • a camera device or a portable terminal having a camera function can provide a high quality image as well as various convenient functions.
  • an image sensor also referred to as a 'camera sensor'
  • HD High Definition
  • the camera device In operation, if a user presses a shutter button, the camera device stores an image acquired by the camera sensor.
  • the shutter button When the user presses the shutter button, there is a shutter delay or a shutter lag between the time of sensing an image to be the full size image, as opposed to a preview image, and the time of capturing the full-size image. That is, there is a time difference from a shutter activation to a time of photographing a subject by the camera sensor. The time difference may occur due to a photographing environment of the subject and a processing time of an image signal processor. Accordingly, there is a problem in that the user cannot acquire images at a desired point in time due to shutter delay (shutter lag).
  • a camera device or a terminal equipped with a camera may include an image signal processor for processing images and an application processor for processing various applications of a device including a camera application.
  • the image signal processor is serially connected to the application processor, so that the image signal processor can convert an image acquired by the camera into serial data, and then transmits the serial data to the application processor.
  • the image acquired by the camera may include an image having HD resolution or greater (Ultra High Definition resolution (UHD)).
  • UHD Ultra High Definition resolution
  • the term "UHD" refers to an image having a resolution of 4K*2K pixels or greater.
  • a camera device having the image signal processor and the application processor or a terminal having the camera function has difficulty in capturing an image having UHD resolution while photographing a moving image in a full HD resolution. This is because it is impossible for an image signal processor of a camera device in the prior art to transmit continuous image data having UHD resolution to an application processor at a desired time.
  • the present invention has been made in view of the above problems and provides additional advantages, by providing an apparatus and a method for capturing an image having UHD resolution photographed by a camera device or a terminal having a camera function at a desired point in time, and for selecting and storing a desired image from the photographed images having UHD resolution.
  • a camera device or a terminal having a camera function processes images acquired from camera sensors to generate a plurality of images for each frame, and transmits the generated images in a set data transmission mode using a hybrid scheme to use the generated images as a preview image and a captured image.
  • the present invention further provides an apparatus and a method capable of capturing an image having UHD resolution while photographing moving images having full HD resolution in a camera device or a terminal having a camera capability.
  • the exemplary embodiment of the present invention provides an apparatus and a method capable of generating images having UHD resolution acquired from a camera by an image signal processor for each frame as images having full HD resolution and compression images having UHD resolution, efficiently transmitting the images in a data transmission mode, and storing and capturing the images by an application processor as a moving image.
  • a method of transmitting a frame image of a camera includes: acquiring an image for each frame from a camera upon operation of the camera; generating the acquired image including a display image and compression images, transmitting the display image for each frame and transmitting the compression images at a preset frame interval; and displaying the acquired display image and buffering the compression images.
  • a camera device includes: a camera; an image signal processor for acquiring a frame image for each frame from the camera upon operation of the camera, generating the acquired image including a display image and a compression image, transmitting the display image for each frame, and transmitting the compression image at a preset frame interval; an application processor for displaying the acquired display image and buffering the compression image; and a display unit for displaying the display image output from the application processor.
  • a portable terminal apparatus includes: a communication unit for performing wireless communication with an external device; a terminal controller for controlling a call and communication of the portable terminal apparatus through the communication unit; a camera for acquiring an image from a camera sensor from each frame upon operation of the camera; an image signal processor for acquiring a frame image for each frame from the camera upon operation of the camera, generating the acquired image including a display image and a compression image, transmitting the display image for each frame, and transmitting the compression image at a preset frame interval; an application processor for displaying the acquired image display and buffering the compression image; and a display unit for displaying the display image output from the application processor.
  • the present invention relates to a camera device or a portable terminal having a camera function, and may process images acquired by an image sensor of the camera to generate a plurality of images, and generate information acquired by the camera and/or sensors as photographed information.
  • the images may include an image for preview (or preview and recording moving image) and a captured image while operating a camera.
  • the photographed information may include information from a camera sensor and/or respective sensors when photographing an image in a corresponding frame during acquisition of the image from the camera, which may be incorporated with meta data and embedded data.
  • the display image may be a YUV image recorded as a preview image and a moving image, and may be controlled and/or modified to a suitable size, by cropping and/or scaling, so that the display image may be displayed on a display unit.
  • the captured image may be a compression coded image as an image (e.g., image having full HD resolution or a resolution greater than the full HD resolution) acquired from the camera sensor, and will be described as a JPEG image in the exemplary embodiment of the present invention.
  • the display image may be a YUV image having full HD resolution and a compression image is a JPEG image acquired by compression-encoding an image having UHD resolution.
  • the compression image refers to a compression image for storing a still image and thus, a captured image and the compression image refer to the same image.
  • the exemplary embodiment of the present invention provides an apparatus which generates a display image and a compression image for each frame by an image signal processor(ISP) upon operation of a camera, transmits the generated display image and compression image in an interleaved data mode to implement zero shutter-lag, receives the display image and compression image, records the display image and compression image as display and moving images by an application processor(AP), and captures a compression image corresponding to a frame at a point in time of a capture request while buffering the compression image to implement the zero shutter-lag, and a method thereof.
  • ISP image signal processor
  • the photographed information includes camera sensor information with respect to every frame image acquired from the camera sensor upon operation of the camera and information generated from other sensors of the camera device.
  • a camera device includes information for setting the camera sensor, and such information may be Exchangeable Image File Format (exif) data.
  • the information for setting the camera sensor may possibly be used only for an acquired image in which an amount of exif data is restrictive.
  • the exemplary embodiment of the present invention provides the apparatus and the method which store photographed information of every frame acquired from the camera sensor and/or other sensors of a device, such as the camera, and efficiently use sensor data using the stored photographed information.
  • the interleaved data mode refers to a mode which multiplexes the display image and the compression image for each frame to transmit the multiplexed image as one frame, which is a mode which multiplexes and transmits the display image, the compression image and the photographed information. That is, the exemplary embodiment of the present invention may multiplex only the display image and the compression image as necessary to transmit the multiplexed image as one frame, multiplex the display image, the compression image, and photographed information to transmit the first multiplexed result as one frame, and multiplex only the display image and the photographed information to transmit the second multiplexed result. That is, in the interleaved data mode, frame information may be configured by only the display image and the compression image, or by only the compression image and the photographed information. The exemplary embodiment of the present invention may have the frame information configured by the display image, the compression image, and the photographed information by way of example.
  • the hybrid interleaved data mode is a mode in which the display image is transmitted for each frame and the compression image is transmitted in units of two frame intervals or greater.
  • the hybrid interleaved data mode may be determined by a bus bandwidth of the image signal processor and the application processor.
  • FIG. 1 is a block diagram illustrating a configuration of a camera device according to an exemplary embodiment of the present invention.
  • a camera 110 and a sensor 120 perform a function of acquiring an image by the sensor 120 when the camera 110 is activated.
  • the sensor 120 may include at least one sensor mounted in the camera 110, which includes at least one of a Global Positioning System (GPS) receiver, a gyro sensor, an acceleration sensor, an altitude sensor, and a direction sensor.
  • GPS Global Positioning System
  • An image signal processor 130 processes an image acquired from the camera 110 to generate a display image on a display unit 160 and a compression image to be stored in response to a capture request.
  • the display image may include a YUV image.
  • the image signal processor 130 color-converts the image acquired from the camera 110 and controls the size of the image to a size suitable for being displayed on the display unit 160, for example, by scaling and/or cropping. That is, the image signal processor 130 generates a YUV image, and may suitably set the YUV image according to a display capability of the display unit 160.
  • the YUV image may include an image displayed on the display unit 160 and an image recorded as a moving image.
  • the exemplary embodiment of the present invention may be implemented such that the display image is an image having a full HD resolution.
  • the image signal processor 130 compresses an image output from the camera 110 in a still image coding scheme (e.g., a JPEG scheme).
  • the compression image may be an image having a full HD resolution to be stored when a user requests capture of such image.
  • the exemplary embodiment of the present invention may be implemented such that the compression image is a JPEG image acquired by compressing a YUV image having UHD resolution.
  • the image signal processor 130 acquires photographed information of the image from the camera 110 and the sensor 120.
  • the photographed information may include various information which is additionally acquired as well as exif information which is acquired when the camera 110 photographs an image.
  • the image signal processor 130 may generate the display image and the compression image for each frame period and may generate the display image, the compression image, and the photographed information.
  • the image signal processor 130 may generate a display image or a compression image having a frame rate of either 30 frames per second or 60 frames per second, or of any preset rate in frames per second (fps), and/or generate the display image, the compression image, and photographed information.
  • the image signal processor 130 multiplexes the generated display image and the compression image (or display image, compression image, and photographed information) as one frame for transmission in an interleaved data mode.
  • the image signal processor 130 transmits the frame information in the interleaved data mode or a hybrid data mode according to a multiplexed bus bandwidth, as explained in greater detail later.
  • the application processor 140 buffers a display image and a compression image, or the display image, the compression image, and photographed image.
  • the application processor 140 controls the display unit 160 to display the display image upon operation of the camera 110.
  • the application processor 140 compresses the moving image and stores the compressed moving image in the storage unit 150.
  • the application processor 140 stores the compression image in the storage unit 150 when the capture request occurs to implement image capture with zero shutter-lag, as explained in greater detail later.
  • the input unit 170 responds to user selections and inputs to generate and provide a camera operation command, a capture command, and/or a moving image photographing command to the application processor 140.
  • the input unit 170 may be a touch panel for detecting a touch input of the user.
  • the input unit 170 may include at least one button or multiple buttons located on or accessible at the exterior of the camera 110.
  • the display unit 160 displays a display image output from the application processor 140 upon operation of the camera 110.
  • the display unit 160 may display photographed information and/or thumbnail images.
  • the display unit 160 may be a LCD or an OLED for displaying data and images generated during the execution of a program.
  • the input unit 170 and the display unit 160 may be implemented as an integral touch screen.
  • the storage unit 150 stores a buffered compression image from the application processor 140 upon receiving a capture request from the user.
  • FIG. 2 is a block diagram illustrating a configuration of the camera 110 according to the exemplary embodiment of the present invention.
  • the image signal processor 130 drives or operates a motor 240, and an actuator 250 controls the operation of an optical unit 210 according to operation of the motor 240.
  • the optical unit 210 may perform an operation such as zoom or focusing via the actuator 250.
  • the optical unit 210 images a peripheral image and an image sensor 220 detects an image imaged by the optical unit 220, and generates and/or converts a detection signal into an electric signal.
  • the image sensor 220 may be an image sensor having a resolution greater than a full HD resolution.
  • the image detected by the image sensor 220 is converted into a digital image by an A/D converter 230, and the digital image is transferred to the image signal processor 130.
  • a flash 260 such as a flash bulb or other lighting devices, may be operated by the image signal processor 130.
  • FIG. 3 is a block diagram illustrating configurations of the image signal processor 130 and the application processor 140 shown in FIG. 1 , according to the exemplary embodiment of the present invention.
  • an image processing part 320 of the image signal processor 130 converts an image output from the camera 110 into a YUV image.
  • the image processing part 320 can include a preprocessing part and a post-processing part.
  • the preprocessing part preprocesses an image acquired from the camera 110.
  • the preprocessing operation includes an extraction and processing of 3A(auto white balance(AWB), auto exposure(AE), auto focusing(AF)), lens shading correction, dead pixel correction, knee correction and etc.
  • the post-processing part performs at least one of color interpolation, noise removal, color compensation, and color transformation on the preprocessed image and generates a YUV data.
  • the image output from the camera 110 is a raw image and has a number of pixels greater than full HD resolution detected by the image sensor 220.
  • the image signal processor 130 includes a still image compressor 340, such as a still image coder, for compressing data and storing the still image in response to a capture request.
  • the still image compressor 340 may use various known still image coding methods such as JPEG and TIFF.
  • the exemplary embodiment of the present invention may be implemented such that the still image compressor 340 is a JPEG coder 340, as shown in FIG. 3 .
  • the JPEG coder 340 compresses an image output from the image processing part 320 into a JPEG image.
  • a scaler 330 scales an image output from the image processing part 320 to an image of a suitable size to be displayed on the display unit 160.
  • the scaler 330 may include elements or a part thereof for scaling, cropping, and/or resizing the image data.
  • the scaler 340 may be configured by a display image generator, which may be configured by a color converter for converting a raw image into a YUV image and a scaled image.
  • An image processing controller 310 controls the operation of the camera 110 under control of the application processing controller 360, and controls an overall operation of the image signal processor 130 in a camera operation mode. Upon operation of the camera 110, the image processing controller 310 may generate photographed information of an image photographed by the camera 110. When the sensor 120 is connected to the image signal processor 130, the image processing controller 310 may input setting information and may communicate an output of the camera 110 to other components, such as a multiplexer 350, as well as information generated from the sensor 120 to generate the photographed information.
  • the image processing controller 310 may include an internal memory. When transmitting the display image and the compression image, or the display image, the compression image, and the photographed information in an interleaved scheme, the compression image may be included in each frame according to a bus bandwidth. Alternatively, the compression image may be included at a preset frame interval, and the compression image may be transmitted only at the preset frame interval.
  • the compression image, the display image, and/or the photographed information are generated for each frame, and the frame may be configured to have a frame rate of 30 frames or 60 frames for one second, or any other preset frame rate.
  • the multiplexer 350 multiplexes the compression image, the display image, and/or the photographed information as one frame and transmits the multiplexed image to the application processor 140 according to an interleaved scheme.
  • the multiplexer 350 may perform a hybrid interleaved data mode transmitting the compression image for each preset frame under control of the image processing controller 310, as explained in greater detail later.
  • the image signal processor 130 may further include a thumbnail image generator for converting the display image into a thumbnail image. To this end, the image signal processor 130 may generate the thumbnail image based on the display image and the JPEG coder 340 may compress an output image of the image processing part 310, then generate an output configured by a header, the thumbnail image, and the compressed JPEG image.
  • the application processing controller 360 controls an operation of the image signal processor 130 according to a command from the input unit 170, buffers an image output from the image signal processor 130 to display the buffered image through the display unit 160, and stores the buffered compression image in the storage unit 150.
  • a demultiplexer 355 demultiplexes and outputs the display image, the compression image, and/or the photographed information from the image signal processor 130 under control of the application processing controller 360.
  • a parser 370 parses and separates the images and information demultiplexed by the demultiplexer 355 as the display image, the compression image, and the photographed information.
  • the application processing controller 360 controls the demultiplexer 355 to software-parse an output of the image signal processor 130 as the display image, the compression image, and/or the photographed information, and so in the alternative exemplary embodiment, the parser 370 may be omitted.
  • a buffer 380 may include a display image buffer 383, a compression image buffer 385, and/or a photographed information buffer 387.
  • the buffer 380 buffers the parsed display image, compression image, and photographed information under the control of the application processing controller 360.
  • the buffer 380 has a size capable of buffering a display image, a compression image, and photographed information of N frames.
  • the structure of the buffer 380 may have a ring buffer structure. The reason to configure the buffer 380 with a ring buffer structure is to maintain the received image and information of each frame for a particular number of frames.
  • the application processor 140 may include a moving image compressor 390, such as a moving image coder, for compressing the display image to a moving image when a storage of the moving image is requested, and the moving image compressor 390 may be configured by various moving image coders known in the art, such as H.264 or MPEG.
  • the moving image compressor 390 may comprise an MPEG coder 390 shown in FIG. 3 . If the storage of a moving image is requested by the user, the application processing controller 360 operates the MPEG coder 390, and the MPEG coder 390 compresses the display image buffered by the display image buffer 383 of the buffer 380 and stores the compressed display image in the storage unit 150.
  • the application processor 140 further include a thumbnail image generator 395 which resizes the display image to generate a thumbnail image.
  • the thumbnail image generator 395 may resize a display image output from the display image buffer 383 to generate a thumbnail image so that the thumbnail image may be included as a JPEG image, which may be displayed on the display unit 160 under the control of the application processing controller 360.
  • the application processor 140 may capture a still image during a preview mode or during a moving image photographing mode.
  • the image signal processor 130 continuously generates and transmits a display image and a compression image (or display image, captured image, photographed information) in an interleaved data mode, Accordingly, the application processor 140 may continuously receive the compression image and the display image.
  • FIG. 3 illustrates coders as a constituent element for compressing a still image and a moving image included in the image signal processor 130 and the application processor 140.
  • a configuration of compressing an image and releasing the compressing may be integrated as a single chip, which may be a Coder and DECoder (CODEC).
  • the still image coder 340 (e.g., the JPEG coder 340) may be a still image CODEC (JPEG CODEC).
  • a moving image coder 390 (e.g., the MPEG coder 390) of the application processor 140 may be a moving image CODEC (MPEG CODEC).
  • the exemplary embodiment of the present invention will be described based on an operation of displaying and compression-storing an image photographed by the camera 110. If compression-storing of images is performed, there is a need to perform image decoding and/or data retrieval or releasing from storage and decompression of the compressed image, and displaying a still image (e.g., JPEG image) stored in the storage unit 150.
  • the application processor 140 may further include a still image decoder (or still image CODEC).
  • the application processor 140 may decode (compression-releases) a compressed image through a still image decoder (still image CODEC) which is not shown, and display the decoded image on the display unit 160.
  • a still image decoder still image CODEC
  • the image signal processor 130 may receive an image output from the camera 110 to generate a display image for preview and a compression image for capture, and generate a display image, a compression image, and photographed information.
  • the application processing controller 360 informs the image signal processor 130 of the camera operation command, and the image processing controller 310 operates the camera 110.
  • the image output from the camera 110 is inputted to an image processing part 320.
  • the camera operation command may initiate performing a capture mode while performing a preview mode, a moving image photographing mode, a still image capture mode, and/or a moving image photographing mode.
  • the image signal processor 130 generates a display image, a compression image, and photographed information at every frame period. Generation of a frame image by the image signal processor 130 may be achieved at a frame rate of 30fps or greater (e.g., 60fps), or any preset frame rate.
  • the image processing part 320 3A processes a raw image received from the camera 110, performs color interpolation and/or performs color conversion of the raw image into a YUV image for every frame period.
  • the image may be an image having a full HD resolution or greater.
  • the JPEG coder 340 compression-encodes an image output from the image processing part 320 in a JPEG scheme to generate a compression image for being stored upon a capture request.
  • the compression data may be generated using a compression coder of another scheme such as JPEG.
  • the exemplary embodiment of the present invention may be implemented such that the compression data are generated in a JPEG scheme.
  • the scaler 330 reduces an image output from the image processing part 320 to a preset data size to generate a display image for preview to be displayed on the display unit 160.
  • a method of reducing the image may include a typical scaling method, a cropping method, and a resizing method.
  • a display image may be generated by one or a combination of at least two of the foregoing methods, or any other known image scaling methods.
  • the display image may include an image to be displayed on the display unit 160 and/or an image to be stored upon moving image capture.
  • the scaler 330 of the image signal processor 130 may control data sizes of a display image for a preview mode and a display image for storing a moving image to a different size.
  • the scaler 340 may be configured by a display image generator.
  • the display image generator may have a configuration including a color conversion function (function of converting a raw image into a YUV image) and a scaling function.
  • the image signal processor 130 acquires various sensor information when photographing the display image at every frame period and an image by the camera 110 upon generation of the compression image to generate the various sensor information as photographed information.
  • the photographed information includes outputs of the camera 110 and the sensor 120.
  • the image signal processor 130 may generate photographed information using information of a photographing time point acquired by the camera 110 and the sensor 120.
  • the image signal processor 130 may first generate photographed information using information acquired by the camera 110, the application processor 140 generates second photographed information using information acquired by the sensor 120, and combines the second photographed information with the first photographed information to generate the photographed information.
  • An image signal processor 130 and/or an application processor 140 receiving an image every frame generated from the camera 110, generate a YUV image for display and a compression image for capture, and generate photographed information of a corresponding frame image (configured by image data and embedded data).
  • the embedded data may include data and information, as meta data, as illustrated in the following Table 1.
  • Flash 4byte Flag capable of checking full light amount of flash, use when flash operates ISO 4byte Store ISO information EV 4byte Store Exposure Value Data type 4byte Store data type of JPEG, YUV, meta data Data size 4byte Store the size of variably changed JPEG, use when JPEG image is stored in a buffer Face detection 4byte Store face detection function on/off Face number 4byte Store the number of recognized faces FD start x 4byte Store start x value of square coordinate in which a face is recognized. Store the number of recognized faces like FD[0].sx, FD[1].sx FD start y 4byte Store start y value of square coordinate in which a face is recognized.
  • the user may set various functions to take a photograph using the foregoing photographed information (meta data).
  • the user may search for a preset value meeting a situation using frequently photographed subject information and camera menu information. In this case, if a human face is detected upon preview, a menu frequently used upon photographing a corresponding person accesses a person database (DB) or a menu which provides an excellent result upon use is automatically set.
  • DB person database
  • a camera user photographs other people such as their family, relations, lover, and talented people such as artists and movie stars.
  • the user may set a menu to assist in setting photograph parameters during photographing (e.g., smile shot, beauty shot, portrait scene mode, AF mode, object tracking, ISO, effect, white balance, exposure, metering, zooming, flash, etc.).
  • the user may confirm at least some settings with respect to whether the camera 110 is photographing scenery photograph to be suited to a scenery photograph (e.g., outdoor visible, flash, ISO, effect, white balance, exposure, metering, landscape scene mode, Fan focus mode, etc.).
  • the photographed image may be stored based on an experience with respect to the user profile; for example, in a case of a romantic photograph, the image may be stored simultaneously with photographing as a romantic album or to be reflected, copied, transmitted and/or saved to a designated social networking service (SNS) or a cloud-based storage and distribution system.
  • SNS social networking service
  • the photographed image may be easily confirmed.
  • the multiplexer 350 multiplexes the display image, the compression image, and the photographed information in an interleaved data mode to be output in one frame to the application processor 140.
  • FIG. 4 is a diagram illustrating the organization of image data in the exemplary embodiment of the present invention, with camera data including YUV data, JPEG data, and embedded data forming a YUV image, a JPEG image, and photographed information, respectively, constituting one frame according to the exemplary embodiment of the present invention.
  • FIGS. 5a to 5f are diagrams illustrated a data transfer format according to the exemplary embodiment of the present invention. More specifically, FIG. 5a illustrates a transfer format of an interleaved data mode.
  • FIGS. 5b to 5f are views illustrating a transfer format of a hybrid interleaved data mode.
  • the image signal processor 130 generates YUV data of a YUV image, JPEG data of a JPEG image, and embedded data and/or meta data as photographed information for additional camera functions.
  • the multiplexer 350 multiplexes the generated YUV image, JPEG image, and photographed information in the format of FIG. 4 and transmits the multiplexed data to the application processor 140.
  • JPEG data representing the compression image conveys an original image having UHD resolution acquired from the camera 110, which are data to be stored as a still image upon a capture request.
  • the size of the JPEG data may be determined by an image sensor 220 of the camera 110, and may have a data size, for example, equal to or greater than 8M bytes.
  • the YUV data represents an image to be previewed by being output to the display unit 160 and/or for recording images, and are scaled to a size smaller than the size of the JPEG data.
  • the YUV data may be used as data to be stored as a moving image other than a preview image. That is, display images stored in the display image buffer 383 are compressed by the MPEG coder 390 while being displayed on the display unit 160 and the compressed display image may be stored in the storage unit 150.
  • the photographed information may be configured by various settings when acquiring an image by the camera 110 and sensor information detected by a sensor 120 located internal to or external to the camera 110.
  • the photographed information is not included in the compression image but is independently stored. That is, the image signal processor 130 generates photographed information using information detected from the camera 110 and the sensor 120.
  • the photographed information is stored corresponding to the display image and the compression image for each frame.
  • information capable of being acquired from the sensor 120 is generated by the application processor 140, and may be merged when the application processor 140 buffers photographed information. Thereafter, the user may efficiently store and process an image using the photographed information.
  • the camera frame data according to the exemplary embodiment of the present invention is generated by the image signal processor 130, and the generated camera frame data are transmitted to the application processor 140 in a transfer format as shown in FIGS. 5a to 5f .
  • the image signal processor 130 multiplexes the display image, the compression image, and the photographed information in a single frame, and transmits the single frame to the application processor 140.
  • the image signal processor 130 converts the frame images and information into parallel data, converts the frame images and information into serial data, and transmits the serial data to the application processor 140.
  • the YUV data may be 2MB ((Full HD): 1920*1080*2 (YUV 422 format is 2 Bytes per pixel)).
  • the JPEG data may represent an image obtained by compression-encoding a UHD image.
  • Meta data, being the photographed information, may be stored in 4K bytes.
  • the interleaved data transmission mode as illustrated in FIG. 5a cannot be used.
  • the image signal processor 130 and/or the application processor 140 does not completely support the interleaved data in an overload state, that is, a state having a limitation to allocation of the bus bandwidth, the interleaved data transmission mode as shown in FIG. 5a may not be used.
  • the compression image is an image to implement zero shutter-lag upon capture of a still image
  • the display image is an image displayed on the display unit 160 and MPEG coded and stored in the storage unit 150 in a moving image photographing mode.
  • the compression image is an image used in the present invention to implement the zero shutter-lag when capturing a still image, and is buffered in the application processor 140.
  • a compression image of a frame corresponding to a generated time point of the capture command among the buffered compression images is accessed and stored as a still image. Accordingly, if the compression image is suitably transmitted in an interleave scheme, the still image may be acquired with zero shutter-lag to satisfy a bus bandwidth necessary between the image signal processor 130 and the application processor 140.
  • the exemplary embodiment of the present invention sets a bus bandwidth in which the image signal processor 130 transmits the compression image in a hybrid interleaved scheme.
  • FIGS. 5b to 5f illustrate implementation examples of a hybrid data mode according to the exemplary embodiment of the present invention.
  • FIG. 5b illustrates a method which transmits a YUV image and photographed information for each frame, and transmits the compression in a first frame (or second frame) at two frame intervals.
  • the compression image has a structure which is transmitted at a frame rate of 15fps.
  • the bandwidth becomes 60% of the interleaved data mode as illustrated in FIG. 5a .
  • FIG. 5c illustrates an example of a hybrid interleaved data mode which transmits a YUV image and photographed information for each frame, transmits a compression image in a first frame (or second frame) and a thumbnail image in another frame at two frame intervals.
  • the compression image has a structure in which the image is transmitted at a frame rate of 15fps.
  • FIG. 5d illustrates an example of a data division transmitting method which transmits the YUV image and photographed information for each frame and makes the compression image to have the same size and transmits the compression image.
  • the compression image may be an image obtained by compressing a YUV image having UHD resolution.
  • the compression image may have a variable data size according to the image photographed by the camera 110. That is, the data size of a compression image generated in each frame has different sizes according to the image. Accordingly, the compression image may be transmitted by making the compression image to have the same size for each frame.
  • the image signal processor 130 includes a memory and appropriate software or instructions to divide data to have substantially or exactly the same size.
  • the image processing controller 310 of the image signal processor 130 may divide a container part (e.g., JPEG container) of the compression image as illustrated in FIG. 5e in the memory, and averages an MIPI limit and transmit data. In this case, if a bandwidth is sufficient and JPEG data is small (JPEG1, JPEG4 of FIG. 5e ) upon division of the data, a plurality of JPEG images are simultaneously transmitted to one frame.
  • the JPEG image is large, such as the block of image data labeled JPEG 5_2 of FIG. 5e , the JPEG image is divided into a plurality of frames and the frames are transmitted so that the problem of the MIPI transmission limit may be solved.
  • the memory may be set to the size of a compression image to be transmitted for each frame.
  • the image processing controller 310 divides a compression image including header information, which is information indicating start and end of a corresponding compression image, having a boundary of a compression image of each frame.
  • FIG. 5f illustrates an example when resolution of a captured image set by the user is lower than the resolution of the display image.
  • the image signal processor 130 transmits only the display image and the photographed information.
  • the image signal processor 130 When the capture of an image is requested in the application processor 140, the image signal processor 130 generates the display image as a compression image.
  • the image signal processor 130 may implement zero shutter-lag upon image capture.
  • the bandwidth has a frame rate of 30fps, it becomes 33% of the interleaved data mode as illustrated in FIG. 5a .
  • a frame interval transmitting the compression image may be set to two frames or greater. If the frame interval is large, a transmission bandwidth may be reduced but the zero shutter-lag may not be exactly implemented. Accordingly, it is preferable that an interval of transmitting the compression image is suitably set as necessary.
  • the multiplexed and transmitted display image, compression image, and photographed information are demultiplexed by the demultiplexer 355 of the application processor 140.
  • the demultiplexed image and information are parsed by the parser 370 and are buffered by the display image buffer 383, the compression image buffer 385, and the photographing information buffer 387 of a ring buffer 380.
  • the buffer 380 has a structure of a ring buffer. If data having the set number of frames are buffered, such data are overwritten.
  • the buffers 383, 385, and 387 have N ring buffer structures capable of storing corresponding data of N frames, respectively, and buffer corresponding data under the control of the application processing controller 360.
  • N buffers capable of storing a JPEG image of N frames are linked with each other by a ring buffer structure, and JPEG images may be stored in corresponding buffers according to a transmitted data mode as illustrated in FIGS. 5a to 5f .
  • Frames of the display image buffer 383, the compression image buffer 385, and the photographing information buffer 387 are synchronized with each other according to a frame count value. Accordingly, the application processing controller 360 may confirm a frame count value to a display image, a compression image, and photographed information of the same frame.
  • the application processing controller 360 buffers the display image in the display image buffer 383 and outputs the display image to be displayed as a preview image.
  • the display image buffered in the display image buffer 383 is transferred to the MPEG coder 390, and the MPEG coder 390 codes a received display image and stores the coded display image in the storage unit 150.
  • the application processing controller 360 detects the capture request command and stores a set or selected compression image among the compression images buffered in the compression image buffer 385 in the storage unit 150. In this case, as shown in FIGS.
  • the application processing controller 360 may implement the zero shutter-lag by selecting and storing a buffering compression image of a frame closest to, or substantially adjacent to or subsequent to, another frame in time when the capture command is generated.
  • the application processing controller 360 confirms a frame when the capture is requested, confirms frame count information of a JPEG image stored by dividing data, and accesses and stores a buffered compression image of the confirmed frame in the storage unit 150.
  • the application processing controller 360 couples the divided compression images and stores the coupled compression images.
  • the stored compression image may be a JPEG image.
  • the stored compression image may include a JPEG header, a thumbnail image, and a JPEG image.
  • FIG. 6 is a flowchart illustrating a method of generating and processing a display image and a compression image for each frame by the camera 110 according to the exemplary embodiment of the present invention.
  • an application processor 140 detects and informs the image signal processor 130 of the camera operation request, and the image signal processor 130 operates the camera 110.
  • the image signal processor 130 operates the camera according to preset information (e.g., focus, zoom, white balance) of the camera 110.
  • the camera 110 performs a preview mode.
  • the user may request a moving image photographing in a state of performing the preview mode.
  • the application processor 140 detects the request in step 611. If moving image photographing mode is not requested, the method loops back to step 611. However, if the moving image photographing mode is requested in step 611, the method performs a moving image photographing mode in subsequent steps.
  • the image signal processor 130 performs the same operation regardless of a preview mode, a capture mode, and a moving image photographing mode. Accordingly, if the camera 110 is operated, the image signal processor 130 acquires an image from the camera 110 in step 613.
  • the camera 110 photographs images having a full HD resolution or greater which may be an image having UHD resolution
  • the image signal processor 130 acquires a frame image in step 613.
  • the image signal processor 130 acquires the image at a preset frame rate, and the frame rate may be 30fps.
  • the image signal processor 130 After acquiring the frame image, the image signal processor 130 generates a display image in a corresponding frame in step 615, and generates a compression image in step 617.
  • the compression image is determined according to the bus bandwidths of the image signal processor 130 and the application processor 140, and an interleaved data mode is used according to the determined bandwidths.
  • the interleaved data mode may be one of transmitting methods as illustrated in FIGS. 5a to 5f .
  • FIG. 7 is a flowchart illustrating a method of generating the compression image in step 617 of FIG. 6 .
  • the image signal processor 130 confirms a transmission mode in step 711. In this case, if the bandwidth is sufficient, the image signal processor 130 detects an interleaved data mode in step 713. If the interleaved data mode is not detected, the method proceeds to step 719. Otherwise, if the interleaved data mode is detected in step 713, the method generates a compression image for each frame as illustrated in FIG. 5a in step 715, multiplexes the compression image with the display image, and transmits the multiplexed image in step 717, and returns to complete step 617 in FIG. 7 .
  • the image signal processor 130 detects the condition of insufficient bandwidth, and determines whether a current mode is a hybrid interleaved data mode which transmits the compression image at a predetermined frame interval in step 719.
  • the image signal processor 130 generates a compression image at a frame interval set as illustrated in FIGS. 5b and 5c , and the method proceeds to step 717.
  • step 719 when the current mode is not the hybrid interleaved data mode, the method proceeds to step 723 to detect if the current mode is a hybrid data mode dividing data in step 723. If the image signal processor 130 detects the hybrid data mode in step 723, the method proceeds to step 724 and generates a compression image for each frame in step 723.
  • the image signal processor 130 integrates or divides the generated compression image of a frame according to the size of the compression image in a memory in step 727, and proceeds to step 717. That is, as shown in FIG. 5e , if the size of the compression image is small; that is, below a predetermined threshold size, the image signal processor 130 integrates a compression image of a plurality of frames in one frame to make a frame compression image.
  • the image signal processor 130 divides the compression image of a corresponding frame so that the compression image is separately transmitted to the frames. Referring back to step 723, if hybrid data mode is not detected, for example, if resolution of a captured image set by the user is lower than that of the display image, the image signal processor 130 does not generate the compression image in step 725, and the method proceeds to step 717.
  • the image signal processor 130 multiplexes the generated compression image and a display image to transmit one frame.
  • the multiplexed display image may have one of the transmitting formats as illustrated in FIGS. 5a to 5d .
  • the photographed information in the form of meta data is not transmitted. That is, FIG. 6 is a diagram illustrating a procedure of an operation when one frame is configured by a display image and a compression image, and photographed information is not transmitted in this case.
  • a frame rate of frame images transmitted from the image signal processor 130 may have a frame rate of 30fps or greater. Accordingly, the image signal processor 130 may generate a display image of 30 frames per second and a compression image of the number of frames determined according to the interleaved data mode. The generated display image and compression image are multiplexed by the multiplexer 350, and the multiplexed image is transferred to the application processor 140. Referring back to FIG. 6 , after step 617, the application processor 140 buffers the display image and the compression image, respectively in step 619. The display image is then displayed through the display unit 160 in step 621 as a preview image and is coded through an MPEG coder 390, and the coded image is stored in the storage unit 150.
  • the application processor 140 detects for the capture request command in step 623. If no capture request command is detected in step 623, the method proceeds to step 627. However, when the capture request command is detected in step 623, the compression image buffer 385 selects a preset compression image from the buffered compression images and stores the selected compression image in the storage unit 150 in step 625 and loops back to step 615. In this case, a method of setting the compression image may be performed using various schemes.
  • the compression image buffer 385 selects a compression image corresponding to a frame when the capture request command is generated and stores the selected compression image in the storage unit 150. For example, in a case where shutter delay (shutter lag) is set to two frames, if the capture request command is generated, the compression image buffer 385 stores compression data of a previous frame of two buffered frames in the storage unit 150 to implement zero shutter-lag. In this case, there is no compression image of a corresponding frame (when the compression image is transmitted as illustrated in FIG. 5b or 5c ), and the compression image buffer 385 selects a buffered compression image of a frame closest to, or substantially adjacent to or subsequent to, another frame in time implementing the zero shutter-lag.
  • shutter delay shutter lag
  • the compression image buffer 385 may set a function of selecting desired compression images while displaying the thumbnail image or the display image.
  • the foregoing operation is repeatedly performed by steps 615-625 until a moving image photographing operation is terminated.
  • the image signal processor 130 and the application processor 140 detect the terminated moving image photographing in step 627. If no termination of a moving image photographing is detected in step 627, the method loops back to step 615. Otherwise, if moving image photographing is detected in step 627, the method terminates a moving image photographing mode and the method ends.
  • the image signal processor 130 and the application processor 140 perform a preview mode. Any additional or remaining operations, except for an operation of storing a moving image by the application processor 140, are performed by the same procedures of FIG. 6 . That is, the operation of the image signal processor 130 may be identically performed in a preview mode or a moving image photographing mode.
  • FIG. 8 is a flowchart illustrating a method of generating and processing a display image, a compression image, and photographed information for each frame by the camera 110 according to the exemplary embodiment of the present invention.
  • FIG. 8 performs substantially the same operations and steps as that of the FIG. 6 except that the image signal processor 130 generates and transmits photographed information, the application processor 140 buffers the transmitted display image, compression image, and photographed information, and adds an output of the sensor 120 to the photographed information transmitted from the image signal processor 130 during buffering to generate and buffer the photographed information.
  • the camera 110 performs a preview mode.
  • the user may request moving image photographing.
  • the application processor 140 detects the request of the moving image photographing in step 811. If moving image photographing mode is not requested, the method loops back to step 811. However, if the moving image photographing mode is requested in step 811, the method performs a moving image photographing mode in subsequent steps. However, if the camera 110 is operated, the image signal processor 130 performs the same operation regardless of a preview mode, a capture mode, and a moving image photographing mode.
  • the image signal processor 130 acquires an image and photographed information from the camera 110 in step 813.
  • the camera 110 photographs an image having a full HD resolution or greater, which may be an image having UHD resolution
  • the image signal processor 130 acquires a frame image in step 813.
  • the image signal processor 130 acquires an image at a preset frame rate, and the frame rate may be 30fps.
  • the image signal processor 130 After acquiring the frame image, the image signal processor 130 generates a display image and photographed information in a corresponding frame in step 815, and generates a compression image in 817.
  • the compression image is determined according to the bus bandwidths of the image signal processor 310 and the application processor 140, and an interleaved data mode is used according to the determined bandwidths.
  • the interleaved data mode is one of the possible transmitting methods used by the present invention as illustrated in FIGS. 5a to 5f .
  • a procedure of generating the compression image in step 817 is performed in the same manner as a procedure of the operation of FIG. 7 described above, and a transfer format becomes one of the formats shown in FIGS. 5a to 5e .
  • a frame rate of frame images transmitted from the image signal processor 130 may be 30fps or greater (e.g., 60 fps). Accordingly, the image signal processor 130 may generate a display image of 30 frames per second and a compression image having the number of frames determined according to the interleaved data mode.
  • the foregoing generated displayed image, compression image, and photographed image are multiplexed by the multiplexer 350, and transferred to the application processor 140.
  • the application processor 140 buffers the display image, the compression image, and the photographed information, respectively in step 819.
  • the display image is then displayed as a preview image through the display unit 160 in step 821 and coded through the MPEG coder 390, and the coded display image is stored in the storage unit 150 in step 821.
  • the application processor 140 detects for the generation of the capture request command in step 823. If no capture request command is detected in step 823, the method proceeds to step 827. However, when the capture request command is detected in step 823, the compression image buffer 385 selects a preset compression image from the buffered compression images and stores the selected preset compression image in the storage unit 150 in step 825, and loops back to step 813. In this case, a method of setting the compression image may be performed in various schemes.
  • the compression image buffer 385 selects a compression image corresponding to a frame when the capture request command is generated and stores the selected compression image in the storage unit 150. For example, in a case where shutter delay (shutter lag) is set to two frames, if the capture request command is generated, the compression image buffer 385 stores compression data of a previous frame of two buffered frames in the storage unit 150 to implement zero shutter-lag. In this case, there is no compression image of a corresponding frame (when the compression image is transmitted as illustrated in FIG. 5b or 5c ), and the compression image buffer 385 selects a buffered compression image of a frame closest to, or substantially adjacent to or subsequent to, another frame in time implementing the zero shutter-lag. Second, the compression image buffer 385 may set a function of selecting desired compression images while displaying the thumbnail image or the display image.
  • step 823 if no capture request command is detected in step 823, and so the moving image photographing may have been terminated, the image signal processor 130 and the application processor 140 detect the terminated moving image photographing in step 827. If no termination of a moving image photographing is detected in step 827, the method loops back to step 813. Otherwise, if moving image photographing is detected in step 827, the method terminates a moving image photographing mode and the method ends. In this case, if the user does not stop operation of the camera, the image signal processor 130 and the application processor 140 perform a preview mode. Any additional or remaining operations except for an operation of storing a moving image by the application processor 140 are performed by the same procedures of FIG. 8 . That is, the operation of the image signal processor 130 may be identically performed in a preview mode or a moving image photographing mode.
  • FIG. 9 is a block diagram illustrating a configuration of a portable terminal apparatus including the camera 110 according to the exemplary embodiment of the present invention.
  • the portable terminal apparatus includes the camera 110 and the sensor 120 as described in the exemplary embodiment of FIG. 1 , to perform a function of acquiring an image through the sensor 120 when the camera 110 is activated.
  • the sensor 120 may include at least one sensor mounted in the camera 110, which includes at least one of a GPS receiver, a gyro sensor, an acceleration sensor, an altitude sensor, and a direction sensor.
  • An image signal processor 130 processes an image acquired from the camera 110 to generate a display image on a display unit 160 and a compression image to be stored in response to a capture request.
  • the display image may include a YUV image.
  • the compression image may include a JPEG image.
  • the image signal processor 130 determines a method of generating and transmitting the compression image according to a bus bandwidth.
  • a transfer format of the compression image may be one of the formats shown in FIGS. 5a to 5f .
  • the application processor 140 performs various application functions of a portable terminal apparatus.
  • the application processor 140 may receive, as input, the output of the sensor 120, and process various sensor applications according to a request of a user.
  • the application processor 140 may receive, as input, the output of GPS reception to process a navigation function and an application detecting regional information according to a location of a camera 110 and/or the portable terminal apparatus.
  • the application processor 140 may execute an application which executes a preset function in a currently executed application according to movement of the portable terminal apparatus.
  • the application processor 140 buffers the display image, the compression image, and the photographed information transmitted from the image signal processor 130 during operation of the camera 110, and controls the display unit 160 to output and display the display image according to the exemplary embodiment of the present invention.
  • the application processor 140 operates the MPEG coder 390 to compression-code the display image in an order determined by the configuration of data in a frame and to store the compression-coded display image as a moving image. If a capture request command of a still image of the user is generated in the preview mode or a moving image photographing mode, the application processor 140 selects a compression image corresponding to a frame when the capture command is generated from the buffered compression images and stores the selected compression image in the storage unit 150 to implement a zero shutter-lag function. In this case, as shown in FIGS.
  • the application processor 140 may select and store a buffering compression image of a frame closest to, or substantially adjacent to or subsequent to, another frame in time when the capture command is generated to implement the zero shutter-lag.
  • the application processor 140 confirms a frame when capture is requested, confirms frame count information of a JPEG image stored by dividing data, and accesses and stores a buffered compression image of the confirmed frame in the storage unit 150.
  • the JPEG image is divided (for example, the JPEG 3 image and the JPEG 5_2 image in FIGS. 5d and 5e , respectively)
  • the application processor 140 couples the divided compression images and stores the coupled compression images.
  • a communication unit 920 performs a communication function with an external device or a base station.
  • the communication unit 920 may include a transmitter having a frequency up converter for up-converting a transmitted signal to an RF band and a power amplifier, a low-noise amplifier for low-noise amplifying an RF received signal, and a frequency down converter for down-converting an RF signal to a baseband.
  • the communication unit 920 may include a modulator for modulating the transmitted signal and transferring the modulated signal to the transmitter and a demodulator for demodulating a signal output from the receiver.
  • the modulator and the demodulator may include a modulator and a demodulator of a WCDMA, GSM, LTE, WIFI, or WIBRO scheme.
  • a terminal controller 910 controls an overall operation of the portable terminal apparatus, and performs voice call and data communication functions.
  • the terminal controller 910 may include a modulator and a demodulator of the communication unit 920.
  • the communication unit 920 may include only an RF communication unit.
  • the terminal controller 910 may perform a function of modulating/demodulating and coding/decoding the communication signal.
  • An audio processor 930 may be connected to a handset of the portable terminal apparatus and processes a call voice during voice communication under the control of the terminal controller 910.
  • the memory 940 may be a work memory in which programs of the terminal controller 910 and the application processor 140 are loaded. In this case, the memory 940 may be configured to include at least a RAM.
  • the input unit 170 generates and provides various operation commands and data input signals of the portable terminal apparatus to the terminal controller 910 and the application processor 140.
  • the input unit 170 may respond to user selections and inputs to generate and provide a camera operation command and a capture command to the application processor 140.
  • the input unit 170 may generate a command requesting display of photographed information and/or thumbnail images according to a user request upon capture of the image.
  • the display unit 160 displays an operation of the portable terminal apparatus and an application execution state under the control of the terminal controller 910 and the application processor 140.
  • the display unit 160 displays a display image output from the application processor 140 in a preview mode according to the exemplary embodiment of the present invention.
  • the display unit 160 may display photographed information and/or thumbnail images output from the application processor 140.
  • the input unit 170 may include a touch panel for detecting a touch input of the user.
  • the display unit 160 may include an LCD or OLED panel for displaying data and images generated during execution of a program.
  • the input unit 170 and the display unit 160 may be an integral touch screen.
  • the input unit 170 may include at least one button or multiple buttons located on or accessible at the exterior of the camera 110.
  • the storage unit 150 may include a program memory storing an operation program of the portable terminal apparatus and programs according to the exemplary embodiment of the present invention, and a data memory storing data for executing a program and data generated during execution of the program.
  • the storage unit 150 may be configured by a non-volatile memory such as a flash memory.
  • the storage unit 150 stores a buffered compression image in the application processor 140 upon a capture request according to the exemplary embodiment of the present invention.
  • a portable terminal apparatus having a configuration as illustrated in FIG. 9 may perform incoming/outgoing call and communication services, and process an image photographed by the camera 110 in a camera photographing mode according to the exemplary embodiment of the present invention.
  • the terminal controller 910 loads a call and communication application in the storage unit 150 and operation programs of the portable terminal apparatus to the memory 940.
  • the application processor 140 loads a program for operating corresponding applications stored in the storage unit 150 to the memory 940.
  • the memory 940 performs a work memory function in which operation programs of the portable terminal apparatus and application programs are loaded.
  • the application processor 140 may share the memory 940 with the terminal controller 910, and the memory 940 may be independently configured and used.
  • the application processor 140 processes an image processing application of a camera 110 and various applications of the portable terminal apparatus. To process the foregoing various applications, the application processor 140 may receive, as input, the output of the sensor 120. The application processor 140 may execute an application using an output of the sensor 120 or control a processing operation of an application according to the output of the sensor 120. The terminal controller 910 may receive, as input, the output of the sensor 120, and perform a call or communication service or control the call or communication service according to the output of the sensor 120.
  • the user may input, select and/or generate an outgoing phone number and a call request signal.
  • the terminal controller 910 having detected the outgoing phone number and call request signal, controls the communication unit 920 to form a call path, and performs an outgoing call function through a base station. If an incoming call is detected and/or received, the terminal controller 910 detects the incoming call, and generates an incoming notification and/or alarm through the audio processor 930, and displays incoming information through the display unit 160. In this case, if the user responds to accept and conduct the incoming call through the input unit 170, the terminal controller 910 forms an incoming call path through the communication unit 920 to perform a call service.
  • the portable terminal apparatus conducts data communication by forming a communication path in an operation similar to the operations for performing an incoming or outgoing call.
  • the compression image may be transmitted in a hybrid transmitting method.
  • the camera 110 and portable terminal apparatus including such a camera 110 includes an image signal processor 130 for acquiring an image from the camera 110 to generate a display image and a compression image, multiplexing the display image and the compression image, and transmitting the multiplexed frame image; and an application processor 140 for buffering the multiplexed frame image and storing a preview using the buffered frame image.
  • an image signal processor 130 for acquiring an image from the camera 110 to generate a display image and a compression image, multiplexing the display image and the compression image, and transmitting the multiplexed frame image
  • an application processor 140 for buffering the multiplexed frame image and storing a preview using the buffered frame image.
  • the exemplary embodiment of the present invention can simultaneously store the image of the full HD resolution and the still image of the UHD resolution by transmitting the compression image at a preset frame interval in a hybrid scheme according to the bus bandwidth.
  • the above-described apparatus and methods according to the present invention can be implemented in hardware, firmware or as software or computer code that can be stored in a non-transitory recording medium such as a CD ROM, a RAM, a ROM, a floppy disk, DVDs, a hard disk, a magnetic storage media, an optical recording media, or a magneto-optical disk or computer code downloaded over a network originally stored on a remote recording medium, a computer readable recording medium, or a non-transitory machine readable medium and to be stored on a local recording medium, so that the methods described herein can be rendered in such software that is stored on the recording medium using a general purpose computer, a digital computer, or a special processor or in programmable or dedicated hardware, such as an ASIC or FPGA.
  • a non-transitory recording medium such as a CD ROM, a RAM, a ROM, a floppy disk, DVDs, a hard disk, a magnetic storage media, an optical recording media, or a magneto
  • the computer, the processor, microprocessor controller or the programmable hardware include memory components, e.g., RAM, ROM, Flash, etc. that may store or receive software or computer code that when accessed and executed by the computer, processor or hardware implement the processing methods described herein.
  • memory components e.g., RAM, ROM, Flash, etc.
  • the execution of the code transforms the general purpose computer into a special purpose computer for executing the processing shown herein.
  • the program may be electronically transferred through any media such as communication signals transmitted by wire/wireless connections, and their equivalents.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Television Signal Processing For Recording (AREA)
  • Closed-Circuit Television Systems (AREA)
EP13155295.2A 2012-02-16 2013-02-14 Vorrichtung und Verfahren zur Übertragung eines Kamerarahmenbildes Active EP2629503B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020120016038A KR101905621B1 (ko) 2012-02-16 2012-02-16 카메라의 프레임 이미지 전송 장치 및 방법

Publications (3)

Publication Number Publication Date
EP2629503A2 true EP2629503A2 (de) 2013-08-21
EP2629503A3 EP2629503A3 (de) 2017-03-08
EP2629503B1 EP2629503B1 (de) 2019-04-24

Family

ID=47826871

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13155295.2A Active EP2629503B1 (de) 2012-02-16 2013-02-14 Vorrichtung und Verfahren zur Übertragung eines Kamerarahmenbildes

Country Status (4)

Country Link
US (1) US9516234B2 (de)
EP (1) EP2629503B1 (de)
KR (1) KR101905621B1 (de)
CN (1) CN103260037B (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3044949A4 (de) * 2013-09-12 2017-04-26 Canon Kabushiki Kaisha Bildaufnahmevorrichtung und steuerungsverfahren dafür

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9300880B2 (en) 2013-12-31 2016-03-29 Google Technology Holdings LLC Methods and systems for providing sensor data and image data to an application processor in a digital image format
US9313343B2 (en) 2014-02-20 2016-04-12 Google Inc. Methods and systems for communicating sensor data on a mobile device
US9307234B1 (en) * 2014-07-23 2016-04-05 American Express Travel Related Services Company, Inc. Interactive latency control with lossless image optimization
KR102423364B1 (ko) 2015-08-12 2022-07-22 삼성전자 주식회사 영상을 제공하는 방법 및 이를 지원하는 전자 장치
WO2017045126A1 (zh) * 2015-09-15 2017-03-23 华为技术有限公司 图像处理方法和终端
KR102399764B1 (ko) * 2015-09-22 2022-05-19 삼성전자 주식회사 전자 장치 및 촬영 방법
KR101954298B1 (ko) * 2015-10-01 2019-03-05 주식회사 케이티 하이브리드 스트리밍 방법, 스트리밍 서버 및 시스템
US10284861B2 (en) * 2016-12-09 2019-05-07 Advanced Micro Devices, Inc. Concurrent image compression and thumbnail generation
CN106650691A (zh) * 2016-12-30 2017-05-10 北京旷视科技有限公司 图像处理方法和图像处理设备
US10455121B2 (en) 2017-05-12 2019-10-22 Microsoft Technology Licensing, Llc Representing advanced color images in legacy containers
KR102434417B1 (ko) * 2017-07-20 2022-08-22 삼성전자주식회사 이미지 데이터 및 이미지 촬영의 제어와 관련된 데이터의 크기에 기반하여, 이미지 데이터 및 이미지 촬영의 제어와 관련된 데이터를 전송하는 방법 및 이를 지원하는 전자 장치
CN110830782A (zh) * 2019-11-20 2020-02-21 南通嘉骏信息科技有限公司 一种基于fpga实现mipi csi-2的raw到yuv的转换方法
KR20230035866A (ko) 2021-09-06 2023-03-14 삼성전자주식회사 연속된 촬영 입력을 처리하는 전자 장치 및 그의 동작 방법
US12526535B2 (en) 2021-12-22 2026-01-13 Samsung Electronics Co., Ltd. Electronic device and method of generating high-definition image
JP7378047B1 (ja) * 2022-12-12 2023-11-13 パナソニックＩｐマネジメント株式会社 決済端末及び決済支援方法
CN117319591B (zh) * 2023-09-25 2024-03-12 深圳市厚朴科技开发有限公司 一种视频传输方法、装置、计算机可读存储介质及设备

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001128050A (ja) * 1999-10-25 2001-05-11 Sanyo Electric Co Ltd ディジタルカメラ
JP3977245B2 (ja) * 2002-12-26 2007-09-19 キヤノン株式会社 再生装置
US7346220B2 (en) * 2003-07-23 2008-03-18 Seiko Epson Corporation Method and apparatus for reducing the bandwidth required to transmit image data
CN100459695C (zh) 2005-08-18 2009-02-04 腾讯科技（深圳）有限公司 视频数据传输方法及其传输系统
US7787048B1 (en) * 2005-09-08 2010-08-31 Nvidia Corporation Motion-adaptive video de-interlacer
CN100386717C (zh) 2006-12-08 2008-05-07 张承 用于远程显示的计算机图像采集方法及装置
JP2009065320A (ja) * 2007-09-05 2009-03-26 Casio Comput Co Ltd 撮像装置
KR101661574B1 (ko) * 2009-11-10 2016-10-14 삼성전자주식회사 셔터-딜레이를 줄이기 위한 카메라 모듈
US8446484B2 (en) * 2010-04-21 2013-05-21 Nokia Corporation Image processing architecture with pre-scaler
TWI418210B (zh) * 2010-04-23 2013-12-01 Alpha Imaging Technology Corp 避免快門延遲之影像擷取模組及影像擷取方法
JP5565226B2 (ja) * 2010-09-10 2014-08-06 富士通株式会社 画像データ処理装置および画像データ処理方法
KR20130094113A (ko) 2012-02-15 2013-08-23 삼성전자주식회사 카메라 데이터 처리 장치 및 방법

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3044949A4 (de) * 2013-09-12 2017-04-26 Canon Kabushiki Kaisha Bildaufnahmevorrichtung und steuerungsverfahren dafür
US10082723B2 (en) 2013-09-12 2018-09-25 Canon Kabushiki Kaisha Image capturing apparatus for generating a high dynamic range video frame from multiple image frames with different characteristics

Also Published As

Publication number Publication date
EP2629503A3 (de) 2017-03-08
US20130215295A1 (en) 2013-08-22
KR20130094633A (ko) 2013-08-26
KR101905621B1 (ko) 2018-10-11
CN103260037A (zh) 2013-08-21
EP2629503B1 (de) 2019-04-24
CN103260037B (zh) 2017-03-01
US9516234B2 (en) 2016-12-06

Similar Documents

Publication Publication Date Title
EP2629503B1 (de) Vorrichtung und Verfahren zur Übertragung eines Kamerarahmenbildes
AU2013200730B2 (en) Data processing apparatus and method using a camera
US9167164B2 (en) Metadata associated with frames in a moving image
KR101893406B1 (ko) 카메라의 이미지처리 장치 및 방법
EP2720451B1 (de) Vorrichtung und Verfahren zur Bildverarbeitung in einer Kameravorrichtung und tragbares Endgerät
EP2654288B1 (de) Bildverarbeitungsvorrichtung und Verfahren einer Kamera
CN103260026B (zh) 在摄像机设备中拍摄移动画面的装置和方法
JPWO2013132828A1 (ja) 通信システムおよび中継装置
KR101894531B1 (ko) 카메라의 연속 촬영 장치 및 방법.
KR101931631B1 (ko) 카메라 장치의 이미지 부호화장치 및 방법
KR101824506B1 (ko) 카메라의 멀티프레임 이미지 촬영 촬영 장치 및 방법
KR102058860B1 (ko) 카메라의 이미지 처리 장치 및 방법

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 1/21 20060101AFI20170201BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170615

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180202

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20181115

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1125509

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190515

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013054186

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190824

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190724

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190725

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190724

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1125509

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190824

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013054186

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

26N No opposition filed

Effective date: 20200127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200214

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200214

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190424

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20260121

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20260122

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20260120

Year of fee payment: 14