WO2023036007A1 - 一种获取图像的方法及电子设备 - Google Patents

一种获取图像的方法及电子设备 Download PDF

Info

Publication number
WO2023036007A1
WO2023036007A1 PCT/CN2022/115551 CN2022115551W WO2023036007A1 WO 2023036007 A1 WO2023036007 A1 WO 2023036007A1 CN 2022115551 W CN2022115551 W CN 2022115551W WO 2023036007 A1 WO2023036007 A1 WO 2023036007A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
electronic device
processing
video
video file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/115551
Other languages
English (en)
French (fr)
Inventor
王宇
朱聪超
肖斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honor Device Co Ltd
Original Assignee
Honor Device Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honor Device Co Ltd filed Critical Honor Device Co Ltd
Priority to EP22866473.6A priority Critical patent/EP4284009B1/en
Priority to US18/549,046 priority patent/US20240171701A1/en
Publication of WO2023036007A1 publication Critical patent/WO2023036007A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/21—Intermediate information storage
    • H04N1/2104—Intermediate information storage for one or a few pictures
    • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
    • H04N1/212—Motion video recording combined with still video recording
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • H04N5/772—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera the recording apparatus and the television camera being placed in the same enclosure
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/91—Television signal processing therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/21—Intermediate information storage
    • H04N1/2104—Intermediate information storage for one or a few pictures
    • H04N1/2112—Intermediate information storage for one or a few pictures using still video cameras
    • H04N1/2137—Intermediate information storage for one or a few pictures using still video cameras with temporary storage before final recording, e.g. in a frame buffer
    • H04N1/2141—Intermediate information storage for one or a few pictures using still video cameras with temporary storage before final recording, e.g. in a frame buffer in a multi-frame buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2101/00—Still video cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
    • H04N2201/0077—Types of the still picture apparatus
    • H04N2201/0084—Digital still camera

Definitions

  • the present application relates to the technical field of photographing, and in particular to a method for acquiring images and electronic equipment.
  • Existing mobile phones generally have photographing and video recording functions, and more and more people use mobile phones to take pictures and videos to record every bit of life.
  • video ie, video recording
  • some wonderful pictures may be collected.
  • the user may hope that the mobile phone can intercept the above-mentioned wonderful picture, and save it as a photo for display to the user. Therefore, there is an urgent need for a solution that can select beautiful images in video recordings and present them to users.
  • the present application provides a method for acquiring images and an electronic device, which can automatically select beautiful images in a video and present them to users.
  • the present application provides a method for acquiring an image, and the method can be applied to an electronic device.
  • the electronic device may receive a user's first operation, and the first operation is used to trigger the start of video recording.
  • the electronic device may display a first interface.
  • the first interface is a viewing interface where the electronic device is recording a video, and the first interface includes a preview image obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation.
  • the electronic device may select the second image from the first video stream.
  • the second image includes an image of the target object, and the first video stream is obtained by an image signal processor (image signal processor, ISP) of the electronic device performing first processing on multiple frames of the first image, and the first processing includes downsampling.
  • the electronic device may cache the first image collected by the camera in the first cache queue.
  • the first buffer queue can buffer n frames of first images collected by the camera, where n ⁇ 1, and n is an integer.
  • the electronic device may take m frames of the first image corresponding to the second image among the n frames of the first image as input, and run a preset original RAW domain image processing algorithm to obtain the third image.
  • the preset RAW domain image processing algorithm has the function of improving image quality. m ⁇ 1, m is an integer.
  • the electronic device may use the ISP to perform second processing on the third image to obtain the fourth image, where the second processing does not include downsampling.
  • the electronic device can display the exciting image obtained by encoding the fourth image after the video recording ends.
  • the image output by the exposure and output of the image sensor (Sensor) of the electronic device can be processed by the ISP to obtain a video stream (ie, the first data stream).
  • the electronic device may select a candidate frame (that is, the second image, that is, a candidate highlight frame) from the video stream processed by the ISP.
  • the electronic device may cache the sensor exposure output image in a first buffer queue (Buffer).
  • the first buffer queue can buffer multiple frames of images. In this way, even if there is a delay from "the Sensor outputs the image corresponding to the candidate frame (that is, the second image)" to "selecting the candidate frame", the image corresponding to the candidate frame can be cached in the first buffer queue. In this way, the electronic device can obtain the image corresponding to the candidate frame from the first buffer queue.
  • the electronic device can also use a preset RAW domain image processing algorithm to process the first image corresponding to the second image, which helps to improve the quality of the wonderful image.
  • the image LR in the video stream corresponding to the preview image and the recording file is down-sampled.
  • the ISP processes the third image to obtain the fourth image to obtain the image HR that has not been down-sampled; therefore, the beautiful image obtained by encoding the image HR is a high-resolution image.
  • the electronic device can automatically select the wonderful images in the video and present them to the user, and the image quality (such as resolution and picture quality) of the above wonderful images can be improved.
  • the above-mentioned electronic device displays the wonderful image obtained by encoding the fourth image, including: after the video recording of the electronic device ends, the video file is displayed in response to the user's operation.
  • the details page, the details page of the video file includes a first preset control; the electronic device responds to the user's click operation on the first preset control, and displays the wonderful images captured by the electronic device during the process of recording the video file.
  • the mobile phone can provide the user with an entrance to trigger the electronic device to display the wonderful images captured during the process of recording the video file by providing an explicit control (ie, the first preset control) on the details page of the video file.
  • the electronic device displays the wonderful image obtained by encoding the fourth image, including: after the video recording of the electronic device ends, the video file is displayed in response to the user's operation. Details page: the electronic device responds to the preset gesture input by the user on the details page of the video file, and displays the wonderful images captured by the electronic device during the process of recording the video file.
  • the mobile phone can provide users with an entrance to trigger the electronic device to display the wonderful images captured during the recording process of the video file through the details page of the video file without explicit controls.
  • n In another possible design of the first aspect, n ⁇ 2.
  • multiple frames of the first image may be cached in the first cache queue.
  • the Sensor output frame can be cached in In Buffer. Therefore, when the electronic device selects a candidate frame (that is, the second image), the Bayer image output by the Sensor may also be cached in the first buffer queue. Moreover, the image content of the frame output by the Sensor will not change much in a short period of time. In this way, m frames of the first image corresponding to the second image can be selected from the Buffer by the electronic device according to the time information of the image cached in the Buffer.
  • the electronic device takes k frames of the first image corresponding to the second image among the n frames of the first image as input, and runs a preset RAW domain image processing algorithm to obtain the third image, It may include: selecting a fifth image corresponding to the second image from the n frames of the first image; using m frames of the first image in the n frames of the first image as input, and running a preset RAW domain image processing algorithm to obtain the third image; Wherein, the m frames of the first image include the fifth image; the images in the m frames of the first image except the fifth image are used to enhance the image quality of the fifth image.
  • the time when the fifth image is output by the image sensor of the camera is the closest to the time when the second image is output by the image sensor of the camera.
  • a frame of image closest to the image content of the second image can be selected from the first cache queue.
  • the fifth image has the highest resolution. In this way, a frame of image with the highest image quality can be selected from the first buffer queue.
  • the above-mentioned target object includes a human face
  • the second image is an image in which the expression of the target object satisfies a preset condition, and the expression of the target object satisfies the preset condition including a smile on a human face.
  • the electronic device can capture wonderful images of smiling faces in the video.
  • the target object includes a human body
  • the second image is an image in which an action of the target object is a preset action.
  • the preset action may be a human body action when the human body jumps to the highest point, a wonderful action during a dance performance or a competitive sports performance, a difficult action during a human body exercise process, and the like.
  • the second image may be a frame of image in the first video stream when the human body poses a preset action.
  • the electronic device can capture wonderful images when the human body performs preset actions during the video recording process.
  • the above-mentioned target object is pre-configured in the electronic device when the electronic device leaves the factory. And/or, the target object is set by the user of the electronic device on the setting interface of the electronic device.
  • the electronic device provides the user with a function of setting a target object on a viewfinder interface where the electronic device has not yet started recording a video.
  • the electronic device may display the second interface.
  • the second interface is a viewfinder interface where the electronic device has not yet started to record video, and the preview image included in the second interface is obtained by the image collected by the camera of the electronic device before the electronic device receives the first operation; the second interface also includes the start video recording interface The switch of the button and the preset function; the first operation is the click operation of the start recording button.
  • the electronic device displays the setting interface of the target object, and the setting interface of the target object includes a plurality of options of the shooting object to be selected.
  • the electronic device determines the subject selected by the user as the target object.
  • the electronic device provides the user with a function of setting a target object on a viewing interface (such as the above-mentioned first interface) where the electronic device has started to record a video.
  • the first interface also includes: a switch of a preset function.
  • the electronic device may display a setting interface of the target object in response to the second operation of the switch of the preset function in the first interface by the user, and the setting interface of the target object includes Multiple options of the subject to be selected; the electronic device determines the subject selected by the user as the target object in response to the user's selection operation on at least one of the options of the subject to be selected.
  • the electronic device may periodically select the second image from the first video stream.
  • the electronic device may use the multiple frames of the first image as input, and run a preset RAW domain image processing algorithm. It should be understood that the images other than the fifth image in the m frames of the first image can enhance the image quality of the fifth image, which is conducive to obtaining information such as noise and texture, and can further improve the image quality of the fifth image .
  • the above-mentioned first processing includes image processing in the RAW domain and image processing in the RGB domain
  • the second processing includes image processing in the RAW domain, image processing in the RGB domain, and image processing in the YUV domain. Image Processing.
  • the electronic device may The second processing is performed on the third image by using the ISP to obtain the fourth image.
  • the first image whose time information is the same as the time information of the second image may not be cached in the first cache queue.
  • the time information of the first image cached in the first cache queue may be quite different from the time information of the second image.
  • the image content of the first image cached in the first cache queue may be quite different from the image content of the second image.
  • the first cache queue includes the first image whose time information and the time difference indicated by the time information of the second image are within the preset time length, it means that the first cache queue and the Sensor output the second image can be found.
  • the first image that is closer in time.
  • the electronic device uses the ISP to perform the second processing on the third image to obtain the fourth image, and can accurately capture the beautiful image corresponding to the second image.
  • the electronic device if the first cache queue does not include the first image whose time information differs from the time indicated by the time information of the second image within a preset duration, the electronic device Then the method of the second aspect can be implemented. The detailed introduction and effect analysis of the method described in the second aspect will not be repeated in this application.
  • the electronic device can not only generate wonderful images online, but can also generate wonderful images offline.
  • the third processing described above may not include downsampling.
  • the method of the present application may further include: the electronic device caches the sixth image in the second cache queue; the electronic device generates a video file in response to the operation of ending the video; wherein, the video file includes multiple frames of the first image generated
  • the electronic device determines the alternative image from the video file in response to the user's selection operation of a frame image in the video file; Select a reference REF image corresponding to the candidate image, and use the REF image to enhance the quality of the candidate image to obtain a ninth image.
  • the electronic device uses the ISP to perform fourth processing on the ninth image to obtain the tenth image, the fourth processing includes image processing in the YUV domain, and the fourth processing does not include down-sampling; the electronic device displays the wonderful image obtained by encoding the tenth image.
  • the fourth processing includes image processing in the YUV domain, and the fourth processing does not include down-sampling; the electronic device displays the wonderful image obtained by encoding the tenth image.
  • the present application provides a method for acquiring an image, and the method can be applied to an electronic device.
  • the electronic device may receive a user's first operation, and the first operation is used to trigger the start of video recording.
  • the electronic device may display a first interface.
  • the first interface is a viewing interface where the electronic device is recording a video, and the first interface includes a preview image obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation.
  • the electronic device may select the second image from the first video stream.
  • the second image includes an image of the target object, and the first video stream is obtained by an image signal processor (image signal processor, ISP) of the electronic device performing first processing on multiple frames of the first image, and the first processing includes downsampling.
  • the electronic device may cache the first image collected by the camera in the first cache queue.
  • the first buffer queue can buffer n frames of first images collected by the camera, where n ⁇ 1, and n is an integer.
  • the electronic device may take m frames of the first image corresponding to the second image among the n frames of the first image as input, and run a preset original RAW domain image processing algorithm to obtain the third image.
  • the preset RAW domain image processing algorithm has the function of improving image quality. m ⁇ 1, m is an integer.
  • the electronic device uses the ISP to perform third processing on the third image to obtain the sixth image; wherein, the third processing includes image processing in the RAW domain and image processing in the RGB domain.
  • the electronic device uses the sixth image to enhance the image quality of the second image to obtain a seventh image.
  • the electronic device uses the ISP to perform fourth processing on the seventh image to obtain the eighth image, the fourth processing includes image processing in the YUV domain, and the fourth processing does not include downsampling. After the video recording is finished, the electronic device displays the exciting image obtained by encoding the eighth image.
  • the foregoing first processing includes image processing in the RAW domain and image processing in the RGB domain.
  • the enhancement of the image quality of the second image by the electronic device using the sixth image includes: image fusion and/or image super-resolution.
  • image super-resolution includes reference frame-based image super-resolution REF-SR.
  • the present application provides a method for acquiring an image, and the method can be applied to an electronic device.
  • the electronic device may receive a user's first operation, and the first operation is used to trigger the start of video recording.
  • the electronic device may display a first interface.
  • the first interface is a viewing interface where the electronic device is recording a video, and the first interface includes a preview image obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation.
  • the electronic device may select the second image from the first video stream.
  • the second image includes an image of the target object, and the first video stream is obtained by an image signal processor (image signal processor, ISP) of the electronic device performing first processing on multiple frames of the first image, and the first processing includes downsampling.
  • the electronic device may cache the first image collected by the camera in the first cache queue.
  • the first buffer queue can buffer n frames of first images collected by the camera, where n ⁇ 1, and n is an integer.
  • the electronic device may take m frames of the first image corresponding to the second image among the n frames of the first image as input, and run a preset original RAW domain image processing algorithm to obtain the third image.
  • the preset RAW domain image processing algorithm has the function of improving image quality. m ⁇ 1, m is an integer.
  • the electronic device uses the ISP to perform third processing on the third image to obtain the sixth image, and cache the sixth image in the second buffer queue.
  • the third processing includes image processing in the RAW domain and image processing in the RGB domain.
  • the third processing does not including downsampling.
  • the electronic device In response to the operation of ending the video recording, the electronic device generates a video file; wherein, the video file includes an image generated by multiple frames of the first image.
  • the electronic device determines the candidate image from the video file in response to the user's selection operation on a frame of image in the video file.
  • the electronic device selects a reference REF image corresponding to the candidate image from the multi-frame sixth images cached in the second buffer queue, and uses the REF image to enhance the quality of the candidate image to obtain a ninth image.
  • the electronic device uses the ISP to perform fourth processing on the ninth image to obtain the tenth image, the fourth processing includes image processing in the YUV domain, and the fourth processing does not include downsampling.
  • the electronic device displays the exciting image obtained by encoding the tenth image.
  • the third process does not include downsampling. Therefore, the sixth image buffered in the second buffer queue is a high-resolution HR image. Moreover, the sixth image is an image with richer image details and higher image quality processed by a preset RAW domain image processing algorithm.
  • the above-mentioned sixth image is generated based on the second image.
  • the second image is an image including the target object in the video file. It can be seen that the sixth image also includes the image of the target object. In this way, after the recording of the electronic device is finished, it is more likely that the user wants to select a frame of image corresponding to the sixth image from the recording file as a wonderful image.
  • the electronic device can cache multiple frames of the sixth image in the second cache queue, so that the second cache queue can be used when there is a user demand to select a frame of image from the video file as a wonderful image.
  • the sixth image in the image is enhanced (including image super-resolution) to obtain a beautiful image with higher image quality (including higher resolution).
  • the details page of the video file includes a second preset control.
  • the operation of selecting a frame of image in the video file is the operation of clicking on the second preset control.
  • the above-mentioned electronic device responds to the user's selection operation of a frame image in the video file, and determines the candidate image from the video file, including: the electronic device responds to the user's click operation on the second preset control in the details page of the video file, from An image corresponding to the sixth image of each frame buffered in the second buffer queue is selected from the video file as a candidate image.
  • the electronic device can provide users with the function of triggering the electronic device to generate and display wonderful images in the video file in the form of explicit controls (such as the second preset control) on the details page of the video file.
  • the playback interface of the video file includes a third preset control.
  • the operation of selecting a frame of image in the video file is the operation of clicking on the third preset control.
  • the above-mentioned electronic device determines the candidate image from the video file in response to the user's selection operation of a frame of image in the video file, including: during the process of playing the video file, the electronic device responds to the third In the click operation of the preset control, a frame of image played by the electronic device when the user clicks on the third preset control is used as a candidate image.
  • the electronic device can provide users with the function of triggering the electronic device to generate and display wonderful images in the video file in the form of explicit controls (such as the second preset control) in the playback interface of the video file.
  • the above-mentioned electronic device determines an alternative image from the video file in response to the user's selection operation on a frame of image in the video file, including: the process of the electronic device playing the video file Among them, in response to the operation of pausing the playback and the operation of enlarging the image, a frame of image corresponding to the operation of enlarging the image is used as the candidate image.
  • the operation of zooming in on the image is used to trigger the electronic device to zoom in on a frame of image in the video file displayed by the electronic device when the electronic device pauses playing the video file.
  • the electronic device when the electronic device plays a video file, when the electronic device plays an image that the user is interested in, the user can click the pause button to control the mobile phone to pause playing the video file, and then the user can control the electronic device to zoom in on the current frame image.
  • the electronic device may use a frame of image enlarged by the user as a candidate image for generating a beautiful image.
  • the above-mentioned electronic device determines an alternative image from the video file in response to the user's selection operation on a frame of image in the video file, including: the process of the electronic device playing the video file , receive the user’s adjustment operation on the progress bar in the playback interface of the video file, and display the image in the video file according to the change of the progress bar; after the electronic device detects that the user’s finger leaves the progress bar, the progress bar corresponds to when the user’s finger leaves the progress bar.
  • a frame of image in the video file is used as an alternative image.
  • the electronic device when the electronic device plays a video file, when the electronic device plays an image that the user is interested in, the user can control the electronic device to display a frame of the image in the video file that the user is interested in through the progress bar.
  • the electronic device detects that the user's finger leaves the progress bar, the electronic device can take the user's finger off the progress bar, and the progress bar corresponds to a frame of image in the video file as a candidate image for generating a beautiful image.
  • the above-mentioned electronic device determines an alternative image from the video file in response to the user's selection operation on a frame of image in the video file, including: the process of the electronic device playing the video file
  • a frame of image is selected from a segment as a candidate image.
  • the electronic device selects a frame of image from the segment as a candidate image for generating a wonderful image, and can obtain a wonderful image that meets user requirements.
  • the above-mentioned electronic device determines an alternative image from the video file in response to the user's selection operation on a frame of image in the video file, including: the process of the electronic device playing the video file
  • the adjustment operation of the progress bar in the playback interface of the video file by the user is received, and the image in the video file is displayed according to the change of the progress bar;
  • the electronic device detects that the user's finger stays on the progress bar, and the progress bar maintains the same position, and records the The duration of the progress bar maintaining the same position, when the duration of the progress bar maintaining the same position is greater than the preset time threshold, the electronic device uses a frame of image in the video file corresponding to the progress bar as an alternative image; or, the electronic device The device detects that the user's finger stays on the progress bar, and the playback interface of the video file keeps displaying the same frame of image.
  • a corresponding frame of image in the video file is used as a
  • the selection of an image frame from a segment as the candidate image includes: the electronic device selecting a frame image with the highest definition in a segment as the candidate image.
  • the present application provides an electronic device, which includes a touch screen, a memory, a display screen, one or more cameras, and one or more processors.
  • the memory, display screen, camera and processor are coupled.
  • the camera is used for collecting images
  • the display screen is used for displaying images collected by the camera or images generated by the processor.
  • Computer program codes are stored in the memory, and the computer program codes include computer instructions.
  • the electronic The device executes the method described in the first aspect, the second aspect, or the third aspect and any possible design manner thereof.
  • the present application provides an electronic device, which includes a touch screen, a memory, a display screen, one or more cameras, and one or more processors.
  • the memory, display screen, camera and processor are coupled.
  • the camera is used to collect images
  • the display screen is used to display images collected by the camera or images generated by the processor
  • computer program codes are stored in the memory, and the computer program codes include computer instructions.
  • the electronic device When the computer instructions are executed by the processor, the electronic device is made to perform the following steps: receiving a first operation from the user; wherein, the first operation is used to trigger the start of video recording; in response to the first operation, displaying a first interface; wherein, the first The interface is a viewing interface where the electronic device is recording a video, the first interface includes a preview image, and the preview image is obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation; The second image; wherein, the second image includes the image of the target object; the first video stream is obtained by the ISP of the electronic device performing the first processing on the multi-frame first image, and the first processing includes downsampling; cached in the first buffer queue The first image collected by the camera; wherein, the first cache queue caches n frames of the first image collected by the camera, n ⁇ 1, and n is an integer; the m frames of the first image corresponding to the second image in the n frames of the first image are used as Input, run the
  • the electronic device when the computer instruction is executed by the processor, the electronic device is made to perform the following steps: after the video recording ends, display the details page of the video file in response to the user's operation, and the details page of the video file A first preset control is included; in response to the user's click operation on the first preset control, the beautiful images captured during the process of recording video files by the electronic device are displayed.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is made to perform the following steps: after the video recording ends, display the details page of the video file in response to the user's operation;
  • the default gesture input on the details page of the video file shows the wonderful images captured by the electronic device during the recording of the video file.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following steps: select a fifth image corresponding to the second image from n frames of first images; The m frames of the first image in the frame of the first image are used as input, and the third image is obtained by running a preset RAW domain image processing algorithm; wherein, the m frames of the first image include the fifth image; the m frames of the first image except the fifth image The other images are used to enhance the image quality of the fifth image.
  • the fifth image is output by the image sensor of the camera at the closest time to the second image output by the image sensor of the camera; or, in the n frames of first images, the resolution of the fifth image is Highest.
  • the target object includes a human face
  • the second image is an image in which the expression of the target object satisfies a preset condition, and the expression of the target object satisfies the preset condition including a smile on a human face
  • the target object includes a human body
  • the second image is an image in which the action of the target object is a preset action.
  • the target object is pre-configured in the electronic device when the electronic device leaves the factory; and/or, the target object is set by the receiving user of the electronic device on the setting interface of the electronic device.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following step: before the electronic device selects the second image from the first video stream, display the second interface; wherein, the second interface is a viewfinder interface where the electronic device has not yet started to record video, and the preview image included in the second interface is obtained by the image collected by the camera of the electronic device before the electronic device receives the first operation; the second interface also Including a button to start video recording and a switch of a preset function; the first operation is to click the button to start video recording; in response to the second operation of the switch of the preset function in the second interface by the user, the setting interface of the target object is displayed, and the target object
  • the setting interface includes a plurality of options of the subject to be selected; in response to the user's selection operation on at least one option of the subject among the options of the subject to be selected, the subject selected by the user is determined as the target object.
  • the above-mentioned first interface further includes: a switch of a preset function.
  • the electronic device is made to perform the following steps: before the electronic device selects the second image from the first video stream, in response to the user's second operation on the switch of the preset function in the first interface, Displaying the setting interface of the target object, the setting interface of the target object includes a plurality of options of the shooting object to be selected; in response to the user's selection operation of at least one option of the shooting object among the options of the plurality of shooting objects to be selected, the user The selected subject is determined as the target subject.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following step: periodically select the second image from the first video stream.
  • the first processing includes image processing in the RAW domain and image processing in the RGB domain
  • the second processing includes image processing in the RAW domain, image processing in the RGB domain, and image processing in the YUV domain deal with.
  • the electronic device when the computer instructions are executed by the processor, the electronic device is caused to perform the following steps: if the first cache queue includes the time information and the time indicated by the time information of the second image For the first image whose difference value is within a preset time length, the ISP is used to perform the second processing on the third image to obtain the fourth image.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following steps: if the first cache queue does not include the time information and the time information indicated by the second image For the first image whose time difference is within the preset duration, the third image is processed by the ISP to obtain the sixth image; wherein, the third processing includes image processing in the RAW domain and image processing in the RGB domain; using the sixth The image quality is enhanced on the second image to obtain the seventh image; the fourth processing is performed on the seventh image by ISP to obtain the eighth image, the fourth processing includes image processing in the YUV domain, and the fourth processing does not include down-sampling; After the end, the wonderful image obtained by encoding the eighth image is displayed.
  • the foregoing third processing does not include downsampling.
  • the electronic device is made to perform the following steps: buffering the sixth image in the second buffer queue; in response to the operation of ending the video recording, generating a video file; wherein, the video file includes multiple frames of the first image Generated image; in the process of playing the video file, in response to the user's selection operation of a frame image in the video file, determine the alternative image from the video file; select and prepare from the multi-frame sixth image cached in the second buffer queue Select the reference REF image corresponding to the image, and use the REF image to enhance the image quality of the candidate image to obtain the ninth image; use the ISP to perform the fourth processing on the ninth image to obtain the tenth image, and the fourth processing includes image processing in the YUV domain , the fourth processing does not include down-sampling; the exciting image obtained by encoding the tenth image is displayed.
  • the present application provides an electronic device, which includes a touch screen, a memory, a display screen, one or more cameras, and one or more processors.
  • the memory, display screen, camera and processor are coupled.
  • the camera is used to collect images
  • the display screen is used to display images collected by the camera or images generated by the processor
  • computer program codes are stored in the memory, and the computer program codes include computer instructions.
  • the electronic device When the computer instructions are executed by the processor, the electronic device is made to perform the following steps: receiving a first operation from the user; wherein, the first operation is used to trigger the start of video recording; in response to the first operation, displaying a first interface; wherein, the first The interface is a viewing interface where the electronic device is recording a video, the first interface includes a preview image, and the preview image is obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation; The second image; wherein, the second image includes the image of the target object; the first video stream is obtained by the ISP of the electronic device performing the first processing on the multi-frame first image, and the first processing includes downsampling; cached in the first buffer queue The first image collected by the camera; wherein, the first cache queue caches n frames of the first image collected by the camera, n ⁇ 1, and n is an integer; the m frames of the first image corresponding to the second image in the n frames of the first image are used as Input, run the
  • the first processing includes image processing in the RAW domain and image processing in the RGB domain.
  • the enhancement of the image quality of the second image by the electronic device using the sixth image includes: image fusion and/or image super-resolution.
  • image super-resolution includes reference frame-based image super-resolution REF-SR.
  • the present application provides an electronic device, which includes a touch screen, a memory, a display screen, one or more cameras, and one or more processors.
  • the memory, display screen, camera and processor are coupled.
  • the camera is used to collect images
  • the display screen is used to display images collected by the camera or images generated by the processor
  • computer program codes are stored in the memory, and the computer program codes include computer instructions.
  • the electronic device When the computer instructions are executed by the processor, the electronic device is made to perform the following steps: receiving a first operation from the user; wherein, the first operation is used to trigger the start of video recording; in response to the first operation, displaying a first interface; wherein, the first The interface is a viewing interface where the electronic device is recording a video, the first interface includes a preview image, and the preview image is obtained from the first image collected by the camera of the electronic device after the electronic device receives the first operation; The second image; wherein, the second image includes the image of the target object; the first video stream is obtained by the ISP of the electronic device performing the first processing on the multi-frame first image, and the first processing includes downsampling; cached in the first buffer queue The first image collected by the camera; wherein, the first cache queue caches n frames of the first image collected by the camera, n ⁇ 1, and n is an integer; the m frames of the first image corresponding to the second image in the n frames of the first image are used as Input, run the
  • the details page of the above-mentioned video file includes a second preset control; the operation of selecting a frame of image in the video file is a click operation on the second preset control.
  • the electronic device When the computer instructions are executed by the processor, the electronic device is made to perform the following steps: in response to the user’s click operation on the second preset control in the details page of the video file, select from the video file each file that is cached in the second cache queue The image corresponding to the sixth image of the frame is used as a candidate image.
  • the playback interface of the video file includes a third preset control.
  • the operation of selecting a frame of image in the video file is the operation of clicking on the third preset control.
  • the electronic device When the computer instruction is executed by the processor, the electronic device is made to perform the following steps: in the process of playing the video file, in response to the user's click operation on the third preset control in the playback interface of the video file, the user clicks the third preset A frame of image played by the electronic device during the control is used as an alternative image.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following steps: in the process of playing the video file, in response to the operation of pausing the playback and the operation of enlarging the image, A frame of image corresponding to the operation of enlarging the image is used as a candidate image.
  • the operation of zooming in on the image is used to trigger the electronic device to zoom in on a frame of image in the video file displayed by the electronic device when the electronic device pauses playing the video file.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is made to perform the following steps: during the process of playing the video file, receiving the user's request for the progress bar in the playback interface of the video file Adjust the operation to display the image in the video file according to the change of the progress bar; after detecting that the user's finger leaves the progress bar, use a frame of image in the video file corresponding to the progress bar when the user's finger leaves the progress bar as an alternative image.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is made to perform the following steps: during the process of playing the video file, receiving the user's request for the progress bar in the playback interface of the video file Adjust the operation to display the image in the video file according to the change of the progress bar; detect that the user's finger stays on the progress bar, the progress bar remains at the same position, record the duration of the progress bar maintaining the same position, when the progress bar remains at the same position When the duration is greater than the preset time threshold, a frame of image in the video file corresponding to the progress bar is used as an alternative image; or, when the user’s finger is detected to stay on the progress bar, the playback interface of the video file keeps displaying the same frame of image, and the video is recorded The playback interface of the file keeps displaying the duration of the same frame of image, and when the duration of the playback interface of the video file keeping displaying the same frame of image is greater than the preset time threshold, a corresponding frame of image in the video file is
  • the electronic device when the computer instruction is executed by the processor, the electronic device is made to perform the following steps: in the process of playing the video file, in response to the user's operation of repeatedly viewing a segment in the video file , select a frame image from a clip as a candidate image.
  • the electronic device when the computer instruction is executed by the processor, the electronic device is caused to perform the following step: selecting a frame of image with the highest definition in a segment as the candidate image.
  • the present application provides a computer-readable storage medium
  • the computer-readable storage medium includes computer instructions, and when the computer instructions are run on the electronic device, the electronic device is made to perform the first aspect or the second aspect and any of them.
  • One possible design approach is described.
  • the present application provides a computer program product.
  • the computer program product runs on a computer, the computer executes the method described in the first aspect or the second aspect and any possible design manner.
  • the computer may be the electronic device described above.
  • Fig. 1 is a flow chart of image processing in a video recording process
  • Fig. 2 is another kind of image processing flowchart in video recording process
  • Fig. 3 is a functional block diagram of a method for acquiring an image provided by an embodiment of the present application
  • FIG. 4 is a schematic diagram of a hardware structure of a mobile phone provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a software architecture of a mobile phone provided by an embodiment of the present application.
  • FIG. 6 is a flow chart of a method for acquiring an image provided in an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a mobile phone video display interface provided by an embodiment of the present application.
  • FIG. 8A is a schematic block diagram of another method for acquiring images provided by the embodiment of the present application.
  • FIG. 8B is a schematic block diagram of another method for acquiring images provided by the embodiment of the present application.
  • FIG. 9 is a schematic diagram of a first cache queue Buffer provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 11 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 12 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 13 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 14 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 15 is a flow chart of another method for acquiring images provided in the embodiment of the present application.
  • FIG. 16 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 17 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 18 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 19 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 20 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 21 is a flow chart of another method for acquiring images provided in the embodiment of the present application.
  • Fig. 22 is a functional block diagram of another method for acquiring images provided by the embodiment of the present application.
  • Fig. 23 is a schematic block diagram of another method for acquiring images provided by the embodiment of the present application.
  • Fig. 24 is a functional block diagram of another method for acquiring images provided by the embodiment of the present application.
  • FIG. 25 is a flow chart of another method for acquiring images provided in the embodiment of the present application.
  • Fig. 26 is a functional block diagram of another method for acquiring images provided by the embodiment of the present application.
  • FIG. 27 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 28 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 29 is a schematic diagram of another mobile phone video display interface provided by the embodiment of the present application.
  • FIG. 30 is a schematic structural diagram of a chip system provided by an embodiment of the present application.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • the image sensor (Sensor) of the electronic equipment is controlled by the exposure, and can continuously output images.
  • Each frame of image is processed by the image signal processor (image signal processor, ISP) or image signal processing algorithm of the electronic device, and then encoded by the encoder (ENCODER), and then the video file can be obtained.
  • the original image output by the image sensor is usually a Bayer (Bayer) format image, and some image sensors can output RGGB, RGBW, CMYK, RYYB, CMY and other format images.
  • the Bayer format image output by the image sensor of the mobile phone is taken as an example for description. It should be noted that image sensors that output images in formats such as RGGB, RGBW, CMYK, RYYB, and CMY, and other electronic devices equipped with such image sensors are also applicable to the technical solutions provided in the embodiments of the present application.
  • RGGB is (red green green blue)
  • RGBW is (red green blue white)
  • CMYK is (cyan magenta yellow black)
  • RYYB is (red yellow yellow blue)
  • CMY is (cyan magenta yellow).
  • FIG. 1 or FIG. 2 shows the processing flow of the preview image and the video file during the video recording process of the mobile phone.
  • the preview image refers to the image finally presented to the user on the display screen during the video recording process of the mobile phone
  • the video file refers to the video stream which is finally saved in the mobile phone in the format of a video file for viewing by the user after the video recording ends.
  • the image processing by the mobile phone's ISP can be divided into three image format domains: image processing in the RAW domain, image processing in the RGB domain, and image processing in the YUV domain.
  • Image processing in the RAW domain can include: black level correction (BLC) correction, linear correction (Linearizaton), lens shading correction (lens shading correction, LSC), bad point repair (defectpixel correction, DPC), RAW noise reduction (Denoise), automatic white balance (automatic white balance, AWB), green channel balance (green imbalance, GIC), dechromatic aberration (CAC) and other processing.
  • BLC black level correction
  • LSC linear correction
  • DPC bad point repair
  • Denoise RAW noise reduction
  • automatic white balance automatic white balance
  • GIC green imbalance
  • CAC dechromatic aberration
  • Image processing in the RGB domain can include: demosaic (Demosiac), color correction CC, dynamic range compression (dynamic range control, DRC), Gamma correction, RGB2YUV (RGB format converted to YUV format).
  • Image processing in the YUV domain can include: UV downsampling, color enhancement CE, spatial domain noise reduction YUVNF, color management 3DLUT, sharpening Sharpness, scaling Scalar.
  • the division of "RAW domain", “RGB domain” and “YUV domain” in the ISP includes but is not limited to the above division methods.
  • demosaicing can also be included in the "RAW domain”.
  • the embodiment of the present application does not limit this.
  • the image after the image sensor (Sensor) outputs the image, the image can be processed by the ISP in the "RAW domain", "RGB domain” and “YUV domain”; "After image processing, it can be divided into two data streams.
  • One stream of data is processed using the processing algorithm 1 shown in Figure 1, and then the display module performs encoding or format conversion to obtain and display a preview image.
  • the other data stream is processed using the processing algorithm 2 shown in Figure 1, and then encoded by the encoder 1 to obtain a video file.
  • the image can be processed by the ISP in the "RAW domain” and "RGB domain”; after the "RGB domain” image processing, Can be divided into two data streams.
  • One data stream is processed by the processing algorithm 1 shown in Figure 2, and then the ISP performs image processing in the "YUV domain", and then the display module performs encoding or format conversion to obtain and display a preview image.
  • the other data stream is processed by the processing algorithm 2 shown in Figure 2, and then the image processing in the "YUV domain” is performed by the ISP, and then encoded by the encoder 1 to obtain a video file.
  • the image processing of the processing algorithm 1 and the processing algorithm 2 can be performed in the RGB domain, and can also be performed in the YUV domain.
  • the ISP can use the processing algorithm 1 to process the image before converting the image from the RGB format to the YUV format. After that, the ISP can convert the image processed by the processing algorithm 1 into the YUV format, and then perform image processing in the "YUV domain" on the image.
  • the ISP can first convert the image from the RGB format to the YUV format, and then use the processing algorithm 1 to process the image in the YUV format. Afterwards, the ISP can perform image processing in the "YUV domain" on the image processed by the processing algorithm 1.
  • processing algorithm 1 may also be called a post-processing algorithm for preview images
  • processing algorithm 2 may also be called a post-processing algorithm for video files.
  • Processing algorithm 1 and processing algorithm 2 may include anti-shake processing, denoising processing, blur processing, color and brightness adjustment and other processing functions.
  • the image output by the Sensor is an image in Bayer (Bayer) format (Bayer image for short).
  • the ISP "RAW domain” input image is a Bayer format image (ie Bayer image)
  • the ISP "RAW domain” output image is an RGB format image (referred to as RGB image).
  • the input image of the "RGB domain” of the ISP is an image in RGB format (that is, an RGB image), and the output image of the "RGB domain” of the ISP is an image in the YUV format (referred to as a YUV image).
  • the input image of the "YUV domain” of the ISP is an image in YUV format (that is, a YUV image), and the output image of the "YUV domain” of the ISP can be encoded (ENCODE) to obtain a preview image or video file.
  • Bayer, RGB and YUV are three expression formats of images.
  • RGB images and YUV images reference may be made to related content in conventional technologies, and details will not be repeated here.
  • the Sensor outputs images the images processed by the ISP and the encoder (ie ENCODER, such as the encoder and encoder 1 of the display module) can be used to record video; therefore, the Sensor output image, the ISP and the encoder ( ENCODER)
  • the data stream in the whole process of image processing (such as the data stream of the video file and the data stream of the preview image) is called the video stream.
  • the way the mobile phone processes images during video recording to obtain preview images and video files includes but is not limited to the ways shown in FIG. 1 and FIG. 2 , and other processing ways will not be described in this embodiment of the present application.
  • the processing method shown in FIG. 1 is taken as an example to introduce the method in the embodiment of the present application.
  • the mobile phone can generally use the hardware processing module of the ISP to process the video stream (such as the data stream of the video file and the preview image) using a relatively simple processing method. data stream); instead of using complex algorithms to improve image quality (such as denoising and brightening).
  • This kind of image processing effect can only meet the requirements of video; while taking pictures has higher requirements for image quality. Therefore, intercepting an image in a video stream cannot obtain an image satisfactory to the user.
  • the ISP down-samples the image output by the Sensor during the recording process.
  • the ISP can down-sample the image frames in the video stream before performing image processing in the YUV domain on the video stream (such as a preview stream or a video stream).
  • down sampling subsampled
  • down sampling down sampled
  • Downsampling the image can reduce the size of the image and reduce the resolution of the image.
  • the number of pixels of the image before downsampling may be 4000*3000, and the number of pixels of the image after downsampling may be 2000*1500.
  • the image output by the "RAW domain" of the ISP is a low-resolution image, recorded as (low resolution, LR).
  • the wonderful images that users hope to obtain during video recording are high-resolution images.
  • the embodiments of the present application provide a method for acquiring images and an electronic device, which can automatically select a beautiful image in a video and present it to the user, and can improve the image quality (such as resolution and picture quality) of the wonderful image.
  • the image output by the image sensor (Sensor) of the electronic device can be processed by the ISP to obtain a video stream (such as a data stream of a video file and a data stream of a preview image).
  • the frame selection module of the electronic device can select a backup frame from the video stream processed by the ISP (such as the "RAW domain” and "RGB domain” of the ISP, referred to as ISP RAW+RGB).
  • the selected frame is an alternative highlight frame (also referred to as a highlight image).
  • a highlight frame or a highlight frame reference may be made to the introduction in the following embodiments, and details are not repeated here.
  • the electronic device can cache the Bayer image (that is, image 1) output by the image sensor (Sensor) in a first buffer queue (Buffer).
  • the first buffer queue can buffer multiple frames of Bayer images. In this way, even if there is a delay from "the Sensor outputs the Bayer image corresponding to the candidate frame” to "the frame selection module selects the candidate frame", the Bayer image corresponding to the candidate frame can be cached in the first buffer queue. In this way, the mobile phone can obtain the Bayer image corresponding to the candidate frame (image 2 shown in FIG. 3 ) from the first cache queue.
  • the electronic device can also use a preset RAW domain image processing algorithm to process image 2 to obtain image 3 , which helps to improve the quality of wonderful images.
  • the image LR in the video stream corresponding to the preview image and the recording file is down-sampled.
  • the image HR obtained by the ISP processing the image 3 shown in FIG. 3 is not down-sampled; therefore, the wonderful image obtained by encoding the image HR by the encoder 2 is a high-resolution image.
  • the electronic device can show the wonderful image shown in FIG. 3 to the user after the video recording is finished.
  • the electronic device can automatically select the wonderful images in the video and present them to the user, and the image quality (such as resolution and picture quality) of the above wonderful images can be improved.
  • the preset RAW domain image processing algorithm is a deep learning network for image quality enhancement in the RAW domain.
  • the preset RAW domain image processing algorithm may also be called a preset image quality enhancement algorithm, a preset image quality enhancement algorithm model, or a preset RAW domain AI model.
  • the aforementioned preset RAW domain image processing algorithm can run on a graphics processing unit (graphics processing unit, GPU), a neural network processor (neural-network processing unit, NPU) of an electronic device, or other devices capable of running a neural network model. in the processor. Any one of the above-mentioned processors may load the preset RAW domain image processing algorithm from the memory before running the preset RAW domain image processing algorithm.
  • graphics processing unit graphics processing unit, GPU
  • NPU neural-network processing unit
  • the preset RAW domain image processing algorithm may be a software image processing algorithm.
  • the preset RAW domain image processing algorithm may be a software algorithm in a hardware abstraction layer (hardware abstraction layer, HAL) algorithm library of the mobile phone.
  • HAL hardware abstraction layer
  • the preset RAW domain image processing algorithm may be a hardware image processing algorithm.
  • the preset RAW domain image processing algorithm may be a hardware image processing algorithm implemented by calling the "RAW domain” image processing algorithm capability in the ISP.
  • the preset RAW domain image processing algorithm may be a hardware image processing algorithm implemented by calling the "RAW domain” and “RGB domain” image processing algorithm capabilities in the ISP.
  • the preset RAW domain image processing algorithm may be a hardware image processing algorithm implemented by calling the "RAW domain”, "RGB domain” and "YUV domain” image processing algorithm capabilities in the ISP.
  • the preset RAW domain image processing algorithm may also be referred to as a preset image processing algorithm.
  • it is called a preset RAW domain image processing algorithm because the input of the preset RAW domain image processing algorithm is a RAW domain image.
  • the output of the preset RAW domain image processing algorithm may be an image in the RAW domain or an image in the RGB domain.
  • the encoders, encoder 1 and encoder 2 in the display module shown in FIG. 1 or FIG. 2 may be three different encoders.
  • the mobile phone can use three different encoders to perform encoding or format conversion to obtain the above-mentioned preview image, video file and snapshot image.
  • the encoder, encoder 1 and encoder 2 in the above display module may be the same encoder.
  • An encoder can include multiple encoding units.
  • the mobile phone can use three different coding units in one coder to perform coding or format conversion respectively to obtain the above preview image, video file and snapped image.
  • the encoder and encoder 1 in the display module may be two different encoding units in the same encoder, and the encoder 2 may be another encoder.
  • the encoding modes of different encoders may be the same or different.
  • the encoding methods of different coding units of the same encoder can be the same or different. Therefore, the image formats output by the encoder in the display module and the encoder 1 may be the same or different.
  • the image output by the encoder in the display module and encoder 1 can be an image in any format such as Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), etc. .
  • JPEG Joint Photographic Experts Group
  • TIFF Tag Image File Format
  • the electronic device in the embodiment of the present application may be a mobile phone, a tablet computer, a smart watch, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook , and cellular phones, personal digital assistants (personal digital assistant, PDA), augmented reality (augmented reality, AR) ⁇ virtual reality (virtual reality, VR) equipment, etc.
  • PDA personal digital assistant
  • augmented reality augmented reality, AR
  • VR virtual reality
  • the form is not particularly limited.
  • FIG. 4 is a schematic structural diagram of an electronic device 400 provided in an embodiment of the present application.
  • the electronic device 400 may include: a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (universal serial bus, USB) interface 430, a charging management module 440, a power management module 441, a battery 442, antenna 1, antenna 2, mobile communication module 450, wireless communication module 460, audio module 470, speaker 470A, receiver 470B, microphone 470C, earphone jack 470D, sensor module 480, button 490, motor 491, indicator 492, camera 493, a display screen 494, and a subscriber identification module (subscriber identification module, SIM) card interface 495, etc.
  • SIM subscriber identification module
  • the above-mentioned sensor module 480 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.
  • sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.
  • the structure shown in this embodiment does not constitute a specific limitation on the electronic device 400 .
  • the electronic device 400 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 410 may include one or more processing units, for example: the processor 410 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit, NPU
  • the controller may be the nerve center and command center of the electronic device 400 .
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 410 for storing instructions and data.
  • the memory in processor 410 is a cache memory.
  • the memory may hold instructions or data that the processor 410 has just used or recycled. If the processor 410 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 410 is reduced, thus improving the efficiency of the system.
  • processor 410 may include one or more interfaces. It can be understood that the interface connection relationship between the modules shown in this embodiment is only for schematic illustration, and does not constitute a structural limitation of the electronic device 400 . In other embodiments, the electronic device 400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 440 is configured to receive charging input from the charger. While the charging management module 440 is charging the battery 442 , it can also supply power to the electronic device through the power management module 441 .
  • the power management module 441 is used for connecting the battery 442 , the charging management module 440 and the processor 410 .
  • the power management module 441 receives the input from the battery 442 and/or the charging management module 440 to provide power for the processor 410 , internal memory 421 , external memory, display screen 494 , camera 493 , and wireless communication module 460 .
  • the wireless communication function of the electronic device 400 can be realized by the antenna 1, the antenna 2, the mobile communication module 450, the wireless communication module 460, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • the antenna 1 of the electronic device 400 is coupled to the mobile communication module 450, and the antenna 2 is coupled to the wireless communication module 460, so that the electronic device 400 can communicate with the network and other devices through wireless communication technology.
  • the electronic device 400 implements a display function through a GPU, a display screen 494, and an application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 494 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 410 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 494 is used to display images, videos and the like.
  • the display screen 494 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode).
  • diode, AMOLED flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 400 can realize the shooting function through an ISP, a camera 493 , a video codec, a GPU, a display screen 494 , and an application processor.
  • the ISP is used to process data fed back by the camera 493 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be located in the camera 493 .
  • Camera 493 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other image signals.
  • the electronic device 400 may include N cameras 493, where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 400 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • Electronic device 400 may support one or more video codecs.
  • the electronic device 400 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.
  • MPEG moving picture experts group
  • the NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • Applications such as intelligent cognition of the electronic device 400 can be implemented through the NPU, such as image recognition, face recognition, voice recognition, text understanding, and the like.
  • the external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 400.
  • the external memory card communicates with the processor 410 through the external memory interface 420 to implement a data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 421 may be used to store computer-executable program code, which includes instructions.
  • the processor 410 executes various functional applications and data processing of the electronic device 400 by executing instructions stored in the internal memory 421 .
  • the processor 410 may execute instructions stored in the internal memory 421, and the internal memory 421 may include a program storage area and a data storage area.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data created during the use of the electronic device 400 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the electronic device 400 may implement audio functions through an audio module 470 , a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, and an application processor. Such as music playback, recording, etc.
  • the keys 490 include a power key, a volume key and the like.
  • the motor 491 can generate a vibrating prompt.
  • the indicator 492 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 495 is used for connecting a SIM card.
  • the SIM card can be connected and separated from the electronic device 400 by inserting it into the SIM card interface 495 or pulling it out from the SIM card interface 495 .
  • the electronic device 400 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 495 can support Nano SIM card, Micro SIM card, SIM card etc.
  • FIG. 5 is a block diagram of the software structure of the mobile phone according to the embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the Android TM system is divided into five layers, from top to bottom are application program layer, application program framework layer, Android runtime (Android runtime) and system library, hardware abstraction layer (hardware abstraction layer, HAL ) and the kernel layer.
  • Android runtime Android runtime
  • HAL hardware abstraction layer
  • this article uses the Android system as an example.
  • other operating systems such as Hongmeng TM system, IOS TM system, etc.
  • the solution of the present application can also be implemented.
  • the application layer can consist of a series of application packages.
  • applications such as call, memo, browser, contacts, gallery, calendar, map, bluetooth, music, video, and short message can be installed in the application layer.
  • an application with a shooting function for example, a camera application
  • a shooting function for example, a camera application
  • the application program layer may be installed in the application program layer.
  • other applications need to use the shooting function, they can also call the camera application to realize the shooting function.
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, etc., which are not limited in this embodiment of the present application.
  • the window manager described above is used to manage window programs.
  • the window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.
  • the above-mentioned content providers are used to store and obtain data, and make these data accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.
  • the above view system can be used to build the display interface of the application.
  • Each display interface can consist of one or more controls.
  • controls may include interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets (Widgets).
  • the resource manager mentioned above provides various resources for the application, such as localized strings, icons, pictures, layout files, video files and so on.
  • the above-mentioned notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify the download completion, message reminder, etc.
  • the notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, sending out prompt sounds, vibrating, and flashing lights, etc.
  • the Android runtime includes a core library and a virtual machine.
  • the Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.
  • the application layer and the application framework layer run in virtual machines.
  • the virtual machine executes the java files of the application program layer and the application program framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • a system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.
  • the surface manager is used to manage the display subsystem, and provides the fusion of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc.
  • the media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.
  • 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is located below the HAL and is the layer between hardware and software.
  • the kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, etc., which are not limited in this embodiment of the present application.
  • a camera service may be set in the application framework layer.
  • the camera application can start the Camera Service by calling the preset API.
  • Camera Service can interact with Camera HAL in Hardware Abstraction Layer (HAL) during operation.
  • Camera HAL is responsible for interacting with hardware devices (such as cameras) that realize shooting functions in mobile phones.
  • Camera HAL hides the implementation details of related hardware devices (such as specific image processing algorithms), and on the other hand, it can provide Call the interface of related hardware devices.
  • the related control commands sent by the user can be sent to the Camera Service.
  • Camera Service can send the received control command to Camera HAL, so that Camera HAL can call the camera driver in the kernel layer according to the received control command, and the camera driver will drive the camera and other hardware devices to respond to the control command to collect images data.
  • the camera can transmit each frame of image data collected to the Camera HAL through the camera driver at a certain frame rate.
  • the transfer process of the control command inside the operating system refer to the specific transfer process of the control flow in FIG. 5 .
  • Camera Service After Camera Service receives the above control command, it can determine the shooting strategy at this time according to the received control command.
  • the shooting strategy sets specific image processing tasks that need to be performed on the collected image data. For example, in the preview mode, Camera Service can set image processing task 1 in the shooting strategy to implement the face detection function. For another example, if the user enables the beautification function in the preview mode, the Camera Service can also set the image processing task 2 in the shooting strategy to realize the beautification function. Furthermore, Camera Service can send the determined shooting strategy to Camera HAL.
  • the Camera HAL When the Camera HAL receives each frame of image data collected by the camera, it can perform corresponding image processing tasks on the above image data according to the shooting strategy issued by the Camera Service, and obtain each frame of the image after image processing. For example, Camera HAL can perform image processing task 1 on each frame of image data received according to shooting strategy 1, and obtain corresponding shooting pictures of each frame. When shooting strategy 1 is updated to shooting strategy 2, Camera HAL can perform image processing task 2 on each frame of image data received according to shooting strategy 2, and obtain corresponding shooting pictures of each frame.
  • the Camera HAL can report each frame of the captured image after image processing to the camera application through the Camera Service, and the camera application can display each frame of the captured image on the display interface, or the camera application can display the captured image in the form of a photo or video Each frame shot is saved in the phone.
  • the transfer process of the above-mentioned shooting picture inside the operating system refer to the specific transfer process of the data stream in FIG. 5 .
  • the working principle of the method for realizing the embodiment of the present application is introduced by each software layer in the mobile phone with reference to FIG. 5 .
  • the camera application When the camera application is running in video recording mode, it can send the capture command issued by the user to the Camera Service.
  • Camera HAL can call the camera driver in the kernel layer according to the video recording command received before, and the camera driver drives the camera and other hardware devices to respond to the video recording command to collect image data.
  • the camera can transmit each frame of image data collected to the Camera HAL through the camera driver at a certain frame rate.
  • the data stream composed of each frame of image transmitted by the camera driver to the Camera HAL based on the video recording instruction may be the video stream (such as the preview stream and the video stream) described in the embodiment of the present application.
  • the frame selection module is included in the HAL.
  • the Camera HAL can call the frame selection module to select an alternative frame (that is, a wonderful frame) from the first cache queue.
  • each frame of image (image 1 shown in FIG. 3 ) exposed and output by the image sensor (Sensor) of the camera may be buffered in the first buffer queue (Buffer).
  • Camera HAL can call the frame selection module to select a frame of image (image 2 as shown in Figure 3 ) that matches the candidate frame (ie highlight frame) from the first cache queue.
  • the first cache queue can be set on any layer of the mobile phone software system, such as the first cache queue can be set in the memory area accessed by the Camera HAL through the software interface.
  • the HAL also includes preset RAW domain image processing algorithms.
  • Camera HAL can call the preset RAW domain image processing algorithm to process the image 2 that matches the above candidate frame, and obtain the processed image frame (image 3 shown in Figure 3).
  • the image 3 can be processed by the ISP to obtain a high-resolution image, which is recorded as (high resolution, HR).
  • the encoder 2 can encode the HR to obtain a wonderful image.
  • the mobile phone can show the wonderful image to the user after the recording is over.
  • An embodiment of the present application provides a method for acquiring an image, and the method can be applied to a mobile phone, where the mobile phone includes a camera.
  • the method can be applied in the video recording process of a mobile phone.
  • the mobile phone can automatically select the wonderful images in the video and present them to the user after the video is over.
  • the action of the mobile phone to automatically select the best image in the video is invisible to the user.
  • the method may include S601-S607.
  • the mobile phone receives a first operation of the user.
  • the first operation is used to trigger the mobile phone to start recording video.
  • the mobile phone may display the viewfinder interface 701 shown in FIG. 7 .
  • the viewfinder interface 701 of the video is the viewfinder interface of the mobile phone that has not started to record.
  • the viewfinder interface 701 of the video includes a button 702 of “Start Video”.
  • the above-mentioned first operation may be the user's click operation on the "start recording" button 702, which is used to trigger the mobile phone to start recording video.
  • the mobile phone displays a first interface.
  • the first interface is a viewfinder interface where the mobile phone is recording a video.
  • the first interface includes a preview image obtained from the first image collected by the camera of the mobile phone after the mobile phone receives the first operation.
  • the first operation is the user's click operation on the button 702 of "Start Video Recording" as an example.
  • the display screen of the mobile phone can display the first interface 703 shown in FIG. 7 .
  • the first interface 703 is a viewfinder interface where the mobile phone is recording a video.
  • the first interface 703 includes a preview image 704 .
  • the preview image 704 is obtained from the first image collected by the camera of the mobile phone after the mobile phone receives the first operation.
  • the embodiment of the present application introduces a method for the mobile phone to obtain the preview image 704 from the first image.
  • the mobile phone can sample the ISP of the mobile phone, and perform RAW image processing, RGB domain image processing, and YUV domain image processing on the first image in sequence to obtain a preview image 704 .
  • the ISP of the mobile phone may perform the above-mentioned RAW image processing, RGB image processing and YUV image processing on each frame of the first image captured by the camera.
  • FIG. 8A shows a schematic diagram of an image processing principle in a recording process provided by an embodiment of the present application.
  • the method for the mobile phone to obtain the preview image 704 from the first image may refer to the processing method of "preview image" shown in FIG. 8A.
  • the image sensor (Sensor) of the mobile phone is controlled by exposure, and can continuously output Bayer images (ie, the first image).
  • Each frame of the Bayer image (that is, the first image) is processed by the ISP of the mobile phone in the RAW domain to obtain an RGB image, and the RGB image is processed by the ISP in the RGB domain to obtain a YUV image.
  • the YUV image is processed by the processing algorithm 1, and then the ISP performs the image processing in the YUV domain, and then sends it to the encoder of the display module for encoding or format conversion to obtain the preview image 704 .
  • the processed multi-frame preview images 704 may form a preview video stream.
  • the first interface 703 further includes a snapshot shutter 702 .
  • the snapshot shutter 702 is used to trigger the mobile phone to capture an image to obtain a photo.
  • the snapshot shutter 702 is used to trigger the mobile phone to capture an image during video recording to obtain a photo. It is conceivable that some wonderful pictures may be collected during the process of recording video (that is, video recording) by the mobile phone. During the video recording process of the mobile phone, the user may hope that the mobile phone can capture the above-mentioned wonderful picture, and save it as a photo for display to the user. The user can click the above-mentioned snapping shutter 702 to realize the function of snapping wonderful images during the video recording process.
  • the mobile phone can automatically select a beautiful image in the video and present it to the user.
  • the method in this embodiment of the present application may further include S603-S607.
  • the first interface 703 may not include the snapshot shutter 702 .
  • the mobile phone can automatically select the beautiful images in the video and present them to the user.
  • the method in this embodiment of the present application may further include S603-S07.
  • the mobile phone can cache the Sensor exposure output Bayer image in a first buffer queue (Buffer). In this way, even if there is a delay between receiving the user's snapping operation and the Snapshot program receiving the snapping command; when the user's snapping operation is received, the Bayer image output by the Sensor can also be cached in the first cache queue. In this way, the mobile phone can acquire this frame of image from the first cache queue. Specifically, in response to the foregoing first operation, the mobile phone may further execute S603.
  • the mobile phone caches the first image captured by the camera in the first cache queue.
  • the first cache queue caches n frames of first images collected by the camera, where n ⁇ 1, and n is an integer.
  • the mobile phone may cache the first image captured by the camera in the first buffer queue (Buffer) shown in FIG. 8A .
  • the first buffer queue may buffer n frames of first images collected by the camera on a first-in-first-out basis.
  • the tail of the first cache queue can perform an enqueue operation for inserting the first image; the queue head of the first cache queue can perform a dequeue operation for deleting the first image.
  • n frames of the first image have been cached in the first buffer queue, each time a frame of the first image is inserted at the tail of the first buffer queue, a frame of the first image is deleted at the head of the first buffer queue.
  • n may be equal to 1.
  • one frame of the first image may be buffered in the first buffer queue.
  • the preset RAW domain image processing algorithm can only input one frame of the first image.
  • n may be greater than 1.
  • multiple frames of the first image may be cached in the first cache queue.
  • the preset RAW domain image processing algorithm may input one frame of the first image, or may input multiple frames of the first image.
  • inputting multiple frames of the first image to the preset RAW domain image processing algorithm can enhance the image quality of the reference frame, which is beneficial to obtain information such as noise and texture, and can further improve the output image of the preset RAW domain image processing algorithm. image quality.
  • the mobile phone selects a second image from the first video stream.
  • the second image includes an image of the target object.
  • the first video stream is obtained by the ISP of the mobile phone performing first processing on multiple frames of the first image, and the first processing includes downsampling.
  • the first video stream may be the data stream of the preview image; or, the first video stream may be the data stream of the recording file.
  • the image can be processed by the ISP in the "RAW domain", “RGB domain” and “YUV domain”; after the "YUV domain” image processing, It can be divided into two data streams: one data stream for preview images, and the other data stream for video files.
  • the first video stream may be a data stream after the ISP performs image processing in "RAW domain” and "RGB domain” on the first image output by the Sensor.
  • the above-mentioned target object may be at least one of human face, human body, food, animal, building and the like. That is to say, the mobile phone can identify the images in the first video stream, and use the image including the target object as the second image.
  • the target object includes a human face
  • the second image may be an image in which the expression of the target object satisfies a preset condition. That is to say, the mobile phone (the frame selection module of the mobile phone) can select the image whose expression of the target object satisfies the preset condition from the first video stream, as the second image shown in FIG. 8A (i.e. the alternative frame shown in FIG. 3 ).
  • the expression of the target object satisfies a preset condition, which may specifically be: a human face may smile. That is to say, the second image may be a frame of image including a smiling face image in the first video stream.
  • the mobile phone can capture wonderful images of smiling faces in the video.
  • the target object includes a human body.
  • the above-mentioned second image may be an image in which the action of the target object satisfies a preset condition. That is to say, the mobile phone (the frame selection module of the mobile phone) can select an image whose action of the target object satisfies the preset condition from the first video stream, as the second image shown in FIG. 8A (that is, the alternative frame shown in FIG. 3 ).
  • the action of the target object satisfies the preset condition is the preset action.
  • the preset action may be a human body action when the human body jumps to the highest point, a wonderful action during a dance performance or a competitive sports performance, a difficult action during a human body exercise process, and the like.
  • the second image may be a frame of image in the first video stream when the human body poses a preset action.
  • the mobile phone can capture wonderful images of the human body in motion during the video recording process when it poses a preset action.
  • the above-mentioned target object may be pre-configured in the mobile phone when the mobile phone leaves the factory. Multiple target objects can be pre-configured in the mobile phone.
  • the above-mentioned target object may also be set by the user in the mobile phone.
  • the mobile phone may receive the type of the target object set by the user on the setting interface.
  • the mobile phone may display the setting interface 1001 shown in (a) in FIG. 10 .
  • the setting interface 1001 includes preset options, such as the option 1002 of “Shooting Target Object”.
  • the "shoot target object” option 1002 is used to trigger the mobile phone to set the target object to be captured during the video recording process.
  • the mobile phone may display a target object setting interface 1003 shown in (b) of FIG. 10 .
  • the target object setting interface 1003 includes multiple options for shooting objects, such as "face” option, "human body” option, “food” option, “animal” option, and "architecture” option.
  • the mobile phone may receive a user's selection operation of an option of any one of the plurality of options for shooting objects. For example, the mobile phone may receive the user's selection operation on the "face” option and the "food” option shown in (b) in FIG. 10 , and display the target object setting interface 1004 shown in (b) in FIG. 10 . Afterwards, the mobile phone may receive the user's click operation on the "OK" button in the target object setting interface 1004 . In response to the user's click operation on the "OK” button, the mobile phone can set the human face and food as the target object.
  • the target object setting interface may not only include options for multiple shooting objects, but also include prompt information for each shooting object.
  • the prompt information may be used to remind the user of the functions that can be realized by the mobile phone after selecting the corresponding shooting object as the target object.
  • the target object setting interface 1003 shown in (b) in FIG. 10 can be replaced with the target object setting interface 1101 shown in FIG. 11 .
  • the target object setting interface 1101 not only includes options for multiple shooting objects (such as "face” option, "human body” option, etc.), but also includes prompt information for each shooting object.
  • the target object setting interface 1101 includes prompt information 1102 for the subject "face” (for example, if you select "face", the mobile phone can capture a wonderful image of a smiling face during the video recording process) and a prompt for the subject "human body”.
  • Information 1103 for example, select "human body", the mobile phone can capture the wonderful image of the preset movement when the human body is moving during the video recording process.
  • the mobile phone can execute the method of the embodiment of the present application only after the preset function is activated, to capture a wonderful image of the target object during the video recording process.
  • the default function is the function of the mobile phone to capture wonderful images of the target object during the video recording process.
  • the mobile phone may enable or disable the aforementioned preset function on the aforementioned target object setting interface 1101 .
  • the target object setting interface 1003 shown in (b) in FIG. 10 the target object setting interface 1004 shown in FIG. 10 (c) and the target object setting interface 1001 shown in FIG. 11 may include preset functions.
  • Switch 1005. The mobile phone may receive the user's selection operation on the switch 1005 of the preset function, and activate or deactivate the preset function.
  • the mobile phone can activate or deactivate the above preset function on the viewfinder interface of the video.
  • the mobile phone can activate or deactivate the above-mentioned preset function on the viewfinder interface where the mobile phone has not yet started recording.
  • the viewfinder interface of the mobile phone that has not started recording includes a switch 1201 with a preset function.
  • the mobile phone can receive the user's operation 1 on the switch 1201 of the preset function shown in (b) in FIG.
  • the viewfinder interface shown. This operation 1 is the third operation.
  • the switch 1201 of the preset function is in an off state, and in the framing interface shown in (c) in FIG.
  • the switch 1201 of the preset function is turned on.
  • the mobile phone can also receive the user's operation 1 on the switch 1201 of the preset function shown in (c) in Figure 12, display the viewfinder interface shown in (b) in Figure 12, and close the preset function.
  • the mobile phone can activate or deactivate the above-mentioned preset function on the viewfinder interface where the mobile phone has started recording.
  • the viewfinder interface of the mobile phone that has started recording includes a switch 1301 for preset functions.
  • the method for activating or deactivating the preset function of the mobile phone on the viewfinder interface that has started recording can refer to the above-mentioned embodiment for the method of activating or deactivating the preset function on the viewfinder interface of the mobile phone that has not started recording, and the embodiment of the present application will not repeat it here.
  • the above-mentioned target object may also be set by the user in the mobile phone.
  • the mobile phone may receive the type of the target object set by the user on the viewfinder interface of the video.
  • the viewfinder interface for video recording may include: in the video recording mode, a viewfinder interface where video recording has started (ie, the first interface) and a viewfinder interface where video recording has not yet started (ie, the second interface).
  • the viewing interfaces shown in (a) in FIG. 12 , (b) in FIG. 12 , (c) in FIG. 12 , and in FIG. 13 all include switches for preset functions.
  • the mobile phone can receive the user's operation 2 of the switch of the preset function (such as any operation such as single-click operation, double-click operation or long-press operation). Wherein, operation 2 is different from operation 1 above. This operation 2 is the second operation.
  • the setting interface of the target object is displayed.
  • the setting interface of the target object includes a plurality of options of the shooting object to be selected. For example, in response to the user's operation 2 on the preset function switch 1301 shown in FIG.
  • the mobile phone may display a target object setting interface 1401 shown at 14 .
  • the mobile phone may determine the object selected by the user as the target object in response to the user's selection operation on the options of multiple objects to be selected in the object setting interface 1401 .
  • the mobile phone may display the target object setting interface 1101 shown in FIG. 11 .
  • the mobile phone may return to the viewing interface shown in FIG. 13 .
  • the above-mentioned target object may be a pre-configured shooting object in the target scene.
  • the target scene may be at least one shooting scene such as a portrait scene, a sports scene, a pet scene, and a food scene.
  • one or more shooting objects can be preset.
  • the pre-configured shooting object (ie, the target object) in a portrait scene may be a human face.
  • the expression of the target object satisfies the preset condition, specifically, it can be a human face smiling. That is, the second image may be a frame image including a smiley image.
  • the mobile phone can capture wonderful images of smiling faces in the video.
  • the pre-configured shooting object in a sports scene may be a human body.
  • the action of the target object is the preset action.
  • the mobile phone can capture wonderful images of the human body in motion during the video recording process when it poses a preset action.
  • the mobile phone can identify the target scene according to the first image collected by the camera in the video recording mode. Then, the mobile phone can recognize whether the expression or movement of the pre-configured shooting object (ie, the target object) in the target scene satisfies the preset condition. If a frame of image in the first video stream includes an image of the target object, and the expression or action of the target object in this frame of image satisfies a preset condition, the mobile phone may select this frame of image as the second image.
  • the above target scene may be pre-configured in the mobile phone when the mobile phone leaves the factory. Multiple target-pair scenarios can be pre-configured in the mobile phone.
  • the above target scene may also be set by the user in the mobile phone.
  • the mobile phone can receive the target scene set by the user on the setting interface.
  • the mobile phone may receive the type of the target object set by the user on the viewfinder interface of the video.
  • the viewfinder interface of the video recording may include: in the video recording mode, a viewfinder interface that has started recording and a viewfinder interface that has not yet started recording.
  • the method for setting the target scene by receiving the user's operation on the mobile phone may refer to the method for setting the target object in the above-mentioned embodiments, which will not be described in detail here in the embodiments of the present application.
  • the setting interface of the target object shown in Figure 11 and Figure 14 can be replaced by the setting interface of the target scene, the option of the shooting object in the setting interface of the target object Can be replaced with options for shooting scenes.
  • the above-mentioned first video stream is obtained by the ISP of the mobile phone performing first processing on multiple frames of the first image.
  • the first processing described above includes downsampling. Downsampling (subsampled) can also be called downsampling (down sampled). Downsampling the image can reduce the size of the image and reduce the resolution of the image. For example, the number of pixels of the image before downsampling may be 4000*3000, and the number of pixels of the image after downsampling may be 2000*1500.
  • the image output by the "RAW domain" of the ISP is a low-resolution image, which is recorded as an LR image.
  • the first video stream includes multiple frames of LR images.
  • the mobile phone After the mobile phone receives the above-mentioned first operation, it can start to select the second image from the first video stream in real time. Or, starting from receiving the above-mentioned first operation, as shown in FIG. 8B , the mobile phone may start to periodically select the second image from the first video stream.
  • the Camera HAL in the HAL of the mobile phone may include a frame selection module. After the Camera HAL receives the recording instruction from the Camera Service, it can start to select the second image (also called a reference frame) from the first video stream in real time or periodically.
  • the second image also called a reference frame
  • the second image is selected from a plurality of down-sampled LR images (that is, the first video stream); therefore, the resolution of the second image is also low. And the wonderful images that users hope to obtain during video recording are high-resolution images. Therefore, in the embodiment of the present application, after the mobile phone selects the second image from the first video stream, it also needs to obtain a high-resolution image with the same or similar image content according to the second image. Specifically, after S604, the method in the embodiment of the present application may further include S605:
  • the mobile phone takes m frames of the first image corresponding to the second image among the n frames of the first image as input, and runs a preset RAW domain image processing algorithm to obtain the third image.
  • the preset RAW domain image processing algorithm has the function of improving image quality. Wherein, m ⁇ 1, m is an integer.
  • the mobile phone executes S603 to cache the image output by the Sensor in the first cache queue.
  • N frames of first images may be cached in the first cache queue.
  • the first image cached in the first cache queue is not down-sampled and is a high-resolution image. Therefore, the mobile phone can find a high-resolution image with the same or similar image content as the second image from the n frames of the first image buffered in the first buffer queue.
  • S605 may include S605a-S605b.
  • the mobile phone selects a fifth image corresponding to the second image from n frames of the first image.
  • the fifth image is output by the image sensor of the camera at the closest time to the second image output by the image sensor of the camera; or, in the n frames of first images, the resolution of the fifth image is Highest.
  • the aforementioned two frame selection criteria such as resolution and time
  • the method in the embodiment of the present application is introduced by taking the time when the fifth image is output by the image sensor of the camera as an example closest to the time when the second image is output by the image sensor of the camera.
  • each frame of the first image above corresponds to a piece of time information
  • the time information records the time when the image sensor outputs the corresponding first image.
  • the time information may also be called a time stamp.
  • the mobile phone (such as the above-mentioned frame selection module) can select the first image whose time stamp in the first buffer queue Buffer is the closest to the time stamp recorded by the second image as the fifth image.
  • the mobile phone may cache the Bayer image output by the Sensor exposure in a first buffer queue Buffer.
  • the first buffer queue can buffer multiple frames of Bayer images. In this way, even if the Sensor outputs the first image corresponding to the second image to the mobile phone (such as the frame selection module) to select the second image from the first video stream, there is a delay.
  • the Bayer image output by the Sensor may also be cached in the first cache queue.
  • the mobile phone may also select a high-resolution image with the same or similar image content as the second image from the Buffer.
  • the Sensor exposure end time may be used as the time stamp; on other platforms, the Sensor start exposure time may be used as the time stamp, which is not limited in this embodiment of the present application.
  • the mobile phone takes m frames of the first image among the n frames of the first image as input, and runs a preset RAW domain image processing algorithm to obtain the third image.
  • the m frames of first images include the above-mentioned fifth image.
  • Other images except the fifth image in the m frames of first images are used to enhance the image quality of the fifth image.
  • m may be equal to 1. That is to say, the m frames of first images are the above-mentioned fifth images.
  • the mobile phone uses the fifth image as an input to run a preset RAW domain image processing algorithm to obtain a third image with higher quality.
  • the preset RAW domain image processing algorithm is an image processing algorithm with single frame input and single frame output.
  • parameters such as data integrity and texture in a frame of image are limited, and running a preset RAW domain image processing algorithm with a frame of image as input cannot effectively improve the image quality of this frame of image.
  • m may be greater than 1.
  • the mobile phone may use the fifth image and at least one frame of images adjacent to the fifth image as input, and run a preset RAW domain image processing algorithm. That is, m frames of first images including the fifth image among n frames of first images may be used as input to run a preset RAW domain image processing algorithm.
  • the preset RAW domain image processing algorithm is an image processing algorithm with multi-frame input and single-frame output. It should be understood that the images other than the fifth image in the m frames of the first image can enhance the image quality of the fifth image, which is conducive to obtaining information such as noise and texture, and can further improve the image quality of the third image .
  • the aforementioned m frames of first images are m adjacent frames of images in the first buffer queue.
  • the m frames of first images may also be m frames of images that are not adjacent but include the fifth image among the n frames of first images buffered in the first buffer queue.
  • the preset RAW domain image processing algorithm described in the embodiment of the present application may be a neural network model with multi-frame input and single-frame output.
  • the preset RAW domain image processing algorithm is a deep learning network for image quality enhancement in the RAW domain.
  • the algorithm processing of the preset RAW domain image processing algorithm is added.
  • the preset RAW domain image processing algorithm Combined with ISP the effect is better, which helps to improve the image quality of the third image.
  • the mobile phone can find a high-resolution image (that is, the fifth image) that has the same or similar image content as the second image from the n frames of the first image cached in the first buffer queue.
  • the mobile phone may use images other than the fifth image in the m frames of the first image to enhance the image quality of the fifth image. In this way, a third image with high resolution and high image quality can be obtained.
  • the mobile phone uses the ISP to perform the second processing on the third image to obtain the fourth image. This second process does not include downsampling.
  • the mobile phone may use the ISP to process the first image to obtain a preview image, and process the third image to obtain a fourth image by means of time division multiplexing. That is to say, the mobile phone uses the ISP to process the first image to obtain the preview image, which will not affect the mobile phone to use the ISP to process the third image to obtain the fourth image.
  • the fourth image is used to generate a wonderful image in the video recording process. In other words, the processing of the wonderful images shown in FIG. 8A or 8B by the mobile phone will not affect the processing of the preview images and video files shown in FIG. 8A or 8B by the mobile phone.
  • the mobile phone may use an ISP hardware module to process the first image to obtain a preview image, and process the third image to obtain a fourth image.
  • the image processing flow of obtaining the preview image from the first image also includes the processing flow of the processing algorithm 1 .
  • the above-mentioned processing algorithm 1 may be included in a hardware module of the ISP.
  • the processing algorithm 1 may be included in other processors of the mobile phone (such as any processor such as CPU, GPU or NPU).
  • the hardware module of the ISP may call the processing algorithm 1 in the above-mentioned other processors to process the first image to obtain the preview image.
  • the fourth image obtained after the mobile phone uses the ISP to perform the second processing not including downsampling on the third image is also a high-resolution image.
  • the fourth image is processed by the ISP, which can also improve the image quality compared with the third image. Therefore, the fourth image is also an image with high resolution and high image quality.
  • the mobile phone displays the exciting image obtained by encoding the fourth image.
  • the mobile phone can use the preset RAW domain image processing algorithm to process m frames to obtain the third image (i.e. execute S605); the mobile phone can use the ISP to process the third image to obtain the fifth image (i.e. execute S606); Afterwards, the mobile phone may use the encoder 2 to encode the wonderful image obtained by encoding the fourth image (ie, the HR image). Moreover, the mobile phone can display the wonderful image to the user after the video recording ends.
  • the mobile phone may receive the user's click operation on the "end recording” button 706 shown in FIG. 7 or FIG. 16, and may end the recording and generate a recording file.
  • the mobile phone may display the video viewfinder interface 1601 shown in FIG. 16 .
  • the viewfinder interface 1601 of the video is the viewfinder interface when the mobile phone has not started to record.
  • the photo in the photo option in the viewfinder interface of the mobile phone is updated from 708 shown in FIG. 7 to 1602 shown in FIG. 16 .
  • the mobile phone may display the album list interface 1701 shown in (a) of FIG. 17 or the album list interface 1703 shown in FIG. 7 in response to the user's start operation on the album application.
  • the album list interface 1701 includes one or more photos and videos that can be saved in the mobile phone.
  • the album list interface 1701 and the album list interface 1703 include video files 1702 recorded by the mobile phone.
  • the mobile phone video recording after the mobile phone video recording is over, the wonderful images captured during the video recording process can be displayed to the user, achieving the effect of getting more than one shot. More than one shot means that after the mobile phone video is over, not only the video file can be obtained, but also the wonderful images captured in the video can be obtained.
  • the beautiful images captured by the mobile phone in the video are visible to the user in the photo album. Users can view the wonderful images captured in the video from the album.
  • the photo album list interface 1703 also includes photos 1704 including wonderful images captured by the mobile phone during recording video files 1702 . Take the target object being a human face as an example. Photo 1704 includes a smiley image.
  • the beautiful images captured by the mobile phone in the video recording are not visible to the user in the photo album. Users can view the beautiful images captured in the video from the album.
  • the photo album list interface 1701 does not include the photo 1704 shown in (b) in FIG. 17 .
  • the mobile phone may send a first prompt message in response to the operation of ending the video recording.
  • the first prompt information is used to request the user to confirm whether to keep the wonderful images captured during the recording process.
  • the mobile phone in response to the user's click operation on the "End Recording" button 706 shown in Figure 16, as shown in Figure 18, the mobile phone can display a first prompt message 1801 on the viewfinder interface 1601, such as "The video is automatically captured for you during the recording process.” A wonderful frame, please confirm if you want to keep it?".
  • the first prompt information 1801 may also include a preview image of the wonderful image, for the user to confirm whether to keep the exciting image according to the preview image.
  • the mobile phone in response to the user's operation 3 on the first prompt message 1801, can save wonderful images in the photo album.
  • operation 3 may be the user's click operation on the "reserve" button in the first prompt information 1801 .
  • the album list interface 1701 does not include the photo 1704 shown in (b) in FIG. 17 .
  • the mobile phone in response to the user's operation 3 on the first prompt message 1801, can also display the wonderful image in other ways. For the manner in which the mobile phone displays wonderful images, reference may be made to the detailed descriptions in the following embodiments, which will not be repeated here.
  • the embodiment combines the above implementation (1), although the mobile phone will not display wonderful images in the photo album. However, in response to the user's operation 3 on the first prompt message 1801, the mobile phone can still display the wonderful image in other ways.
  • the mobile phone displays wonderful images reference may be made to the detailed descriptions in the following embodiments, which will not be repeated here.
  • the mobile phone can at least include the following ways (a) and way (b):
  • the video file can be generated.
  • the details page of the video file may include a first preset control.
  • the mobile phone may receive a user's click operation on the first preset control.
  • the mobile phone can display the wonderful images captured during the shooting of the video file (ie video).
  • the mobile phone in response to the user's click operation on the video file 1702 shown in (a) in Figure 17 or (b) in Figure 17, as shown in (a) in Figure 19, the mobile phone can display the details of the video file 1702 Page 1901.
  • the details page 1901 includes a first preset control 1902 .
  • the mobile phone in response to the user's click operation on the first preset control 1902, as shown in (b) in Figure 19, the mobile phone can display the details page 1903 of the wonderful image captured during the shooting of the video file 1702, and present the wonderful image to the user .
  • the mobile phone in response to the user's click operation on the return button on the details page 1903, as shown in (c) in FIG. 19, the mobile phone can display the details page 1904 of the video file 1702.
  • the mobile phone can receive the preset gesture input by the user on the details page of the video file. In response to the preset gesture, the mobile phone can display wonderful images captured during the shooting of the video file (ie video).
  • the aforementioned preset gesture may be any sliding gesture such as an upward or downward sliding gesture, an S-shaped sliding gesture, or a ⁇ -shaped sliding gesture.
  • the mobile phone in response to the user's click operation on the video file 1702 shown in (a) in Figure 17 or (b) in Figure 17, as shown in (a) in Figure 20, the mobile phone can display the details of the video file 1702 Page 1901.
  • the mobile phone may receive a preset gesture input by the user on the details page 1901 (such as an upward sliding gesture).
  • the mobile phone can display the details page 1903 of the wonderful image captured during shooting the video file 1702 to show the wonderful image to the user.
  • the details page of the video file 1702 may include the first preset control 1902 or may not include the first preset control 1902 .
  • the details page 1901 shown in (a) in FIG. 20 includes a first preset control 1902 .
  • the details page 2001 shown in (b) in FIG. 20 does not include the first preset control 1902 .
  • the mobile phone may also display the detail page 1903 of the wonderful image captured during the shooting of the video file 1702, and present the wonderful image to the user.
  • the triggering methods for displaying the wonderful image to the user include but not limited to the above-mentioned methods (a) and (b), and other triggering methods will not be described here in this embodiment of the application.
  • the mobile phone may cache the Bayer image (ie, image 1) output by the Sensor exposure in a first buffer queue (Buffer).
  • the first buffer queue can buffer multiple frames of Bayer images. In this way, even if there is a delay from "the Sensor outputs the Bayer image corresponding to the candidate frame” to "the frame selection module selects the candidate frame", the Bayer image corresponding to the candidate frame can be cached in the first buffer queue. In this way, the mobile phone can obtain the Bayer image corresponding to the candidate frame (image 2 shown in FIG. 3 ) from the first cache queue.
  • the mobile phone can also use the preset RAW domain image processing algorithm to process image 2 to obtain image 3, which helps to improve the quality of wonderful images.
  • image LR in the video stream corresponding to the preview image and the recording file is down-sampled.
  • image HR obtained by the ISP processing the image 3 shown in FIG. 3 is not down-sampled; therefore, the wonderful image obtained by encoding the image HR by the encoder 2 is a high-resolution image.
  • the mobile phone can show the wonderful image shown in Figure 3 to the user after the video recording is finished.
  • the mobile phone can automatically select the wonderful images in the video and present them to the user, and the image quality (such as resolution and picture quality) of the above-mentioned beautiful images can be improved.
  • the first image whose time information is the same as that of the second image may not be cached in the first cache queue.
  • the time information of the first image cached in the first cache queue may be quite different from the time information of the second image.
  • the image content of the first image cached in the first cache queue may be quite different from the image content of the second image. Therefore, in this case, the first image (or the first image with the highest definition in the first cache queue) with the time in the first cache queue closest to the time of the second image is directly used as the first image for encoding to generate a wonderful image. image (that is, the fourth image described in S606 and S607), it may not be possible to accurately capture the wonderful image corresponding to the second image.
  • the mobile phone can still select the fifth image described in S605a-S605b from the first cache queue according to the second image according to the method described in the above embodiment.
  • the fifth image described in S605a-S605b can no longer be used as the base image for generating the highlight image, but the second image is used as the base image for generating the highlight image, including the fifth image described in S606 and S607
  • the m frames of the first image can be used to enhance the image quality of the second image.
  • the method in this embodiment of the present application may include S601-S605 and S2101-S2104.
  • S601-S605 and S2101-S2104.
  • the mobile phone uses the ISP to perform third processing on the third image to obtain a sixth image.
  • This third processing includes image processing in RAW domain and RGB domain.
  • the third processing described in S2101 is different from the second processing described in S606.
  • the second processing described in S606 may include image processing in the RAW domain, RGB domain, and YUV domain; while the third processing described in S2101 includes image processing in the RAW domain and RGB domain, but does not include image processing in the YUV domain. Therefore, the sixth image obtained by executing S2101 on the mobile phone is different from the fourth image obtained by executing S606 on the mobile phone.
  • the mobile phone uses the sixth image to enhance the image quality of the second image to obtain a seventh image.
  • FIG. 22 shows a schematic diagram of an image processing principle in a video recording process provided by an embodiment of the present application.
  • the image sensor (Sensor) of the mobile phone is controlled by exposure, and can continuously output Bayer images (ie, the first image).
  • Bayer images ie, the first image.
  • the mobile phone after the mobile phone uses the preset RAW domain image processing algorithm to process m frames of the first image to obtain the third image, the mobile phone will not directly use the ISP to process the third image to obtain the code for encoding and generating a wonderful image. image.
  • the image content of the first image and the image content of the second image cached in the first cache queue may be quite different. Therefore, in this embodiment, the first image (or the first image with the highest definition in the first cache queue) that is the closest to the second image in the first cache queue will not be directly used as the key for encoding to generate a wonderful image. image. Instead, use the second image as the base image for generating a great image.
  • the mobile phone may use the sixth image with higher image quality to enhance the image quality of the second image to obtain an image with higher resolution and substantially the same image content as that of the second image.
  • the mobile phone may perform RAW image processing and RGB domain image processing on the third image to obtain a sixth image.
  • the third image is obtained by the mobile phone using the preset RAW domain image processing algorithm to process the first image of m frames
  • the sixth image is obtained by the mobile phone using the ISP to process the third image; therefore, compared to the image obtained by using the ISP
  • the second image the sixth image may include more image details. Therefore, as shown in FIG. 22, the mobile phone can use the sixth image as a reference (Refence, REF) image, and use the sixth image (that is, the REF image) to enhance the image quality of the second image to obtain a higher image quality. Seventh image.
  • REF Reference
  • the mobile phone uses the ISP to perform fourth processing on the seventh image to obtain an eighth image.
  • This fourth process includes image processing in the YUV domain, which does not include downsampling.
  • the mobile phone may use the ISP to process the first image to obtain a preview image, process the third image to obtain a sixth image, and process the seventh image to obtain an eighth image by means of time division multiplexing. That is to say, the mobile phone uses the ISP to process the first image to obtain the preview image, which will not affect the mobile phone to use the ISP to process the third image to obtain the sixth image, and process the seventh image to obtain the eighth image.
  • the eighth image is used to generate a wonderful image in the video recording process. In other words, the processing of the wonderful images shown in FIG. 22 by the mobile phone will not affect the processing of the preview images and video files shown in FIG. 22 by the mobile phone.
  • the mobile phone displays the exciting image obtained by encoding the eighth image.
  • the mobile phone can use the ISP to process the seventh image to obtain the eighth image (that is, execute S2103); after that, the mobile phone can use the encoder 2 to encode the eighth image to obtain a wonderful image. Moreover, the mobile phone can display the wonderful image to the user after the video recording ends.
  • the mobile phone performs S2104 to display the wonderful image obtained by encoding the eighth image.
  • the mobile phone executes S607 to display the exciting image obtained by encoding the fourth image.
  • the mobile phone may perform S605a to find the first image whose time information and the time indicated by the time information of the second image are within a preset duration from the first cache queue cache.
  • the mobile phone executes S605a and finds the above image from the first cache queue, the mobile phone uses the found image as the fifth image. Then, the mobile phone can execute S605b and S606-S607.
  • the mobile phone may use the first frame of the first image whose time information is closest to the time indicated by the time information of the second image as the fifth image. Then, the mobile phone can execute S605b and S2101-S2104.
  • the image quality enhancement described in S2102 above may include image fusion.
  • the image quality enhancement shown in FIG. 22 can be replaced by the image fusion shown in FIG. 23 .
  • S2102 can be replaced with S2102'.
  • S2102' The mobile phone performs image fusion on the sixth image and the second image to obtain a seventh image.
  • the third processing described in S2101 may or may not include downsampling.
  • the resolution of the sixth image obtained by the mobile phone using the ISP to perform the third processing on the third image is lower than the resolution of the third image. Since the second image is an image after the first processing including downsampling; therefore, the resolution of the second image is also relatively low. In this case, both the sixth image and the second image may be images with lower resolution. Of course, the resolution of the sixth image may be the same as or different from the resolution of the second image.
  • the sixth image and the second image may both be images with lower resolution; however, since the third image is obtained by processing m frames of the first image with the preset RAW domain image processing algorithm of the mobile phone, and the sixth image is obtained by the mobile phone using The third image is obtained by ISP processing; therefore, the sixth image may include more image details than the second image obtained by ISP processing. Therefore, in this embodiment, the mobile phone executes S1202 to supplement richer image details in the sixth image to the second image to obtain a seventh image. In this way, the image quality of the seventh image can be improved.
  • the mobile phone may use the sixth image to enhance the quality of the second image through a fusion network (also referred to as an image fusion network) to obtain the seventh image.
  • a fusion network also referred to as an image fusion network
  • the method for the mobile phone to perform image enhancement through the fusion network can refer to related methods in the conventional technology, which will not be described in detail here in the embodiment of the present application.
  • registration may be performed on the sixth image and the second image.
  • the mobile phone may use the registered sixth image to perform image fusion (Fusion) on the registered second image.
  • image fusion Fusion
  • the registration of the second image and the sixth image can improve the success rate and effect of image quality enhancement of the mobile phone.
  • registration can include two ways: global registration and local registration.
  • Global registration generally uses feature point detection and matching. Take the registration of the sixth image and the second image by the mobile phone as an example.
  • the mobile phone can detect matching feature points (such as pixel points) in the sixth image and the second image.
  • the phone can then filter for matching feature points. If the number of good feature points in the matched feature points is greater than the preset threshold 1, the mobile phone can consider that the global registration effect is better and fusion can be performed.
  • the Local registration generally uses the optical flow method. Take the registration of the sixth image and the second image by the mobile phone as an example. The mobile phone may first calculate the optical flow for the sixth image and the second image. Then, the mobile phone may make a difference between the second image transformed by optical flow registration and the sixth image transformed by optical flow registration. If the difference is less than the preset threshold 2, the mobile phone can consider that the local registration effect is better and fusion can be performed.
  • the mobile phone before the mobile phone registers the sixth image and the second image, it may first compare the texture similarity between the sixth image and the second image. If the texture similarity between the sixth image and the second image is higher than the preset similarity threshold, it indicates that the texture similarity between the sixth image and the second image is relatively high. In this case, the registration success rate of the mobile phone to the sixth image and the second image is relatively high. By adopting this solution, the success rate of mobile phone registration can be improved.
  • the mobile phone will not register the sixth image with the second image. In this way, the impact of invalid registration on power consumption of the mobile phone is reduced. In this case, the mobile phone may directly use the sixth image as the seventh image.
  • the image quality enhancement mentioned above can implement functions such as noise removal, definition improvement, change or expansion of dynamic range (Dynamic Range), image super-resolution, and the like.
  • the image quality enhancement described in S2102 above may include image super-resolution.
  • the image quality enhancement shown in FIG. 22 can be replaced by the image super-resolution shown in FIG. 24 .
  • S2102 can be replaced by S2102′′.
  • S2102′′ The mobile phone uses the sixth image to perform image super-resolution on the second image to obtain a seventh image.
  • the third processing described in S2101 may not include downsampling.
  • the resolution of the sixth image is higher than that of the third image.
  • the sixth image is a high resolution (high resolution, LR) image.
  • the ISP down-samples the image output by the Sensor during the recording process. Downsampling an image reduces the resolution of the image.
  • the second image in the recording file is an LR image.
  • the mobile phone can use the sixth image as a guide image to enhance the image quality of the second image (including image super-resolution), thereby increasing the resolution of the second image.
  • the mobile phone executes S2101, and may use the sixth image with a resolution of 4k as a guide image to enhance the quality of the fourth image with a resolution of 1080p.
  • the resolution of the enhanced second image (that is, the seventh image) may be 4k.
  • the image super-resolution described in the embodiments of the present application may be reference frame-based image super-resolution (Refence super resolution, REF SR).
  • REF SR reference frame-based image super-resolution
  • the mobile phone may execute S2102′′; if the resolution of the sixth image is lower than or equal to the resolution of the second image, the mobile phone may execute S2102'.
  • the mobile phone can also use the second image as the basic image for generating wonderful images, and use the image with higher image quality obtained by processing the first image in the first cache queue through the preset RAW domain image processing algorithm as the guide image to improve the second image.
  • the image quality of the image is very important.
  • the method can be applied to the identification and generation of highlight frames after the mobile phone video is finished.
  • the mobile phone can automatically select a reference (REF) image meeting the requirements in the video recording, and cache it in the second cache queue.
  • the mobile phone may determine a candidate image from the video file in response to the user's selection operation on any frame of the video file, and the candidate image may be a frame of image preferred by the user in the video file.
  • the mobile phone can use the above REF to enhance the image quality of the image candidate image, so as to generate a wonderful image and present it to the user.
  • the action of generating and buffering the REF image during the recording process of the mobile phone, and the action of generating a wonderful image after the recording of the mobile phone is not visible to the user.
  • the mobile phone video is finished, not only the recorded video can be presented to the user, but also the wonderful images in the video can be presented, achieving the effect of getting more than one shot.
  • the method in this embodiment of the present application may include S601-S605 and S2501-S2506.
  • S601-S605 and S2501-S2506.
  • the mobile phone uses the ISP to perform third processing on the third image to obtain a sixth image, and cache the sixth image in a second cache queue.
  • the third processing includes image processing in the RAW domain and the RGB domain, and the third processing does not include downsampling.
  • the third processing described in S2501 is different from the second processing described in S606.
  • the second processing described in S606 may include image processing in the RAW domain, RGB domain, and YUV domain; while the third processing described in S2101 includes image processing in the RAW domain and RGB domain, but does not include image processing in the YUV domain. Therefore, the sixth image obtained by executing S2501 on the mobile phone is different from the fourth image obtained by executing S606 on the mobile phone. Multiple frames of the sixth image may be cached in the second cache queue.
  • the third processing described in S2501 does not include downsampling. Therefore, the sixth image buffered in the second buffer queue is a high-resolution HR image. Moreover, the sixth image is an image with richer image details and higher image quality processed by a preset RAW domain image processing algorithm.
  • the above sixth image is generated based on the second image by the mobile phone executing S605 and S2501.
  • the second image is an image including the target object in the video file. It can be seen that the sixth image also includes the image of the target object. In this way, after the recording of the mobile phone is finished, it is more likely that the user wants to select a frame of image corresponding to the sixth image from the video file as a wonderful image.
  • the mobile phone can cache multiple frames of the sixth image in the second cache queue, so that the second cache can be used when there is a user demand to select a frame of image from the video file as a wonderful image.
  • the sixth image in the queue performs image quality enhancement (including image super-resolution) on this frame of image to obtain a beautiful image with higher image quality (including higher resolution).
  • the mobile phone generates a video file in response to the operation of ending the video recording.
  • the mobile phone in response to the user's click operation (that is, the first operation) on the "start recording” button 702 shown in FIG. 7 , the mobile phone may execute S601-S605 and S2501.
  • the operation of ending the video recording in S2502 may be the user's click operation on the "end video recording” button 706 shown in FIG. 7 or FIG. 16 .
  • the mobile phone can end the recording and generate a recording file.
  • the mobile phone may display the video viewfinder interface 1601 shown in FIG. 16 .
  • the viewfinder interface 1601 of the video is the viewfinder interface when the mobile phone has not started to record. Compared with the video viewfinder interface 701 shown in FIG. 7 , the photo in the photo option in the viewfinder interface of the mobile phone is updated from 708 shown in FIG. 7 to 1602 shown in FIG. 16 .
  • the mobile phone may display the album list interface 1703 shown in the album list interface 1701 shown in (a) of FIG. 17 in response to the user's start operation on the album application.
  • the album list interface 1701 includes one or more photos and videos that can be saved in the mobile phone.
  • the album list interface 1701 and the album list interface 1703 include video files 1702 recorded by the mobile phone.
  • the album list interface 1701 does not include the photo 1704 shown in (b) in FIG. 17 .
  • the mobile phone determines the candidate image from the video file in response to the user's selection operation on an image in the video file.
  • the resolution of the images (including the alternative images) in the video file is lower.
  • the sixth image is an HR image
  • the alternative image is an LR image.
  • the mobile phone selects a reference (REF) image corresponding to the candidate image from the second cache queue, and uses the REF image to enhance the quality of the candidate image to obtain a ninth image.
  • REF reference
  • the time at which the REF image is output by the image sensor of the camera is the closest to the time at which the candidate image is output by the image sensor of the camera.
  • the time information of the sixth image is the same as the time information of the third image.
  • the time information of the third image is the same as the time information of the fifth image in the m first images.
  • the mobile phone may cache the sixth image with high image quality and high resolution in the second cache queue Buffer.
  • the second buffer queue can buffer multiple frames of the sixth image. In this way, when there is a user demand to filter out a frame of image from the video file as a wonderful image, the sixth image in the second cache queue that is closest to the time of this frame of image can be used as a guide image, and this frame of image Perform image quality enhancement (including image super-resolution) to obtain beautiful images with higher image quality.
  • the sixth image cached in the second cache queue is an HR image
  • the REF image is also an HR image. Therefore, the ninth image obtained by using the REF image to enhance the image quality of the candidate image is also an HR image.
  • the image quality enhancement described in S2504 may include image super-resolution.
  • the image super-resolution can be REF SR.
  • REF SR for the detailed description of the REF SR, reference may be made to the introduction in the foregoing embodiments, and details are not repeated here.
  • the image quality enhancement described in S2504 may include image fusion.
  • image fusion for a detailed description of image fusion, reference may be made to the introduction in the foregoing embodiments, and details are not repeated here.
  • the mobile phone uses the ISP to execute the fourth image on the ninth image to obtain the tenth image.
  • the fourth image includes image processing in the YUV domain, and the fourth processing does not include downsampling.
  • the mobile phone may use the ISP to process the first image to obtain a preview image, process the third image to obtain a sixth image, and process the ninth image to obtain a tenth image by means of time division multiplexing. That is to say, the mobile phone uses the ISP to process the first image to obtain the preview image, which will not affect the mobile phone to use the ISP to process the third image to obtain the sixth image, and process the ninth image to obtain the tenth image.
  • the tenth image is used to generate a wonderful image in the video recording process. In other words, the processing of the wonderful images shown in FIG. 26 by the mobile phone will not affect the processing of the preview images and video files shown in FIG. 26 by the mobile phone.
  • the mobile phone displays the exciting image obtained by encoding the tenth image.
  • FIG. 26 shows a schematic diagram of an image processing principle in a recording process provided by an embodiment of the present application.
  • the image sensor (Sensor) of the mobile phone is controlled by exposure, and can continuously output Bayer images (ie, the first image).
  • Bayer images ie, the first image.
  • the mobile phone may execute S2501 to process the third image to obtain the sixth image shown in FIG. 26 , and cache the sixth image in the second cache queue shown in FIG. 26 .
  • the mobile phone can generate a video file as shown in FIG. 26 .
  • the mobile phone may execute S2503 to receive the user's response to a frame of image in the video file, and determine the candidate image shown in FIG. 26 from the video file.
  • the mobile phone may execute S2504 to select the REF image shown in FIG. 26 from the second cache queue, and use the REF image to perform the image quality enhancement shown in FIG. 26 on the candidate image to obtain the ninth image.
  • the mobile phone may execute S2505 to process the ninth image to obtain the tenth image shown in FIG. 26 .
  • the mobile phone may use the encoder 2 shown in FIG. 26 to encode the tenth image information, and perform S2506 to display the encoded wonderful image.
  • the mobile phone performs S2506 to display the wonderful image obtained by encoding the tenth image.
  • the mobile phone executes S607 to display the exciting image obtained by encoding the fourth image. I won't go into details here.
  • the mobile phone implements the solutions described in S601-S607, which can obtain wonderful images during the video recording process of the mobile phone, and display the wonderful images after the video recording ends.
  • the solutions described in S601-S607 it is possible to realize the multi-shot online video recording of the mobile phone. More than one shot means that after the mobile phone video is over, not only the video file can be obtained, but also the wonderful images captured in the video can be obtained.
  • the mobile phone executes S601-S605 and S2501-S2506, which can obtain and cache multi-frame guide images that can be used to support the mobile phone to obtain wonderful images during the recording process; after that, the cached guide can be used during the playback of video files after the mobile phone records.
  • the image enhances the image quality of the candidate image selected by the user to obtain a wonderful image.
  • the mobile phone video recording function can be realized offline.
  • the mobile phone can notify that it supports the above-mentioned online scheme and offline scheme. That is to say, in some embodiments, the method in this embodiment of the application may include S601-S605, S606-S607/S2101-S2104, and S2501-S2506.
  • this embodiment of the present application introduces the "user's selection operation of an image in the video file" described in S2503, and the method for the mobile phone to determine a candidate image from the video file in response to the selection operation.
  • the implementation of the above-mentioned user's selection operation of a frame image in the video file can at least include the following methods (A)-mode (E):
  • the user's selection operation of a frame of image in the video file can be the user's click operation on the second preset control in the details page of the video file (such as single-click operation, double-click operation, or long-press operation, etc.) an operation).
  • the second preset control is used to trigger the mobile phone to automatically generate one or more frames of wonderful images in the video file.
  • the number of one or more frames of wonderful images in the video file is the same as the number of the sixth images buffered in the second buffer queue.
  • S2503 may include: in response to the user's click operation on the second preset control, the mobile phone may determine a frame of image corresponding to each frame of the sixth image cached in the second cache queue from the video file. Wherein, one frame of image corresponding to each frame of the sixth image is a candidate image.
  • the time information of the sixth image is the same as the time information of a frame of image corresponding to the sixth image.
  • the REF image corresponding to the candidate image in S2504 is specifically the sixth image corresponding to the candidate image.
  • the mobile phone in response to the user's click operation on the video file 1702 shown in (a) in FIG. 17 , as shown in (a) in FIG. 27 , the mobile phone can display the details page 2701 of the video file 1702.
  • the detail page 2701 includes a second preset control 2702 .
  • the list page 2703 of the wonderful images the list page 2703 includes two frames of wonderful images.
  • a frame of the sixth image is cached in the second cache queue
  • the details page 2704 of the wonderful image is displayed to the user.
  • the user's selection operation of a frame of image in the video file may be the user's click operation on the third preset control in the playback interface of the video file (such as click operation, any operation such as double-click operation or long-press operation).
  • the third preset control in the playback interface of the video file
  • the mobile phone displays a frame of image played by the mobile phone when the user clicks on the third preset control (or an image that is played before this frame of image and is separated from this frame of image by p frames) as an alternative image.
  • the mobile phone may receive the click operation of the play button on the details page of the video file shown in (a) in FIG. 28 .
  • the mobile phone can start to play the video file.
  • the mobile phone can display the playback interface of the video file shown in (b) in FIG. 28 or (c) in FIG. 28 .
  • the playback interface of the video file shown in (b) in FIG. 28 or (c) in FIG. 28 both includes a third preset control 2801 .
  • the embodiment of the present application does not limit the position of the third preset control 2801 in the playback interface of the video file.
  • the mobile phone can display the details page of the wonderful image shown in (c) in Figure 27 2704.
  • the mobile phone may display the prompt information 2901 shown in FIG. 29 .
  • the prompt information 2901 is used to request the user to confirm whether to save the wonderful image, for example, the prompt information 2901 may be "a wonderful image has been generated for you, please confirm whether to save it?".
  • the prompt message 2901 wonderful images can also be previewed.
  • the mobile phone In response to the user's click operation on the "Yes” button in the prompt information 2901, the mobile phone can save the wonderful image in the photo album; in response to the user's click operation on the "No” button in the prompt information 2901, the mobile phone will not save the wonderful image, and the mobile phone Continue to play video files.
  • the user's selection operation of a frame of image in the video file may include the operation of pausing playback input by the user during the process of playing the video file on the mobile phone and the operation of enlarging a frame of image after pausing the playback.
  • the above-mentioned operation of pausing playback may be the user's click operation on the pause playback button in the playback interface of the video file.
  • the above-mentioned operation of pausing playback may also be an adjustment operation of the user on the progress bar in the playback interface of the video file.
  • the mobile phone can display a frame of image in the video file corresponding to the current state of the progress bar.
  • the mobile phone may use a frame of image enlarged by the user as the candidate image.
  • a clip can contain multiple frames of images.
  • the mobile phone may select an image from the segment as a candidate image.
  • the method for the mobile phone to select a frame of image from the clip as a candidate image can refer to the method for selecting the second image from the first video stream in the above embodiment.
  • the candidate image may be a frame image with the highest clarity and completeness in the segment.
  • the user's selection operation of a frame of image in the video file may include the operation of pausing playback input by the user during the process of playing the video file by the mobile phone.
  • the above-mentioned operation of pausing playback may be the user's click operation on the pause playback button in the playback interface of the video file.
  • the above-mentioned operation of pausing playback may also be an adjustment operation of the user on the progress bar in the playback interface of the video file.
  • the mobile phone can display a frame of image in the video file corresponding to the current state of the progress bar.
  • the operation of pausing playback is the user's adjustment operation on the progress bar.
  • the user can adjust the progress bar in the video file playback interface to control the mobile phone to display a frame of image in the video file that the user is interested in.
  • the mobile phone can receive the user's adjustment operation on the return progress bar, and display the images in the video file according to the change of the progress bar.
  • the mobile phone can take the user's finger from the progress bar, and the progress bar corresponds to a frame of image in the video file as an alternative image.
  • the user's selection operation of a frame image in the video file and the method for the mobile phone to determine a candidate image from the video file in response to the selection operation include but are not limited to mode (A)-mode (The method shown in D) and other methods will not be described here in this embodiment of the application.
  • the electronic device may include: the above-mentioned display screen, a camera, a memory, and one or more processors.
  • the display screen, camera, memory and processor are coupled.
  • the memory is used to store computer program code comprising computer instructions.
  • the processor executes the computer instructions, the electronic device can execute various functions or steps performed by the mobile phone in the foregoing method embodiments.
  • the structure of the electronic device reference may be made to the structure of the mobile phone shown in FIG. 4 .
  • the chip system 3000 includes at least one processor 3001 and at least one interface circuit 3002 .
  • the processor 3001 and the interface circuit 3002 can be interconnected through lines.
  • interface circuit 3002 may be used to receive signals from other devices, such as memory of an electronic device.
  • the interface circuit 3002 may be used to send signals to other devices (such as the processor 3001).
  • the interface circuit 3002 can read instructions stored in the memory, and send the instructions to the processor 3001 .
  • the electronic device may be made to execute various steps in the foregoing embodiments.
  • the chip system may also include other discrete devices, which is not specifically limited in this embodiment of the present application.
  • the embodiment of the present application also provides a computer storage medium, the computer storage medium includes computer instructions, and when the computer instructions are run on the above-mentioned electronic device, the electronic device is made to perform the various functions or steps performed by the mobile phone in the above-mentioned method embodiment .
  • the embodiment of the present application also provides a computer program product, which, when the computer program product is run on a computer, causes the computer to execute each function or step performed by the mobile phone in the method embodiment above.
  • the disclosed devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium.
  • the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • aforementioned storage medium comprises: various mediums that can store program codes such as U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Television Signal Processing For Recording (AREA)
  • Studio Devices (AREA)

Abstract

一种获取图像的方法及电子设备，涉及拍摄技术领域，可自动选择出录像中的精彩图像并呈现给用户。该方案包括：电子设备接收用户的第一操作；响应于第一操作显示第一界面(即正在录制视频的取景界面)，第一界面包括预览图像；从第一视频流中选择出第二图像(即包括目标对象的图像)，第一视频流是ISP对多帧第一图像执行包括下采样的第一处理得到的；在第一缓存队列缓存第一图像；将第一缓存队列缓存的第一图像中与第二图像对应的m帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像；采样ISP对第三图像执行不包括下采样的第二处理得到第四图像；在录像结束后，展示对第四图像进行编码得到的精彩图像。

Description

一种获取图像的方法及电子设备
本申请要求于2021年09月07日提交国家知识产权局、申请号为202111044551.4、发明名称为“一种视频中选取图像的方法及电子设备”的中国专利申请的优先权，以及于2021年12月29日提交国家知识产权局、申请号为202111649415.8、发明名称为“一种获取图像的方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及拍摄技术领域，尤其涉及一种获取图像的方法及电子设备。
背景技术
现有的手机一般具有拍照和录像功能，越来越来的人使用手机拍摄照片和视频来记录生活的点点滴滴。其中，手机录制视频(即录像)的过程中，可能会采集到的一些精彩的画面。在手机录像的过程中，用户可能会希望手机可以截取上述精彩的画面，并保存成照片展示给用户。因此，亟待一种可以选择录像中精彩图像并呈现给用户的方案。
发明内容
本申请提供一种获取图像的方法及电子设备，可以自动选择出录像中的精彩图像并呈现给用户。
为达到上述目的，本申请的实施例采用如下技术方案：
第一方面，本申请提供一种获取图像的方法，该方法可以应用于电子设备。该方法中，电子设备可接收用户的第一操作，该第一操作用于触发开始录制视频。响应于第一操作，电子设备可显示第一界面。该第一界面是电子设备正在录制视频的取景界面，该第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的。电子设备可以从第一视频流中选择出第二图像。该第二图像包括目标对象的图像，该第一视频流是电子设备的图像信号处理器(image signal processor，ISP)对多帧第一图像执行第一处理得到的，第一处理包括下采样。电子设备可以在第一缓存队列缓存摄像头采集的第一图像。该第一缓存队列可缓存摄像头采集的n帧第一图像，n≥1，n为整数。之后，电子设备可以将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像。该预设RAW域图像处理算法具备提升图像画质的功能。m≥1，m为整数。然后，电子设备可以采用ISP对第三图像执行第二处理得到第四图像，该第二处理不包括下采样。最后，电子设备可以在录像结束后，展示对第四图像进行编码得到的精彩图像。
本申请中，电子设备(如电子设备)的图像传感器(Sensor)曝光输出的图像可以由ISP处理得到视频流(即第一数据流)。电子设备可以从ISP处理后的视频流中选择出备选帧(即第二图像，即备选的精彩帧)。并且，电子设备可以将Sensor曝光输出图像缓存在一个第一缓存队列(Buffer)中。该第一缓存队列可以缓存多帧图像。 如此，即使从“Sensor输出备选帧(即第二图像)对应的图像”到“选择出备选帧”存在延迟，该备选帧对应的图像也可以缓存第一缓存队列中。如此，电子设备则可以从第一缓存队列中得到备选帧对应的图像。
并且，电子设备还可以采用预设RAW域图像处理算法处理于第二图像对应的第一图像，有助于提升精彩图像的画质。其中，预览图像和录像文件对应的视频流中的图像LR是经过下采样的。而ISP处理第三图像得到第四图像得到的是未经过下采样的图像HR；因此，编码图像HR得到的精彩图像是高分辨率的图像。
综上所述，采用本方案，电子设备可以自动选择出录像中的精彩图像并呈现给用户，并且可以提升上述精彩图像的图像质量(如分辨率和画质)。
在第一方面的一种可能的设计方式中，上述电子设备在录像结束后，展示对第四图像进行编码得到的精彩图像，包括：电子设备录像结束后，响应于用户的操作显示录像文件的详情页，录像文件的详情页包括第一预设控件；电子设备响应于用户对第一预设控件的点击操作，展示电子设备录制录像文件的过程中抓拍的精彩图像。该设计方式中，手机可以通过在录像文件的详情页提供显性控件(即第一预设控件)的方式，为用户提供触发电子设备展示录制录像文件的过程中抓拍的精彩图像的入口。
在第一方面的另一种可能的设计方式中，电子设备在录像结束后，展示对第四图像进行编码得到的精彩图像，包括：电子设备录像结束后，响应于用户的操作显示录像文件的详情页；电子设备响应于用户在录像文件的详情页输入的预设手势，展示电子设备录制录像文件的过程中抓拍的精彩图像。该设计方式中，手机可以通过在录像文件的详情页以无显性控件的方式，为用户提供触发电子设备展示录制录像文件的过程中抓拍的精彩图像的入口。
在第一方面的另一种可能的设计方式中，n≥2。换言之，第一缓存队列中可以缓存多帧第一图像。如此，即使从“Sensor输出备选帧(即第二图像)对应的图像”到“选择出备选帧”存在延迟(如120ms-160ms)；在这段延迟时长内Sensor出帧都可以缓存在Buffer中。因此，电子设备选择出备选帧(即第二图像)时，Sensor输出的Bayer图像也可以缓存在第一缓存队列中。并且，短时间内Sensor出帧的图像内容不会发生太大变化。如此，可以由电子设备的根据Buffer中缓存的图像的时间信息，从Buffer中选择出第二图像对应的m帧第一图像。
在第一方面的另一种可能的设计方式中，电子设备将n帧第一图像中与第二图像对应的k帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像，可以包括：从n帧第一图像选择出与第二图像对应的第五图像；将n帧第一图像中的m帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像；其中，m帧第一图像包括第五图像；m帧第一图像中除第五图像的其他图像，用于对第五图像进行画质增强。
其中，在n帧第一图像中，第五图像由摄像头的图像传感器输出的时间与第二图像由摄像头的图像传感器输出的时间最近。这样，可以从第一缓存队列中选择出与第二图像的图像内容最接近的一帧图像。或者，n帧第一图像中，第五图像的清晰度最高。这样，可以从第一缓存队列中选择出图像质量最高的一帧图像。
在第一方面的另一种可能的设计方式中，上述目标对象包括人脸，第二图像为目 标对象的表情满足预设条件的图像，目标对象的表情满足预设条件包括人脸微笑。在该设计方式中，电子设备可以在录像中抓拍到人脸微笑的精彩图像。
在第一方面的另一种可能的设计方式中，上述目标对象包括人体，第二图像为目标对象的动作为预设动作的图像。例如，该预设动作可以为人体跳跃至最高点时的人体动作、舞蹈表演或者运动竞技表演过程中的精彩动作，人体运动锻炼过程中的高难度动作等。也就是说，第二图像可以是第一视频流中包括人体摆出预设动作时的一帧图像。在该实例中，电子设备可以在录像过程中抓拍到人体运动过程中摆出预设动作时的精彩图像。
在第一方面的另一种可能的设计方式中，上述目标对象在所述电子设备出厂时，预先配置在所述电子设备中。和/或，目标对象是电子设备接收用户在所述电子设备的设置界面设置的。
在第一方面的另一种可能的设计方式中，电子设备为用户提供在电子设备还未开始录制视频的取景界面设置目标对象的功能。
具体的，在电子设备从第一视频流中选择出第二图像之前，电子设备可以显示第二界面。该第二界面是电子设备还未开始录制视频的取景界面，第二界面包括的预览图像是电子设备接收到第一操作前由电子设备的摄像头采集的图像得到的；第二界面还包括开始录像按钮和预设功能的开关；第一操作是对开始录像按钮的点击操作。电子设备响应于用户对第二界面中预设功能的开关的第二操作，显示目标对象的设置界面，目标对象的设置界面包括多个待选择的拍摄对象的选项。电子设备响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
在第一方面的另一种可能的设计方式中，电子设备为用户提供在电子设备已开始录制视频的取景界面(如上述第一界面)设置目标对象的功能。第一界面还包括：预设功能的开关。在电子设备从第一视频流中选择出第二图像之前，电子设备可以响应于用户对第一界面中预设功能的开关的第二操作，显示目标对象的设置界面，目标对象的设置界面包括多个待选择的拍摄对象的选项；电子设备响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
在第一方面的另一种可能的设计方式中，电子设备可以周期性地从第一视频流中选择出第二图像。
在第一方面的另一种可能的设计方式中，m≥2。具体的，电子设备可以将该多帧第一图像作为输入，运行预设RAW域图像处理算法。应理解，m帧第一图像中除第五图像之外的其他图像，可以对第五图像起到画质增强的作用，有利于获取噪声和纹理等信息，可以进一步提升第五图像的画质。
在第一方面的另一种可能的设计方式中，上述第一处理包括RAW域的图像处理和RGB域的图像处理，第二处理包括RAW域的图像处理、RGB域的图像处理和YUV域的图像处理。
在第一方面的另一种可能的设计方式中，若第一缓存队列中包括时间信息与第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，电子设备可以采用 ISP对第三图像执行第二处理得到第四图像。
由于第一缓存队列的缓存空间有限；因此，第一缓存队列中可能并未缓存时间信息与第二图像的时间信息相同的第一图像。换言之，第一缓存队列中缓存的第一图像的时间信息与第二图像的时间信息可能相差较多。在这种情况下，第一缓存队列中缓存的第一图像的图像内容与第二图像的图像内容可能会存在较大差异。
如果第一缓存队列中包括时间信息与第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，则表示从第一缓存队列中可以找到与Sensor输出第二图像的时间较为接近的第一图像。这种情况下，电子设备才采用ISP对第三图像执行第二处理得到第四图像，可以准确抓拍到与第二图像对应的精彩图像。
在第一方面的另一种可能的设计方式中，若第一缓存队列中不包括时间信息与第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，电子设备则可以执行第二方面的方法。第二方面所述的方法的详细介绍和效果分析本申请这里不予赘述。
在第一方面的另一种可能的设计方式中，电子设备不仅可以在线生成精彩图像，还可以离线生成精彩图像。上述第三处理可以不包括下采样。具体的，本申请的方法还可以包括：电子设备将第六图像缓存在第二缓存队列中；电子设备响应于结束录像的操作，生成录像文件；其中，录像文件包括由多帧第一图像生成的图像；电子设备播放录像文件的过程中，响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像；电子设备从第二缓存队列缓存的多帧第六图像中选择与备选图像对应的参考REF图像，并采用REF图像对备选图像进行画质增强，得到第九图像。电子设备采用ISP对第九图像执行第四处理得到第十图像，第四处理包括YUV域的图像处理，第四处理不包括下采样；电子设备展示对第十图像进行编码得到的精彩图像。该设计方式的详细介绍和效果分析，可以参考本申请对第三方面所述的方法的详细介绍和效果分析，本申请这里不予赘述。
第二方面，本申请提供一种获取图像的方法，该方法可以应用于电子设备。该方法中，电子设备可接收用户的第一操作，该第一操作用于触发开始录制视频。响应于第一操作，电子设备可显示第一界面。该第一界面是电子设备正在录制视频的取景界面，该第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的。电子设备可以从第一视频流中选择出第二图像。该第二图像包括目标对象的图像，该第一视频流是电子设备的图像信号处理器(image signal processor，ISP)对多帧第一图像执行第一处理得到的，第一处理包括下采样。电子设备可以在第一缓存队列缓存摄像头采集的第一图像。该第一缓存队列可缓存摄像头采集的n帧第一图像，n≥1，n为整数。之后，电子设备可以将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像。该预设RAW域图像处理算法具备提升图像画质的功能。m≥1，m为整数。电子设备采用ISP对第三图像执行第三处理得到第六图像；其中，第三处理包括RAW域的图像处理和RGB域的图像处理。电子设备采用第六图像对第二图像进行画质增强，得到第七图像。电子设备采用ISP对第七图像执行第四处理得到第八图像，第四处理包括YUV域的图像处理，第四处理不包括下采样。电子设备在录像结束后，展示对第八图像进行编码得到的精彩图像。
在第二方面的一种可能的设计方式中，上述第一处理包括RAW域的图像处理和RGB域的图像处理。
在第二方面的另一种可能的设计方式中，上述电子设备采用第六图像对第二图像进行的画质增强包括：图像融合和/或图像超分辨。其中，图像超分辨包括基于参考帧的图像超分辨REF-SR。
第三方面，本申请提供一种获取图像的方法，该方法可以应用于电子设备。该方法中，电子设备可接收用户的第一操作，该第一操作用于触发开始录制视频。响应于第一操作，电子设备可显示第一界面。该第一界面是电子设备正在录制视频的取景界面，该第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的。电子设备可以从第一视频流中选择出第二图像。该第二图像包括目标对象的图像，该第一视频流是电子设备的图像信号处理器(image signal processor，ISP)对多帧第一图像执行第一处理得到的，第一处理包括下采样。电子设备可以在第一缓存队列缓存摄像头采集的第一图像。该第一缓存队列可缓存摄像头采集的n帧第一图像，n≥1，n为整数。之后，电子设备可以将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像。该预设RAW域图像处理算法具备提升图像画质的功能。m≥1，m为整数。电子设备采用ISP对第三图像执行第三处理得到第六图像，并将第六图像缓存在第二缓存队列中，第三处理包括RAW域的图像处理和RGB域的图像处理，第三处理不包括下采样。电子设备响应于结束录像的操作，生成录像文件；其中，录像文件包括由多帧第一图像生成的图像。电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像。电子设备从第二缓存队列缓存的多帧第六图像中选择与备选图像对应的参考REF图像，并采用REF图像对备选图像进行画质增强，得到第九图像。电子设备采用ISP对第九图像执行第四处理得到第十图像，第四处理包括YUV域的图像处理，第四处理不包括下采样。电子设备展示对第十图像进行编码得到的精彩图像。
一方面，第三处理不包括下采样。因此，缓存在第二缓存队列中的第六图像是高分辨率的HR图像。并且，该第六图像是经过预设RAW域图像处理算法处理过的图像细节较为丰富、图像质量较高的图像。
另一方面，上述第六图像是基于第二图像生成的。第二图像是录像文件中包括目标对象的图像。由此可见，该第六图像中也包括目标对象的图像。如此，电子设备录像结束后，用户想要从录像文件中筛选出第六图像对应的一帧图像作为精彩图像的可能性较大。
因此，本申请中，电子设备可以将多帧第六图像缓存在第二缓存队列中，用于当出现从录像文件中筛选出一帧图像作为精彩图像的用户需求时，可以采用第二缓存队列中的第六图像对这一帧图像进行画质增强(包括图像超分辨)，以得到图像质量较高(包括分辨率较高)的精彩图像。
在第三方面的一种可能的设计方式中，上述录像文件的详情页包括第二预设控件。对录像文件中一帧图像的选择操作为对第二预设控件的点击操作。
上述电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备响应于用户对录像文件的详情页中第二预设控件的点击操 作，从录像文件中选择与第二缓存队列中缓存的每一帧第六图像对应的图像，作为备选图像。
该设计方式中，电子设备可以在录像文件的详情页中以显性控件(如第二预设控件)的方式，为用户提供触发电子设备生成并展示录像文件中精彩图像的功能。
在第三方面的另一种可能的设计方式中，上述录像文件的播放界面包括第三预设控件。对录像文件中一帧图像的选择操作为对第三预设控件的点击操作。
上述电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备在播放录像文件的过程中，响应于用户对录像文件的播放界面中第三预设控件的点击操作，将用户点击第三预设控件时电子设备播放的一帧图像作为备选图像。
该设计方式中，电子设备可以在录像文件的播放界面中以显性控件(如第二预设控件)的方式，为用户提供触发电子设备生成并展示录像文件中精彩图像的功能。
在第三方面的另一种可能的设计方式中，上述电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备在播放录像文件的过程中，响应于暂停播放的操作和放大图像的操作，将放大图像的操作对应的一帧图像作为备选图像。
其中，放大图像的操作用于触发电子设备放大电子设备暂停播放录像文件时，电子设备所显示的录像文件中的一帧图像。
该设计方式中，电子设备播放录像文件的过程中，当电子设备播放到用户感兴趣的图像，用户则可以点击暂停按钮控制手机暂停播放录像文件，然后用户可以控制电子设备放大当前帧图像。电子设备响应于上述暂停播放的操作和放大图像的操作，可以将用户放大的一帧图像作为备选图像用于生成精彩图像。
在第三方面的另一种可能的设计方式中，上述电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备在播放录像文件的过程中，接收用户对录像文件的播放界面中进度条的调整操作，根据进度条的变化显示录像文件中的图像；电子设备检测到用户手指离开进度条后，将用户手指离开进度条时进度条对应录像文件中的一帧图像作为备选图像。
该设计方式中，电子设备播放录像文件的过程中，当电子设备播放到用户感兴趣的图像，用户则可以通过进度条控制电子设备显示录像文件中用户感兴趣的一帧图像。当电子设备检测到用户手指离开进度条后，电子设备可以将用户手指离开进度条时，该进度条对应录像文件中的一帧图像作为备选图像用于生成精彩图像。
在第三方面的另一种可能的设计方式中，上述电子设备响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备在播放录像文件的过程中，响应于用户反复查看录像文件中一个片段的操作，从一个片段中选择一帧图像作为备选图像。
其中，如果用户反复查看录像文件中的一个片段，则表示用户对该片段的图像感兴趣。这种情况下，电子设备从该片段中选择一帧图像作为备选图像用于生成精彩图像，可得到符合用户需求的精彩图像。
在第三方面的另一种可能的设计方式中，上述电子设备响应于用户对录像文件中 一帧图像的选择操作，从录像文件中确定备选图像，包括：电子设备在播放录像文件的过程中，接收用户对录像文件的播放界面中进度条的调整操作，根据进度条的变化显示录像文件中的图像；电子设备检测到用户手指停留在进度条上，进度条保持同一位置，记录所述进度条保持同一位置的持续时间，当所述进度条保持同一位置的持续时间大于预设时间阈值时，电子设备将所述进度条对应录像文件中的一帧图像作为备选图像；或者，电子设备检测到用户手指停留在进度条上，录像文件的播放界面保持显示同一帧图像，记录录像文件的播放界面保持显示同一帧图像的持续时间，当录像文件的播放界面保持显示同一帧图像的持续时间大于预设时间阈值时，将录像文件中的对应一帧图像作为备选图像。
在第三方面的另一种可能的设计方式中，上述从一个片段中选择一帧图像作为备选图像，包括：电子设备选择一个片段中清晰度最高的一帧图像作为备选图像。
第四方面，本申请提供一种电子设备，该电子设备包括触摸屏、存储器、显示屏、一个或多个摄像头和一个或多个处理器。该存储器、显示屏、摄像头与处理器耦合。其中，摄像头用于采集图像，显示屏用于显示摄像头采集的图像或者处理器生成的图像，存储器中存储有计算机程序代码，计算机程序代码包括计算机指令，当计算机指令被处理器执行时，使得电子设备执行如第一方面、第二方面或第三方面及其任一种可能的设计方式所述的方法。
第五方面，本申请提供一种电子设备，该电子设备包括触摸屏、存储器、显示屏、一个或多个摄像头和一个或多个处理器。该存储器、显示屏、摄像头与处理器耦合。其中，摄像头用于采集图像，显示屏用于显示摄像头采集的图像或者处理器生成的图像，存储器中存储有计算机程序代码，计算机程序代码包括计算机指令。当计算机指令被处理器执行时，使得电子设备执行下步骤：接收用户的第一操作；其中，第一操作用于触发开始录制视频；响应于第一操作，显示第一界面；其中，第一界面是电子设备正在录制视频的取景界面，第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的；从第一视频流中选择出第二图像；其中，第二图像包括目标对象的图像；第一视频流是电子设备的ISP对多帧第一图像执行第一处理得到的，第一处理包括下采样；在第一缓存队列缓存摄像头采集的第一图像；其中，第一缓存队列缓存摄像头采集的n帧第一图像，n≥1，n为整数；将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；采用ISP对第三图像执行第二处理得到第四图像；其中，第二处理不包括下采样；在录像结束后，展示对第四图像进行编码得到的精彩图像。
在第五方面的一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：录像结束后，响应于用户的操作显示录像文件的详情页，录像文件的详情页包括第一预设控件；响应于用户对第一预设控件的点击操作，展示电子设备录制录像文件的过程中抓拍的精彩图像。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：录像结束后，响应于用户的操作显示录像文件的详情页；响应于用户在录像文件的详情页输入的预设手势，展示电子设备录制录像文件的过程中抓拍 的精彩图像。
在第五方面的另一种可能的设计方式中，n≥2。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：从n帧第一图像选择出与第二图像对应的第五图像；将n帧第一图像中的m帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像；其中，m帧第一图像包括第五图像；m帧第一图像中除第五图像的其他图像，用于对第五图像进行画质增强。
其中，在n帧第一图像中，第五图像由摄像头的图像传感器输出的时间与第二图像由摄像头的图像传感器输出的时间最近；或者，n帧第一图像中，第五图像的清晰度最高。
在第五方面的另一种可能的设计方式中，上述目标对象包括人脸，第二图像为目标对象的表情满足预设条件的图像，目标对象的表情满足预设条件包括人脸微笑；和/或，目标对象包括人体，第二图像为目标对象的动作为预设动作的图像。
在第五方面的另一种可能的设计方式中，上述目标对象在电子设备出厂时，预先配置在电子设备中；和/或，目标对象是电子设备接收用户在电子设备的设置界面设置的。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：上述在电子设备从第一视频流中选择出第二图像之前，显示第二界面；其中，第二界面是电子设备还未开始录制视频的取景界面，第二界面包括的预览图像是电子设备接收到第一操作前由电子设备的摄像头采集的图像得到的；第二界面还包括开始录像按钮和预设功能的开关；第一操作是对开始录像按钮的点击操作；响应于用户对第二界面中预设功能的开关的第二操作，显示目标对象的设置界面，目标对象的设置界面包括多个待选择的拍摄对象的选项；响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
在第五方面的另一种可能的设计方式中，上述第一界面还包括：预设功能的开关。当计算机指令被处理器执行时，使得电子设备执行下步骤：在电子设备从第一视频流中选择出第二图像之前，响应于用户对第一界面中预设功能的开关的第二操作，显示目标对象的设置界面，目标对象的设置界面包括多个待选择的拍摄对象的选项；响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：周期性地从第一视频流中选择出第二图像。
在第五方面的另一种可能的设计方式中，m≥2。
在第五方面的另一种可能的设计方式中，第一处理包括RAW域的图像处理和RGB域的图像处理，第二处理包括RAW域的图像处理、RGB域的图像处理和YUV域的图像处理。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：若第一缓存队列中包括时间信息与第二图像的时间信息所指示的 时间的差值在预设时长内的第一图像，采用ISP对第三图像执行第二处理得到第四图像。
在第五方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：若第一缓存队列中不包括时间信息与第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，采用ISP对第三图像执行第三处理得到第六图像；其中，第三处理包括RAW域的图像处理和RGB域的图像处理；采用第六图像对第二图像进行画质增强，得到第七图像；采用ISP对第七图像执行第四处理得到第八图像，第四处理包括YUV域的图像处理，第四处理不包括下采样；在录像结束后，展示对第八图像进行编码得到的精彩图像。
在第五方面的另一种可能的设计方式中，上述第三处理不包括下采样。当计算机指令被处理器执行时，使得电子设备执行下步骤：将第六图像缓存在第二缓存队列中；响应于结束录像的操作，生成录像文件；其中，录像文件包括由多帧第一图像生成的图像；播放录像文件的过程中，响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像；从第二缓存队列缓存的多帧第六图像中选择与备选图像对应的参考REF图像，并采用REF图像对备选图像进行画质增强，得到第九图像；采用ISP对第九图像执行第四处理得到第十图像，第四处理包括YUV域的图像处理，第四处理不包括下采样；展示对第十图像进行编码得到的精彩图像。
第六方面，本申请提供一种电子设备，该电子设备包括触摸屏、存储器、显示屏、一个或多个摄像头和一个或多个处理器。该存储器、显示屏、摄像头与处理器耦合。其中，摄像头用于采集图像，显示屏用于显示摄像头采集的图像或者处理器生成的图像，存储器中存储有计算机程序代码，计算机程序代码包括计算机指令。当计算机指令被处理器执行时，使得电子设备执行下步骤：接收用户的第一操作；其中，第一操作用于触发开始录制视频；响应于第一操作，显示第一界面；其中，第一界面是电子设备正在录制视频的取景界面，第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的；从第一视频流中选择出第二图像；其中，第二图像包括目标对象的图像；第一视频流是电子设备的ISP对多帧第一图像执行第一处理得到的，第一处理包括下采样；在第一缓存队列缓存摄像头采集的第一图像；其中，第一缓存队列缓存摄像头采集的n帧第一图像，n≥1，n为整数；将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；采用ISP对第三图像执行第三处理得到第六图像；其中，第三处理包括RAW域的图像处理和RGB域的图像处理；采用第六图像对第二图像进行画质增强，得到第七图像；采用ISP对第七图像执行第四处理得到第八图像，第四处理包括YUV域的图像处理，第四处理不包括下采样；在录像结束后，展示对第八图像进行编码得到的精彩图像。
在第六方面的另一种可能的设计方式中，上述第一处理包括RAW域的图像处理和RGB域的图像处理。
在第六方面的另一种可能的设计方式中，上述电子设备采用第六图像对第二图像进行的画质增强包括：图像融合和/或图像超分辨。其中，图像超分辨包括基于参考帧 的图像超分辨REF-SR。
第七方面，本申请提供一种电子设备，该电子设备包括触摸屏、存储器、显示屏、一个或多个摄像头和一个或多个处理器。该存储器、显示屏、摄像头与处理器耦合。其中，摄像头用于采集图像，显示屏用于显示摄像头采集的图像或者处理器生成的图像，存储器中存储有计算机程序代码，计算机程序代码包括计算机指令。当计算机指令被处理器执行时，使得电子设备执行下步骤：接收用户的第一操作；其中，第一操作用于触发开始录制视频；响应于第一操作，显示第一界面；其中，第一界面是电子设备正在录制视频的取景界面，第一界面包括预览图像，预览图像是电子设备接收到第一操作后由电子设备的摄像头采集的第一图像得到的；从第一视频流中选择出第二图像；其中，第二图像包括目标对象的图像；第一视频流是电子设备的ISP对多帧第一图像执行第一处理得到的，第一处理包括下采样；在第一缓存队列缓存摄像头采集的第一图像；其中，第一缓存队列缓存摄像头采集的n帧第一图像，n≥1，n为整数；将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；采用ISP对第三图像执行第三处理得到第六图像，并将第六图像缓存在第二缓存队列中，第三处理包括RAW域的图像处理和RGB域的图像处理，第三处理不包括下采样；响应于结束录像的操作，生成录像文件；其中，录像文件包括由多帧第一图像生成的图像；响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像；从第二缓存队列缓存的多帧第六图像中选择与备选图像对应的参考REF图像，并采用REF图像对备选图像进行画质增强，得到第九图像；采用ISP对第九图像执行第四处理得到第十图像，第四处理包括YUV域的图像处理，第四处理不包括下采样；展示对第十图像进行编码得到的精彩图像。
在第七方面的一种可能的设计方式中，上述录像文件的详情页包括第二预设控件；对录像文件中一帧图像的选择操作为对第二预设控件的点击操作。
当计算机指令被处理器执行时，使得电子设备执行下步骤：响应于用户对录像文件的详情页中第二预设控件的点击操作，从录像文件中选择与第二缓存队列中缓存的每一帧第六图像对应的图像，作为备选图像。
在第七方面的另一种可能的设计方式中，上述录像文件的播放界面包括第三预设控件。对录像文件中一帧图像的选择操作为对第三预设控件的点击操作。
当计算机指令被处理器执行时，使得电子设备执行下步骤：在播放录像文件的过程中，响应于用户对录像文件的播放界面中第三预设控件的点击操作，将用户点击第三预设控件时电子设备播放的一帧图像作为备选图像。
在第七方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：在播放录像文件的过程中，响应于暂停播放的操作和放大图像的操作，将放大图像的操作对应的一帧图像作为备选图像。
其中，放大图像的操作用于触发电子设备放大电子设备暂停播放录像文件时，电子设备所显示的录像文件中的一帧图像。
在第七方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：在播放录像文件的过程中，接收用户对录像文件的播放界面中进 度条的调整操作，根据进度条的变化显示录像文件中的图像；检测到用户手指离开进度条后，将用户手指离开进度条时进度条对应录像文件中的一帧图像作为备选图像。
在第七方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：在播放录像文件的过程中，接收用户对录像文件的播放界面中进度条的调整操作，根据进度条的变化显示录像文件中的图像；检测到用户手指停留在进度条上，进度条保持同一位置，记录所述进度条保持同一位置的持续时间，当进度条保持同一位置的持续时间大于预设时间阈值时，将进度条对应录像文件中的一帧图像作为备选图像；或者，检测到用户手指停留在进度条上，录像文件的播放界面保持显示同一帧图像，记录录像文件的播放界面保持显示同一帧图像的持续时间，当录像文件的播放界面保持显示同一帧图像的持续时间大于预设时间阈值时，将录像文件中的对应一帧图像作为备选图像。
在第七方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：在播放录像文件的过程中，响应于用户反复查看录像文件中一个片段的操作，从一个片段中选择一帧图像作为备选图像。
在第七方面的另一种可能的设计方式中，当计算机指令被处理器执行时，使得电子设备执行下步骤：选择一个片段中清晰度最高的一帧图像作为备选图像。
第八方面，本申请提供一种计算机可读存储介质，该计算机可读存储介质包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备执行如第一方面或第二方面及其任一种可能的设计方式所述的方法。
第九方面，本申请提供一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得该计算机执行如第一方面或第二方面及任一种可能的设计方式所述的方法。该计算机可以是上述电子设备。
可以理解地，上述提供的第四方面至第七方面及其任一种可能的设计方式所述的电子设备，第八方面所述的计算机存储介质，第九方面所述的计算机程序产品所能达到的有益效果，可参考第一方面和第二方面及其任一种可能的设计方式中的有益效果，此处不再赘述。
附图说明
图1为一种录像过程中的图像处理流程图；
图2为另一种录像过程中的图像处理流程图；
图3为本申请实施例提供的一种获取图像的方法原理框图；
图4为本申请实施例提供的一种手机的硬件结构示意图；
图5为本申请实施例提供的一种手机的软件架构示意图；
图6为本申请实施例提供的一种获取图像的方法流程图；
图7为本申请实施例提供的一种手机录像的显示界面示意图；
图8A为本申请实施例提供的另一种获取图像的方法原理框图；
图8B为本申请实施例提供的另一种获取图像的方法原理框图；
图9为本申请实施例提供的一种第一缓存队列Buffer的示意图；
图10为本申请实施例提供的另一种手机录像的显示界面示意图；
图11为本申请实施例提供的另一种手机录像的显示界面示意图；
图12为本申请实施例提供的另一种手机录像的显示界面示意图；
图13为本申请实施例提供的另一种手机录像的显示界面示意图；
图14为本申请实施例提供的另一种手机录像的显示界面示意图；
图15为本申请实施例提供的另一种获取图像的方法流程图；
图16为本申请实施例提供的另一种手机录像的显示界面示意图；
图17为本申请实施例提供的另一种手机录像的显示界面示意图；
图18为本申请实施例提供的另一种手机录像的显示界面示意图；
图19为本申请实施例提供的另一种手机录像的显示界面示意图；
图20为本申请实施例提供的另一种手机录像的显示界面示意图；
图21为本申请实施例提供的另一种获取图像的方法流程图；
图22为本申请实施例提供的另一种获取图像的方法原理框图；
图23为本申请实施例提供的另一种获取图像的方法原理框图；
图24为本申请实施例提供的另一种获取图像的方法原理框图；
图25为本申请实施例提供的另一种获取图像的方法流程图；
图26为本申请实施例提供的另一种获取图像的方法原理框图；
图27为本申请实施例提供的另一种手机录像的显示界面示意图；
图28为本申请实施例提供的另一种手机录像的显示界面示意图；
图29为本申请实施例提供的另一种手机录像的显示界面示意图；
图30为本申请实施例提供的一种芯片系统的结构示意图。
具体实施方式
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
目前，电子设备录像过程中，电子设备的图像传感器(Sensor)受到曝光的控制，可以不断输出图像。每一帧图像经过电子设备的图像信号处理器(image signal processor，ISP)或图像信号处理算法处理，然后经过编码器(ENCODER)进行编码，便可以得到视频文件。以手机为代表的电子设备，其图像传感器输出的原始图像通常为拜耳(Bayer)格式图像，也有部分图像传感器可以输出RGGB、RGBW、CMYK、RYYB、CMY等格式图像。本申请实施例中，以手机的图像传感器输出Bayer格式图像为例进行描述。需要注意的是，输出RGGB、RGBW、CMYK、RYYB、CMY等格式图像的图像传感器以及搭载所述图像传感器的其他电子设备也适用于本申请实施例提供的技术方案。
其中，RGGB为(red green green blue)，RGBW为(red green blue white)，CMYK为(cyan magenta yellow black)，RYYB为(red yellow yellow blue)，CMY为(cyan magenta yellow)。
请参考图1或图2，其示出手机录像过程中预览图像和录像文件的处理流程。其中，预览图像是指手机录像过程中最终用于在显示屏上呈现给用户的图像，录像文件是指最终用于录像结束后以视频文件的格式保存在手机中可供用户查看的视频流。
如图1或图2所示，手机的ISP处理图像可以分为三个图像格式域的处理：RAW 域的图像处理、RGB域的图像处理和YUV域的图像处理。
RAW域的图像处理可以包括：黑电平(black level correction，BLC)纠正、线性纠正(Linearizaton)、镜头阴影纠正(lens shading correction，LSC)、坏点修复(defectpixel correction，DPC)、RAW降噪(Denoise)、自动白平衡(automatic white balance，AWB)、绿通道平衡(green imbalance，GIC)、去色差(CAC)等处理。
RGB域的图像处理可以包括：去马赛克(Demosiac)、色彩纠正CC、动态范围压缩(dynamic range control，DRC)、Gamma校正、RGB2YUV(RGB格式转换为YUV格式)。
YUV域的图像处理可以包括：UV下采样、色彩增强CE、空间域降噪YUVNF、色彩管理3DLUT、锐化Sharpness、缩放Scalar。
需要说明的是，ISP中“RAW域”、“RGB域”和“YUV域”的划分包括但不限于上述划分方式。例如，去马赛克(Demosiac)还可以包括在“RAW域”中。本申请实施例对此不作限制。
在一种实现方式中，如图1所示，图像传感器(Sensor)输出图像后，可以由ISP对图像进行“RAW域”、“RGB域”和“YUV域”的图像处理；在“YUV域”图像处理后，可以分为两路数据流。一路数据流采用图1所示的处理算法1进行处理，然后由显示模组进行编码或格式转换后得到并显示预览图像。另一路数据流采用图1所示的处理算法2进行处理，然后经过编码器1可编码得到录像文件。
在一种实现方式中，如图2所示，图像传感器(Sensor)输出图像后，可以由ISP对图像进行“RAW域”和“RGB域”的图像处理；在“RGB域”图像处理后，可以分为两路数据流。一路数据流采用图2所示的处理算法1进行处理，然后由ISP进行“YUV域”的图像处理，再由显示模组进行编码或格式转换后得到并显示预览图像。另一路数据流采用图2所示的处理算法2进行处理，然后由ISP进行“YUV域”的图像处理，再经过编码器1可编码得到录像文件。
其中，处理算法1和处理算法2的图像处理可以在RGB域进行，也可以在YUV域进行。
例如，以处理算法1处理图像为例。在ISP对图像进行“RGB域”的图像处理之后，ISP可以采用处理算法1在图像由RGB格式转换为YUV格式之前，对图像进行处理。之后，ISP可以将处理算法1处理后的图像转换为YUV格式，再对图像进行“YUV域”的图像处理。
又例如，仍以处理算法1处理图像为例。在ISP对图像进行“RGB域”的图像处理之后，ISP可以先将图像由RGB格式转换为YUV格式，再采用处理算法1对YUV格式的图像进行处理。之后，ISP可以对处理算法1处理后的图像进行“YUV域”的图像处理。
需要说明的说，上述处理算法1也可以称为预览图像的后处理算法，处理算法2也可以称为录像文件的后处理算法。处理算法1和处理算法2可以包括防抖处理、去噪处理、虚化处理、色彩和亮度调整等处理功能。其中，Sensor输出的图像为拜耳(Bayer)格式的图像(简称Bayer图像)。图1或图2中，ISP的“RAW域”输入图 像为Bayer格式的图像(即Bayer图像)，ISP的“RAW域”输出图像为RGB格式的图像(简称RGB图像)。图1或图2中，ISP的“RGB域”输入图像为RGB格式的图像(即RGB图像)，ISP的“RGB域”输出图像为YUV格式的图像(简称YUV图像)。图1或图2中，ISP的“YUV域”输入图像为YUV格式的图像(即YUV图像)，ISP的“YUV域”输出的图像经过编码(ENCODE)可以得到预览图像或录像文件。
其中，Bayer、RGB和YUV是图像的三种表达格式。Bayer图像、RGB图像和YUV图像的详细介绍可以参考常规技术中的相关内容，这里不予赘述。
应注意，由于Sensor输出图像，ISP和编码器(即ENCODER，如显示模组的编码器和编码器1)处理图像均可以用于录制视频；因此，可以将Sensor输出图像、ISP和编码器(ENCODER)处理图像的整个过程中的数据流(如录像文件的数据流和预览图像的数据流)称为视频流。
需要说明的是，手机在录像过程中处理图像得到预览图像和录像文件的方式包括但不限于图1和图2所示的方式，其他的处理方式本申请实施例这里不予赘述。以下实施例中，以图1所示的处理方式为例，介绍本申请实施例的方法。
手机录像的过程中，可能会采集到的一些精彩的画面。用户可能会希望手机可以截取到上述精彩的画面，并保存成照片展示给用户。但是，手机录像过程中，每秒需要处理大量图像(如30帧图像)。如此，留给每一帧图像的运算资源和时间都是有限的；因此，手机一般可以使用ISP的硬件处理模块，采用较为简单的处理方式来处理视频流(如录像文件的数据流和预览图像的数据流)；而不会使用复杂的算法来提升画质(如去噪和提亮)。这样的图像处理效果，只能满足视频的要求；而拍照对画质的要求则更高。因此，截取视频流中的图像，并不能得到用户满意的图像。
并且，考虑到功耗和存储空间等因素，录像(即录制视频)所选择的分辨率相比于Sensor出图的分辨率较低。因此，在录像过程中ISP对Sensor输出的图像是进行了下采样的。一般而言，ISP可以在对视频流(如预览流或录像流)做YUV域的图像处理前，对视频流中的图像帧进行下采样。其中，下采样(subsampled)也可以称为降采样(down sampled)。对图像进行下采样，可以缩小图像，降低图像的分辨率。例如，下采样前的图像的像素点的个数可以是4000*3000，下采样后的图像的像素点的个数可以是2000*1500。如此，如图1或图2所示，由ISP的“RAW域”输出的图像是低分辨率图像,记录为(low resolution，LR)。而用户希望在录像过程中获得的精彩图像是高分辨率的图像。
基于此，本申请实施例提供一种获取图像的方法及电子设备，可以自动选择出录像中的精彩图像并呈现给用户，并且可以提升上述精彩图像的图像质量(如分辨率和画质)。
其中，电子设备(如手机)的图像传感器(Sensor)曝光输出的图像可以由ISP处理得到视频流(如录像文件的数据流和预览图像的数据流)。本方案中，如图3所示，可以由电子设备的选帧模块从ISP(如ISP的“RAW域”和“RGB域”，简称为ISP RAW+RGB)处理后的视频流中选择出备选帧，即备选的精彩帧(也称为精彩图像)。精彩帧或精彩帧的定义可以参考以下实施例中的介绍，这里不予赘述。
如图3所示，电子设备可以将图像传感器(Sensor)曝光输出Bayer图像(即图 像1)缓存在一个第一缓存队列(Buffer)中。该第一缓存队列可以缓存多帧Bayer图像。如此，即使从“Sensor输出备选帧对应的Bayer图像”到“选帧模块选择出备选帧”存在延迟，该备选帧对应的Bayer图像也可以缓存第一缓存队列中。如此，手机则可以从第一缓存队列中得到备选帧对应的Bayer图像(如图3所示的图像2)。
并且，如图3所示，电子设备还可以采用预设RAW域图像处理算法处理图像2得到图像3，有助于提升精彩图像的画质。结合上述描述可知：预览图像和录像文件对应的视频流中的图像LR是经过下采样的。而图3所示ISP处理图像3得到的图像HR是未经过下采样的；因此，编码器2对图像HR进行编码得到的精彩图像是高分辨率的图像。最后，电子设备可以在录像结束后，向用户展示经过图3所示的精彩图像。
综上所述，采用本方案，电子设备可以自动选择出录像中的精彩图像并呈现给用户，并且可以提升上述精彩图像的图像质量(如分辨率和画质)。
其中，预设RAW域图像处理算法是一个RAW域的画质增强的深度学习网络。该预设RAW域图像处理算法也可以称为预设画质增强算法、预设画质增强算法模型或者预设RAW域AI模型。
示例性的，上述预设RAW域图像处理算法可以运行在电子设备的图形处理器(graphics processing unit，GPU)、神经网络处理器(neural-network processing unit，NPU)或者其他具备运行神经网络模型能力的处理器中。上述任一种处理器在运行预设RAW域图像处理算法之前，可以从内存加载该预设RAW域图像处理算法。
在一些实施例中，预设RAW域图像处理算法可以是软件图像处理算法。该预设RAW域图像处理算法可以是手机的硬件抽象层(hardware abstraction layer，HAL)算法库中的一种软件算法。
在另一些实施例中，预设RAW域图像处理算法可以是硬件图像处理算法。该预设RAW域图像处理算法可以是调用ISP中的“RAW域”图像处理算法能力实现的一种硬件图像处理算法。或者，该预设RAW域图像处理算法可以是调用ISP中的“RAW域”和“RGB域”图像处理算法能力实现的一种硬件图像处理算法。或者，该预设RAW域图像处理算法可以是调用ISP中的“RAW域”、“RGB域”和“YUV域”图像处理算法能力实现的一种硬件图像处理算法。
需要说明的是，预设RAW域图像处理算法也可以称为预设图像处理算法。本申请实施例中之所以称之为预设RAW域图像处理算法，是因为该预设RAW域图像处理算法输入的是RAW域的图像。该预设RAW域图像处理算法输出的可以是RAW域的图像，也可以是RGB域的图像。
图1或图2所示的显示模组中的编码器、编码器1和编码器2可以是三个不同的编码器。手机可以采用三个不同的编码器进行编码或者格式转换得到上述预览图像、录像文件和抓拍图像。或者，上述显示模组中的编码器、编码器1和编码器2可以是同一个编码器。一个编码器可以包括多个编码单元。手机可以采用一个编码器中三个不同的编码单元分别进行编码或者格式转换得到上述预览图像、录像文件和抓拍图像。或者，显示模组中的编码器和编码器1可以是同一个编码器中不同的两个编码单元，编码器2可以是另一个编码器。
其中，不同编码器的编码方式可以相同，也可以不同。同一编码器的不同编码单 元的编码方式可以相同，也可以不同。因此，上述显示模组中的编码器和编码器1输出的图像格式可以相同，也可以不同。例如，显示模组中的编码器和编码器1输出的图像可以是联合图像专家组(Joint Photographic Experts Group，JPEG)、标签图像文件格式(Tag Image File Format，TIFF)等任一种格式的图像。
示例性的，本申请实施例中的电子设备可以是手机、平板电脑、智能手表、桌面型、膝上型、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本，以及蜂窝电话、个人数字助理(personal digital assistant，PDA)、增强现实(augmented reality，AR)\虚拟现实(virtual reality，VR)设备等包括摄像头的设备，本申请实施例对该电子设备的具体形态不作特殊限制。
下面将结合附图对本申请实施例的实施方式进行详细描述。请参考图4，为本申请实施例提供的一种电子设备400的结构示意图。如图4所示，电子设备400可以包括：处理器410，外部存储器接口420，内部存储器421，通用串行总线(universal serial bus，USB)接口430，充电管理模块440，电源管理模块441，电池442，天线1，天线2，移动通信模块450，无线通信模块460，音频模块470，扬声器470A，受话器470B，麦克风470C，耳机接口470D，传感器模块480，按键490，马达491，指示器492，摄像头493，显示屏494，以及用户标识模块(subscriber identification module，SIM)卡接口495等。
其中，上述传感器模块480可以包括压力传感器，陀螺仪传感器，气压传感器，磁传感器，加速度传感器，距离传感器，接近光传感器，指纹传感器，温度传感器，触摸传感器，环境光传感器和骨传导传感器等传感器。
可以理解的是，本实施例示意的结构并不构成对电子设备400的具体限定。在另一些实施例中，电子设备400可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器410可以包括一个或多个处理单元，例如：处理器410可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
控制器可以是电子设备400的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器410中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器410中的存储器为高速缓冲存储器。该存储器可以保存处理器410刚用过或循环使用的指令或数据。如果处理器410需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器410的等待时间，因而提高了系统的效率。
在一些实施例中，处理器410可以包括一个或多个接口。可以理解的是，本实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备400的结构限定。在另一些实施例中，电子设备400也可以采用上述实施例中不同的接口连接 方式，或多种接口连接方式的组合。
充电管理模块440用于从充电器接收充电输入。充电管理模块440为电池442充电的同时，还可以通过电源管理模块441为电子设备供电。
电源管理模块441用于连接电池442、充电管理模块440与处理器410。电源管理模块441接收电池442和/或充电管理模块440的输入，为处理器410，内部存储器421，外部存储器，显示屏494，摄像头493，和无线通信模块460等供电。
电子设备400的无线通信功能可以通过天线1，天线2，移动通信模块450，无线通信模块460，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。在一些实施例中，电子设备400的天线1和移动通信模块450耦合，天线2和无线通信模块460耦合，使得电子设备400可以通过无线通信技术与网络以及其他设备通信。
电子设备400通过GPU，显示屏494，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏494和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器410可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏494用于显示图像，视频等。该显示屏494包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrixorganic light emitting diode，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。
电子设备400可以通过ISP，摄像头493，视频编解码器，GPU，显示屏494以及应用处理器等实现拍摄功能。
ISP用于处理摄像头493反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头493中。
摄像头493用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备400可以包括N个摄像头493，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备400在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备400可以支持一种或多种 视频编解码器。这样，电子设备400可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备400的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口420可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备400的存储能力。外部存储卡通过外部存储器接口420与处理器410通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器421可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器410通过运行存储在内部存储器421的指令，从而执行电子设备400的各种功能应用以及数据处理。例如，在本申请实施例中，处理器410可以通过执行存储在内部存储器421中的指令，内部存储器421可以包括存储程序区和存储数据区。
其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备400使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器421可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
电子设备400可以通过音频模块470，扬声器470A，受话器470B，麦克风470C，耳机接口470D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
按键490包括开机键，音量键等。马达491可以产生振动提示。指示器492可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
SIM卡接口495用于连接SIM卡。SIM卡可以通过插入SIM卡接口495，或从SIM卡接口495拔出，实现和电子设备400的接触和分离。电子设备400可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口495可以支持Nano SIM卡，Micro SIM卡，SIM卡等。
以下实施例中的方法均可以在具有上述硬件结构的电子设备400中实现。以下实施例中，以电子设备400是手机为例，介绍本申请实施例的方法。图5是本申请实施例的手机的软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android TM系统分为五层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，硬件抽象层(hardware abstraction layer，HAL)以及内核层。应理解：本文以Android系统举例来说明，在其他操作系统中(例如鸿蒙 TM系统，IOS TM系统等)，只要各个功能模块实现的功能和本申请的实施例类似也能实现本申请的方案。
应用程序层可以包括一系列应用程序包。
如图5所示，应用程序层中可以安装通话，备忘录，浏览器，联系人，图库，日历，地图，蓝牙，音乐，视频，短信息等应用。
在本申请实施例中，应用程序层中可以安装具有拍摄功能的应用，例如，相机应用。当然，其他应用需要使用拍摄功能时，也可以调用相机应用实现拍摄功能。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
例如，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，资源管理器，通知管理器等，本申请实施例对此不做任何限制。
例如，上述窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。上述内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。上述视图系统可用于构建应用程序的显示界面。每个显示界面可以由一个或多个控件组成。一般而言，控件可以包括图标、按钮、菜单、选项卡、文本框、对话框、状态栏、导航栏、微件(Widget)等界面元素。上述资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。上述通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，振动，指示灯闪烁等。
如图5所示，Android runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
其中，表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。2D图形引擎是2D绘图的绘图引擎。
内核层位于HAL之下，是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动等，本申请实施例对此不做任何限制。
在本申请实施例中，仍如图5所示，以相机应用举例，可应用程序框架层中设置有相机服务(Camera Service)。相机应用可通过调用预设的API启动Camera Service。 Camera Service在运行过程中可以与硬件抽象层(HAL)中的Camera HAL交互。其中，Camera HAL负责与手机中实现拍摄功能的硬件设备(例如摄像头)进行交互，Camera HAL一方面隐藏了相关硬件设备的实现细节(例如具体的图像处理算法)，另一方面可向Android系统提供调用相关硬件设备的接口。
示例性的，相机应用运行时可将用户下发的相关控制命令(例如预览、放大、拍照、录像或者抓拍指令)发送至Camera Service。一方面，Camera Service可将接收到的控制命令发送至Camera HAL，使得Camera HAL可根据接收到的控制命令调用内核层中的相机驱动，由相机驱动来驱动摄像头等硬件设备响应该控制命令采集图像数据。例如，摄像头可按照一定的帧率，将采集到的每一帧图像数据通过相机驱动传递给Camera HAL。其中，控制命令在操作系统内部的传递过程可参见图5中控制流的具体传递过程。
另一方面，Camera Service接收到上述控制命令后，可根据接收到的控制命令确定此时的拍摄策略，拍摄策略中设置了需要对采集到的图像数据执行的具体图像处理任务。例如，在预览模式下，Camera Service可在拍摄策略中设置图像处理任务1用于实现人脸检测功能。又例如，如果在预览模式下用户开启了美颜功能，则Camera Service还可以在拍摄策略中设置图像处理任务2用于实现美颜功能。进而，Camera Service可将确定出的拍摄策略发送至Camera HAL。
当Camera HAL接收到摄像头采集到的每一帧图像数据后，可根据Camera Service下发的拍摄策略对上述图像数据执行相应的图像处理任务，得到图像处理后的每一帧拍摄画面。例如，Camera HAL可根据拍摄策略1对接收到的每一帧图像数据执行图像处理任务1，得到对应的每一帧拍摄画面。当拍摄策略1更新为拍摄策略2后，Camera HAL可根据拍摄策略2对接收到的每一帧图像数据执行图像处理任务2，得到对应的每一帧拍摄画面。
后续，Camera HAL可将经过图像处理后的每一帧拍摄画面通过Camera Service上报给相机应用，相机应用可将每一帧拍摄画面显示在显示界面中，或者，相机应用以照片或视频的形式将每一帧拍摄画面保存在手机内。其中，上述拍摄画面在操作系统内部的传递过程可参见图5中数据流的具体传递过程。
本申请实施例这里结合图5介绍手机中各个软件层实现本申请实施例的方法的工作原理。相机应用在录像模式下运行时，可将用户下发的抓拍指令发送至Camera Service。在录像模式下，Camera HAL可根据之前接收到的录像指令调用内核层中的相机驱动，由相机驱动来驱动摄像头等硬件设备响应该录像指令采集图像数据。例如，摄像头可按照一定的帧率，将采集到的每一帧图像数据通过相机驱动传递给Camera HAL。其中，基于录像指令由相机驱动传递给Camera HAL的每一帧图像组成的数据流可以为本申请实施例中所述的视频流(如预览流和录像流)。HAL中包括选帧模块。Camera HAL可以调用选帧模块从该第一缓存队列中选择出备选帧(即精彩帧)。
本申请实施例中，摄像头的图像传感器(Sensor)曝光输出的每一帧图像(如图3所示的图像1)可以缓存在第一缓存队列(Buffer)中。Camera HAL可以调用选帧模块从该第一缓存队列中选择出与备选帧(即精彩帧)匹配的一帧图像(如图3所示的图像2)。其中，第一缓存队列可以设置在手机软件系统的任何一层，如第一缓存 队列可以设置在Camera HAL通过软件接口访问的内存区域。
HAL中还包括预设RAW域图像处理算法。Camera HAL可以调用预设RAW域图像处理算法处理与上述备选帧匹配的图像2,得到处理后的图像帧(如图3所示的图像3)。之后，如图3所示，可由ISP处理该图像3得到高分辨率图像,记录为(high resolution，HR)。编码器2可以对该HR进行编码得到精彩图像。手机可以在录像结束后，向用户展示精彩图像。
本申请实施例提供一种获取图像的方法，该方法可以应用于手机，该手机包括摄像头。
在本申请一实施例中，该方法可以应用于手机录像过程中。在手机录像过程中，手机可以自动选择出录像中的精彩图像，并在录像结束后呈现给用户。手机自动选择录像中的精彩图像的动作对用户不可见。手机录像结束后，不仅可以为用户呈现录制的视频，还可以呈现录像中的精彩图像，实现一拍多得的效果。具体的，如图6所示，该方法可以包括S601-S607。
S601、手机接收用户的第一操作。该第一操作用于触发手机开始录制视频。
示例性的，手机可以显示图7所示的录像的取景界面701。该录像的取景界面701是手机还未开始录像的取景界面。该录像的取景界面701包括“开始录像”按钮702。上述第一操作可以是用户对“开始录像”按钮702的点击操作，用于触发手机开始录制视频。
S602、响应于第一操作，手机显示第一界面。该第一界面是手机正在录制视频的取景界面。该第一界面包括预览图像，预览图像是手机接收到第一操作后由手机的摄像头采集的第一图像得到的。
示例性的，以第一操作是用户对“开始录像”按钮702的点击操作为例。手机响应于用户对“开始录像”按钮702的点击操作，手机的显示屏可显示图7所示的第一界面703。该第一界面703是手机正在录制视频的取景界面。如图7所示，该第一界面703包括预览图像704。该预览图像704是手机接收到第一操作后由手机的摄像头采集的第一图像得到的。
其中，本申请实施例这里介绍手机由第一图像得到预览图像704的方法。手机的摄像头采集第一图像之后，手机显示第一界面之前，手机可以采样手机的ISP，依次对第一图像进行RAW的图像处理、RGB域的图像处理和YUV域的图像处理，得到预览图像704。应注意，手机的ISP可以对摄像头采集的每一帧第一图像均执行上述RAW的图像处理、RGB域的图像处理和YUV域的图像处理。
例如，请参考图8A，其示出本申请实施例提供的一种录像过程中的图像处理原理示意图。手机由第一图像得到预览图像704的方法，可以参考图8A所示“预览图像”的处理方法。
如图8A所示，手机的图像传感器(Sensor)受到曝光的控制，可以不断输出Bayer图像(即第一图像)。每一帧Bayer图像(即第一图像)由手机的ISP进行RAW域的图像处理得到RGB图像，RGB图像由ISP进行RGB域的图像处理得到YUV图像。YUV图像由处理算法1进行处理，之后由ISP进行YUV域的图像处理后，送至显示模组的编码器进行编码或格式转换，便可以得到预览图像704。处理后的多帧预览图像704 可以形成一段预览的视频流。
其中，预览图像的处理流程中，RAW的图像处理、RGB域的图像处理和YUV域的图像处理的详细描述，可以参考上述实施例中的相关介绍，这里不予赘述。图8A所述的录像文件的处理方式可以参考上述实施例对录像文件处理方式的介绍，这里不予赘述。
在一些实施例中，如图7所示，第一界面703还包括抓拍快门702。该抓拍快门702用于触发手机抓拍图像得到照片。具体的，该抓拍快门702用于触发手机在录像的过程中抓拍图像得到照片。可以想到的是，手机录制视频(即录像)的过程中，可能会采集到的一些精彩的画面。在手机录像的过程中，用户可能会希望手机可以抓拍到上述精彩的画面，并保存成照片展示给用户。用户点击上述抓拍快门702便可以实现录像过程中抓拍精彩图像的功能。该实施例中，在手机录像过程中，即使未接收到用户对上述抓拍快门702的点击操作，手机也可以自动选择出录像中的精彩图像并呈现给用户。具体的，在上述S602之后，本申请实施例的方法还可以包括S603-S607。
在另一些实施例中，第一界面703可以不包括抓拍快门702。在手机录像过程中，手机可以自动选择出录像中的精彩图像并呈现给用户。具体的，在上述S602之后，本申请实施例的方法还可以包括S603-S07。
为了保证手机可以自动选择出录像中的精彩图像并呈现给用户；手机可以将Sensor曝光输出Bayer图像缓存在一个第一缓存队列(Buffer)中。如此，即使从接收到用户的抓拍操作到Snapshot程序接收到抓拍指令，存在延迟时长；接收到用户的抓拍操作时，Sensor输出的Bayer图像也可以缓存在第一缓存队列中。这样，手机便可以从第一缓存队列中获取这一帧图像。具体的，响应于上述第一操作，手机还可以执行S603。
S603、手机在第一缓存队列缓存摄像头采集的第一图像。该第一缓存队列缓存摄像头采集的n帧第一图像，n≥1，n为整数。
示例性的，手机响应于上述第一操作，手机可以在图8A所示的第一缓存队列(Buffer)中缓存摄像头采集的第一图像。例如，该第一缓存队列可以以先进先出的原则缓存摄像头采集的n帧第一图像。如图9所示，第一缓存队列的队尾可以执行入队操作，用于插入第一图像；第一缓存队列的队头可以执行出队操作，用于删除第一图像。在第一缓存队列中已缓存n帧第一图像的情况下，第一缓存队列的队尾每插入一帧第一图像，第一缓存队列的队头则删除一帧第一图像。
在一些实施例中，n可以等于1。在这种情况下，第一缓存队列中可以缓存一帧第一图像。如此，手机在执行S605时，预设RAW域图像处理算法只能输入一帧第一图像。
在另一些实施例中，n可以大于1。在这种情况下，第一缓存队列中可以缓存多帧第一图像。如此，手机在执行S605时，预设RAW域图像处理算法可以输入一帧第一图像，也可以输入多帧第一图像。其中，向预设RAW域图像处理算法输入多帧第一图像，可以对参考帧起到画质增强的作用，有利于获取噪声和纹理等信息，可以进一步提升预设RAW域图像处理算法输出图像的画质。
示例性的，n可以为预设正整数。假设Sensor每秒钟可以曝光a帧Bayer图像，图3所示的延迟时长为b秒，则Sensor在延迟时长b秒内可以曝光出b/(1/a)＝a*b帧Bayer图像。n可以为大于或者等于a*b的整数。
S604、手机从第一视频流中选择出第二图像。第二图像包括目标对象的图像。第一视频流是手机的ISP对多帧第一图像执行第一处理得到的，第一处理包括下采样。
具体的，上述第一视频流可以是上述预览图像的数据流；或者，第一视频流可以是上述录像文件的数据流。例如，如图8A所示，图像传感器(Sensor)输出图像后，可以由ISP对图像进行“RAW域”、“RGB域”和“YUV域”的图像处理；在“YUV域”图像处理后，可以分为两路数据流：一路预览图像的数据流，另一路录像文件的数据流。该第一视频流可以是由ISP对Sensor输出的第一图像进行“RAW域”和“RGB域”的图像处理后的数据流。
示例性的，上述目标对象可以是人脸、人体、美食、动物、建筑等至少一种。也就是说，手机可以识别第一视频流中的图像，将包括目标对象的图像作为第二图像。
在一些实施例中，目标对象包括人脸，上述第二图像可以是目标对象的表情满足预设条件的图像。也就是说，手机(手机的选帧模块)可以从第一视频流中选择目标对象的表情满足预设条件的图像，作为图8A所示的第二图像(即图3所示的备选帧)。
例如，目标对象的表情满足预设条件，具体可以为：可以为人脸微笑。也就是说，第二图像可以是第一视频流中包括笑脸图像的一帧图像。在该实例中，手机可以在录像中抓拍到人脸微笑的精彩图像。
在另一些实施例中，目标对象包括人体。上述第二图像可以是目标对象的动作满足预设条件的图像。也就是说，手机(手机的选帧模块)可以从第一视频流中选择目标对象的动作满足预设条件的图像，作为图8A所示的第二图像(即图3所示的备选帧)。
例如，目标对象的动作满足预设条件，具体可以为：人体的动作为预设动作。例如，该预设动作可以为人体跳跃至最高点时的人体动作、舞蹈表演或者运动竞技表演过程中的精彩动作，人体运动锻炼过程中的高难度动作等。也就是说，第二图像可以是第一视频流中包括人体摆出预设动作时的一帧图像。在该实例中，手机可以在录像过程中抓拍到人体运动过程中摆出预设动作时的精彩图像。
在一些实施例中，上述目标对象可以是手机出厂时，预先配置在手机中的。手机中可以预先配置多个目标对象。
在另一些实施例中，上述目标对象还可以是用户在手机中设置的。例如，手机可以接收用户在设置界面设置的目标对象的类型。
示例性的，手机可以显示图10中的(a)所示的设置界面1001。该设置界面1001包括预设选项，如“拍摄目标对象”选项1002。该“拍摄目标对象”选项1002用于触发手机设置录像过程中需抓拍的目标对象。响应于用户对“拍摄目标对象”选项1002的点击操作，手机可以显示图10中的(b)所示的目标对象设置界面1003。该目标对象设置界面1003包括多个拍摄对象的选项，如“人脸”选项、“人体”选项、“美食”选项、“动物”选项和“建筑”选项等。手机可以接收用户多该多个拍摄对象的选项中任一拍摄对象的选项的选择操作。例如，手机可以接收用户对图10中的(b)所示的“人脸”选项和“美食”选项的选择操作，显示图10中的(b)所示的目标对象设置界面1004。之后，手机可以接收用户对目标对象设置界面1004中“确定”按钮的点击操作。响应于用户对“确定”按钮的点击操作，手机可以将人脸和美食设置为目标对象。
在一种可能的实现方式中，上述目标对象设置界面不仅可以包括多个拍摄对象的选项，还可以包括每个拍摄对象的提示信息。该提示信息可以用于向用户提示选择对应拍摄对象作为目标对象后，手机可以实现的功能。例如，图10中的(b)所示的目标对象设置界面1003可以替换为图11所示的目标对象设置界面1101。该目标对象设置界面1101不仅包括多个拍摄对象的选项(如“人脸”选项、“人体”选项等)，还包括每个拍摄对象的提示信息。例如，目标对象设置界面1101包括拍摄对象“人脸”的提示信息1102(如，选择“人脸”,手机在录像过程中可抓拍到人脸微笑的精彩图像)和拍摄对象“人体”的提示信息1103(如，选择“人体”,手机在录像过程中可抓拍到人体运动时摆出预设运动的精彩图像)。通过各个拍摄对象的提示信息，用户可以明确选择对应拍摄对象作为目标对象后手机能够实现的功能。这样，可以便于用户依据该用户需要手机在录像过程中实现的功能来设置符合用户需求的目标对象。
在一些实施例中，手机启动预设功能后，才可以执行本申请实施例的方法，在录像过程中抓拍目标对象的精彩图像。该预设功能为手机在录像过程中抓拍目标对象的精彩图像的功能。
在一种实现方式中，手机可以在上述目标对象设置界面1101启动或关闭上述预设功能。例如，图10中的(b)所示的目标对象设置界面1003、图10中的(c)所示的目标对象设置界面1004和图11所示的目标对象设置界面1001可以包括预设功能的开关1005。手机可以接收用户对预设功能的开关1005的选择操作，启动或关闭预设功能。
在另一种实现方式中，手机可以在录像的取景界面启动或关闭上述预设功能。
例如，手机可以在该手机还未开始录像的取景界面启动或关闭上述预设功能。如图12中的(a)或图12中的(b)所示，手机还未开始录像的取景界面包括预设功能的开关1201。手机可以接收用户对图12中的(b)所示预设功能的开关1201的操作1(如单击操作、双击操作或者长按操作等任一种操作)，显示图12中的(c)所示的取景界面。该操作1是第三操作。其中，假设在图12中的(a)或图12中的(b)所示的取景界面中，预设功能的开关1201处于关闭状态，图12中的(c)所示的取景界面中的预设功能的开关1201处于开启状态。当然，手机还可以接收用户对图12中的(c)所示的预设功能的开关1201的操作1，显示图12中的(b)所示的取景界面，关闭所述预设功能。
又例如，手机可以在该手机已开始录像的取景界面启动或关闭上述预设功能。如图13所示，手机已开始录像的取景界面包括预设功能的开关1301。手机在已开始录像的取景界面启动或关闭预设功能的方法，可以参考上述实施例对手机在未开始录像的取景界面启动或关闭预设功能的方法，本申请实施例这里不予赘述。
在另一些实施例中，上述目标对象还可以是用户在手机中设置的。例如，手机可以接收用户在录像的取景界面设置的目标对象的类型。该录像的取景界面可以包括：录像模式下，已开始录像的取景界面(即第一界面)和还未开始录像的取景界面(即第二界面)。
例如，图12中的(a)、图12中的(b)、图12中的(c)和图13所示的取景界面均包括预设功能的开关。手机可以接收用户对该预设功能的开关的操作2(如单击 操作、双击操作或者长按操作等任一种操作)。其中，操作2与上述操作1不同。该操作2是第二操作。响应于该操作2，显示目标对象的设置界面。该目标对象的设置界面包括多个待选择的拍摄对象的选项。例如，手机响应于用户对图13所示的预设功能的开关1301的操作2，可以显示以14所示的目标对象设置界面1401。手机可以响应于用户对目标对象设置界面1401中多个待选择的拍摄对象的选项的选择操作，将用户选择的拍摄对象确定为目标对象。或者，响应于用户对图13所示的预设功能的开关1301的操作2，手机可以显示图11所示的目标对象的设置界面1101。响应于用户对目标对象的设置界面1101中“取消”按钮或“确定”按钮的点击操作，手机可返回图13所示的取景界面。
在另一些实施例中，上述目标对象可以是目标场景下预先配置的拍摄对象。例如，目标场景可以是人像场景、运动场景、宠物场景、美食场景等至少一种拍摄场景。针对不同的拍摄场景，可以预设一个或多个拍摄对象。
例如，人像场景下预先配置的拍摄对象(即目标对象)可以是人脸。目标对象的表情满足预设条件，具体可以为：可以为人脸微笑。也就是说，第二图像可以是包括笑脸图像的一帧图像。在该实例中，手机可以在录像中抓拍到人脸微笑的精彩图像。
又例如，运动场景下预先配置的拍摄对象可以是人体。目标对象的动作为预设动作。其中，预设动作的描述可参考上述实施例中的介绍，这里不予赘述。在该实例中，手机可以在录像过程中抓拍到人体运动过程中摆出预设动作时的精彩图像。
在该实施例中，手机在录像模式下，可以根据摄像头采集到的第一图像识别出目标场景。然后，手机可以识别目标场景下预先配置的拍摄对象(即目标对象)的表情或动作是否满足预设条件。如果第一视频流中一帧图像中包括目标对象的图像，且这帧图像中目标对象的表情或动作满足预设条件，手机则可以选择这帧图像作为第二图像。
其中，上述目标场景可以是手机出厂时，预先配置在手机中的。手机中可以预先配置多个目标对场景。或者，上述目标场景还可以是用户在手机中设置的。例如，手机可以接收用户在设置界面设置的目标场景。又例如，手机可以接收用户在录像的取景界面设置的目标对象的类型。该录像的取景界面可以包括：录像模式下，已开始录像的取景界面和还未开始录像的取景界面。其中，手机接收用户的操作设置目标场景的方法，可以参考上述实施例中设置目标对象的方法，本申请实施例这里不予赘述。。例如，图10中的(b)、图10中的(c)、图11和图14所示的目标对象的设置界面可以替换为目标场景的设置界面，目标对象的设置界面中拍摄对象的选项可以替换为拍摄场景的选项。
其中，上述第一视频流是手机的ISP对多帧第一图像执行第一处理得到的。上述第一处理包括下采样。下采样(subsampled)也可以称为降采样(down sampled)。对图像进行下采样，可以缩小图像，降低图像的分辨率。例如，下采样前的图像的像素点的个数可以是4000*3000，下采样后的图像的像素点的个数可以是2000*1500。如此，如图1、图2、图3和图8A中任一附图所示，由ISP的“RAW域”输出的图像是低分辨率图像,记录为LR图像。第一视频流包括多帧LR图像。手机从接收到上述第一操作开始，便可以开始实时地从该第一视频流中选择出第二图像。或者，手机从接收 到上述第一操作开始，如图8B所示，可以开始周期性地从该第一视频流中选择出第二图像。
本申请实施例中，如图5所示，手机的HAL中的Camera HAL可以包括一个选帧模块。Camera HAL接收到来自Camera Service的录像指令后，便可以开始实时或者周期性的从第一视频流中选择出所述第二图像(也称为参考帧)。
应理解，第二图像是从多帧经过下采样的LR图像(即第一视频流)中选择出的；因此，该第二图像的分辨率也较低。而用户希望在录像过程中获得的精彩图像是高分辨率的图像。因此，本申请实施例中，手机从第一视频流中选择出第二图像后，还需要依据第二图像得到图像内容相同或相似的高分辨率图像。具体的，S604之后，本申请实施例的方法还可以包括S605：
S605、手机将n帧第一图像中与第二图像对应的m帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像。该预设RAW域图像处理算法具备提升图像画质的功能。其中，m≥1，m为整数。
示例性的，手机执行S603将Sensor输出的图像缓存在第一缓存队列中。该第一缓存队列中可以缓存n帧第一图像。并且，该第一缓存队列中缓存的第一图像并未执行下采样，是高分辨率图像。因此，手机可以从第一缓存队列缓存的n帧第一图像中找到与第二图像的图像内容相同或相似的高分辨率图像。具体的，如图15所示，S605可以包括S605a-S605b。
S605a、手机从n帧第一图像选择出与第二图像对应的第五图像。
其中，在n帧第一图像中，第五图像由摄像头的图像传感器输出的时间与第二图像由摄像头的图像传感器输出的时间最近；或者，n帧第一图像中，第五图像的清晰度最高。当然，也可以同时参考前述两种选帧依据(如清晰度和时间)，使用不同的权重并进行加权，作为对第五图像的选择依据。以下实施例中，以第五图像由摄像头的图像传感器输出的时间与第二图像由摄像头的图像传感器输出的时间最近为例，介绍本申请实施例的方法。
在一些实施例中，上述每一帧第一图像对应一个时间信息，该时间信息记录有图像传感器Sensor输出对应第一图像的时间。其中，该时间信息也可以称为时间戳。如此，手机(如上述选帧模块)则可以选择第一缓存队列Buffer中时间戳与第二图像的时间戳记录的时间最近的第一图像作为上述第五图像。
本申请实施例中，手机可以将Sensor曝光输出Bayer图像缓存在一个第一缓存队列Buffer中。该第一缓存队列可以缓存多帧Bayer图像。如此，即使从Sensor输出第二图像对应的第一图像到手机(如选帧模块)从第一视频流中选择出第二图像，存在延迟时长。手机从第一视频流中选择出第二图像时，Sensor输出的Bayer图像也可以缓存在第一缓存队列中。手机也可以从Buffer中选择与第二图像的图像内容相同或相似的高分辨率图像。
需要说明的是，在一些平台，可以将Sensor曝光结束时间作为时间戳；在另一些平台可以将Sensor开始曝光时间作为时间戳，本申请实施例对此不作限制。
S605b、手机将n帧第一图像中的m帧第一图像作为输入，运行预设RAW域图像处理算法得到第三图像。该m帧第一图像包括上述第五图像。该m帧第一图像中除第五 图像的其他图像，用于对第五图像进行画质增强。
在一些实施例中，m可以等于1。也就是说，m帧第一图像是上述第五图像。但是，手机将上述第五图像作为输入运行预设RAW域图像处理算法，便可以得到画质较高的第三图像。预设RAW域图像处理算法是一个单帧输入单帧输出的图像处理算法。但是，一帧图像中的数据的完整性和纹理等参数均有限，将一帧图像作为输入运行预设RAW域图像处理算法，并不能有效提升这一帧图像的画质。
基于此，在另一些实施例中，m可以大于1。具体的，手机可以将该第五图像以及该第五图像相邻的至少一帧图像作为输入，运行预设RAW域图像处理算法。即可以将n帧第一图像中、包括第五图像在内的m帧第一图像作为输入运行预设RAW域图像处理算法。预设RAW域图像处理算法是一个多帧输入单帧输出的图像处理算法。应理解，m帧第一图像中除第五图像之外的其他图像，可以对第五图像起到画质增强的作用，有利于获取噪声和纹理等信息，可以进一步提升第三图像的画质。
在一些实施例中，上述m帧第一图像为第一缓存队列中相邻的m帧图像。在另一些实施例中，m帧第一图像也可以是第一缓存队列缓存的n帧第一图像中，不相邻但包括第五图像的m帧图像。
也就是说，本申请实施例中所述的预设RAW域图像处理算法可以是一个多帧输入、单帧输出的神经网络模型。其中，预设RAW域图像处理算法是一个RAW域的画质增强的深度学习网络。本方案中，增加了预设RAW域图像处理算法的算法处理，相比于完全采用ISP的硬件RAW域的图像处理、RGB域的图像处理和YUV域的图像处理，预设RAW域图像处理算法与ISP结合的效果更好，有助于提升第三图像的图像质量。
需要说明的是，第一缓存队列中缓存的第一图像并未执行下采样，是高分辨率图像。因此，手机可以从第一缓存队列缓存的n帧第一图像中找到与第二图像的图像内容相同或相似的高分辨率图像(即第五图像)。并且，手机可以采用m帧第一图像中除第五图像之外的其他图像对第五图像进行画质增强。如此，便可以得到高分辨率、且图像质量较高的第三图像。
S606、手机采用ISP对第三图像执行第二处理得到第四图像。该第二处理不包括下采样。
本申请实施例中，手机可以通过时分复用的方式，采用ISP处理第一图像得到预览图像，处理第三图像得到第四图像。也就是说，手机采用ISP处理第一图像得到预览图像，并不会影响手机采用ISP处理第三图像得到第四图像。该第四图像用于生成录像过程中的精彩图像。换言之，手机处理图8A或图8B所示的精彩图像，并不会影响手机处理图8A或图8B所示的预览图像和录像文件。
示例性的，手机可以采用ISP的硬件模块处理第一图像得到预览图像，处理第三图像得到第四图像。其中，由第一图像得到预览图像的图像处理流程中还包括处理算法1的处理流程。在一些实施例中，上述处理算法1可以包含在ISP的硬件模块中。
在另一些实施例中，处理算法1可以包含在手机的其他处理器(如CPU、GPU或者NPU等任一处理器)中。在该实施例中，ISP的硬件模块可以调用上述其他处理器中的处理算法1，来处理第一图像得到预览图像。
需要强调的是，上述第三图像是高分辨率、且图像质量较高的图像。因此，手机 采用ISP对第三图像执行不包括下采样的第二处理后得到的第四图像，也是高分辨率的图像。并且，第四图像经过ISP的处理，相比于第三图像也可以提升图像质量。因此，该第四图像也是高分辨率、且图像质量较高的图像。
S607、手机在录像结束后，展示对第四图像进行编码得到的精彩图像。
其中，如图8A或图8B所示，手机可以采用预设RAW域图像处理算法处理m帧得到第三图像(即执行执行S605)；手机可以采用ISP处理第三图像得到第五图像(即执行S606)；之后，手机可以采用编码器2对第四图像(即HR图像)进行编码得到的精彩图像。并且，手机可以在录像结束后，向用户展示该精彩图像。
示例性的，响应于用户对图7所示的“开始录像”按钮702的单击操作(即第一操作)，手机可以执行S601-S606，并对第四图像进行编码得到的精彩图像。之后，手机可以接收用户对图7或图16所示的“结束录像”按钮706的点击操作，可结束录像并生成录像文件。例如，手机响应于用户对图16所示“结束录像”按钮706的点击操作，可以显示图16所示的录像的取景界面1601。录像的取景界面1601是手机未开始录像的取景界面。与图7所示的录像的取景界面701相比，手机的取景界面中的照片选项中的照片由图7所示的708更新为图16所示的1602。
手机可以响应于用户对相册应用的启动操作，显示图17中的(a)所示的相册列表界面1701或图7所示的相册列表界面1703。该相册列表界面1701包括手机中可以保存的一个或多个照片和视频。例如，相册列表界面1701和相册列表界面1703包括手机录制的录像文件1702。
其中，手机录像结束后，可以向用户展示录像过程中抓拍的精彩图像，实现一拍多得的效果。一拍多得是指手机录像结束后不仅可以得到录像文件，还可以得到录像中抓拍的精彩图像。
在上述实施例的实现方式(1)中，手机在录像中抓拍的精彩图像在相册中对用户可见。用户可以从相册中查看录像中抓拍的精彩图像。例如，如图7所示，相册列表界面1703还包括手机在录制录像文件1702过程中抓拍的包括精彩图像的照片1704。以目标对象是人脸为例。照片1704包括笑脸图像。
在上述实施例的实现方式(2)中，手机在录像中抓拍的精彩图像在相册中对用户不可见。用户可以从相册中不能查看到录像中抓拍的精彩图像。例如，如图7所示，相册列表界面1701不包括图17中的(b)所示的照片1704。
在一些实施例中，手机响应于结束录像的操作，可以发出第一提示信息。该第一提示信息用于请求用户确认是否保留录像过程中抓拍的精彩图像。例如，响应于用户对图16所示的“结束录像”按钮706的点击操作，如图18所示，手机可在取景界面1601显示第一提示信息1801，如“录像过程中自动为您抓拍到一帧精彩图像，请确认是否保留？”。其中，第一提示信息1801还可以包括上述精彩图像的预览图，以供用户根据该预览图确认是否保留精彩图像。
该实施例结合上述实现方式(2)，响应于用户对第一提示信息1801的操作3，手机可以在相册中保存精彩图像。例如，操作3可以是用户对第一提示信息1801中“保留”按钮的点击操作。如图7所示，相册列表界面1701不包括图17中的(b)所示的照片1704。结合实现方式(2)，响应于用户对第一提示信息1801的操作3，手机还 可以以其他方式展示该精彩图像。手机展示精彩图像的方式可以参考以下实施例中的详细描述，这里不予赘述。
应注意，实施例结合上述实现方式(1)，虽然手机不会相册中展示精彩图像。但是，响应于用户对第一提示信息1801的操作3，手机还是可以以其他方式展示该精彩图像。手机展示精彩图像的方式可以参考以下实施例中的详细描述，这里不予赘述。
无论手机在录像中抓拍的精彩图像在相册中是否对用户可见。手机在录像结束后，向用户展示该精彩图像的触发方式至少可以包括以下方式(a)和方式(b)：
方式(a)：手机录像结束后，可生成录像文件。该录像文件的详情页可以包括第一预设控件。手机可以接收用户对该第一预设控件的点击操作。响应于用户对该第一预设控件的点击操作，手机可展示拍摄该录像文件(即录像)过程中抓拍的精彩图像。
例如，响应于用户对图17中的(a)或图17中的(b)所示的录像文件1702的点击操作，如图19中的(a)所示，手机可显示录像文件1702的详情页1901。如图19中的(a)所示，该详情页1901包括第一预设控件1902。响应于用户对该第一预设控件1902的点击操作，如图19中的(b)所示，手机可显示拍摄录像文件1702过程中抓拍的精彩图像的详情页1903，向用户展示该精彩图像。
其中，响应于用户对详情页1903中返回按钮的点击操作，如图19中的(c)所示，手机可显示录像文件1702的详情页1904。
方式(b)：手机录像结束后，可生成录像文件。手机可接收用户在该录像文件的详情页输入的预设手势。响应于该预设手势，手机可显示拍摄该录像文件(即录像)过程中抓拍的精彩图像。
示例性的，上述预设手势可以是向上或向下的滑动手势、S型滑动手势、√型滑动手势等任一种滑动手势。
例如，响应于用户对图17中的(a)或图17中的(b)所示的录像文件1702的点击操作，如图20中的(a)所示，手机可显示录像文件1702的详情页1901。之后，手机可接收用户在详情页1901输入的预设手势(如向上滑动手势)。响应于该预设手势，手机可显示拍摄录像文件1702过程中抓拍的精彩图像的详情页1903，向用户展示该精彩图像。
在方式(b)中，录像文件1702的详情页可以包括第一预设控件1902，也可以不包括第一预设控件1902。例如，图20中的(a)所示的详情页1901包括第一预设控件1902。例如，图20中的(b)所示的详情页2001不包括第一预设控件1902。手机响应于用户在详情页2001输入的预设手势(如向上滑动手势)，也可以可显示拍摄录像文件1702过程中抓拍的精彩图像的详情页1903，向用户展示该精彩图像。
需要说明的是，手机录像结束后，向用户展示该精彩图像的触发方式包括但不限于上述方式(a)和方式(b)，其他的触发方式本申请实施例这里不予赘述。
本申请实施例中，手机可以可以将Sensor曝光输出Bayer图像(即图像1)缓存在一个第一缓存队列(Buffer)中。该第一缓存队列可以缓存多帧Bayer图像。如此，即使从“Sensor输出备选帧对应的Bayer图像”到“选帧模块选择出备选帧”存在延迟，该备选帧对应的Bayer图像也可以缓存第一缓存队列中。如此，手机则可以从第一缓存队列中得到备选帧对应的Bayer图像(如图3所示的图像2)。
并且，如图3所示，手机还可以采用预设RAW域图像处理算法处理图像2得到图像3，有助于提升精彩图像的画质。结合上述描述可知：预览图像和录像文件对应的视频流中的图像LR是经过下采样的。而图3所示ISP处理图像3得到的图像HR是未经过下采样的；因此，编码器2对图像HR进行编码得到的精彩图像是高分辨率的图像。最后，手机可以在录像结束后，向用户展示经过图3所示的精彩图像。
综上所述，采用本方案，手机可以自动选择出录像中的精彩图像并呈现给用户，并且可以提升上述精彩图像的图像质量(如分辨率和画质)。
在一些情况下，由于第一缓存队列的缓存空间有限；因此，第一缓存队列中可能并未缓存时间信息与第二图像的时间信息相同的第一图像。换言之，第一缓存队列中缓存的第一图像的时间信息与第二图像的时间信息可能相差较多。在这种情况下，第一缓存队列中缓存的第一图像的图像内容与第二图像的图像内容可能会存在较大差异。因此，在这种情况下，直接将第一缓存队列中时间与第二图像的时间最近的第一图像(或者第一缓存队列中清晰度最高的第一图像)作为用于编码生成精彩图像的图像(即S606和S607中所述的第四图像)，则可能无法准确抓拍到与第二图像对应的精彩图像。
基于此，在一些实施例中，手机仍然可以按照上述实施例所述的方法，依据第二图像从第一缓存队列中选择出S605a-S605b所述的第五图像。但是，S605a-S605b所述的第五图像不能再作为用于生成精彩图像的基础图像，而是将第二图像作为用于生成精彩图像的基础图像，包括S606和S607中所述的第五图像的m帧第一图像则可以用于对第二图像进行画质增强。
具体的，如图21所示，本申请实施例的方法可以包括S601-S605和S2101-S2104。其中，S601-S605的详细描述可以参考上述实施例中的介绍，这里不再赘述。
S2101、手机采用ISP对第三图像执行第三处理得到第六图像。该第三处理包括RAW域和RGB域的图像处理。
需要说明的是，S2101所述的第三处理与S606所示的第二处理不同。S606所述的第二处理可以包括RAW域、RGB域和YUV域的图像处理；而S2101所述的第三处理包括RAW域和RGB域的图像处理，不包括YUV域的图像处理。因此，手机执行S2101得到的第六图像与手机执行S606得到的第四图像不同。
S2102、手机采用第六图像对第二图像进行画质增强，得到第七图像。
请参考图22，其示出本申请实施例提供的一种录像过程中的图像处理原理示意图。如图22所示，手机的图像传感器(Sensor)受到曝光的控制，可以不断输出Bayer图像(即第一图像)。图22中对应S2101-S2105的内容介绍，可以参考上述实施例中对S601-S605的详细描述，这里不予赘述。图22中生成并显示预览图像以及生成录像文件的方法，可以参考上述实施例中的相关描述，这里不予赘述。
不同的是，如图22所示，手机采用预设RAW域图像处理算法处理m帧第一图像得到第三图像之后，该手机不会直接采用ISP处理第三图像得到用于编码生成精彩图像的图像。结合上述描述，考虑到第一缓存队列中缓存的第一图像的图像内容与第二图像的图像内容可能会存在较大差异。因此，该实施例中不会直接将第一缓存队列中时间与第二图像的时间最近的第一图像(或者第一缓存队列中清晰度最高的第一图像) 作为用于编码生成精彩图像的图像。而是将第二图像作为用于生成精彩图像的基础图像。之后，手机可以采用图像质量更高的第六图像对第二图像进行画质增强，得到分辨较高且图像内容与第二图像的图像内容基本相同的图像。
具体的，如图22所示，手机可以对第三图像执行RAW的图像处理和RGB域的图像处理，得到第六图像。应理解，第三图像是手机采用预设RAW域图像处理算法处理m帧第一图像得到的，而第六图像是手机采用ISP处理第三图像得到的；因此，相比于采用ISP处理得到的第二图像，第六图像可能会包括更多的图像细节。由此，如图22所示，手机可以将第六图像作为参考(Refence，REF)图像，采用该第六图像(即REF图像)对第二图像进行画质增强，以得到图像质量更高的第七图像。
S2103、手机采用ISP对第七图像执行第四处理得到第八图像。该第四处理包括YUV域的图像处理，该第四处理不包括下采样。
本申请实施例中，手机可以通过时分复用的方式，采用ISP处理第一图像得到预览图像，处理第三图像得到第六图像，处理第七图像得到第八图像。也就是说，手机采用ISP处理第一图像得到预览图像，并不会影响手机采用ISP处理第三图像得到第六图像，处理第七图像得到第八图像。该第八图像用于生成录像过程中的精彩图像。换言之，手机处理图22所示的精彩图像，并不会影响手机处理图22所示的预览图像和录像文件。
S2104、手机在录像结束后，展示对第八图像进行编码得到的精彩图像。
其中，如图22所示，手机可以采用ISP处理第七图像得到第八图像(即执行S2103)；之后，手机可以采用编码器2对第八图像进行编码得到的精彩图像。并且，手机可以在录像结束后，向用户展示该精彩图像。
需要说明的是，手机执行S2104展示对第八图像进行编码得到的精彩图像的方法，可以参考上述实施例中，手机执行S607展示对第四图像进行编码得到的精彩图像的方法，本申请实施例这里不予赘述。
在一些实施例中，在S604之后，手机可以执行S605a从第一缓存队列缓存中查找到时间信息与第二图像的时间信息所指示的时间的差值在预设时长内的第一图像。
如果手机执行S605a从第一缓存队列缓存中查找到上述图像，手机则将查找到的图像作为第五图像。然后，手机可以执行S605b和S606-S607。
如果手机执行S605a不能从第一缓存队列缓存中查找到上述图像，手机则可以将时间信息与第二图像的时间信息所指示的时间最近的一帧第一图像作为第五图像。然后，手机可以执行S605b和S2101-S2104。
在一些实施例中，上述S2102中所述的画质增强可以包括图像融合。例如，图22所示的画质增强可以替换为图23所示的图像融合。在该实施例中，S2102可以替换为S2102′。S2102′：手机对第六图像和第二图像进行图像融合，得到第七图像。在该实施例中，S2101所述的第三处理可以包括下采样，也可以不包括下采样。
在S2101所述的第三处理包括下采样的情况下，手机采用ISP对第三图像执行第三处理得到的第六图像的分辨率低于第三图像的分辨率。由于第二图像是经过包括下采样发第一处理后的图像；因此，第二图像的分辨率也会相对较低。在这种情况下，第六图像与第二图像可能都是分辨率较低的图像。当然，第六图像的分辨与与第二图 像的分辨率可能相同，也可能不同。
虽然第六图像与第二图像可能都是分辨率较低的图像；但是，由于第三图像是手机采用预设RAW域图像处理算法处理m帧第一图像得到的，而第六图像是手机采用ISP处理第三图像得到的；因此，相比于采用ISP处理得到的第二图像，第六图像可能会包括更多的图像细节。因此，本实施例中，手机执行S1202，可以将第六图像中较为丰富的图像细节补充至第二图像中，得到第七图像。这样，可以提升第七图像的图像质量。
示例性的，手机可以通过融合网络(也称为图像融合网络)，利用上述第六图像对第二图像进行画质增强得到第七图像。其中，手机通过融合网络进行图像增强的方法，可以参考常规技术中的相关方法，本申请实施例这里不予赘述。
在一些实施例中，手机执行S2102之前，可以对上述第六图像和第二图像进行配准。之后，手机可以利用配准后的第六图像，对配准后的第二图像进行图像融合(Fusion)。其中，手机进行融合(Fusion)之前，对第二图像和第六图像进行配准，可以提升手机进行画质增强的成功率和效果。
一般而言，配准可以包括两种方式：全局配准和局部配准。
全局配准一般使用特征点检测和匹配。以手机对第六图像和第二图像进行配准为例。手机可以检测第六图像和第二图像中匹配的特征点(如像素点)。然后，手机可以筛选匹配的特征点。如果匹配的特征点中好的特征点个数大于预设阈值1，则手机可以认为全局配准效果较好，可以进行融合。
局部配准一般使用光流法。以手机对第六图像和第二图像进行配准为例。手机可以先对第六图像和第二图像计算光流。然后，手机可以将经过光流配准变换后的第二图像，与经过光流配准变换后的第六图像做差。如果差异小于预设阈值2，则手机可以认为局部配准效果较好，可以融合。
在另一些实施例中，手机对第六图像和第二图像进行配准之前，可以先对比第六图像和第二图像的纹理相似度。如果第六图像和第二图像的纹理相似度高于预设相似度阈值，则表示第六图像和第二图像的纹理相似度较高。在这种情况下，手机对第六图像和第二图像进行配准的成功率较高。采用本方案，可以提升手机配准的成功率。
如果第六图像和第二图像的纹理相似度低于或等于预设相似度阈值，则表示第六图像和第二图像的纹理相似度较低。在这种情况下，手机则不会对第六图像和第二图像进行配准。这样，减少无效配准影响手机功耗。在这种情况下，手机可以直接将对将第六图像作为第七图像。
示例性的，上述画质增强可以实现噪声去除、清晰度提升、动态范围(Dynamic Range)的改变或拓展、图像超分辨等功能。
在另一些实施例中，上述S2102中所述的画质增强可以包括图像超分辨。例如，图22所示的画质增强可以替换为图24所示的图像超分辨。在该实施例中，S2102可以替换为S2102〃。S2102〃：手机采用第六图像对第二图像进行图像超分辨，得到第七图像。在该实施例中，S2101所述的第三处理可以不包括下采样。
在S2101所述的第三处理不包括下采样的情况下，第六图像的分辨率高于第三图像的分辨率。如图24所示，第六图像是高分辨率(high resolution，LR)图像。一 般而言，虑到功耗和存储空间等因素，录像(即录制视频)所选择的分辨率相比于Sensor出图的分辨率较低。因此，在录像过程中ISP对Sensor输出的图像是进行了下采样的。对图像进行下采样，可以降低图像的分辨率。如此，如图24所示，录像文件中的第二图像是LR图像。而用户希望在录像过程中获得的精彩图像是高分辨率的图像。基于此，手机可以将第六图像作为引导图像，对第二图像进行画质增强(包括图像超分辨)，可以提升第二图像的分辨率。
例如，假设第二图像的分辨率为1080p，上述第六图像的分辨率为4k。手机执行S2101，可以将分辨率为4k的第六图像作为引导图像，对分辨率为1080p的第四图像进行画质增强。画质增强后的第二图像(即第七图像)的分辨率可以为4k。
示例性的，本申请实施例中所述的图像超分辨可以是基于参考帧的图像超分辨(Refence super resolution，REF SR)。
在另一些实施例中，如果第六图像的分辨率高于第二图像的分辨率，手机可以执行S2102〃；如果第六图像的分辨率低于或等于第二图像的分辨率，手机可以执行S2102′。
本申请实施例中，即使由于第一缓存队列中并未缓存时间信息与第二图像的时间信息相同的第一图像，导致第一缓存队列中缓存的第一图像的图像内容与第二图像的图像内容存在较大差异。手机也可以将第二图像作为用于生成精彩图像的基础图像，采用经过预设RAW域图像处理算法处理第一缓存队列中第一图像得到的图像质量更高的图像作为引导图像，提升第二图像的图像质量。
在本申请一实施例中，该方法可以应用于手机录像结束后的精彩帧识别和生成。在手机录像过程中，手机可以自动选择出录像中的满足要求的参考(REF)图像，并将其缓存在第二缓存队列中。录像结束后，手机可以响应于用户对录像文件中任一一帧图像的选择操作，从录像文件中确定出备选图像，该备选图像可以是录像文件中用户偏好的一帧图像。之后，手机可以采用上述REF对图像备选图像进行画质增强，用来生成精彩图像并呈现给用户。
其中，手机录像过程中生成并缓存REF图像的动作，以及手机录像结束后生成精彩图像的动作对用户不可见。手机录像结束后，不仅可以为用户呈现录制的视频，还可以呈现录像中的精彩图像，实现一拍多得的效果。
具体的，如图25所示，本申请实施例的方法可以包括S601-S605和S2501-S2506。其中，S601-S605的详细描述可以参考上述实施例中的介绍，这里不再赘述。
S2501、手机采用ISP对第三图像执行第三处理得到第六图像，并将第六图像缓存在第二缓存队列中。该第三处理包括RAW域和RGB域的图像处理，该第三处理不包括下采样。
其中，S2501所述的第三处理与S606所示的第二处理不同。S606所述的第二处理可以包括RAW域、RGB域和YUV域的图像处理；而S2101所述的第三处理包括RAW域和RGB域的图像处理，不包括YUV域的图像处理。因此，手机执行S2501得到的第六图像与手机执行S606得到的第四图像不同。第二缓存队列中可以缓存多帧第六图像。
一方面，S2501所述的第三处理不包括下采样。因此，缓存在第二缓存队列中的第六图像是高分辨率的HR图像。并且，该第六图像是经过预设RAW域图像处理算法处 理过的图像细节较为丰富、图像质量较高的图像。
另一方面，上述第六图像是手机执行S605和S2501，基于第二图像生成的。第二图像是录像文件中包括目标对象的图像。由此可见，该第六图像中也包括目标对象的图像。如此，手机录像结束后，用户想要从录像文件中筛选出第六图像对应的一帧图像作为精彩图像的可能性较大。
因此，本申请实施例中，手机可以将多帧第六图像缓存在第二缓存队列中，用于当出现从录像文件中筛选出一帧图像作为精彩图像的用户需求时，可以采用第二缓存队列中的第六图像对这一帧图像进行画质增强(包括图像超分辨)，以得到图像质量较高(包括分辨率较高)的精彩图像。
S2502、手机响应于结束录像的操作，生成录像文件。
示例性的，响应于用户对图7所示的“开始录像”按钮702的单击操作(即第一操作)，手机可以执行S601-S605和S2501。例如，S2502中结束录像的操作可以为用户对图7或图16所示的“结束录像”按钮706的点击操作。手机响应于用户对图7或图16所示的“结束录像”按钮706的点击操作，可结束录像并生成录像文件。例如，手机响应于用户对图16所示“结束录像”按钮706的点击操作，可以显示图16所示的录像的取景界面1601。录像的取景界面1601是手机未开始录像的取景界面。与图7所示的录像的取景界面701相比，手机的取景界面中的照片选项中的照片由图7所示的708更新为图16所示的1602。
手机可以响应于用户对相册应用的启动操作，显示图17中的(a)所示的相册列表界面1701所示的相册列表界面1703。该相册列表界面1701包括手机中可以保存的一个或多个照片和视频。例如，相册列表界面1701和相册列表界面1703包括手机录制的录像文件1702。
在该实施例中，手机录像结束时还未生成精彩图像。因此，如图7所示，相册列表界面1701不包括图17中的(b)所示的照片1704。
S2503、手机播放录像文件的过程中，响应于用户对录像文件中一帧图像的选择操作，从录像文件中确定备选图像。
其中，相比于第二图像中缓存的第六图像的分辨率，录像文件中的图像(包括备选图像)的分辨率较低。如图26所示，第六图像是HR图像，备选图像是LR图像。S2503的详细介绍可以参考以下实施例中的描述，这里不予赘述。
S2504、手机从第二缓存队列中选择与备选图像对应的参考(REF)图像，并采用REF图像对备选图像进行画质增强，得到第九图像。
其中，在第二缓存队列中缓存的多帧第六图像中，REF图像由摄像头的图像传感器输出的时间与备选图像由摄像头的图像传感器输出的时间最近。其中，第六图像的时间信息与第三图像的时间信息相同。该第三图像的时间信息与m个第一图像中第五图像的时间信息相同。
本申请实施例中，手机可以将画质高且分辨率高的第六图像缓存在第二缓存队列Buffer中。该第二缓存队列可以缓存多帧第六图像。如此，当出现从录像文件中筛选出一帧图像作为精彩图像的用户需求时，可以将第二缓存队列中时间与这一帧图像的时间最近的第六图像作为引导图像，对这一帧图像进行画质增强(包括图像超分辨)， 以得到图像质量较高的精彩图像。
其中，第二缓存队列中缓存的第六图像是HR图像，该REF图像也是HR图像。因此，采用REF图像对备选图像进行画质增强得到的第九图像也是HR图像。
示例性的，S2504所述的画质增强可以包括图像超分辨。例如，该图像超分辨可以是REF SR。其中，REF SR的详细描述可以参考上述实施例中的介绍，这里不予赘述。
在另一些实施例中，S2504所述的画质增强可以包括图像融合。其中，图像融合的详细描述可以参考上述实施例中的介绍，这里不予赘述。
S2505、手机采用ISP对第九图像执行第四图像得到第十图像。该第四图像包括YUV域的图像处理，该第四处理不包括下采样。
本申请实施例中，手机可以通过时分复用的方式，采用ISP处理第一图像得到预览图像，处理第三图像得到第六图像，处理第九图像得到第十图像。也就是说，手机采用ISP处理第一图像得到预览图像，并不会影响手机采用ISP处理第三图像得到第六图像，处理第九图像得到第十图像。该第十图像用于生成录像过程中的精彩图像。换言之，手机处理图26所示的精彩图像，并不会影响手机处理图26所示的预览图像和录像文件。
S2506、手机展示对第十图像进行编码得到的精彩图像。
请参考图26，其示出本申请实施例提供的一种录像过程中的图像处理原理示意图。如图26所示，手机的图像传感器(Sensor)受到曝光的控制，可以不断输出Bayer图像(即第一图像)。图26中对应S2501-S2505的内容介绍，可以参考上述实施例对S601-S605的详细描述，这里不予赘述。图26中生成并显示预览图像以及生成录像文件的方法，可以参考上述实施例中的相关描述，这里不予赘述。
手机可以执行S2501处理第三图像得到图26所示的第六图像，并将第六图像缓存在图26所示的第二缓存队列中。录像结束后，手机可生成图26所示的录像文件。在手机播放图26所示的录像文件的过程中，手机可以执行S2503接收用户对录像文件中一帧图像的，从录像文件中确定出图26所示的备选图像。然后，手机可以执行S2504从第二缓存队列中选择出图26所示的REF图像，并采用REF图像对备选图像进行图26所示的画质增强，得到第九图像。之后，手机可执行S2505，处理第九图像得到图26所示的第十图像。最后，手机可采用图26所示的编码器2对第十图像信息编码，执行S2506展示编码得到的精彩图像。
需要说明的是，手机执行S2506展示对第十图像进行编码得到的精彩图像的方法，可以参考上述实施例中，手机执行S607展示对第四图像进行编码得到的精彩图像的方法，本申请实施例这里不予赘述。
手机执行S601-S607所述的方案，可以在手机录像过程中获取精彩图像，并在录像结束后展示精彩图像。换言之，采用S601-S607所述的方案，可实现手机在线录像的一拍多得。一拍多得是指手机录像结束后不仅可以得到录像文件，还可以得到录像中抓拍的精彩图像。
手机执行S601-S605和S2501-S2506，可以在录像过程中获取可用于支持手机获取精彩图像的多帧引导图像并缓存起来；之后，手机录像结束后播放录像文件的过程中，可采用缓存的引导图像对用户选择的备选图像进行画质增强，得到精彩图像。换 言之，采用S601-S605和S2501-S2506所述的方案，可在录像结束后，通过离线方式实现手机录像的一拍多得功能。
需要说明的是，手机可以通知支持上述在线方案和离线方案。也就是说，在一些实施例中，本申请实施例的方法可以包括S601-S605、S606-S607/S2101-S2104，以及S2501-S2506。
示例性的，本申请实施例这里介绍S2503中所述的“用户对录像文件中一帧图像的选择操作”，以及手机响应于该选择操作从录像文件中确定备选图像的方法。上述用户对录像文件中一帧图像的选择操作的实现方式，至少可以包括以下方式(A)-方式(E)：
方式(A)：用户对录像文件中一帧图像的选择操作,可以是用户对该录像文件的详情页中第二预设控件的点击操作(如单击操作、双击操作或者长按操作等任一种操作)。该第二预设控件的用于触发手机自动生成录像文件中的一帧或多帧精彩图像。该录像文件中一帧或多帧精彩图像的数量与第二缓存队列中缓存的第六图像的数量相同。
具体的，S2503可以包括：响应于用户对第二预设控件的点击操作，手机可以从录像文件中确定出第二缓存队列中缓存的每一帧第六图像对应的一帧图像。其中，每一帧第六图像对应的一帧图像均为备选图像。第六图像的时间信息和与该第六图像对应的一帧图像的时间信息相同。在该方式(A)中，S2504中与备选图像对应的REF图像具体为该备选图像对应的第六图像。
例如，响应于用户对图17中的(a)所示的录像文件1702的点击操作，如图27中的(a)所示，手机可显示录像文件1702的详情页2701。如图27中的(a)所示，该详情页2701包括第二预设控件2702。假设第二缓存队列中缓存了两帧第六图像，响应于用户对该第二预设控件2702的点击操作，如图27中的(b)所示，手机可显示拍摄录像文件1702过程中抓拍的精彩图像的列表页2703，列表页2703包括两帧精彩图像。假设第二缓存队列中缓存了一帧第六图像，响应于用户对该第二预设控件2702的点击操作，如图27中的(c)所示，手机可显示拍摄录像文件1702过程中抓拍的精彩图像的详情页2704，向用户展示该精彩图像。
方式(B)：用户对录像文件中一帧图像的选择操作,可以是手机播放录像文件的过程中，用户对该录像文件的播放界面中第三预设控件的点击操作(如单击操作、双击操作或者长按操作等任一种操作)。手机播放录像文件的过程中，当手机播放到用户感兴趣的图像，用户则可以点击第三预设控件。手机响应于用户对第三预设控件的点击操作，将用户点击第三预设控件时手机播放的一帧图像(或者在这一帧图像之前播放，与这一帧图像间隔p帧的图像)作为备选图像。
例如，手机可以接收图28中的(a)所示的录像文件的详情页中播放按钮的点击操作。响应于用户对录像文件的详情页中播放按钮的点击操作，手机可以开始播放录像文件。如，手机可显示图28中的(b)或图28中的(c)所示的录像文件的播放界面。图28中的(b)或图28中的(c)所示的录像文件的播放界面均包括第三预设控件2801。本申请实施例对第三预设控件2801在录像文件的播放界面中的位置不做限制。
响应于用户对图28中的(b)或图28中的(c)所示的第三预设控件2801的点击操作，手机可显示图27中的(c)所示的精彩图像的详情页2704，或者，手机可显示图29所示的提示信息2901。该提示信息2901用于请求用户确认是否保存精彩图像，如提示信息2901可以为“为您生成一张精彩图像，请确认是否保存？”。该提示信息2901中还可以预览精彩图像。响应于用户对提示信息2901中“是”按钮的点击操作，手机可在相册中保存该精彩图像；响应于用户对提示信息2901中“否”按钮的点击操作，手机不会保存精彩图像，手机继续播放录像文件。
方式(C)：用户对录像文件中一帧图像的选择操作,可以包括手机播放录像文件的过程中，用户输入的暂停播放的操作和暂停播放后放大一帧图像的操作。
示例性的，上述暂停播放的操作可以是用户对录像文件的播放界面中暂停播放按钮的点击操作。上述暂停播放的操作还可以是用户对录像文件的播放界面中进度条的调整操作。手机响应于用户对进度条的调整操作，可以显示进度条当前状态下对应录像文件中的一帧图像。
以暂停播放的操作是用户对录像文件的播放界面中暂停播放按钮的点击操作为例。手机播放录像文件的过程中，当手机播放到用户感兴趣的图像，用户则可以点击暂停按钮控制手机暂停播放录像文件，然后用户可以控制手机放大当前帧图像。手机响应于上述暂停播放的操作和放大图像的操作，可以将用户放大的一帧图像作为备选图像。
方式(D)：用户对录像文件中一帧图像的选择操作,可以为手机播放录像文件的过程中，用户反复查看一个片段的操作。一个片段可以包括多帧图像。手机播放录像文件的过程中，如果接收到用户反复查看录像文件中一个片段的操作，手机则可以从该片段中选择出一帧图像作为备选图像。其中，手机则可以从该片段中选择出一帧图像作为备选图像的方法，可以参考上述实施例中，手机从第一视频流中选择出第二图像的方法。或者，备选图像可以是该片段中清晰度和完整度最高的一帧图像。
方式(E)：用户对录像文件中一帧图像的选择操作,可以包括手机播放录像文件的过程中，用户输入的暂停播放的操作。
示例性的，上述暂停播放的操作可以是用户对录像文件的播放界面中暂停播放按钮的点击操作。上述暂停播放的操作还可以是用户对录像文件的播放界面中进度条的调整操作。手机响应于用户对进度条的调整操作，可以显示进度条当前状态下对应录像文件中的一帧图像。
以暂停播放的操作是用户对进度条的调整操作为例。手机播放录像文件的过程中，用户可以调整录像文件的播放界面中的进度条，以控制手机显示录像文件中用户感兴趣的一帧图像。手机可以接收用户对还进度条的调整操作，根据进度条的变化显示录像文件中的图像。当手机检测到用户手指离开进度条后，手机可以将用户手指离开进度条时，该进度条对应录像文件中的一帧图像作为备选图像。
需要说明的是，S2503中“用户对录像文件中一帧图像的选择操作”，以及手机响应于该选择操作从录像文件中确定备选图像的方法，包括但不限于方式(A)-方式(D)所示的方法，其他的方法本申请实施例这里不予赘述。
本申请另一些实施例提供了一种电子设备，该电子设备可以包括：上述显示屏、摄像头、存储器和一个或多个处理器。该显示屏、摄像头、存储器和处理器耦合。该 存储器用于存储计算机程序代码，该计算机程序代码包括计算机指令。当处理器执行计算机指令时，电子设备可执行上述方法实施例中手机执行的各个功能或者步骤。该电子设备的结构可以参考图4所示的手机的结构。
本申请实施例还提供一种芯片系统，如图30所示，该芯片系统3000包括至少一个处理器3001和至少一个接口电路3002。处理器3001和接口电路3002可通过线路互联。例如，接口电路3002可用于从其它装置(例如电子设备的存储器)接收信号。又例如，接口电路3002可用于向其它装置(例如处理器3001)发送信号。示例性的，接口电路3002可读取存储器中存储的指令，并将该指令发送给处理器3001。当所述指令被处理器3001执行时，可使得电子设备执行上述实施例中的各个步骤。当然，该芯片系统还可以包含其他分立器件，本申请实施例对此不作具体限定。
本申请实施例还提供一种计算机存储介质，该计算机存储介质包括计算机指令，当所述计算机指令在上述电子设备上运行时，使得该电子设备执行上述方法实施例中手机执行的各个功能或者步骤。
本申请实施例还提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机执行上述方法实施例中手机执行的各个功能或者步骤。
通过以上实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等 各种可以存储程序代码的介质。
以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (28)

  1. 一种获取图像的方法，其特征在于，应用于电子设备，所述方法包括：
    所述电子设备接收用户的第一操作；其中，所述第一操作用于触发开始录制视频；
    响应于所述第一操作，所述电子设备显示第一界面；其中，所述第一界面是所述电子设备正在录制视频的取景界面，所述第一界面包括预览图像，所述预览图像是所述电子设备接收到所述第一操作后由所述电子设备的摄像头采集的第一图像得到的；
    所述电子设备从第一视频流中选择出第二图像；其中，所述第二图像包括目标对象的图像；所述第一视频流是所述电子设备的图像信号处理器ISP对多帧第一图像执行第一处理得到的，所述第一处理包括下采样；
    所述电子设备在第一缓存队列缓存所述摄像头采集的第一图像；其中，所述第一缓存队列缓存所述摄像头采集的n帧第一图像，n≥1，n为整数；
    所述电子设备将所述n帧第一图像中与所述第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，所述预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；
    所述电子设备采用ISP对所述第三图像执行第二处理得到第四图像；其中，所述第二处理不包括所述下采样；
    所述电子设备在录像结束后，展示对所述第四图像进行编码得到的精彩图像。
  2. 根据权利要求1所述的方法，其特征在于，所述电子设备在录像结束后，展示对所述第四图像进行编码得到的精彩图像，包括：
    所述电子设备录像结束后，响应于用户的操作显示录像文件的详情页，所述录像文件的详情页包括第一预设控件；
    所述电子设备响应于用户对所述第一预设控件的点击操作，展示所述电子设备录制所述录像文件的过程中抓拍的所述精彩图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述电子设备在录像结束后，展示对所述第四图像进行编码得到的精彩图像，包括：
    所述电子设备录像结束后，响应于用户的操作显示录像文件的详情页；
    所述电子设备响应于用户在所述录像文件的详情页输入的预设手势，展示所述电子设备录制所述录像文件的过程中抓拍的所述精彩图像。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，n≥2。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述电子设备将所述n帧第一图像中与所述第二图像对应的k帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像，包括：
    所述电子设备从所述n帧第一图像选择出与所述第二图像对应的第五图像；
    所述电子设备将所述n帧第一图像中的m帧第一图像作为输入，运行所述预设RAW域图像处理算法得到所述第三图像；其中，所述m帧第一图像包括所述第五图像；所述m帧第一图像中除所述第五图像的其他图像，用于对所述第五图像进行画质增强；
    其中，在所述n帧第一图像中，所述第五图像由所述摄像头的图像传感器输出的时间与所述第二图像由所述摄像头的图像传感器输出的时间最近；或者，所述n帧第一图像中，所述第五图像的清晰度最高。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，
    所述目标对象包括人脸，所述第二图像为所述目标对象的表情满足预设条件的图像，所述目标对象的表情满足预设条件包括人脸微笑；和/或，
    所述目标对象包括人体，所述第二图像为所述目标对象的动作为预设动作的图像。
  7. 根据权利要求1-6中任一项所述的方法，其特征在于，所述目标对象在所述电子设备出厂时，预先配置在所述电子设备中；和/或，
    所述目标对象是所述电子设备接收用户在所述电子设备的设置界面设置的。
  8. 根据权利要求1-6中任一项所述的方法，其特征在于，在所述电子设备从第一视频流中选择出第二图像之前，所述方法还包括：
    所述电子设备显示第二界面；其中，所述第二界面是所述电子设备还未开始录制视频的取景界面，所述第二界面包括的预览图像是所述电子设备接收到所述第一操作前由所述电子设备的摄像头采集的图像得到的；所述第二界面还包括开始录像按钮和预设功能的开关；所述第一操作是对所述开始录像按钮的点击操作；
    所述电子设备响应于用户对所述第二界面中所述预设功能的开关的第二操作，显示所述目标对象的设置界面，所述目标对象的设置界面包括多个待选择的拍摄对象的选项；
    所述电子设备响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
  9. 根据权利要求1-6中任一项所述的方法，其特征在于，所述第一界面还包括：预设功能的开关；在所述电子设备从第一视频流中选择出第二图像之前，所述方法还包括：
    所述电子设备响应于用户对所述第一界面中所述预设功能的开关的第二操作，显示所述目标对象的设置界面，所述目标对象的设置界面包括多个待选择的拍摄对象的选项；
    所述电子设备响应于用户对多个待选择的拍摄对象的选项中至少一个拍摄对象的的选项的选择操作，将用户选择的拍摄对象确定为目标对象。
  10. 根据权利要求1-9中任一项所述的方法，其特征在于，所述电子设备从第一视频流中选择出第二图像，包括：
    所述电子设备周期性地从所述第一视频流中选择出所述第二图像。
  11. 根据权利要求1-10中任一项所述的方法，其特征在于，m≥2。
  12. 根据权利要求1-11中任一项所述的方法，其特征在于，所述第一处理包括RAW域的图像处理和RGB域的图像处理，所述第二处理包括RAW域的图像处理、RGB域的图像处理和YUV域的图像处理。
  13. 根据权利要求1-12中任一项所述的方法，其特征在于，所述电子设备采用ISP对所述第三图像执行第二处理得到第四图像，包括：
    若所述第一缓存队列中包括时间信息与所述第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，所述电子设备采用所述ISP对所述第三图像执行所述第二处理得到第四图像。
  14. 根据权利要求13所述的方法，其特征在于，所述方法还包括：
    若所述第一缓存队列中不包括时间信息与所述第二图像的时间信息所指示的时间的差值在预设时长内的第一图像，所述电子设备采用所述ISP对所述第三图像执行第三处理得到第六图像；其中，所述第三处理包括RAW域的图像处理和RGB域的图像处理；
    所述电子设备采用所述第六图像对所述第二图像进行画质增强，得到第七图像；
    所述电子设备采用所述ISP对所述第七图像执行第四处理得到第八图像，所述第四处理包括YUV域的图像处理，所述第四处理不包括下采样；
    所述电子设备在录像结束后，展示对所述第八图像进行编码得到的精彩图像。
  15. 根据权利要求1-14中任一项所述的方法，其特征在于，所述第三处理不包括下采样；所述方法还包括：
    所述电子设备将所述第六图像缓存在第二缓存队列中；
    所述电子设备响应于结束录像的操作，生成录像文件；其中，所述录像文件包括由多帧第一图像生成的图像；
    所述电子设备播放所述录像文件的过程中，响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像；
    所述电子设备从所述第二缓存队列缓存的多帧第六图像中选择与所述备选图像对应的参考REF图像，并采用所述REF图像对所述备选图像进行画质增强，得到第九图像；
    所述电子设备采用所述ISP对所述第九图像执行第四处理得到第十图像，所述第四处理包括YUV域的图像处理，所述第四处理不包括下采样；
    所述电子设备展示对所述第十图像进行编码得到的精彩图像。
  16. 一种获取图像的方法，其特征在于，应用于电子设备，所述方法包括：
    所述电子设备接收用户的第一操作；其中，所述第一操作用于触发开始录制视频；
    响应于所述第一操作，所述电子设备显示第一界面；其中，所述第一界面是所述电子设备正在录制视频的取景界面，所述第一界面包括预览图像，所述预览图像是所述电子设备接收到所述第一操作后由所述电子设备的摄像头采集的第一图像得到的；
    所述电子设备从第一视频流中选择出第二图像；其中，所述第二图像包括目标对象的图像；所述第一视频流是所述电子设备的图像信号处理器ISP对多帧第一图像执行第一处理得到的，所述第一处理包括下采样；
    所述电子设备在第一缓存队列缓存所述摄像头采集的第一图像；其中，所述第一缓存队列缓存所述摄像头采集的n帧第一图像，n≥1，n为整数；
    所述电子设备将所述n帧第一图像中与所述第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，所述预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；
    所述电子设备采用所述ISP对所述第三图像执行第三处理得到第六图像；其中，所述第三处理包括RAW域的图像处理和RGB域的图像处理；
    所述电子设备采用所述第六图像对所述第二图像进行画质增强，得到第七图像；
    所述电子设备采用所述ISP对所述第七图像执行第四处理得到第八图像，所述第四处理包括YUV域的图像处理，所述第四处理不包括下采样；
    所述电子设备在录像结束后，展示对所述第八图像进行编码得到的精彩图像。
  17. 根据权利要求16所述的方法，其特征在于，所述第一处理包括RAW域的图像处理和RGB域的图像处理。
  18. 根据权利要求16或17所述的方法，其特征在于，所述电子设备采用所述第六图像对所述第二图像进行的画质增强包括：图像融合和/或图像超分辨；
    其中，所述图像超分辨包括基于参考帧的图像超分辨REF-SR。
  19. 一种获取图像的方法，其特征在于，应用于电子设备，所述方法包括：
    所述电子设备接收用户的第一操作；其中，所述第一操作用于触发开始录制视频；
    响应于所述第一操作，所述电子设备显示第一界面；其中，所述第一界面是所述电子设备正在录制视频的取景界面，所述第一界面包括预览图像，所述预览图像是所述电子设备接收到所述第一操作后由所述电子设备的摄像头采集的第一图像得到的；
    所述电子设备从第一视频流中选择出第二图像；其中，所述第二图像包括目标对象的图像；所述第一视频流是所述电子设备的图像信号处理器ISP对多帧第一图像执行第一处理得到的，所述第一处理包括下采样；
    所述电子设备在第一缓存队列缓存所述摄像头采集的第一图像；其中，所述第一缓存队列缓存所述摄像头采集的n帧第一图像，n≥1，n为整数；
    所述电子设备将所述n帧第一图像中与所述第二图像对应的m帧第一图像作为输入，运行预设原始RAW域图像处理算法得到第三图像；其中，所述预设RAW域图像处理算法具备提升图像画质的功能；m≥1，m为整数；
    所述电子设备采用所述ISP对所述第三图像执行第三处理得到第六图像，并将所述第六图像缓存在第二缓存队列中，所述第三处理包括RAW域的图像处理和RGB域的图像处理，所述第三处理不包括下采样；
    所述电子设备响应于结束录像的操作，生成录像文件；其中，所述录像文件包括由多帧第一图像生成的图像；
    所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像；
    所述电子设备从所述第二缓存队列缓存的多帧第六图像中选择与所述备选图像对应的参考REF图像，并采用所述REF图像对所述备选图像进行画质增强，得到第九图像；
    所述电子设备采用所述ISP对所述第九图像执行第四处理得到第十图像，所述第四处理包括YUV域的图像处理，所述第四处理不包括下采样；
    所述电子设备展示对所述第十图像进行编码得到的精彩图像。
  20. 根据权利要求19所述的方法，其特征在于，所述录像文件的详情页包括第二预设控件；所述对所述录像文件中一帧图像的选择操作为所述对所述第二预设控件的点击操作；
    所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像，包括：
    所述电子设备响应于用户对所述录像文件的详情页中所述第二预设控件的点击操作，从所述录像文件中选择与所述第二缓存队列中缓存的每一帧第六图像对应的图像， 作为所述备选图像。
  21. 根据权利要求19或20所述的方法，其特征在于，所述录像文件的播放界面包括第三预设控件；所述对所述录像文件中一帧图像的选择操作为所述对所述第三预设控件的点击操作；
    所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像，包括：
    所述电子设备在播放所述录像文件的过程中，响应于用户对所述录像文件的播放界面中所述第三预设控件的点击操作，将用户点击所述第三预设控件时所述电子设备播放的一帧图像作为所述备选图像。
  22. 根据权利要求19-21中任一项所述的方法，其特征在于，所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像，包括：
    所述电子设备在播放所述录像文件的过程中，响应于暂停播放的操作和放大图像的操作，将所述放大图像的操作对应的一帧图像作为所述备选图像；
    其中，所述放大图像的操作用于触发所述电子设备放大所述电子设备暂停播放所述录像文件时，所述电子设备所显示的所述录像文件中的一帧图像。
  23. 根据权利要求19-22中任一项所述的方法，其特征在于，所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像，包括：
    所述电子设备在播放所述录像文件的过程中，接收用户对所述录像文件的播放界面中进度条的调整操作，根据所述进度条的变化显示所述录像文件中的图像；
    所述电子设备检测到用户手指离开所述进度条后，将用户手指离开所述进度条时所述进度条对应所述录像文件中的一帧图像作为所述备选图像。
  24. 根据权利要求19-23中任一项所述的方法，其特征在于，所述电子设备响应于用户对所述录像文件中一帧图像的选择操作，从所述录像文件中确定备选图像，包括：
    所述电子设备在播放所述录像文件的过程中，响应于用户反复查看所述录像文件中一个片段的操作，从所述一个片段中选择一帧图像作为所述备选图像。
  25. 根据权利要求19-24中任一项所述的方法，其特征在于，所述从所述一个片段中选择一帧图像作为所述备选图像，包括：
    所述电子设备选择所述一个片段中清晰度最高的一帧图像作为所述备选图像。
  26. 一种电子设备，其特征在于，包括：触摸屏、存储器、摄像头、显示屏、一个或多个处理器；所述触摸屏、所述存储器、所述摄像头、所述显示屏与所述处理器耦合；其中，所述存储器中存储有计算机程序代码，所述计算机程序代码包括计算机指令，当所述计算机指令被所述处理器执行时，使得所述电子设备执行如权利要求1-25任一项所述的方法。
  27. 一种计算机可读存储介质，其特征在于，包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1-25中任一项所述的方法。
  28. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得所述计算机执行如权利要求1-25中任一项所述的方法。
PCT/CN2022/115551 2021-09-07 2022-08-29 一种获取图像的方法及电子设备 Ceased WO2023036007A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22866473.6A EP4284009B1 (en) 2021-09-07 2022-08-29 Method for acquiring image, and electronic device
US18/549,046 US20240171701A1 (en) 2021-09-07 2022-08-29 Image obtaining method and electronic device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111044551 2021-09-07
CN202111044551.4 2021-09-07
CN202111649415.8A CN115802148B (zh) 2021-09-07 2021-12-29 一种获取图像的方法及电子设备
CN202111649415.8 2021-12-29

Publications (1)

Publication Number Publication Date
WO2023036007A1 true WO2023036007A1 (zh) 2023-03-16

Family

ID=85473382

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/115551 Ceased WO2023036007A1 (zh) 2021-09-07 2022-08-29 一种获取图像的方法及电子设备

Country Status (4)

Country Link
US (1) US20240171701A1 (zh)
EP (1) EP4284009B1 (zh)
CN (2) CN118264896A (zh)
WO (1) WO2023036007A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116150421A (zh) * 2023-04-23 2023-05-23 深圳竹云科技股份有限公司 图像的展示方法、装置、计算机设备和存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114202000A (zh) * 2020-08-31 2022-03-18 华为技术有限公司 一种业务处理方法及设备
USD992593S1 (en) * 2021-01-08 2023-07-18 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
US12368967B2 (en) * 2023-03-08 2025-07-22 Meta Platforms Technologies, Llc Distributed image signal processor (ISP) system for a head-mounted device
WO2025097338A1 (zh) * 2023-11-08 2025-05-15 影石创新科技股份有限公司 拍摄方法、拍摄设备及图像处理设备
CN119277183B (zh) * 2024-04-11 2025-08-29 荣耀终端股份有限公司 图像抓拍方法、设备及存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102148983A (zh) * 2010-02-08 2011-08-10 杨占昆 一种解决处理高清图像资源占用过高的方法
CN105827951A (zh) * 2016-01-29 2016-08-03 维沃移动通信有限公司 一种运动对象拍照方法及移动终端
CN105827971A (zh) * 2016-03-31 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
CN108401110A (zh) * 2018-03-18 2018-08-14 广东欧珀移动通信有限公司 图像的获取方法、装置、存储介质及电子设备
CN108427938A (zh) * 2018-03-30 2018-08-21 广东欧珀移动通信有限公司 图像处理方法、装置、存储介质和电子设备
CN110049254A (zh) * 2019-04-09 2019-07-23 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
CN110944109A (zh) * 2018-09-21 2020-03-31 华为技术有限公司 一种拍照方法、装置与设备
CN111741211A (zh) * 2019-03-25 2020-10-02 华为技术有限公司 图像显示方法和设备

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5531467B2 (ja) * 2009-07-03 2014-06-25 ソニー株式会社 撮像装置、および画像処理方法、並びにプログラム
KR101797041B1 (ko) * 2012-01-17 2017-12-13 삼성전자주식회사 디지털 영상 처리장치 및 그 제어방법
US10250799B2 (en) * 2014-05-21 2019-04-02 Google Technology Holdings LLC Enhanced image capture
US9715903B2 (en) * 2014-06-16 2017-07-25 Qualcomm Incorporated Detection of action frames of a video stream
US20170127039A1 (en) * 2015-11-02 2017-05-04 Mediatek Inc. Ultrasonic proximity detection system
US10225511B1 (en) * 2015-12-30 2019-03-05 Google Llc Low power framework for controlling image sensor mode in a mobile image capture device
US10547776B2 (en) * 2016-09-23 2020-01-28 Apple Inc. Devices, methods, and graphical user interfaces for capturing and recording media in multiple modes
CN108234870B (zh) * 2017-12-27 2019-10-11 Oppo广东移动通信有限公司 图像处理方法、装置、终端及存储介质
CN108234873A (zh) * 2018-01-10 2018-06-29 北京小米移动软件有限公司 一种获取图像的方法和装置
KR102198337B1 (ko) * 2018-12-07 2021-01-04 삼성전자주식회사 전자 장치, 전자 장치의 제어 방법 및 컴퓨터 판독 매체.
US11064113B1 (en) * 2018-12-27 2021-07-13 Gopro, Inc. Image capture device with an automatic image capture capability
CN109862433A (zh) * 2019-03-27 2019-06-07 努比亚技术有限公司 游戏视频的生成方法、装置、终端及计算机可读存储介质
CN110180181B (zh) * 2019-04-22 2022-12-23 努比亚技术有限公司 精彩时刻视频的截图方法、装置及计算机可读存储介质
KR102933769B1 (ko) * 2021-03-08 2026-03-05 삼성전자주식회사 전자 장치의 촬영 방법 및 그 전자 장치
CN115525188A (zh) * 2022-02-28 2022-12-27 荣耀终端有限公司 拍摄方法和电子设备

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102148983A (zh) * 2010-02-08 2011-08-10 杨占昆 一种解决处理高清图像资源占用过高的方法
CN105827951A (zh) * 2016-01-29 2016-08-03 维沃移动通信有限公司 一种运动对象拍照方法及移动终端
CN105827971A (zh) * 2016-03-31 2016-08-03 维沃移动通信有限公司 一种图像处理方法及移动终端
CN108401110A (zh) * 2018-03-18 2018-08-14 广东欧珀移动通信有限公司 图像的获取方法、装置、存储介质及电子设备
CN108427938A (zh) * 2018-03-30 2018-08-21 广东欧珀移动通信有限公司 图像处理方法、装置、存储介质和电子设备
CN110944109A (zh) * 2018-09-21 2020-03-31 华为技术有限公司 一种拍照方法、装置与设备
US20210227151A1 (en) * 2018-09-21 2021-07-22 Huawei Technologies Co., Ltd. Photographing Method, Apparatus, And Device
CN111741211A (zh) * 2019-03-25 2020-10-02 华为技术有限公司 图像显示方法和设备
CN110049254A (zh) * 2019-04-09 2019-07-23 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4284009A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116150421A (zh) * 2023-04-23 2023-05-23 深圳竹云科技股份有限公司 图像的展示方法、装置、计算机设备和存储介质
CN116150421B (zh) * 2023-04-23 2023-07-18 深圳竹云科技股份有限公司 图像的展示方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
CN118264896A (zh) 2024-06-28
US20240171701A1 (en) 2024-05-23
EP4284009B1 (en) 2025-10-29
EP4284009A1 (en) 2023-11-29
CN115802148A (zh) 2023-03-14
EP4284009A4 (en) 2024-10-09
CN115802148B (zh) 2024-04-12

Similar Documents

Publication Publication Date Title
US11949978B2 (en) Image content removal method and related apparatus
CN115802148B (zh) 一种获取图像的方法及电子设备
CN116320783B (zh) 一种录像中抓拍图像的方法及电子设备
WO2021147482A1 (zh) 一种长焦拍摄的方法及电子设备
CN115802146B (zh) 一种录像中抓拍图像的方法及电子设备
CN115689963B (zh) 一种图像处理方法及电子设备
CN117119285B (zh) 一种拍摄方法
US12335600B2 (en) Method for capturing image during video recording and electronic device
US12556809B2 (en) Shooting method and related device
WO2023231697A1 (zh) 一种拍摄方法及相关设备
WO2024179100A1 (zh) 一种拍摄方法
WO2023035868A1 (zh) 拍摄方法及电子设备
CN116828099B (zh) 一种拍摄方法、介质和电子设备
CN115802147B (zh) 一种录像中抓拍图像的方法及电子设备
WO2026081623A1 (zh) 一种动态照片的处理方法及相关装置
WO2025139057A1 (zh) 图像拍摄方法、电子设备以及计算机可读存储介质
CN119653224A (zh) 视频处理方法、电子设备和计算机可读存储介质

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2022866473

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022866473

Country of ref document: EP

Effective date: 20230824

WWE Wipo information: entry into national phase

Ref document number: 18549046

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2022866473

Country of ref document: EP