WO2024007769A1 - 帧率调节方法、装置、设备及系统 - Google Patents

帧率调节方法、装置、设备及系统 Download PDF

Info

Publication number
WO2024007769A1
WO2024007769A1 PCT/CN2023/096968 CN2023096968W WO2024007769A1 WO 2024007769 A1 WO2024007769 A1 WO 2024007769A1 CN 2023096968 W CN2023096968 W CN 2023096968W WO 2024007769 A1 WO2024007769 A1 WO 2024007769A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame rate
target
video
event
speed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/096968
Other languages
English (en)
French (fr)
Inventor
李旭旭
闫昌达
王瀛
黄耀谆
王霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2024007769A1 publication Critical patent/WO2024007769A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Definitions

  • the present application relates to the field of image processing, and in particular to a frame rate adjustment method, device, equipment and system.
  • Frame rate is used to indicate the frequency at which the shooting device captures video footage.
  • the unit of frame rate is Frames per Second (FPS).
  • FPS Frames per Second
  • shooting equipment uses a fixed frame rate to shoot and collect videos of targets. If the shooting device uses a higher frame rate to shoot a target that moves slower than the shooting device, the shooting device will collect more redundant images, which increases the power consumption of the shooting device. If the shooting device uses a lower frame rate to shoot a target that moves faster than the shooting device, the shooting device cannot fully capture the motion information of the target, resulting in unsmooth motion of the target in the collected video.
  • the shooting device can re-encode the captured video to generate videos with different frame rates.
  • the shooting equipment uses a lower frame rate to shoot videos, which has caused the loss of motion information.
  • the motion information cannot be compensated through subsequent encoding, and there is a problem of unsmooth videos.
  • Embodiments of the present application provide a frame rate adjustment method, device, equipment and system, thereby improving the smoothness of videos collected by shooting equipment.
  • the first aspect provides a frame rate adjustment method.
  • the frame rate adjustment method is executed by the shooting device, and specifically includes the following steps: the shooting device uses the first frame rate to shoot the shooting target to obtain the first video.
  • the shooting device determines the second frame rate based on the relative motion state of the shooting device itself and the photographed target.
  • the relative motion state is used to indicate how slowly the image of the target in the first video changes.
  • the second frame rate is a frame rate suitable for photographing a target in the relative motion state relative to the shooting device, and the second frame rate may be greater than or less than the first frame rate.
  • the shooting device dynamically adjusts the shooting frame rate according to the relative motion state between itself and the target in the first video.
  • the shooting frame rate is increased.
  • the shooting frame rate is increased.
  • the shooting frame rate is reduced to ensure the smoothness of the second video. Therefore, if the target in the first video and the shooting device are relatively stationary but change (for example, the brightness changes) or there is relative movement between the target and the shooting device in the first video, the shooting device can use the third method that is suitable for the speed of the image change. Collecting the second video at a second frame rate improves the accuracy of frame rate adjustment, thereby further improving the smoothness of the second video collected by the shooting device according to the adjusted second frame rate.
  • the relative motion state indicates that the image of the target in the first video changes slowly, and the shooting device uses a frame rate lower than the first frame rate to shoot the target, and the video obtained can also satisfy the human eye's video fluency.
  • the second frame rate is smaller than the first frame rate.
  • the relative motion state indicates that the image of the target in the first video changes quickly, and the video obtained by the shooting device using the first frame rate to shoot the target cannot meet the smoothness of the video by the human eye. demand, the second frame rate is greater than the first frame rate.
  • the shooting device first determines the changing speed of the image of the target in the first video, and then determines the second frame rate based on the changing speed of the image of the target in the first video and the first correlation relationship.
  • the first correlation relationship is used to indicate the relationship between the relative motion state and the frame rate.
  • the second frame rate determined by the shooting device according to the first association relationship not only ensures the smoothness of the second video, but also avoids the need for shooting The problem of high power consumption of shooting equipment caused by too high frame rate.
  • the shooting device can determine the change speed of the image in the first video, use the change speed of the image to represent the relative motion state, and then use the change speed of the image to indicate the relative motion state, and then based on the change speed of the image and The first association determines the second frame rate.
  • the first correlation relationship is used to indicate the relationship between the relative motion state and the frame rate. For example, if each numerical value of the change speed of the image represents a different relative motion state, then the shooting device queries the first association relationship according to the change speed of the image, and determines the frame rate corresponding to the change speed of the image and the second frame rate. For another example, different value intervals of the change speed of the image represent different relative motion states, then the shooting device determines the value interval to which the image change speed belongs, and then queries the first association relationship to determine the frame rate corresponding to the value interval. is the second frame rate. Therefore, this embodiment provides more than one way of determining the second frame rate based on the changing speed of the image. The method of determining the second frame rate can be selected based on the accuracy requirements of frame rate adjustment and the performance of the shooting equipment, thereby improving Provides flexibility in frame rate adjustment.
  • This embodiment does not limit the way in which the shooting device determines the changing speed of the image of the target in the first video.
  • the shooting device can determine the changing speed of the image of the target in the first video based on different types of sensing information collected by different sensors.
  • the shooting device includes a dynamic vision sensor (DVS), and the shooting device determines the changing speed of the image of the target in the first video based on the asynchronous events output by the DVS.
  • Asynchronous events are time-stamped signals output by DVS when local pixel-level brightness changes (called events) exceed a set threshold. Because DVS can sense and output pixel-level brightness changes in the target's image, it improves the accuracy of the shooting device in determining the changing speed of the image.
  • the shooting device determines the event accumulation speed according to the asynchronous event, and determines the change speed of the image of the target in the first video according to the event accumulation speed.
  • the event accumulation speed is used to indicate the speed at which DVS outputs asynchronous events.
  • the above event accumulation rate is the total number of asynchronous events on the photosensitive array of the DVS per unit time.
  • the photosensitive array of the DVS of the shooting device can also be called the target surface.
  • the unit time can be a time period of a preset duration, and the preset duration can be flexibly adjusted according to the needs of the shooting equipment and the target.
  • the above event accumulation speed is the number of pixels in the area where the event exists in the event frame.
  • An event frame is an event image output based on multiple asynchronous events accumulated per unit time.
  • the shooting device will, based on the asynchronous event, Generate event frames.
  • the shooting device can also perform morphological filtering on the event frame to repair smearing or defects in the edge texture area of the event frame due to target movement in the first video, and remove sparse and discrete background noise. This reduces the impact of smear, incompleteness and background noise on the calculation of event accumulation speed, ensuring the accuracy of event accumulation speed.
  • the shooting equipment includes an inertial measurement unit (Inertial Measurement Unit, IMU), the shooting device determines the moving speed of the shooting device based on the inertial data collected by the IMU, and determines the changing speed of the image of the target in the first video based on the moving speed of the shooting device.
  • IMU Inertial Measurement Unit
  • the shooting device calculates the vector length of the motion speed of the shooting device based on the acceleration and angular velocity collected by the IMU. Since the motion speed of the shooting device itself will cause relative motion between the shooting device and the target, if the target moves slowly, the shooting device can use the vector length of the motion speed of the shooting device to represent the changing speed of the image of the target in the first video. Therefore, the changing speed of the image of the target in the first video is determined through the inertial data collected by the IMU, which is common in shooting equipment, simplifying the steps of determining the changing speed of the image of the target in the first video, and improving the efficiency of the frame rate adjustment method. applicability.
  • the shooting device includes a DVS and an IMU.
  • the shooting device determines the changing speed of the image of the target in the first video based on the inertial data collected by the IMU and the asynchronous events output by the DVS.
  • the shooting device determines the first change speed of the image of the target in the first video based on the inertial data collected by the IMU, and determines the second change speed of the image of the target in the first video based on the asynchronous event output by the DVS. Then the shooting device queries the frame rate corresponding to the first change speed and the frame rate corresponding to the second change speed in the first association relationship. The shooting device determines the change speed corresponding to the higher frame rate among the first change speed and the second change speed as the change speed of the image of the target in the first video, so that the shooting device determines the change speed according to the change speed of the image of the target in the first video.
  • the second frame rate can capture a second video with higher fluency, improving the accuracy of frame rate adjustment.
  • a second aspect provides a frame rate adjustment device, which includes various modules for executing the frame rate adjustment method in the first aspect or any possible implementation of the first aspect.
  • the frame rate adjustment device described in the second aspect may be a terminal device or a network device, or may be a chip (system) or other component or component that can be disposed in the terminal device or network device, or may include Terminal equipment or network equipment is not limited in this application.
  • a computing device including a memory and a processor.
  • the memory is used to store a set of computer instructions.
  • the processor executes the set of computer instructions, it is used to execute any of the first aspects. Operational steps of a frame rate adjustment method in a possible design.
  • a computer-readable storage medium including: computer software instructions; when the computer software instructions are run in a computing device, the computing device is caused to execute the method described in any possible implementation manner in the first aspect. operating steps.
  • a computer program product is provided.
  • the computer program product When the computer program product is run on a computer, it causes the computer to perform the operation steps of the method described in any possible implementation manner in the first aspect.
  • Figure 1 is a schematic structural diagram of a frame rate adjustment system provided by an embodiment of the present application.
  • Figure 2 is a schematic flowchart of a frame rate adjustment method provided by an embodiment of the present application.
  • FIG. 3 is a schematic flowchart of another frame rate adjustment method provided by an embodiment of the present application.
  • Figure 4 is a schematic flowchart of a second frame rate determination step provided by an embodiment of the present application.
  • Figure 5 is a schematic diagram of an event frame denoising comparison provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of a frame rate adjustment device provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • Embodiments of the present application provide a frame rate adjustment method, particularly a method for adjusting the shooting frame rate of the shooting device based on the relative motion state of the shooting device and the shooting target, that is, the shooting device collects the first video according to the first frame rate.
  • the second frame rate is determined based on the relative motion state of the shooting device and the target in the first video.
  • the shooting device adjusts the shooting frame rate to the second frame rate and then collects the second video, where the relative motion state is used Indicates how quickly the image of the target in the first video changes.
  • the shooting device adjusts the shooting frame rate according to the relative motion state of the shooting device and the target in the first video, that is, the speed of change of the image of the target in the first video, so that the shooting device can capture the target itself according to the second frame rate.
  • Changes for example: brightness changes
  • image information of the relative movement of the target and the shooting device thereby improving the accuracy of frame rate adjustment and ensuring the smoothness of the second video collected by the shooting device according to the adjusted second frame rate.
  • FIG. 1 is a schematic structural diagram of a frame rate adjustment system 100 provided by an embodiment of the present application.
  • the frame rate adjustment system 100 includes a control device 110 , a computing device 120 , a data storage system 130 and a shooting device 140 .
  • the control device 110 may be a terminal, such as a mobile phone, a tablet computer, a laptop, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (Mixed Reality, MR) device, or an extended reality (Extended) device. Reality, ER), vehicle camera equipment, etc.
  • a terminal such as a mobile phone, a tablet computer, a laptop, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (Mixed Reality, MR) device, or an extended reality (Extended) device.
  • VR virtual reality
  • AR augmented reality
  • MR mixed reality
  • Extended reality Extended reality
  • the computing device 120 may be a terminal, or other computing device, such as a server or a cloud device.
  • the computing device 120 may be a graphics processing unit (GPU), a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (Application-Specific Integrated Circuit) specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor can be a microprocessor or any conventional processor, etc.
  • the training device 720 may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or a or A plurality of integrated circuits used to control the execution of the program of this application.
  • GPU graphics processing unit
  • NPU neural network processing unit
  • ASIC application-specific integrated circuit
  • the data storage system 130 is used to store data that the control device 110 needs to call, as well as data output by the control device 110 .
  • the shooting device 140 is used to collect sensing data using sensors, and send the collected sensing data to the computing device 120 .
  • the shooting device 140 includes an image sensor 141, an inertial measurement unit (Inertial Measurement Unit, IMU) 142, and a dynamic vision sensor (Dynamic Vision Sensor, DVS) 143.
  • Sensing data includes inertial data collected by the inertial measurement unit 142, video collected by the image sensor 141, asynchronous events output by the dynamic vision sensor 143, etc.
  • the image sensor 141 is used to convert the light image on the photosensitive surface into an electrical signal proportional to the light image by utilizing the photoelectric conversion function of the optoelectronic device.
  • the image sensor 141 may be, but is not limited to, a complementary metal oxide semiconductor (Complementary Metal-Oxide Semiconductor). Oxide-Semiconductor (CMOS), Charge-coupled Device (CCD), etc.
  • CMOS complementary Metal-Oxide Semiconductor
  • CCD Charge-coupled Device
  • the video captured by the image sensor 141 may include the shooting device 140
  • the first video is collected according to the first frame rate
  • the second video is collected according to the second frame rate.
  • the first frame rate and the second frame rate are used to indicate the frame rate at which the shooting device 140 shoots the target, and the first frame rate and the second frame rate are different.
  • the computing device 120 is configured to calculate the relative motion state of the target in the first video using inertial data and/or asynchronous events, and determine the second frame rate according to the relative motion state. In turn, the computing device 120 sends the determined second frame rate to the control device 110 .
  • the control device 110 is used to implement the function of adjusting the shooting frame rate of the shooting device 140 according to the second frame rate, and to instruct the shooting device 140 to shoot the second video using the second frame rate.
  • the above-mentioned frame rate adjustment system 100 is an example of a system that applies the frame rate adjustment method.
  • the frame rate adjustment method provided by the embodiment of the present application can be applied to various systems with shooting functions.
  • the frame rate adjustment system 100 may also be a system used for portrait photography, movie shooting, or aerial photography, as well as a mobile phone, a tablet computer, and the like.
  • Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, devices, modules, etc. shown in Figure 1 does not constitute any limitation.
  • the control device 110, the computing device 120 and the shooting device 140 can Set on the same physical device.
  • the shooting device 140 may be a camera
  • the control device 110 may be a control component that controls the lens
  • the computing device 120 may be a processor provided in the camera.
  • the control device 110 may be a control component in a mobile phone for controlling a camera
  • the computing device 120 may be a processor in the mobile phone
  • the shooting device 140 may be a camera in the mobile phone.
  • the control device 110, the computing device 120, and the photographing device 140 may be disposed on the same physical device or may be disposed on different physical devices.
  • the control device 110 is a monitoring device used by the director to monitor and control the camera
  • the computing device 120 is a computer connected to the monitoring device
  • the shooting device 140 is a camera connected to the monitoring device and the computer.
  • the control device 110 is a monitoring device used by the director to monitor and control the camera.
  • the computing device 120 is connected to the control device 110 , and the computing device 120 is used to process the data transmitted by the control device 110 , for example, perform image processing on the video acquired by the control device 110 .
  • the shooting device 140 is connected to the monitoring device and the computing device 120 through a wired connection or a wireless connection.
  • the shooting device 140 is a camera used to shoot running horses, rolling footballs and other moving objects.
  • the default shooting frame rate of the camera is the first frame rate, the camera shoots the moving object based on the first frame rate, and sends the captured first video to the computing device 120 .
  • the camera also uses the inertial measurement unit 142 to collect inertial data, and/or uses the dynamic vision sensor 143 to collect asynchronous events, and sends the inertial data and/or asynchronous events to the computing device 120 .
  • the computing device 120 determines the relative motion state of the camera and the moving object in the first video according to the inertial data and/or asynchronous events.
  • the relative motion state represents the relative motion speed of the moving object and the camera, and determines the user according to the relative motion state.
  • Second frame rate for shooting football The computing device 120 sends the second frame rate to the monitoring device, and the monitoring device adjusts the shooting frame rate of the camera according to the second frame rate, so that the monitoring device uses the second frame rate to shoot the football to obtain the second video.
  • the control device 110 is a control component in the mobile phone used to control the camera
  • the computing device 120 is the processor of the mobile phone
  • the shooting device 140 is the camera of the mobile phone.
  • the user uses the operation interface of the mobile phone to open the shooting application, the shooting application starts the camera, and uses the default first frame rate to shoot the display playing the video to obtain the first video.
  • the camera uses the inertial measurement unit 142 to collect inertial data, and/or uses the dynamic vision sensor 143 to collect asynchronous events, and sends the inertial data and/or asynchronous events to the processor.
  • the processor determines the relative motion state between the camera and the display in the first video based on inertial data and/or asynchronous events.
  • the relative motion state is used to indicate the speed of change of the image of the target in the first video.
  • the target in the first video refers to the display,
  • the speed of change of the target image refers to the speed of change of the brightness of the display.
  • the processor determines a second frame rate for photographing the display based on the relative motion state.
  • the processor sends the second frame rate to the control component, and the control component Adjust the shooting frame rate of the camera so that the camera uses the second frame rate to shoot the display to obtain the second video.
  • the relative movement speed of the moving object in the first video and the shooting device 140 will cause the image of the video collected by the shooting device 140 to change, and the display will play The brightness change during the video will also cause the image of the video collected by the shooting device 140 to change.
  • the computing device 120 determines a second frame rate for capturing the second video of the target based on the relative motion state used as an indicator indicating how quickly the image of the target changes in the first video.
  • the computing device 120 can adjust the shooting frame rate to the second frame rate suitable for shooting the target, so that the shooting device 140 adopts
  • the second video shot at the second frame rate has sufficient smoothness and prevents the shooting device 140 from using an excessively high frame rate to shoot targets with slow image changes, thereby reducing the power consumption of the shooting device 140 .
  • control device 110 can be further subdivided into an architecture as shown in Figure 1.
  • the control device 110 is configured with a computing module 111 and an I/O interface 112. and processing module 113.
  • the I/O interface 112 is used for data interaction with external devices.
  • the photographing device 140 inputs data to the I/O interface 112 .
  • Input data can include images, videos, inertial data, asynchronous events, etc.
  • the computing module 111 is used to process input data from the computing device 120 .
  • the computing module 111 is configured to generate a second frame rate adjustment instruction according to the second frame rate input by the computing device 120 , and send the second frame rate adjustment instruction to the shooting device 140 through the I/O interface 112 .
  • the processing module 113 is configured to perform processing according to input data received by the I/O interface 112 or the computing device 120 .
  • the processing module 113 is used to perform image signal processing (ISP) on the input data received from the I/O interface 112 to generate a variable frame rate media file.
  • ISP image signal processing
  • control device 110 When the control device 110 performs image processing on input data, or when the calculation module 111 of the control device 110 performs calculations and other related processes, the control device 110 can call data, codes, etc. in the data storage system 130 for corresponding processing. , the data and instructions obtained by corresponding processing can also be stored in the data storage system 130 .
  • the I/O interface 112 returns the processing result to the shooting device 140, thereby providing it to the user so that the user can view the processing result.
  • the photographing device 140, the control device 110 and the computing device 120 in FIG. 1 are taken as an example of the same physical device, that is, the photographing device 140 includes the control device 110 and the computing device 120.
  • the specific steps of the frame rate adjustment method will be described below with the shooting device 140 as the execution subject.
  • Step 210 The shooting device 140 collects the first video according to the first frame rate.
  • the shooting device 140 sets the shooting frame rate to the first frame rate, and collects the first video according to the first frame rate.
  • the first frame rate is used to indicate the shooting frame rate of the shooting device 140 shooting the target.
  • the first frame rate may be a default frame rate of the shooting device 140 .
  • Step 220 The shooting device 140 determines the second frame rate according to the relative motion state between the shooting device 140 and the target in the first video.
  • the shooting device 140 uses the computing device 120 to determine the relative motion state of the shooting device 140 and the target in the first video according to the changing speed of the image of the target in the first video, and determines the second frame rate according to the relative motion state.
  • the second frame rate is used to indicate the shooting frame rate at which the shooting device 140 shoots the target.
  • the change speed of the image of the target in the first video may refer to the change speed of the target's position in the shooting screen of the shooting device 140 caused by the relative movement of the target and the shooting device 140 .
  • the target When the soccer ball is marked as moving the change speed of the image of the target is the position change speed of the soccer ball in the shooting screen of the shooting device 140 .
  • the shooting device 140 can use the motion speed of the shooting device 140 itself as the difference between the shooting device 140 and the first video.
  • the relative movement speed of the target Since the motion of the shooting device 140 will cause the shooting picture to change, the different relative movement speeds of the shooting device 140 and the target in the first video will cause the position of the target's image in the shooting picture of the shooting device 140 to change, causing the target to change in the shooting picture of the shooting device 140 .
  • the computing device 120 determines the movement speed of the shooting device 140 based on the inertial data collected by the inertial measurement unit 142, and the corresponding relationship between the movement speed of the shooting device 140 and the change speed of the image of the target in the first video, and determines the image of the target in the first video. the speed of change.
  • the change speed of the image of the target in the first video may also refer to the change speed of the brightness of the target in the shooting screen of the shooting device 140 due to changes in the target itself.
  • the target is a display that plays a video
  • the brightness of the display continues to change due to the played video
  • the brightness of the display changes in the shooting frame of the shooting device 140 .
  • the change speed of the image of the target is the brightness change speed of the screen displayed on the display in the shooting screen of the shooting device 140 .
  • the computing device 120 may determine the change speed of the image of the target in the first video according to an asynchronous event output by the dynamic vision sensor 143 according to the brightness change in the captured image.
  • DVS also called event camera
  • DVS is a new type of neuromorphic sensor. Unlike traditional image sensors that output signals in a "frame" manner, DVS identifies the status of each pixel and outputs an asynchronous event signal, that is, an asynchronous event when the pixel brightness changes. Therefore, the dynamic vision sensor 143 can record changes that occur in the shooting images of the shooting device 140 and has a high motion information capture capability.
  • the computing device 120 may also determine the changing speed of the image of the target in the first video based on the inertial data collected by the inertial measurement unit 142 and the asynchronous events output by the dynamic vision sensor 143 . Therefore, the shooting device 140 uses the computing device 120 to more accurately determine the first video in the case where the target itself changes, or the shooting device 140 itself is in a moving state, or when the target itself changes and the shooting device 140 itself is in a moving state. The speed of change of the target's image.
  • the computing device 120 determines the changing speed of the image of the target in the first video, it then determines the relative motion state of the shooting device 140 and the target in the first video based on the changing speed of the image of the target in the first video.
  • the relative motion state includes the speed of image change of multiple targets, such as three speed levels: static state, low speed state and high speed state.
  • the static state, low speed state and high speed state respectively include different image change speeds. value range.
  • the computing device 120 determines the relative motion state of the target in the first video according to the value interval in which the change speed of the image of the target in the first video belongs to a static state, a low-speed state, or a high-speed state.
  • the computing device 120 determines a second frame rate for photographing the target based on the relative motion state and the first association relationship.
  • the first association relationship is used to indicate the relationship between the relative motion state and the frame rate.
  • the frame rate corresponding to the static state is A
  • the frame rate corresponding to the low-speed state is B
  • the frame rate corresponding to the high-speed state is C.
  • the unit of the frame rate is Hertz (HZ), and the values of A, B, and C increase in sequence. big.
  • each value of the change speed of the target image in the first video indicates that the target is in a relative motion state.
  • Each value of the change speed of the target image corresponds to the relative motion state.
  • Each relative motion state corresponds to Frame rate corresponds one to one.
  • the shooting device 140 can adjust the relative motion status and frame according to its own performance.
  • the rate division granularity can be flexibly set.
  • Step 230 The shooting device 140 collects the second video according to the second frame rate.
  • the shooting device 140 sets the shooting frame rate of the image sensor 141 to the second frame rate, and collects the second video according to the second frame rate.
  • the computing device 120 outputs the calculated second frame rate to the computing module 111 , and the computing module 111 sends a second frame rate adjustment instruction to the image sensor 141 of the shooting device 140 through the I/O interface 112 .
  • the second frame rate adjustment instruction is used to instruct the shooting device 140 to set the shooting frame rate of the image sensor 141 to the second frame rate.
  • the shooting device 140 dynamically adjusts the shooting frame rate according to the relative motion state between itself and the target in the first video, and increases when the relative motion state indicates that the image of the target changes rapidly.
  • the shooting frame rate ensures the smoothness of the second video.
  • the shooting device 140 reduces the shooting frame rate when the image change speed of the relative motion state indicator target is slow, thereby ensuring the smoothness of the second video and avoiding excessive power consumption of the shooting device 140 .
  • the shooting device 140 adjusts the shooting frame rate according to the speed of image change of the target, the target and the shooting device 140 in the first video are relatively stationary but change themselves (for example, the brightness changes) or the target and the shooting device in the first video
  • the second frame rate determined by the shooting device 140 can ensure the smoothness of the second video, thereby further improving the smoothness of the second video and improving the applicability of the frame rate adjustment.
  • the shooting device 140 may also perform the step of: the shooting device 140 outputs the second video.
  • the processing module 113 is used to perform image signal processing on the signal of the second video to achieve imaging of the second video, and output the image of the second video.
  • Step 220 may include the following steps.
  • Step 221 The shooting device 140 determines the first changing speed of the image of the target in the first video based on the inertial data collected by the inertial measurement unit 142.
  • the shooting device 140 uses the computing device 120 to pre-integrate the inertial data to obtain the motion speed V (Vx, Vy, Vz) of the shooting device 140, and then calculates the vector length according to V (Vx, Vy, Vz) to determine the target in the first video
  • the first change speed of the image is the vector length V1.
  • the inertial data collected by the inertial measurement unit 142 includes angular velocity (gx, gy, gz) and acceleration (ax, ay, az). Examples of inertial data are shown in Table 1.
  • the computing device 120 calculates the movement speed V (Vx, Vy ,Vz) and vector length V1 are shown in Table 2.
  • Step 222 The shooting device 140 determines the second changing speed of the image of the target in the first video according to the asynchronous event output by the dynamic vision sensor 143.
  • the shooting device 140 uses the computing device 120 to determine an event accumulation speed according to the asynchronous events output by the dynamic vision sensor 143, and determines a second change speed of the image of the target in the first video according to the event accumulation speed.
  • the computing device 120 first initializes the event frame formation parameters, generates the event frame according to the event frame formation parameters, and then determines the event accumulation speed according to the event frame.
  • p is the polarity of the event
  • p The value of can be 1 or -1, 1 and -1 are used to indicate that the brightness of the pixel position increases or decreases respectively.
  • An example of an asynchronous event can be shown in Table 3.
  • the shooting device 140 After the previous event frame is output, the shooting device 140 performs a cumulative count on the asynchronous events, and the value of the cumulative count is n.
  • the shooting device 140 forms an event frame when t ⁇ t0+T0, or n ⁇ N0.
  • the computing device 120 performs a filtering process on the event frames before determining the event accumulation rate based on the event frames.
  • the computing device 120 performs morphological filtering such as median filtering, mean filtering, binarization processing, or dilation erosion in the spatial domain on the event frame to remove stray background noise and enhance the edge contour of the scene signal.
  • morphological filtering such as median filtering, mean filtering, binarization processing, or dilation erosion in the spatial domain on the event frame to remove stray background noise and enhance the edge contour of the scene signal.
  • Figure 5 represents an event frame with noise
  • (b) represents an event frame after asynchronous spatiotemporal neighborhood denoising
  • (c) represents an event frame after morphological denoising.
  • the DVS circuit does not process the brightness changes of the pixels in time, resulting in a lag in the asynchronous event output time, and the edge texture area of the event frame will form smear or incompleteness.
  • the incomplete edges can be effectively repaired through morphological opening and closing operations, and sparse and discrete background noise can be more thoroughly removed, thereby improving the accuracy of the image change speed of the target in the first video.
  • the computing device 120 may also calculate the percentage of the event area accounting for the entire photosensitive array area of the dynamic vision sensor 143, and determine the event accumulation rate as this percentage.
  • V2 Nt/(W*H), where W is the wide resolution of the dynamic vision sensor 143, and H is the high resolution of the dynamic vision sensor 143.
  • Step 223 The shooting device 140 determines the relative motion state according to the first change speed and the second change speed.
  • the shooting device 140 uses the computing device 120 to determine that the change speed of the image of the target in the first video is the higher change speed of the corresponding relative motion state among the first change speed V1 and the second change speed V2.
  • relative motion states include stationary state, low speed state and high speed state.
  • the relative motion state refers to the contrast between the static state, the low-speed state and the high-speed state.
  • the high-speed state is higher than the low-speed state
  • the low-speed state is higher than the static state.
  • the first change speed V1 is set with value intervals A1, A2 and A3.
  • the value interval of A1 is [0,0.5], and the value interval of A2 is (0.5, 2].
  • the value interval of A3 is (2,100].
  • the value interval of B1 is [0,1000]
  • the value interval of B2 is is (1000, 10000]
  • the value range of B3 is (10000, 100000].
  • A1, A2 and A3 correspond to the static state, low speed state and high speed state respectively
  • B1, B2 and B3 respectively correspond to the static state, low speed state and High speed state.
  • the endpoint value of each value interval of the above-mentioned first change speed V1 and the second change speed V2 is only an example, and this embodiment does not limit the endpoint value of each value interval.
  • the value interval corresponding to the first change speed V1 is A1
  • the value interval corresponding to the second change speed V2 is B2
  • the The relative motion state corresponding to the first change speed V1 is a stationary state
  • the relative motion state corresponding to the second change speed V2 is a low speed state
  • the computing device 120 determines that the relative motion state is a low speed state.
  • Step 224 The shooting device 140 determines the second frame rate according to the relative motion state and the first association relationship.
  • the shooting device 140 uses the computing device 120 to query the first association relationship according to the relative motion state and determine the second frame rate.
  • the first correlation relationship is used to indicate the relationship between the relative motion state and the frame rate.
  • the relative motion state includes a static state, a low-speed state, and a high-speed state
  • the shooting frame rate includes three gears corresponding to the static state, the low-speed state, and the high-speed state, such as 5HZ, 30HZ and 200HZ.
  • the computing device 120 determines that the second frame rate is 30 HZ.
  • the number of shooting frame rate gears is not limited to the 3 gears mentioned in the example. It can be set to 2 or any number more than 2. to meet the needs of different application scenarios.
  • the shooting device 140 may also utilize the computing device 120 to adjust event frame formation parameters according to the relative motion state.
  • the dynamic vision sensor 143 is sensitive to the motion information in the shooting picture.
  • the movement speed or state change of the target in the shooting picture will cause the density of output asynchronous events to change in real time.
  • the faster the movement speed the denser the number of asynchronous events generated in the same time. , on the contrary, the sparser it is. Therefore, the shooting device 140 uses the computing device 120 to adjust the event frame formation parameters according to the relative motion state, which can effectively avoid the asynchronous events being too sparse in the shooting picture where the moving target is too small or lacks texture, and the moving target is too large or the texture is rich in the scene. Asynchronous events are too dense.
  • the shooting device 140 can use the computing device 120 to adjust the acquisition frame rate of the image sensor 141 and the event frame formation parameters of the dynamic vision sensor 143 according to the relative motion state, or use the computing device 120 to individually adjust the acquisition frame rate according to the relative motion state.
  • the acquisition frame rate of the image sensor 141 is adjusted, or the computing device 120 is used to individually adjust the event frame formation parameters of the dynamic vision sensor 143 according to the relative motion state.
  • the computing device 120 determines the overall proportion p of the event area in the photosensitive array of the dynamic vision sensor 143 per unit time or the accumulated number of events n, and when p exceeds the threshold 6% or n exceeds the threshold 20000, decrease N0 or T0 value, thereby increasing the output frame rate of the event frame, conversely increasing the value of N0 or T0, thereby reducing the output frame rate of the event frame.
  • the computing device 120 Based on the method of determining the second frame rate shown in FIG. 4 , the computing device 120 performs fusion processing on the inertial data collected by the inertial measurement unit 142 and the asynchronous events output by the dynamic vision sensor 143 to obtain the relative motion state of the target in the first video, and This is used as the basis for the speed of change of the target image in the first video.
  • the shooting device 140 uses a lower frame rate corresponding to the static state or the low-speed state to shoot the second video, so as to reduce the shooting situation of the shooting device 140 in a shooting scene with small movement (for example: the target is in a stationary state, the shooting device 140 The power consumption of the shooting device 140 when shooting videos and processing images when in a handheld shooting state.
  • the shooting device 140 uses a higher frame rate corresponding to the high-speed state to shoot the second video, thereby improving the performance of the shooting device 140 in shooting scenes with large movements (for example, the target is in a running state, jumping, extreme sports, etc.). 140 in high-speed motion), the picture fineness and smoothness.
  • the shooting device 140 may determine the changing speed of the image of the target in the first video only based on the inertial data collected by the inertial measurement unit 142 , determine the relative motion state based on the first changing speed, and then determine the relative motion state based on the relative motion. The state and the first association determine the second frame rate.
  • the difference between this implementation and the method of determining the second frame rate shown in FIG. 4 is that the shooting device 140 does not perform step 222, and only determines the relative motion state according to the first change speed in step 223, which will not be described again.
  • the shooting device 140 may determine the changing speed of the image of the target in the first video only based on the asynchronous event output by the dynamic vision sensor 143, determine the relative motion state based on the first changing speed, and then determine the relative motion state based on the relative motion. The state and the first association determine the second frame rate.
  • the difference between this implementation and the method of determining the second frame rate shown in FIG. 4 is that the shooting device 140 does not perform step 221, and only determines the relative motion state according to the second change speed in step 223, which will not be described again.
  • the frame rate adjustment method provided by this embodiment is described in detail above with reference to FIGS. 2 to 5 .
  • the frame rate adjustment device provided by this embodiment will be described below with reference to FIG. 6 .
  • FIG. 6 is a schematic diagram of a possible frame rate adjustment device provided in this embodiment.
  • the frame rate adjustment device can be used to implement the functions of the execution device in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments.
  • the frame rate adjustment device may be the computing device 120 as shown in FIG. 1 , or may be a module (such as a chip) applied to the server.
  • the frame rate adjustment device 600 includes an acquisition module 610 and a processing module 620.
  • the frame rate adjustment device 600 is used to implement the functions of the computing device in the method embodiment shown in FIG. 2, FIG. 3 or FIG. 4.
  • the collection module 610 is configured to collect the first video according to a first frame rate, where the first frame rate is used to indicate the frame rate of the shooting target of the shooting device.
  • the processing module 620 is configured to determine the second frame rate according to the relative motion state of the shooting device and the target in the first video.
  • the relative motion state is used to indicate how fast the image of the target in the first video changes
  • the second frame rate is used to indicate the frame rate of the target captured by the shooting device
  • the second frame rate is greater than or less than the first frame rate.
  • the collection module 610 is also used to collect the second video according to the second frame rate.
  • the processing module 620 when determining the second frame rate based on the relative motion state of the shooting device and the target in the first video, is specifically used to: determine the changing speed of the image of the target in the first video; The changing speed of the image of the target in a video and the first correlation determine the second frame rate.
  • the first correlation relationship is used to indicate the relationship between the relative motion state and the frame rate.
  • the shooting device 140 includes a dynamic vision sensor 143.
  • the processing module 620 is specifically configured to: determine the first time according to the asynchronous event output by the dynamic vision sensor 143. The speed of change of the image of the target in the video.
  • the processing module 620 when determining the changing speed of the image of the target in the first video based on the asynchronous event output by the dynamic vision sensor 143, is specifically configured to: determine the event accumulation speed based on the asynchronous event; Determine how fast the image of the target in the first video changes.
  • the event accumulation speed is used to indicate the speed at which the dynamic vision sensor 143 outputs asynchronous events.
  • the event accumulation rate includes the total number of asynchronous events on the photosensitive array of the dynamic vision sensor 143 per unit time.
  • the event accumulation speed includes the number of pixels in the area where the event exists in the event frame, and the event frame is an event image output based on multiple asynchronous events accumulated within a unit time.
  • the processing module 620 when determining the event accumulation speed based on asynchronous events, is specifically configured to: when the continuous duration of the asynchronous events output by the dynamic vision sensor 143 exceeds the preset duration and/or the accumulated number of asynchronous events exceeds the preset duration.
  • the processing module 620 When setting the number, generate event frames based on asynchronous events; perform morphological filtering on the event frames; determine the number of pixels in the area where asynchronous events exist in the event frames after morphological filtering, and obtain the event accumulation speed.
  • the shooting device 140 includes an inertial measurement unit 142.
  • the processing module 620 is specifically configured to: based on the inertial data collected by the inertial measurement unit 142 and asynchronous events. Determine how fast the image of the target in the first video changes.
  • the frame rate adjustment device 600 in the embodiment of the present application can be implemented by a GPU, an NPU, an ASIC, or a programmable logic device (PLD).
  • PLD programmable logic device
  • the above PLD can be a complex programmable logical device (complex programmable logical device). device (CPLD), field-programmable gate array (FPGA), general array logic (GAL) or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL general array logic
  • the frame rate adjustment device 600 may correspond to performing the method described in the embodiment of the present application, and the above and other operations and/or functions of the various units in the frame rate adjustment device 600 are respectively to implement FIG. 2 and FIG.
  • the corresponding processes of each method in Figure 3 or Figure 4 will not be repeated here for the sake of simplicity.
  • FIG. 7 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • Computing device 700 includes memory 701, processor 702, communication interface 703, and bus 704. Among them, the memory 701, the processor 702, and the communication interface 703 realize communication connections between each other through the bus 704.
  • Memory 701 may be a read-only memory, a static storage device, a dynamic storage device, or a random access memory.
  • the memory 701 may store computer instructions. When the computer instructions stored in the memory 701 are executed by the processor 702, the processor 702 and the communication interface 703 are used for steps in the frame rate adjustment method.
  • the memory can also store a data set. For example, a part of the storage resources in the memory 701 is divided into an area for storing the program that implements the function of the frame rate adjustment device 600 in the embodiment of the present application.
  • the processor 702 can be a general-purpose CPU, application specific integrated circuit (ASIC), GPU or any combination thereof.
  • Processor 702 may include one or more chips.
  • Processor 702 may include an NPU.
  • the processor 702 executes the method for realizing the functions of the acquisition module 610 and the processing module 620 according to the program in the memory 701 that implements the functions of the frame rate adjustment device 600 .
  • the communication interface 703 uses a transceiver module, such as but not limited to a transceiver, to implement communication between the computing device 700 and other devices or communication networks.
  • a transceiver module such as but not limited to a transceiver, to implement communication between the computing device 700 and other devices or communication networks.
  • the first video or the second video can be obtained through the communication interface 703, and the second frame rate can be fed back.
  • Bus 704 may include a path that carries information between various components of computing device 700 (eg, memory 701, processor 702, communication interface 703).
  • the computing device 700 may be a computer (for example, a server) in a cloud data center, a computer in an edge data center, or a terminal.
  • computing device 120 may be deployed on each computing device 700.
  • a GPU is used to implement the functions of computing device 120 .
  • computing device 120 may communicate with control device 110 via bus 704 .
  • the computing device 120 may communicate with the control device 110 through the communication network.
  • the method steps in this embodiment can be implemented by hardware or by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules.
  • Software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or other well-known in the art any other form of storage media.
  • An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and storage media may be located in an ASIC. Additionally, the ASIC can be located in the terminal device.
  • the processor and storage media can also exist in the network as discrete components device or terminal device.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment, or other programmable device.
  • the computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.
  • the computer program or instructions may be transmitted from a website, computer, A server or data center transmits via wired or wireless means to another website site, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media.
  • the available media may be magnetic media, such as floppy disks, hard disks, and magnetic tapes; they may also be optical media, such as digital video discs (DVDs); they may also be semiconductor media, such as solid state drives (solid state drives). ,SSD).
  • SSD solid state drives

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

本申请提供一种帧率调节方法、装置、设备及系统，涉及图像处理领域。该方法包括：拍摄设备采用默认拍摄帧率对目标进行拍摄时，确定目标与拍摄设备的相对运动状态，根据该相对运动状态对拍摄设备的拍摄帧率进行调节，在相对运动状态指示目标的图像变化快慢程度较快时提高拍摄帧率，在相对运动状态指示目标的图像变化快慢程度较慢时降低拍摄帧率。由此，第一视频中的目标自身状态发生变化或与拍摄设备存在相对运动的情况下，拍摄设备都能使用适用于图像变化快慢程度的第二帧率采集第二视频，提高了帧率调节的准确度，提高了拍摄设备根据调节后的第二帧率采集的第二视频的流畅度。

Description

帧率调节方法、装置、设备及系统
本申请要求于2022年07月06日提交国家知识产权局、申请号为202210799229.0、申请名称为“帧率调节方法、装置、设备及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，尤其涉及一种帧率调节方法、装置、设备及系统。
背景技术
帧率(Frame rate)是用于指示拍摄设备拍摄视频的画面的频率。帧率的单位为每秒显示帧数(Frames per Second，FPS)。拍摄设备拍摄视频时采用的帧率越高，人眼感知视频越流畅。反之，拍摄设备拍摄视频时采用的帧率越低，人眼感知视频的卡顿现象越严重。
目前，拍摄设备采用一种固定的帧率对目标进行拍摄采集视频。如果拍摄设备采用较高的帧率拍摄相对拍摄设备运动速度较慢的目标，则拍摄设备会采集到较多的冗余图像，增加了拍摄设备的功耗。如果拍摄设备采用较低的帧率拍摄相对拍摄设备运动速度较快的目标，则拍摄设备无法全面地捕捉目标的运动信息，导致采集到的视频中目标的运动不流畅。
通常，拍摄设备可以对采集的视频进行重新编码，生成不同帧率的视频。但是，拍摄设备采用较低的帧率拍摄视频已造成运动信息的丢失，无法通过后续编码对运动信息进行补偿，存在视频不流畅的问题。
发明内容
本申请实施例提供一种帧率调节方法、装置、设备及系统，由此提高拍摄设备采集的视频的流畅度。
第一方面，提供一种帧率调节方法。该帧率调节方法由拍摄设备执行，具体包括如下步骤：拍摄设备采用第一帧率对拍摄的目标进行拍摄，获得第一视频。拍摄设备根据拍摄设备自身与拍摄的目标的相对运动状态确定第二帧率。相对运动状态用于指示第一视频中的目标的图像变化快慢程度较慢的程度。第二帧率是适用于拍摄相对拍摄设备处于该相对运动状态的目标的帧率，第二帧率可以大于或小于第一帧率。
由此，拍摄设备根据自身与第一视频中目标的相对运动状态对拍摄帧率进行动态调节，在相对运动状态指示目标的图像变化快慢程度较快时提高拍摄帧率，在相对运动状态指示目标的图像变化快慢程度较慢时降低拍摄帧率，保证了第二视频的流畅度。因此，第一视频中的目标与拍摄设备相对静止但自身发生变化(例如：亮度变化)或第一视频中的目标与拍摄设备存在相对运动，拍摄设备都能使用适用于图像变化快慢程度的第二帧率采集第二视频，提高了帧率调节的准确度，从而进一步提高了拍摄设备根据调节后的第二帧率采集的第二视频的流畅度。
例如，该相对运动状态指示第一视频中的目标的图像变化快慢程度较慢，拍摄设备采用比第一帧率更低的帧率对目标进行拍摄获得的视频也能满足人眼对视频流畅度的需求，则第二帧率小于第一帧率。又如，该相对运动状态指示第一视频中的目标的图像变化快慢程度较快，拍摄设备采用第一帧率对目标进行拍摄获得的视频无法满足人眼对视频流畅度 的需求，则第二帧率大于第一帧率。
作为一种可能的实现方式，拍摄设备先确定第一视频中目标的图像的变化速度，再根据第一视频中目标的图像的变化速度和第一关联关系确定第二帧率。其中，第一关联关系用于指示相对运动状态与帧率的关系。
基于上述实现方式，由于第一关联关系用于指示相对运动状态与帧率的关系，因此拍摄设备根据第一关联关系确定的第二帧率既保证了第二视频的流畅度，又避免了拍摄帧率过高导致的拍摄设备功耗高的问题。
由于相对运动状态用于指示第一视频中目标的图像的变化快慢程度，拍摄设备可以确定第一视频中的图像变化速度，利用图像的变化速度来表示相对运动状态，再根据图像的变化速度和第一关联关系确定第二帧率。
其中，第一关联关系用于指示相对运动状态与帧率的关系。例如，图像的变化速度的每个数值表示不同的相对运动状态，则拍摄设备根据图像的变化速度查询第一关联关系，确定与该图像的变化速度对应的帧率为第二帧率。又如，图像的变化速度的不同取值区间表示不同的相对运动状态，则拍摄设备确定图像的变化速度所属的取值区间，然后查询第一关联关系，确定与该取值区间对应的帧率为第二帧率。由此，本实施例提供的根据图像的变化速度确定第二帧率的方式不止一种，可以根据帧率调节的精确度需求和拍摄设备的性能情况选择第二帧率的确定方式，从而提高了帧率调节的灵活性。
本实施例不对拍摄设备确定第一视频中目标的图像的变化速度的方式进行限定，拍摄设备可以根据不同传感器采集的不同类型的传感信息确定第一视频中目标的图像的变化速度。
作为一种可能的实现方式，拍摄设备包括动态视觉传感器(Dynamic Vision Sensor，DVS)，拍摄设备根据DVS输出的异步事件确定第一视频中目标的图像的变化速度。异步事件是DVS在局部像素级亮度变化(称为事件)超过设定阈值时输出的带有时间戳的信号。由于DVS能够感知并输出目标的图像中像素级的亮度变化，从而提高了拍摄设备确定图像的变化速度的精确度。
可选地，拍摄设备根据异步事件确定事件积累速度，根据事件积累速度确定第一视频中目标的图像的变化速度。其中，事件积累速度用于指示DVS输出异步事件的速度。
例如，上述事件积累速度是单位时间内DVS的感光阵列上的所述异步事件的总数。拍摄设备的DVS的感光阵列也可以称为靶面。单位时间可以是预设时长的时间段，该预设时长可以根据拍摄设备和目标的需求进行灵活调节。
又如，上述事件积累速度是事件帧中存在事件的区域的像素个数。事件帧是基于单位时间内累积的多个异步事件输出的事件图像。
以事件积累速度是事件帧中存在事件的区域的像素个数为例，拍摄设备在DVS输出异步事件的连续时长超过预设时长和/或异步事件的累积数量超过预设数量时，根据异步事件生成事件帧。拍摄设备还可以对事件帧进行形态学滤波处理，来修复第一视频中由于目标运动导致事件帧的边缘纹理区域存在拖影或残缺，并去除稀疏离散的背景噪声。从而减少了拖影、残缺和背景噪声对事件积累速度的计算带来的影响，保证了事件积累速度的准确率。
作为另一种可能的实现方式，拍摄设备包括惯性测量单元(Inertial Measurement Unit， IMU)，拍摄设备根据IMU采集的惯性数据确定拍摄设备的运动速度，并根据拍摄设备的运动速度确定第一视频中目标的图像的变化速度。
可选地，拍摄设备根据IMU采集的加速度和角速度计算拍摄设备的运动速度的向量长度。由于拍摄设备自身的运动速度会导致拍摄设备与目标发生相对运动，若目标的运动速度较慢，拍摄设备可以使用拍摄设备的运动速度的向量长度来表示第一视频中目标的图像的变化速度。从而通过在拍摄设备中较普遍的IMU采集的惯性数据确定第一视频中目标的图像的变化速度，简化了第一视频中目标的图像的变化速度的确定步骤，并提高了帧率调节方法的适用性。
作为又一种可能的实现方式，拍摄设备包括DVS和IMU，拍摄设备根据IMU采集的惯性数据和DVS输出的异步事件确定第一视频中目标的图像的变化速度，
例如，拍摄设备根据IMU采集的惯性数据确定第一视频中目标的图像的第一变化速度，并根据DVS输出的异步事件确定第一视频中目标的图像的第二变化速度。然后拍摄设备分别在第一关联关系中查询第一变化速度对应的帧率和第二变化速度对应的帧率。拍摄设备确定第一变化速度和第二变化速度中对应的帧率较高的变化速度作为第一视频中目标的图像的变化速度，使拍摄设备根据第一视频中目标的图像的变化速度确定的第二帧率能够采集流畅度较高的第二视频，提高了帧率调节的准确度。
第二方面，提供了一种帧率调节装置，所述装置包括用于执行第一方面或第一方面任一种可能实现方式中的帧率调节方法的各个模块。
需要说明的是，第二方面所述的帧率调节装置可以是终端设备或网络设备，也可以是可设置于终端设备或网络设备中的芯片(系统)或其他部件或组件，还可以是包含终端设备或网络设备的装置，本申请对此不做限定。
此外，第二方面所述的帧率调节装置的技术效果可以参考第一方面所述的帧率调节方法的技术效果，此处不再赘述。
第三方面，提供了一种计算设备，包括存储器和处理器，所述存储器用于存储一组计算机指令，当所述处理器执行所述一组计算机指令时，用于执行第一方面中任一种可能设计中的帧率调节方法的操作步骤。
此外，第三方面所述的计算设备的技术效果可以参考第一方面所述的帧率调节方法的技术效果，此处不再赘述。
第四方面，提供一种计算机可读存储介质，包括：计算机软件指令；当计算机软件指令在计算设备中运行时，使得计算设备执行如第一方面中任意一种可能的实现方式中所述方法的操作步骤。
第五方面，提供一种计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行如第一方面中任意一种可能的实现方式中所述方法的操作步骤。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1是本申请实施例提供的一种帧率调节系统的结构示意图；
图2为本申请实施例提供的一种帧率调节方法的流程示意图；
图3为本申请实施例提供的另一种帧率调节方法的流程示意图；
图4为本申请实施例提供的一种第二帧率确定步骤的流程示意图；
图5为本申请实施例提供的一种事件帧去噪对比示意图；
图6为本申请实施例提供的一种帧率调节装置的示意图；
图7为本申请实施例提供的一种计算设备的结构示意图。
具体实施方式
本申请实施例提供一种帧率调节方法，尤其是一种基于拍摄设备与拍摄目标的相对运动状态对拍摄设备的拍摄帧率进行调节的方法，即拍摄设备根据第一帧率采集第一视频的过程中，根据拍摄设备与第一视频中目标的相对运动状态确定第二帧率，拍摄设备将拍摄帧率调节至第二帧率后进行第二视频的采集，其中，相对运动状态用于指示第一视频中目标的图像的变化快慢程度。由此，拍摄设备根据拍摄设备与第一视频中目标的相对运动状态即第一视频中目标的图像的变化快慢程度对拍摄帧率进行调节，使拍摄设备能够根据第二帧率采集目标的自身变化(例如：亮度变化)以及目标与拍摄设备的相对运动的图像信息，从而提高了帧率调节的准确度，保证了拍摄设备根据调节后的第二帧率采集的第二视频的流畅度。
下面结合附图对本申请实施例的实施方式进行说明。
图1是本申请实施例提供的一种帧率调节系统100的结构示意图。如图1所示，帧率调节系统100包括控制设备110、计算设备120、数据存储系统130和拍摄设备140。
控制设备110可以是终端，如手机、平板电脑、笔记本电脑、虚拟现实(virtual reality，VR)设备、增强现实(augmented reality，AR)设备、混合现实(Mixed Reality，MR)设备、扩展现实(Extended Reality，ER)、车载摄像设备等。
计算设备120可以是终端，还可以是其他计算设备，如服务器或者云端设备等。计算设备120可以是图形处理单元(graphic processing unit，GPU)、中央处理器(central processing unit，CPU)、其他通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。训练设备720可以是图形处理器(graphics processing unit，GPU)、神经网络处理器(neural network processing unit，NPU)、微处理器、特定应用集成电路(application-specific integrated circuit，ASIC)、或一个或多个用于控制本申请方案程序执行的集成电路。
数据存储系统130用于存储控制设备110需要调用的数据，以及控制设备110输出的数据。
拍摄设备140用于使用传感器采集传感数据，并将采集的传感数据发送至计算设备120。作为一种可能的实现方式，拍摄设备140包括图像传感器141、惯性测量单元(Inertial Measurement Unit，IMU)142和动态视觉传感器(Dynamic Vision Sensor，DVS)143。传感数据包括惯性测量单元142采集的惯性数据、图像传感器采集141的视频以及动态视觉传感器143输出的异步事件等。图像传感器141用于利用光电器件的光电转换功能将感光面上的光像转换为与光像成相应比例关系的电信号，图像传感器141可以是但不限于是互补金属氧化物半导体(Complementary Metal-Oxide-Semiconductor，CMOS)、电荷耦合元件(Charge-coupled Device，CCD)等。图像传感器141采集的视频可以包括拍摄设备140根 据第一帧率采集的第一视频，以及根据第二帧率采集的第二视频。其中，第一帧率和第二帧率用于指示拍摄设备140拍摄目标的帧率，第一帧率和第二帧率不同。
计算设备120用于利用惯性数据和/或异步事件计算第一视频中目标的相对运动状态，并根据相对运动状态确定第二帧率。进而，计算设备120向控制设备110发送确定的第二帧率。控制设备110用于实现根据第二帧率调节拍摄设备140的拍摄帧率的功能，以及指示拍摄设备140采用第二帧率拍摄第二视频的功能。
上述帧率调节系统100是对帧率调节方法应用的系统的一个举例，本申请实施例提供的帧率调节方法可以应用于各种具有拍摄功能的系统。在可能的示例中，帧率调节系统100还可以是用于进行人像摄影、电影拍摄或航空摄影的系统以及手机、平板电脑等。图1仅是本申请实施例提供的一种系统架构的示意图，图1中所示设备、器件、模块等之间的位置关系不构成任何限制，控制设备110、计算设备120和拍摄设备140可以设置于同一物理设备。示例地，拍摄设备140可以为相机，控制设备110为控制镜头的控制组件，计算设备120为相机内设置的处理器。控制设备110可以是手机中用于控制摄像头的控制组件，计算设备120是手机中的处理器，拍摄设备140是手机中的摄像头。控制设备110、计算设备120和拍摄设备140可以设置于同一物理设备还可以设置于不同的物理设备。示例地，控制设备110是导演对摄影机进行监视和控制的监控设备，计算设备120是与监控设备连接的计算机，拍摄设备140是与监控设备以及计算机连接摄影机。
例如，对于电影拍摄的场景：控制设备110是导演对摄影机进行监视和控制的监控设备。计算设备120与控制设备110连接，计算设备120用于对控制设备110传输的数据进行处理，例如对控制设备110获取的视频进行图像处理。拍摄设备140通过有线连接或无线连接方式与监控设备和计算设备120连接，拍摄设备140是用于对奔跑的马匹、滚动的足球等运动物体进行拍摄的摄影机。摄影机的默认的拍摄帧率为第一帧率，摄影机基于第一帧率对运动物体进行拍摄，并将拍摄到的第一视频发送至计算设备120。此外，摄影机还利用惯性测量单元142采集惯性数据，和/或利用动态视觉传感器143采集异步事件，并将惯性数据和/或异步事件发送至计算设备120。计算设备120根据惯性数据和/或异步事件确定摄影机与第一视频中的运动物体的相对运动状态，该相对运动状态表示运动物体与摄影机的相对运动速度的快慢程度，并根据相对运动状态确定用于对足球进行拍摄的第二帧率。计算设备120将第二帧率发送至监控设备，监控设备根据第二帧率对摄影机的拍摄帧率进行调节，使监控设备采用第二帧率对足球进行拍摄，得到第二视频。
又如，对于使用手机拍摄播放视频的显示器的场景：控制设备110、计算设备120和拍摄设备140设置于同一个手机中。控制设备110是手机中用于控制摄像头的控制组件，计算设备120是手机的处理器，拍摄设备140是手机的摄像头。用户利用手机的操作界面打开拍摄应用程序，拍摄应用程序启动摄像头，采用默认的第一帧率对正在播放视频的显示器进行拍摄，获得第一视频。摄像头利用惯性测量单元142采集惯性数据，和/或利用动态视觉传感器143采集异步事件，并将惯性数据和/或异步事件发送至处理器。处理器根据惯性数据和/或异步事件确定摄像头与第一视频中显示器相对运动状态，该相对运动状态用于指示第一视频中目标的图像的变化快慢程度，第一视频中的目标指显示器，目标的图像的变化快慢程度是指显示器的亮度变化的快慢程度。处理器根据相对运动状态确定用于对显示器进行拍摄的第二帧率。处理器将第二帧率发送至控制组件，控制组件根据第二帧率 对摄像头的拍摄帧率进行调节，使摄像头采用第二帧率对显示器进行拍摄，得到第二视频。
在上述电影拍摄的场景下或拍摄播放视频的显示器的场景下，第一视频中的运动物体与拍摄设备140的相对运动速度的快慢程度会造成拍摄设备140采集的视频的图像发生变化，显示器播放视频时的亮度变化也会造成拍摄设备140采集的视频的图像发生变化。计算设备120根据用于指示第一视频中目标的图像的变化快慢程度的相对运动状态作为指标，来确定用于对目标采集第二视频的第二帧率。因此，在目标与拍摄设备140发生相对运动或目标自身发生亮度等变化的情况下，计算设备120都能将拍摄帧率调节至适用于对目标进行拍摄的第二帧率，使拍摄设备140采用第二帧率拍摄的第二视频具有足够的流畅度，又避免拍摄设备140采用过高帧率对图像变化快慢程度较慢的目标进行拍摄，降低了拍摄设备140的功耗。
进一步地，根据控制设备110所执行的功能，还可以进一步将控制设备110细分为如图1所示的架构，如图1所示，控制设备110配置有计算模块111、I/O接口112和处理模块113。I/O接口112用于与外部设备进行数据交互。拍摄设备140向I/O接口112输入数据。输入数据可以包括图像、视频、惯性数据和异步事件等。
计算模块111用于对计算设备120的输入数据进行处理。在本申请实施例中，计算模块111用于根据计算设备120输入的第二帧率生成第二帧率调整指令，并通过I/O接口112向拍摄设备140发送第二帧率调整指令。
处理模块113用于根据I/O接口112或计算设备120接收到的输入数据进行处理。在本申请实施例中，处理模块113用于对从I/O接口112接收到的输入数据进行图像信号处理(Image Signal Processing，ISP)，生成可变帧率的媒体文件。
在控制设备110对输入数据进行图像处理，或者在控制设备110的计算模块111执行计算等相关的处理过程中，控制设备110可以调用数据存储系统130中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据和指令等存入数据存储系统130中。
最后，I/O接口112将处理结果返回给拍摄设备140，从而提供给用户，以便用户查看处理结果。
接下来请参考图2，对帧率调节方法进行详细阐述。在这里以图1中的拍摄设备140、控制设备110和计算设备120为同一物理设备举例，即拍摄设备140包括控制设备110和计算设备120。下文中以拍摄设备140为执行主体对帧率调节方法的具体步骤进行说明。
步骤210、拍摄设备140根据第一帧率采集第一视频。
拍摄设备140将拍摄帧率设置为第一帧率，根据第一帧率采集第一视频。其中，第一帧率用于指示拍摄设备140拍摄目标的拍摄帧率。第一帧率可以是拍摄设备140的默认帧率。
步骤220、拍摄设备140根据拍摄设备140与第一视频中目标的相对运动状态确定第二帧率。
拍摄设备140利用计算设备120，根据第一视频中目标的图像的变化速度确定拍摄设备140与第一视频中目标的相对运动状态，根据相对运动状态确定第二帧率。其中，第二帧率用于指示拍摄设备140对目标进行拍摄的拍摄帧率。
第一种可能的实现方式，上述第一视频中目标的图像的变化速度可以是指目标与拍摄设备140的相对运动导致的目标在拍摄设备140的拍摄画面中的位置变化速度。例如，目 标为运动的足球时，目标的图像的变化速度是足球在拍摄设备140的拍摄画面中的位置变化速度。
可选地，针对拍摄设备140自身处于运动状态，目标处于静止状态或运动速度相对拍摄设备140较慢的情况，拍摄设备140可以将拍摄设备140自身的运动速度作为拍摄设备140与第一视频中目标的相对运动速度。由于拍摄设备140的运动会导致拍摄画面发生变化，拍摄设备140与第一视频中目标的不同相对运动速度会导致目标的图像在拍摄设备140的拍摄画面中的位置发生变化，使目标在拍摄设备140中的图像发生变化。因此，拍摄设备140的运动速度与第一视频中目标的图像的变化速度存在对应关系。
计算设备120根据惯性测量单元142采集的惯性数据确定拍摄设备140的运动速度，以及拍摄设备140的运动速度与第一视频中目标的图像的变化速度的对应关系，确定第一视频中目标的图像的变化速度。
第二种可能的实现方式，上述第一视频中目标的图像的变化速度还可以是指目标自身变化导致目标在拍摄设备140的拍摄画面中的亮度变化速度。例如，目标为播放视频的显示器时，显示器由于播放的视频的亮度持续变化，拍摄设备140的拍摄画面中显示器的亮度存在变化。目标的图像的变化速度是显示器显示的画面在拍摄设备140的拍摄画面中的亮度变化速度。
可选地，针对第一视频中目标处于运动状态，例如目标相对拍摄设备140存在相对运动，或目标自身存在变化而导致拍摄设备140的拍摄画面中目标的图像存在亮度变化的情况，计算设备120可以根据动态视觉传感器143根据拍摄图像中的亮度变化输出的异步事件确定第一视频中目标的图像的变化速度。
其中，DVS又叫事件相机，是一种新型的神经拟态传感器。不同于传统的图像传感器以“帧”的方式输出信号，DVS是识别每个像素的状态，当像素亮度变化时输出异步的事件信号即异步事件。因此动态视觉传感器143能够记录拍摄设备140的拍摄画面中发生的变化，具有较高的运动信息捕捉能力。
第三种的实现方式，计算设备120还可以根据惯性测量单元142采集的惯性数据以及动态视觉传感器143输出的异步事件确定第一视频中目标的图像的变化速度。从而使拍摄设备140利用计算设备120在目标自身存在变化，或拍摄设备140自身处于运动状态，以及目标自身存在变化且拍摄设备140自身处于运动状态的情况下，更加精确地确定的第一视频中目标的图像的变化速度。
计算设备120确定第一视频中目标的图像的变化速度后，再根据第一视频中目标的图像的变化速度，确定拍摄设备140与第一视频中目标的相对运动状态。
在一些实施例中，相对运动状态包括多个目标的图像变化的快慢程度，例如静止状态、低速状态和高速状态三种快慢程度，静止状态、低速状态和高速状态分别包括不同的图像的变化速度的取值区间。计算设备120根据第一视频中目标的图像的变化速度属于静止状态、低速状态或高速状态的取值区间，来确定第一视频中目标的相对运动状态。
计算设备120根据相对运动状态和第一关联关系，确定用于对目标进行拍摄的第二帧率。第一关联关系用于指示相对运动状态与帧率的关系。例如，静止状态对应的帧率为A，低速状态对应的帧率为B，高速状态对应的帧率为C，其中，帧率的单位为赫兹(HZ)，A、B、C的数值依次增大。
本实施例中第一视频中目标的相对运动状态与相对运动状态的对应关系，以及相对运动状态与帧率的对应关系只是举例说明，不予限定。例如，第一视频中目标的图像的变化速度的每个取值表示目标处于一个相对运动状态，目标的图像的变化速度的每个取值与相对运动状态一一对应，每个相对运动状态与帧率一一对应。考虑到相对运动状态和帧率的划分粒度越小，拍摄设备140进行帧率调节的频率越高，对拍摄设备140的性能消耗较高，拍摄设备140可以根据自身性能高低对相对运动状态和帧率的划分粒度进行灵活设置。
上述确定第一视频中目标的图像的变化速度的具体步骤，以及拍摄设备140确定第二帧率的具体步骤请参考图4，在此不再赘述。
步骤230、拍摄设备140根据第二帧率采集第二视频。
拍摄设备140将图像传感器141拍摄帧率设置为第二帧率，根据第二帧率采集第二视频。例如，计算设备120将计算得到的第二帧率输出至计算模块111，计算模块111通过I/O接口112向拍摄设备140的图像传感器141发送第二帧率调节指令。其中，第二帧率调节指令用于指示拍摄设备140将图像传感器141的拍摄帧率设置为第二帧率。
基于本实施例提供的上述帧率调节方法，拍摄设备140根据自身与第一视频中目标的相对运动状态对拍摄帧率进行动态调节，在相对运动状态指示目标的图像变化快慢程度较快时提高拍摄帧率，保证了第二视频的流畅度。拍摄设备140在相对运动状态指示目标的图像变化快慢程度较慢时降低拍摄帧率，在保证第二视频的流畅度且避免拍摄设备140的功耗过高。由于拍摄设备140是根据目标的图像变化快慢程度对拍摄帧率进行调节，第一视频中的目标与拍摄设备140相对静止但自身发生变化(例如：亮度变化)或第一视频中的目标与拍摄设备140存在相对运动的情况下，拍摄设备140确定的第二帧率都能保证第二视频的流畅度，从而进一步提高了第二视频的流畅度，且提高了帧率调节的适用性。
作为一种可能的实现方式，为了方便用户观看第二视频，如图3所示，拍摄设备140在执行步骤230后，还可以执行步骤：拍摄设备140输出第二视频。
拍摄设备140的图像传感器141采用第二帧率采集第二视频的信号后，利用处理模块113对第二视频的信号进行图像信号处理实现第二视频的成像，并输出第二视频的图像。
上文结合图2对帧率调节方法的整体流程进行了说明，接下来结合图4对基于动态视觉传感器143输出的异步事件和惯性测量单元142采集的惯性数据确定第二帧率的步骤进行举例说明。步骤220可以包括如下步骤。
步骤221、拍摄设备140根据惯性测量单元142采集的惯性数据确定第一视频中目标的图像的第一变化速度。
拍摄设备140利用计算设备120对惯性数据进行预积分，得到拍摄设备140的运动速度V(Vx,Vy,Vz)，然后根据V(Vx,Vy,Vz)计算向量长度，确定第一视频中目标的图像的第一变化速度为向量长度V1。
其中，惯性测量单元142采集的惯性数据包括角速度(gx,gy,gz)和加速度(ax,ay,az)，惯性数据的示例如表1所示，计算设备120计算运动速度V(Vx,Vy,Vz)和向量长度V1的示例如表2所示。
表1

表2
步骤222、拍摄设备140根据动态视觉传感器143输出的异步事件确定第一视频中目标的图像的第二变化速度。
拍摄设备140利用计算设备120，根据动态视觉传感器143输出的异步事件确定事件积累速度，根据事件积累速度确定第一视频中目标的图像的第二变化速度。
作为一种可能的实现方式，计算设备120首先对事件帧形成参数进行初始化，根据事件帧形成参数生成事件帧，再根据事件帧确定事件积累速度。
动态视觉传感器143输出的异步事件可以表示为一个四元组(x，y，p，t),如(32，88，1，1596163547341660)，其中(x,y)＝(340，104)为事件发生的像素坐标位置，t代表事件发生的时间，可以是绝对时间戳t＝1596163547341660，也可以是以某一时刻tx＝1596163547000000为参考的相对时间戳t＝341660，p为事件的极性，p的取值可以是1或-1，1和-1分别用于指示该像素位置亮度增加或降低。异步事件的示例可以如表3所示。
表3

例如，事件帧形成参数包括事件帧起始时间t0、压帧时间间隔T0和局部区域包括的异步事件的个数N0，t0＝1648103440010042，T0＝1ms，N0＝10000。应说明的是，本实施例中t0＝1648103440010042，T0＝1ms，N0＝10000为一个示例，并未限定t0和T0的值。
拍摄设备140在上一个事件帧输出后，对异步事件进行累积计数，累积计数的值为n。拍摄设备140在t≥t0+T0，或n≥N0时形成一个事件帧。
可选地，计算设备120在根据事件帧确定事件积累速度之前对事件帧进行滤波处理。例如，计算设备120对事件帧进行空域的中值滤波、均值滤波、二值化处理或膨胀腐蚀等形态学滤波去除杂散的背景噪声，强化场景信号的边缘轮廓。请参考图5，(a)表示带噪声事件帧，(b)表示异步时空邻域去噪后的事件帧，(c)表示形态学去噪后的事件帧。目标的图像的变化快慢程度较快时，DVS电路对像素的亮度变化处理不及时导致异步事件输出时间滞后，所形成事件帧的边缘纹理区域会形成拖影或残缺。通过形态学开闭运算可以有效修复残缺的边缘，同时更彻底去除稀疏离散的背景噪声，从而提高了第一视频中目标的图像的变化速度的准确度。
本实施例中不对事件积累速度的具体含义进行限定。
例如，事件积累速度可以是指单位时间内动态视觉传感器143的感光阵列上的异步事件的总数。若V2表示第二变化速度即事件积累速度，事件积累速度计算方式可以是：V2＝N/T0。V2的示例如表4所示。
表4
又如，事件积累速度可以是指事件帧中存在事件的区域的像素个数。若V2表示第二变化速度即事件积累速度，Nt表示事件帧中存在事件的区域的像素个数，计算设备120在单个事件帧内对图5的(c)中白色的事件区域的像素个数进行累加，当像素个数累积达到N0即Nt＝N0时，确定V2＝Nt。可选地，计算设备120还可以计算事件区域占整个动态视觉传感器143的感光阵列区域的百分比，确定事件积累速度为该百分比。示例地， V2＝Nt/(W*H)，其中，W为动态视觉传感器143的宽分辨率，H为动态视觉传感器143的高分辨率，则在Nt＝N0时，V2＝10000/640/480＝3.25％。
步骤223、拍摄设备140根据第一变化速度和第二变化速度确定相对运动状态。
拍摄设备140利用计算设备120确定第一视频中目标的图像的变化速度为第一变化速度V1和第二变化速度V2中对应的相对运动状态较高的变化速度。
假设相对运动状态包括静止状态、低速状态和高速状态。相对运动状态的高低是指静止状态、低速状态和高速状态的对比，例如，高速状态高于低速状态，低速状态高于静止状态。
示例地，如表5所示，对于第一变化速度V1设置有取值区间A1、A2和A3，A1的取值区间为[0,0.5]，A2的取值区间为(0.5，2]，A3的取值区间为(2,100]。如表6所示，对于第二变化速度V2设置有取值区间B1、B2和B3，B1的取值区间为[0,1000]，B2的取值区间为(1000，10000]，B3的取值区间为(10000,100000]。其中，A1、A2和A3分别对应静止状态、低速状态和高速状态，B1、B2和B3分别对应静止状态、低速状态和高速状态。
表5
表6
可选地，上述第一变化速度V1和第二变化速度V2的每个取值区间的端点值仅为示例，本实施例对每个取值区间的端点值不作限定。
假设计算设备120确定的第一变化速度V1＝0.209095，第二变化速度V2＝18000，第一变化速度V1对应的取值区间为A1，第二变化速度V2对应的取值区间为B2，则第一变化速度V1对应的相对运动状态为静止状态，第二变化速度V2对应的相对运动状态为低速状态，计算设备120确定相对运动状态为低速状态。
步骤224、拍摄设备140根据相对运动状态和第一关联关系确定第二帧率。
拍摄设备140利用计算设备120，根据相对运动状态查询第一关联关系，确定第二帧率。其中，第一关联关系用于指示相对运动状态与帧率的关系。
示例地，如表5和表6所示，假设相对运动状态包括静止状态、低速状态和高速状态，拍摄帧率包括与静止状态、低速状态和高速状态分别对应的三个档位，例如5HZ、30HZ和200HZ。例如，计算设备120确定的相对运动状态为低速状态，则计算设备120确定第二帧率为30HZ。
拍摄帧率的档位数量不限于例中所述的3个档位，可设2个或多于2个的任意个数， 以适应不同应用场景需要。
可选地，拍摄设备140还可以利用计算设备120，根据相对运动状态对事件帧形成参数进行调节。
动态视觉传感器143对拍摄画面中的运动信息敏感，拍摄画面中目标的运动速度或状态变化会导致输出异步事件的稠密度实时变化，运动速度越快，相同时间内生成的异步事件数就越稠密，反之越稀疏。因此拍摄设备140利用计算设备120，根据相对运动状态对事件帧形成参数进行调节，能有效避免运动目标过小或缺少纹理的拍摄画面中异步事件过于稀疏，以及运动目标过大或纹理丰富场景，异步事件过于稠密的情况。在一些实施例中，拍摄设备140可以利用计算设备120根据相对运动状态对图像传感器141的采集帧率以及动态视觉传感器143的事件帧形成参数进行调节，或利用计算设备120根据相对运动状态单独对图像传感器141的采集帧率进行调节，又或者利用计算设备120根据相对运动状态单独对动态视觉传感器143的事件帧形成参数进行调节。
作为一种可能的实现方式，拍摄设备140的视场大小和分辨率一定时，运动区域越大、纹理越丰富的拍摄画面单位时间内产生的异步事件数越多，异步事件形成参数的阈值应当提高。例如，计算设备120确定单位时间内事件区域在动态视觉传感器143的感光阵列的整体占比p或已累积的事件数n，当p超过阈值6％或n超过阈值20000时，减小N0或T0的数值，从而提高事件帧的输出帧率，反之增大N0或T0的数值，从而降低事件帧的输出帧率。
基于图4示出的确定第二帧率的方式，计算设备120对惯性测量单元142采集的惯性数据和动态视觉传感器143输出的异步事件做融合处理得到第一视频中目标的相对运动状态，并以此作为第一视频中目标的图像的变化快慢的依据。对于静止状态或低速状态，拍摄设备140采用与静止状态或低速状态对应的较低帧率拍摄第二视频，降低拍摄设备140在运动较小的拍摄场景(例如：目标处于静止状态、拍摄设备140处于手持拍摄状态等)下拍摄设备140拍摄视频、处理图像的功耗。对于高速状态拍摄设备140采用与高速状态对应的较高帧率拍摄第二视频，从而提升拍摄设备140在运动较大的拍摄场景(例如：目标处于奔跑、跳跃、极限运动等运动状态，拍摄设备140处于高速运动状态)下的画面精细度和流畅度。
在另一种可能的实现方式中，拍摄设备140可以仅根据惯性测量单元142采集的惯性数据确定第一视频中目标的图像的变化速度，根据第一变化速度确定相对运动状态，再根据相对运动状态和第一关联关系确定第二帧率。该实现方式与图4所示的确定第二帧率的方式的区别在于，拍摄设备140不执行步骤222，且在步骤223中仅根据第一变化速度确定相对运动状态，在此不再赘述。
在又一种可能的实现方式中，拍摄设备140可以仅根据动态视觉传感器143输出的异步事件确定第一视频中目标的图像的变化速度，根据第一变化速度确定相对运动状态，再根据相对运动状态和第一关联关系确定第二帧率。该实现方式与图4所示的确定第二帧率的方式的区别在于，拍摄设备140不执行步骤221，且在步骤223中仅根据第二变化速度确定相对运动状态，在此不再赘述。
上文结合图2-图5详细描述了根据本实施例所提供的帧率调节方法，下面将结合图6，描述本实施例所提供的帧率调节装置。
图6为本实施例提供的可能的帧率调节装置的示意图。帧率调节装置可以用于实现上述方法实施例中执行设备的功能，因此也能实现上述方法实施例所具备的有益效果。在本实施例中，该帧率调节装置可以是如图1所示的计算设备120，还可以是应用于服务器的模块(如芯片)。
帧率调节装置600包括采集模块610和处理模块620。帧率调节装置600用于实现上述图2、图3或图4中所示的方法实施例中计算设备的功能。
采集模块610，用于根据第一帧率采集第一视频，第一帧率用于指示拍摄设备拍摄目标的帧率。
处理模块620，用于根据拍摄设备与第一视频中目标的相对运动状态确定第二帧率。其中，相对运动状态用于指示第一视频中目标的图像的变化快慢程度，第二帧率用于指示所述拍摄设备拍摄目标的帧率，第二帧率大于或小于所述第一帧率。
采集模块610，还用于根据第二帧率采集第二视频。
作为一种可能的实现方式，处理模块620在根据拍摄设备与第一视频中目标的相对运动状态确定第二帧率时，具体用于：确定第一视频中目标的图像的变化速度；根据第一视频中目标的图像的变化速度和第一关联关系确定第二帧率。其中，第一关联关系用于指示相对运动状态与帧率的关系。
作为一种可能的实现方式，拍摄设备140包括动态视觉传感器143，处理模块620在确定第一视频中目标的图像的变化速度时，具体用于：根据动态视觉传感器143输出的异步事件确定第一视频中目标的图像的变化速度。
作为一种可能的实现方式，处理模块620在根据动态视觉传感器143输出的异步事件确定第一视频中目标的图像的变化速度时，具体用于：根据异步事件确定事件积累速度；根据事件积累速度确定第一视频中目标的图像的变化速度。其中，事件累积速度用于指示动态视觉传感器143输出异步事件的速度。
例如，事件积累速度包括单位时间内动态视觉传感器143的感光阵列上的异步事件的总数。
又如，所述事件累积速度包括事件帧中存在事件的区域的像素个数，事件帧是基于单位时间内累积的多个异步事件输出的事件图像。
作为一种可能的实现方式，处理模块620在根据异步事件确定事件积累速度时，具体用于：在动态视觉传感器143输出异步事件的连续时长超过预设时长和/或异步事件的累积数量超过预设数量时，根据异步事件生成事件帧；对事件帧进行形态学滤波处理；确定形态学滤波后的事件帧中存在异步事件的区域的像素个数，得到事件累积速度。
作为一种可能的实现方式，拍摄设备140包括惯性测量单元142，处理模块620在确定第一视频中目标的图像的变化速度时，具体用于：根据惯性测量单元142采集的惯性数据和异步事件确定第一视频中目标的图像的变化速度。
应理解的是，本申请实施例的帧率调节装置600可以通过GPU、NPU、ASIC实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。在通过软件实现图2、图3或图4所示的方法时，帧率调节装置600及其各个模块也 可以为软件模块。
根据本申请实施例的帧率调节装置600可对应于执行本申请实施例中描述的方法，并且帧率调节装置600中的各个单元的上述和其它操作和/或功能分别为了实现图2、图3或图4中的各个方法的相应流程，为了简洁，在此不再赘述。
本申请实施例还提供了一种计算设备，请参考图7，图7为本申请实施例提供的一种计算设备的结构示意图。计算设备700包括存储器701、处理器702、通信接口703以及总线704。其中，存储器701、处理器702、通信接口703通过总线704实现彼此之间的通信连接。
存储器701可以是只读存储器，静态存储设备，动态存储设备或者随机存取存储器。存储器701可以存储计算机指令，当存储器701中存储的计算机指令被处理器702执行时，处理器702和通信接口703用于帧率调节方法中的步骤。存储器还可以存储数据集合，例如：存储器701中的一部分存储资源被划分成一个区域，用于存储实现本申请实施例中实现帧率调节装置600的功能的程序。
处理器702可以采用通用的CPU，应用专用集成电路(application specific integrated circuit，ASIC)，GPU或其任意组合。处理器702可以包括一个或多个芯片。处理器702可以包括NPU。处理器702根据存储器701中实现帧率调节装置600的功能的程序，执行实现采集模块610的功能和处理模块620的功能的方法。
通信接口703使用例如但不限于收发器一类的收发模块，来实现计算设备700与其他设备或通信网络之间的通信。例如，可以通过通信接口703获取第一视频或第二视频，以及反馈第二帧率。
总线704可包括在计算设备700各个部件(例如，存储器701、处理器702、通信接口703)之间传送信息的通路。
计算设备700可以为云数据中心中的计算机(例如：服务器)，或边缘数据中心中的计算机，或终端。
每个计算设备700上都可以部署计算设备120的功能。例如，GPU用于实现计算设备120的功能。
对于同一个计算设备700内部署的计算设备120的功能和控制设备110的功能，计算设备120可以通过总线704与控制设备110进行通信。
对于不同计算设备700内部署的计算设备120的功能和控制设备110的功能，计算设备120可以通过通信网络与控制设备110进行通信。
本实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于终端设备中。当然，处理器和存储介质也可以作为分立组件存在于网络 设备或终端设备中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (18)

  1. 一种帧率调节方法，其特征在于，所述方法由拍摄设备执行，所述方法包括：
    根据第一帧率采集第一视频，所述第一帧率用于指示所述拍摄设备拍摄目标的帧率；
    根据所述拍摄设备与所述第一视频中目标的相对运动状态确定第二帧率，所述相对运动状态用于指示所述第一视频中目标的图像的变化快慢程度，所述第二帧率用于指示所述拍摄设备拍摄所述目标的帧率，所述第二帧率大于或小于所述第一帧率；
    根据所述第二帧率采集第二视频。
  2. 根据权利要求1所述的方法，其特征在于，根据所述拍摄设备与所述第一视频中目标的相对运动状态确定第二帧率，包括：
    确定所述第一视频中目标的图像的变化速度；
    根据所述第一视频中目标的图像的变化速度和第一关联关系确定所述第二帧率，所述第一关联关系用于指示所述第一视频中目标的相对运动状态与帧率的关系。
  3. 根据权利要求2所述的方法，其特征在于，所述拍摄设备包括动态视觉传感器DVS；所述确定所述第一视频中目标的图像的变化速度，包括：
    根据所述DVS输出的异步事件确定所述第一视频中目标的图像的变化速度。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述DVS输出的异步事件确定所述第一视频中目标的图像的变化速度，包括：
    根据所述异步事件确定事件积累速度，所述事件累积速度用于指示所述DVS输出所述异步事件的速度；
    根据所述事件积累速度确定所述第一视频中目标的图像的变化速度。
  5. 根据权利要求4所述的方法，其特征在于，所述事件积累速度包括单位时间内所述DVS的感光阵列上的所述异步事件的总数。
  6. 根据权利要求4所述的方法，其特征在于，所述事件累积速度包括事件帧中存在事件的区域的像素个数，所述事件帧是基于单位时间内累积的多个异步事件输出的事件图像。
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述异步事件确定事件积累速度，包括：
    在所述DVS输出异步事件的连续时长超过预设时长和/或所述异步事件的累积数量超过预设数量时，根据所述异步事件生成所述事件帧；
    对所述事件帧进行形态学滤波处理；
    确定形态学滤波后的事件帧中存在异步事件的区域的像素个数，得到事件累积速度。
  8. 根据权利要求3所述的方法，其特征在于，所述拍摄设备还包括惯性测量单元IMU；所述确定所述第一视频中目标的图像的变化速度，包括：
    根据所述IMU采集的惯性数据和所述异步事件确定所述第一视频中目标的图像的变化速度。
  9. 一种帧率调节装置，其特征在于，所述装置包括：
    采集模块，用于根据第一帧率采集第一视频，所述第一帧率用于指示拍摄设备拍摄目标的帧率；
    处理模块，用于根据所述拍摄设备与所述第一视频中目标的相对运动状态确定第二帧率，所述相对运动状态用于指示所述第一视频中目标的图像的变化快慢程度，所述第二帧 率用于指示所述拍摄设备拍摄所述目标的帧率，所述第二帧率大于或小于所述第一帧率；
    所述采集模块，还用于根据所述第二帧率采集第二视频。
  10. 根据权利要求9所述的装置，其特征在于，所述处理模块具体用于：
    确定所述第一视频中目标的图像的变化速度；
    根据所述第一视频中目标的图像的变化速度和第一关联关系确定所述第二帧率，所述第一关联关系用于指示相对运动状态与帧率的关系。
  11. 根据权利要求10所述的装置，其特征在于，所述拍摄设备包括动态视觉传感器DVS，所述处理模块具体用于：
    根据所述DVS输出的异步事件确定所述第一视频中目标的图像的变化速度。
  12. 根据权利要求11所述的装置，其特征在于，所述处理模块具体用于：
    根据所述异步事件确定事件积累速度，所述事件累积速度用于指示所述DVS输出所述异步事件的速度；
    根据所述事件积累速度确定所述第一视频中目标的图像的变化速度。
  13. 根据权利要求12所述的装置，其特征在于，所述事件积累速度包括单位时间内所述DVS的感光阵列上的所述异步事件的总数。
  14. 根据权利要求12所述的装置，其特征在于，所述事件累积速度包括事件帧中存在事件的区域的像素个数，所述事件帧是基于单位时间内累积的多个异步事件输出的事件图像。
  15. 根据权利要求14所述的装置，其特征在于，所述处理模块具体用于：
    在所述DVS输出异步事件的连续时长超过预设时长和/或所述异步事件的累积数量超过预设数量时，根据所述异步事件生成所述事件帧；
    对所述事件帧进行形态学滤波处理；
    确定形态学滤波后的事件帧中存在异步事件的区域的像素个数，得到事件累积速度。
  16. 根据权利要求11所述的装置，其特征在于，所述拍摄设备还包括惯性测量单元IMU，所述处理模块具体用于：
    根据所述IMU采集的惯性数据和所述异步事件确定所述拍摄设备的运动速度；
    根据所述拍摄设备的运动速度确定所述第一视频中目标的图像的变化速度。
  17. 一种计算设备，其特征在于，所述计算设备包括存储器和至少一个处理器，所述存储器用于存储一组计算机指令；当所述处理器执行所述一组计算机指令时，执行上述权利要求1至8中任一所述的方法的操作步骤。
  18. 一种帧率调节系统，其特征在于，所述帧率调节系统包括如权利要求17所述的计算设备，所述计算设备用于执行上述权利要求1至8中任一所述的方法的操作步骤。
PCT/CN2023/096968 2022-07-06 2023-05-29 帧率调节方法、装置、设备及系统 Ceased WO2024007769A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210799229.0A CN117425064A (zh) 2022-07-06 2022-07-06 帧率调节方法、装置、设备及系统
CN202210799229.0 2022-07-06

Publications (1)

Publication Number Publication Date
WO2024007769A1 true WO2024007769A1 (zh) 2024-01-11

Family

ID=89454118

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/096968 Ceased WO2024007769A1 (zh) 2022-07-06 2023-05-29 帧率调节方法、装置、设备及系统

Country Status (2)

Country Link
CN (1) CN117425064A (zh)
WO (1) WO2024007769A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117750191A (zh) * 2024-02-19 2024-03-22 荣耀终端有限公司 一种摄像头控制方法、电子设备及存储介质
CN120568194A (zh) * 2025-07-31 2025-08-29 浙江德施曼科技智能股份有限公司 设备控制方法、设备、存储介质及程序产品

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120431501A (zh) * 2024-02-04 2025-08-05 索尼集团公司 电子设备、用于电子设备的方法、计算机可读存储介质和计算机程序产品
CN120475249A (zh) * 2024-02-05 2025-08-12 荣耀终端股份有限公司 一种图像处理方法及电子设备
CN119815171B (zh) * 2025-03-13 2025-10-24 闪极科技(深圳)有限公司 拍摄方法、可穿戴设备及可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104967803A (zh) * 2015-07-01 2015-10-07 广东欧珀移动通信有限公司 一种视频录制方法及装置
CN107396019A (zh) * 2017-08-11 2017-11-24 维沃移动通信有限公司 一种慢动作视频录制方法及移动终端
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system
CN112153301A (zh) * 2020-09-27 2020-12-29 维沃移动通信有限公司 拍摄方法和电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104967803A (zh) * 2015-07-01 2015-10-07 广东欧珀移动通信有限公司 一种视频录制方法及装置
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system
CN107396019A (zh) * 2017-08-11 2017-11-24 维沃移动通信有限公司 一种慢动作视频录制方法及移动终端
CN112153301A (zh) * 2020-09-27 2020-12-29 维沃移动通信有限公司 拍摄方法和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117750191A (zh) * 2024-02-19 2024-03-22 荣耀终端有限公司 一种摄像头控制方法、电子设备及存储介质
CN120568194A (zh) * 2025-07-31 2025-08-29 浙江德施曼科技智能股份有限公司 设备控制方法、设备、存储介质及程序产品

Also Published As

Publication number Publication date
CN117425064A (zh) 2024-01-19

Similar Documents

Publication Publication Date Title
CN112738414B (zh) 一种拍照方法、电子设备及存储介质
CN117425064A (zh) 帧率调节方法、装置、设备及系统
US8988529B2 (en) Target tracking apparatus, image tracking apparatus, methods of controlling operation of same, and digital camera
CN110445988A (zh) 图像处理方法、装置、存储介质及电子设备
CN110445989A (zh) 图像处理方法、装置、存储介质及电子设备
JP7681690B2 (ja) 撮影方法、撮影装置、電子機器及び可読記憶媒体
WO2021109409A1 (zh) 图像拍摄方法、装置、设备及存储介质
CN114390209A (zh) 拍摄方法、拍摄装置、电子设备和可读存储介质
CN107948514A (zh) 图像虚化处理方法、装置及移动设备
JP2013135468A (ja) 場面を取り込む一連の画像フレーム内の画像フレームの露光をその場面内の動きのレベルに基づいて最適化するためのカメラおよび方法
CN110445986A (zh) 图像处理方法、装置、存储介质及电子设备
US11361448B2 (en) Image processing apparatus, method of controlling image processing apparatus, and storage medium
CN111798513A (zh) 一种基于事件相机的合成孔径成像方法及系统
US20250285239A1 (en) Noise removal for surveillance camera image by means of ai-based object recognition
CN110198419A (zh) 图像处理方法、装置、存储介质及电子设备
CN112822412A (zh) 曝光方法和电子设备
CN108401109B (zh) 图像获取方法、装置、存储介质及电子设备
CN102377984A (zh) 监视影像录制方法和监视系统以及计算机程序产品
CN110351508B (zh) 基于录像模式的防抖处理方法和装置、电子设备
CN113014817B (zh) 高清高帧视频的获取方法、装置及电子设备
CN107979729A (zh) 一种显示预览图像的方法及设备
US20240357234A1 (en) Image blur degree determining method and related device thereof
JP2015103918A (ja) 撮像装置及び撮像装置の制御方法
CN115564695A (zh) 一种处理方法及电子设备
TWI887017B (zh) 電子裝置與其長曝光影片產生方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23834543

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23834543

Country of ref document: EP

Kind code of ref document: A1