WO2024078032A1 - 信号处理方法、装置、设备、存储介质及计算机程序 - Google Patents

信号处理方法、装置、设备、存储介质及计算机程序 Download PDF

Info

Publication number
WO2024078032A1
WO2024078032A1 PCT/CN2023/103954 CN2023103954W WO2024078032A1 WO 2024078032 A1 WO2024078032 A1 WO 2024078032A1 CN 2023103954 W CN2023103954 W CN 2023103954W WO 2024078032 A1 WO2024078032 A1 WO 2024078032A1
Authority
WO
WIPO (PCT)
Prior art keywords
signal
event signal
event
frame
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/103954
Other languages
English (en)
French (fr)
Inventor
刘沛
田晶铎
付轶帆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP23876241.3A priority Critical patent/EP4589549A4/en
Publication of WO2024078032A1 publication Critical patent/WO2024078032A1/zh
Priority to US19/177,148 priority patent/US20250240538A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/47—Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/44—Event detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H04N25/41—Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/581—Control of the dynamic range involving two or more exposures acquired simultaneously
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0117—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal

Definitions

  • the present application relates to the field of image processing technology, and in particular to a signal processing method, apparatus, device, storage medium and computer program.
  • the image signals obtained by using traditional image sensors to collect signals from target scenes may be blurred, overexposed, or underexposed.
  • the blurred image signals are caused by the displacement of objects in the target scene relative to the image sensor during the exposure time
  • the overexposed image signals are caused by the whitening of objects due to the excessive brightness of the target scene or the long exposure time
  • the underexposed image signals are caused by the lack of imaging details due to the low brightness of the target scene or the short exposure time.
  • DVS dynamic vision sensors
  • the present application provides a signal processing method, device, equipment, storage medium and computer program, which can improve the quality of images.
  • the technical solution is as follows:
  • a signal processing method is provided.
  • an image signal and a first event signal of a target scene are obtained, the image signal indicates the brightness information of multiple pixels corresponding to the target scene during the exposure time, the first event signal indicates the motion information of the multiple pixels during the exposure time, and the first event signal is an event signal in a frame format or an event signal in a stream format.
  • the first event signal is format-converted in the time dimension and/or space dimension to obtain a second event signal, the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the second event signal is fused with the image signal to obtain a fused signal.
  • the second event signal is obtained by converting the format of the first event signal in the time dimension and/or the space dimension. Since the second event signal is an event signal in a frame format, that is, the format of the second event signal is similar to the format of the image signal. Therefore, the second event signal has a resolution, and the resolution of the second event signal is the same as the resolution of the image signal. In this way, the second event signal can be better fused with the image signal. Moreover, since the image signal indicates the brightness information of multiple pixels during the exposure time, the event signal indicates the motion information of the multiple pixels during the exposure time. Therefore, the event signal is fused with the image signal, and the obtained fused signal includes both the brightness information of the multiple pixels and the motion information of the multiple pixels. In this way, the quality of the image can be improved by a dense fused signal that has both brightness information and motion information.
  • the exposure time refers to the exposure time of the image sensor.
  • the format of the second event signal is any one of the event frame format, the time plane format and the voxel grid format.
  • the event signal in the event frame format is a frame of event signal consisting of the accumulated value of the event polarity corresponding to each pixel within a period of time or the total number of event polarities.
  • the event signal in the time plane format is a frame of event signal consisting of the maximum timestamp corresponding to the polarity event of each pixel within a period of time.
  • the event signal in the voxel grid format is a frame of event signal consisting of the product of the accumulated value of the event polarity corresponding to each pixel within a period of time and the weight of the event signal in the time dimension.
  • the first event signal includes an event signal in a frame format and an event signal in a stream format.
  • the first event signal is converted in time dimension and/or space dimension to obtain the second event signal in different ways, which will be described in the following two situations.
  • the first event signal is an event signal in a frame format
  • the first event signal includes M frame event signals
  • the second event signal includes N frame event signals, where M and N are both integers greater than or equal to 1, and M is greater than or equal to N.
  • the M frame event signals are converted into a frame signal.
  • the sequence number is divided into N groups of event signals, each of the N groups of event signals includes at least one frame of event signals with continuous frame sequence numbers.
  • Each of the N groups of event signals is format-converted in terms of time dimension and/or space dimension to obtain the N frames of event signals.
  • a group of event signals is selected from the N groups of event signals as the target group event signals.
  • Mode 1 Performing format conversion of the time dimension and the space dimension on at least one frame of event signal included in the target group of event signals to obtain a frame of event signal after format conversion.
  • traversing the at least one frame event signal in sequence according to the above method to obtain the pixel value of each pixel in the one frame event signal after format conversion is only an example.
  • the pixel value of each pixel in the one frame event signal after format conversion can also be determined in other ways.
  • each frame event signal in the at least one frame event signal is interpolated to obtain at least one frame event signal after interpolation processing.
  • the weight of each frame event signal after interpolation processing in the at least one frame event signal after interpolation processing in the time dimension is determined.
  • the target pixel value of each pixel included in each frame event signal after interpolation processing is determined. Then, the target pixel values of the pixels at the same position in the at least one frame event signal after interpolation processing are accumulated to obtain the one frame event signal after format conversion.
  • the at least one frame event signal includes more pixels. In this way, when determining the target pixel value of any pixel in each frame event signal, it is not necessary to consider the pixel values of other pixels adjacent to the pixel in the spatial dimension, thereby improving the efficiency of signal processing.
  • the one-frame event signal obtained after format conversion by the above-mentioned method 1 is an event signal in voxel grid format. That is, when the first event signal is an event signal in frame format, according to the method provided by the above-mentioned method 1, the accumulated value of the event polarity corresponding to each pixel and the product of the weight of the first event signal in the time dimension are used as the pixel value after conversion of each pixel to obtain an event signal in voxel grid format.
  • Mode 2 Performing a format conversion of a spatial dimension on at least one frame of event signals included in the target group of event signals to obtain a frame of event signals after format conversion.
  • each frame event signal in the at least one frame event signal is interpolated to obtain at least one frame event signal after interpolation processing.
  • the pixel values of the pixels at the same position in the at least one frame event signal after interpolation processing are accumulated to obtain a frame event signal after format conversion.
  • the at least one frame event signal includes more pixels. In this way, there is no need to determine the pixel value of any pixel in each frame event signal in the spatial dimension, that is, there is no need to consider the pixel values of other pixels adjacent to the pixel in the spatial dimension, thereby improving the efficiency of signal processing.
  • the event signal of one frame obtained after the format conversion by the above method 2 is an event signal in an event frame format. That is, in the case where the first event signal is an event signal in a frame format, according to the method provided in the above method 2, the accumulated value of the event polarity corresponding to each pixel is used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format.
  • the total number of event polarities corresponding to each pixel can also be used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format, and the embodiment of the present application does not limit this.
  • the event frame format is simpler than the voxel grid format, converting the first event signal into a second event signal in the event frame format can improve the efficiency of signal processing.
  • Mode 3 Performing a time-dimensional format conversion on at least one frame of event signals included in the target group event signals to obtain a format-converted A frame event signal.
  • each frame event signal in the at least one frame event signal records the brightness change of the pixel
  • the maximum frame number in the frame numbers of the at least one frame event signal is determined as the target pixel value of the pixel. If some frame event signals in the at least one frame event signal record the brightness change of the pixel, the maximum frame number in the frame numbers of the part of the frame event signals is determined as the target pixel value of the pixel. If the at least one frame event signal does not record the brightness change of the pixel, the target pixel value of the pixel is determined to be 0.
  • the target pixel values of each pixel constitute a frame event signal after format conversion.
  • the event signal recording the brightness change of the pixel is directly selected from the at least one frame event signal according to the above method, and the maximum frame number in the frame numbers of the selected event signal is determined as the target pixel value of the pixel, or, when the at least one frame event signal does not record the brightness change of the pixel, determining the target pixel value of the pixel to be 0 is only an example.
  • the pixel value of each pixel in the frame event signal after format conversion can also be determined in other ways. For example, the at least one frame event signal is sorted in order from small to large according to the frame number to obtain the sorting result of the at least one frame event signal. Based on the sorting result and the pixel value of each pixel included in each frame event signal of the at least one frame event signal, the target pixel value of each pixel is determined.
  • the one-frame event signal obtained after format conversion by the above method 3 is an event signal in a time plane format. That is, when the first event signal is an event signal in a frame format, according to the method provided by the above method 3, the maximum frame number of the event signal in the at least one frame event signal recording the brightness change of each pixel is used as the target pixel value of each pixel to obtain a frame event signal in a time plane format.
  • Mode 4 Based on the image signal, format conversion of the time dimension and the space dimension is performed on at least one frame event signal included in the target group event signal to obtain a frame event signal after format conversion.
  • Each frame event signal in the at least one frame event signal is split according to the polarity of the event to obtain at least one frame positive event signal and at least one frame negative event signal.
  • the pixel values of each pixel in each frame positive event signal in the at least one frame positive event signal and the pixel values of each pixel in each frame negative event signal in the at least one frame negative event signal are determined.
  • the target pixel value of each pixel in each frame positive event signal is determined.
  • the target pixel value of each pixel in each frame negative event signal is determined.
  • a frame event signal after format conversion is determined.
  • the first event signal is formatted in combination with the acquired image signal, so that the converted second event signal can more accurately indicate the brightness information of the pixel at different moments within the exposure time.
  • the first event signal is an event signal in a stream format
  • the first event signal includes event signals at H moments
  • the H moments are within the exposure time
  • the second event signal includes N frame event signals
  • H and N are both integers greater than or equal to 1.
  • the exposure time is divided into N sub-time periods, and each of the N sub-time periods includes an event signal at at least one of the H moments.
  • the event signal included in each of the N sub-time periods is format-converted in the time dimension and/or space dimension to obtain the N frame event signal.
  • a sub-time period is selected from the N sub-time periods as the target sub-time period.
  • Mode 1 Perform format conversion of the time dimension and the space dimension on the event signal of at least one moment included in the target sub-time period to obtain a frame of event signal after format conversion.
  • the event signal of one frame is obtained as an event signal in a voxel grid format. That is, in the case where the first event signal is an event signal in a stream format, according to the method provided by the above method 1, the accumulated value of the event polarity corresponding to each pixel and the product of the weight of the first event signal in the time dimension are used as the pixel value after the conversion of each pixel to obtain the format conversion. Event signal in voxel grid format for the next frame.
  • Mode 2 performing a format conversion of a spatial dimension on an event signal of at least one moment included in a target sub-time period to obtain a frame of event signal after the format conversion.
  • the event polarity of each pixel included in the event signal at each moment in the at least one moment is determined.
  • the event polarity of the pixels at the same position in the event signal at the at least one moment is accumulated to obtain a frame of event signal after format conversion.
  • the event signal of one frame obtained after the format conversion by the above-mentioned method 2 is an event signal in an event frame format. That is, in the case where the first event signal is an event signal in a stream format, according to the method provided in the above-mentioned method 2, the accumulated value of the event polarity corresponding to each pixel is used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format after the format conversion.
  • the total number of event polarities corresponding to each pixel can also be used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format after the format conversion, and the embodiment of the present application does not limit this.
  • Mode 3 performing format conversion in the time dimension on the event signal of at least one moment included in the target sub-time period to obtain a frame of event signal after format conversion.
  • the spatial position coordinates of the pixel after transformation are determined from the correspondence between the spatial position coordinates before transformation and the spatial position coordinates after transformation based on the spatial position coordinates of the pixel in the event signal of each moment. If the event signal of each moment in the event signal of at least one moment records the brightness change of the pixel, the maximum timestamp among the timestamps of the event signal of at least one moment is determined as the pixel value at the spatial position coordinates after transformation of the pixel.
  • the maximum timestamp among the timestamps of the event signal of the some moments is determined as the pixel value at the spatial position coordinates after transformation of the pixel.
  • the pixel values at the spatial position coordinates after transformation of each pixel constitute a frame of event signal after format transformation.
  • any pixel in the event signal of the at least one moment directly selecting the event signal recording the brightness change of the pixel from the event signal of the at least one moment according to the above method, and determining the maximum timestamp in the timestamps of the selected event signal as the pixel value at the spatial position coordinates after the pixel is converted is only an example.
  • the pixel values of each pixel in the event signal of this frame after the format conversion can also be determined in other ways.
  • the event signals of the at least one moment are sorted in the order of the timestamps from small to large to obtain the sorting result of the event signal of the at least one moment.
  • the spatial position coordinates of each pixel in the event signal of the at least one moment are determined. Based on the sorting result and the timestamp of the event signal of each moment in the at least one moment, the pixel value of each pixel after the format conversion is determined.
  • the one-frame event signal obtained after format conversion by the above method 3 is an event signal in a time plane format. That is, when the first event signal is an event signal in a stream format, according to the method provided by the above method 3, the timestamp corresponding to the last polarity event of each pixel is used as the target pixel value corresponding to each pixel to obtain a one-frame event signal in a time plane format after format conversion.
  • Mode 4 Based on the image signal, format conversion of the event signal of at least one moment included in the target sub-time period is performed in terms of time dimension and space dimension to obtain a frame of event signal after format conversion.
  • the negative polarity value of each pixel included in the event signal at each moment in the at least one moment, and the image signal determine a frame of event signal after format conversion.
  • the first event signal is formatted in combination with the acquired image signal, so that the converted second event signal can more accurately indicate the brightness information of the pixel at different moments within the exposure time.
  • a mask area in a frame of event signals is determined, the mask area indicates an area where pixels with motion information are located in the corresponding frame of event signals, pixel values of each pixel within the mask area are fused with pixel values of corresponding pixels in the image signal, and pixel values of each pixel outside the mask area are set to pixel values of corresponding pixels in the image signal, so as to obtain a frame of fused signal.
  • each pixel outside the mask area is shielded, and there is no need to fuse the event signal and the image signal for each pixel outside the mask area.
  • the target scene is an autonomous driving scene
  • the fused signal is input into a neural network model to obtain scene perception information of the autonomous driving scene.
  • a cloud server receives an image signal and a first event signal of a target scene sent by a signal processing device, wherein the image signal indicates the brightness information of multiple pixels corresponding to the target scene during the exposure time, and the first event signal indicates the motion information of the multiple pixels during the exposure time, and the first event signal is an event signal in a frame format or an event signal in a stream format.
  • the cloud server performs format conversion of the first event signal in a time dimension and/or a space dimension to obtain a second event signal, wherein the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the second event signal is fused with the image signal to obtain a fused signal.
  • the cloud server sends the fused signal to the signal processing device.
  • the first event signal includes an event signal in a frame format and an event signal in a stream format.
  • the first event signal is converted in time dimension and/or space dimension to obtain the second event signal in different ways, which will be described in the following two situations.
  • the first event signal is an event signal in a frame format
  • the first event signal includes M frame event signals
  • the second event signal includes N frame event signals, where M and N are both integers greater than or equal to 1, and M is greater than or equal to N.
  • the M frame event signals are divided into N groups of event signals according to the frame sequence number, and each group of event signals in the N groups of event signals includes at least one frame event signal with a continuous frame sequence number.
  • Each group of event signals in the N groups of event signals is format-converted in the time dimension and/or space dimension to obtain the N frame event signals.
  • the first event signal is an event signal in a stream format
  • the first event signal includes event signals at H moments
  • the H moments are within the exposure time
  • the second event signal includes N frame event signals
  • H and N are both integers greater than or equal to 1.
  • the exposure time is divided into N sub-time periods, and each of the N sub-time periods includes an event signal at at least one moment in the H moments.
  • the event signal included in each of the N sub-time periods is format-converted in the time dimension and/or space dimension to obtain the N frame event signal.
  • a mask area in a frame of event signals is determined, the mask area indicates an area where pixels with motion information are located in the corresponding frame of event signals, pixel values of each pixel within the mask area are fused with pixel values of corresponding pixels in the image signal, and pixel values of each pixel outside the mask area are set to pixel values of corresponding pixels in the image signal, so as to obtain a frame of fused signal.
  • a signal processing device wherein the signal processing device has the function of implementing the signal processing method in the first aspect.
  • the signal processing device includes at least one module, and the at least one module is used to implement the signal processing method provided in the first aspect.
  • a cloud server comprising a communication interface and one or more processors;
  • the communication interface is used to receive an image signal of a target scene and a first event signal sent by a signal processing device, wherein the image signal indicates brightness information of a plurality of pixels corresponding to the target scene within an exposure time, and the first event signal indicates motion information of the plurality of pixels within the exposure time, and the first event signal is an event signal in a frame format or an event signal in a stream format;
  • the one or more processors are used to perform format conversion of the first event signal in a time dimension and/or a space dimension to obtain a second event signal, where the second event signal is an event signal in a frame format, and a resolution of the second event signal is the same as a resolution of the image signal;
  • the one or more processors are used to fuse the second event signal with the image signal to obtain a fused signal
  • the one or more processors are used to send the fused signal to the signal processing device through the communication interface.
  • a signal processing system which includes a signal processing device and a cloud server, wherein the signal processing device is used to send an image signal and a first event signal of a target scene to the cloud server, and the cloud server is used to implement the signal processing method provided in the second aspect above.
  • a signal processing device comprising a processor and a memory, the memory being used to store a computer program for executing the signal processing method provided in the first aspect.
  • the processor is configured to execute the computer program stored in the memory to implement the signal processing method described in the first aspect.
  • the signal processing device may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.
  • a computer-readable storage medium stores instructions.
  • the instructions When executed on a computer, When executed, the computer executes the steps of the signal processing method described in the first aspect or the second aspect.
  • a computer program product comprising instructions is provided, and when the instructions are executed on a computer, the computer executes the steps of the signal processing method described in the first aspect or the second aspect.
  • a computer program is provided, and when the computer program is executed on a computer, the computer executes the steps of the signal processing method described in the first aspect or the second aspect.
  • FIG1 is a schematic diagram of a first event signal provided in an embodiment of the present application.
  • FIG2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG3 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG4 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG5 is a schematic diagram of the architecture of a signal processing system provided in an embodiment of the present application.
  • FIG6 is a schematic diagram of a signal processing device provided in an embodiment of the present application.
  • FIG7 is a flow chart of a signal processing method provided in an embodiment of the present application.
  • FIG8 is a schematic diagram of an event signal splitting method provided in an embodiment of the present application.
  • FIG9 is a flowchart of another signal processing method provided in an embodiment of the present application.
  • FIG10 is a schematic diagram of the structure of a signal processing device provided in an embodiment of the present application.
  • FIG11 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application.
  • FIG12 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application.
  • FIG. 13 is a schematic diagram of the structure of another terminal device provided in an embodiment of the present application.
  • Image sensor It can collect signals of the target scene to obtain the image signal of the target scene, that is, the image sensor can convert the optical signal into an electrical signal, which can be a digital signal.
  • CMOS image sensors include two shutter modes: rolling shutter and global shutter.
  • DVS It can independently perceive the brightness change of each pixel in the multiple pixels corresponding to the target scene, and output the spatial position coordinates, current timestamp and brightness change information of the pixel for the pixel whose brightness change exceeds the change threshold, so as to obtain the event signal of the target scene.
  • Stream format event signal It is a set of four-dimensional arrays in the form of (x, y, t, p).
  • x and y are positive integers, representing the spatial position coordinates of the pixel
  • t is a positive real number, representing the timestamp of the brightness change of the pixel
  • p represents the polarity of the brightness change.
  • Event signal in frame format A two-dimensional array obtained by projecting the positions of each pixel whose brightness changes over a period of time onto the same two-dimensional plane.
  • the value of any pixel position in the two-dimensional array can be expressed as E(x, y, Mi ).
  • x and y are positive integers, representing the spatial position coordinates of the pixel
  • Mi is a positive integer, representing the frame number of the event signal.
  • E(x, y, Mi ) -1
  • E(x, y, Mi ) 1
  • E(x, y, Mi ) 1
  • Figure 1 is a schematic diagram of a first event signal provided by an embodiment of the present application.
  • the left figure is an event signal in a stream format over a period of time
  • the right figure is an event signal in a frame format.
  • a black dot represents a negative polarity event occurring in a pixel
  • a white dot represents a positive polarity event occurring in a pixel
  • a white triangle represents no event occurring in a pixel.
  • Coupled sensor A new type of sensor formed by coupling DVS with an image sensor. By collecting signals from the target scene through the coupled sensor, the image signal and event signal of the target scene can be obtained.
  • the signal processing method provided in the embodiments of the present application can be applied to a variety of scenarios, such as autonomous driving, terminal device imaging, and target object monitoring.
  • Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • the on-board perception device when the on-board perception device includes a DVS and an image sensor, the on-board perception device obtains an image signal of the target scene through the image sensor, and obtains a first event signal of the target scene through the DVS. Then, the first event signal is formatted according to the method provided in the embodiment of the present application to obtain a second event signal, and the second event signal is fused with the image signal to obtain a fused signal. Finally, the fused signal is input into the neural network model to obtain scene perception information, thereby realizing the perception of road conditions, vehicles, pedestrians, environmental changes and other information in the autonomous driving scenario.
  • FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • a terminal device such as a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a pocket pc (PPC), or a tablet computer
  • the terminal device simultaneously obtains the image signal and the first event signal of the target scene through the coupling sensor, and performs format conversion on the first event signal according to the method provided in the embodiment of the present application to obtain the second event signal, and then fuses the second event signal with the image signal to obtain a fused signal.
  • the fused signal is input into the image processor to obtain a real-time picture of the scene.
  • Figure 4 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • the image processing device obtains a fusion signal according to the method provided by an embodiment of the present application, and the fusion signal is a fusion signal corresponding to the current exposure time. Then, based on the fusion signal corresponding to the current exposure time and the fusion signal corresponding to the previous exposure time, video interpolation processing is performed to reduce image delay and improve image accuracy.
  • FIG. 5 is a schematic diagram of the architecture of a signal processing system provided in an embodiment of the present application.
  • the system includes an image sensor 501, a DVS502 and a signal processing device 503.
  • the image sensor 501 and the DVS502 form a coupled sensor.
  • the image sensor 501 and the DVS502 can be other devices independent of the signal processing device 503, that is, the image sensor 501, the DVS502 and the signal processing device 503 are three independent devices respectively.
  • the image sensor 501 and the DVS502 are integrated into the signal processing device 503, that is, the image sensor 501, the DVS502 and the signal processing device 503 are used as a whole device, which is not limited in the embodiment of the present application.
  • the image sensor 501 is used to output the image signal of the target scene.
  • the DVS 502 is used to output the first event signal of the target scene.
  • FIG6 is a schematic diagram of a signal processing device provided in an embodiment of the present application.
  • the signal processing device includes an input module, a conversion module, a fusion module and an output module.
  • the input module is used to input the image signal and the first event signal of the target scene, and the first event signal is an event signal in a stream format or an event signal in a frame format.
  • the conversion module is used to convert the format of the first event signal to obtain a second event signal.
  • the fusion module is used to fuse the second event signal with the image signal to obtain a fused signal.
  • the output module is used to output the fused signal.
  • the above-mentioned input module, conversion module, fusion module and output module can all be deployed on the signal processing device.
  • the input module can also be deployed on the signal processing device, and the conversion module, fusion module and output module can all be deployed on the cloud server.
  • the input module, conversion module, fusion module and output module are preferentially deployed on the chip for algorithm hardening.
  • related software can also be developed on the operating system.
  • FIG7 is a flow chart of a signal processing method provided in an embodiment of the present application.
  • the execution subject of the signal processing method provided in an embodiment of the present application is a signal processing device, and the signal processing device includes any one of the vehicle-mounted sensing device, terminal device and image processing device mentioned above. Please refer to Figure 7, the method includes the following steps.
  • Step 701 Acquire an image signal and a first event signal of a target scene, wherein the image signal indicates brightness information of a plurality of pixels corresponding to the target scene within an exposure time, and the first event signal indicates motion information of the plurality of pixels within the exposure time.
  • the first event signal is an event signal in a frame format or an event signal in a stream format.
  • the image signal of the target scene is acquired through the image sensor, and the first event signal of the target scene is acquired through the DVS.
  • the image signal and the first event signal of the target scene are acquired through a coupling sensor.
  • the image signal and the first event signal of the target scene can also be acquired by other means, which is not limited in the embodiments of the present application.
  • the exposure time refers to the exposure time of the image sensor.
  • the brightness information of each pixel within the exposure time is processed according to a relevant algorithm to obtain a frame of image signal of the target scene.
  • the first event signal includes an event signal in a stream format and an event signal in a frame format.
  • the event signal in a stream format includes event signals at H moments within the exposure time. For an event signal at any of the H moments, the pixel value of each pixel in the event signal at that moment is the event polarity corresponding to each pixel at that moment.
  • the event signal in a frame format includes M frame event signals within the exposure time. For any frame event signal in the M frame event signals, the pixel value of each pixel in the frame event signal is the event polarity corresponding to each pixel within a period of time.
  • Step 702 Perform format conversion of the time dimension and/or space dimension on the first event signal to obtain a second event signal, where the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the format of the second event signal is any one of an event frame format, a time plane format and a voxel grid format.
  • the event signal in the event frame format is a frame of event signal consisting of the accumulated value of the event polarity corresponding to each pixel within a period of time or the total number of event polarities.
  • the event signal in the time plane format is a frame of event signal consisting of the maximum timestamp corresponding to the polarity event of each pixel within a period of time.
  • the event signal in the voxel grid format is a frame of event signal consisting of the product of the accumulated value of the event polarity corresponding to each pixel within a period of time and the weight of the event signal in the time dimension.
  • the first event signal includes an event signal in a frame format and an event signal in a stream format.
  • the first event signal is converted in time dimension and/or space dimension to obtain the second event signal in different ways, which will be described in the following two situations.
  • the first event signal is an event signal in a frame format
  • the first event signal includes M frame event signals
  • the second event signal includes N frame event signals, where M and N are both integers greater than or equal to 1, and M is greater than or equal to N.
  • the M frame event signals are divided into N groups of event signals according to the frame sequence number, and each group of event signals in the N groups of event signals includes at least one frame event signal with a continuous frame sequence number.
  • Each group of event signals in the N groups of event signals is format-converted in the time dimension and/or space dimension to obtain the N frame event signals.
  • the M frames of event signals are divided into N groups of event signals according to the following formula (1).
  • ⁇ t INT(M/N) (1)
  • ⁇ t represents the offset in the time dimension, that is, the ⁇ t frame event signals with consecutive frame numbers in the M frame event signals are regarded as a group of event signals
  • N represents the total number of event signals after conversion, which is usually set in advance
  • INT(M/N) represents the floor function.
  • the M-frame event signal can also be divided into N groups of event signals in other ways. For example, since the total number of converted event signals (i.e., N) is set in advance, that is, the frame number of the converted event signal is known, and the signal processing device stores the correspondence between the frame number of the converted event signal and the offset ⁇ t on the time dimension.
  • N the total number of converted event signals
  • the offset ⁇ t on the time dimension corresponding to each frame number is obtained from the correspondence between the frame number and the offset ⁇ t on the time dimension, and then according to the offset ⁇ t on the time dimension corresponding to each frame number, the ⁇ t frame event signals with continuous frame numbers in the M-frame event signal are taken as a group of event signals.
  • the 4 frame event signals of frame numbers 1-4 in the acquired 20 frame event signals are divided into one group of event signals
  • the 8 frame event signals of frame numbers 5-12 are divided into one group of event signals
  • the 6 frame event signals of frame numbers 13-18 are divided into one group of event signals
  • the 2 frame event signals of frame numbers 19-20 are divided into one group of event signals, thereby obtaining 4 groups of event signals.
  • a group of event signals is selected from the N groups of event signals as the target group event signals.
  • Mode 1 Performing format conversion of the time dimension and the space dimension on at least one frame of event signal included in the target group of event signals to obtain a frame of event signal after format conversion.
  • the pixel value of the pixel in the frame event signal and the pixel value of the adjacent pixel in the frame event signal are accumulated to obtain the pixel value of the pixel in the spatial dimension of the frame event signal.
  • the weight of the frame event signal in the time dimension and the pixel value of the pixel in the spatial dimension of the frame event signal are multiplied to obtain the target pixel value of the pixel in the frame event signal, and the at least one frame event signal is traversed in sequence to obtain the target pixel value of the pixel in each frame event signal.
  • the target pixel values of the pixel in the at least one frame event signal are accumulated respectively to obtain the pixel value of the pixel in the frame event signal after format conversion.
  • the pixel value of each pixel in the frame event signal after format conversion can be determined according to the above steps to obtain the frame event signal after format conversion.
  • the pixel value of any pixel in a frame of event signal after format conversion is determined according to the following formula (2).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after the format conversion
  • E(x, y, Mi ) represents the pixel value of the pixel (x, y) in the event signal with the frame number Mi in the at least one frame event signal
  • ⁇ x represents the offset along the X-axis in the spatial dimension
  • ⁇ y represents the offset along the Y-axis in the spatial dimension, which is usually set in advance.
  • traversing the at least one frame event signal in sequence according to the above method to obtain the pixel value of each pixel in the one frame event signal after format conversion is only an example.
  • the pixel value of each pixel in the one frame event signal after format conversion can also be determined in other ways.
  • each frame event signal in the at least one frame event signal is interpolated to obtain at least one frame event signal after interpolation processing.
  • the weight of each frame event signal after interpolation processing in the at least one frame event signal after interpolation processing in the time dimension is determined.
  • the target pixel value of each pixel included in each frame event signal after interpolation processing is determined. Then, the target pixel values of the pixels at the same position in the at least one frame event signal after interpolation processing are accumulated to obtain a frame event signal after format conversion.
  • the frame event signal For each frame event signal, based on the pixel values of every two adjacent pixels in the spatial dimension of the frame event signal, the frame event signal is interpolated to obtain the interpolated event signal. For any pixel in the interpolated event signal of each frame, the pixel is interpolated. The pixel value in the event signal after the frame interpolation processing and the weight of the event signal after the frame interpolation processing in the time dimension are multiplied to obtain the target pixel value of the pixel in the event signal after the frame interpolation processing, and the event signal after the at least one frame interpolation processing is traversed in sequence to obtain the target pixel value of the pixel in each frame interpolation processing event signal.
  • the target pixel values corresponding to the pixel in the event signal after the at least one frame interpolation processing are accumulated to obtain the pixel value of the pixel in the frame event signal after the format conversion. That is, by interpolating the at least one frame event signal, the at least one frame event signal includes more pixels. In this way, when determining the target pixel value of any pixel in each frame event signal, there is no need to consider the pixel values of other pixels adjacent to the pixel in the spatial dimension, thereby improving the efficiency of signal processing.
  • the event signal can be interpolated by a nearest neighbor interpolation method or a bilinear interpolation method, which is not limited in the embodiments of the present application.
  • the one-frame event signal obtained after format conversion by the above-mentioned method 1 is an event signal in a voxel grid format. That is, when the first event signal is an event signal in a frame format, according to the method provided by the above-mentioned method 1, the accumulated value of the event polarity corresponding to each pixel and the product of the weight of the first event signal in the time dimension are used as the pixel value after conversion of each pixel to obtain an event signal in a voxel grid format.
  • Mode 2 Performing a format conversion of a spatial dimension on at least one frame of event signals included in the target group of event signals to obtain a frame of event signals after format conversion.
  • the pixel value of the pixel in the frame event signal is accumulated with the pixel value of the adjacent pixel in the frame event signal to obtain the pixel value of the pixel in the spatial dimension in the frame event signal, and the at least one frame event signal is traversed in sequence to obtain the pixel value of the pixel in the spatial dimension in each frame event signal. Then, the pixel value of the pixel in the spatial dimension in the at least one frame event signal is accumulated to obtain the pixel value of the pixel in the frame event signal after format conversion. In this way, for each pixel in the at least one frame event signal, the pixel value of each pixel in the frame event signal after format conversion can be determined according to the above steps to obtain the frame event signal after format conversion.
  • the pixel value of any pixel in a frame of event signal after format conversion is determined according to the following formula (3).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after the format conversion.
  • each frame event signal in the at least one frame event signal is interpolated to obtain at least one frame event signal after interpolation processing.
  • the pixel values of the pixels at the same position in the at least one frame event signal after interpolation processing are accumulated to obtain a frame event signal after format conversion.
  • interpolation processing is performed on the frame event signal based on the pixel values of every two adjacent pixels in the frame event signal in the spatial dimension to obtain the interpolated event signal.
  • the pixel values of the pixel in the at least one frame event signal after interpolation processing are accumulated to obtain the pixel value of the pixel in the frame event signal after format conversion. That is, by interpolating the at least one frame event signal, the at least one frame event signal includes more pixels. In this way, there is no need to determine the pixel value of any pixel in each frame event signal in the spatial dimension, that is, there is no need to consider the pixel values of other pixels adjacent to the pixel in the spatial dimension, thereby improving the efficiency of signal processing.
  • the event signal of one frame obtained after the format conversion by the above method 2 is an event signal in an event frame format. That is, in the case where the first event signal is an event signal in a frame format, according to the method provided in the above method 2, the accumulated value of the event polarity corresponding to each pixel is used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format.
  • the total number of event polarities corresponding to each pixel can also be used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format, and the embodiment of the present application does not limit this.
  • the event frame format is simpler than the voxel grid format, converting the first event signal into a second event signal in the event frame format can improve the efficiency of signal processing.
  • Mode 3 performing format conversion in a time dimension on at least one frame of event signal included in the target group of event signals to obtain a frame of event signal after format conversion.
  • each frame event signal in the at least one frame event signal records the brightness change of the pixel
  • the maximum frame number in the frame numbers of the at least one frame event signal is determined as the target pixel value of the pixel. If some frame event signals in the at least one frame event signal record the brightness change of the pixel, the maximum frame number in the frame numbers of the some frame event signals is determined as the target pixel value of the pixel. If the at least one frame event signal does not record the brightness change of the pixel, the target pixel value of the pixel is determined to be 0.
  • the target pixel values of each pixel constitute a frame event signal after format conversion.
  • the event signal recording the brightness change of the pixel is directly selected from the at least one frame event signal according to the above method, and the maximum frame number in the frame numbers of the selected event signal is determined as the target pixel value of the pixel, or, when the at least one frame event signal does not record the brightness change of the pixel, determining the target pixel value of the pixel to be 0 is only an example.
  • the pixel value of each pixel in the frame event signal after format conversion can also be determined in other ways. For example, the at least one frame event signal is sorted in order from small to large according to the frame number to obtain the sorting result of the at least one frame event signal. Based on the sorting result and the pixel value of each pixel included in each frame event signal in the at least one frame event signal, the target pixel value of each pixel is determined.
  • any pixel in the first frame event signal in the sorting result determine whether the pixel value of the pixel in the first frame event signal is 0. If the pixel value of the pixel in the first frame event signal is not 0, determine the frame number of the first frame event signal as the target pixel value of the pixel. If the pixel value of the pixel in the first frame event signal is 0, determine that the target pixel value of the pixel is also 0. In this way, for each pixel in the first frame event signal, the target pixel value of each pixel in the first frame event signal can be determined according to the above steps. For any pixel in the second frame event signal in the sorting result, determine whether the pixel value of the pixel in the second frame event signal is 0.
  • the target pixel value of each pixel in the second frame event signal can be determined according to the above steps. Then, traverse the at least one frame event signal in sequence according to the same method to obtain a frame event signal after format conversion.
  • the target group event signal includes three frames of event signals with frame numbers 8, 9, and 10.
  • the target pixel value of pixel (1, 1) is 8; after traversing the event signal with frame number 9, the target pixel value of pixel (1, 1) remains unchanged at 8; after traversing the event signal with frame number 10, the target pixel value of pixel (1, 1) is 10.
  • this frame event signal is obtained by performing format conversion in the time dimension on the three frame event signals with frame numbers 8, 9, and 10 in the target group event signal.
  • the one-frame event signal obtained after format conversion by the above method 3 is an event signal in a time plane format. That is, when the first event signal is an event signal in a frame format, according to the method provided by the above method 3, the maximum frame number of the event signal in the at least one frame event signal recording the brightness change of each pixel is used as the target pixel value of each pixel to obtain a frame event signal in a time plane format.
  • Mode 4 Based on the image signal, format conversion of the time dimension and the space dimension is performed on at least one frame event signal included in the target group event signal to obtain a frame event signal after format conversion.
  • Each frame event signal in the at least one frame event signal is split according to the polarity of the event to obtain at least one frame positive event signal and at least one frame negative event signal.
  • the pixel values of each pixel in each frame positive event signal in the at least one frame positive event signal and the pixel values of each pixel in each frame negative event signal in the at least one frame negative event signal are determined.
  • the target pixel value of each pixel in each frame positive event signal is determined.
  • the target pixel value of each pixel in each frame negative event signal is determined.
  • a frame event signal after format conversion is determined.
  • the event signal of each frame is split according to the polarity of the event to obtain a frame of positive event signal and a frame of negative event signal.
  • the process includes: for any pixel in the frame event signal, determining whether the pixel value of the pixel in the frame event signal is a positive value. When the pixel value of the pixel in the frame event signal is a positive value, the pixel value of the pixel is kept unchanged. When the pixel value of the pixel in the frame event signal is not a positive value, the pixel value of the pixel is set to 0, thereby obtaining a frame of positive event signal corresponding to the frame event signal. Similarly, for any pixel in the frame event signal, determining whether the pixel value of the pixel in the frame event signal is a negative value.
  • the pixel value of the pixel in the frame event signal is a negative value
  • the pixel value of the pixel is kept unchanged.
  • the pixel value of the pixel in the frame event signal is not a negative value
  • the pixel value of the pixel is set to 0, thereby obtaining a frame of negative event signal corresponding to the frame event signal.
  • Figure 8 is a schematic diagram of an event signal splitting provided by an embodiment of the present application.
  • the frame event signal includes four pixels: pixel (1, 1), pixel (1, 2), pixel (2, 1) and pixel (2, 2).
  • a frame of positive event signal and a frame of negative event signal obtained by splitting the frame event signal are shown in Figure 8.
  • the pixel value of the pixel in the positive event signal of the frame is accumulated with the pixel value of the adjacent pixel in the positive event signal of the frame to obtain the pixel value of the pixel in the spatial dimension of the positive event signal of the frame.
  • the pixel value of the pixel in the spatial dimension of the positive event signal of the frame is multiplied by the positive change threshold value to obtain the target pixel value of the pixel in the positive event signal of the frame, and the at least one frame of positive event signals is traversed in sequence to obtain the target pixel value of the pixel in each frame of the positive event signal.
  • the target pixel value of the pixel in each frame of the negative event signal is determined in a similar manner.
  • the target pixel value of the pixel in the at least one frame of the positive event signal and the target pixel value of the pixel in the at least one frame of the negative event signal are accumulated, and the accumulated calculation result is multiplied by the pixel value of the pixel in the image signal to obtain the pixel value of the pixel in the frame event signal after the format conversion.
  • the pixel value of each pixel in the frame event signal after the format conversion can be determined according to the above steps to obtain a frame event signal after the format conversion.
  • the pixel value of any pixel in the frame event signal after format conversion is determined according to the following formula (4).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after the format conversion
  • E + (x, y, Mi ) represents the pixel value of the pixel (x, y) in the positive event signal with the frame number Mi in the at least one frame positive event signal
  • C + represents the positive change threshold, which is usually set in advance.
  • E - (x, y, Mi ) represents the pixel value of the pixel (x, y) in the negative event signal with the frame number Mi in the at least one frame of the negative event signal, represents the pixel value of the pixel (x, y) in the spatial dimension in the negative event signal of the frame number Mi
  • C - represents the negative change threshold, which is usually set in advance, represents the target pixel value of the pixel (x, y) in the negative event signal of the frame number Mi
  • L(u, v) represents the pixel value of the pixel (u, v) in the image signal
  • Represents the exponential function which is used to convert the accumulated calculation results from the logarithmic domain to the linear domain.
  • the first event signal is formatted in combination with the acquired image signal, so that the converted second event signal can more accurately indicate the brightness information of the pixel at different moments within the exposure time.
  • the first event signal is an event signal in a stream format
  • the first event signal includes event signals at H moments
  • the H moments are within the exposure time
  • the second event signal includes N frame event signals
  • H and N are both integers greater than or equal to 1.
  • the exposure time is divided into N sub-time periods, and each of the N sub-time periods includes an event signal at at least one of the H moments.
  • the event signal included in each of the N sub-time periods is format-converted in the time dimension and/or space dimension to obtain the N frame event signal.
  • the process of dividing the exposure time into N sub-time periods is similar to the process of dividing M frame event signals into N groups of event signals according to the frame sequence numbers in the first case above. Therefore, you can refer to the relevant content of the first case above and will not repeat it here.
  • a sub-time period is selected from the N sub-time periods as the target sub-time period.
  • Mode 1 Perform format conversion of the time dimension and the space dimension on the event signal of at least one moment included in the target sub-time period to obtain a frame of event signal after format conversion.
  • the weight of the event signal at the moment in the time dimension is determined based on the timestamp of the event signal at the moment, the start time of the target sub-time period, and the duration of the target sub-time period. For any pixel in the event signal at the moment, the event polarity of the pixel in the event signal at the moment is multiplied by the weight of the event signal at the moment in the time dimension to obtain the target pixel value of the pixel in the event signal at the moment, and the event signal of the at least one moment is traversed in sequence to obtain the target pixel value of the pixel in the event signal at each moment.
  • the target pixel values corresponding to the pixel in the event signal of the at least one moment are accumulated to obtain the pixel value of the pixel in the event signal of the frame after the format conversion.
  • the pixel value of each pixel in the event signal of the frame after the format conversion can be determined according to the above steps, thereby obtaining a frame of event signal after the format conversion.
  • the pixel value of any pixel in the frame event signal after format conversion is determined according to the following formula (5).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after the format conversion
  • tstart represents the start time of the target sub-time period
  • tend represents the end time of the target sub-time period
  • tj represents the timestamp of the event signal at time tj in the event signal at the at least one time
  • p j ⁇ (ux j ) ⁇ (vy j ) represents the event polarity corresponding to the pixel (u, v) in the event signal at time t j
  • ⁇ (ux j ) and ⁇ (vy j ) represent the polarity function
  • the one-frame event signal obtained after format conversion by the above method 1 is an event signal in voxel grid format. That is, when the first event signal is an event signal in stream format, according to the method provided in the above method 1, the accumulated value of the event polarity corresponding to each pixel and the product of the weight of the first event signal in the time dimension are used as the pixel value after conversion of each pixel to obtain a one-frame event signal in voxel grid format after format conversion.
  • Mode 2 performing a format conversion of a spatial dimension on an event signal of at least one moment included in a target sub-time period to obtain a frame of event signal after the format conversion.
  • the event polarities corresponding to the pixel in the event signal at the at least one moment are accumulated to obtain the pixel value of the pixel in the event signal of the frame after the format conversion.
  • the pixel value of each pixel in the event signal of the frame after the format conversion can be determined according to the above steps, thereby obtaining the event signal of the frame after the format conversion.
  • the pixel value of any pixel in the frame event signal after format conversion is determined according to the following formula (6).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after the format conversion.
  • the event signal of one frame obtained after the format conversion by the above-mentioned method 2 is an event signal in an event frame format. That is, in the case where the first event signal is an event signal in a stream format, according to the method provided in the above-mentioned method 2, the accumulated value of the event polarity corresponding to each pixel is used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format after the format conversion.
  • the total number of event polarities corresponding to each pixel can also be used as the pixel value after the conversion of each pixel to obtain an event signal of one frame in an event frame format after the format conversion, and the embodiment of the present application does not limit this.
  • Mode 3 performing format conversion in the time dimension on the event signal of at least one moment included in the target sub-time period to obtain a frame of event signal after format conversion.
  • the spatial position coordinates of the pixel after transformation are determined from the correspondence between the spatial position coordinates before transformation and the spatial position coordinates after transformation based on the spatial position coordinates of the pixel in the event signal of each moment. If the event signal of each moment in the event signal of at least one moment records the brightness change of the pixel, the maximum timestamp among the timestamps of the event signal of at least one moment is determined as the pixel value at the spatial position coordinates after transformation of the pixel.
  • the maximum timestamp among the timestamps of the event signal of the some moments is determined as the pixel value at the spatial position coordinates after transformation of the pixel.
  • the pixel values at the spatial position coordinates after transformation of each pixel constitute a frame of event signal after format transformation.
  • any pixel in the event signal of the at least one moment directly selecting the event signal recording the brightness change of the pixel from the event signal of the at least one moment according to the above method, and determining the maximum timestamp in the timestamps of the selected event signal as the pixel value at the spatial position coordinates after the pixel is converted is only an example.
  • the pixel values of each pixel in the event signal of this frame after the format conversion can also be determined in other ways. For example, the event signals of the at least one moment are sorted in the order of the timestamps from small to large to obtain the sorting result of the event signal of the at least one moment.
  • the spatial position coordinates of each pixel in the event signal of the at least one moment are determined. Based on the sorting result and the timestamp of the event signal of each moment in the at least one moment, the pixel value of each pixel after the format conversion is determined.
  • the spatial position coordinates of the pixel after transformation are determined from the correspondence between the spatial position coordinates before transformation and the spatial position coordinates after transformation. Then, the timestamp of the event signal at the first moment is determined as the pixel value at the spatial position coordinates of the pixel after transformation. In this way, for each pixel in the event signal at the first moment, the pixel value at the spatial position coordinates of each pixel after transformation can be determined according to the above steps.
  • the event signals at at least one moment are traversed in sequence according to the same method, so as to obtain the pixel value of the pixel in this frame event signal after format transformation.
  • the pixel value of each pixel in this frame event signal after format transformation can be determined according to the above steps to obtain a frame event signal after format transformation.
  • the one-frame event signal obtained after format conversion by the above method 3 is an event signal in a time plane format. That is, when the first event signal is an event signal in a stream format, according to the method provided by the above method 3, the timestamp corresponding to the last polarity event of each pixel is used as the target pixel value corresponding to each pixel to obtain a one-frame event signal in a time plane format after format conversion.
  • Mode 4 Based on the image signal, format conversion of the event signal of at least one moment included in the target sub-time period is performed in terms of time dimension and space dimension to obtain a frame of event signal after format conversion.
  • the event signal of the at least one moment includes the event polarity of each pixel, and the positive change threshold, and determines the positive polarity value of each pixel included in the event signal at each moment. Based on the event polarity of each pixel included in the event signal at each moment in the at least one moment, and the negative change threshold, determine the negative polarity value of each pixel included in the event signal at each moment. Based on the positive polarity value of each pixel included in the event signal at each moment in the at least one moment, the negative polarity value of each pixel included in the event signal at each moment, and the image signal, determine a frame of event signal after format conversion.
  • the absolute value of the event polarity of the pixel in the event signal at the moment is multiplied by the positive change threshold to obtain the positive polarity value of the pixel in the event signal at the moment.
  • the absolute value of the event polarity of the pixel in the event signal at the moment is multiplied by the negative change threshold to obtain the negative polarity value of the pixel in the event signal at the moment.
  • the positive polarity value corresponding to the pixel in the event signal at the at least one moment and the negative polarity value of the pixel in the event signal at the at least one moment are accumulated, and the accumulated calculation result is multiplied by the pixel value of the pixel in the image signal to obtain the pixel value of the pixel in the event signal of the frame after the format conversion.
  • the pixel value of each pixel in the event signal of the frame after the format conversion can be determined according to the above steps to obtain a frame of event signal after the format conversion.
  • the pixel value of any pixel in the frame event signal after format conversion is determined according to the following formula (7).
  • K(u, v, Ni ) represents the pixel value of the pixel (u, v) in the frame event signal with the frame number Ni obtained after format conversion
  • C + ⁇ ( pj -1) ⁇ ( uxj ) ⁇ ( vyj ) represents the positive polarity value corresponding to the pixel (u, v) in the event signal at time tj
  • C - ⁇ ( pj +1) ⁇ ( uxj ) ⁇ ( vyj ) represents the negative polarity value corresponding to the pixel (u, v) in the event signal at time tj .
  • the first event signal is formatted in combination with the acquired image signal, so that the converted second event signal can more accurately indicate the brightness information of the pixel at different moments within the exposure time.
  • Step 703 Fuse the second event signal with the image signal to obtain a fused signal.
  • a mask area in a frame of event signals is determined, the mask area indicates an area where pixels with motion information are located in the corresponding frame of event signals, pixel values of each pixel within the mask area are fused with pixel values of corresponding pixels in the image signal, and pixel values of each pixel outside the mask area are set to pixel values of corresponding pixels in the image signal, so as to obtain a frame of fused signal.
  • a frame event signal is selected from the N frame event signal, and taking the frame event signal as an example, the process of fusing the frame event signal with the image signal to obtain a frame fused signal is introduced.
  • the pixel value of each pixel in the frame event signal may be 0 or not. If the pixel value of a certain pixel is 0, it indicates that there is no motion information of the pixel at the time indicated by the frame event signal. If the pixel value of a certain pixel is not 0, it indicates that there is motion information of the pixel at the time indicated by the frame event signal. In this way, the area where the pixels with motion information in the frame event signal are located is determined as the mask area corresponding to the frame event signal.
  • a mask signal corresponding to the frame event signal is generated. That is, for any pixel in the frame event signal, when the pixel value of the pixel is 0, the value of the mask array corresponding to the pixel is set to the first value. When the pixel value of the pixel is not 0, the value of the mask array corresponding to the pixel is set to the second value.
  • the values of the mask array corresponding to each pixel in the frame event signal constitute the mask signal corresponding to the frame event signal.
  • the area surrounded by the mask array whose value in the mask signal is the second value is the mask area corresponding to the frame event signal.
  • the first value and the second value are preset, for example, the first value is 0, and the second value is 1. Moreover, the first value and the second value can be adjusted according to different requirements.
  • a pixel with the same spatial position coordinates as the pixel is selected from the image signal to obtain the pixel corresponding to the pixel in the image signal.
  • the pixel value of the pixel in the frame event signal is fused with the pixel value of the pixel in the image signal to obtain the pixel value of the pixel in this frame fusion signal.
  • the pixel value of the pixel in the image signal is determined as the pixel value of the pixel in this frame fusion signal. In this way, for each pixel in the frame event signal, the pixel value of each pixel in this frame fusion signal can be determined according to the above steps to obtain a frame fusion signal.
  • each pixel outside the mask area is shielded, without the need to Each pixel outside the area is fused with the event signal and the image signal.
  • the pixel value of the pixel in the frame event signal when the pixel value of the pixel in the frame event signal is fused with the pixel value of the pixel in the image signal, the pixel value of the pixel in the frame event signal can be directly added to the pixel value of the pixel in the image signal, and the pixel value of the pixel in the frame event signal can also replace the pixel value of the pixel in the image signal.
  • the pixel value of the pixel in the frame event signal and the pixel value of the pixel in the image signal can also be fused in other ways, and the embodiments of the present application are not limited to this.
  • the fused signal can also be used as input for downstream tasks, including video frame insertion, image deblurring, image super-resolution, target object monitoring, depth estimation and other scenarios.
  • the fused signal is input into the neural network model to obtain the scene perception information of the autonomous driving scene. That is, the fused signal is used as the input of the downstream task of the autonomous driving scene, so as to realize the perception of road conditions, vehicles, pedestrians and environmental changes in the autonomous driving scene.
  • the second event signal and the image signal can be preprocessed respectively to improve the image quality of the second event signal and the image signal, thereby further improving the image quality of the fused signal.
  • the second event signal is filtered to eliminate noise and bad pixels in the second event signal.
  • the image signal is interpolated, denoised, de-mosaiced, and white-balanced.
  • the filtering process for the second event signal includes median filtering, Gaussian filtering, etc.
  • the second event signal and the image signal can also be preprocessed in other ways, which is not limited in the embodiment of the present application.
  • a second event signal is obtained by converting the format of the first event signal in the time dimension and/or the space dimension. Since the second event signal is an event signal in a frame format, that is, the format of the second event signal is similar to the format of the image signal. Therefore, the second event signal has a resolution, and the resolution of the second event signal is the same as the resolution of the image signal. In this way, the second event signal can be better fused with the image signal. Moreover, since the image signal indicates the brightness information of multiple pixels during the exposure time, the event signal indicates the motion information of the multiple pixels during the exposure time. Therefore, the event signal is fused with the image signal, and the obtained fused signal includes both the brightness information of the multiple pixels and the motion information of the multiple pixels. In this way, the quality of the image can be improved by a dense fused signal that has both brightness information and motion information.
  • FIG9 is a flow chart of another signal processing method provided in an embodiment of the present application, wherein the interactive execution subjects of the signal processing method provided in an embodiment of the present application are a signal processing device and a cloud server.
  • the method includes the following steps.
  • Step 901 The signal processing device obtains an image signal and a first event signal of a target scene, and sends the image signal and the first event signal of the target scene to a cloud server.
  • the image signal indicates brightness information of a plurality of pixels corresponding to the target scene within an exposure time
  • the first event signal indicates motion information of the plurality of pixels within the exposure time.
  • the first event signal is an event signal in a frame format or an event signal in a stream format.
  • Step 902 The cloud server receives the image signal and the first event signal of the target scene sent by the signal processing device, and performs format conversion of the first event signal in the time dimension and/or space dimension to obtain a second event signal, where the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the first event signal includes an event signal in a frame format and an event signal in a stream format.
  • the cloud server performs format conversion of the time dimension and/or space dimension on the first event signal to obtain the second event signal in different ways, which will be described in the following two situations.
  • the first event signal is an event signal in a frame format
  • the first event signal includes M frame event signals
  • the second event signal includes N frame event signals, where M and N are both integers greater than or equal to 1, and M is greater than or equal to N.
  • the M frame event signals are divided into N groups of event signals according to the frame sequence number, and each group of event signals in the N groups of event signals includes at least one frame event signal with a continuous frame sequence number.
  • Each group of event signals in the N groups of event signals is format-converted in the time dimension and/or space dimension to obtain the N frame event signals.
  • the first event signal is an event signal in a stream format
  • the first event signal includes event signals at H moments
  • the H moments are within the exposure time
  • the second event signal includes N frame event signals
  • H and N are both integers greater than or equal to 1.
  • the exposure time is divided into N sub-time periods, and each of the N sub-time periods includes an event signal at at least one moment in the H moments.
  • the event signal included in each of the N sub-time periods is format-converted in the time dimension and/or space dimension to obtain the N frame event signal.
  • the process of converting the format of the first event signal by the cloud server is similar to the process of converting the format of the first event signal by the signal processing device in step 702.
  • the process of format conversion of the first event signal is similar, so the relevant contents of the above step 702 may be referred to and will not be described again here.
  • Step 903 The cloud server fuses the second event signal with the image signal to obtain a fused signal.
  • the cloud server performs the following operations on each of the N frames of event signals: determining a mask area in a frame of event signal, the mask area indicating an area where pixels with motion information are located in the corresponding frame of event signal, fusing the pixel values of each pixel within the mask area with the pixel values of the corresponding pixels in the image signal, and setting the pixel values of each pixel outside the mask area to the pixel values of the corresponding pixels in the image signal to obtain a frame of fused signal.
  • Step 904 The cloud server sends the fused signal to the signal processing device.
  • the cloud server After the cloud server obtains the fused signal according to the above steps, it sends the fused signal to the signal processing device. After the signal processing device receives the fused signal sent by the cloud server, it uses the fused signal as the input of the downstream task.
  • the downstream task includes any one of video interpolation, image deblurring, image super-resolution, target object monitoring and depth estimation.
  • the fused signal is input into the neural network model to obtain the scene perception information of the autonomous driving scene. That is, the fused signal is used as the input of the downstream task of the autonomous driving scene, so as to realize the perception of road conditions, vehicles, pedestrians and environmental changes in the autonomous driving scene.
  • a second event signal is obtained by converting the format of the first event signal in the time dimension and/or the space dimension. Since the second event signal is an event signal in a frame format, that is, the format of the second event signal is similar to the format of the image signal. Therefore, the second event signal has a resolution, and the resolution of the second event signal is the same as the resolution of the image signal. In this way, the second event signal can be better fused with the image signal. Moreover, since the image signal indicates the brightness information of multiple pixels during the exposure time, the event signal indicates the motion information of the multiple pixels during the exposure time. Therefore, the event signal is fused with the image signal, and the obtained fused signal includes both the brightness information of the multiple pixels and the motion information of the multiple pixels. In this way, the quality of the image can be improved by a dense fused signal that has both brightness information and motion information.
  • FIG10 is a schematic diagram of the structure of a signal processing device provided in an embodiment of the present application, and the signal processing device can be implemented by software, hardware or a combination of both to form part or all of the signal processing equipment.
  • the device includes: an acquisition module 1001 , a conversion module 1002 and a fusion module 1003 .
  • the acquisition module 1001 is used to acquire an image signal and a first event signal of a target scene, wherein the image signal indicates brightness information of a plurality of pixels corresponding to the target scene during an exposure time, and the first event signal indicates motion information of the plurality of pixels during an exposure time, and the first event signal is an event signal in a frame format or an event signal in a stream format.
  • the detailed implementation process refers to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.
  • the conversion module 1002 is used to convert the format of the first event signal in the time dimension and/or the space dimension to obtain a second event signal, where the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
  • the fusion module 1003 is used to fuse the second event signal with the image signal to obtain a fusion signal.
  • the detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
  • the first event signal is an event signal in a frame format, and the first event signal includes M frame event signals, and the second event signal includes N frame event signals, M and N are both integers greater than or equal to 1, and M is greater than or equal to N;
  • the conversion module 1002 is specifically used to:
  • each group of event signals in the N groups of event signals includes at least one frame of event signals with consecutive frame sequence numbers;
  • a format conversion of a time dimension and/or a space dimension is performed on each group of the N groups of event signals to obtain the N frames of event signals.
  • the first event signal is an event signal in a stream format, and the first event signal includes event signals at H moments, the H moments are within the exposure time, the second event signal includes N frame event signals, and H and N are both integers greater than or equal to 1;
  • the conversion module 1002 is specifically used to:
  • each of the N sub-time periods includes an event signal of at least one of the H moments;
  • the event signal included in each of the N sub-time periods is format-converted in terms of time dimension and/or space dimension to obtain the N frames of event signals.
  • the format of the second event signal is any one of an event frame format, a time plane format and a voxel grid format.
  • the second event signal includes N frame event signals, where N is an integer greater than or equal to 1;
  • the fusion module 1003 is specifically configured to:
  • the pixel values of each pixel within the mask area are fused with the pixel values of the corresponding pixels in the image signal, and the pixel values of each pixel outside the mask area are set to the pixel values of the corresponding pixels in the image signal to obtain a frame of fused signal.
  • the target scene is an autonomous driving scene; the device further includes:
  • the input module is used to input the fusion signal into the neural network model to obtain scene perception information of the autonomous driving scene.
  • a second event signal is obtained by converting the format of the first event signal in the time dimension and/or the space dimension. Since the second event signal is an event signal in a frame format, that is, the format of the second event signal is similar to the format of the image signal. Therefore, the second event signal has a resolution, and the resolution of the second event signal is the same as the resolution of the image signal. In this way, the second event signal can be better fused with the image signal. Moreover, since the image signal indicates the brightness information of multiple pixels during the exposure time, the event signal indicates the motion information of the multiple pixels during the exposure time. Therefore, the event signal is fused with the image signal, and the obtained fused signal includes both the brightness information of the multiple pixels and the motion information of the multiple pixels. In this way, the quality of the image can be improved by a dense fused signal that has both brightness information and motion information.
  • the signal processing device provided in the above embodiment processes the signal
  • only the division of the above functional modules is used as an example.
  • the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the signal processing device provided in the above embodiment and the signal processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
  • FIG 11 is a schematic diagram of the structure of a computer device according to an embodiment of the present application, wherein the computer device is the above-mentioned signal processing device or cloud server.
  • the computer device includes at least one processor 1101, a communication bus 1102, a memory 1103 and at least one communication interface 1104.
  • the processor 1101 may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or may be one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
  • the communication bus 1102 is used to transmit information between the above components.
  • the communication bus 1102 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
  • the memory 1103 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.
  • the memory 1103 may exist independently and be connected to the processor 1101 via the communication bus 1102.
  • the memory 1103 may also be integrated with the processor 1101.
  • the communication interface 1104 uses any transceiver-like device for communicating with other devices or communication networks.
  • the communication interface 1104 includes a wired communication interface and may also include a wireless communication interface.
  • the wired communication interface may be, for example, an Ethernet interface.
  • the Ethernet interface may be an optical interface, an electrical interface, or a combination thereof.
  • the wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof, etc.
  • WLAN wireless local area network
  • the processor 1101 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 11 .
  • a computer device may include multiple processors, such as processor 1101 and processor 1105 shown in FIG11. Each of these processors may be a single-core processor or a multi-core processor.
  • the processor here may refer to one or more devices, circuits, and/or processing cores for processing data (such as computer program instructions).
  • the computer device may further include an output device 1106 and an input device 1107.
  • the output device 1106 communicates with the processor 1101 and may display information in a variety of ways.
  • the output device 1106 may be a liquid crystal display (LCD).
  • the input device 1107 may be a computer display device, such as a computer monitor, a display device, such as a LCD, a light emitting diode (LED), a cathode ray tube (CRT) display device, or a projector.
  • the input device 1107 communicates with the processor 1101 and may receive user input in a variety of ways.
  • the input device 1107 may be a mouse, a keyboard, a touch screen device, or a sensor device.
  • the memory 1103 is used to store the program code 1110 for executing the solution of the present application, and the processor 1101 can execute the program code 1110 stored in the memory 1103.
  • the program code 1110 may include one or more software modules, and the computer device may implement the signal processing method provided in the embodiment of FIG. 7 or FIG. 9 above through the processor 1101 and the program code 1110 in the memory 1103.
  • the cloud server includes a communication interface and one or more processors;
  • the communication interface is used to receive an image signal of a target scene and a first event signal sent by a signal processing device, wherein the image signal indicates brightness information of a plurality of pixels corresponding to the target scene within an exposure time, and the first event signal indicates motion information of the plurality of pixels within the exposure time, and the first event signal is an event signal in a frame format or an event signal in a stream format.
  • the detailed implementation process refers to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.
  • One or more processors are used to convert the format of the first event signal in the time dimension and/or the space dimension to obtain a second event signal, the second event signal is an event signal in a frame format, and the resolution of the second event signal is the same as the resolution of the image signal.
  • the detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
  • One or more processors are used to fuse the second event signal with the image signal to obtain a fused signal.
  • the detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
  • One or more processors are used to send the fused signal to the signal processing device through the communication interface.
  • the detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.
  • FIG. 12 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application.
  • the terminal device may be the above-mentioned signal processing device.
  • the terminal device includes a sensor unit 1210 , a computing unit 1220 , a storage unit 1240 and an interaction unit 1230 .
  • the sensor unit 1210 generally includes a visual sensor (such as a camera), a depth sensor, an IMU, a laser sensor, etc.;
  • the computing unit 1220 usually includes a CPU, a GPU, a cache, a register, etc., and is mainly used to run an operating system;
  • the storage unit 1240 mainly includes a memory and an external storage, and is mainly used for reading and writing local and temporary data;
  • the interaction unit 1230 mainly includes a display screen, a touch panel, a speaker, a microphone, etc., and is mainly used to interact with the user, obtain input, and implement the presentation algorithm effect, etc.
  • Fig. 13 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application.
  • the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and the like.
  • the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100.
  • the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently.
  • the components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc.
  • different processing units can be independent devices or integrated in one or more processors.
  • the processor 110 can execute a computer program to implement any method in the embodiments of the present application.
  • the controller may be the nerve center and command center of the terminal device 100.
  • the controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
  • the processor 110 may also be provided with a memory for storing instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory may store instructions or data that the processor 110 has just used or is cyclically using. The instruction or data can be directly called from the memory when it is used for the second time, thus avoiding repeated access and reducing the waiting time of the processor 110, thereby improving the efficiency of the system.
  • the processor 110 may include one or more interfaces.
  • the interface may include an inter-integrated circuit (IC) interface, an inter-integrated circuit sound (IS) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.
  • IC inter-integrated circuit
  • IS inter-integrated circuit sound
  • PCM pulse code modulation
  • UART universal asynchronous receiver/transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100.
  • the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 140 is used to receive charging input from a charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.
  • the wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
  • the terminal device 100 can use a wireless communication function to communicate with other devices.
  • the terminal device 100 can communicate with a second electronic device, the terminal device 100 establishes a screen projection connection with the second electronic device, and the terminal device 100 outputs the projection data to the second electronic device.
  • the projection data output by the terminal device 100 may be audio and video data.
  • Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas.
  • antenna 1 can be reused as a diversity antenna for a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module 150 can provide solutions for wireless communications including 1G/3G/4G/5G applied to the terminal device 100.
  • the mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.
  • the mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 2.
  • at least some of the functional modules of the mobile communication module 150 can be set in the processor 110.
  • at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194.
  • the modem processor may be an independent device.
  • the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) and the like applied to the terminal device 100.
  • WLAN wireless local area networks
  • BT wireless fidelity
  • GNSS global navigation satellite system
  • FM frequency modulation
  • NFC near field communication
  • IR infrared
  • the wireless communication module 160 can be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via antenna 1, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through antenna 2.
  • the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), and wireless communication technology. access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc.
  • the GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation system
  • the terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor.
  • the GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, etc.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (QLED), etc.
  • the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
  • the display screen 194 may be used to display various interfaces of the system output of the terminal device 100 .
  • the terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
  • the ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.
  • the ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the object generates an optical image through the lens and projects it onto the photosensitive element.
  • the photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor.
  • CMOS complementary metal oxide semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • the DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.
  • the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.
  • Video codecs are used to compress or decompress digital videos.
  • the terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, MPEG4, etc.
  • MPEG Moving Picture Experts Group
  • NPU is a neural network (NN) computing processor.
  • NN neural network
  • applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.
  • the internal memory 121 can be used to store computer executable program codes, which include instructions.
  • the processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121.
  • the internal memory 121 may include a program storage area and a data storage area.
  • the program storage area can store an operating system, an application required for at least one function (such as the indoor positioning method in the embodiment of the present application, etc.).
  • the data storage area can store data created during the use of the terminal device 100 (such as audio data, phone book, etc.).
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
  • UFS universal flash storage
  • the terminal device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.
  • the audio module 170 can be used to play the sound corresponding to the video. For example, when the display screen 194 displays the video playing screen, the audio module 170 outputs the sound of the video playing.
  • the audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. Signal.
  • the speaker 170A also called a "horn" is used to convert audio electrical signals into sound signals.
  • the receiver 170B also called a “handset”, is used to convert audio electrical signals into sound signals.
  • Microphone 170C also called “microphone” or “microphone”, is used to convert sound signals into electrical signals.
  • the earphone interface 170D is used to connect a wired earphone.
  • the earphone interface 170D may be the USB interface 130, or may be a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals.
  • the pressure sensor 180A can be disposed on the display screen 194.
  • the gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100.
  • the air pressure sensor 180C is used to measure air pressure.
  • the acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in all directions (including three axes or six axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.
  • the distance sensor 180F is used to measure the distance.
  • the ambient light sensor 180L is used to sense the brightness of ambient light.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the temperature sensor 180J is used to detect the temperature.
  • the touch sensor 180K is also called a "touch panel”.
  • the touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen”.
  • the touch sensor 180K is used to detect touch operations acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to the touch operation can be provided through the display screen 194.
  • the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.
  • the key 190 includes a power key, a volume key, etc.
  • the key 190 may be a mechanical key or a touch key.
  • the terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.
  • Motor 191 can generate vibration prompts.
  • the indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.
  • the SIM card interface 195 is used to connect a SIM card.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device.
  • the computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer instructions can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (digital subscriber line, DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.
  • the computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated.
  • the available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)).
  • a magnetic medium e.g., a floppy disk, a hard disk, a magnetic tape
  • an optical medium e.g., a digital versatile disc (DVD)
  • DVD digital versatile disc
  • SSD solid state disk
  • the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.
  • an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored.
  • the instructions When the instructions are executed on a computer, the computer executes the steps of the above-mentioned signal processing method.
  • the embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned signal processing method.
  • a computer program is provided, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned signal processing method.
  • the information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data for analysis, stored data, displayed data, etc.
  • signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
  • the image signal and the first event signal of the target scene involved in the embodiments of the present application are obtained with full authorization.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Television Systems (AREA)

Abstract

本申请公开了一种信号处理方法、装置、设备、存储介质及计算机程序，属于图像处理技术领域。所述方法包括：获取目标场景的图像信号和第一事件信号，第一事件信号为帧格式或流格式的事件信号，对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，将第二事件信号与图像信号进行融合，以得到融合信号。由于图像信号指示多个像素在曝光时间内的亮度信息，事件信号指示该多个像素在曝光时间内的运动信息。因此将事件信号与图像信号进行融合，得到的融合信号既包括该多个像素的亮度信息，又包括该多个像素的运动信息。这样，通过稠密的、兼具亮度信息和运动信息的融合信号能够提升图像的质量。

Description

信号处理方法、装置、设备、存储介质及计算机程序
本申请要求于2022年10月14日提交的申请号为202211259723.4、发明名称为“信号处理方法、装置、设备、存储介质及计算机程序”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，特别涉及一种信号处理方法、装置、设备、存储介质及计算机程序。
背景技术
传统的图像传感器具有高冗余、高延迟、高噪声、低动态范围等缺点，所以，通过传统的图像传感器对目标场景进行信号采集得到的图像信号会存在模糊、过曝、欠曝等情况。其中，图像信号模糊是由于目标场景中的物体在曝光时间内相对于图像传感器发生了位移，图像信号过曝是由于目标场景的亮度过高或曝光时间过长而使物体泛白，图像信号欠曝是由于目标场景的亮度过低或曝光时间过短而导致成像细节缺失。
与传统的图像传感器相比，动态视觉传感器(dynamic vision sensor，DVS)具备高动态范围、高时间分辨率、低功耗等特性。通过DVS对目标场景进行信号采集得到的事件信号能够很好地捕捉到目标场景中的运动信息。因此，DVS受到了广泛地关注。但是，通过事件信号进行后续的目标检测、图像质量提升等任务的效果欠佳。
发明内容
本申请提供了一种信号处理方法、装置、设备、存储介质及计算机程序，可以提升图像的质量。所述技术方案如下：
第一方面，提供了一种信号处理方法。在该方法中，获取目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。将第二事件信号与图像信号进行融合，以得到融合信号。
通过对第一事件信号进行时间维度和/或空间维度的格式转化，得到第二事件信号。由于第二事件信号为帧格式的事件信号，即第二事件信号的格式与图像信号的格式类似。所以，第二事件信号存在分辨率，且第二事件信号的分辨率与图像信号的分辨率相同。这样，能够将第二事件信号与图像信号更好的进行融合。而且，由于图像信号指示多个像素在曝光时间内的亮度信息，事件信号指示该多个像素在曝光时间内的运动信息。因此，将事件信号与图像信号进行融合，得到的融合信号既包括该多个像素的亮度信息，又包括该多个像素的运动信息。这样，通过稠密的、兼具亮度信息和运动信息的融合信号能够提升图像的质量。
其中，曝光时间是指图像传感器的曝光时间。第二事件信号的格式为事件帧格式、时间面格式和体素网格格式中的任一种。事件帧格式的事件信号是由一段时间内各个像素所对应的事件极性的累加值或事件极性的总数量构成的一帧事件信号。时间面格式的事件信号是由一段时间内各个像素发生极性事件所对应的最大时间戳构成的一帧事件信号。体素网格格式的事件信号是由一段时间内各个像素所对应的事件极性的累加值与事件信号在时间维度上的权重之间的乘积构成的一帧事件信号。
第一事件信号包括帧格式的事件信号和流格式的事件信号。在不同的情况下，对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号的方式不同，接下来将分为以下两种情况分别进行说明。
第一种情况，第一事件信号为帧格式的事件信号，且第一事件信号包括M帧事件信号，第二事件信号包括N帧事件信号，M和N均为大于等于1的整数，且M大于等于N。此时，将该M帧事件信号按照帧 序号划分为N组事件信号，该N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号。对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
由于对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化的方式相同，因此，从该N组事件信号中选择一组事件信号作为目标组事件信号，接下来以目标组事件信号为例，将按照不同的方式对目标组事件信号包括的至少一帧事件信号进行格式转化的方式分别进行说明。
方式1，对目标组事件信号包括的至少一帧事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一帧事件信号中每帧事件信号在时间维度上的权重。确定该至少一帧事件信号中每帧事件信号包括的各个像素在空间维度上的像素值。基于该至少一帧事件信号中每帧事件信号在时间维度上的权重、以及每帧事件信号包括的各个像素在空间维度上的像素值，确定每帧事件信号包括的各个像素的目标像素值。然后，将该至少一帧事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
需要说明的是，按照上述方法依次遍历该至少一帧事件信号，以得到各个像素在格式转化后的一帧事件信号中的像素值仅为一种示例。可选地，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，对该至少一帧事件信号中的每帧事件信号进行插值处理，以得到至少一帧插值处理后的事件信号。确定该至少一帧插值处理后的事件信号中每帧插值处理后的事件信号在时间维度上的权重。基于该至少一帧插值处理后的事件信号中每帧插值处理后的事件信号在时间维度上的权重、以及每帧插值处理后的事件信号包括的各个像素的像素值，确定每帧插值处理后的事件信号包括的各个像素的目标像素值。然后，将该至少一帧插值处理后的事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
也即是，通过对该至少一帧事件信号进行插值处理，使得该至少一帧事件信号包括更多的像素。这样，在确定每帧事件信号中任一像素的目标像素值时，就无需考虑在空间维度上与该像素相邻的其他像素的像素值，从而能够提高信号处理的效率。
通过上述方式1进行格式转化后得到的一帧事件信号为体素网格格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式1所提供的方法，将各个像素所对应的事件极性的累加值，以及第一事件信号在时间维度上的权重之间的乘积作为各个像素转化后的像素值，以得到一帧体素网格格式的事件信号。
方式2，对目标组事件信号包括的至少一帧事件信号进行空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一帧事件信号中每帧事件信号包括的各个像素在空间维度上的像素值。将该至少一帧事件信号中同一位置的像素在空间维度上的像素值累加，以得到格式转化后的一帧事件信号。
需要说明的是，按照上述方法依次遍历该至少一帧事件信号，以得到各个像素在格式转化后的这一帧事件信号中的像素值仅为一种示例。可选地，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，对该至少一帧事件信号中的每帧事件信号进行插值处理，以得到至少一帧插值处理后的事件信号。将该至少一帧插值处理后的事件信号中同一位置的像素的像素值进行累加，以得到格式转化后的一帧事件信号。
也即是，通过对该至少一帧事件信号进行插值处理，使得该至少一帧事件信号包括更多的像素。这样，无需确定每帧事件信号中任一像素在空间维度上的像素值，即无需考虑在空间维度上与该像素相邻的其他像素的像素值，从而能够提高信号处理的效率。
通过上述方式2进行格式转化后得到的一帧事件信号为事件帧格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式2所提供的方法，将各个像素所对应的事件极性的累加值作为各个像素转化后的像素值，以得到一帧事件帧格式的事件信号。当然，在实际应用中，还能够将各个像素所对应的事件极性的总数量作为各个像素转化后的像素值，以得到一帧事件帧格式的事件信号，本申请实施例对此不做限定。
由于事件帧格式比体素网格格式更简单，所以，将第一事件信号转化为事件帧格式的第二事件信号，能够提高信号处理的效率。
方式3，对目标组事件信号包括的至少一帧事件信号进行时间维度的格式转化，以得到格式转化后的 一帧事件信号。
对于该至少一帧事件信号中同一位置的像素，如果该至少一帧事件信号中的每帧事件信号均记录有该像素的亮度变化情况，则将该至少一帧事件信号的帧序号中的最大帧序号确定为该像素的目标像素值。如果该至少一帧事件信号中存在部分帧事件信号记录有该像素的亮度变化情况，则将该部分帧事件信号的帧序号中的最大帧序号确定为该像素的目标像素值。如果该至少一帧事件信号均未记录该像素的亮度变化情况，则确定该像素的目标像素值为0。将各个像素的目标像素值构成格式转化后的一帧事件信号。
需要说明的是，对于该至少一帧事件信号中的任一像素，按照上述方法直接从该至少一帧事件信号中选择记录有该像素的亮度变化情况的事件信号，并将选择出的事件信号的帧序号中的最大帧序号确定为该像素的目标像素值，或者，在该至少一帧事件信号均未记录该像素的亮度变化情况时，确定该像素的目标像素值为0仅为一种示例。可选地，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，将该至少一帧事件信号按照帧序号从小到大的顺序进行排序，以得到该至少一帧事件信号的排序结果。基于该排序结果，以及该至少一帧事件信号中每帧事件信号包括的各个像素的像素值，确定各个像素的目标像素值。
通过上述方式3进行格式转化后得到的一帧事件信号为时间面格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式3所提供的方法，将该至少一帧事件信号中记录有各个像素的亮度变化情况的事件信号的最大帧序号作为各个像素的目标像素值，以得到一帧时间面格式的事件信号。
方式4，基于图像信号，对目标组事件信号包括的至少一帧事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
将该至少一帧事件信号中的每帧事件信号按照事件的极性进行拆分，以得到至少一帧正事件信号和至少一帧负事件信号。确定该至少一帧正事件信号中每帧正事件信号包括的各个像素在空间维度上的像素值，以及该至少一帧负事件信号中每帧负事件信号包括的各个像素在空间维度上的像素值。基于该至少一帧正事件信号中每帧正事件信号包括的各个像素在空间维度上的像素值，以及正变化阈值，确定每帧正事件信号包括的各个像素的目标像素值。基于该至少一帧负事件信号中每帧负事件信号包括的各个像素在空间维度上的像素值，以及负变化阈值，确定每帧负事件信号包括的各个像素的目标像素值。基于该至少一帧正事件信号中每帧正事件信号包括的各个像素的目标像素值、该至少一帧负事件信号中每帧负事件信号包括的各个像素的目标像素值，以及图像信号，确定格式转化后的一帧事件信号。
按照上述方式4所提供的方法，结合获取到的图像信号，对第一事件信号进行格式转化，使得转化后的第二事件信号能够更准确地指示像素在曝光时间内的不同时刻的亮度信息。
第二种情况，第一事件信号为流格式的事件信号，且第一事件信号包括H个时刻的事件信号，该H个时刻位于曝光时间内，第二事件信号包括N帧事件信号，H和N均为大于等于1的整数。此时，将曝光时间划分为N个子时间段，该N个子时间段中的每个子时间段包括该H个时刻中至少一个时刻的事件信号。对该N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
由于对该N个子时间段中的每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化的方式相同，因此，从该N个子时间段中选择一个子时间段作为目标子时间段，接下来以目标子时间段为例，将按照不同的方式对目标子时间段包括的至少一个时刻的事件信号进行格式转化的方式分别进行说明。
方式1，对目标子时间段包括的至少一个时刻的事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号在时间维度上的权重。确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。基于该至少一个时刻中每个时刻的事件信号在时间维度上的权重、以及每个时刻的事件信号包括的各个像素的事件极性，确定每个时刻的事件信号包括的各个像素的目标像素值。然后，将该至少一个时刻的事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
通过上述方式1进行格式转化后得到的一帧事件信号为体素网格格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式1所提供的方法，将各个像素所对应的事件极性的累加值，以及第一事件信号在时间维度上的权重之间的乘积作为各个像素转化后的像素值，以得到格式转化 后的一帧体素网格格式的事件信号。
方式2，对目标子时间段包括的至少一个时刻的事件信号进行空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。将该至少一个时刻的事件信号中同一位置的像素的事件极性累加，以得到格式转化后的一帧事件信号。
通过上述方式2进行格式转化后得到的一帧事件信号为事件帧格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式2所提供的方法，将各个像素所对应的事件极性的累加值作为各个像素转化后的像素值，以得到格式转化后的一帧事件帧格式的事件信号。当然，在实际应用中，还能够将各个像素所对应的事件极性的总数量作为各个像素转化后的像素值，以得到格式转化后的一帧事件帧格式的事件信号，本申请实施例对此不做限定。
方式3，对目标子时间段包括的至少一个时刻的事件信号进行时间维度的格式转化，以得到格式转化后的一帧事件信号。
对于该至少一个时刻的事件信号中同一位置的像素，基于该像素在每个时刻的事件信号中的空间位置坐标，从转化前的空间位置坐标与转化后的空间位置坐标之间的对应关系中，确定该像素转化后的空间位置坐标。如果该至少一个时刻的事件信号中的每个时刻的事件信号均记录有该像素的亮度变化情况，则将该至少一个时刻的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值。如果该至少一个时刻的事件信号中存在部分时刻的事件信号记录有该像素的亮度变化情况，则将该部分时刻的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值。将各个像素转化后的空间位置坐标上的像素值构成格式转化后的一帧事件信号。
需要说明的是，对于该至少一个时刻的事件信号中的任一像素，按照上述方法直接从该至少一个时刻的事件信号中选择记录有该像素的亮度变化情况的事件信号，并将选择出的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值仅为一种示例。可选地，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，将该至少一个时刻的事件信号按照时间戳从小到大的顺序进行排序，以得到该至少一个时刻的事件信号的排序结果。基于该至少一个时刻的事件信号中各个像素的空间位置坐标，确定格式转化后的这一帧事件信号中各个像素的空间位置坐标。基于该排序结果，以及该至少一个时刻中每个时刻的事件信号的时间戳，确定各个像素转化后的像素值。
通过上述方式3进行格式转化后得到的一帧事件信号为时间面格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式3所提供的方法，将各个像素最后发生极性事件所对应的时间戳作为各个像素对应的目标像素值，以得到格式转化后的一帧时间面格式的事件信号。
方式4，基于图像信号，对目标子时间段包括的至少一个时刻的事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。基于该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性，以及正变化阈值，确定每个时刻的事件信号包括的各个像素的正极性值。基于该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性，以及负变化阈值，确定每个时刻的事件信号包括的各个像素的负极性值。基于该至少一个时刻中每个时刻的事件信号包括的各个像素的正极性值、该至少一个时刻中每个时刻的事件信号包括的各个像素的负极性值，以及图像信号，确定格式转化后的一帧事件信号。
按照上述方式4所提供的方法，结合获取到的图像信号，对第一事件信号进行格式转化，使得转化后的第二事件信号能够更准确地指示像素在曝光时间内的不同时刻的亮度信息。
对该N帧事件信号中每帧事件信号执行以下操作：确定一帧事件信号中的掩码区域，掩码区域指示对应的一帧事件信号中存在运动信息的像素所处的区域，将位于掩码区域内的各个像素的像素值与图像信号中相应像素的像素值进行融合，将位于掩码区域之外的各个像素的像素值设置为图像信号中相应像素的像素值，以得到一帧融合信号。
也即是，通过该N帧事件信号中每帧事件信号对应的掩码区域，将位于掩码区域之外的各个像素屏蔽，无需对位于掩码区域之外的各个像素进行事件信号与图像信号的融合。
可选地，目标场景为自动驾驶场景，按照上述步骤得到融合信号之后，将融合信号输入至神经网络模型，以得到自动驾驶场景的场景感知信息。
第二方面，提供了一种信号处理方法。在该方法中，云服务器接收信号处理设备发送的目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。云服务器对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。将第二事件信号与图像信号进行融合，以得到融合信号。云服务器将融合信号发送给信号处理设备。
第一事件信号包括帧格式的事件信号和流格式的事件信号。在不同的情况下，对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号的方式不同，接下来将分为以下两种情况分别进行说明。
第一种情况，第一事件信号为帧格式的事件信号，且第一事件信号包括M帧事件信号，第二事件信号包括N帧事件信号，M和N均为大于等于1的整数，且M大于等于N。此时，将该M帧事件信号按照帧序号划分为N组事件信号，该N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号。对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
第二种情况，第一事件信号为流格式的事件信号，且第一事件信号包括H个时刻的事件信号，该H个时刻位于曝光时间内，第二事件信号包括N帧事件信号，H和N均为大于等于1的整数。此时，将曝光时间划分为N个子时间段，该N个子时间段中的每个子时间段包括该H个时刻中至少一个时刻的事件信号。对该N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
对该N帧事件信号中每帧事件信号执行以下操作：确定一帧事件信号中的掩码区域，掩码区域指示对应的一帧事件信号中存在运动信息的像素所处的区域，将位于掩码区域内的各个像素的像素值与图像信号中相应像素的像素值进行融合，将位于掩码区域之外的各个像素的像素值设置为图像信号中相应像素的像素值，以得到一帧融合信号。
第三方面，提供了一种信号处理装置，所述信号处理装置具有实现上述第一方面中信号处理方法行为的功能。所述信号处理装置包括至少一个模块，该至少一个模块用于实现上述第一方面所提供的信号处理方法。
第四方面，提供了一种云服务器，所述云服务器包括通信接口和一个或多个处理器；
所述通信接口，用于接收信号处理设备发送的目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号；
所述一个或多个处理器，用于对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同；
所述一个或多个处理器，用于将第二事件信号与图像信号进行融合，以得到融合信号；
所述一个或多个处理器用于通过所述通信接口将融合信号发送给信号处理设备。
第五方面，提供了一种信号处理系统，所述信号处理系统包括信号处理设备和云服务器，所述信号处理设备用于将目标场景的图像信号和第一事件信号发送给所述云服务器，所述云服务器用于实现上述第二方面所提供的信号处理方法。
第六方面，提供了一种信号处理设备，所述信号处理设备包括处理器和存储器，所述存储器用于存储执行上述第一方面所提供的信号处理方法的计算机程序。所述处理器被配置为用于执行所述存储器中存储的计算机程序，以实现上述第一方面所述的信号处理方法。
可选地，所述信号处理设备还可以包括通信总线，该通信总线用于该处理器与存储器之间建立连接。
第七方面，提供了一种计算机可读存储介质，所述存储介质内存储有指令，当所述指令在计算机上运 行时，使得计算机执行上述第一方面或第二方面所述的信号处理方法的步骤。
第八方面，提供了一种包含指令的计算机程序产品，当所述指令在计算机上运行时，使得计算机执行上述第一方面或第二方面所述的信号处理方法的步骤。或者说，提供了一种计算机程序，当所述计算机程序在计算机上运行时，使得计算机执行上述第一方面或第二方面所述的信号处理方法的步骤。
上述第二方面至第八方面所获得的技术效果与第一方面中对应的技术手段获得的技术效果近似，在这里不再赘述。
附图说明
图1是本申请实施例提供的一种第一事件信号的示意图；
图2是本申请实施例提供的一种应用场景的示意图；
图3是本申请实施例提供的另一种应用场景的示意图；
图4是本申请实施例提供的再一种应用场景的示意图；
图5是本申请实施例提供的一种信号处理系统的架构示意图；
图6是本申请实施例提供的一种信号处理设备的示意图；
图7是本申请实施例提供的一种信号处理方法的流程图；
图8是本申请实施例提供的一种事件信号拆分的示意图；
图9是本申请实施例提供的另一种信号处理方法的流程图；
图10是本申请实施例提供的一种信号处理装置的结构示意图；
图11是本申请实施例提供的一种计算机设备的结构示意图；
图12是本申请实施例提供的一种终端设备的结构示意图；
图13是本申请实施例提供的另一种终端设备的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
在对本申请实施例提供的信号处理方法进行详细地解释说明之前，先对本申请实施例涉及的术语、业务场景和系统架构进行介绍。
为了便于理解，首先对本申请实施例涉及的术语进行解释。
图像传感器：能够对目标场景进行信号采集，以得到目标场景的图像信号，即通过图像传感器能够将光学信号转换为电信号，该电信号可以为数字信号。
目前广泛应用的两类图像传感器分别是电荷耦合器件(charged couple device，CCD)和互补金属氧化物半导体(complementary metal oxide semiconductor，CMOS)。其中，CMOS图像传感器包括卷帘快门(rolling shutter)和全局快门(global shutter)两种快门方式。
DVS：能够独立地感知目标场景对应的多个像素中每个像素的亮度变化，对亮度变化超过变化阈值的像素输出该像素的空间位置坐标、当前时间戳和该像素的亮度变化信息，从而得到目标场景的事件信号。
流格式的事件信号：是一种形如(x，y，t，p)四维数组的集合。其中，x和y为正整数，表示像素的空间位置坐标；t为正实数，表示该像素的亮度发生变化的时间戳；p表示亮度变化的极性，当p＝-1时，指示该像素对应的亮度从大变小，即该像素发生负极性事件，当p＝1时，指示该像素对应的亮度从小变大，即该像素发生正极性事件。
帧格式的事件信号：将一段时间内亮度发生变化的各个像素位置投影到同一个二维平面得到的二维数组。该二维数组中任一像素位置的值可以表示为E(x，y，Mi)。其中，x和y为正整数，表示该像素的空间位置坐标；Mi为正整数，代表事件信号的帧序号。以E(x，y，Mi)的取值范围为{-1，0，1}为例，当E(x，y，Mi)＝-1时，指示帧序号为Mi的事件信号中(x，y)位置的像素对应的亮度从大变小，即该像素发生负极性事件，当E(x，y，Mi)＝1时，指示帧序号为Mi的事件信号中(x，y)位置的像素对应的亮度从小变大，即该像素发生正极性事件，当E(x，y，Mi)＝0时，指示帧序号为Mi的事件信号中(x， y)位置的像素对应的亮度没有发生变化，即该像素无事件发生。
请参考图1，图1是本申请实施例提供的一种第一事件信号的示意图。在图1中，左图为一段时间内的流格式的事件信号，右图为一帧帧格式的事件信号。黑色圆点代表像素发生负极性事件，白色圆点代表像素发生正极性事件，白色三角形代表像素无事件发生。
耦合传感器：将DVS与图像传感器耦合形成的新型传感器。通过耦合传感器对目标场景进行信号采集，能够得到目标场景的图像信号和事件信号。
其次对本申请实施例涉及的业务场景和系统架构进行介绍。
本申请实施例提供的信号处理方法能够应用于多种场景，比如自动驾驶、终端设备成像和目标对象监控等场景。
请参考图2，图2是本申请实施例提供的一种应用场景的示意图。在图2中，在车载感知设备包括DVS与图像传感器的情况下，车载感知设备通过图像传感器获取目标场景的图像信号，通过DVS获取目标场景的第一事件信号。然后，按照本申请实施例提供的方法对第一事件信号进行格式转化，以得到第二事件信号，并将第二事件信号与图像信号进行融合，以得到融合信号。最后，将融合信号输入至神经网络模型，以得到场景感知信息，从而实现在自动驾驶场景中感知路况、车辆、行人及环境变化等信息。
请参考图3，图3是本申请实施例提供的另一种应用场景的示意图。在图3中，在个人计算机(personal computer，PC)、手机、智能手机、个人数字助手(personal digital assistant，PDA)、掌上电脑(pocket pc，PPC)、平板电脑等终端设备搭载有耦合传感器的情况下，终端设备通过耦合传感器同时获取目标场景的图像信号和第一事件信号，并按照本申请实施例提供的方法对第一事件信号进行格式转化，以得到第二事件信号，进而将第二事件信号与图像信号进行融合，以得到融合信号。然后，将融合信号输入至图像处理器，以得到场景的实时画面。
请参考图4，图4是本申请实施例提供的再一种应用场景的示意图。在图4中，在目标对象监控场景，图像处理设备按照本申请实施例提供的方法得到融合信号，该融合信号是当前曝光时间对应的融合信号。然后，基于当前曝光时间对应的融合信号，以及上一曝光时间对应的融合信号进行视频插帧处理，从而降低图像延时并提高图像精度。
请参考图5，图5是本申请实施例提供的一种信号处理系统的架构示意图。该系统包括图像传感器501、DVS502和信号处理设备503。图像传感器501和DVS502组成耦合传感器。图像传感器501以及DVS502可以是独立于信号处理设备503之外的其他设备，即图像传感器501、DVS502和信号处理设备503分别是3个各自独立的设备。或者，将图像传感器501以及DVS502集成在信号处理设备503内，即将图像传感器501、DVS502和信号处理设备503作为一个整体的设备，本申请实施例对此不做限定。
图像传感器501用于输出目标场景的图像信号。DVS502用于输出目标场景的第一事件信号。请参考图6，图6是本申请实施例提供的一种信号处理设备的示意图。信号处理设备包括输入模块、转化模块、融合模块和输出模块。输入模块用于输入目标场景的图像信号和第一事件信号，第一事件信号为流格式的事件信号或者帧格式的事件信号。转化模块用于对第一事件信号进行格式转化，以得到第二事件信号。融合模块用于将第二事件信号与图像信号进行融合，以得到融合信号。输出模块用于输出融合信号。
需要说明的是，上述提及的输入模块、转化模块、融合模块和输出模块可以均部署在信号处理设备上。当然，还能够将输入模块部署在信号处理设备上，将转化模块、融合模块和输出模块均部署在云服务器上。
可选地，为了最大限度地发挥算法效率，输入模块、转化模块、融合模块和输出模块优先部署在芯片上进行算法硬化，对于没有进行算法硬化的芯片，也可以在操作系统上开发相关软件。
本领域技术人员应能理解上述信号处理设备仅为举例，其他现有的或今后可能出现的信号处理设备如可适用于本申请实施例，也应包含在本申请实施例保护范围以内，并在此以引用方式包含于此。
需要说明的是，本申请实施例描述的业务场景和系统架构是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本领域普通技术人员可知，随着新业务场景的出现和系统架构的演变，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
图7是本申请实施例提供的一种信号处理方法的流程图，本申请实施例提供的信号处理方法的执行主体为信号处理设备，该信号处理设备包括上述所提及的车载感知设备、终端设备和图像处理设备中的任一 种。请参考图7，该方法包括如下步骤。
步骤701：获取目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。
基于上文描述，通过图像传感器获取目标场景的图像信号，通过DVS获取目标场景的第一事件信号。或者，通过耦合传感器获取目标场景的图像信号和第一事件信号。当然，在实际应用中，还能够通过其他的方式获取目标场景的图像信号和第一事件信号，本申请实施例对此不做限定。其中，曝光时间是指图像传感器的曝光时间。
对于目标场景对应的多个像素中的各个像素，按照相关算法分别对各个像素在曝光时间内的亮度信息进行处理，以得到目标场景的一帧图像信号。
第一事件信号包括流格式的事件信号和帧格式的事件信号。其中，流格式的事件信号包括位于曝光时间内的H个时刻的事件信号，对于该H个时刻中任一时刻的事件信号，该时刻的事件信号中各个像素的像素值为各个像素在该时刻对应的事件极性。帧格式的事件信号包括位于曝光时间内的M帧事件信号，对于该M帧事件信号中的任一帧事件信号，该帧事件信号中各个像素的像素值为各个像素在一段时间内对应的事件极性。
步骤702：对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。
其中，第二事件信号的格式为事件帧格式、时间面格式和体素网格格式中的任一种。事件帧格式的事件信号是由一段时间内各个像素所对应的事件极性的累加值或事件极性的总数量构成的一帧事件信号。时间面格式的事件信号是由一段时间内各个像素发生极性事件所对应的最大时间戳构成的一帧事件信号。体素网格格式的事件信号是由一段时间内各个像素所对应的事件极性的累加值与事件信号在时间维度上的权重之间的乘积构成的一帧事件信号。
基于上文描述，第一事件信号包括帧格式的事件信号和流格式的事件信号。在不同的情况下，对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号的方式不同，接下来将分为以下两种情况分别进行说明。
第一种情况，第一事件信号为帧格式的事件信号，且第一事件信号包括M帧事件信号，第二事件信号包括N帧事件信号，M和N均为大于等于1的整数，且M大于等于N。此时，将该M帧事件信号按照帧序号划分为N组事件信号，该N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号。对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
在一些实施例中，按照下述公式(1)将该M帧事件信号划分为N组事件信号。
Δt＝INT(M/N)      (1)
其中，在上述公式(1)中，Δt代表时间维度上的偏移量，即将该M帧事件信号中帧序号连续的Δt帧事件信号作为一组事件信号，N代表转化后的事件信号的总数量，通常是事先设置的，INT(M/N)代表向下取整函数。
需要说明的是，按照上述公式(1)将该M帧事件信号划分为N组事件信号仅为一种示例。在另一些实施例中，还能够按照其他的方式将该M帧事件信号划分为N组事件信号。示例地，由于转化后的事件信号的总数量(即N)是事先设置的，即转化后的事件信号的帧序号是已知的，且信号处理设备存储有转化后的事件信号的帧序号与时间维度上的偏移量Δt之间的对应关系。这样，在将该M帧事件信号按照帧序号划分为N组事件信号时，从帧序号与时间维度上的偏移量Δt之间的对应关系中，获取每个帧序号所对应的时间维度上的偏移量Δt，进而按照每个帧序号所对应的时间维度上的偏移量Δt，将该M帧事件信号中帧序号连续的Δt帧事件信号作为一组事件信号。
假设，M帧事件信号的总数量为20，即M＝20，事先设置的转化后的事件信号的总数量为4，即N＝4，信号处理设备存储的帧序号Ni与时间维度上的偏移量Δti之间的对应关系如表1所示。此时，按照帧序号Ni与时间维度上的偏移量Δti之间的对应关系，将获取到的该20帧事件信号中帧序号1-4这4帧事件信号划分为一组事件信号，将帧序号5-12这8帧事件信号划分为一组事件信号，将帧序号13-18这6帧事件信号划分为一组事件信号，将帧序号19-20这2帧事件信号划分为一组事件信号，从而得到4组事件信号。
表1
由于对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化的方式相同，因此，从该N组事件信号中选择一组事件信号作为目标组事件信号，接下来以目标组事件信号为例，将按照不同的方式对目标组事件信号包括的至少一帧事件信号进行格式转化的方式分别进行说明。
方式1，对目标组事件信号包括的至少一帧事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一帧事件信号中每帧事件信号在时间维度上的权重。确定该至少一帧事件信号中每帧事件信号包括的各个像素在空间维度上的像素值。基于该至少一帧事件信号中每帧事件信号在时间维度上的权重、以及每帧事件信号包括的各个像素在空间维度上的像素值，确定每帧事件信号包括的各个像素的目标像素值。然后，将该至少一帧事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
对于每帧事件信号中的任一像素，将该像素在该帧事件信号中的像素值与相邻像素在该帧事件信号中的像素值进行累加，以得到该帧事件信号中该像素在空间维度上的像素值。然后，将该帧事件信号在时间维度上的权重、以及该帧事件信号中该像素在空间维度上的像素值相乘，以得到该帧事件信号中该像素的目标像素值，依次遍历该至少一帧事件信号，以得到每帧事件信号中该像素的目标像素值。然后，将该像素分别在该至少一帧事件信号中的目标像素值进行累加，从而得到该像素在格式转化后的一帧事件信号中的像素值。这样，对于该至少一帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的一帧事件信号中的像素值，从而得到格式转化后的一帧事件信号。
示例地，按照下述公式(2)确定任一像素在格式转化后的一帧事件信号中的像素值。
其中，在上述公式(2)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值，E(x，y，Mi)代表在该至少一帧事件信号中帧序号为Mi的事件信号中像素(x，y)的像素值，代表在帧序号为Mi的事件信号中像素(x，y)在空间维度上的像素值，代表帧序号为Mi的事件信号在时间维度上的权重，Δx代表空间维度上沿X轴方向的偏移量，Δy代表空间维度上沿Y轴方向的偏移量，通常是事先设置的。
需要说明的是，按照上述方法依次遍历该至少一帧事件信号，以得到各个像素在格式转化后的一帧事件信号中的像素值仅为一种示例。在另一些实施例中，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，对该至少一帧事件信号中的每帧事件信号进行插值处理，以得到至少一帧插值处理后的事件信号。确定该至少一帧插值处理后的事件信号中每帧插值处理后的事件信号在时间维度上的权重。基于该至少一帧插值处理后的事件信号中每帧插值处理后的事件信号在时间维度上的权重、以及每帧插值处理后的事件信号包括的各个像素的像素值，确定每帧插值处理后的事件信号包括的各个像素的目标像素值。然后，将该至少一帧插值处理后的事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
对于每帧事件信号，基于该帧事件信号中在空间维度上每相邻两个像素的像素值，对该帧事件信号进行插值处理，以得到插值处理后的事件信号。对于每帧插值处理后的事件信号中的任一像素，将该像素在 该帧插值处理后的事件信号中的像素值、以及该帧插值处理后的事件信号在时间维度上的权重相乘，以得到该帧插值处理后的事件信号中该像素的目标像素值，依次遍历该至少一帧插值处理后的事件信号，以得到每帧插值处理后的事件信号中该像素的目标像素值。然后，将该像素分别在该至少一帧插值处理后的事件信号中对应的目标像素值进行累加，从而得到该像素在格式转化后的这一帧事件信号中的像素值。即，通过对该至少一帧事件信号进行插值处理，使得该至少一帧事件信号包括更多的像素。这样，在确定每帧事件信号中任一像素的目标像素值时，就无需考虑在空间维度上与该像素相邻的其他像素的像素值，从而能够提高信号处理的效率。
示例地，在实际应用中，能够通过最近邻插值法或双线性插值法对事件信号进行插值处理，本申请实施例对此不做限定。
通过上述方式1进行格式转化后得到的一帧事件信号为体素网格格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式1所提供的方法，将各个像素所对应的事件极性的累加值，以及第一事件信号在时间维度上的权重之间的乘积作为各个像素转化后的像素值，以得到一帧体素网格格式的事件信号。
方式2，对目标组事件信号包括的至少一帧事件信号进行空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一帧事件信号中每帧事件信号包括的各个像素在空间维度上的像素值。将该至少一帧事件信号中同一位置的像素在空间维度上的像素值累加，以得到格式转化后的一帧事件信号。
对于每帧事件信号中的任一像素，将该像素在该帧事件信号中的像素值与相邻像素在该帧事件信号中的像素值进行累加，以得到该帧事件信号中该像素在空间维度上的像素值，依次遍历该至少一帧事件信号，以得到每帧事件信号中该像素在空间维度上的像素值。然后，将该至少一帧事件信号中该像素在空间维度上的像素值进行累加，从而得到该像素在格式转化后的一帧事件信号中的像素值。这样，对于该至少一帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，从而得到格式转化后的一帧事件信号。
示例地，按照下述公式(3)确定任一像素在格式转化后的一帧事件信号中的像素值。
其中，在上述公式(3)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值。
需要说明的是，按照上述方法依次遍历该至少一帧事件信号，以得到各个像素在格式转化后的这一帧事件信号中的像素值仅为一种示例。在另一些实施例中，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，对该至少一帧事件信号中的每帧事件信号进行插值处理，以得到至少一帧插值处理后的事件信号。将该至少一帧插值处理后的事件信号中同一位置的像素的像素值进行累加，以得到格式转化后的一帧事件信号。
对于每帧事件信号，基于该帧事件信号中在空间维度上每相邻两个像素的像素值，对该帧事件信号进行插值处理，以得到插值处理后的事件信号。对于每帧插值处理后的事件信号中的任一像素，将该像素分别在该至少一帧插值处理后的事件信号中的像素值进行累加，从而得到该像素在格式转化后的这一帧事件信号中的像素值。即，通过对该至少一帧事件信号进行插值处理，使得该至少一帧事件信号包括更多的像素。这样，无需确定每帧事件信号中任一像素在空间维度上的像素值，即无需考虑在空间维度上与该像素相邻的其他像素的像素值，从而能够提高信号处理的效率。
通过上述方式2进行格式转化后得到的一帧事件信号为事件帧格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式2所提供的方法，将各个像素所对应的事件极性的累加值作为各个像素转化后的像素值，以得到一帧事件帧格式的事件信号。当然，在实际应用中，还能够将各个像素所对应的事件极性的总数量作为各个像素转化后的像素值，以得到一帧事件帧格式的事件信号，本申请实施例对此不做限定。
由于事件帧格式比体素网格格式更简单，所以，将第一事件信号转化为事件帧格式的第二事件信号，能够提高信号处理的效率。
方式3，对目标组事件信号包括的至少一帧事件信号进行时间维度的格式转化，以得到格式转化后的一帧事件信号。
对于该至少一帧事件信号中同一位置的像素，如果该至少一帧事件信号中的每帧事件信号均记录有该像素的亮度变化情况，则将该至少一帧事件信号的帧序号中的最大帧序号确定为该像素的目标像素值。如果该至少一帧事件信号中存在部分帧事件信号记录有该像素的亮度变化情况，则将该部分帧事件信号的帧序号中的最大帧序号确定为该像素的目标像素值。如果该至少一帧事件信号均未记录该像素的亮度变化情况，则确定该像素的目标像素值为0。将各个像素的目标像素值构成格式转化后的一帧事件信号。
需要说明的是，对于该至少一帧事件信号中的任一像素，按照上述方法直接从该至少一帧事件信号中选择记录有该像素的亮度变化情况的事件信号，并将选择出的事件信号的帧序号中的最大帧序号确定为该像素的目标像素值，或者，在该至少一帧事件信号均未记录该像素的亮度变化情况时，确定该像素的目标像素值为0仅为一种示例。在另一些实施例中，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，将该至少一帧事件信号按照帧序号从小到大的顺序进行排序，以得到该至少一帧事件信号的排序结果。基于该排序结果，以及该至少一帧事件信号中每帧事件信号包括的各个像素的像素值，确定各个像素的目标像素值。
对于该排序结果中的第一帧事件信号中的任一像素，确定该像素在第一帧事件信号中的像素值是否为0。在该像素在第一帧事件信号中的像素值不为0的情况下，将第一帧事件信号的帧序号确定为该像素的目标像素值。在该像素在第一帧事件信号中的像素值为0的情况下，确定该像素的目标像素值也为0。这样，对于第一帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在第一帧事件信号中的目标像素值。对于该排序结果中的第二帧事件信号中的任一像素，确定该像素在第二帧事件信号中的像素值是否为0。在该像素在第二帧事件信号中的像素值不为0的情况下，将第二帧事件信号的帧序号更新为该像素的目标像素值。在该像素在第二帧事件信号中的像素值为0的情况下，确定该像素的目标像素值保持不变。这样，对于第二帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在第二帧事件信号中的目标像素值。然后，按照相同的方法依次遍历该至少一帧事件信号，从而得到格式转化后的一帧事件信号。
假设，目标组事件信号包括帧序号为8、9、10的3帧事件信号。以像素(1，1)为例，E(1，1，8)＝4，E(1，1，9)＝0，E(1，1，10)＝3。也即是，像素(1，1)在帧序号为8的事件信号中的像素值为4，像素(1，1)在帧序号为9的事件信号中的像素值为0，像素(1，1)在帧序号为10的事件信号中的像素值为3。这样，在遍历帧序号为8的事件信号之后，像素(1，1)的目标像素值为8；在遍历帧序号为9的事件信号之后，像素(1，1)的目标像素值保持不变仍为8；在遍历帧序号为10的事件信号之后，像素(1，1)的目标像素值为10。这样，确定出像素(1，1)在格式转化后的这一帧事件信号中的像素值为10，且这一帧事件信号是对目标组事件信号中帧序号为8、9、10的3帧事件信号进行时间维度的格式转化得到的。
通过上述方式3进行格式转化后得到的一帧事件信号为时间面格式的事件信号。也即是，在第一事件信号为帧格式的事件信号的情况下，按照上述方式3所提供的方法，将该至少一帧事件信号中记录有各个像素的亮度变化情况的事件信号的最大帧序号作为各个像素的目标像素值，以得到一帧时间面格式的事件信号。
方式4，基于图像信号，对目标组事件信号包括的至少一帧事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
将该至少一帧事件信号中的每帧事件信号按照事件的极性进行拆分，以得到至少一帧正事件信号和至少一帧负事件信号。确定该至少一帧正事件信号中每帧正事件信号包括的各个像素在空间维度上的像素值，以及该至少一帧负事件信号中每帧负事件信号包括的各个像素在空间维度上的像素值。基于该至少一帧正事件信号中每帧正事件信号包括的各个像素在空间维度上的像素值，以及正变化阈值，确定每帧正事件信号包括的各个像素的目标像素值。基于该至少一帧负事件信号中每帧负事件信号包括的各个像素在空间维度上的像素值，以及负变化阈值，确定每帧负事件信号包括的各个像素的目标像素值。基于该至少一帧正事件信号中每帧正事件信号包括的各个像素的目标像素值、该至少一帧负事件信号中每帧负事件信号包括的各个像素的目标像素值，以及图像信号，确定格式转化后的一帧事件信号。
其中，将每帧事件信号按照事件的极性进行拆分，以得到一帧正事件信号和一帧负事件信号的实现过 程包括：对于该帧事件信号中的任一像素，确定该像素在该帧事件信号中的像素值是否为正值。在该像素在该帧事件信号中的像素值为正值的情况下，保持该像素的像素值不变。在该像素在该帧事件信号中的像素值不为正值的情况下，将该像素的像素值置为0，从而得到该帧事件信号对应的一帧正事件信号。同理，对于该帧事件信号中的任一像素，确定该像素在该帧事件信号中的像素值是否为负值。在该像素在该帧事件信号中的像素值为负值的情况下，保持该像素的像素值不变。在该像素在该帧事件信号中的像素值不为负值的情况下，将该像素的像素值置为0，从而得到该帧事件信号对应的一帧负事件信号。
示例地，请参考图8，图8是本申请实施例提供的一种事件信号拆分的示意图。在图8中，以帧序号为1的一帧事件信号为例，该帧事件信号包括像素(1，1)、像素(1，2)、像素(2，1)和像素(2，2)这4个像素。E(1，1，1)＝0，E(1，2，1)＝1，E(2，1，1)＝-1，E(2，2，1)＝0。也即是，像素(1，1)在该帧事件信号中的像素值为0，像素(1，2)在该帧事件信号中的像素值为1，像素(2，1)在该帧事件信号中的像素值为-1，像素(2，2)在该帧事件信号中的像素值为0。此时，将该帧事件信号拆分得到的一帧正事件信号和一帧负事件信号如图8所示。
对于每帧正事件信号中的任一像素，将该像素在该帧正事件信号中的像素值与相邻像素在该帧正事件信号中的像素值进行累加，以得到该帧正事件信号中该像素在空间维度上的像素值。将该帧正事件信号中该像素在空间维度上的像素值与正变化阈值相乘，以得到该帧正事件信号中该像素的目标像素值，依次遍历该至少一帧正事件信号，以得到每帧正事件信号中该像素的目标像素值。同理，按照类似的方法确定出每帧负事件信号中该像素的目标像素值。然后，将该像素分别在该至少一帧正事件信号中的目标像素值，以及该像素分别在该至少一帧负事件信号中的目标像素值进行累加，并将累加得到的计算结果与图像信号中该像素的像素值相乘，从而得到该像素在格式转化后的这一帧事件信号中的像素值。这样，对于该至少一帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，从而得到格式转化后的一帧事件信号。
示例地，按照下述公式(4)确定任一像素在格式转化后的这一帧事件信号中的像素值。
其中，在上述公式(4)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值，E+(x，y，Mi)代表在该至少一帧正事件信号中帧序号为Mi的正事件信号中像素(x，y)的像素值，代表在帧序号为Mi的正事件信号中像素(x，y)在空间维度上的像素值，C+代表正变化阈值，通常是事先设置的，代表在帧序号为Mi的正事件信号中像素(x，y)的目标像素值，E-(x，y，Mi)代表在该至少一帧负事件信号中帧序号为Mi的负事件信号中像素(x，y)的像素值，代表在帧序号为Mi的负事件信号中像素(x，y)在空间维度上的像素值，C-代表负变化阈值，通常是事先设置的，代表在帧序号为Mi的负事件信号中像素(x，y)的目标像素值，L(u，v)代表在图像信号中像素(u，v)的像素值，代表指数函数，用于将累加得到的计算结果从对数域转化至线性域。
按照上述方式4所提供的方法，结合获取到的图像信号，对第一事件信号进行格式转化，使得转化后的第二事件信号能够更准确地指示像素在曝光时间内的不同时刻的亮度信息。
第二种情况，第一事件信号为流格式的事件信号，且第一事件信号包括H个时刻的事件信号，该H个时刻位于曝光时间内，第二事件信号包括N帧事件信号，H和N均为大于等于1的整数。此时，将曝光时间划分为N个子时间段，该N个子时间段中的每个子时间段包括该H个时刻中至少一个时刻的事件信号。对该N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
其中，将曝光时间划分为N个子时间段的过程与上述第一种情况中将M帧事件信号按照帧序号划分为N组事件信号的过程相似，所以，可以参考上述第一种情况的相关内容，此处不再赘述。
由于对该N个子时间段中的每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化的方式相同，因此，从该N个子时间段中选择一个子时间段作为目标子时间段，接下来以目标子时间段为例，将按照不同的方式对目标子时间段包括的至少一个时刻的事件信号进行格式转化的方式分别进行说明。
方式1，对目标子时间段包括的至少一个时刻的事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号在时间维度上的权重。确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。基于该至少一个时刻中每个时刻的事件信号在时间维度上的权重、以及每个时刻的事件信号包括的各个像素的事件极性，确定每个时刻的事件信号包括的各个像素的目标像素值。然后，将该至少一个时刻的事件信号中同一位置的像素的目标像素值累加，以得到格式转化后的一帧事件信号。
对于该至少一个时刻中每个时刻的事件信号，基于该时刻的事件信号的时间戳、目标子时间段的起始时间，以及目标子时间段的时长，确定该时刻的事件信号在时间维度上的权重。对于该时刻的事件信号中的任一像素，将该像素在该时刻的事件信号中的事件极性与该时刻的事件信号在时间维度上的权重相乘，以得到该时刻的事件信号中该像素的目标像素值，依次遍历该至少一个时刻的事件信号，以得到每个时刻的事件信号中该像素的目标像素值。然后，将该像素分别在该至少一个时刻的事件信号中对应的目标像素值进行累加，从而得到该像素在格式转化后的这一帧事件信号中的像素值。这样，对于该至少一个时刻的事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，从而得到格式转化后的一帧事件信号。
示例地，按照下述公式(5)确定任一像素在格式转化后的这一帧事件信号中的像素值。
其中，在上述公式(5)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值，t起代表目标子时间段的起始时间，t止代表目标子时间段的终止时间，tj代表该至少一个时刻的事件信号中时刻tj的事件信号的时间戳，代表时刻tj的事件信号在时间维度上的权重，pjδ(u-xj)δ(v-yj)代表在时刻tj的事件信号中像素(u，v)对应的事件极性，δ(u-xj)和δ(v-yj)代表示性函数，以δ(u-xj)为例进行说明，在(u-xj)为0时δ(u-xj)等于1，在(u-xj)不为0时δ(u-xj)等于0。
通过上述方式1进行格式转化后得到的一帧事件信号为体素网格格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式1所提供的方法，将各个像素所对应的事件极性的累加值，以及第一事件信号在时间维度上的权重之间的乘积作为各个像素转化后的像素值，以得到格式转化后的一帧体素网格格式的事件信号。
方式2，对目标子时间段包括的至少一个时刻的事件信号进行空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。将该至少一个时刻的事件信 号中同一位置的像素的事件极性累加，以得到格式转化后的一帧事件信号。
对于每个时刻的事件信号中的任一像素，将该像素分别在该至少一个时刻的事件信号中对应的事件极性进行累加，以得到该像素在格式转化后的这一帧事件信号中的像素值。这样，对于该至少一个时刻的事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，从而得到格式转化后的一帧事件信号。
示例地，按照下述公式(6)确定任一像素在格式转化后的这一帧事件信号中的像素值。
其中，在上述公式(6)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值。
通过上述方式2进行格式转化后得到的一帧事件信号为事件帧格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式2所提供的方法，将各个像素所对应的事件极性的累加值作为各个像素转化后的像素值，以得到格式转化后的一帧事件帧格式的事件信号。当然，在实际应用中，还能够将各个像素所对应的事件极性的总数量作为各个像素转化后的像素值，以得到格式转化后的一帧事件帧格式的事件信号，本申请实施例对此不做限定。
方式3，对目标子时间段包括的至少一个时刻的事件信号进行时间维度的格式转化，以得到格式转化后的一帧事件信号。
对于该至少一个时刻的事件信号中同一位置的像素，基于该像素在每个时刻的事件信号中的空间位置坐标，从转化前的空间位置坐标与转化后的空间位置坐标之间的对应关系中，确定该像素转化后的空间位置坐标。如果该至少一个时刻的事件信号中的每个时刻的事件信号均记录有该像素的亮度变化情况，则将该至少一个时刻的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值。如果该至少一个时刻的事件信号中存在部分时刻的事件信号记录有该像素的亮度变化情况，则将该部分时刻的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值。将各个像素转化后的空间位置坐标上的像素值构成格式转化后的一帧事件信号。
需要说明的是，对于该至少一个时刻的事件信号中的任一像素，按照上述方法直接从该至少一个时刻的事件信号中选择记录有该像素的亮度变化情况的事件信号，并将选择出的事件信号的时间戳中的最大时间戳确定为该像素转化后的空间位置坐标上的像素值仅为一种示例。在另一些实施例中，还能够按照其他的方式，来确定各个像素在格式转化后的这一帧事件信号中的像素值。示例地，将该至少一个时刻的事件信号按照时间戳从小到大的顺序进行排序，以得到该至少一个时刻的事件信号的排序结果。基于该至少一个时刻的事件信号中各个像素的空间位置坐标，确定格式转化后的这一帧事件信号中各个像素的空间位置坐标。基于该排序结果，以及该至少一个时刻中每个时刻的事件信号的时间戳，确定各个像素转化后的像素值。
对于该排序结果中的第一个时刻的事件信号中的任一像素，基于该像素在第一个时刻的事件信号中的空间位置坐标，从转化前的空间位置坐标与转化后的空间位置坐标之间的对应关系中，确定该像素转化后的空间位置坐标。然后，将第一个时刻的事件信号的时间戳确定为该像素转化后的空间位置坐标上的像素值。这样，对于第一个时刻的事件信号中的各个像素，均能够按照上述步骤确定出各个像素转化后的空间位置坐标上的像素值。然后，按照相同的方法依次遍历该至少一个时刻的事件信号，从而得到该像素在格式转化后的这一帧事件信号中的像素值。这样，对于该至少一个时刻的事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，以得到格式转化后的一帧事件信号。
通过上述方式3进行格式转化后得到的一帧事件信号为时间面格式的事件信号。也即是，在第一事件信号为流格式的事件信号的情况下，按照上述方式3所提供的方法，将各个像素最后发生极性事件所对应的时间戳作为各个像素对应的目标像素值，以得到格式转化后的一帧时间面格式的事件信号。
方式4，基于图像信号，对目标子时间段包括的至少一个时刻的事件信号进行时间维度和空间维度的格式转化，以得到格式转化后的一帧事件信号。
确定该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性。基于该至少一个时刻中每个 时刻的事件信号包括的各个像素的事件极性，以及正变化阈值，确定每个时刻的事件信号包括的各个像素的正极性值。基于该至少一个时刻中每个时刻的事件信号包括的各个像素的事件极性，以及负变化阈值，确定每个时刻的事件信号包括的各个像素的负极性值。基于该至少一个时刻中每个时刻的事件信号包括的各个像素的正极性值、该至少一个时刻中每个时刻的事件信号包括的各个像素的负极性值，以及图像信号，确定格式转化后的一帧事件信号。
对于每个时刻的事件信号中的任一像素，将该像素在该时刻的事件信号中的事件极性的绝对值与正变化阈值相乘，以得到该像素在该时刻的事件信号中的正极性值。将该像素在该时刻的事件信号中的事件极性的绝对值与负变化阈值相乘，以得到该像素在该时刻的事件信号中的负极性值。然后，将该像素分别在该至少一个时刻的事件信号中对应的正极性值，以及该像素分别在该至少一个时刻的事件信号中的负极性值进行累加，并将累加得到的计算结果与图像信号中该像素的像素值相乘，以得到该像素在格式转化后的这一帧事件信号中的像素值。这样，对于该至少一个时刻的事件信号中的各个像素，均能够按照上述步骤确定出各个像素在格式转化后的这一帧事件信号中的像素值，以得到格式转化后的一帧事件信号。
示例地，按照下述公式(7)确定任一像素在格式转化后的这一帧事件信号中的像素值。
其中，在上述公式(7)中，K(u，v，Ni)代表在格式转化后得到的帧序号为Ni的这一帧事件信号中像素(u，v)的像素值，C+δ(pj-1)δ(u-xj)δ(v-yj)代表在时刻tj的事件信号中像素(u，v)对应的正极性值，C-δ(pj+1)δ(u-xj)δ(v-yj)代表在时刻tj的事件信号中像素(u，v)对应的负极性值。
按照上述方式4所提供的方法，结合获取到的图像信号，对第一事件信号进行格式转化，使得转化后的第二事件信号能够更准确地指示像素在曝光时间内的不同时刻的亮度信息。
步骤703：将第二事件信号与图像信号进行融合，以得到融合信号。
对该N帧事件信号中每帧事件信号执行以下操作：确定一帧事件信号中的掩码区域，掩码区域指示对应的一帧事件信号中存在运动信息的像素所处的区域，将位于掩码区域内的各个像素的像素值与图像信号中相应像素的像素值进行融合，将位于掩码区域之外的各个像素的像素值设置为图像信号中相应像素的像素值，以得到一帧融合信号。
由于将该N帧事件信号中每帧事件信号与图像信号进行融合的过程相同，因此，从该N帧事件信号中选择一帧事件信号，并以该帧事件信号为例，对该帧事件信号与图像信号进行融合以得到一帧融合信号的过程进行介绍。
该帧事件信号中各个像素的像素值可能为0，也可能不为0。如果某一像素的像素值为0，则表明该像素在该帧事件信号所指示的时刻不存在运动信息。如果某一像素的像素值不为0，则表明该像素在该帧事件信号所指示的时刻存在运动信息。这样，将该帧事件信号中存在运动信息的像素所处的区域确定为该帧事件信号对应的掩码区域。
示例地，基于该帧事件信号中各个像素的像素值，生成该帧事件信号所对应的掩码信号。即对于该帧事件信号中的任一像素，在该像素的像素值为0的情况下，将该像素对应的掩码数组的值置为第一数值。在该像素的像素值不为0的情况下，将该像素对应的掩码数组的值置为第二数值。将该帧事件信号中各个像素对应的掩码数组的值构成该帧事件信号所对应的掩码信号。掩码信号中值为第二数值的掩码数组所围成的区域为该帧事件信号对应的掩码区域。
其中，第一数值和第二数值是事先设置的，例如，第一数值为0，第二数值为1。而且，第一数值和第二数值还能够按照不同的需求来调整。
基于该帧事件信号中任一像素的空间位置坐标，从图像信号中选择与该像素的空间位置坐标相同的像素，以得到该像素在图像信号中对应的像素。在该像素位于该帧事件信号对应的掩码区域内的情况下，将该像素在该帧事件信号中的像素值与该像素在图像信号中的像素值进行融合，以得到该像素在这一帧融合信号中的像素值。在该像素位于该帧事件信号对应的掩码区域之外的情况下，将图像信号中该像素的像素值确定为该像素在这一帧融合信号中的像素值。这样，对于该帧事件信号中的各个像素，均能够按照上述步骤确定出各个像素在这一帧融合信号中的像素值，以得到一帧融合信号。
也即是，通过该帧事件信号对应的掩码区域，将位于掩码区域之外的各个像素屏蔽，无需对位于掩码 区域之外的各个像素进行事件信号与图像信号的融合。
其中，将该像素在该帧事件信号中的像素值与该像素在图像信号中的像素值进行融合时，能够直接将该像素在该帧事件信号中的像素值与该像素在图像信号中的像素值相加，还能够将该像素在该帧事件信号中的像素值替换该像素在图像信号中的像素值。当然，在实际应用中，还能够通过其他的方式将该像素在该帧事件信号中的像素值与该像素在图像信号中的像素值进行融合，本申请实施例对此不做限定。
需要说明的是，按照上述步骤701-703得到融合信号之后，还能够将融合信号作为下游任务的输入。其中，下游任务包括视频插帧、图像去模糊处理、图像超分处理、目标对象监控和深度估计等场景。
示例地，以目标场景为自动驾驶场景为例，按照上述步骤得到融合信号之后，将融合信号输入至神经网络模型，以得到自动驾驶场景的场景感知信息。即，将融合信号作为自动驾驶场景这一下游任务的输入，以此来实现在自动驾驶场景中感知路况、车辆、行人及环境变化等信息。
可选地，将第二事件信号与图像信号进行融合之前，还能够分别对第二事件信号和图像信号进行预处理，以此来提高第二事件信号和图像信号的图像质量，从而进一步提高融合信号的图像质量。示例地，对第二事件信号进行滤波处理，以此来消除第二事件信号中的噪声和坏点。对图像信号进行插值处理、降噪处理、去马赛克处理和白平衡处理等。
其中，对第二事件信号的滤波处理包括中值滤波、高斯滤波等。当然，在实际应用中，还能够按照其他的方式对第二事件信号和图像信号进行预处理，本申请实施例对此不做限定。
在本申请实施例中，通过对第一事件信号进行时间维度和/或空间维度的格式转化，得到第二事件信号。由于第二事件信号为帧格式的事件信号，即第二事件信号的格式与图像信号的格式类似。所以，第二事件信号存在分辨率，且第二事件信号的分辨率与图像信号的分辨率相同。这样，能够将第二事件信号与图像信号更好的进行融合。而且，由于图像信号指示多个像素在曝光时间内的亮度信息，事件信号指示该多个像素在曝光时间内的运动信息。因此，将事件信号与图像信号进行融合，得到的融合信号既包括该多个像素的亮度信息，又包括该多个像素的运动信息。这样，通过稠密的、兼具亮度信息和运动信息的融合信号能够提升图像的质量。
图9是本申请实施例提供的另一种信号处理方法的流程图，本申请实施例提供的信号处理方法的交互执行主体为信号处理设备和云服务器。请参考图9，该方法包括如下步骤。
步骤901：信号处理设备获取目标场景的图像信号和第一事件信号，并将目标场景的图像信号和第一事件信号发送给云服务器。图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。
其中，信号处理设备获取目标场景的图像信号和第一事件信号的过程参考上述步骤701中的相关描述，此处不再赘述。
步骤902：云服务器接收信号处理设备发送的目标场景的图像信号和第一事件信号，对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。
基于上文描述，第一事件信号包括帧格式的事件信号和流格式的事件信号。在不同的情况下，云服务器对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号的方式不同，接下来将分为以下两种情况分别进行说明。
第一种情况，第一事件信号为帧格式的事件信号，且第一事件信号包括M帧事件信号，第二事件信号包括N帧事件信号，M和N均为大于等于1的整数，且M大于等于N。此时，将该M帧事件信号按照帧序号划分为N组事件信号，该N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号。对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
第二种情况，第一事件信号为流格式的事件信号，且第一事件信号包括H个时刻的事件信号，该H个时刻位于曝光时间内，第二事件信号包括N帧事件信号，H和N均为大于等于1的整数。此时，将曝光时间划分为N个子时间段，该N个子时间段中的每个子时间段包括该H个时刻中至少一个时刻的事件信号。对该N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
在上述两种情况中，云服务器对第一事件信号进行格式转化的过程与上述步骤702中信号处理设备对 第一事件信号进行格式转化的过程相似，所以，可以参考上述步骤702的相关内容，此处不再赘述。
步骤903：云服务器将第二事件信号与图像信号进行融合，以得到融合信号。
云服务器对该N帧事件信号中每帧事件信号执行以下操作：确定一帧事件信号中的掩码区域，掩码区域指示对应的一帧事件信号中存在运动信息的像素所处的区域，将位于掩码区域内的各个像素的像素值与图像信号中相应像素的像素值进行融合，将位于掩码区域之外的各个像素的像素值设置为图像信号中相应像素的像素值，以得到一帧融合信号。
步骤904：云服务器将融合信号发送给信号处理设备。
云服务器按照上述步骤得到融合信号之后，将融合信号发送给信号处理设备。信号处理设备接收到云服务器发送的融合信号之后，将融合信号作为下游任务的输入。其中，下游任务包括视频插帧、图像去模糊处理、图像超分处理、目标对象监控和深度估计中的任一种。
示例地，以目标场景为自动驾驶场景为例，按照上述步骤得到融合信号之后，将融合信号输入至神经网络模型，以得到自动驾驶场景的场景感知信息。即，将融合信号作为自动驾驶场景这一下游任务的输入，以此来实现在自动驾驶场景中感知路况、车辆、行人及环境变化等信息。
在本申请实施例中，通过对第一事件信号进行时间维度和/或空间维度的格式转化，得到第二事件信号。由于第二事件信号为帧格式的事件信号，即第二事件信号的格式与图像信号的格式类似。所以，第二事件信号存在分辨率，且第二事件信号的分辨率与图像信号的分辨率相同。这样，能够将第二事件信号与图像信号更好的进行融合。而且，由于图像信号指示多个像素在曝光时间内的亮度信息，事件信号指示该多个像素在曝光时间内的运动信息。因此，将事件信号与图像信号进行融合，得到的融合信号既包括该多个像素的亮度信息，又包括该多个像素的运动信息。这样，通过稠密的、兼具亮度信息和运动信息的融合信号能够提升图像的质量。
图10是本申请实施例提供的一种信号处理装置的结构示意图，该信号处理装置可以由软件、硬件或者两者的结合实现成为信号处理设备的部分或者全部。参见图10，该装置包括：获取模块1001、转化模块1002和融合模块1003。
获取模块1001，用于获取目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
转化模块1002，用于对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
融合模块1003，用于将第二事件信号与图像信号进行融合，以得到融合信号。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
可选地，第一事件信号为帧格式的事件信号，且第一事件信号包括M帧事件信号，第二事件信号包括N帧事件信号，M和N均为大于等于1的整数，且M大于等于N；转化模块1002具体用于：
将该M帧事件信号按照帧序号划分为N组事件信号，该N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号；
对该N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
可选地，第一事件信号为流格式的事件信号，且第一事件信号包括H个时刻的事件信号，该H个时刻位于曝光时间内，第二事件信号包括N帧事件信号，H和N均为大于等于1的整数；转化模块1002具体用于：
将曝光时间划分为N个子时间段，该N个子时间段中的每个子时间段包括该H个时刻中至少一个时刻的事件信号；
对该N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到该N帧事件信号。
可选地，第二事件信号的格式为事件帧格式、时间面格式和体素网格格式中的任一种。
可选地，第二事件信号包括N帧事件信号，N为大于等于1的整数；融合模块1003具体用于：
对该N帧事件信号中每帧事件信号执行以下操作：
确定一帧事件信号中的掩码区域，掩码区域指示对应的一帧事件信号中存在运动信息的像素所处的区域；
将位于掩码区域内的各个像素的像素值与图像信号中相应像素的像素值进行融合，将位于掩码区域之外的各个像素的像素值设置为图像信号中相应像素的像素值，以得到一帧融合信号。
可选地，目标场景为自动驾驶场景；该装置还包括：
输入模块，用于将融合信号输入至神经网络模型，以得到自动驾驶场景的场景感知信息。
在本申请实施例中，通过对第一事件信号进行时间维度和/或空间维度的格式转化，得到第二事件信号。由于第二事件信号为帧格式的事件信号，即第二事件信号的格式与图像信号的格式类似。所以，第二事件信号存在分辨率，且第二事件信号的分辨率与图像信号的分辨率相同。这样，能够将第二事件信号与图像信号更好的进行融合。而且，由于图像信号指示多个像素在曝光时间内的亮度信息，事件信号指示该多个像素在曝光时间内的运动信息。因此，将事件信号与图像信号进行融合，得到的融合信号既包括该多个像素的亮度信息，又包括该多个像素的运动信息。这样，通过稠密的、兼具亮度信息和运动信息的融合信号能够提升图像的质量。
需要说明的是：上述实施例提供的信号处理装置在对信号进行处理时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的信号处理装置与信号处理方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
请参考图11，图11是根据本申请实施例示出的一种计算机设备的结构示意图，该计算机设备为上述的信号处理设备或者云服务器。该计算机设备包括至少一个处理器1101、通信总线1102、存储器1103以及至少一个通信接口1104。
处理器1101可以是一个通用中央处理器(central processing unit，CPU)、网络处理器(network processor，NP)、微处理器、或者可以是一个或多个用于实现本申请方案的集成电路，例如，专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)、现场可编程逻辑门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合。
通信总线1102用于在上述组件之间传送信息。通信总线1102可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
存储器1103可以是只读存储器(read-only memory，ROM)，也可以是随机存取存储器(random access memory，RAM)，也可以是电可擦可编程只读存储器(electrically erasable programmable read-only Memory，EEPROM)、光盘(包括只读光盘(compact disc read-only memory，CD-ROM)、压缩光盘、激光盘、数字通用光盘、蓝光光盘等)、磁盘存储介质或者其它磁存储设备，或者是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其它介质，但不限于此。存储器1103可以是独立存在，并通过通信总线1102与处理器1101相连接。存储器1103也可以和处理器1101集成在一起。
通信接口1104使用任何收发器一类的装置，用于与其它设备或通信网络通信。通信接口1104包括有线通信接口，还可以包括无线通信接口。其中，有线通信接口例如可以为以太网接口。以太网接口可以是光接口、电接口或其组合。无线通信接口可以为无线局域网(wireless local area networks，WLAN)接口、蜂窝网络通信接口或其组合等。
在具体实现中，作为一种实施例，处理器1101可以包括一个或多个CPU，如图11中所示的CPU0和CPU1。
在具体实现中，作为一种实施例，计算机设备可以包括多个处理器，如图11中所示的处理器1101和处理器1105。这些处理器中的每一个可以是一个单核处理器，也可以是一个多核处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(如计算机程序指令)的处理核。
在具体实现中，作为一种实施例，计算机设备还可以包括输出设备1106和输入设备1107。输出设备1106和处理器1101通信，可以以多种方式来显示信息。例如，输出设备1106可以是液晶显示器(liquid crystal  display，LCD)、发光二级管(light emitting diode，LED)显示设备、阴极射线管(cathode ray tube，CRT)显示设备或投影仪(projector)等。输入设备1107和处理器1101通信，可以以多种方式接收用户的输入。例如，输入设备1107可以是鼠标、键盘、触摸屏设备或传感设备等。
在一些实施例中，存储器1103用于存储执行本申请方案的程序代码1110，处理器1101可以执行存储器1103中存储的程序代码1110。该程序代码1110中可以包括一个或多个软件模块，该计算机设备可以通过处理器1101以及存储器1103中的程序代码1110，来实现上文图7或图9实施例提供的信号处理方法。
也即是，对于云服务器来说，该云服务器包括通信接口和一个或多个处理器；
通信接口，用于接收信号处理设备发送的目标场景的图像信号和第一事件信号，图像信号指示目标场景对应的多个像素在曝光时间内的亮度信息，第一事件信号指示该多个像素在曝光时间内的运动信息，第一事件信号为帧格式的事件信号或流格式的事件信号。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
一个或多个处理器，用于对第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，第二事件信号为帧格式的事件信号，第二事件信号的分辨率与图像信号的分辨率相同。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
一个或多个处理器，用于将第二事件信号与图像信号进行融合，以得到融合信号。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
一个或多个处理器，用于通过通信接口将融合信号发送给信号处理设备。详细实现过程参考上述各个实施例中对应的内容，此处不再赘述。
请参考图12，图12是本申请实施例提供的一种终端设备的结构示意图。该终端设备可以为上述的信号处理设备。该终端设备包括传感器单元1210、计算单元1220、存储单元1240和交互单元1230。
传感器单元1210，通常包括视觉传感器(如相机)、深度传感器、IMU、激光传感器等；
计算单元1220，通常包括CPU、GPU、缓存、寄存器等，主要用于运行操作系统；
存储单元1240，主要包括内存和外部存储，主要用于本地和临时数据的读写等；
交互单元1230，主要包括显示屏、触摸板、扬声器、麦克风等，主要用于和用户进行交互，获取用于输入，并实施呈现算法效果等。
为便于理解，下面将对本申请实施例提供的一种终端设备100的结构进行示例说明。参见图13，图13是本申请实施例提供的一种终端设备的结构示意图。
如图13所示，终端设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L等。
可以理解的是，本申请实施例示意的结构并不构成对终端设备100的具体限定。在本申请另一些实施例中，终端设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。处理器110可以执行计算机程序，以实现本申请实施例中任一种方法。
其中，控制器可以是终端设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再 次使用该指令或数据，可从所述存储器中直接调用，避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I1C)接口，集成电路内置音频(inter-integrated circuit sound，I1S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对终端设备100的结构限定。在本申请另一些实施例中，终端设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头193，和无线通信模块160等供电。
终端设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
在一些可行的实施方式中，终端设备100可以使用无线通信功能和其他设备通信。例如，终端设备100可以和第二电子设备通信，终端设备100与第二电子设备建立投屏连接，终端设备100输出投屏数据至第二电子设备等。其中，终端设备100输出的投屏数据可以为音视频数据。
天线1和天线2用于发射和接收电磁波信号。终端设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在终端设备100上的包括1G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线2转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在终端设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线1接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，终端设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得终端设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple  access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
终端设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，终端设备100可以包括1个或N个显示屏194，N为大于1的正整数。
在一些可行的实施方式中，显示屏194可用于显示终端设备100的系统输出的各个界面。
终端设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，终端设备100可以包括1个或N个摄像头193，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。
视频编解码器用于对数字视频压缩或解压缩。终端设备100可以支持一种或多种视频编解码器。这样，终端设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG1，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现终端设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展终端设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行终端设备100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如本申请实施例中的室内定位方法等)等。存储数据区可存储终端设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
终端设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。在一些可行的实施方式中，音频模块170可用于播放视频对应的声音。例如，显示屏194显示视频播放画面时，音频模块170输出视频播放的声音。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频 信号。
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。陀螺仪传感器180B可以用于确定终端设备100的运动姿态。气压传感器180C用于测量气压。
加速度传感器180E可检测终端设备100在各个方向上(包括三轴或六轴)加速度的大小。当终端设备100静止时可检测出重力的大小及方向。还可以用于识别终端设备姿态，应用于横竖屏切换，计步器等应用。
距离传感器180F，用于测量距离。
环境光传感器180L用于感知环境光亮度。
指纹传感器180H用于采集指纹。
温度传感器180J用于检测温度。
触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于终端设备100的表面，与显示屏194所处的位置不同。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。终端设备100可以接收按键输入，产生与终端设备100的用户设置以及功能控制有关的键信号输入。
马达191可以产生振动提示。
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
SIM卡接口195用于连接SIM卡。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意结合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络或其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如：同轴电缆、光纤、数据用户线(digital subscriber line，DSL))或无线(例如：红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质，或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如：软盘、硬盘、磁带)、光介质(例如：数字通用光盘(digital versatile disc，DVD))或半导体介质(例如：固态硬盘(solid state disk，SSD))等。值得注意的是，本申请实施例提到的计算机可读存储介质可以为非易失性存储介质，换句话说，可以是非瞬时性存储介质。
也即是，本申请实施例还提供了一种计算机可读存储介质，该计算机可读存储介质内存储有指令，当该指令在计算机上运行时，使得计算机执行上述信号处理方法的步骤。
本申请实施例还提供了一种包含指令的计算机程序产品，当该指令在计算机上运行时，使得计算机执行上述信号处理方法的步骤。或者说，提供了一种计算机程序，当计算机程序在计算机上运行时，使得计算机执行上述信号处理方法的步骤。
应当理解的是，本文提及的“多个”是指两个或两个以上。在本申请实施例的描述中，除非另有说明，“/”表示或的意思，例如，A/B可以表示A或B；本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，为了便于清楚描述本申请实施例的技术方案，在本申请实施例中，采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分。本领域技术人员可以理解“第一”、“第二”等字样并不对数量和执行次序进行限定，并且“第一”、“第二”等字样也并不限定一定不同。
需要说明的是，本申请实施例所涉及的信息(包括但不限于用户设备信息、用户个人信息等)、数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)以及信号，均为经用户授权或者经过各方充分授权的，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。例如，本申请实施例中涉及到的目标场景的图像信号和第一事件信号都是在充分授权的情况下获取的。
以上所述为本申请提供的实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (20)

  1. 一种信号处理方法，其特征在于，所述方法包括：
    获取目标场景的图像信号和第一事件信号，所述图像信号指示所述目标场景对应的多个像素在曝光时间内的亮度信息，所述第一事件信号指示所述多个像素在所述曝光时间内的运动信息，所述第一事件信号为帧格式的事件信号或流格式的事件信号；
    对所述第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，所述第二事件信号为帧格式的事件信号，所述第二事件信号的分辨率与所述图像信号的分辨率相同；
    将所述第二事件信号与所述图像信号进行融合，以得到融合信号。
  2. 如权利要求1所述的方法，其特征在于，所述第一事件信号为帧格式的事件信号，且所述第一事件信号包括M帧事件信号，所述第二事件信号包括N帧事件信号，所述M和所述N均为大于等于1的整数，且所述M大于等于所述N；所述对所述第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，包括：
    将所述M帧事件信号按照帧序号划分为N组事件信号，所述N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号；
    对所述N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  3. 如权利要求1所述的方法，其特征在于，所述第一事件信号为流格式的事件信号，且所述第一事件信号包括H个时刻的事件信号，所述H个时刻位于所述曝光时间内，所述第二事件信号包括N帧事件信号，所述H和所述N均为大于等于1的整数；所述对所述第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，包括：
    将所述曝光时间划分为N个子时间段，所述N个子时间段中的每个子时间段包括所述H个时刻中至少一个时刻的事件信号；
    对所述N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  4. 如权利要求1-3任一所述的方法，其特征在于，所述第二事件信号的格式为事件帧格式、时间面格式和体素网格格式中的任一种。
  5. 如权利要求1-4任一所述的方法，其特征在于，所述第二事件信号包括N帧事件信号，所述N为大于等于1的整数；所述将所述第二事件信号与所述图像信号进行融合，以得到融合信号，包括：
    对所述N帧事件信号中每帧事件信号执行以下操作：
    确定一帧事件信号中的掩码区域，所述掩码区域指示对应的所述一帧事件信号中存在运动信息的像素所处的区域；
    将位于所述掩码区域内的各个像素的像素值与所述图像信号中相应像素的像素值进行融合，将位于所述掩码区域之外的各个像素的像素值设置为所述图像信号中相应像素的像素值，以得到一帧融合信号。
  6. 如权利要求1-5任一所述的方法，其特征在于，所述目标场景为自动驾驶场景；所述方法还包括：
    将所述融合信号输入至神经网络模型，以得到所述自动驾驶场景的场景感知信息。
  7. 如权利要求1-6任一所述的方法，其特征在于，所述方法的执行主体为云服务器；所述获取目标场景的图像信号和第一事件信号包括：
    接收信号处理设备发送的所述目标场景的图像信号和第一事件信号；
    所述将所述第二事件信号与所述图像信号进行融合，以得到融合信号之后，所述方法还包括：
    将所述融合信号发送给所述信号处理设备。
  8. 一种信号处理装置，其特征在于，所述装置包括：
    获取模块，用于获取目标场景的图像信号和第一事件信号，所述图像信号指示所述目标场景对应的多个像素在曝光时间内的亮度信息，所述第一事件信号指示所述多个像素在所述曝光时间内的运动信息，所述第一事件信号为帧格式的事件信号或流格式的事件信号；
    转化模块，用于对所述第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，所述第二事件信号为帧格式的事件信号，所述第二事件信号的分辨率与所述图像信号的分辨率相同；
    融合模块，用于将所述第二事件信号与所述图像信号进行融合，以得到融合信号。
  9. 如权利要求8所述的装置，其特征在于，所述第一事件信号为帧格式的事件信号，且所述第一事件信号包括M帧事件信号，所述第二事件信号包括N帧事件信号，所述M和所述N均为大于等于1的整数，且所述M大于等于所述N；所述转化模块具体用于：
    将所述M帧事件信号按照帧序号划分为N组事件信号，所述N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号；
    对所述N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  10. 如权利要求8所述的装置，其特征在于，所述第一事件信号为流格式的事件信号，且所述第一事件信号包括H个时刻的事件信号，所述H个时刻位于所述曝光时间内，所述第二事件信号包括N帧事件信号，所述H和所述N均为大于等于1的整数；所述转化模块具体用于：
    将所述曝光时间划分为N个子时间段，所述N个子时间段中的每个子时间段包括所述H个时刻中至少一个时刻的事件信号；
    对所述N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  11. 如权利要求8-10任一所述的装置，其特征在于，所述第二事件信号的格式为事件帧格式、时间面格式和体素网格格式中的任一种。
  12. 如权利要求8-11任一所述的装置，其特征在于，所述第二事件信号包括N帧事件信号，所述N为大于等于1的整数；所述融合模块具体用于：
    对所述N帧事件信号中每帧事件信号执行以下操作：
    确定一帧事件信号中的掩码区域，所述掩码区域指示对应的所述一帧事件信号中存在运动信息的像素所处的区域；
    将位于所述掩码区域内的各个像素的像素值与所述图像信号中相应像素的像素值进行融合，将位于所述掩码区域之外的各个像素的像素值设置为所述图像信号中相应像素的像素值，以得到一帧融合信号。
  13. 如权利要求8-12任一所述的装置，其特征在于，所述目标场景为自动驾驶场景；所述装置还包括：
    输入模块，用于将所述融合信号输入至神经网络模型，以得到所述自动驾驶场景的场景感知信息。
  14. 一种云服务器，其特征在于，所述云服务器包括通信接口和一个或多个处理器；
    所述通信接口，用于接收信号处理设备发送的目标场景的图像信号和第一事件信号，所述图像信号指示所述目标场景对应的多个像素在曝光时间内的亮度信息，所述第一事件信号指示所述多个像素在所述曝光时间内的运动信息，所述第一事件信号为帧格式的事件信号或流格式的事件信号；
    所述一个或多个处理器，用于对所述第一事件信号进行时间维度和/或空间维度的格式转化，以得到第二事件信号，所述第二事件信号为帧格式的事件信号，所述第二事件信号的分辨率与所述图像信号的分辨率相同；
    所述一个或多个处理器，用于将所述第二事件信号与所述图像信号进行融合，以得到融合信号；
    所述一个或多个处理器用于通过所述通信接口将所述融合信号发送给所述信号处理设备。
  15. 如权利要求14所述的云服务器，其特征在于，所述第一事件信号为帧格式的事件信号，且所述第一事件信号包括M帧事件信号，所述第二事件信号包括N帧事件信号，所述M和所述N均为大于等于1的整数，且所述M大于等于所述N；所述一个或多个处理器具体用于：
    将所述M帧事件信号按照帧序号划分为N组事件信号，所述N组事件信号中的每组事件信号包括帧序号连续的至少一帧事件信号；
    对所述N组事件信号中的每组事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  16. 如权利要求14所述的云服务器，其特征在于，所述第一事件信号为流格式的事件信号，且所述第一事件信号包括H个时刻的事件信号，所述H个时刻位于所述曝光时间内，所述第二事件信号包括N帧事件信号，所述H和所述N均为大于等于1的整数；所述一个或多个处理器具体用于：
    将所述曝光时间划分为N个子时间段，所述N个子时间段中的每个子时间段包括所述H个时刻中至少一个时刻的事件信号；
    对所述N个子时间段中每个子时间段包括的事件信号进行时间维度和/或空间维度的格式转化，以得到所述N帧事件信号。
  17. 如权利要求14-16任一所述的云服务器，其特征在于，所述第二事件信号包括N帧事件信号，所述N为大于等于1的整数；所述一个或多个处理器具体用于：
    对所述N帧事件信号中每帧事件信号执行以下操作：
    确定一帧事件信号中的掩码区域，所述掩码区域指示对应的所述一帧事件信号中存在运动信息的像素所处的区域；
    将位于所述掩码区域内的各个像素的像素值与所述图像信号中相应像素的像素值进行融合，将位于所述掩码区域之外的各个像素的像素值设置为所述图像信号中相应像素的像素值，以得到一帧融合信号。
  18. 一种信号处理设备，其特征在于，所述信号处理设备包括存储器和处理器，所述存储器用于存储计算机程序，所述处理器被配置为用于执行所述存储器中存储的计算机程序，以实现权利要求1-6任一项所述的方法。
  19. 一种计算机可读存储介质，其特征在于，所述存储介质内存储有指令，当所述指令在计算机或处理器上运行时，使得所述计算机或所述处理器执行权利要求1-7任一所述的方法的步骤。
  20. 一种计算机程序，其特征在于，所述计算机程序包括指令，当所述指令在所述计算机上运行时，使得所述计算机执行权利要求1-7任一项所述的方法。
PCT/CN2023/103954 2022-10-14 2023-06-29 信号处理方法、装置、设备、存储介质及计算机程序 Ceased WO2024078032A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23876241.3A EP4589549A4 (en) 2022-10-14 2023-06-29 SIGNAL PROCESSING METHOD AND APPARATUS, DEVICE, RECORDING MEDIUM AND COMPUTER PROGRAM
US19/177,148 US20250240538A1 (en) 2022-10-14 2025-04-11 Signal Processing Method, Apparatus, and Device, Storage Medium, and Computer Program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211259723.4A CN117893856A (zh) 2022-10-14 2022-10-14 信号处理方法、装置、设备、存储介质及计算机程序
CN202211259723.4 2022-10-14

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/177,148 Continuation US20250240538A1 (en) 2022-10-14 2025-04-11 Signal Processing Method, Apparatus, and Device, Storage Medium, and Computer Program

Publications (1)

Publication Number Publication Date
WO2024078032A1 true WO2024078032A1 (zh) 2024-04-18

Family

ID=90638186

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/103954 Ceased WO2024078032A1 (zh) 2022-10-14 2023-06-29 信号处理方法、装置、设备、存储介质及计算机程序

Country Status (4)

Country Link
US (1) US20250240538A1 (zh)
EP (1) EP4589549A4 (zh)
CN (1) CN117893856A (zh)
WO (1) WO2024078032A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250159371A1 (en) * 2023-11-09 2025-05-15 Omnivision Technologies, Inc. Hybrid image sensors with on-chip image deblur
CN118494511B (zh) * 2024-07-17 2024-10-11 比亚迪股份有限公司 车载数据处理方法、装置、计算机设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113724142A (zh) * 2020-05-26 2021-11-30 杭州海康威视数字技术股份有限公司 图像复原系统及方法
CN113727079A (zh) * 2020-05-25 2021-11-30 华为技术有限公司 一种图像信号处理方法及装置、电子设备
US20220222496A1 (en) * 2021-01-13 2022-07-14 Fotonation Limited Image processing system
CN114972412A (zh) * 2022-05-12 2022-08-30 深圳锐视智芯科技有限公司 一种姿态估计方法、装置、系统及可读存储介质
CN115564695A (zh) * 2022-09-29 2023-01-03 联想(北京)有限公司 一种处理方法及电子设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111247801B (zh) * 2017-09-28 2022-06-14 苹果公司 用于事件相机数据处理的系统和方法
KR20230127287A (ko) * 2020-12-31 2023-08-31 후아웨이 테크놀러지 컴퍼니 리미티드 포즈 추정 방법 및 관련 장치
CN112801027B (zh) * 2021-02-09 2024-07-12 北京工业大学 基于事件相机的车辆目标检测方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113727079A (zh) * 2020-05-25 2021-11-30 华为技术有限公司 一种图像信号处理方法及装置、电子设备
CN113724142A (zh) * 2020-05-26 2021-11-30 杭州海康威视数字技术股份有限公司 图像复原系统及方法
US20220222496A1 (en) * 2021-01-13 2022-07-14 Fotonation Limited Image processing system
CN114972412A (zh) * 2022-05-12 2022-08-30 深圳锐视智芯科技有限公司 一种姿态估计方法、装置、系统及可读存储介质
CN115564695A (zh) * 2022-09-29 2023-01-03 联想(北京)有限公司 一种处理方法及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4589549A4

Also Published As

Publication number Publication date
CN117893856A (zh) 2024-04-16
EP4589549A4 (en) 2025-12-31
EP4589549A1 (en) 2025-07-23
US20250240538A1 (en) 2025-07-24

Similar Documents

Publication Publication Date Title
CN114946169B (zh) 一种图像获取方法以及装置
EP3686845B1 (en) Image processing method and device and apparatus
EP4036854B1 (en) Image processing method and apparatus, and electronic device
WO2020192461A1 (zh) 一种延时摄影的录制方法及电子设备
US20240119566A1 (en) Image processing method and apparatus, and electronic device
CN116095476B (zh) 摄像头的切换方法、装置、电子设备及存储介质
US20250240538A1 (en) Signal Processing Method, Apparatus, and Device, Storage Medium, and Computer Program
CN113052056B (zh) 一种视频处理的方法以及装置
CN113096022A (zh) 图像虚化处理方法、装置、存储介质与电子设备
WO2022033344A1 (zh) 视频防抖方法、终端设备和计算机可读存储介质
CN113744139A (zh) 图像处理方法、装置、电子设备及存储介质
CN113468929B (zh) 运动状态识别方法、装置、电子设备和存储介质
WO2024082976A1 (zh) 文本图像的ocr识别方法、电子设备及介质
CN117880645A (zh) 一种图像处理的方法、装置、电子设备及存储介质
CN112188094B (zh) 图像处理方法及装置、计算机可读介质及终端设备
CN116095509B (zh) 生成视频帧的方法、装置、电子设备及存储介质
CN108427938A (zh) 图像处理方法、装置、存储介质和电子设备
CN117135455A (zh) 一种图像处理方法及电子设备
CN116723264A (zh) 确定目标位置信息的方法、设备及存储介质
CN116055894B (zh) 基于神经网络的图像去频闪方法和装置
CN116051450B (zh) 眩光信息获取方法、装置、芯片、电子设备及介质
WO2024093854A1 (zh) 一种图像处理方法及电子设备
CN117593236A (zh) 图像的显示方法、装置和终端设备
CN119277212B (zh) 图像处理方法和装置
CN119255090B (zh) 图像处理方法、电子设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23876241

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023876241

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023876241

Country of ref document: EP

Effective date: 20250416

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 2023876241

Country of ref document: EP