WO2024055764A1 - 图像处理方法及装置 - Google Patents

图像处理方法及装置 Download PDF

Info

Publication number
WO2024055764A1
WO2024055764A1 PCT/CN2023/110156 CN2023110156W WO2024055764A1 WO 2024055764 A1 WO2024055764 A1 WO 2024055764A1 CN 2023110156 W CN2023110156 W CN 2023110156W WO 2024055764 A1 WO2024055764 A1 WO 2024055764A1
Authority
WO
WIPO (PCT)
Prior art keywords
isp
image
images
parameters
original
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/110156
Other languages
English (en)
French (fr)
Inventor
邱珏沁
闻明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP23864498.3A priority Critical patent/EP4576018A4/en
Publication of WO2024055764A1 publication Critical patent/WO2024055764A1/zh
Priority to US19/078,875 priority patent/US20250247629A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • Embodiments of the present application relate to the field of image processing, and in particular, to an image processing method and device.
  • the terminal device can perform image processing on the images collected by the camera, and apply the image processing results to scenarios such as autonomous driving and security monitoring.
  • the firmware (FW) part of the image signal processor (image signal processor, ISP) is based on fixed algorithm parameters and performs image processing on the images collected by the camera to obtain the processed image.
  • image signal processor image signal processor
  • image processing methods based on fixed algorithm parameters are relatively simple and cannot adapt to all scenarios.
  • Embodiments of the present application provide an image processing method and device.
  • the device can generate ISP parameters that satisfy the corresponding scenarios based on the original images obtained in different scenarios to achieve dynamic adjustment of the ISP parameters, thereby performing adaptive image enhancement for different scenarios.
  • embodiments of the present application provide an image processing method.
  • the method includes: the device acquires a first original image collected by a camera in a first scene.
  • the device obtains the first ISP parameter based on the first original image.
  • the device performs image processing on the first original image collected by the camera based on the first ISP parameter to obtain the first image.
  • the device outputs the first image to the target application, so that the target application performs image task processing on the first image.
  • the device obtains the second original image collected by the camera in the second scene.
  • the device obtains the second ISP parameter based on the second original image.
  • the device performs image processing on the second original image collected by the camera based on the second ISP parameter to obtain the second image.
  • the device outputs the second image to the target application, so that the target application performs image task processing on the second image.
  • embodiments of the present application can obtain ISP parameters that meet the needs of different scenarios, thereby realizing dynamic adjustment of ISP parameters, and performing image processing on original images collected in different scenarios based on different ISP parameters, thereby improving the accuracy of image processing.
  • the ISP parameters in the embodiments of the present application are automatically obtained, that is, the corresponding ISP parameters can be predicted based on the original images obtained in different scenarios.
  • the device can periodically acquire ISP parameters to reduce the computing pressure of the device.
  • the target application may be a sensing application, a display application, etc., which is not limited in this application.
  • image processing characters for CV perception applications may include but are not limited to 2D/3D target detection, lane line detection, scene semantic segmentation, etc., which are not limited in this application.
  • the image content of the first original image and the second original image are different, and/or the image attributes of the first original image and the second original image are different.
  • the contents and/or attributes of images taken in different scenes are different. This application can predict the corresponding ISP parameters based on different original images to obtain better image processing results, thereby adapting to Different scene requirements.
  • the first ISP parameter and the second ISP parameter are used to adjust at least one of the following image attributes of the image: brightness, color, noise, sharpness, and contrast.
  • different ISP parameters can be used to adjust different image attributes to obtain better image effects.
  • the image attributes of the first image and the second image meet the image task requirements of the target application.
  • the image processing in this application is all to meet the needs of the target application, so that the target application can better recognize the processed image, and improve the accuracy of the image recognition (or other image processing processes) of the target application.
  • obtaining the first ISP parameter based on the first original image includes: inputting the first original image to the ISP parameter prediction model, and obtaining the first ISP parameter predicted by the ISP parameter prediction model based on the first original image.
  • ISP parameter; based on the second original image, obtaining the second ISP parameter includes: inputting the second original image to the ISP parameter prediction model, and obtaining the second ISP parameter predicted by the ISP parameter prediction model based on the second original image.
  • the ISP parameter prediction model runs on the neural network processor NPU.
  • the prediction model in the embodiment of the present application runs on the general computing power of the NPU and requires only a small amount of calculation.
  • this application can be deployed on edge devices without occupying high-priority tasks on the AI processor.
  • the method before obtaining the first original image collected by the camera in the first scene, the method further includes: obtaining the ISP parameter prediction model from the cloud.
  • the device only needs the ISP system of the chip on sale to support the dynamic rewriting of parameters.
  • This solution can download updates over the air (that is, obtain and install them from the cloud) for products already in circulation on the market. Support is provided to meet compatibility needs.
  • image processing is performed on the first original image based on the first ISP parameter to obtain the first image, including: replacing the ISP parameter currently saved in the ISP memory with the first ISP parameter, so that the ISP changes from Obtain the first ISP parameter from the ISP memory, perform image processing on the first original image based on the first ISP parameter, and output the first image; perform image processing on the second original image based on the second ISP parameter, and obtain the second image, It includes: replacing the first ISP parameter currently saved in the ISP memory with the second ISP parameter, so that the ISP obtains the second ISP parameter from the ISP memory, and performs image processing on the second original image based on the second ISP parameter, and outputs the second ISP parameter.
  • Two images In this way, in the embodiment of the present application, the traditional ISP processing algorithm is still used on the image, which can ensure the controllability and interpretability of the entire processing process.
  • embodiments of the present application provide a model training method.
  • the method includes: inputting N original images to an image signal processor ISP parameter prediction model, obtaining N ISP parameters output by the ISP parameter prediction model, the weight of the ISP parameter prediction model is the first weight, and the N ISP parameters are the ISP parameter predictions.
  • the model is obtained based on the first weight value, and N is an integer greater than 1.
  • Input N ISP parameters and N original images to the proxy model and obtain N images output by the proxy model.
  • the proxy network can realize the function of ISP, and realize the end-to-end loop training process between the ISP parameter prediction model and the target application through joint training of the differentiable proxy network and the FW parameter prediction network.
  • the model training method can be executed on a computing node.
  • the preset condition includes: the difference between the preset ground truth labels of the N original images and the ground truth labels included in the image task processing result of the target application is less than a preset threshold.
  • the method before inputting N original images to the ISP parameter prediction model, the method also includes: inputting M original images and M ISP parameters to the ISP, and obtaining M images output by the image processor; wherein, M original images and M ISP parameters correspond one to one, M is an integer greater than 1; input M original images and M ISP parameters to the proxy model, and obtain M images output by the proxy model; the current weight of the proxy model is the third weight; if the proxy model The similarity between the output M images and the M images output by the image processor is less than the threshold, adjust the weight of the proxy model to the fourth weight, and start again from the step of inputting M original images and M ISP parameters to the ISP.
  • the processing results of the proxy network can be made close to or the same as the processing results of the ISP, so that the proxy network can realize the function of the ISP during the joint training process.
  • inventions of the present application provide an image processing device.
  • the device includes: ISP parameter prediction module and ISP module.
  • the ISP parameter prediction module is used to obtain the first original image collected by the camera in the first scene; the ISP parameter prediction module is also used to obtain the first image signal processor ISP parameters based on the first original image; the image processing ISP module , used to perform image processing on the first original image based on the first ISP parameter to obtain the first image; the ISP module is also used to output the first image to the target application, so that the target application performs image task processing on the first image;
  • the ISP parameter prediction module is also used to obtain the second original image collected by the camera in the second scene; the ISP parameter prediction module is also used to obtain the second ISP parameter based on the second original image; the ISP module is also used to obtain the second ISP parameter based on the second original image.
  • the second ISP parameter is used to perform image processing on the second original image to obtain the second image; the ISP module is also used to output the second image to the
  • the image content of the first original image and the second original image are different, and/or the image attributes of the first original image and the second original image are different.
  • the first ISP parameter and the second ISP parameter are used to adjust at least one of the following image attributes of the image: brightness, color, noise, sharpness, and contrast.
  • the image attributes of the first image and the second image meet the image task requirements of the target application.
  • the ISP parameter prediction module is used to replace the ISP parameters currently saved in the ISP memory with the first ISP parameter; the ISP module is used to obtain the first ISP parameter from the ISP memory, and calculate the first ISP parameter based on the first ISP parameter.
  • An ISP parameter which performs image processing on the first original image and outputs the first image; an ISP parameter prediction module, which is used to replace the first ISP parameter currently saved in the ISP memory with the second ISP parameter; the ISP module, which is used to obtain the first ISP parameter from the ISP Obtain the second ISP parameter in the memory, perform image processing on the second original image based on the second ISP parameter, and output the second image.
  • embodiments of the present application provide a model training system, including an image signal processor ISP parameter prediction model, a proxy model and a target application.
  • the system inputs N original images to the ISP parameter prediction model and obtains N ISP parameters output by the ISP parameter prediction model.
  • the weight of the ISP parameter prediction model is the first weight
  • the N ISP parameters are the values of the ISP parameter prediction model based on the first weight.
  • N is an integer greater than 1.
  • the system inputs N ISP parameters and N original images to the proxy model and obtains N images output by the proxy model.
  • the system inputs N images to the target application.
  • the system adjusts the weight of the ISP parameter prediction model to the second weight, and again starts from the step of inputting N original images to the ISP parameter prediction model until the N output by the proxy model The images meet the image task requirements of the target application.
  • the system also includes an ISP; before inputting N original images to the ISP parameter prediction model, input M original images and M ISP parameters to the ISP to obtain M images output by the image processor; where , M original images and M ISP parameters correspond one to one, M is an integer greater than 1; input M original images and M ISP parameters to the proxy model, and obtain M images output by the proxy model; the current weight of the proxy model is the third weight; if the similarity between the M images output by the proxy model and the M images output by the image processor is less than the threshold, adjust the weight of the proxy model to the fourth weight, and the fourth weight is different from the third weight.
  • the execution starts from inputting M original images and M ISP parameters to the image processor, obtaining M images output by the image processor, until the similarity between the M images output by the proxy model and the M images output by the image processor degree is greater than the threshold.
  • embodiments of the present application provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein one or more computer programs are stored on the memory, and when the computer program is Or a method that, when executed by multiple processors, causes the electronic device to execute the instructions of the first aspect or the method in any possible implementation of the first aspect.
  • embodiments of the present application provide a camera module, including: one or more processors; a memory; and one or more computer programs, wherein one or more computer programs are stored in the memory.
  • the computer program is A method that, when executed by one or more processors, causes the camera module to execute the instructions of the method in the first aspect or any possible implementation of the first aspect.
  • embodiments of the present application provide a computer-readable medium for storing a computer program, where the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.
  • embodiments of the present application provide a computer program, which includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.
  • embodiments of the present application provide a chip, which includes a processing circuit and transceiver pins.
  • the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive the signal, so as to Control the sending pin to send signals.
  • Figure 1 is a schematic diagram of the hardware structure of electronic equipment
  • Figure 2 is a schematic diagram of the hardware structure of the electronic device
  • Figure 3 is a schematic flowchart of an exemplary image processing method
  • Figure 4 is a schematic flow chart of an exemplary image processing method
  • Figure 5 is a schematic flowchart of an exemplary image processing method
  • Figure 6 is a schematic structural diagram of an exemplary proxy network
  • Figure 7 is a schematic diagram of an exemplary agent network training process
  • Figure 8 is a schematic diagram of the training process of the FW parameter prediction network
  • Figure 9 is a schematic diagram of the system structure of FW parameter prediction network training exemplarily shown.
  • Figure 10 is a schematic diagram of an exemplary image processing flow
  • Figure 11 is a schematic structural diagram of an exemplary device.
  • a and/or B can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations.
  • first and second in the description and claims of the embodiments of this application are used to distinguish different objects, rather than to describe a specific order of objects.
  • first target object, the second target object, etc. are used to distinguish different target objects, rather than to describe a specific order of the target objects.
  • multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.
  • the image processing method in the embodiment of this application is applied to electronic devices, where the electronic device can be a mobile phone, a tablet, a vehicle-mounted device, a security device, a computer, a wearable device, a smart home device, etc., which is not limited by this application.
  • FIG. 1 shows a schematic structural diagram of an electronic device 100 .
  • the electronic device 100 shown in FIG. 1 is only an example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, and two or more components may be combined. , or can have different component configurations.
  • the various components shown in Figure 1 may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the electronic device is a mobile phone as an example. In other embodiments, the electronic device may be any device described above, which is not limited in this application.
  • the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2.
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, And subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) wait.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • NPU neural-network processing unit
  • different processing units can be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the electronic device 100 .
  • the controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • the processor 110 may also be provided with a memory for storing instructions and data.
  • the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
  • processor 110 may include one or more interfaces.
  • Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • UART universal asynchronous receiver and transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (derail clock line, SCL).
  • processor 110 may include multiple sets of I2C buses.
  • the processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces.
  • the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device 100 .
  • the I2S interface can be used for audio communication.
  • processor 110 may include multiple sets of I2S buses.
  • the processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 .
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.
  • the PCM interface can also be used for audio communications to sample, quantize and encode analog signals.
  • audio module 170 The wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is generally used to connect the processor 110 and the wireless communication module 160 .
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function.
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
  • the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 .
  • MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc.
  • the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100 .
  • the processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100 .
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193, display screen 194, wireless communication module 160, audio module 170, sensor module 180, etc.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
  • the USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other electronic devices, such as AR devices, etc.
  • the interface connection relationships between the modules illustrated in the embodiments of the present application are only schematic illustrations and do not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 140 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .
  • the charging management module 140 may receive wireless charging input through the wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142, it can also provide power to the electronic device through the power management module 141.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, wireless communication module 160, etc.
  • the power management module 141 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters.
  • the power management module 141 may also be provided in the processor 110 .
  • the power management module 141 and the charging management module 140 may also be provided in the same device.
  • the wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.
  • the mobile communication module 150 can provide solutions for wireless communication including 2G/3G/4G/5G applied on the electronic device 100 .
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
  • the mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation.
  • at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 .
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194.
  • the modem processor may be a stand-alone device.
  • the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), and global navigation satellites.
  • WLAN wireless local area networks
  • System global navigation satellite system, GNSS
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication, NFC
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi) -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like.
  • the GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 194 is used to display images, videos, etc.
  • Display 194 includes a display panel.
  • the display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED organic light-emitting diode
  • FLED flexible light-emitting diode
  • Miniled MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc.
  • the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
  • the electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.
  • the ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the electrical signal is processed by the ISP and converted into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, white balance, contrast and other attributes.
  • the ISP may be provided in the camera 193.
  • the optimized attributes of ISPs of different manufacturers and models can be set according to actual needs. For example, the ISP of electronic device A can optimize the noise and brightness of the image.
  • the ISP of electronic device B can optimize the brightness and contrast of the image, which is not limited by this application.
  • Camera 193 is used to capture still images or video.
  • the object passes through the lens to produce an optical image that is projected onto the photosensitive element.
  • the photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
  • the image captured by the camera 193 may be called a Raw image (or simply a Raw image), and may be understood as is the original image file.
  • ISP can perform image processing on Raw images and output RGB images or YUV images.
  • RGB and YUV are both color spaces, used to represent colors.
  • "Y” in YUV represents brightness, which is the grayscale value; while "U” and “V” represent image color and saturation, which are used to specify the color of pixels.
  • RGB color mode is a color standard in the industry. It obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. of.
  • Video codecs are used to compress or decompress digital video.
  • Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • MPEG moving picture experts group
  • MPEG2 MPEG2, MPEG3, MPEG4, etc.
  • NPU is a neural network (NN) computing processor.
  • NN neural network
  • Intelligent cognitive applications of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, etc.
  • the NPU may be used to support the FW parameter prediction network in the FW parameter setting module, that is, the NPU executes the steps performed by the FW parameter prediction network in the FW parameter setting module.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.
  • Internal memory 121 may be used to store computer executable program code, which includes instructions.
  • the processor 110 executes instructions stored in the internal memory 121 to execute various functional applications and data processing of the electronic device 100 .
  • the internal memory 121 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.).
  • the storage data area may store data created during use of the electronic device 100 (such as audio data, phone book, etc.).
  • the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.
  • the electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .
  • Speaker 170A also called “speaker” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 can listen to music through the speaker 170A, or listen to hands-free calls.
  • Receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear.
  • Microphone 170C also called “microphone” or “microphone” is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with the human mouth and input the sound signal to the microphone 170C.
  • the electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.
  • the headphone interface 170D is used to connect wired headphones.
  • the headphone interface 170D may be a USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA Cellular Telecommunications Industry Association of the USA
  • the pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals.
  • pressure sensor 180A may be disposed on display screen 194 .
  • pressure sensors 180A there are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.
  • a capacitive pressure sensor may include at least two parallel plates of conductive material.
  • the electronic device 100 determines the intensity of the pressure based on the change in capacitance.
  • the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A.
  • the electronic device 100 may also calculate the touched position based on the detection signal of the pressure sensor 180A.
  • touch operations with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.
  • the gyro sensor 180B may be used to determine the motion posture of the electronic device 100 .
  • the angular velocity of electronic device 100 about three axes ie, x, y, and z axes
  • the gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shake of the electronic device 100 through reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
  • Air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • Magnetic sensor 180D includes a Hall sensor.
  • the electronic device 100 may utilize the magnetic sensor 180D to detect opening and closing of the flip holster.
  • the electronic device 100 may detect the opening and closing of the flip according to the magnetic sensor 180D. Then, based on the detected opening and closing status of the leather case or the opening and closing status of the flip cover, features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices and be used in horizontal and vertical screen switching, pedometer and other applications.
  • Distance sensor 180F for measuring distance.
  • Electronic device 100 can measure distance via infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may utilize the distance sensor 180F to measure distance to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • the electronic device 100 emits infrared light outwardly through the light emitting diode.
  • Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 .
  • the electronic device 100 can use the proximity light sensor 180G to detect when the user holds the electronic device 100 close to the ear for talking, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in holster mode, and pocket mode automatically unlocks and locks the screen.
  • the ambient light sensor 180L is used to sense ambient light brightness.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touching.
  • Fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc.
  • Temperature sensor 180J is used to detect temperature.
  • the electronic device 100 utilizes the temperature detected by the temperature sensor 180J to execute the temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the low temperature from causing the electronic device 100 to shut down abnormally. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 180K also called “touch panel”.
  • the touch sensor 180K can be disposed on the display screen 194.
  • the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near the touch sensor 180K.
  • the touch sensor can pass the detected touch operation to the application processor to determine the touch event type.
  • Visual output related to the touch operation may be provided through display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a location different from that of the display screen 194 .
  • Bone conduction sensor 180M can acquire vibration signals.
  • the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human body's vocal part.
  • the bone conduction sensor 180M can also contact the human body's pulse and receive blood pressure beating signals.
  • the bone conduction sensor 180M can also be provided in an earphone and combined into a bone conduction earphone.
  • the audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibrating bone obtained by the bone conduction sensor 180M to implement the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.
  • the buttons 190 include a power button, a volume button, etc.
  • Key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .
  • the motor 191 can generate vibration prompts.
  • the motor 191 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback.
  • touch operations acting on different applications can correspond to different vibration feedback effects.
  • the motor 191 can also respond to different vibration feedback effects for touch operations in different areas of the display screen 194 .
  • Different application scenarios such as time reminders, receiving information, alarm clocks, games, etc.
  • the touch vibration feedback effect can also be customized.
  • the indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc.
  • the SIM card interface 195 is used to connect a SIM card.
  • the SIM card can be connected to or separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 .
  • the electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different.
  • the SIM card interface 195 is also compatible with different types of SIM cards.
  • the SIM card interface 195 is also compatible with external memory cards.
  • the electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.
  • the electronic device 100 uses an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .
  • the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of this application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .
  • the system may be a Linux system, which is not limited in this application.
  • FIG. 2 is a software structure block diagram of the electronic device 100 according to the embodiment of the present application.
  • the layered architecture of the electronic device 100 divides the software into several layers, and each layer has clear roles and division of labor.
  • the layers communicate through software interfaces.
  • the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer.
  • the application layer can include a series of application packages.
  • the application package can include camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.
  • the application framework layer provides an application programming interface (API) and programming framework for applications in the application layer.
  • API application programming interface
  • the application framework layer includes some predefined functions.
  • the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, FW parameter setting module, etc.
  • a window manager is used to manage window programs.
  • the window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.
  • Content providers are used to store and retrieve data and make this data accessible to applications.
  • Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, etc.
  • a view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
  • the phone manager is used to provide communication functions of the electronic device 100 .
  • call status management including connected, hung up, etc.
  • the resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.
  • the notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, etc.
  • the notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.
  • the FW parameter setting module may include a FW parameter prediction network.
  • the FW parameter prediction network is used to predict the corresponding FW parameters based on scene requirements.
  • the FW parameter setting module can be located at the application layer.
  • the FW parameters obtained by the FW parameter prediction network can also be called ISP parameters, FW hyperparameters, or ISP hyperparameters, which can be set according to actual needs and are not limited in this application. It can be understood that during subsequent use, ISP can obtain image processing parameters based on FW hyperparameters and preset image processing parameter functions. ISP can process Raw images based on image processing parameters.
  • the FW parameter setting module can also run at the application layer, which can be understood as a piece of code running on the NPU, which is not limited in this application.
  • the user can update the FW parameter setting module by updating the system of the electronic device.
  • the user can operate the electronic device, and the electronic device can obtain an updated version from the cloud and update the system based on the updated version to install the FW parameter setting module.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.
  • the application layer and the application framework layer run in a virtual machine.
  • the virtual machine executes the Java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • System libraries can include multiple functional modules. For example: surface manager (surface manager), media libraries (Media Libraries), 3D graphics processing libraries (for example: OpenGL ES), 2D graphics engines (for example: SGL), etc.
  • the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc.
  • the media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.
  • 2D Graphics Engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer includes at least display driver, camera driver, audio driver, sensor driver, etc.
  • the components included in the system framework layer, system library and runtime layer shown in Figure 2 do not constitute specific limitations on the electronic device 100.
  • the electronic device 100 may include more or fewer components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently.
  • a camera in an electronic device collects an image, and the image is called a Raw image.
  • the ISP can obtain the FW parameters from storage (which can also be called shared memory, for example, it can be a Double Data Rate memory, which is not limited in this application).
  • the FW parameters are set before the electronic equipment leaves the factory, and a set of FW parameters is stored in the DDR.
  • the FW parameter can be understood as an array.
  • the FW parameter can be understood as an array used to adjust at least one image attribute during ISP processing.
  • the ISP can adjust the brightness attribute in the Raw image based on the first parameter in the FW parameter, so that the brightness of the generated RGB (or YUV, which will not be described in detail below) image is brighter or darker.
  • Applications in electronic devices can obtain RGB images and process the RGB images accordingly.
  • the perception application can perform AI recognition on RGB images to identify people or other matters in RGB images, which is not limited by this application.
  • each module may also include interactions with other modules.
  • the ISP may obtain Raw images through a camera driver. This application does not limit this, and the description will not be repeated below.
  • the FW parameters are set before the electronic device leaves the factory, it can be understood as a fixed FW parameter. That is, the ISP performs the same processing on all acquired Raw images.
  • the first parameter in the FW parameter indicates that the brightness (i.e., brightness attribute) of the Raw image is adjusted to the first threshold.
  • the ISP will adjust the brightness parameter of the Raw image to the first threshold based on the FW parameter. threshold.
  • this adjustment method is too simple and cannot be applied to all scenarios. For example, in the car In the mobile device, the brightness of the image acquired by the camera in the tunnel is relatively dark.
  • the RGB image it generates may be The brightness is still dark, so that downstream applications (such as perception applications) may not be able to accurately identify objects in the image.
  • FIG. 4 is a schematic diagram illustrating the principle of an image processing method according to an embodiment of the present application.
  • the FW parameters running on the AI (Artificial Intelligence) processor for example, it can be an NPU or a TPU, etc., which is not limited in this application
  • the setting module (specifically, the FW parameter prediction network) can predict the corresponding optimal FW parameters based on the Raw image (to be different from the FW parameters in Figure 3, this application calls them dynamic FW parameters), and the ISP can predict based on the AI processor.
  • the dynamic FW parameters are obtained, image processing is performed on the Raw image, and the corresponding RGB image is generated.
  • Perception applications can also be other applications
  • the image processing method in the embodiment of the present application can dynamically adjust the FW parameters according to the scene requirements, so that the ISP can adaptively process the images obtained by the camera in different scenes, so as to obtain a processing effect that is more in line with the scene requirements and the image is clearer. As a result, the recognition accuracy of downstream applications is improved.
  • vehicle-mounted cameras i.e., cameras inherited from vehicle-mounted equipment
  • vehicle-mounted equipment such as MCUs in vehicle-mounted equipment
  • real-time multi-type perception of the road environment such as 2D/3D target detection, Lane line detection, scene semantic segmentation, etc.
  • relevant descriptions in prior art embodiments which are not limited in this application.
  • computer vision perception tasks are usually carried out by deep models based on CNN (Convolutional Neutral Network) or transformer architecture.
  • CNN Convolutional Neutral Network
  • transformer architecture For the same original image (i.e.
  • the FW parameter prediction network in the FW parameter setting module can be used for each Raw The image predicts a set of the most suitable FW parameters, so that the RGB or YUV image obtained after ISP processing of the Raw image can obtain the best prediction effect on one or more perception tasks, thereby improving assisted driving or autonomous driving scenarios. Perceptual accuracy.
  • the ATR Adaptive Tone Reproduction
  • the noise module predicts a set of parameters with weak noise reduction strength to reduce the loss of image details due to smearing.
  • the DRC Dynamic Range Compression
  • assisted driving or autonomous driving application scenarios are used as examples for explanation.
  • the specific implementation methods for other application scenarios are the same. This application will not illustrate each example one by one, and the description will not be repeated below.
  • FIG. 5 is a schematic flowchart of an exemplary image processing method.
  • the image processing method in the embodiment of the present application can be divided into an offline training stage and an online deployment stage.
  • the offline stage includes but is not limited to two stages:
  • the proxy network can be understood as a network that replaces the ISP, and its processing results are the same or similar to those of the ISP. It can be understood that the process of proxy network training is a process in which the image processing results of the proxy network gradually approach the processing results of the ISP.
  • the FW parameter prediction network training phase can be understood as training the FW parameter prediction network based on the trained proxy network to obtain a FW parameter prediction network with more accurate prediction results.
  • the prediction result is the FW parameter.
  • the online deployment phase optionally deploys the parameter prediction network obtained in the offline training phase to the target device.
  • the target device can be terminal equipment, vehicle-mounted equipment, security monitoring equipment, etc., and is not limited in this application.
  • the offline training phase can be executed by the computing node.
  • Computing nodes can be any devices such as computers, servers, computers, etc.
  • the steps in the offline phase may be executed by the GPU processor in the computing node, which is not limited by this application.
  • the offline training phase can be executed at any point in time to obtain a trained parameter prediction network.
  • the electronic device can be installed with a FW parameter setting module that supports the parameter prediction network before leaving the factory.
  • offline training The training phase can also be executed periodically (for example, once a month) to update the parameter prediction network.
  • the computing node can upload the updated parameter prediction network to the cloud (the cloud includes one or more servers), and the cloud can provide the electronic device with Push the system update version, which includes the parameter prediction network.
  • the electronic device can update the FW parameter setting module based on the system update version to obtain the new parameter prediction network.
  • the operator needs to prepare data and models before starting the offline training phase.
  • the data prepared includes but is not limited to:
  • N can be any value.
  • the operator can control the image sensor (such as a vehicle-mounted camera) to collect images to obtain N Raw images, which are hereinafter referred to as training Raw images.
  • the scene covered by the collected images optionally fits the working environment of the electronic equipment (such as vehicle-mounted equipment or terminal equipment) in the deployment stage as much as possible.
  • the working environment may involve tunnels and highways in different weather environments.
  • the operator can control the image sensor to collect images of multiple tunnels and images of different highways in different weather environments. This application does not Make limitations.
  • the difference between the images can optionally be the difference of each parameter between the images, such as the noise parameter, brightness parameter, etc. mentioned above.
  • the images involved in the embodiments of the present application are arrays to the computer.
  • it can actually be understood as describing the elements of the image.
  • the corresponding array or can be further understood as the parameters corresponding to the array describing the image.
  • any processing of the image such as processing the image by ISP, can be understood as processing the pixel values corresponding to the image, and the description will not be repeated below.
  • the images collected by the cameras may be the same or different.
  • vehicle-mounted device A and vehicle-mounted device B are vehicle-mounted devices of the same manufacturer but different models, and their camera parameters are different, the collected Raw images will be different.
  • the different images may optionally be different in attributes (such as brightness, color temperature, etc.).
  • different attributes may optionally mean that the type, quantity, and/or intensity of the attributes (which may also be high or low, light or dark, etc.) are different.
  • the camera of vehicle-mounted device A and the camera of vehicle-mounted device B capture the same picture in the same environment.
  • the attributes of the Raw image collected by vehicle-mounted camera A include but are not limited to: attribute A (for example, color temperature), attribute B (for example, white balance), and attribute C (for example, brightness), while the Raw image collected by the camera of vehicle-mounted device B
  • the attributes of the image include but are not limited to: attribute A, attribute B, attribute D (for example, noise), and attribute E (for example, Gamma).
  • the attribute A of the image captured by the camera of the vehicle-mounted device A and the attribute A of the image captured by the camera of the vehicle-mounted device B are the same or different in size (can also be high or low), which is not limited in this application.
  • operators can perform training operations on different types and/or different models of equipment to obtain parameter prediction networks corresponding to different types and/or different models of equipment.
  • This application only takes the training process corresponding to a single device as an example.
  • the computing node can repeatedly execute the training phase described in the embodiments of this application based on different devices to obtain the corresponding parameter prediction network. This application will not give examples one by one.
  • b.N group of FW parameters For example, the operator can use a sampling algorithm such as Latin-Hypercube (the corresponding formula can be set according to actual needs, which is not limited in this application) to sample N sets of FW parameters.
  • the FW parameters are arrays.
  • N groups of FW parameters can also be understood as N arrays.
  • each set of training FW parameters is optionally a set of vectors with a length of n.
  • the value of n is related to the processing capability of the ISP.
  • the attributes processed by different models of ISP may be the same or different.
  • the ISP of model A is used to process q attributes such as brightness and color temperature
  • n is related to q.
  • one parameter in the FW parameters can be used to adjust an attribute, and multiple parameters in the FW parameters can also be used to adjust an attribute, which is not limited in this application.
  • the FW parameters during the preparation process are called training FW parameters.
  • the number of Raw images participating in training is equal to N (that is, the number of groups of FW parameters) is used as an example for explanation.
  • the number of Raw images may also be greater than N or less than N.
  • the corresponding FW parameters can be the same or different.
  • the greater the number of FW parameters, that is, the more samples participating in training the closer the network (such as the agent network) obtained by training is to the user's needs.
  • CV Computer Vision
  • the CV perception network is used to predict specific computer vision tasks, such as 2D/3D box prediction for vehicles, pedestrians, traffic signs and other objects in road scenes, semantic segmentation of instances in the scene, and specific object prediction. Tracking and more.
  • specific computer vision tasks such as 2D/3D box prediction for vehicles, pedestrians, traffic signs and other objects in road scenes, semantic segmentation of instances in the scene, and specific object prediction. Tracking and more.
  • the CV perception network involved in the embodiments of this application does not specify a specific perception task, as long as the task has a clear objective function and the perception network has end-to-end trainable characteristics. For most supervised CV tasks and most deep models built based on CNN, these two prerequisites are easily satisfied.
  • the default CV sensing network in the embodiment of this application receives RGB or YUV images as input, and passes through a series of computing modules or sub-networks (such as backbone( Backbone Network), RPN (Region Proposal Network, Candidate Region Generation Network), etc.) and then output the prediction results required for specific tasks, such as 2D/3D detection boxes, object category labels, prediction confidence, segmentation masks, object motion trajectories, etc. .
  • a series of computing modules or sub-networks such as backbone( Backbone Network), RPN (Region Proposal Network, Candidate Region Generation Network), etc.
  • d.True value label For example, the operator can set a true value label for the Raw image used for training, which is used to identify the real result of the Raw image passing through the CV perception network.
  • Ground-truth labels include but are not limited to: 2D/3D detection boxes, object category labels, prediction confidence, segmentation masks, object motion trajectories and other labels. Its specific settings are set according to the CV-aware network. For example, if the CV perception network is used for object category recognition on RGB images, the operator can set the true value label for the Raw image in advance, that is, set the object category label included in the Raw image through manual identification. For example, the Raw image includes images of kittens and puppies, and the object category tags set by the operator for the Raw image include kitten tags and puppy tags.
  • the purpose of the true value label is to compare it with the actual output result of the CV during the training process to determine the deviation between the result output by the CV perception network and the real result (i.e., the true value label). Specific details will be described in detail in the following examples
  • the agent network may be a convolutional neural network (CNN), a recurrent neural network (RNN), or some other deep model (such as a transformer, etc.).
  • CNN convolutional neural network
  • RNN recurrent neural network
  • the present invention does not limit the specific model type used. It only needs to ensure that it receives a Raw image and a set of FW parameters as input and has differentiable properties.
  • the proxy network is also set with an initial weight.
  • the weight of the proxy network is called the proxy network weight
  • the weight of the FW parameter prediction network is called FW parameter prediction.
  • Network weight For example, the weights involved in the embodiment of this application may be a set of arrays.
  • Figure 6 is a schematic structural diagram of an exemplary proxy network. Please refer to Figure 6. This structure is the structure of the Transformer model, which includes the Encoder (encoding) and Decoder (decoding) parts.
  • Encoder includes but is not limited to: Multi-Head Attention layer, Layer Normalization (Norm in Figure 6) and Feed forward neural network (Feed forward neural network) ( Figure 6) 6 Feed Forwad) and other modules.
  • Decoder includes but is not limited to: Masked multi-head attention layer, multi-head attention layer, layer normalization (i.e. (Norm) in Figure 6) and feed-forward neural network (i.e. Feed in Figure 6 forwad) and other modules. Among them, each module corresponds to a set of weight values, and the weight values of all modules are combined to form the weight value of the agent network.
  • the FW parameter prediction network is a deep network deployed on the AI processor. It reads the Raw image data to be processed in the video stream collected by the current camera from storage (such as DDR) on demand, and predicts the best FW parameters. (It can also be understood as network reasoning). The FW parameter prediction network writes the predicted parameters (a set of vectors with a length of N, where N is the number of dynamically configurable FW parameters) into the shared memory of the ISP Firmware, overwriting the original FW parameters.
  • the FW parameter prediction network is similar to the proxy network, and can be a convolutional neural network (CNN), a recursive neural network (RNN), or some other deep models (such as transformer, etc.).
  • CNN convolutional neural network
  • RNN recursive neural network
  • the present invention does not limit the specific model type used, but only needs to ensure that it has differentiable properties.
  • the FW parameter prediction network is also set with initial weights.
  • the initial weight can be set arbitrarily by the operator, and is not limited in this application.
  • the weight of the FW parameter prediction network can be called the FW parameter prediction network weight to distinguish it from other weights.
  • the proxy network weight please refer to the relevant description of the proxy network weight and will not be repeated here.
  • the proxy network is used to use a differentiable model to proxy the non-differentiable image processing algorithms in ISP, thereby allowing first-order optimization algorithms such as gradient descent algorithms to apply to the FW parameter prediction network, proxy network and CV sensing network.
  • the complete data path is jointly optimized end-to-end.
  • Figure 7 is a schematic diagram of an exemplary agent network training process. Please refer to Figure 7. Specific details include but are not limited to:
  • ISP performs image processing on the training Raw image based on the training FW parameters to obtain the RGB_ISP image.
  • ISP is a SoC subsystem composed of two parts: Firmware and Hardware (some ISPs also include some software parts running on AI processing).
  • the Firmware part is responsible for some scheduling work required to execute the image processing algorithm, such as reading FW parameters from the designated memory area, controlling the switch status and execution sequence of each module, modifying the status bits of each register, etc., while executing some low-intensity logical operations.
  • the Hardware part is responsible for performing high-intensity operations in image processing algorithms, and its calculation logic is controlled by the status bits in the register.
  • the operator prepares N training FW parameters and N training Raw images.
  • the operator can preset the correspondence between N training FW parameters and N training Raw images.
  • Raw image 1 corresponds to FW parameter 1
  • Raw image 2 corresponds to FW parameter 2
  • Raw image N corresponds to FW parameter N.
  • the corresponding relationship can be set arbitrarily, just ensure that each Raw image corresponds to a FW parameter, and the corresponding FW parameters between different Raw images should be as different as possible to increase the difference between samples.
  • the computing node may include N ISPs, each ISP is used to perform image processing on a Raw image based on FW parameters to obtain N RGB images (which may also be YUV images, this application does not limit the Repeat the explanation again).
  • the ISP's processing of Raw images based on FW parameters may specifically include but is not limited to: the ISP itself carries an image processing parameter algorithm or an image processing parameter function.
  • the ISP may be based on a preset image processing parameter algorithm and FW. parameters (i.e. FW hyperparameters) to obtain the corresponding image processing parameters.
  • Raw images are arrays, and the arrays include multiple grayscale values.
  • the ISP can adjust at least one gray value in the data based on the image processing parameters to change the properties of the Raw image. For example, based on the image processing parameters, ISP adjusts some grayscale values below the threshold (which can be set according to actual needs) in the Raw image to the desired value, so that the brightness of the generated RGB image is higher than that before adjustment.
  • the threshold which can be set according to actual needs
  • ISP adjusts some grayscale values below the threshold (which can be set according to actual needs) in the Raw image to the desired value, so that the brightness of the generated RGB image is higher than that before adjustment.
  • the RGB image obtained by ISP is called the RGB_ISP image in the embodiment of the present application.
  • the image processing process of ISP can be understood as correcting (or processing) the attributes of the corresponding Raw image based on the FW parameters to obtain an RGB image.
  • the attributes of the RGB image are those processed based on the FW parameters.
  • Raw image 1 includes color temperature A
  • ISP performs image processing on Raw image 1 based on FW parameters (the FW parameters include color temperature thresholds, which may also be called color temperature correction parameters or color temperature correction thresholds, and are not limited in this application) to obtain RGB Image 1.
  • the color temperature B in the RGB image 1 is the same as or different from the color temperature A in the Raw image 1.
  • ISP can also perform image processing on each of the N training Raw images based on each parameter of the N training FW parameters to obtain N ⁇ N RGB_ISP images, so as to Expand the size of the training set.
  • Specific training methods can be set according to actual needs and are not limited in this application.
  • the ISP involved in the embodiment of this application can be any model of ISP in any type of equipment. Assuming that the processes involved in Figure 7 are all executed by the ISP of the electronic device of model A, then the proxy network obtained by the process of Figure 7 and the FW parameter prediction network obtained in the following embodiments all correspond to model A. of electronic equipment. If you need to obtain the FW parameter prediction network corresponding to the electronic device of model B, you need to re-execute the training processes in Figure 7 and the following embodiments based on the electronic device of model B. For example, the ISP involved in S701 will Replace the ISP for electronic equipment with model B. Examples will not be repeated in the embodiments of this application, and descriptions will not be repeated below.
  • all devices such as ISP
  • modules involved in the training phase in the embodiment of the present application can be integrated into the computing node and run by the computing node.
  • the ISP in the computing node is the ISP of the electronic device of model A.
  • the computing node may include multiple independent devices or modules.
  • the ISP and the proxy network are different devices, and the proxy network can be on the GPU of the computing node.
  • the ISP can be executed by the ISP in the A-type electronic device and output to the GPU of the computing node.
  • the system structure of the computing node can be set according to actual needs, which is not limited in this application.
  • the proxy network outputs an RGB_DL image based on the training FW parameters and the training Raw image.
  • the agent network may be composed of a depth perception model (it may also be other neural networks, which is not limited in this application).
  • the computing node uses N Raw images and N training FW parameters as the input of the agent network.
  • the agent network is based on the training Practice FW parameters, N Raw images and initial proxy network weights for inference (or operation), and output RGB_DL images.
  • the operator has preset the corresponding relationship between N Raw images and N FW parameters.
  • the proxy network when the proxy network is processing, it is still processed according to the set corresponding relationship. It can be understood that when the proxy network is processing, the corresponding relationship between the Raw image and the FW parameters needs to be consistent with the corresponding relationship between the Raw image and the FW parameters when the ISP performs image processing, in order to be comparable.
  • the computing node can determine the number of Raw images and FW parameters (ie, the number of groups) input into the proxy network each time based on the processing capability of the proxy network.
  • the Raw image and FW parameters input each time maintain the corresponding relationship as described above.
  • the number of Raw images (and FW parameters) input can be the same or different each time.
  • the computing node can input 20 Raw images and 20 FW parameters to the proxy network each time.
  • the numerical values are only illustrative examples and can be set according to actual needs, and are not limited in this application.
  • the proxy network can operate on multiple Raw images and FW parameters input each time based on the current weight (which may be an initial weight or an updated weight). During the operation process, it is similar to ISP and processes the Raw image based on the FW parameters corresponding to each Raw image to obtain the processing result.
  • the current weight which may be an initial weight or an updated weight.
  • the proxy network processes N Raw images and N sets of FW parameters (which may be processed in batches) to obtain N RGB_DL images.
  • the proxy network is set with initial weights, and the initial weights are arbitrarily set by the operator.
  • the results (i.e., arrays) output by the agent network in the initial stage may be meaningless.
  • the process shown in Figure 7 in the embodiment of the present application can be understood as The proxy network is continuously trained to update the proxy network weight, so that the result output by the proxy network gradually approaches the result processed by the ISP in S701 (i.e., RGB_ISP image).
  • S703 Determine whether the similarity between the RGB_IGP image and the RGB_DL image is greater than the threshold.
  • the computing node (specifically, it may be the CPU of the computing node) obtains N RGB_ISP images output by the ISP and N RGB_DL images output by the proxy network.
  • the computing node can obtain the similarity between the two corresponding images based on the correspondence between the N RGB_ISP images and the N RGB_DL images to obtain N similarities.
  • the computer can obtain the overall similarity between the RGB_ISP image processed by the ISP and the RGB_DL image processed by the proxy network based on the N similarities.
  • the correspondence between N RGB_ISP images and N RGB_DL images refers to the RGB images obtained after processing based on the same Raw image and the same FW parameter.
  • the calculation node obtains the similarity between the two corresponding images based on RGB_ISP image 1 and RGB_DL image 1.
  • RGB_ISP image 1 and RGB_DL image 1 are obtained by the ISP and proxy network based on Raw image 1 and FW parameter 1 respectively.
  • each image is an array.
  • RGB_ISP image 1 and RGB_DL image 1 are recorded as I out_ISP
  • RGB_DL image 1 is I out_DL
  • Raw image 1 is recorded as I Raw
  • the FW parameter is p.
  • the computing node calculates the difference between I out_ISP and I out_DL (which may also be called error, similarity difference, etc., and is not limited in this application), which is recorded as L_1.
  • the specific calculation method may refer to existing technical embodiments, and is not limited in this application.
  • the computing node can obtain the similarity difference between N RGB_ISP images and N RGB_DL images, so as to obtain N similarity difference values, such as L_1...L_N.
  • the calculation node calculates the average of N similarity difference values, which is the similarity value between the RGB_ISP image processed by the ISP and the RGB_DL image processed by the proxy network, recorded as L.
  • calculation methods such as arithmetic mean can also be used to obtain the similarity value, which is not limited in this application.
  • the computing node detects whether L is greater than (or equal to) a preset threshold.
  • the preset threshold can be set according to actual needs and is not limited in this application.
  • the preset threshold is used to indicate the degree of similarity between images. The smaller the preset threshold, the greater the similarity between the two images. Correspondingly, if L is smaller, the similarity between the two images is higher.
  • the computing node detects that the L value is less than the threshold, it can also be understood that the similarity is greater than the threshold. Then execute S704.
  • the computing node detects that the L value is greater than or equal to the threshold, it can also be understood that the similarity is less than the threshold, that is, the difference between the image output by the proxy network and the image output by the ISP is large, then S705 is executed.
  • the computing node can save the current proxy network weight and lock the proxy network weight to prevent the proxy network weight from changing with subsequent processing.
  • the weight of the proxy network needs to be readjusted, so that the proxy network re-executes S702 based on the new weight.
  • the computing node uses optimization algorithms such as the gradient descent algorithm to calculate the corresponding gradient value based on the current weight of the agent network (denoted as w t ). And according to the gradient value, the current agent network weight (w t ) is updated by gradient descent to obtain the updated agent network weight, which is recorded as w t+1 .
  • optimization algorithms such as the gradient descent algorithm to calculate the corresponding gradient value based on the current weight of the agent network (denoted as w t ).
  • the current agent network weight (w t ) is updated by gradient descent to obtain the updated agent network weight, which is recorded as w t+1 .
  • the agent network can repeatedly execute S702.
  • the agent network processes the training FW parameters and the training Raw image based on the updated weight (w t+1 ).
  • the computing node can compare the output result of the proxy network with the output result of the ISP again (S701 only needs to be executed once), and based on the comparison result, execute S704 or S705.
  • the number of loops depends on S703.
  • the size of the L value in that is, the similarity between the image of the output result of the proxy network and the output result of the ISP.
  • the calculation node detects that the similarity is greater than the threshold, that is, the L value is less than the threshold, then the proxy is saved. Network weights, training is over.
  • the FW parameter prediction network is a deep network deployed on the AI processor. It reads the Raw image data to be processed in the video stream collected by the current camera from storage (such as DDR) on demand, and Predict the optimal FW parameters (which can also be understood as network reasoning).
  • the FW parameter prediction network writes the predicted parameters (a set of vectors with a length of N, where N is the number of dynamically configurable FW parameters) into the shared memory of the ISP Firmware, overwriting the original FW parameters.
  • the training of the FW parameter prediction network in the embodiment of the present application can be understood as a process of making the output results of the FW parameter prediction network gradually more accurate.
  • the FW parameter prediction network is the same as the proxy network. It is also set with initial weights. In the process of training the FW parameter prediction network, it is similar to the training of the proxy network. In the process, the network weight is predicted by the FW parameters. Continuous updates are performed so that the output results of the FW parameter prediction network gradually optimize the output FW parameters.
  • Figure 8 is a schematic diagram of an exemplary FW parameter prediction network training process. Please refer to Figure 8. Specific details include but are not limited to:
  • the FW parameter prediction network obtains the predicted FW parameters based on the Raw image.
  • the computing node can input N Raw images (that is, N Raw images in the preparation phase) to the FW parameter prediction network, and the FW parameter prediction network can be based on the current weight (which can be the initial weight or the updated weight). weight), process (or operate) the input N Raw images to obtain N predicted FW parameters.
  • N Raw images that is, N Raw images in the preparation phase
  • the FW parameter prediction network can be based on the current weight (which can be the initial weight or the updated weight). weight)
  • process (or operate) the input N Raw images to obtain N predicted FW parameters.
  • the input samples in the training phase of the FW parameter prediction network that is, the N Raw images are the same as the input samples in the training phase of the proxy network, are used as an example for explanation.
  • the input samples in the FW parameter pre-storage network training phase may be M images among N Raw images.
  • it can also be other M Raw images that are different from the agent network training stage, and this application does not limit it.
  • the computing node can input N Raw images to the FW parameter prediction network in batches based on the processing capability of the FW parameter prediction network.
  • the number of Raw images input can be the same or different each time.
  • the computing node can input 20 Raw images to the FW parameter prediction network at a time.
  • the numerical values are only illustrative examples and can be set according to actual needs, and are not limited in this application.
  • the N predicted FW parameters output by the FW parameter prediction network correspond one-to-one to the input Raw image.
  • the FW parameter prediction network can be based on the Raw image 1 and output the predicted FW parameter 1. Based on Raw image 2, output predicted FW parameter 2.
  • the FW parameter prediction network is set with initial weights, which are arbitrarily set by the operator.
  • the results (i.e., arrays) output by the FW parameter prediction network in the initial stage may be meaningless, but in the embodiment of this application, the results output by the FW parameter prediction network will still be used. is called the predicted FW parameter.
  • the process shown in Figure 8 in the embodiment of the present application can be understood as continuously training the FW parameter prediction network to update the weight of the FW parameter prediction network to continuously optimize the output results of the FW parameter prediction network.
  • the proxy network obtains the RGB image based on the predicted FW parameters and Raw image.
  • N Raw images and N predicted FW parameters are in one-to-one correspondence.
  • the corresponding relationship is the corresponding relationship described in S801.
  • the proxy network can process (or operate) N Raw images based on the current weight (that is, the weight obtained after training in Figure 7) according to the correspondence between N Raw images and N predicted FW parameters. , get N RGB images. For example, based on the current weight, the proxy network can combine the predicted FW parameter 1 and the predicted FW parameter 1 according to the corresponding relationship between the Raw image 1 and the predicted FW parameter 1 (that is, the predicted FW parameter 1 is obtained by the FW parameter prediction network based on the Raw image 1).
  • Raw image 1 is processed (or operated) as a pair of input parameters to obtain RGB image 1.
  • the computer can obtain the correspondence between the Raw image 1, the predicted FW parameter 1, and the RGB image 1.
  • the proxy network has been trained in Figure 7.
  • the result obtained after the proxy network processes the Raw image based on the current weight is the same as that processed by the specified model of ISP.
  • the results are the same or close (i.e. similar). That is to say, in S802, the proxy network can be regarded as an ISP to perform image processing on the Raw image to obtain the corresponding RGB image.
  • the proxy network has differentiable properties to achieve end-to-end joint training between the FW prediction network and the CV sensing network.
  • the computing node can input N Raw images and N FW prediction parameters to the proxy network in batches.
  • the specific implementation method can be referred to the above, and will not be described again here.
  • the CV perception network obtains the perception task label based on the RGB image.
  • the operator is pre-set with a CV sensing network.
  • the function of the CV perception network can be to output the corresponding perception task label based on the input Raw image.
  • the number and type of perception task labels depend on the task type of the CV perception network. For example, if the task type of the CV perception network is to identify object types, the CV perception network can output the corresponding perception task label based on the input Raw image, and the perception task label includes the object type label.
  • the computing node after the computing node obtains the N RGB images output by the proxy network, it can input the N RGB images (it can be in batches.
  • the specific processing method can be referred to the above, and will not be described again here).
  • the CV perception network can process (or recognize) N RGB images and obtain N perception task labels.
  • N RGB images are in one-to-one correspondence with N perception task labels.
  • each perception task label may include multiple sub-task labels.
  • the type and number of sub-task labels are associated with the task type of the CV perception network. For detailed description, please refer to the above and will not be repeated here.
  • the computing node can obtain the correspondence between N RGB images and N perception task labels.
  • the computing node also obtains the correspondence between N RGB images and N Raw images.
  • the computing node can obtain the correspondence between N perception task labels and N Raw images. For example, perception task label 1 corresponds to Raw image 1, perception task label 2 corresponds to Raw image 2, and so on.
  • the operator sets a true value label for each Raw image among the N Raw images.
  • the computing node can obtain the correspondence between N true value labels and N Raw images in advance.
  • the computing node can obtain the The correspondence between N perception task labels and N truth value labels (the correspondence is one-to-one).
  • perception task label 1 corresponds to ground truth label 1
  • perception task label 2 corresponds to ground truth label 2.
  • S804 Determine whether the difference between the true value label and the perception task label is less than a threshold.
  • the computing node can obtain the correspondence between N perception task labels and N truth value labels.
  • the computing node can obtain the difference between the two corresponding perception task labels and the truth value label (which can also be called a difference value, loss function, etc.) based on the corresponding relationship between the N perception task labels and the N truth value labels.
  • This application without limitation) to obtain N difference values.
  • the manner in which the calculation node obtains the difference between the true value label and the perception task label may refer to existing technical embodiments, and is not limited in this application.
  • the calculation node can obtain the average value of N differences (it can also be the arithmetic average, which is not limited in this application), which is recorded as Q.
  • the computing node detects whether Q is greater than (or equal to) a preset threshold.
  • the preset threshold can be set according to actual needs and is not limited in this application.
  • a preset threshold is used to indicate the degree of difference between two labels. Among them, the smaller the preset threshold is, the smaller the difference between the two labels is. Correspondingly, if the Q value is smaller, the difference between the two labels is smaller.
  • the computing node detects that the Q value is less than the threshold, it can also be understood that the difference is less than the threshold. Then execute S704.
  • the computing node detects that the Q value is greater than or equal to the threshold, it can also be understood that the difference is greater than the threshold, that is, the difference between the perception task label output by the CV perception network based on the RGB image output by the agent network and the true value label. is larger, it can also be understood that the enhancement effect of the RGB image obtained by the proxy network based on the predicted FW parameters is low, making the results recognized by the CV sensing network inaccurate. It can also be understood that the predicted FW parameters cannot correct the Raw image. Achieve the enhancement effect that can be recognized by the CV perception network.
  • the computing node executes S806.
  • the computing node can save the FW parameter prediction network, that is, save the current structure of the FW parameter prediction network (such as a neural network structure) and the current weight of the FW parameter prediction network, recorded as w f .
  • the weight of the FW parameter prediction network needs to be readjusted so that the FW parameter prediction network Re-execute S801 based on the updated weight.
  • the computing node uses an optimization algorithm such as the gradient descent algorithm (which can be set according to actual needs and is not limited in this application) to predict the current weight (w f ) of the network based on the FW parameters and calculate the corresponding gradient value. And according to the gradient value, the current FW parameter prediction network weight (w f ) is updated by gradient descent to obtain the updated FW parameter prediction network weight, which is recorded as w f+1 .
  • the gradient descent algorithm which can be set according to actual needs and is not limited in this application
  • the current FW parameter prediction network weight (w f ) is updated by gradient descent to obtain the updated FW parameter prediction network weight, which is recorded as w f+1 .
  • the FW parameter prediction network can repeatedly execute S801.
  • the FW parameter prediction network processes the Raw image based on the updated weight (w f+1 ) to obtain the corresponding predicted FW parameters.
  • the number of loops of the process in Figure 8 depends on the size of the Q value in S804, that is, the degree of difference between the perceived task label and the true value label. In any cycle, if the Q value is less than the threshold, the calculation node saves the FW parameter prediction network and the corresponding FW parameter prediction network weight, and the training ends.
  • Fig. 9 is a schematic diagram of the system structure of FW parameter prediction network training.
  • the computing node inputs the Raw image into the FW parameter prediction network.
  • the FW parameter prediction network outputs predicted FW parameters based on the input Raw image.
  • the predicted FW parameters and Raw images are used as inputs to the proxy network, and the proxy network outputs RGB images.
  • the RGB image is used as the input of the CV sensing network, and the computing node can obtain the Q value (i.e., the difference value or loss function) based on the output of the CV sensing network.
  • the Q value does not meet the conditions, that is, it is greater than the threshold, the system will return to the starting position and repeat the process in Figure 9 until the Q value is less than the threshold.
  • the image processing method also includes an online deployment stage.
  • the online deployment stage The FW parameter prediction network and its weights saved in Figure 8 are deployed to the target device.
  • the target device is the device corresponding to the ISP during the training process.
  • the ISP is an electronic device of model A
  • the operator can deploy the FW parameter prediction network and its weights saved by the computing node to the electronic device of model A.
  • model A electronic equipment can also obtain and install the FW parameter prediction network and its weights through the cloud, which is not limited by this application.
  • the target device loads the FW parameter prediction network and its weights into the memory, and works in the operating system as a resident process of the AI processor (such as NPU). It can also be understood as the FW parameter setting module in Figure 2 (which can also be called the FW parameter prediction module), or the FW parameter setting (prediction) application running at the application layer.
  • the AI processor such as NPU
  • the FW parameter setting module in Figure 2 which can also be called the FW parameter prediction module
  • the FW parameter setting (prediction) application running at the application layer.
  • FIG. 10 is a schematic diagram of an exemplary image processing flow.
  • a trigger signal can be sent to the AI processor to call the FW parameter setting module on the AI processor.
  • the FW parameter setting module includes the FW parameter prediction network.
  • the camera will be called at the same time, and the camera will start collecting images in response to the call of the camera application. And place the collected images in shared memory.
  • the FW parameter setting module reads Raw images from shared memory in response to calls from the camera application.
  • shared memory is only used as an example for explanation. In other embodiments, it can also be other storage, which is not limited by this application.
  • the FW parameter prediction network in the FW parameter setting module can predict the network weight based on the FW parameters, process the input Raw image, and predict the corresponding FW parameters.
  • the FW parameter setting module writes the FW parameters into the ISP memory corresponding to the ISP FW (i.e., the ISP software part).
  • the ISP memory may be part of the shared memory or may be located in other storage, which is not limited in this application.
  • the FW parameters may have been saved in the ISP memory, and the saved FW parameters may be the initial FW parameters, or may be the last written by the FW parameter setting module, which is not limited in this application.
  • the FW parameter setting module can delete the FW parameters currently saved in the ISP memory and write the newly predicted FW parameters.
  • ISP FW can read the FW parameters (that is, the newly written FW parameters) from the ISP memory.
  • ISP FW can indicate FW parameters to ISP HW.
  • ISP FW can modify the register status of FW HW to write FW parameters into the register of FW HW.
  • ISP HW reads the Raw image from the shared memory and performs image processing on the Raw image based on the obtained FW parameters (for example, read from the register) to obtain an RGB image.
  • ISP HW can output the acquired RGB image to the camera application.
  • the camera application can display the acquired RGB image on the display of the electronic device.
  • the autonomous driving application after the autonomous driving application obtains the RGB image output by the ISP HW, it can recognize the RGB image based on the application's CV perception network to obtain the recognition result, and based on the recognition As a result, automatic driving operation is performed.
  • the FW parameter prediction network in the FW parameter setting module can predict the Raw image of each frame collected by the camera, that is, for each Raw image , execute the process in Figure 10.
  • the FW parameter prediction network in the FW parameter setting module can be based on a preset period (for example, it can be based on the interval length, or it can be based on the image collected by the camera).
  • the frame rate setting can be set according to actual needs and is not limited in this application) to predict the Raw images collected by the camera.
  • the FW parameter setting module can make predictions every 1s to reduce the workload of the AI processor.
  • the FW parameter setting module can work in sleep mode after being called by the application, and only make a prediction when the application layer sends a wake-up signal to it.
  • the CPU such as an autonomous driving application
  • the autonomous driving application can send instruction information to the FW parameter setting module to trigger it to perform FW parameter prediction, and This parameter prediction can be done only once.
  • the FW parameter setting module can also be triggered after the electronic device is started.
  • the input of the FW parameter setting module may be a Raw image with a lower resolution after downsampling.
  • the downsampling operation may be performed by the AI processor or by the ISP HW, which is not limited in this application.
  • the ISP HW may obtain the original Raw image from the shared memory, and after downsampling the Raw image, output the Raw image with a lower resolution to the shared memory.
  • the FW parameter setting module may read the Raw image with a lower resolution from the shared memory and perform subsequent processing.
  • the Raw image read by the FW parameter setting module from the shared memory may also be the original Raw image obtained by other encoding or compression, which is not limited in this application.
  • FIG. 11 shows a schematic block diagram of a device 1100 according to an embodiment of the present application.
  • the device 1100 may include: a processor 1101 and a transceiver/transceiver pin 1102, and optionally, a memory 1103.
  • bus 1104 includes a power bus, a control bus, and a status signal bus in addition to a data bus.
  • bus 1104 various buses are referred to as bus 1104 in the figure.
  • the memory 1103 may be used for instructions in the foregoing method embodiments.
  • the processor 1101 can be used to execute instructions in the memory 1103, and control the receiving pin to receive signals, and control the transmitting pin to send signals.
  • the device 1100 may be an electronic device, a camera module, or a chip of an electronic device in the above method embodiment.
  • This embodiment also provides a computer storage medium that stores computer instructions.
  • the electronic device When the computer instructions are run on an electronic device, the electronic device causes the electronic device to execute the above related method steps to implement the method in the above embodiment.
  • This embodiment also provides a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to perform the above related steps to implement the method in the above embodiment.
  • inventions of the present application also provide a device.
  • This device may be a chip, a camera assembly (or a camera module), and the device may include a connected processor and a memory; where the memory is used to store computer execution instructions, When the device is running, the processor can execute computer execution instructions stored in the memory, so that the chip executes the methods in each of the above method embodiments.
  • the electronic equipment, computer storage media, computer program products or chips provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the corresponding methods provided above. The beneficial effects of the method will not be repeated here.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined or can be integrated into another device, or some features can be ignored, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • a unit described as a separate component may or may not be physically separate.
  • a component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the above integrated units can be implemented in the form of hardware or software functional units.
  • Integrated units may be stored in a readable storage medium if they are implemented in the form of software functional units and sold or used as independent products.
  • the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.
  • the steps of the methods or algorithms described in connection with the disclosure of the embodiments of this application can be implemented in hardware or by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules.
  • Software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read only memory (Read Only Memory, ROM), erasable programmable read only memory ( Erasable Programmable ROM (EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), register, hard disk, removable hard disk, compact disc (CD-ROM) or any other form of storage media well known in the art.
  • An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and storage media may be located in an ASIC.
  • Computer-readable media includes computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
  • Storage media can be any available media that can be accessed by a general purpose or special purpose computer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Signal Processing (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Neurology (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例提供了一种图像处理方法及装置。该方法包括：基于摄像头在不同场景下采集到的原始图像，生成对应的ISP参数，以基于不同的ISP参数，对不同原始图像进行图像处理，得到处理后的图像。从而实现ISP参数的动态调整，以基于不同的ISP参数对不同场景下采集的原始图像进行图像处理，提升图像处理的精准度，实现对不同场景进行自适应的图像增强。

Description

图像处理方法及装置
本申请要求于2022年09月14日提交中国国家知识产权局、申请号为202211115044.X、申请名称为“图像处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理领域，尤其涉及一种图像处理方法及装置。
背景技术
目前，随着图像处理技术的不断发展，其所应用的场景也越来越广泛。例如，终端设备可通过对摄像头采集到的图像进行图像处理，并将图像处理结果应用于自动驾驶、安防监控等场景中。
在已有技术的图像处理方法中，图像信号处理器(image signal processor，ISP)的固件(Firmware，FW)部分基于固定的算法参数，对摄像头采集到的图像进行图像处理，以得到处理后的图像。但是，基于固定算法参数的图像处理方法，其处理方法较为单一，无法适应所有场景。
发明内容
本申请实施例提供一种图像处理方法及装置。在该方法中，装置可基于根据不同场景下获取到的原始图像，生成满足对应场景的ISP参数，以实现ISP参数的动态调整，从而针对不同的场景进行自适应的图像增强。
第一方面，本申请实施例提供一种图像处理方法。该方法包括：装置获取摄像头在第一场景下采集到的第一原始图像。装置基于第一原始图像，获取第一ISP参数。装置基于第一ISP参数，对摄像头所采集到的第一原始图像进行图像处理，得到第一图像。装置将第一图像输出至目标应用，使得目标应用对第一图像进行图像任务处理。装置获取摄像头在第二场景下采集到的第二原始图像。装置基于第二原始图像，获取第二ISP参数。装置基于第二ISP参数，对摄像头采集到的第二原始图像进行图像处理，得到第二图像。装置将第二图像输出至目标应用，使得目标应用对第二图像进行图像任务处理。这样，本申请实施例可以获取到满足不同场景需求的ISP参数，从而实现ISP参数的动态调整，以基于不同的ISP参数对不同场景下采集的原始图像进行图像处理，提升图像处理的精准度，实现对不同场景进行自适应的图像增强。
示例性的，本申请实施例中的ISP参数为自动获取，即，可基于不同场景中所获取到的原始图像，预测出对应的ISP参数。
示例性的，装置可以周期性地进行ISP参数的获取，以降低装置的计算压力。
示例性的，目标应用可以是感知类应用、显示类应用等，本申请不做限定。
示例性的，不同的目标应用的图像任务处理不相同。例如，CV感知应用的图像处理人物可以包括但不限于2D/3D目标检测、车道线检测、场景语义分割等，本申请不做限定。
在一种可能的实现方式中，第一原始图像与第二原始图像的图像内容不同，和/或，第一原始图像与第二原始图像的图像属性不同。本申请实施例中，不同场景下拍摄的图像，其内容和/或属性不相同，本申请均可以根据不同的原始图像，预测出对应的ISP参数，以得到更好的图像处理结果，从而适应不同的场景需求。
在一种可能的实现方式中，第一ISP参数与第二ISP参数用于调整图像的以下至少一种图像属性：亮度、颜色、噪声、锐度、对比度。这样，本申请实施例中，不同的ISP参数可以实现对不同的图像属性进行调整，以得到更好的图像效果。
在一种可能的实现方式中，第一图像与第二图像的图像属性满足目标应用的图像任务需求。本申请的图像处理均是为了满足目标应用的需求，使得目标应用能够更好的识别经过处理后得到的图像，提升目标应用的图像识别(或其它图像处理流程)的精准度。
在一种可能的实现方式中，基于第一原始图像，获取第一ISP参数，包括：将第一原始图像输入至ISP参数预测模型，获取ISP参数预测模型基于第一原始图像预测出的第一ISP参数；基于第二原始图像，获取第二ISP参数，包括：将第二原始图像输入至ISP参数预测模型，获取ISP参数预测模型基于第二原始图像预测出的第二ISP参数。这样，本申请实施例中通过预先训练的ISP参数预测模型，可基于不同场景下的原始图像，实现ISP参数的自动化预测，以生成满足不同场景需求的ISP参数。
在一种可能的实现方式中，ISP参数预测模型运行在神经网路处理器NPU上。这样，本申请实施例中的预测模型运行在NPU的通用算力上，且只需要很小的计算量。并且，本申请能在边缘设备进行部署，不挤占AI处理器上的高优先级任务。
在一种可能的实现方式中，获取摄像头在第一场景下采集到的第一原始图像之前，方法还包括：从云端获取ISP参数预测模型。这样，本申请实施例中，装置只需要在售芯片的ISP系统支持参数的动态刷写，该方案就可以通过空中下载更新的方式(即从云端获取并安装)对已经在市面上流通的产品进行支持，以满足兼容性需求。
在一种可能的实现方式中，基于第一ISP参数，对第一原始图像进行图像处理，得到第一图像，包括：将ISP内存中当前保存的ISP参数替换为第一ISP参数，使得ISP从ISP内存中获取第一ISP参数，并基于第一ISP参数，对第一原始图像进行图像处理，输出第一图像；基于第二ISP参数，对第二原始图像进行图像处理，得到第二图像，包括：将ISP内存中当前保存的第一ISP参数替换为第二ISP参数，使得ISP从ISP内存中获取第二ISP参数，并基于第二ISP参数，对第二原始图像进行图像处理，输出第二图像。这样，在本申请实施例中，最终作用在图像上的仍然是传统的ISP处理算法，能够确保整个处理过程的可控性和可解释性。
第二方面，本申请实施例提供一种模型训练方法。该方法包括：向图像信号处理器ISP参数预测模型输入N个原始图像，获取ISP参数预测模型输出的N个ISP参数，ISP参数预测模型的权重为第一权重，N个ISP参数为ISP参数预测模型基于第一权重值得到的，N为大于1的整数。向代理模型输入N个ISP参数与N个原始图像，获取代理模型输出的N个图像。向目标应用输入N个图像，获取目标应用输出的图像任务处理结果。若图像任务处理结果未满足预设条件，调整ISP参数预测模型的权重为第二权重，再次从向ISP参数预测模型输入N个原始图像的步骤开始执行，直至目标应用输出的图像任务处理结果满足预设条件。这样，代理网络可以实现ISP的功能，通过可微化的代理网络与FW参数预测网络联合训练，实现ISP参数预测模型与目标应用之间的端到端循环训练流程。
示例性的，模型训练方法可以在计算节点上执行。
在一种可能的实现方式中，预设条件包括：N个原始图像的预设真值标签与目标应用的图像任务处理结果中包括的真值标签之间的差值小于预设阈值。这样，本申请实施例中可基于原始图像的真实标签和目标应用输出的标签之间的差异，判断代理网络所输出的图像是否满足目标应用的图像任务需求，以进一步判断代理网络所使用的ISP，即ISP参数预测网络所预测的ISP参数是否符合需求。
在一种可能的实现方式中，向ISP参数预测模型输入N个原始图像之前，方法还包括：向ISP输入M个原始图像和M个ISP参数，获取图像处理器输出的M个图像；其中，M个原始图像和M个 ISP参数一一对应，M为大于1的整数；向代理模型输入M个原始图像和M个ISP参数，获取代理模型输出的M个图像；代理模型的当前权重为第三权重；若代理模型输出的M个图像与图像处理器输出的M个图像之间的相似度小于阈值，调整代理模型的权重为第四权重，再次从向ISP输入M个原始图像和M个ISP参数的步骤开始执行，直至代理模型输出的M个图像与图像处理器输出的M个图像之间的相似度大于阈值。这样，通过训练代理网络，可以使得代理网络的处理结果逼近或与ISP的处理结果相同，从而使得代理网络在联合训练过程中实现ISP的功能。
第三方面，本申请实施例提供一种图像处理装置。该装置包括：ISP参数预测模块和ISP模块。ISP参数预测模块，用于获取摄像头在第一场景下采集到的第一原始图像；ISP参数预测模块，还用于基于第一原始图像，获取第一图像信号处理器ISP参数；图像处理ISP模块，用于基于第一ISP参数，对第一原始图像进行图像处理，得到第一图像；ISP模块，还用于将第一图像输出至目标应用，使得目标应用对第一图像进行图像任务处理；ISP参数预测模块，还用于获取摄像头在第二场景下采集到的第二原始图像；ISP参数预测模块，还用于基于第二原始图像，获取第二ISP参数；ISP模块，还用于基于第二ISP参数，对第二原始图像进行图像处理，得到第二图像；ISP模块，还用于将第二图像输出至目标应用，使得目标应用对第二图像进行图像任务处理。
在一种可能的实现方式中，第一原始图像与第二原始图像的图像内容不同，和/或，第一原始图像与第二原始图像的图像属性不同。
在一种可能的实现方式中，第一ISP参数与第二ISP参数用于调整图像的以下至少一种图像属性：亮度、颜色、噪声、锐度、对比度。
在一种可能的实现方式中，第一图像与第二图像的图像属性满足目标应用的图像任务需求。
在一种可能的实现方式中，ISP参数预测模块，用于将ISP内存中当前保存的ISP参数替换为第一ISP参数；ISP模块，用于从ISP内存中获取第一ISP参数，并基于第一ISP参数，对第一原始图像进行图像处理，输出第一图像；ISP参数预测模块，用于将ISP内存中当前保存的第一ISP参数替换为第二ISP参数；ISP模块，用于从ISP内存中获取第二ISP参数，并基于第二ISP参数，对第二原始图像进行图像处理，输出第二图像。
第四方面，本申请实施例提供一种模型训练系统，包括图像信号处理器ISP参数预测模型、代理模型和目标应用。该系统向ISP参数预测模型输入N个原始图像，获取ISP参数预测模型输出的N个ISP参数，ISP参数预测模型的权重为第一权重，N个ISP参数为ISP参数预测模型基于第一权重值得到的，N为大于1的整数。该系统向代理模型输入N个ISP参数与N个原始图像，获取代理模型输出的N个图像。该系统向目标应用输入N个图像。若N个图像未满足目标应用的图像任务需求，该系统调整ISP参数预测模型的权重为第二权重，再次从向ISP参数预测模型输入N个原始图像的步骤开始执行，直至代理模型输出的N个图像满足目标应用的图像任务需求。
在一种可能的实现方式中，系统还包括ISP；向ISP参数预测模型输入N个原始图像之前，向ISP输入M个原始图像和M个ISP参数，获取图像处理器输出的M个图像；其中，M个原始图像和M个ISP参数一一对应，M为大于1的整数；向代理模型输入M个原始图像和M个ISP参数，获取代理模型输出的M个图像；代理模型的当前权重为第三权重；若代理模型输出的M个图像与图像处理器输出的M个图像之间的相似度小于阈值，调整代理模型的权重为第四权重，第四权重与第三权重不相同，再次从向图像处理器输入M个原始图像和M个ISP参数，获取图像处理器输出的M个图像开始执行，直至代理模型输出的M个图像与图像处理器输出的M个图像之间的相似度大于阈值。
第五方面，本申请实施例提供一种电子设备，包括：一个或多个处理器；存储器；以及一个或多个计算机程序，其中一个或多个计算机程序存储在存储器上，当计算机程序被一个或多个处理器执行时，使得电子设备执行第一方面或第一方面的任意可能的实现方式中的方法的指令的方法。
第六方面，本申请实施例提供一种摄像模组，包括：一个或多个处理器；存储器；以及一个或多个计算机程序，其中一个或多个计算机程序存储在存储器上，当计算机程序被一个或多个处理器执行时，使得摄像模组执行第一方面或第一方面的任意可能的实现方式中的方法的指令的方法。
第七方面，本申请实施例提供了一种计算机可读介质，用于存储计算机程序，该计算机程序包括用于执行第一方面或第一方面的任意可能的实现方式中的方法的指令。
第八方面，本申请实施例提供了一种计算机程序，该计算机程序包括用于执行第一方面或第一方面的任意可能的实现方式中的方法的指令。
第九方面，本申请实施例提供了一种芯片，该芯片包括处理电路、收发管脚。其中，该收发管脚、和该处理电路通过内部连接通路互相通信，该处理电路执行第一方面或第一方面的任一种可能的实现方式中的方法，以控制接收管脚接收信号，以控制发送管脚发送信号。
附图说明
图1是电子设备的硬件结构示意图；
图2是电子设备的硬件结构示意图；
图3为示例性示出的图像处理方法的流程示意图；
图4为示例性示出的图像处理方法的流程示意图；
图5为示例性示出的图像处理方法的流程示意图；
图6为示例性示出的代理网络的结构示意图；
图7为示例性示出的代理网络训练流程示意图；
图8为示例性示出的FW参数预测网络训练流程示意图；
图9为示例性示出的FW参数预测网络训练的系统结构示意图；
图10为示例性示出的图像处理流程示意图；
图11为示例性示出的一种装置的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
本申请实施例的说明书和权利要求书中的术语“第一”和“第二”等是用于区别不同的对象，而不是用于描述对象的特定顺序。例如，第一目标对象和第二目标对象等是用于区别不同的目标对象，而不是用于描述目标对象的特定顺序。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
在本申请实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。例如，多个处理单元是指两个或两个以上的处理单元；多个系统是指两个或两个以上的系统。
本申请实施例中的图像处理方法应用于电子设备，其中，电子设备可以是手机、平板、车载设备、安保设备、电脑、可穿戴设备、智能家居设备等，本申请不做限定。
图1示出了电子设备100的结构示意图。应该理解的是，图1所示电子设备100仅是电子设备的一个范例，并且电子设备100可以具有比图中所示的更多的或者更少的部件，可以组合两个或多个的部件，或者可以具有不同的部件配置。图1中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。需要说明的是，本申请实施例中仅以电子设备为手机为例进行说明，在其他实施例中，电子设备可以是上文所述的任意设备，本申请不做限定。
电子设备100可以包括：处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K，充电器，闪光灯，摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现电子设备100的触摸功能。
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170 与无线通信模块160可以通过PCM总线接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备100的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块160，音频模块170，传感器模块180等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为电子设备100充电，也可以用于电子设备100与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如AR设备等。
可以理解的是，本申请实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块140可以通过电子设备100的无线充电线圈接收无线充电输入。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为电子设备供电。
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块141也可以设置于处理器110中。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，电信号经由ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，白平衡，对比度等属性进行算法优化。在一些实施例中，ISP可以设置在摄像头193中。在本申请实施例中，不同厂商不同型号的ISP，其所优化的属性可根据实际需求设置。例如，电子设备A的ISP可对图像的噪点和亮度进行优化。电子设备B的ISP可对图像的亮度和对比度进行优化，本申请不做限定。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。
在本申请实施例中，摄像头193捕捉的图像可以称为Raw图像(或简称为Raw图)，可以理解为 是原始图像文件。ISP可对Raw图像进行图像处理，并输出RGB图像或YUV图像。其中，RGB和YUV都是色彩空间，用于表示颜色。YUV中的“Y”表示明亮度，也就是灰阶值；而“U”和“V”表示的则是影像色彩及饱和度，用于指定像素的颜色。RGB色彩模式是工业界的一种颜色标准，是通过对红(R)、绿(G)、蓝(B)三个颜色通道的变化以及它们相互之间的叠加来得到各式各样的颜色的。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。在本申请实施例中，NPU可用于支持FW参数设置模块中的FW参数预测网络，即，由NPU执行FW参数设置模块中的FW参数预测网络所执行的步骤。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行电子设备100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备100可以通过扬声器170A收听音乐，或收听免提通话。
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备100接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。电子设备100可以设置至少一个麦克风170C。在另一些实施例中，电子设备100可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备100还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180A，电极之间的电容改变。电子设备100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，电子设备100根据压力传感器180A检测所述触摸操作强度。电子设备100也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触 摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。
陀螺仪传感器180B可以用于确定电子设备100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定电子设备100围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器180B检测电子设备100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备100的抖动，实现防抖。陀螺仪传感器180B还可以用于导航，体感游戏场景。
气压传感器180C用于测量气压。在一些实施例中，电子设备100通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。
磁传感器180D包括霍尔传感器。电子设备100可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当电子设备100是翻盖机时，电子设备100可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。
加速度传感器180E可检测电子设备100在各个方向上(一般为三轴)加速度的大小。当电子设备100静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。
距离传感器180F，用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备100可以利用距离传感器180F测距以实现快速对焦。
接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备100通过发光二极管向外发射红外光。电子设备100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备100附近有物体。当检测到不充分的反射光时，电子设备100可以确定电子设备100附近没有物体。电子设备100可以利用接近光传感器180G检测用户手持电子设备100贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式，口袋模式自动解锁与锁屏。
环境光传感器180L用于感知环境光亮度。电子设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测电子设备100是否在口袋里，以防误触。
指纹传感器180H用于采集指纹。电子设备100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
温度传感器180J用于检测温度。在一些实施例中，电子设备100利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，电子设备100执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备100对电池142加热，以避免低温导致电子设备100异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备100对电池142的输出电压执行升压，以避免低温导致的异常关机。
触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于电子设备100的表面，与显示屏194所处的位置不同。
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180M也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180M获取的血压跳动信号解析心率信息，实现心率检测功能。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备100可以接收按键输入，产生与电子设备100的用户设置以及功能控制有关的键信号输入。
马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和电子设备100的接触和分离。电子设备100可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口195可以支持Nano SIM卡，Micro SIM卡，SIM卡等。同一个SIM卡接口195可以同时插入多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容不同类型的SIM卡。SIM卡接口195也可以兼容外部存储卡。电子设备100通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，电子设备100采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在电子设备100中，不能和电子设备100分离。
电子设备100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以分层架构的Android系统为例，示例性说明电子设备100的软件结构。在其他实施例中，例如电子设备100为车载设备，则系统可以是Linux系统，本申请不做限定。
图2是本申请实施例的电子设备100的软件结构框图。
电子设备100的分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。
如图2所示，应用程序包可以包括相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图2所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器，FW参数设置模块等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供电子设备100的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。
FW参数设置模块可包含FW参数预测网络。FW参数预测网络用于基于场景需求，预测对应的FW参数。FW参数设置模块可位于应用层。在本申请实施例中，FW参数预测网络所获取到的FW参数也可以称为是ISP参数、FW超参数、或者是ISP超参数，可根据实际需求设置，本申请不做限定。可以理解为，ISP在后续使用过程中，可以基于FW超参数，以及预设的图像处理参数函数，得到图像处理参数。ISP可以基于图像处理参数对Raw图像进行处理。
可选地，在其他实施例中，FW参数设置模块也可以运行于应用层，其可以理解为是运行在NPU上的一段代码，本申请不做限定。
可选地，用户可以通过对电子设备的系统进行更新，以更新FW参数设置模块。例如，对于未安装FW参数设置模块的电子设备，用户可以对电子设备进行操作，电子设备可从云端获取到的更新版本，并基于更新版本对系统进行更新，以安装FW参数设置模块。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动等。
可以理解的是，图2示出的系统框架层、系统库与运行时层包含的部件，并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。
为更好的理解本申请实施例中的技术方案，在说明本申请实施例的图像处理方法之前，结合图3对已有技术实施例中的图像处理方法进行简单介绍。请参照图3，示例性的，电子设备中的摄像头采集图像，图像称为Raw图像。ISP可从存储(也可以称为共享内存，例如可以是双倍速率(Double Data Rate)存储器，本申请不做限定)中获取FW参数。该FW参数是电子设备出厂前设置的，DDR中存储一套FW参数。FW参数可以理解为是数组。FW参数可以理解为是用于调整ISP处理过程中至少一种图像属性的数组。例如，ISP可基于FW参数中的第一参数，对Raw图像中的亮度属性进行调整，使得生成的RGB(也可以是YUV，下文中不再赘述)图像的亮度更加明亮或者更加暗淡，本申请不做限定。
电子设备中的应用(例如感知应用，也可以是其它应用，例如安防应用等，本申请不做限定，下文中不再重复说明)可获取到RGB图像，并对RGB图像进行相应处理。例如，感知应用可以对RGB图像进行AI识别，以识别RGB图像中的人物或其它事务等，本申请不做限定。
需要说明的是，各模块之间可能还包括与其它模块之间的交互，例如，ISP可能是通过摄像头驱动获取到Raw图像，本申请不做限定，下文中不再重复说明。
在图3所示的已有技术中，由于FW参数是电子设备出厂前设置的，其可以理解为是固定FW参数。即，ISP对所有获取到的Raw图像均是作相同的处理。例如，FW参数中的第一参数指示将Raw图像的亮度(即亮度属性)调整到第一阈值，则对于每个Raw图像，ISP均会基于FW参数，将Raw图像的亮度参数调整到第一阈值。而该种调整方式过于单一，无法适用于全场景中。举例说明，在车 载设备中，摄像头在隧道内获取到的图像亮度较暗，如果ISP仍然按照预设的参数，将摄像头在隧道内获取到的Raw图像的亮度参数调整到第一阈值，可能其生成的RGB图像的亮度仍然较暗，使得下游应用(例如感知应用)可能无法准确的识别到图像中的物体。
图4为本申请实施例提供一种图像处理方法的原理示意图。请参照图4，在本申请实施例中的图像处理方法中，AI(Artificial Intelligence，人工智能)处理器(例如可以是NPU，也可以是TPU等，本申请不做限定)上运行的FW参数设置模块(具体是FW参数预测网络)可以基于Raw图像预测出对应的最佳FW参数(为区别于图3中的FW参数，本申请称其为动态FW参数)，ISP可基于AI处理器预测出的动态FW参数，对Raw图像进行图像处理，并生成对应的RGB图像。感知应用(也可以是其它应用)可基于RGB图像进行相应处理，例如图像识别等。本申请实施例中的图像处理方法可以根据场景需求，动态调整FW参数，使得ISP可以对摄像头在不同场景下获取到的图像进行适应性处理，以得到效果更加符合场景需求，图像更加清晰的处理结果，提升下游应用的识别准确性。
例如，以辅助驾驶或自动驾驶感知系统的应用场景为例。在辅助驾驶或自动驾驶业务中，车载相机(即车载设备上继承的摄像头)及车载设备(例如为车载设备中的MCU)需要对道路环境进行实时的多类型感知，如2D/3D目标检测、车道线检测、场景语义分割等。具体检测方式可参照已有技术实施例中的相关描述，本申请不做限定。目前，此类计算机视觉感知任务通常由基于CNN(Convolutional Neutral Network，卷积神经网络)或transformer架构的深度模型承载。对于摄像头采集到的同一张原始图像(即Raw图像)，当配置不同的FW参数时，能够得到具有不同图像效果的输出图像(例如RGB或YUV图像。此时，将这些图像输入至CNN或transformer感知模型时，也将得到不同的预测结果，从而影响车载感知系统甚至规控系统的准确性与可靠性。在本申请实施例中，FW参数设置模块中的FW参数预测网络可以为每个Raw图像预测出一组最适合的FW参数，从而使该Raw图像经过ISP处理后得到的RGB或YUV图像在一个或多个感知任务上得到最佳的预测效果，从而提升辅助驾驶或自动驾驶场景下的感知精度。例如，对于极低光照场景，期望针对ATR(Adaptive Tone Reproduction，自适应阶调还原)模块预测出一组亮度提升较强的参数，以提升图像暗部的细节还原能力，同时针对降噪模块预测出一组降噪强度较弱的参数，以减少因涂抹而导致图像细节丢失。对于雨雾场景，期望针对DRC(Dynamic Range Compression，动态范围压缩)模块预测出一组对比度提升较强的参数，以弥补环境能见度较低而导致的物体轮廓丢失。
需要说明的是，本申请实施例中以辅助驾驶或自动驾驶应用场景为例进行说明，对于其它应用场景其具体实现方式是相同的，本申请不再逐一举例说明，下文中不再重复说明。
图5为示例性示出的图像处理方法的流程示意图。请参照图5，本申请实施例中的图像处理方法可以分为离线训练阶段和在线部署阶段。离线阶段包括但不限于两个阶段：
1)代理网络训练阶段
2)FW参数预测网络训练阶段
其中，代理网络可以理解为是代替ISP的网络，其处理结果与ISP处理结果相同或相似。可以理解为，代理网络训练的过程，是将代理网络对图像的处理结果逐渐逼近ISP的处理结果的过程。
FW参数预测网络训练阶段可以理解为是基于训练好的代理网络，对FW参数预测网络进行训练，以得到预测结果更加准确的FW参数预测网络。其中，预测结果即为FW参数。
示例性的，在线部署阶段可选地为将离线训练阶段获取到的参数预测网络部署到目标设备中。目标设备可以是终端设备、车载设备、安防监控设备等，本申请不做限定。
下面对图5所示的离线训练阶段和在线部署阶段进行详细说明。
离线训练阶段：
在本申请实施例中，离线训练阶段可以由计算节点执行。计算节点可以是计算机、服务器、电脑等任意设备。具体地，离线阶段中的步骤可以是由计算节点中的GPU处理器执行的，本申请不做限定。
在本申请实施例中，离线训练阶段可以是在任意时间点执行，以得到训练好的参数预测网络。一个示例中，电子设备在出厂前可安装支持参数预测网络的FW参数设置模块。另一个示例中，离线训 练阶段也可以是周期性执行的(例如一个月执行一次)以更新参数预测网络，计算节点可将更新后的参数预测网络上传至云端(云端包括一个或多个服务器)，云端可向电子设备推送系统更新版本，系统更新版本中包括参数预测网络，电子设备可基于系统更新版本，更新FW参数设置模块，以获取到新的参数预测网络。
在本申请实施例中，离线训练阶段开始之前操作人员需要进行数据及模型准备。
准备的数据包括但不限于：
a.N个Raw图像。N可以为任意数值。示例性的，操作人员可控制图像传感器(例如车载摄像头)采集图像，以获取到N个Raw图像，以下称为训练Raw图像。采集到的图像覆盖的场景可选地尽量贴合部署阶段的电子设备(例如车载设备或终端设备)所处的工作环境。例如，对于车载设备，其所工作环境可能涉及到隧道以及不用天气环境下的公路等，操作人员可以控制图像传感器采集多个隧道的图像以及不同天气环境下的不同公路的图像等，本申请不做限定。
当然，参与训练的Raw图像的数量越多，且图像之间相差越大，则参数预测网络的训练结果越准确。示例性的，图像之间的差别可选地为图像之间的各参数的差值，例如包括上文所述的噪声参数、亮度参数等。
本申请实施例中所涉及到的图像，例如Raw图像或RGB图像等，对于计算机而言是数组，相应的，本申请实施例中在描述图像时，实际上也可以理解为是在描述图像所对应的数组，或者是可以进一步理解为是在描述图像的数组所对应的参数。在本申请实施例中，对图像的任意处理过程，例如ISP等对图像进行处理，则可以理解为是对图像所对应的像素值进行处理，下文中不再重复说明。
在一种可能的实现方式中，对于不同类型的电子设备或者是同一类型但不同型号的电子设备，其摄像头采集到的图像可以相同或不同。例如，车载设备A与车载设备B是同一厂家不同型号的车载设备，其摄像头参数不相同，则采集到的Raw图像不同。其中，图像不同可选地为属性(例如亮度、色温等)不同。
在本申请实施例中，属性不同可选地为属性的种类、数量和/或强弱(也可以是高低、明暗等)不相同。例如，车载设备A的摄像头与车载设备B的摄像头在相同环境下采集相同画面。车载摄像头A采集到的Raw图像的属性包括但不限于：属性A(例如是色温)、属性B(例如是白平衡)和属性C(例如是亮度)，而车载设备B的摄像头采集到的Raw图像的属性包括但不限于：属性A、属性B、属性D(例如是噪声)和属性E(例如是Gamma)。其中，车载设备A的摄像头采集到的图像的属性A与车载设备B的摄像头采集到的图像的属性A的大小(也可以是高低)相同或不同，本申请不做限定。相应的，在本申请实施例中，在训练阶段，操作人员可以对不同类型和/或不同型号的设备执行训练操作，以获取到对应于不同类型和/或不同型号的设备的参数预测网络。本申请仅以单一设备对应的训练过程为例进行说明，实际上，在训练阶段，计算节点可基于不同的设备反复执行本申请实施例中所述的训练阶段，以得到对应的参数预测网络，本申请不再逐一举例说明。
b.N组FW参数。示例性的，操作人员可利用Latin-Hypercube等采样算法(可根据实际需求设置对应的公式，本申请不做限定)采样得到N组FW参数。如上文所述，FW参数为数组，相应的，N组FW参数也可以理解为是N个数组。
在本申请实施例中，每组训练FW参数可选地为一组长度为n的向量。在本申请实施例中，n的数值与ISP的处理能力相关。如上文所述，不同型号的ISP所处理的属性可能相同或不同，例如，型号A的ISP用于处理亮度、色温等q个属性，则n与q相关。可选地，FW参数中的一个参数可以用于实现对一种属性的调整，FW参数中也可以通过多个参数实现对一种属性的调整，本申请不做限定。
需要说明的是，为区分于在线部署过程中的FW参数，本申请实施例中将准备过程中的FW参数称为训练FW参数。
在本申请实施例中，是以参与训练的Raw图像的数量等于N(即FW参数的组数量)为例进行说明的。在其他实施例中，Raw图像的数量也可以大于N或小于N。只需要保证每个Raw图像对应一组FW参数即可(对应的FW参数可以相同或不同)。当然，FW参数的数量越多，即参加训练的样本越多，则训练所得到的网络(例如代理网络)越接近用户需求。
c.CV(Computer Vision，计算机视觉)感知网络(也可以简称为CV网络、CV任务网络、感知网络或感知模型等，本申请不做限定)。CV感知网络用于对特定的计算机视觉任务进行预测，例如对道路场景中的车辆、行人、交通指示牌等物体进行2D/3D框的预测、对场景中的实例进行语义分割、对特定对象进行跟踪等等。需要说明的是，本申请实施例中涉及到的CV感知网络并未对具体的感知任务进行指定，只要该任务具备明确的目标函数，且该感知网络具备端到端可训练的特性即可。对于大部分有监督CV任务以及大部分基于CNN构建的深度模型而言，这两点前提很容易得到满足。进一步需要说明的是，考虑到在线端部署时的功耗和实时性要求，本申请实施例中默认CV感知网络接收RGB或YUV图像作为输入，经过一系列的计算模块或子网络(如backbone(主干网络)、RPN(Region Proposal Network，候选区域生成网络)等)后输出特定任务所需的预测结果，如2D/3D检测框、物体类别标签、预测置信度、分割掩模、物体运动轨迹等。
d.真值标签。示例性的，操作人员可为训练所用的Raw图像设置真值标签，用于标识Raw图像经过CV感知网络的真实结果。真值标签包括但不限于：2D/3D检测框、物体类别标签、预测置信度、分割掩模、物体运动轨迹等标签。其具体设置是根据CV感知网络设置的。举例说明，若CV感知网络用于对RGB图像进行物体类别识别，操作人员可预先为Raw图像设置真值标签，即，通过人工鉴定的方式，设置Raw图像中包括的物体类别标签。例如，Raw图像中包括小猫和小狗的图像，操作人员为该Raw图像设置的物体类别标签包括小猫标签和小狗标签。真值标签的作用是用于与训练过程中CV实际输出的结果进行比较，以确定CV感知网络输出的结果与真实结果(即真值标签)之间的偏差。具体细节将在下面实施例中详细说明。
e.代理网络。在本申请实施例中，代理网络可以是卷积神经网络(CNN)、递归神经网络(RNN)，也可以是其他的一些深度模型(如transformer等)。本发明并未对其所采用的具体模型类型进行限制，只需确保其接收一张Raw图像以及一组FW参数作为输入，且具有可微分性质即可。
示例性的，代理网络还设置有初始权重，为区别于FW参数预测网络的权重，本申请实施例中将代理网络的权重称为代理网络权重，将FW参数预测网络的权重称为FW参数预测网络权重。示例性的，本申请实施例中所涉及到的权重可以是一组数组。以Transformer模型构成的代理网路为例。图6为示例性示出的代理网络的结构示意图。请参照图6，该结构为Transformer模型的结构，其中包括Encoder(编码)和Decoder(解码)部分。Encoder包括但不限于：多头注意力层(Multi-Head Attention layer)，层归一化(Layer Normalization)(即图6中的(Norm))和前馈神经网络(Feed forward neural network)(即图6中的Feed Forwad)等模块。Decoder包括但不限于：标记多头注意力(Masked multi-head attention)层，多头注意力层，层归一化(即图6中的(Norm))和前馈神经网络(即图6中的Feed forwad)等模块。其中，每个模块对应一组权重值，所有模块的权重值组合，构成代理网络的权重值。
f.FW参数预测网络。FW参数预测网络是一个部署于AI处理器上的深度网络，其按需从存储(例如DDR)中读取当前摄像头采集到的视频流中待处理的Raw图像数据，并进行最佳FW参数预测(也可以理解为是网络推理)。FW参数预测网络将预测出的参数(一组长度为N的向量，N为可动态配置的FW参数个数)写入ISP Firmware的共享内存中，覆盖原有的FW参数。
本申请实施例中，FW参数预测网络与代理网络类似，可以是卷积神经网络(CNN)、递归神经网络(RNN)，也可以是其他的一些深度模型(如transformer等)。本发明并未对其所采用的具体模型类型进行限制，只需确保其具有可微分性质即可。
示例性的，FW参数预测网络同样设置有初始权重。初始权重可以是操作人员任意设置的，本申请不做限定。FW参数预测网络的权重可以称为FW参数预测网络权重，以区别于其它权重。其它未描述部分可以参照代理网络权重的相关描述，此处不再赘述。
1)代理网络训练阶段
示例性的，代理网络用于使用一个可微分的模型，对ISP中不可微的图像处理算法进行代理，从而允许梯度下降算法等一阶优化算法对FW参数预测网络、代理网络以及CV感知网络的完整数据通路进行端到端联合优化。
图7为示例性示出的代理网络训练流程示意图。请参照图7，具体包括但不限于：
S701，ISP基于训练FW参数对训练Raw图像进行图像处理，得到RGB_ISP图像。
示例性的，ISP是一套由Firmware和Hardware两部分组件(某些ISP中还会包括部分运行在AI处理上的软件部分)构成的SoC子系统。其中，Firmware部分负责执行图像处理算法所需的一些调度工作，例如从指定内存区域中读取FW参数、控制各模块的开关状态以及执行顺序、修改各寄存器的状态位等，同时执行部分低强度的逻辑运算。Hardware部分负责执行图像处理算法中的高强度运算，其计算逻辑受到寄存器中的状态位控制。
如上文所述，在准备阶段，操作人员准备N个训练FW参数和N个训练Raw图像。操作人员可预先设置N个训练FW参数与N个训练Raw图像之间的对应关系。例如，Raw图像1与FW参数1对应，Raw图像2与FW参数2对应……Raw图像N与FW参数N对应。对应关系可任意设置，只需保证每个Raw图像对应一个FW参数，不同的Raw图像之间对应的FW参数尽量不同，以增加样本之间的差异度。
示例性的，计算节点可以包括N个ISP，每个ISP用于基于FW参数对一个Raw图像进行图像处理，以得到N个RGB图像(也可以是YUV图像，本申请不做限定，下文中不再重复说明)。在本申请实施例中，ISP基于FW参数对Raw图像的处理，可以具体包括但不限于：ISP自身携带图像处理参数算法或者是图像处理参数函数，ISP可基于预设的图像处理参数算法和FW参数(即FW超参数)，获取到对应的图像处理参数。Raw图像为数组，数组中包括多个灰度值。ISP可基于图像处理参数，对数据中的至少一个灰度值进行调整，以改变Raw图像的属性。例如，ISP基于图像处理参数，将Raw图像中的部分低于阈值(可根据实际需求设置)的灰度值调整到期望值，使得生成的RGB图像的亮度比未调整前的亮度更高。对于ISP的具体处理，具体细节可以参照已有技术实施例中的相关内容，本文不再赘述，下文中不再重复说明。
在本申请实施例中，为区别于其它RGB图像，本申请实施例中将ISP获取到的RGB图像称为RGB_ISP图像。ISP的图像处理过程可以理解为是基于FW参数对对应的Raw图像的属性进行修正(或处理)，以得到RGB图像。RGB图像的属性即为基于FW参数处理后的属性。例如，Raw图像1中包括色温A，ISP基于FW参数(FW参数中包括色温阈值，也可以称为色温修正参数或色温修正阈值，本申请不做限定)对Raw图像1进行图像处理，得到RGB图像1。其中，RGB图像1中的色温B与Raw图像1中的色温A的高低相同或不同。ISP的图像处理的具体细节可参照已有技术实施例中的相关描述，本申请不再赘述。
在一种可能的实现方式中，ISP也可以基于N个训练FW参数中的每个参数，对N个训练Raw图像中的每个图像分别进行图像处理，以得到N×N个RGB_ISP图像，以扩充训练集的规模。具体训练方式可根据实际需求设置，本申请不做限定。
需要说明的是，如上文所述，不同型号或不同类型的电子设备的ISP的处理和输出结果可能是不相同的。因此，本申请实施例中所涉及到的ISP可以是任意类型的设备中的任意型号的ISP。假设图7中所涉及到的流程均是针对A型号的电子设备的ISP执行的，则图7流程所获取到的代理网络以及下文实施例中获取到的FW参数预测网络都是对应于A型号的电子设备的。若需要获取到对应于B型号的电子设备的FW参数预测网络，则需要基于B型号的电子设备，重新执行图7以及下文实施例中的各训练流程，例如在S701中所涉及到的ISP将替换为B型号的电子设备的ISP。本申请实施例中不再重复举例说明，下文中不再重复说明。
在一种可能的实现方式中，本申请实施例中的训练阶段所涉及到的器件(例如ISP)或模块可以全部集成于计算节点中，由计算节点运行。例如，计算节点中的ISP即为A型号的电子设备的ISP。
在另一种可能的实现方式中，计算节点可以是包括多个独立的器件或模块。例如，ISP以及代理网络是不同的器件，代理网络可以是在计算节点的GPU上。ISP可以是A型号电子设备中的ISP执行，并输出给计算节点的GPU的，计算节点的系统结构可以根据实际需求设置，本申请不做限定。
S702，代理网络基于训练FW参数和训练Raw图像，输出RGB_DL图像。
示例性的，如上文所述，代理网络可以是深度感知模型构成的(也可以是其它神经网络，本申请不做限定)。计算节点将N个Raw图像以及N训练FW参数作为代理网络的输入，代理网络基于训 练FW参数、N个Raw图像以及初始代理网络权重进行推理(或运算)，输出RGB_DL图像。
在本申请实施例中，如上文所述，操作人员已预先设置好N个Raw图像与N个FW参数之间的对应关系。相应的，代理网络在处理时，仍然是按照已设置好的对应关系进行处理的。可以理解为，代理网络在处理时，Raw图像与FW参数之间的对应关系需要与ISP进行图像处理时的Raw图像与FW参数之间的对应关系一致，才能具备可对比性。
在本申请实施例中，计算节点可以根据代理网络的处理能力，确定每次输入代理网络的Raw图像和FW参数(即组的数量)的数量。每次输入的Raw图像和FW参数是保持上文所述的对应关系的。每次输入的Raw图像(和FW参数)的数量可以相同或不同。例如，在本申请实施例中，计算节点可以每次向代理网络输入20个Raw图像和20个FW参数。数值仅为示意性举例，可根据实际需求设置，本申请不做限定。
示例性的，代理网络可基于当前的权重(可以是初始权重，也可能是更新后的权重)对每次输入的多个Raw图像和FW参数进行运算。在运算过程中，其与ISP类似，是基于每个Raw图像所对应的FW参数，对Raw图像进行处理，以得到处理结果。
示例性的，代理网络对N个Raw图像与N组FW参数进行处理(可以是分批次处理)，可得到N个RGB_DL图像。
示例性的，如上文所述，代理网络设置有初始权重，初始权重是操作人员任意设置的。代理网络在初始阶段(例如第一次或前几次训练过程)所输出的结果(即数组)可能是毫无意义的，相应的，本申请实施例中图7所示的流程即可理解为是通过不断地对代理网络进行训练，以更新代理网络权重，以使得代理网络输出的结果逐渐逼近S701中ISP所处理的结果(即RGB_ISP图像)。
S703，判断RGB_IGP图像与RGB_DL图像之间的相似度是否大于阈值。
示例性的，计算节点(具体可以是计算节点的CPU)获取到ISP输出的N个RGB_ISP图像和代理网络输出的N个RGB_DL图像。计算节点可基于N个RGB_ISP图像和N个RGB_DL图像之间的对应关系，获取对应的两个图像之间的相似度，以得到N个相似度。计算机电可基于N个相似度，获取ISP处理的RGB_ISP图像与代理网络处理的RGB_DL图像之间的整体相似度。
其中，N个RGB_ISP图像和N个RGB_DL图像之间的对应关系是指基于同一个Raw图像和同一个FW参数处理后得到的RGB图像。例如，计算节点基于RGB_ISP图像1和RGB_DL图像1，获取对应的两个图像之间的相似度。其中，RGB_ISP图像1和RGB_DL图像1分别是ISP和代理网络基于Raw图像1和FW参数1获取到的。
下面以具体示例对两个图像之间的相似度获取方式进行详细说明。示例性的，如上文所述，每个图像为一个数组。以RGB_ISP图像1和RGB_DL图像1为例，RGB_ISP图像1记为Iout_ISP，RGB_DL图像1即为Iout_DL，Raw图像1记为IRaw，FW参数即为p。
示例性的，计算节点计算Iout_ISP与Iout_DL之间的差值(也可以称为误差、相似度差值等，本申请不做限定)，记为L_1。具体计算方式可参照已有技术实施例，本申请不做限定。
示例性的，计算节点可基于上述方式，获取到N个RGB_ISP图像和N个RGB_DL图像之间两两图像之间的相似度差值，以得到N个相似度差值，例如L_1……L_N。计算节点计算N个相似度差值的平均值，即为ISP处理的RGB_ISP图像与代理网络处理的RGB_DL图像之间的相似度值，记为L。可选地，在其他实施例中，也可以使用算术平均等计算方式求取相似度值，本申请不做限定。
示例性的，计算节点检测L是否大于(或等于)预设阈值。预设阈值可根据实际需求设置，本申请不做限定。预设阈值用于指示图像之间的相似度程度，预设阈值越小，则两个图像之间的相似度越大。相应的，如果L越小，则两个图像之间的相似度越高。
一个示例中，若计算节点检测到L值小于阈值，也可以理解为是相似度大于阈值。则执行S704。
另一个示例中，若计算节点检测到L值大于或等于阈值，也可以理解是相似度小于阈值，即代理网络输出的图像与ISP输出的图像之间的差距较大，则执行S705。
S704，保存代理网络权重。
示例性的，在代理网络输出的图像与ISP输出的图像差异较小的情况下，即可认为代理网络的处理能力是接近或与ISP相同的。计算节点可保存当前的代理网络权重，并锁定代理网络权重，避免代理网络权重随着后续的处理变换。
S705，更新代理网络权重。
示例性的，在代理网络输出的图像与ISP输出的图像之间的差异较大的情况下，则需要重新调整代理网络的权重，以使得代理网络基于新的权重重新执行S702。
计算节点利用梯度下降算法等优化算法，基于代理网络当前的权重(记为wt)，计算对应的梯度值。并根据梯度值，对当前的代理网络权重(wt)进行梯度下降更新，以得到更新后的代理网络权重，记为wt+1。梯度下降算法的具体计算方法可参照已有技术实施例中的相关内容，本申请不再赘述。需要说明的是，在其他实施例中，也可以利用其它算法，本申请不做限定。
示例性的，计算节点获取到的更新后的权重(wt+1)后，代理网络可以重复执行S702。其中，在重新执行S702的过程中，代理网络是基于更新后的权重(wt+1)对训练FW参数以及训练Raw图像进行处理的。具体处理方式可参照上文实施例中的相关内容，此处不再赘述。计算节点获取到代理网络的输出结果后，可再次将代理网络的输出结果与ISP的输出结果(S701只需要执行一次)进行比较，并基于比较结果，执行S704或S705，循环的次数取决于S703中的L值的大小，即代理网络的输出结果与ISP的输出结果的图像之间的相似度，在任意一次循环中，计算节点检测到相似度大于阈值，即L值小于阈值，则保存代理网络权重，训练结束。
2)FW参数预测网络训练阶段
本申请实施例中，FW参数预测网络是一个部署于AI处理器上的深度网络，其按需从存储(例如DDR)中读取当前摄像头采集到的视频流中待处理的Raw图像数据，并进行最佳FW参数预测(也可以理解为是网络推理)。FW参数预测网络将预测出的参数(一组长度为N的向量，N为可动态配置的FW参数个数)写入ISP Firmware的共享内存中，覆盖原有的FW参数。
本申请实施例中对FW参数预测网络的训练，可以理解为是使得FW参数预测网络的输出结果逐渐准确的过程。如上文所述，FW参数预测网络与代理网络相同，其同样是设置有初始权重，在FW参数预测网络训练的过程中，与代理网络的训练类似，其过程中是通过对FW参数预测网络权重进行不断的更新，从而使得FW参数预测网络的输出结果逐渐优化输出的FW参数。
图8为示例性示出的FW参数预测网络训练流程示意图。请参照图8，具体包括但不限于：
S801，FW参数预测网络基于Raw图像，获取预测FW参数。
示例性的，计算节点可以向FW参数预测网络输入N个Raw图像(即准备阶段中的N个Raw图像)，FW参数预测网络可基于当前的权重(可以是初始权重，也可能是更新后的权重)，对输入的N个Raw图像进行处理(或运算)，得到N个预测FW参数。
需要说明的是，在本申请实施例中，以FW参数预测网络训练阶段的输入样本，即N个Raw图像与代理网络训练阶段的输入样本相同为例进行说明。在其他实施例中，FW参数预存网络训练阶段的输入样本可以是N个Raw图像中的M个图像。当然，也可以是与代理网络训练阶段不相同的其它M个Raw图像，本申请不做限定。
一种可能的实现方式中，计算节点可以根据FW参数预测网络的处理能力，分批次向FW参数预测网络输入N个Raw图像。每次输入的Raw图像的数量可以相同或不同。例如，在本申请实施例中，计算节点可以每次向FW参数预测网络输入20个Raw图像。数值仅为示意性举例，可根据实际需求设置，本申请不做限定。
示例性的，FW参数预测网络输出的N个预测FW参数是与输入的Raw图像一一对应的。例如， FW参数预测网络可基于Raw图像1，输出预测FW参数1。基于Raw图像2，输出预测FW参数2。
与代理网络类似，FW参数预测网络设置有初始权重，初始权重是操作人员任意设置的。FW参数预测网络在初始阶段(例如第一次或前几次训练过程)所输出的结果(即数组)可能是毫无意义的，但是在本申请实施例中仍将FW参数预测网络输出的结果称为预测FW参数。相应的，本申请实施例中图8所示的流程即可理解为是通过不断地对FW参数预测网络进行训练，以更新FW参数预测网络权重，以不断优化FW参数预测网络输出的结果。
S802，代理网络基于预测FW参数和Raw图像，得到RGB图像。
示例性的，计算节点获取到FW参数预测网络输出的N个预测FW参数后，可向代理网络输入N个Raw图像和N个预测FW参数。其中，N个Raw图像与N个预测FW参数是一一对应的。其对应关系即为S801中所述的对应关系。
代理网络可基于当前的权重(即为图7中经过训练后获取到的权重)，按照N个Raw图像与N个预测FW参数之间的对应关系，对N个Raw图像进行处理(或运算)，得到N个RGB图像。例如，代理网络可基于当前的权重，按照Raw图像1与预测FW参数1之间的对应关系(即预测FW参数1是FW参数预测网络基于Raw图像1获取到的)，将预测FW参数1和Raw图像1作为一对输入参数进行处理(或运算)，得到RGB图像1。示例性的，计算机可以获取到Raw图像1、预测FW参数1以及RGB图像1之间的对应关系。
需要说明的是，如上文所述，图7中已经对代理网络完成训练，代理网络基于当前的权重对Raw图像进行处理后所得到的结果(即RGB图像)是与指定型号的ISP所处理的结果相同或接近(即相似)的。也就是说，在S802中，代理网络可看做是ISP，以对Raw图像进行图像处理，得到对应的RGB图像。而与ISP不同的时，代理网络具有可微分性质，以实现FW预测网络与CV感知网络之间的端到端联合训练。
在一种可能的实现方式中，计算节点可以分批次向代理网络输入N个Raw图像和N个FW预测参数，具体实现方式可参照上文，此处不再赘述。
S803，CV感知网络基于RGB图像，得到感知任务标签。
示例性的，如上文所述，准确阶段时，操作人员预先设置有CV感知网络。CV感知网络的作用可以是基于输入的Raw图像，输出对应的感知任务标签。感知任务标签的数量以及类型，取决于CV感知网络的任务类型。例如，CV感知网络的任务类型是识别物体类型，则CV感知网络可基于输入的Raw图像，输出对应的感知任务标签，感知任务标签中包括物体类型标签。
在本申请实施例中，计算节点获取到代理网络输出的N个RGB图像后，可将N个RGB图像输入(可以是分批次的，具体处理方式可参照上文，此处不再赘述)到CV感知网络中。CV感知网络可对N个RGB图像进行处理(或识别)，得到N个感知任务标签。其中，N个RGB图像与N个感知任务标签是一一对应的。可选地，每个感知任务标签可以包括多个子任务标签，子任务标签的类型和数量与CV感知网络的任务类型相关联，具体描述可参照上文，此处不再赘述。
示例性的，计算节点在本步骤中可以获取到N个RGB图像与N个感知任务标签之间的对应关系。在S802中，计算节点还获取到N个RGB图像与N个Raw图像之间的对应关系。相应的，计算节点可获取到N个感知任务标签与N个Raw图像之间的对应关系。例如，感知任务标签1与Raw图像1对应，感知任务标签2与Raw图像2对应等。
示例性的，如上文所述，在准备阶段，操作人员针对N个Raw图像中的每个Raw图像设置有真值标签。计算节点可预先获取到N个真值标签与N个Raw图像之间的对应关系。
相应的，在本申请实施例中，计算节点可基于N个感知任务标签与N个Raw图像之间的对应关系，以及，N个真值标签与N个Raw图像之间的对应关系，获取到N个感知任务标签与N个真值标签的对应关系(对应关系为一一对应)。例如，感知任务标签1与真值标签1对应，感知任务标签2与真值标签2对应。
S804，判断真值标签与感知任务标签之间的差异是否小于阈值。
示例性的，如上文所述，计算节点可获取到N个感知任务标签与N个真值标签的对应关系。计 算节点可基于N个感知任务标签与N个真值标签的对应关系，获取对应的两个感知任务标签和真值标签之间的差值(也可以称为差异值、损失函数等，本申请不做限定)，以得到N个差值。计算节点获取真值标签与感知任务标签之间的差值的方式可参照已有技术实施例，本申请不做限定。
示例性的，计算节点可获取N个差值的平均值(也可以是算术平均，本申请不做限定)，记为Q。
示例性的，计算节点检测Q是否大于(或等于)预设阈值。预设阈值可根据实际需求设置，本申请不做限定。预设阈值用于指示两个标签之间的差异程度。其中，预设阈值越小，则两个标签之间的差异越小。相应的，如果Q值越小，则两个标签之间的差异越小。
一个示例中，若计算节点检测到Q值小于阈值，也可以理解为是差异度小于阈值。则执行S704。
另一个示例中，若计算节点检测到Q值大于或等于阈值，也可以理解是差异度大于阈值，即CV感知网络基于代理网络输出的RGB图像输出的感知任务标签与真值标签之间的差异较大，也可以理解为是代理网络基于预测FW参数得到的RGB图像的增强效果较低，使得CV感知网络识别到的结果不准确，还可以理解为是，预测FW参数对Raw图像的修正无法达到CV感知网络能够识别到的增强效果。相应的，计算节点执行S806。
S805，保存FW参数预测网络。
示例性的，在CV感知网络输出的感知任务标签与真值任务标签之间的差异较小(即小于阈值)的情况下，即可认为FW参数预测网络所得到的预测FW参数对Raw图像的增强效果是满足CV感知网路的需求的。相应的，计算节点可保存FW参数预测网络，即保存当前的FW参数预测网络的结构(例如神经网络结构)以及FW参数预测网络当前的权重，记为wf。
S806，更新FW参数预测网络权重。
示例性的，在CV感知网络输出的感知任务标签与真值任务标签之间的差异较大(即大于阈值)的情况下，则需要重新调整FW参数预测网络的权重，以使得FW参数预测网络基于更新后的权重重新执行S801。
示例性的，计算节点利用梯度下降算法等优化算法(可根据实际需求设置，本申请不做限定)，基于FW参数预测网络当前的权重(wf)，计算对应的梯度值。并根据梯度值，对当前的FW参数预测网络权重(wf)进行梯度下降更新，以得到更新后的FW参数预测网络权重，记为wf+1。梯度下降算法的具体计算方法可参照已有技术实施例中的相关内容，本申请不再赘述。需要说明的是，在其他实施例中，也可以利用其它算法，本申请不做限定。
示例性的，计算节点获取到的更新后的权重(wf+1)后，FW参数预测网络可以重复执行S801。其中，在重新执行S801的过程中，FW参数预测网络是基于更新后的权重(wf+1)对Raw图像进行处理，以得到对应的预测FW参数的。具体处理方式可参照上文实施例中的相关内容，此处不再赘述。示例性的，图8中的流程的循环次数取决于S804中的Q值的大小，即感知任务标签与真值标签之间的差异度。在任意一次循环中，若Q值小于阈值，则计算节点保存FW参数预测网络以及对应的FW参数预测网络权重，训练结束。
结合图8，图9为示例性示出的FW参数预测网络训练的系统结构示意图。请参照图9，示例性的，计算节点将Raw图像输入到FW参数预测网络。FW参数预测网络基于输入的Raw图像，输出预测FW参数。预测FW参数和Raw图像作为代理网络的输入，代理网络输出RGB图像。RGB图像作为CV感知网络的输入，计算节点可基于CV感知网络的输出，得到Q值(即差异值或损失函数)。在Q值不满足条件，即大于阈值的情况下，系统将重新返回到起始位置重复执行图9中的流程，直至Q值小于阈值。
请继续参照图5，在本申请实施例中，图像处理方法还包括在线部署阶段。其中，在线部署阶段 是将图8中保存的FW参数预测网络及其权重部署到目标设备中。其中，目标设备即为训练过程中ISP所对应的设备。例如，ISP为A型号的电子设备，则操作人员可将计算节点保存的FW参数预测网络及其权重部署到A型号的电子设备上。当然，A型号的电子设备也可以通过云端获取并安装FW参数预测网络及其权重，本申请不做限定。
示例性的，目标设备将FW参数预测网络及其权重加载到内存中，作为AI处理器(例如NPU)的一个常驻进程工作于操作系统中。也可以理解为是图2中的FW参数设置模块(也可以称为FW参数预测模块)，或者是运行于应用层的FW参数设置(预测)应用。
图10为示例性示出的图像处理流程示意图。请参照图10，示例性的，相机应用启动后，可向AI处理器发送触发信号，以调用AI处理器上的FW参数设置模块。其中，FW参数设置模块中包括FW参数预测网络。示例性的，相机应用启动后，同时会调用摄像头，摄像头响应于相机应用的调用，开始采集图像。并将采集到的图像置于共享内存中。FW参数设置模块响应于相机应用的调用，从共享内存中读取Raw图像。本申请实施例中仅以共享内存为例进行说明，在其他实施例中，也可以是其它存储，本申请不做限定。
示例性的，FW参数设置模块中的FW参数预测网络可基于FW参数预测网络权重，对输入的Raw图像进行处理，预测出对应的FW参数。FW参数设置模块将FW参数写入到ISP FW(即ISP软件部分)所对应的ISP内存中。ISP内存可以是共享内存中的一部分，也可以位于其它存储，本申请不做限定。示例性的，ISP内存中可能已保存有FW参数，已保存的FW参数可能是初始的FW参数，也可能是FW参数设置模块上一次写入的，本申请不做限定。相应的，FW参数设置模块可将ISP内存中当前保存的FW参数删除，并写入新预测到的FW参数。
ISP FW可从ISP内存中读取到FW参数(即新写入的FW参数)。ISP FW可将FW参数指示给ISP HW。例如，ISP FW可以修改FW HW的寄存器状态，以将FW参数写入到FW HW的寄存器中。ISP HW从共享内存中读取Raw图像，并且基于获取到的FW参数(例如是从寄存器中读取到的)，对Raw图像进行图像处理，得到RGB图像。ISP HW可将获取到RGB图像输出到相机应用。相机应用可在电子设备的显示屏中显示获取到的RGB图像。
在一种可能实现方式中，以自动驾驶应用为例，自动驾驶应用获取到ISP HW输出的RGB图像后，可以基于应用的CV感知网络，对RGB图像进行识别，以得到识别结果，并基于识别结果，执行自动驾驶操作。
在另一种可能的实现方式中，FW参数设置模块被应用调用之后，FW参数设置模块中的FW参数预测网络可以对摄像头采集到的每一帧的Raw图像进行预测，即针对每个Raw图像，执行图10中的流程。
在又一种可能的实现方式中，FW参数设置模块被应用调用之后，FW参数设置模块中的FW参数预测网络可以按照预设周期(例如可以是根据间隔时长，也可以是根据摄像头采集图像的帧率设置，可根据实际需求设置，本申请不做限定)，对摄像头采集的Raw图像进行预测。例如，FW参数设置模块可以每隔1s进行一次预测，以降低AI处理器的工作负载。
在又一种可能的实现方式中，FW参数设置模块被应用调用后可工作在休眠模式，当应用层向其发送唤醒信号时才进行一次预测，例如，为降低AI处理器的工作负载，仅当CPU(例如自动驾驶应用)检测到行车环境发生显著变化(如进出隧道、车库，昼夜切换等)时，自动驾驶应用可向FW参数设置模块发送指示信息，以触发其进行FW参数预测，而该参数预测可以只进行一次。
在又一种可能的实现方式中，FW参数设置模块也可以在电子设备启动后触发。
在又一种可能的实现方式中，为了降低AI处理器的负载，FW参数设置模块的输入可以是下采样后具有较低分辨率的Raw图像。其中，下采样操作可以是由AI处理器执行的，也可以是由ISP HW执行的，本申请不做限定。例如，ISP HW可以从共享内存中获取原始Raw图像，并将Raw图像进行下采样后，将较低分辨率的Raw图像输出到共享内存中。FW参数设置模块可以从共享内存中读取具有较低分辨率的Raw图像并进行后续处理。可选地，FW参数设置模块从共享内存中读取到的Raw图像也可以是原始Raw图像经过其他编码或压缩得到的，本申请不做限定。
示例性的，图11示出了本申请实施例的一种装置1100的示意性框图装置1100可包括：处理器1101和收发器/收发管脚1102，可选地，还包括存储器1103。
装置1100的各个组件通过总线1104耦合在一起，其中总线1104除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图中将各种总线都称为总线1104。
可选地，存储器1103可以用于前述方法实施例中的指令。该处理器1101可用于执行存储器1103中的指令，并控制接收管脚接收信号，以及控制发送管脚发送信号。
装置1100可以是上述方法实施例中的电子设备、摄像头模组或电子设备的芯片。
其中，上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。
本实施例还提供一种计算机存储介质，该计算机存储介质中存储有计算机指令，当该计算机指令在电子设备上运行时，使得电子设备执行上述相关方法步骤实现上述实施例中的方法。
本实施例还提供了一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述相关步骤，以实现上述实施例中的方法。
另外，本申请的实施例还提供一种装置，这个装置具体可以是芯片，摄像头组件(或摄像头模组)，该装置可包括相连的处理器和存储器；其中，存储器用于存储计算机执行指令，当装置运行时，处理器可执行存储器存储的计算机执行指令，以使芯片执行上述各方法实施例中的方法。
其中，本实施例提供的电子设备、计算机存储介质、计算机程序产品或芯片均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
通过以上实施方式的描述，所属领域的技术人员可以了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
本申请各个实施例的任意内容，以及同一实施例的任意内容，均可以自由组合。对上述内容的任意组合均在本申请的范围之内。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。
结合本申请实施例公开内容所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(Random Access Memory，RAM)、闪存、只读存储器(Read Only Memory，ROM)、可擦除可编程只读存储器(Erasable Programmable ROM，EPROM)、电可擦可编程只读存储器(Electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本申请实施例所描述的功能可以用硬件、软件、固件或它们的任意组合来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。计算机可读介质包括计算机存储介质和通信介质，其中通信介质包括便于从一个地方向另一个地方传送计算机程序的任何介质。存储介质可以是通用或专用计算机能够存取的任何可用介质。
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。

Claims (23)

  1. 一种图像处理方法，其特征在于，包括：
    获取摄像头在第一场景下采集到的第一原始图像；
    基于所述第一原始图像，获取第一图像信号处理器(Image Signal Processing,ISP)参数；
    基于所述第一ISP参数，对所述第一原始图像进行图像处理，得到第一图像；
    将所述第一图像输出至目标应用，使得所述目标应用对所述第一图像进行图像任务处理；
    获取所述摄像头在第二场景下采集到的第二原始图像；
    基于所述第二原始图像，获取第二ISP参数；
    基于所述第二ISP参数，对所述第二原始图像进行图像处理，得到第二图像；
    将所述第二图像输出至所述目标应用，使得所述目标应用对所述第二图像进行图像任务处理。
  2. 根据权利要求1所述的方法，其特征在于，所述第一原始图像与所述第二原始图像的图像内容不同，和/或，所述第一原始图像与所述第二原始图像的图像属性不同。
  3. 根据权利要求2所述的方法，其特征在于，所述第一ISP参数与所述第二ISP参数用于调整图像的以下至少一种图像属性：
    亮度、颜色、噪声、锐度、对比度。
  4. 根据权利要求2所述的方法，其特征在于，所述第一图像与所述第二图像的图像属性满足所述目标应用的图像任务需求。
  5. 根据权利要求1所述的方法，其特征在于，所述基于所述第一原始图像，获取第一ISP参数，包括：
    将所述第一原始图像输入至ISP参数预测模型，获取所述ISP参数预测模型基于所述第一原始图像预测出的所述第一ISP参数；
    所述基于所述第二原始图像，获取第二ISP参数，包括：
    将所述第二原始图像输入至所述ISP参数预测模型，获取所述ISP参数预测模型基于所述第二原始图像预测出的所述第二ISP参数。
  6. 根据权利要求5所述的方法，其特征在于，所述ISP参数预测模型运行在神经网路处理器NPU上。
  7. 根据权利要求5所述的方法，其特征在于，所述获取摄像头在第一场景下采集到的第一原始图像之前，所述方法还包括：
    从云端获取所述ISP参数预测模型。
  8. 根据权利要求1至7任一项所述的方法，其特征在于，
    所述基于所述第一ISP参数，对所述第一原始图像进行图像处理，得到第一图像，包括：
    将ISP内存中当前保存的ISP参数替换为所述第一ISP参数，使得ISP从所述ISP内存中获取所述第一ISP参数，并基于所述第一ISP参数，对所述第一原始图像进行图像处理，输出所述第一图像；
    所述基于所述第二ISP参数，对所述第二原始图像进行图像处理，得到第二图像，包括：
    将所述ISP内存中当前保存的所述第一ISP参数替换为所述第二ISP参数，使得所述ISP从所述ISP内存中获取所述第二ISP参数，并基于所述第二ISP参数，对所述第二原始图像进行图像处理，输出所述第二图像。
  9. 一种模型训练方法，其特征在于，包括：
    向图像信号处理器ISP参数预测模型输入N个原始图像，获取所述ISP参数预测模型输出的N个ISP参数，所述ISP参数预测模型的权重为第一权重，所述N个ISP参数为所述ISP参数预测模型基于所述第一权重值得到的，N为大于1的整数；
    向代理模型输入所述N个ISP参数与所述N个原始图像，获取所述代理模型输出的N个图像；
    向目标应用输入所述N个图像，获取所述目标应用输出的图像任务处理结果；
    若所述图像任务处理结果未满足预设条件，调整所述ISP参数预测模型的权重为第二权重，再次从所述向ISP参数预测模型输入N个原始图像的步骤开始执行，直至所述目标应用输出的图像任务处理结果满足所述预设条件。
  10. 根据权利要求9所述的方法，其特征在于，所述预设条件包括：
    所述N个原始图像的预设真值标签与所述目标应用的图像任务处理结果中包括的真值标签之间的差值小于预设阈值。
  11. 根据权利要求9所述的方法，其特征在于，所述向ISP参数预测模型输入N个原始图像之前，方法还包括：
    向ISP输入M个原始图像和M个ISP参数，获取所述图像处理器输出的M个图像；其中，M个原始图像和M个ISP参数一一对应，M为大于1的整数；
    向所述代理模型输入M个原始图像和M个ISP参数，获取所述代理模型输出的M个图像；所述代理模型的当前权重为第三权重；
    若所述代理模型输出的M个图像与所述图像处理器输出的M个图像之间的相似度小于阈值，调整所述代理模型的权重为第四权重，再次从所述向ISP输入M个原始图像和M个ISP参数的步骤开始执行，直至所述代理模型输出的M个图像与所述图像处理器输出的M个图像之间的相似度大于阈值。
  12. 一种图像处理装置，其特征在于，包括：
    图像信号处理器ISP参数预测模块，用于获取摄像头在第一场景下采集到的第一原始图像；
    所述ISP参数预测模块，还用于基于所述第一原始图像，获取第一图像信号处理器ISP参数；
    图像处理ISP模块，用于基于所述第一ISP参数，对所述第一原始图像进行图像处理，得到第一图像；
    所述ISP模块，还用于将所述第一图像输出至目标应用，使得所述目标应用对所述第一图像进行图像任务处理；
    所述ISP参数预测模块，还用于获取所述摄像头在第二场景下采集到的第二原始图像；
    所述ISP参数预测模块，还用于基于所述第二原始图像，获取第二ISP参数；
    所述ISP模块，还用于基于所述第二ISP参数，对所述第二原始图像进行图像处理，得到第二图像；
    所述ISP模块，还用于将所述第二图像输出至所述目标应用，使得所述目标应用对所述第二图像进行图像任务处理。
  13. 根据权利要求12所述的装置，其特征在于，所述第一原始图像与所述第二原始图像的图像内容不同，和/或，所述第一原始图像与所述第二原始图像的图像属性不同。
  14. 根据权利要求13所述的装置，其特征在于，所述第一ISP参数与所述第二ISP参数用于调整图像的以下至少一种图像属性：
    亮度、颜色、噪声、锐度、对比度。
  15. 根据权利要求13所述的装置，其特征在于，所述第一图像与所述第二图像的图像属性满足所述目标应用的图像任务需求。
  16. 根据权利要求12至15任一项所述的装置，其特征在于，
    所述ISP参数预测模块，用于将ISP内存中当前保存的ISP参数替换为所述第一ISP参数；
    所述ISP模块，用于从所述ISP内存中获取所述第一ISP参数，并基于所述第一ISP参数，对所述第一原始图像进行图像处理，输出所述第一图像；
    所述ISP参数预测模块，用于将所述ISP内存中当前保存的所述第一ISP参数替换为所述第二ISP参数；
    所述ISP模块，用于从所述ISP内存中获取所述第二ISP参数，并基于所述第二ISP参数，对所述第二原始图像进行图像处理，输出所述第二图像。
  17. 一种模型训练系统，其特征在于，包括图像信号处理器ISP参数预测模型、代理模型和目标应用；
    向所述ISP参数预测模型输入N个原始图像，获取所述ISP参数预测模型输出的N个ISP参数，所述ISP参数预测模型的权重为第一权重，所述N个ISP参数为所述ISP参数预测模型基于所述第一权重值得到的，N为大于1的整数；
    向所述代理模型输入所述N个ISP参数与所述N个原始图像，获取所述代理模型输出的N个图像；
    向所述目标应用输入所述N个图像；
    若所述N个图像未满足所述目标应用的图像任务需求，调整所述ISP参数预测模型的权重为第二权重，再次从所述向ISP参数预测模型输入N个原始图像的步骤开始执行，直至所述代理模型输出的N个图像满足所述目标应用的图像任务需求。
  18. 根据权利要求17所述的系统，其特征在于，所述系统还包括ISP；
    所述向ISP参数预测模型输入N个原始图像之前，向所述ISP输入M个原始图像和M个ISP参数，获取所述图像处理器输出的M个图像；其中，M个原始图像和M个ISP参数一一对应，M为大于1的整数；
    向所述代理模型输入M个原始图像和M个ISP参数，获取所述代理模型输出的M个图像；所述代理模型的当前权重为第三权重；
    若所述代理模型输出的M个图像与所述图像处理器输出的M个图像之间的相似度小于阈值，调整所述代理模型的权重为第四权重，所述第四权重与所述第三权重不相同，再次从所述向图像处理器输入M个原始图像和M个ISP参数，获取所述图像处理器输出的M个图像开始执行，直至所述代理模型输出的M个图像与所述图像处理器输出的M个图像之间的相似度大于阈值。
  19. 一种电子设备，其特征在于，包括：
    一个或多个处理器；
    存储器；
    以及一个或多个计算机程序，其中所述一个或多个计算机程序存储在所述存储器上，当所述计算机程序被所述一个或多个处理器执行时，使得所述电子设备执行权利要求1-8任一项所述的方法。
  20. 一种摄像模组，其特征在于，包括：
    一个或多个处理器；
    存储器；
    以及一个或多个计算机程序，其中所述一个或多个计算机程序存储在所述存储器上，当所述计算机程序被所述一个或多个处理器执行时，使得所述摄像模组执行权利要求1-8任一项所述的方法。
  21. 一种芯片，其特征在于，包括一个或多个接口电路和一个或多个处理器；所述接口电路用于从电子设备的存储器接收信号，并向所述处理器发送所述信号，所述信号包括存储器中存储的计算机指令；当所述处理器执行所述计算机指令时，使得所述电子设备执行权利要求1-8任一项所述的方法。
  22. 一种计算机存储介质，其特征在于，包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1-8任一项所述的方法。
  23. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得所述计算机执行如权利要求1-8任一项所述的方法。
PCT/CN2023/110156 2022-09-14 2023-07-31 图像处理方法及装置 Ceased WO2024055764A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23864498.3A EP4576018A4 (en) 2022-09-14 2023-07-31 IMAGE PROCESSING METHOD AND APPARATUS
US19/078,875 US20250247629A1 (en) 2022-09-14 2025-03-13 Image Processing Method and Apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211115044.XA CN117726929A (zh) 2022-09-14 2022-09-14 图像处理方法及装置
CN202211115044.X 2022-09-14

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/078,875 Continuation US20250247629A1 (en) 2022-09-14 2025-03-13 Image Processing Method and Apparatus

Publications (1)

Publication Number Publication Date
WO2024055764A1 true WO2024055764A1 (zh) 2024-03-21

Family

ID=90203939

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/110156 Ceased WO2024055764A1 (zh) 2022-09-14 2023-07-31 图像处理方法及装置

Country Status (4)

Country Link
US (1) US20250247629A1 (zh)
EP (1) EP4576018A4 (zh)
CN (1) CN117726929A (zh)
WO (1) WO2024055764A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119277026A (zh) * 2024-10-21 2025-01-07 南京优宇旭科技有限公司 固定交通环境视觉内容管理系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118175433A (zh) * 2024-05-13 2024-06-11 成都云创天下科技有限公司 一种基于同一视频画面内不同场景isp自动调优的方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108632512A (zh) * 2018-05-17 2018-10-09 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
WO2021204202A1 (zh) * 2020-04-10 2021-10-14 华为技术有限公司 图像自动白平衡的方法及装置
CN114615495A (zh) * 2020-12-09 2022-06-10 Oppo广东移动通信有限公司 模型量化方法、装置、终端及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111656781A (zh) * 2018-01-30 2020-09-11 高通股份有限公司 用于使用参考图像进行图像信号处理器调谐的系统和方法
US10796200B2 (en) * 2018-04-27 2020-10-06 Intel Corporation Training image signal processors using intermediate loss functions
CN114677719A (zh) * 2020-12-09 2022-06-28 华为技术有限公司 图像信号处理的方法、装置和计算机可读存储介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108632512A (zh) * 2018-05-17 2018-10-09 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
WO2021204202A1 (zh) * 2020-04-10 2021-10-14 华为技术有限公司 图像自动白平衡的方法及装置
CN114615495A (zh) * 2020-12-09 2022-06-10 Oppo广东移动通信有限公司 模型量化方法、装置、终端及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4576018A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119277026A (zh) * 2024-10-21 2025-01-07 南京优宇旭科技有限公司 固定交通环境视觉内容管理系统

Also Published As

Publication number Publication date
EP4576018A1 (en) 2025-06-25
US20250247629A1 (en) 2025-07-31
EP4576018A4 (en) 2025-11-26
CN117726929A (zh) 2024-03-19

Similar Documents

Publication Publication Date Title
EP4064284A1 (en) Voice detection method, prediction model training method, apparatus, device, and medium
WO2021136050A1 (zh) 一种图像拍摄方法及相关装置
CN114242037B (zh) 一种虚拟人物生成方法及其装置
CN115798390B (zh) 屏幕显示方法和终端设备
WO2022179604A1 (zh) 一种分割图置信度确定方法及装置
WO2021258814A1 (zh) 视频合成方法、装置、电子设备及存储介质
WO2021104485A1 (zh) 一种拍摄方法及电子设备
US20250247629A1 (en) Image Processing Method and Apparatus
WO2021057626A1 (zh) 图像处理方法、装置、设备及计算机存储介质
CN115589051B (zh) 充电方法和终端设备
CN111768765B (zh) 语言模型生成方法和电子设备
WO2023179123A1 (zh) 蓝牙音频播放方法、电子设备及存储介质
CN115705241B (zh) 应用的调度方法及电子设备
CN114444000A (zh) 页面布局文件的生成方法、装置、电子设备以及可读存储介质
WO2025036044A1 (zh) 资源调度的方法、装置和电子设备
CN111835904A (zh) 一种基于情景感知和用户画像开启应用的方法及电子设备
WO2023207667A1 (zh) 一种显示方法、汽车和电子设备
CN113380240B (zh) 语音交互方法和电子设备
CN110058729A (zh) 调节触摸检测的灵敏度的方法和电子设备
CN117995137B (zh) 一种调节显示屏色温的方法、电子设备及相关介质
CN115964231A (zh) 基于负载模型的评估方法和装置
CN116489272A (zh) 语音消息播放方法及电子设备
CN114115772B (zh) 灭屏显示的方法及装置
WO2023098467A1 (zh) 语音解析方法、电子设备、可读存储介质及芯片系统
CN112770002A (zh) 一种心跳管控的方法和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23864498

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023864498

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023864498

Country of ref document: EP

Effective date: 20250319

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 2023864498

Country of ref document: EP