WO2025062831A1 - Dispositif de traitement d'image, procédé de traitement d'image et programme - Google Patents

Dispositif de traitement d'image, procédé de traitement d'image et programme Download PDF

Info

Publication number
WO2025062831A1
WO2025062831A1 PCT/JP2024/026801 JP2024026801W WO2025062831A1 WO 2025062831 A1 WO2025062831 A1 WO 2025062831A1 JP 2024026801 W JP2024026801 W JP 2024026801W WO 2025062831 A1 WO2025062831 A1 WO 2025062831A1
Authority
WO
WIPO (PCT)
Prior art keywords
color
image
unit
processing
space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2024/026801
Other languages
English (en)
Japanese (ja)
Inventor
将嗣 福永
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Original Assignee
Sony Semiconductor Solutions Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp filed Critical Sony Semiconductor Solutions Corp
Publication of WO2025062831A1 publication Critical patent/WO2025062831A1/fr
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/12—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with one sensor only
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/85—Camera processing pipelines; Components thereof for processing colour signals for matrixing

Definitions

  • This technology relates to an image processing device. More specifically, it relates to an image processing device that processes images in various color spaces, an image processing method in the image processing device, and a program for causing a computer to execute the method.
  • the above-mentioned conventional technology aims to improve the quality of Bayer array images by correcting the sensitivity difference.
  • image processing device it becomes difficult to improve image quality when performing processes such as white balance correction on images whose color space is not RGB space.
  • the color space is not RGB space
  • color reproducibility cannot be maintained unless the gain during white balance correction is increased, but if the gain is large, noise increases.
  • reducing the gain reduces noise, but color reproducibility decreases.
  • problems such as a trade-off between improving color reproducibility and reducing noise arise, making it difficult to improve image quality.
  • This technology was developed in light of these circumstances, and aims to improve image quality in image processing devices that process images.
  • the present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image processing device including a color base conversion unit that performs color base conversion processing on an image to convert the color space from a first color space to a second color space, and an image quality adjustment processing unit that performs a predetermined image quality adjustment processing on the image after the color base conversion processing, as well as an image processing method in the image processing device and a program for causing a computer to execute the method. This brings about the effect of improving image quality.
  • the image processing device may further include a demosaic processing unit that converts a RAW image into a demosaic image, and the image may be the RAW image or the demosaic image. This has the effect of improving the image quality of the demosaic image.
  • the system may further include an AWB unit that estimates a white balance gain, the color base conversion unit performs the color base conversion process on the detection value of the RAW image and supplies it to the AWB unit, and the image quality adjustment processing unit may include a color reproduction unit that amplifies the color signals in the demosaic image using the white balance gain and corrects them using a color reproduction matrix. This improves color reproducibility and reduces noise.
  • the first color space may be a color space different from the RGB space
  • the second color space may be the RGB space. This provides the effect of converting from a space other than the RGB space to the RGB space.
  • a pixel array unit that generates the image may be further provided. This provides the effect of processing being performed within the image sensor.
  • a plurality of pixels including clear pixels may be arranged in the pixel array section, and the clear pixels may be provided with transparent filters. This provides the effect of converting the color space of an image including the clear pixels.
  • the image quality adjustment processing unit may execute at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing. This brings about the effect of adjusting the image quality.
  • FIG. 1 is a block diagram showing a configuration example of an imaging device according to a first embodiment of the present technology
  • 1 is a block diagram showing a configuration example of an image sensor according to a first embodiment of the present technology
  • FIG. 1 is a block diagram showing an example of the configuration of an image sensor according to a conventional technique.
  • FIG. 1 is a diagram showing an example of processing up to color reproduction in the prior art.
  • FIG. 1 is a diagram showing details of processing up to color reproduction in the prior art.
  • FIG. 1 is a diagram showing an example of a Bayer array in the prior art.
  • 1 is a graph showing an example of spectral characteristics in the prior art.
  • 1 is an example of a factor graph in the prior art.
  • 1 is an example of a constrained color reproduction matrix in the prior art.
  • FIG. 4 is a diagram showing an example of a plot of detection values according to the first embodiment of the present technology;
  • FIG. 4 is a flowchart showing an example of a calibration method according to the first embodiment of the present technology.
  • 5 is a diagram illustrating an example of a method for calculating correction matrix coefficients in the first embodiment of the present technology.
  • FIG. 4A to 4C are diagrams illustrating an example of a white balance gain and a color correction matrix according to the first embodiment of the present technology.
  • FIG. 11 is a block diagram showing a configuration example of an image sensor according to a second embodiment of the present technology.
  • FIG. 11 is a diagram illustrating an example of a color base conversion process according to a second embodiment of the present technology.
  • 1 is a block diagram showing a schematic configuration example of a vehicle control system;
  • FIG. 4 is an explanatory diagram showing an example of an installation position of an imaging unit.
  • First embodiment example of performing color basis conversion processing on a demosaic image
  • Second embodiment example of performing color basis transformation processing on detected values of a RAW image
  • First embodiment [Configuration example of imaging device] 1 is a block diagram showing a configuration example of an imaging device 100 according to a first embodiment of the present technology.
  • the imaging device 100 is a device for capturing image data, and includes an optical unit 110, an image sensor 200, and a DSP (Digital Signal Processing) circuit 120.
  • the imaging device 100 further includes a display unit 130, an operation unit 140, a bus 150, a frame memory 160, a storage unit 170, and a power supply unit 180.
  • the imaging device 100 may be a digital camera or an electronic device having an imaging function (such as a smartphone or a personal computer).
  • the imaging device 100 is an example of an image processing device described in the claims.
  • the optical unit 110 collects light from a subject and guides it to the image sensor 200.
  • the image sensor 200 generates image data through photoelectric conversion.
  • the image sensor 200 processes the generated image data and supplies the processed data to the DSP circuit 120 via a signal line 209.
  • the bus 150 is a common path for the optical unit 110, image sensor 200, DSP circuit 120, display unit 130, operation unit 140, frame memory 160, storage unit 170, and power supply unit 180 to exchange data with each other.
  • FIG. 2 is a block diagram showing an example of a configuration of an image sensor 200 according to the first embodiment of the present technology.
  • the image sensor 200 includes a pixel array unit 210, a demosaic processing unit 220, a color basis conversion unit 230, a photometry and AWB unit 241, an amplifier unit 242, a color reproduction unit 250, and a white clip processing unit 260.
  • the image sensor 200 further includes a gamma correction unit 271, an edge enhancement unit 272, a YUV conversion unit 273, a color conversion unit 274, and a DNN (Deep Neural Network) monitor 275.
  • DNN Deep Neural Network
  • some or all of the downstream processing of the pixel array section 210 can be performed by a circuit external to the image sensor 200 (such as the DSP circuit 120).
  • a plurality of pixels 211 are arranged in a two-dimensional grid. Each of the pixels 211 generates a pixel signal by photoelectric conversion.
  • the pixel array section 210 generates a RAW image in a color space different from RGB.
  • a RAW image is an image having one piece of color information for each pixel, and is also called a mosaic image.
  • a plurality of pixels including clear pixels are arranged in an array conforming to the Bayer array.
  • a clear pixel is a pixel provided with a transparent filter whose transmittance is equal to or greater than a predetermined value (such as 0.5) across the entire visible light range.
  • a clear pixel is also called a clarity pixel.
  • a clear pixel is referred to as a C1 pixel, and pixels of different colors other than the clear pixel are referred to as C0 and C2 pixels.
  • an R (Red) pixel and a B (Blue) pixel are used as the C0 and C2 pixels.
  • An R pixel is a pixel provided with a color filter that selectively transmits light in the red wavelength range
  • a B pixel is a pixel provided with a color filter that selectively transmits light in the blue wavelength range.
  • R and G (Green) pixels, or Y (Yellow) and Cy (Cyan) pixels can also be used as C0 and C2 pixels.
  • the G pixel is a pixel equipped with a color filter that selectively transmits the green wavelength range
  • the Y pixel is a pixel equipped with a color filter that selectively transmits the yellow wavelength range.
  • the Cy pixel is a pixel equipped with a color filter that selectively transmits the cyan wavelength range.
  • the pixels C0 to C2 are arranged in an array in which the position of the G pixel in the Bayer array is replaced with the C1 pixel.
  • C0C1C2 space The color space in which these pixels are arranged is hereinafter referred to as the "C0C1C2 space.”
  • C0C1C2 space Within the C0C1C2 space, the one in which the C0 and C2 pixels are R and B pixels is referred to as the "RCB space.”
  • the pixel array unit 210 supplies the RAW image in the C0C1C2 space to the demosaic processing unit 220 via various circuits such as an ADC (Analog to Digital Converter).
  • ADC Analog to Digital Converter
  • the demosaic processing unit 220 converts the RAW image into a demosaic image.
  • a demosaic image is an image in which missing color information among the three colors is supplemented for each pixel.
  • the demosaic processing unit 220 supplies the demosaic image to the color basis conversion unit 230.
  • the color basis conversion unit 230 performs color basis conversion processing on the demosaic image, converting the color space from C0C1C2 space to RGB space using a basis conversion matrix. Details of the basis conversion matrix will be described later.
  • the color basis conversion unit 230 supplies the demosaic image after color basis conversion processing to the photometry and AWB unit 241 and the amplifier unit 242. This converted demosaic image contains three color signals of R, G, and B for each pixel. Note that although the color basis conversion unit 230 converts the C0C1C2 space to the RGB space, the color space to be converted is not limited to the C0C1C2 space.
  • the output timing to the amplifier 242 is delayed relative to the output timing to the photometry and AWB unit 241 in accordance with the processing delay of the photometry and AWB unit 241.
  • a buffer can also be placed between the color base conversion unit 230 and the amplifier 242.
  • the amplifier 242 amplifies the color signals in the demosaic image by a white balance gain.
  • An amplifier that amplifies the corresponding color signal for each color is arranged in the amplifier 242. Since the demosaic image includes three color signals for each pixel, three amplifiers are arranged in the amplifier 242.
  • the amplifier 242 supplies the amplified color signals to the color reproduction unit 250.
  • the color reproduction unit 250 corrects the color signal from the amplifier unit 242 using a color reproduction matrix (CCM).
  • CCM color reproduction matrix
  • This CCM is also called a linear matrix.
  • the color reproduction unit 250 holds in advance a set of correction matrix coefficients in the CCM for each of a number of color temperatures.
  • the color reproduction unit 250 then performs correction using the CCM of the correction matrix coefficients corresponding to the correlated color temperature estimated by the photometry and AWB unit 241, and supplies the corrected demosaic image to the white clip processing unit 260. Details of the processing content of the color reproduction unit 250 will be described later.
  • the circuits following the color reproduction unit 250 are an example of the image quality adjustment processing unit described in the claims.
  • the parts enclosed by dotted lines i.e., the color base conversion unit 230, the photometry/AWB unit 241, the amplifier unit 242, the color reproduction unit 250, and the white clip processing unit 260, differ from the prior art.
  • a comparative example will be considered: a prior art image sensor 200 in which the parts enclosed by dotted lines are replaced with a general configuration.
  • FIG. 3 is a block diagram showing an example of the configuration of an image sensor 200 in the conventional technology.
  • the color base conversion unit 230 and the white clip processing unit 260 are not provided.
  • the pixel array unit 210 also generates a raw image in RGB space and supplies it to the photometry/AWB unit 241 and the amplifier unit 242.
  • the photometry/AWB unit 241 detects the raw image, estimates the white balance gain and correlated color temperature, and supplies it to the amplifier unit 242 and the color reproduction unit 250.
  • the amplifier 242 amplifies the color signals in the RAW image using a white balance gain and supplies the amplified color signals to the demosaic processor 220.
  • the demosaic processor 220 converts the RAW image into a demosaic image and supplies the amplified color signals to the color reproduction unit 250.
  • the color reproduction unit 250 corrects the color signals in the demosaic image using CCM and supplies the amplified color signals to the gamma correction unit 271.
  • RGB or YUV signals with spatial phases aligned at each pixel position are widely and commonly used as input vectors for signal processing with recognition algorithms such as CNN (Convolutional Neural Network)/RNN (Recurrent Neural Networks) or Attention-based Transformer.
  • recognition algorithms such as CNN (Convolutional Neural Network)/RNN (Recurrent Neural Networks) or Attention-based Transformer.
  • CNN Convolutional Neural Network
  • RNN Recurrent Neural Networks
  • Attention-based Transformer Most of these input signals are generated by inheriting the technology of digital still cameras and camcorders that were previously used mainly for ornamental purposes.
  • the figure shows a schematic diagram of these common and essential signal processes.
  • pixel data is input in a sequential, time-series manner from a pixel array having a Bayer pattern, passes through a photometry/AWB section 241 and an amplifier section 242, and is subjected to demosaic processing to align the spatial phase and number of pixels at the pixel positions of each RGB channel.
  • processing to adjust the color gamut such as a color reproduction matrix, is performed, and the signal is sent to subsequent processing (gamma correction, edge emphasis, YUV conversion, etc.) to become the final output signal.
  • this configuration example is only a part of the whole, and various pixel corrections (defect correction, black level adjustment, various shading corrections, etc.) before white balance are omitted as they are not directly related to the present invention.
  • demosaic processing is realized using a wide variety of algorithms, with many examples of the Bayer pattern being particularly representative. There are also many arrays other than the Bayer pattern, including those described in WO 2016/098641 and WO 2013/145487.
  • the common purpose of these demosaic processes is to generate basic signals for adjusting various spatial frequency components in the subsequent stage, such as color reproduction, luminance reproduction, and edge processing, under conditions where the spatial phase of the pixel positions is aligned in the subsequent signal processing.
  • Figure 4 shows part of the process in Figure 3, focusing on the CCM used in the amplifier 242 and color reproduction unit 250 before and after demosaic processing used in a typical RGB-Bayer type image sensor.
  • Fig. 5 shows a specific example of the configuration of Fig. 4 using specific basic function processing (low-pass filter, minimum value extraction processing, average value calculation, NR, multiplier, etc.).
  • LPF indicates a low-pass filter
  • min indicates minimum value extraction processing
  • mean indicates average value calculation processing
  • x indicates a multiplier.
  • + indicates an adder
  • Gm indicates the statistics of Gr and Gb (average value, etc.).
  • a shows the spectral characteristics of each of the R, Gr, Gb, and B pixels.
  • b shows the spectral characteristics of each of the R, Cr, Cg, and B pixels.
  • the Cr and Cg pixels are clear pixels.
  • the vertical axis of a and b shows relative sensitivity, and the horizontal axis shows wavelength.
  • a white clip is often provided after the color reproduction process to maintain the white color.
  • the above signal processing is valid when focusing on the RGB sensor.
  • Figure 11 shows an example of a color solid defined in RGB space (e.g., sRGB space) and the white point of a certain light source, as well as representative colors such as magenta (Mg), cyan (Cy), and yellow.
  • RGB space e.g., sRGB space
  • Mg magenta
  • Cy cyan
  • yellow yellow
  • the calculation using the color reproduction matrix rotates the axis connecting the white point and the origin (black) as the rotation axis, and further expands and contracts the solid arbitrarily along the rotation axis.
  • These geometric operations are mainly determined as conversion coefficients.
  • Figure 13 shows the white balance gain values, color reproduction matrix coefficients, and their determinants calculated under light source D for the RGGB sensor and RCCB sensor with the characteristics of Figure 7.
  • a shows the white balance gain values, color reproduction matrix coefficients, and determinants calculated under light source D for the RGGB sensor.
  • b shows the white balance gain values, color reproduction matrix coefficients, and determinants calculated under light source D for the RCCB sensor.
  • the present invention provides an image processing device that can properly process RAW signals generated in color spaces other than RGB, eliminate the trade-off between color reproduction and noise, and generate a correct white point while providing compatibility with RGB sensors in a conventional signal processing pipeline for RGB sensors.
  • FIG. 2 shows the signal processing and pipeline architecture in the first embodiment of the present invention.
  • the blocks enclosed by the dotted lines are different from the conventional technology.
  • the amplifier 242 in the front stage of the demosaic processing is placed in the rear stage.
  • a false color suppression process using color differences that imposes constraints on the input signal of the demosaic processing (such as aligning the signal levels of each pixel channel in the achromatic area in advance) as in the conventional method is adopted.
  • the above problem can be solved by first performing some kind of basis transformation on any color channel (three components) with a uniform spatial phase into RGB space, and then reapplying white balance and color reproduction matrix in RGB space.
  • the color basis transformation unit 230 performs transformation T (color basis transforming in the figure) in FIG. 14 to transform the pixel signal into the RGB space.
  • T color basis transforming in the figure
  • FIG. 7 a method using FIG. 7 can be used. That is, consider the transformation for each wavelength of the spectral characteristics of different color filters. First, each wavelength is discretized (for example, sampled in 5 nm increments), and the channel components (C0, C1, C2) and (R, G, B) at each wavelength are regarded as coordinate values in the respective color spaces. Assume that (R, C, B) at b in FIG. 7 is transformed to (R, G, B) at a in the same figure.
  • transformation T [...], which is a linear transformation from three dimensions to three dimensions. That is, consider the following equation so that the root mean square of the distance from the value (C0, C1, C2) at the wavelength ⁇ of each spectral characteristic to (R, G, B) at the same wavelength ⁇ through transformation T is minimized, for example, in the visible light wavelength range.
  • (C0, C1, C2) ⁇ is the coordinates at wavelength ⁇ of the spectral characteristics of an arbitrary color filter
  • (R, G, B) ⁇ is the coordinates at wavelength ⁇ of the spectral characteristics of the RGB sensor.
  • d[...] indicates the distance between two points.
  • the possible range of ⁇ is the visible light region (approximately 380 nm to 700 nm, in 5 nm increments).
  • the color base conversion unit 230 in FIG. 2 with multiple matrix calculators to generate a signal using a conversion T that takes into account multiple light source spectra, but this is not a good idea as it would unnecessarily increase costs. Therefore, one representative conversion T is placed in the color base conversion unit 230.
  • This can be the commonly used D light source (D65, etc.), or a conversion (Original) obtained from the spectral characteristics of an image sensor that does not take the light source into account. In this way, we consider how to estimate the correlated color temperature for an arbitrary color filter and an arbitrary light source.
  • the detection values used here are R, G, B values that have already been converted by the color base conversion unit 230, but here we distinguish that they have been converted into an R'G'B' space that actually contains errors, taking into account light source dependency.
  • the detection value data of each window region obtained in this way is plotted on a plane such as B'/G'-R'/G', as shown in FIG. 17.
  • a plane such as B'/G'-R'/G'
  • FIG. 17 an example is shown in which detection values obtained for each arbitrary window under light source A of an image sensor having an arbitrary color filter are plotted as black circles.
  • the center of gravity of the plotted values is determined as the position of light source A (white circle in the figure).
  • the photometry/AWB unit 241 repeats this series of operations for each other known light source.
  • the conversion T is originally dependent on the light source, by reconstructing the blackbody locus (obtained by fitting, etc.) based on the information obtained by calibration with the actual light source, the correlated color temperature can be estimated as a unique conversion independent of the light source for this conversion T.
  • FIG. 18 is a flowchart showing an example of a calibration method in the first embodiment of the present technology.
  • This calibration is started, for example, before the start of imaging.
  • the photometry/AWB unit 241 acquires the detection values of each window (step S901) and converts these detection values to R'G'B' (step S902).
  • the photometry/AWB unit 241 then plots the converted values on the B'/G'-R'/G' plane (step S903).
  • the photometry/AWB unit 241 determines the center of gravity of the plotted values as the coordinates of the set light source (step S904).
  • the photometry/AWB unit 241 ends the calibration.
  • the photometry/AWB unit 241 can estimate the correlated color temperature and white balance gain value from each window detection value obtained from an arbitrary subject (for example, by determining the barycentric coordinates of each detection value as in the calibration described above), as in a normal AWB.
  • This estimation is the same as that obtained in the generally known RGB space. For example, there is the estimation method described in International Publication No. 2010/095873, although the claimed purpose is different.
  • the color base conversion unit 230 and photometry/AWB unit 241 in FIG. 2 can therefore estimate the correlated color temperature and white balance gain from any color space by using any conversion T.
  • the output signal value (main line signal) of the color base conversion unit 230 which is purposely distinguished from the R'G'B', will have errors due to light source dependency if left as is. This error is absorbed by preparing correction coefficients for each correlated color temperature in the next stage, the color reproduction unit 250.
  • the correlated color temperature estimated by the photometry/AWB unit 241 in the same figure can be replaced with a previously prepared CCM as a correction.
  • the CCM correction coefficients can be generated by calibration.
  • FIG. 19 is a diagram showing an example of a method for calculating correction matrix coefficients in the first embodiment of the present technology.
  • the external circuit of the image sensor 200 prepares an appropriate color chart and acquires color patch values under known light sources with the image sensor 200 having an arbitrary color filter and the target RGB sensor.
  • the external circuit then obtains the least squares distance for each color patch between the value obtained by applying transformation T to the value of each patch and the value of the target RGB sensor.
  • This method is similar to the method for obtaining transformation T described above.
  • the external circuit can obtain CCM (Color Correction Matrix) coefficients for each correlated color temperature.
  • the external circuit stores the obtained CCM coefficients in a register or the like within the image sensor 200. These CCM coefficients are read out by the color reproduction unit 250.
  • an image sensor with any color filter can achieve color reproduction and white point of the same quality as the target RGB sensor.
  • the CCM in the figure is a simple 3x3 matrix and does not impose the above-mentioned constraint (the sum of row components is 1.0). Instead, the white clip in Figure 10 is introduced in the latter stage.
  • the target RGB values do not have to be the values of the actual sensor. It is common for users to target any RGB value they want.
  • Figure 20 is an example showing the composite coefficients of conversion T x CCM and white balance gain values under a D65 light source for an RCCB sensor with the characteristics of Figure 7.
  • the method of the present invention allows for conversion with smaller values.
  • the value of the determinant indicating the degree of conversion is also small, spatial distortion before and after conversion can be suppressed.
  • the present invention uses conversion T, and further corrects the error of conversion T through various calibrations, thereby achieving color reproduction similar to that of the target RGB sensor with high accuracy, low distortion, and reduced noise. This can further improve image quality.
  • the color base conversion unit 230 converts the color space from the C0C1C2 space to the RGB space, and after the conversion, the amplifier 242 amplifies the signal using the white balance gain, making it possible to achieve both improved color reproducibility and reduced noise. This makes it possible to improve image quality.
  • the color base conversion unit 231 detects the RAW image and converts the color space of the detected values, so that the same functions as those of the first embodiment can be realized in software using conventional hardware such as that shown in FIG. 3.
  • the drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs.
  • the drive system control unit 12010 functions as a control device for a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force for the vehicle.
  • the outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000.
  • the image capturing unit 12031 is connected to the outside-vehicle information detection unit 12030.
  • the outside-vehicle information detection unit 12030 causes the image capturing unit 12031 to capture images outside the vehicle, and receives the captured images.
  • the outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, or characters on the road surface, based on the received images.
  • the imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received.
  • the imaging unit 12031 can output the electrical signal as an image, or as distance measurement information.
  • the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared light.
  • the microcomputer 12051 can calculate control target values for the driving force generating device, steering mechanism, or braking device based on information inside and outside the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040, and output control commands to the drive system control unit 12010.
  • the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.
  • ADAS Advanced Driver Assistance System
  • the microcomputer 12051 can also perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.
  • the microcomputer 12051 can also output control commands to the body system control unit 12020 based on information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.
  • the audio/image output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying the occupants of the vehicle or the outside of the vehicle of information.
  • an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices.
  • the display unit 12062 may include, for example, at least one of an on-board display and a head-up display.
  • FIG. 24 shows an example of the installation position of the imaging unit 12031.
  • the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.
  • the imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 12100.
  • the imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100.
  • the imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100.
  • the imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100.
  • the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.
  • FIG. 24 shows an example of the imaging ranges of the imaging units 12101 to 12104.
  • Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose
  • imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively
  • imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door.
  • an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.
  • At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information.
  • at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple imaging elements, or an imaging element having pixels for phase difference detection.
  • the microcomputer 12051 can obtain the distance to each solid object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and can extract as a preceding vehicle, in particular, the closest solid object on the path of the vehicle 12100 that is traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km/h or faster). Furthermore, the microcomputer 12051 can set the inter-vehicle distance that should be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving, which runs autonomously without relying on the driver's operation.
  • automatic braking control including follow-up stop control
  • automatic acceleration control including follow-up start control
  • the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data to automatically avoid obstacles.
  • the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see.
  • the microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of a collision, it can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by forcibly decelerating or steering to avoid a collision via the drive system control unit 12010.
  • At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays.
  • the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured image of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian.
  • the audio/image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian.
  • the audio/image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.
  • the technology disclosed herein can be applied to the imaging unit 12031.
  • the image sensor 200 in FIG. 2 can be applied to the imaging unit 12031.
  • the processing procedures described in the above embodiments may be regarded as a method having a series of these procedures, or as a program for causing a computer to execute the series of procedures or a recording medium for storing the program.
  • Examples of the recording medium that can be used include a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, and a Blu-ray (registered trademark) Disc. It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.
  • the present technology can also be configured as follows. (1) a color base conversion unit that performs color base conversion processing on an image to convert a color space from a first color space to a second color space; an image quality adjustment processing unit that performs a predetermined image quality adjustment process on the image after the color basis conversion process. (2) further comprising a demosaic processing unit that converts a RAW image into a demosaic image; The image processing device according to (1), wherein the image is the RAW image or the demosaic image.
  • a photometry and AWB unit that detects the demosaic image after the color basis conversion process and estimates a white balance gain; an amplifier that amplifies a color signal in the demosaic image after the color basis conversion process by the white balance gain and supplies the amplified color signal to the image quality adjustment processing unit,
  • the color basis conversion unit performs the color basis conversion process on the demosaic image
  • the image quality adjustment processing unit a color reproduction unit that corrects the amplified color signal using a color reproduction matrix;
  • the image processing device further comprising: a white clip processing unit that limits the corrected color signal by a predetermined threshold value.
  • the color base conversion unit performs the color base conversion process on the detection value of the RAW image and supplies the result to the AWB unit;
  • the first color space is a color space different from an RGB space;
  • the image processing device according to any one of (1) to (4), wherein the second color space is an RGB space.
  • a plurality of pixels including clear pixels are arranged in the pixel array section, The image processing device according to (6), wherein the clear pixels are provided with transparent filters.
  • the image quality adjustment processing unit executes at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing.
  • (10) a color basis conversion step of performing a color basis conversion process on the image to convert a color space from a first color space to a second color space; and a program for causing a computer to execute an image quality adjustment processing procedure for performing a predetermined image quality adjustment processing on the image after the color basis conversion processing.
  • Imaging device 110
  • Optical section 120
  • DSP circuit 130
  • Display section 140
  • Operation section 150
  • Bus 160
  • Frame memory 170
  • Power supply section 200
  • Image sensor 210
  • Pixel array section 211
  • Pixel 220 Demosaicing section 230
  • 231 Color base conversion section
  • 241 Photometry and AWB section
  • Amplification section 243
  • AWB section 250
  • 251 Color reproduction section
  • White clip processing section 271
  • Gamma correction section 272
  • Edge enhancement section 273
  • YUV conversion section 274
  • Color conversion section 275 DNN monitor

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Of Color Television Signals (AREA)
  • Studio Devices (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

La présente invention améliore la qualité d'image dans un dispositif de traitement d'image pour traiter une image.　Ce dispositif de traitement d'image est pourvu d'une unité de transformation de base de couleur et d'une unité de traitement de réglage de qualité d'image. Dans le dispositif de traitement d'image, l'unité de transformation de base de couleur parmi l'unité de transformation de base de couleur et l'unité de traitement de réglage de qualité d'image effectue, sur une image, un traitement de transformation de base de couleur pour transformer un espace de couleur d'un premier espace de couleur en un second espace de couleur. L'unité de traitement de réglage de qualité d'image parmi l'unité de transformation de base de couleur et l'unité de traitement de réglage de qualité d'image effectue un traitement de réglage de qualité d'image prédéterminé sur l'image soumise au traitement de transformation de base de couleur par transformation de base de couleur.
PCT/JP2024/026801 2023-09-20 2024-07-26 Dispositif de traitement d'image, procédé de traitement d'image et programme Pending WO2025062831A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023151797A JP2025044312A (ja) 2023-09-20 2023-09-20 画像処理装置、画像処理方法、および、プログラム
JP2023-151797 2023-09-20

Publications (1)

Publication Number Publication Date
WO2025062831A1 true WO2025062831A1 (fr) 2025-03-27

Family

ID=95072741

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/026801 Pending WO2025062831A1 (fr) 2023-09-20 2024-07-26 Dispositif de traitement d'image, procédé de traitement d'image et programme

Country Status (2)

Country Link
JP (1) JP2025044312A (fr)
WO (1) WO2025062831A1 (fr)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001285700A (ja) * 2000-03-29 2001-10-12 Nec Corp 撮像装置
JP2006270135A (ja) * 2005-03-22 2006-10-05 Acutelogic Corp 電子的撮像装置における色再現補正装置
JP2008302075A (ja) * 2007-06-08 2008-12-18 Olympus Corp 画像処理装置並びにスコープおよびこれらを備える内視鏡装置
JP2009081709A (ja) * 2007-09-26 2009-04-16 Seiko Epson Corp 画像処理システムおよび画像処理装置
JP2009171062A (ja) * 2008-01-11 2009-07-30 Sony Corp 画像処理装置、撮像装置、方法およびプログラム
JP2012209913A (ja) * 2011-03-16 2012-10-25 Toshiba Corp 固体撮像装置及びカメラモジュール
JP2013012979A (ja) * 2011-06-30 2013-01-17 Nikon Corp 画像処理装置及び画像処理方法
JP2021170767A (ja) * 2020-04-16 2021-10-28 パナソニックＩｐマネジメント株式会社 画像処理装置、画像処理方法、および画像処理システム

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001285700A (ja) * 2000-03-29 2001-10-12 Nec Corp 撮像装置
JP2006270135A (ja) * 2005-03-22 2006-10-05 Acutelogic Corp 電子的撮像装置における色再現補正装置
JP2008302075A (ja) * 2007-06-08 2008-12-18 Olympus Corp 画像処理装置並びにスコープおよびこれらを備える内視鏡装置
JP2009081709A (ja) * 2007-09-26 2009-04-16 Seiko Epson Corp 画像処理システムおよび画像処理装置
JP2009171062A (ja) * 2008-01-11 2009-07-30 Sony Corp 画像処理装置、撮像装置、方法およびプログラム
JP2012209913A (ja) * 2011-03-16 2012-10-25 Toshiba Corp 固体撮像装置及びカメラモジュール
JP2013012979A (ja) * 2011-06-30 2013-01-17 Nikon Corp 画像処理装置及び画像処理方法
JP2021170767A (ja) * 2020-04-16 2021-10-28 パナソニックＩｐマネジメント株式会社 画像処理装置、画像処理方法、および画像処理システム

Also Published As

Publication number Publication date
JP2025044312A (ja) 2025-04-02

Similar Documents

Publication Publication Date Title
US11910099B2 (en) Real-time HDR video for vehicle control
US10432847B2 (en) Signal processing apparatus and imaging apparatus
US11172173B2 (en) Image processing device, image processing method, program, and imaging device
TWI775808B (zh) 攝像裝置、攝像模組、攝像系統及攝像裝置之控制方法
EP3474534B1 (fr) Appareil de traitement d'images, système d'imagerie et système de traitement d'images
WO2014084199A1 (fr) Dispositif de capture d'image
WO2018179671A1 (fr) Dispositif de traitement d'images, procédé de traitement d'images et dispositif de capture d'images
CN109479092B (zh) 图像处理设备和图像处理方法
CN109644241B (zh) 图像处理设备和图像处理方法
WO2022249562A1 (fr) Dispositif de traitement de signal, procédé et programme
WO2020075357A1 (fr) Élément d'imagerie à semi-conducteur, dispositif d'imagerie, et procédé de contrôle d'élément d'imagerie à semi-conducteur
US10630952B2 (en) Image sensor
WO2025062831A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et programme
JP4974897B2 (ja) 車載撮像装置
JP2012010141A (ja) 画像処理装置
WO2021166601A1 (fr) Dispositif d'imagerie et procédé d'imagerie
JP7708176B2 (ja) 画像処理装置および画像処理方法
WO2017169079A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et élément de capture d'image
US20240338852A1 (en) Color discrimination device and color discrimination method
CA3033242C (fr) Video hdr en temps reel pour la commande de vehicules
WO2022219874A1 (fr) Dispositif et procédé de traitement de signaux, et programme

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24867913

Country of ref document: EP

Kind code of ref document: A1