JP2022008037A - Image processing device, method for controlling image processing device, and program - Google Patents

Abstract

An object of the present invention is to improve the accuracy of image inference even when image characteristics change due to correction applied to the image. [Solution] An image processing device uses an image processing unit that performs multiple image processes on a training image and a teacher image, and a training image and a teacher image that have undergone multiple image processes, respectively, to create a learning model. The apparatus includes a learning means that performs machine learning, and a control means that performs control to store a plurality of parameters for a learning model learned in each of a plurality of image processes in association with a correction condition of the image processing. [Selection diagram] Figure 3

Description

The present invention relates to an image processing apparatus, a control method and a program of the image processing apparatus.

In recent years, a method of inferring an image with improved resolution, contrast, etc. using a neural network has been used. As a related technique, the technique of Patent Document 1 has been proposed. The technique of Patent Document 1 calculates an error between a gamma-corrected correct image and an output image, and updates the parameters of the neural network based on the calculated error.

Japanese Unexamined Patent Publication No. 2019-12152

C. Chen, Q. Chen, J. et al. Xu, and V. Koltun, "Learning to see in the dark," https: // arxiv. org / abs / 1805.01934.

In the technique of Patent Document 1 described above, there is only one type of network parameter of the neural network used when making an inference for correcting blur due to aberration or diffraction. Further, in the technique of Non-Patent Document 1, it is premised that inference such as noise reduction is performed using one kind of network parameter. Therefore, when image processing whose characteristics change depending on the correction conditions is used, if the conditions at the time of learning and the correction conditions at the time of inference are different, appropriate inference cannot be performed in noise reduction or the like.

An object of the present invention is to improve the inference accuracy of an image even when the characteristics of the image are changed by the correction applied to the image.

In order to achieve the above object, the image processing apparatus according to claim 1 of the present application includes an image processing means that performs a plurality of image processing on a training image and a teacher image, and the image processing means that has been subjected to the plurality of image processing, respectively. A learning means for performing machine learning of a learning model using a training image and the teacher image, and a plurality of parameters for the learning model learned in each of the plurality of image processings are used as correction conditions for the image processing. It is characterized by comprising a control means for controlling storage in association with each other.

According to the present invention, the inference accuracy can be improved even when the characteristics of the image are changed by the correction applied to the image.

It is a figure which shows an example of the system configuration including an image processing apparatus. It is a block diagram which shows the function of the learning processing of an image processing apparatus. It is a flowchart which shows an example of the flow of the learning process of an embodiment. It is a figure which shows an example of a neural network. It is a block diagram which shows the function of the inference processing of an image processing apparatus. It is a flowchart which shows an example of the flow of inference processing. It is a block diagram which shows the function of an image processing part. It is a figure which shows the 1st example of an analog gain. It is a figure which shows the 2nd example of an analog gain. It is a flowchart which shows an example of the flow of the learning process of 1st Embodiment. It is a flowchart which shows the other example of the flow of the learning process of 1st Embodiment. It is a flowchart which shows an example of the flow of the inference processing of 1st Embodiment. It is a figure which shows an example of the relationship between the F value and the amount of light, and the relationship between an F value and a correction value. It is a flowchart which shows an example of the flow of the learning process of 2nd Embodiment. It is a figure which shows another example of the relationship between an F value and a correction value. It is a flowchart which shows an example of the flow of the inference processing of 2nd Embodiment. It is a flowchart which shows an example of the flow of the learning process of 3rd Embodiment. It is a flowchart which shows an example of the flow of the inference processing of 3rd Embodiment. It is a figure which shows the state of the peripheral light falloff. It is a flowchart which shows an example of the flow of the learning process of 4th Embodiment. It is a figure which shows the example of the table data of the peripheral light falloff correction data. It is a flowchart which shows an example of the flow of the inference processing of 4th Embodiment. It is a figure which shows the sensor structure. It is a figure which shows the output voltage of each channel with respect to the amount of light incident on a photodiode. It is a flowchart which shows an example of the flow of the learning process of 5th Embodiment. It is a flowchart which shows an example of the flow of the inference processing of 5th Embodiment. It is a block diagram which shows the function of the learning processing of an image processing apparatus. It is a flowchart which shows an example of the flow of the learning process of 6th Embodiment. It is a block diagram which shows the function of the inference processing of an image processing apparatus. It is a flowchart which shows an example of the flow of the inference processing of 6th Embodiment.

Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the respective embodiments.

Hereinafter, the configuration common to each embodiment will be described. FIG. 1 is a diagram showing an example of a system configuration including an image processing device. In the system configuration of FIG. 1, the image processing device 100 is connected to the image pickup device 120, the display device 130, and the storage device 140. The system configuration is not limited to the example of FIG. For example, the image processing device 100 may be configured to include one or both of the display device 130 and the storage device 140. Further, the image processing device 100 may be incorporated in the image pickup device 120. The image processing device 100 may be an edge computer, a local server, a cloud server, or the like.

The image processing apparatus 100 performs various image processing, as well as learning processing and inference processing. Hereinafter, the learning process will be described as being a process by deep learning. However, the image processing apparatus 100 may perform machine learning by using a machine learning method other than deep learning. For example, the image processing apparatus 100 may perform learning processing by using an arbitrary machine learning method such as a support vector machine, a decision tree, or logistics regression.

In the learning process, a learning set in which a training image and a teacher image are paired is input to the neural network, and network parameters (hereinafter, parameters) such as weights and biases of the neural network are adjusted. By inputting many learning sets into the neural network and performing machine learning, the parameters are optimized so that the feature distribution of the image output by the neural network approaches the feature distribution of the teacher image. When an unknown image is input to the neural network in which the trained parameters are set, the inference result (inference image) is output from the neural network.

The image pickup apparatus 120 will be described. The image pickup device 120 includes a lens unit 121, an image pickup element 122, and a camera control circuit 123. The image pickup apparatus 120 may include other elements. The lens unit 121 includes a diaphragm, an optical lens, a motor, and the like. The motor drives the aperture and the optical lens. The lens unit 121 may be configured to include a plurality of optical lenses. The lens unit 121 operates based on the control signal output by the camera control circuit 123, and can optically enlarge / reduce the image, adjust the focal length, and the like. The aperture of the lens unit 121 can control the aperture area. As a result, the aperture value (F value) is controlled, and the amount of incident light can be adjusted.

The light transmitted through the lens unit 121 is imaged on the image pickup device 122. The image pickup device 122 converts the incident light into an electric signal like a CCD sensor, a CMOS sensor, or the like. The image sensor 122 is driven based on a control signal from the camera control circuit 123. The image pickup device 122 has functions such as resetting the charge in each pixel, controlling the timing of reading, gain processing applied to the read signal, converting an analog signal to a digital signal, and the like. The image pickup device 120 transmits the digital image signal output by the image pickup device 122 to the image signal receiving circuit 101 of the image processing device 100. In the example of FIG. 1, the image pickup device 120 and the image processing device 100 are connected wirelessly or by wire so as to be communicable, and the image processing device 100 receives the image signal transmitted by the image pickup device 120. The camera control circuit 123 performs drive control of the lens unit 121 and the image pickup element 122 based on the drive control signal received from the camera communication connection circuit 102 of the image processing device 100.

Next, the image processing apparatus 100 will be described. The image signal receiving circuit 101 receives an image signal from the image pickup apparatus 120. The camera communication connection circuit 102 transmits a drive control signal for driving the lens unit 121 and the image pickup device 122 to the image pickup device 120. The image processing circuit 103 performs image processing on the image signal received by the image signal receiving circuit 101. This image processing may be performed by the CPU 105. Further, the image processing circuit 103 has a function of performing noise reduction processing using a neural network. The image processing circuit 103 may be realized by a circuit such as a microprocessor, DSP, FPGA, or ASIC that performs image processing, or may be realized by a CPU 105.

The frame memory 104 is a memory for temporarily storing an image signal. The frame memory 104 is an element capable of temporarily storing an image signal and reading the stored image signal at high speed. When the amount of data of the image signal becomes large, it is preferable that a high-speed and large-capacity memory is applied to the frame memory 104. For example, as the frame memory 104, DDR3-SDRAM (Double Data Rate 3-Synchronous Dynamic RAM) or the like can be applied. When DDR3- SDRAM is applied to the frame memory 104, various processing becomes possible. For example, DDR3- SDRAM is a suitable element for performing image processing such as synthesizing images different in time and cutting out a necessary region. However, the frame memory 104 is not limited to DDR3- SDRAM.

The CPU 105 reads out the control program and various parameters stored in the ROM 106 and expands them in the RAM 107. The processing of each embodiment is realized by the CPU 105 executing the program expanded in the RAM 107. The metadata extraction circuit 108 extracts metadata such as lens drive conditions and sensor drive conditions. The GPU 109 is a graphics processing unit, which is a processor capable of high-speed arithmetic processing. The GPU 109 is used to generate a screen to be displayed on the display device 130, but is also suitably used for deep learning arithmetic processing. The image generated by the GPU 109 is output to the display device 130 via the display drive circuit 110 and the display device connection circuit 111. As a result, the image is displayed on the display device 130.

The storage drive circuit 112 drives the storage device 140 via the storage connection circuit 113. The storage device 140 stores a large amount of image data. The image data stored in the storage device 140 includes a training image (learning image) and a teacher image (correct image) corresponding to the training image. The training image and the teacher image form one pair (learning set). A plurality of learning sets are stored in the storage device 140. The storage device 140 may store the learned parameters generated by the learning process. Hereinafter, when the learning process is performed under the control of the CPU 105, the image data stored in advance in the storage device 140 is used, and when the inference process is performed, the image signal acquired from the image pickup device 120 is used. Explain as a thing. However, the present invention is not limited to the above example. For example, when the inference process is performed, the image signal stored in the storage device 140, an external storage device, or the like may be used.

The training image used in the learning process may be an image of the Bayer arrangement, an image taken by a three-plate type image pickup sensor, an image taken by a vertical color separation type image pickup sensor, or the like. Further, the training image may be an image of an arrangement other than the Bayer arrangement (honeycomb structure, color filter arrangement having low periodicity, etc.). When the training image is an image of the Bayer arrangement, it may be an image of 1ch of the Bayer arrangement or an image separated for each color channel. The same applies to the teacher image. Training images and teacher images are not limited to the examples described above.

Next, the learning process will be described. 2A and 2B are block diagrams showing a function of learning processing of the image processing apparatus 100. First, FIG. 2A will be described. The training image holding unit 202 holds the training image acquired from the storage device 201. The teacher image holding unit 203 holds the teacher image acquired from the storage device 201. The training image holding unit 202 and the teacher image holding unit 203 may be realized by, for example, a RAM 107 or the like.

The teacher image and the training image are images showing the same subject. The training image is an image containing noise, and the teacher image is a correct image containing no noise. However, the teacher image may contain some noise. For example, the training image may be generated by the CPU 105 or the image processing circuit 103 adding noise to the teacher image. Further, the training image may be an image taken in a situation where noise may occur in the same subject as the correct image (for example, an image taken with a high sensitivity setting). The training image may be an image in which the sensitivity of the image taken in low illuminance is corrected to the same degree as that of the teacher image. Further, the teacher image may be an image taken with low sensitivity (sensitivity lower than the training image).

The neural network 204 is a multi-layered neural network. Details will be described later. When learning (deep learning) for the neural network 204 is performed, it is preferable to use a training image including various noise patterns and subjects in order to improve the inference accuracy. By learning the neural network 204 using a training image containing various noise patterns and subjects, inference processing when an unknown image including noise patterns and subjects not included in the training image is input. The accuracy of is improved. When the number of training images used for deep learning of the neural network 204 is insufficient, for example, padding processing such as cutting, rotation, and inversion may be applied to the training images. In this case, the same padding process is performed on the teacher image. Further, it is preferable that the training image and the teacher image are each divided by the upper limit value (saturation luminance value) of the signal and normalized. When the training image is input, the neural network 204 generates and outputs an output image.

The image processing unit 205 performs image processing on the output image and the teacher image of the neural network 204. Examples of image processing include ISO sensitivity correction, F value correction, color suppression processing, color bending correction, color balance correction, peripheral light falloff correction, interchannel gain correction, flicker correction, and compression and decompression processing. When the training process of the neural network 204 is completed, the trained parameters are generated. As a result, inference processing can be performed by applying the learned parameters to the neural network 204. It is preferable that the image processing unit 205 matches the condition of the correction performed in the learning process and the condition of the correction performed in the inference process. This improves the inference accuracy when the inference process is performed. In FIG. 2A, the image processing unit 205 is on the output side of the neural network 204, but as shown in FIG. 2B, the image processing unit 205 may be on the input side of the neural network 204. The image processing unit 205 is realized by, for example, an image processing circuit 103, a CPU 105, or the like.

The error evaluation unit 206 calculates the error between the output image corrected by the image processing unit 205 and the teacher image. The teacher image and the training image have the same arrangement of color components. The error evaluation unit 206 may calculate an error (loss error) by using, for example, the mean square error for each pixel or the sum of the absolute values of the differences of each pixel. Further, the error evaluation unit 206 may evaluate the error by calculating the error using another index such as the coefficient of determination. The parameter adjustment unit 207 updates each parameter of the neural network 204 so that the error calculated by the error evaluation unit 206 becomes small. The error evaluation unit 206 and the parameter adjustment unit 207 may be realized by, for example, the CPU 105, the GPU 109, or the cooperative operation of the CPU 105 and the GPU 109. The error evaluation unit 206 and the parameter adjustment unit 207 correspond to the learning means.

The parameter adjustment unit 207 may update each parameter of the neural network 204 by using, for example, an error back propagation method. The parameter adjusting unit 207 may update each parameter of the neural network 204 by using another method. At this time, the parameter adjusting unit 207 may fix or change the update amount of each parameter of the neural network 204. As described above, each parameter of the neural network 204 is updated so that the error between the output image processed by the image processing unit 205 and the teacher image becomes small. This improves the inference accuracy when an unknown image is input to the neural network 204.

The parameter storage areas 208-1 to 208-n (n is an integer of 2 or more) are areas for storing updated parameters. The parameter storage areas 208-1 to 208-n (hereinafter collectively referred to as parameter storage areas 208) are areas for storing the parameters of the neural network 204 that has been trained according to the correction conditions of the image processing. That is, in each parameter storage area 208, the trained parameters that have been trained under the correction conditions of different image processing are stored in association with the correction conditions of the respective image processing. The parameter storage area 208 may be a part of the storage area of the RAM 107 or a part of the storage area of the storage device 201. The control unit 209 performs various controls related to the learning process. The control unit 209 is realized by the CPU 105.

Next, the flow of the learning process will be described. FIG. 3 is a flowchart showing an example of the flow of the learning process. In FIG. 3, the step is described as S, which is the same in the other flowcharts. In S301, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S302, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. The noise amount and noise pattern of the training image may be different from the noise amount and noise pattern of other training images, and may be the same.

In S303, the image processing unit 205 performs image processing on the output image from the neural network 304 and the teacher image. As described above, the image processing unit 205 may be on the input side or the output side of the neural network 204. That is, the processing of S303 may be performed before the processing of S302.

In S304, the error evaluation unit 206 calculates the error between the output image subjected to the image processing and the teacher image. In S305, the parameter adjustment unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S306, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the number of learnings reaches a predetermined number of times, the control unit 209 may determine that the predetermined end condition is satisfied. Further, the control unit 209 may determine that the predetermined end condition is satisfied when the calculated error is equal to or less than the predetermined value. Further, the control unit 209 may determine that the predetermined end condition is satisfied when the degree of reduction of the above error is less than or equal to the predetermined degree or when the user gives an end instruction. When the control unit 209 determines NO in S306, the control unit 209 returns the flow to S301. In this case, the processes of S301 to S305 are performed using the learning set of the new training image and the correct answer image. On the other hand, if the control unit 209 determines YES in S306, the control unit 209 advances the flow to S307.

In S307, the control unit 209 stores the updated parameters (learned parameters) in the parameter storage area 208 that differs depending on the image processing conditions. At this time, the control unit 209 may store not only the parameters updated by the learning process but also information related to the model structure of the neural network 204 in association with the parameters in the parameter storage area 208. In S308, the control unit 209 determines whether or not the parameters for the correction conditions of all the image processing have been acquired. If the control unit 209 determines YES in S308, the control unit 209 ends the processing of the flowchart of FIG. On the other hand, if the control unit 209 determines NO in S308, the control unit 209 advances the flow to S309. In S309, the control unit 209 changes the correction conditions for image processing. Then, the control unit 209 returns the flow to S301. As a result, the learning process in which the image processing conditions are changed is newly performed.

Next, the neural network 204 will be described. FIG. 4 is a diagram showing an example of the neural network 204. The neural network 204 of each embodiment is described as being a convolutional neural network (CNN). However, the neural network 204 is not limited to CNN. For example, the neural network 204 may be a GAN (Generative Adversarial Network), a skip connection, an RNN (Recurrent Neural Network), or the like. Further, instead of the neural network 204, a learning model machine-learned by another machine learning method may be used. As the learning model, for example, the above-mentioned support vector machine, decision tree, or the like can be applied.

When the input image 401 is input to the neural network 204, the convolution operation 402 is performed. The input image 401 may be image data or data representing a feature map of the image data. The convolution matrix 403 is a filter that performs a convolution operation on the input image 401. Bias 404 is added to the result output by the convolution operation of the input image 401 and the convolution matrix 403. The feature map 405 is the calculation result of the convolution operation in which the bias 404 is added. The number of layers, the number of neurons, the coupling coefficient, the weight, and the like of the intermediate layer of the neural network 204 may be arbitrary values. Further, when the neural network 204 is implemented in a programming circuit such as FPGA, the connection and weight between neurons may be reduced. This point is the same when the GPU 109 processes the neural network 204. Further, the learning process and the inference process may be executed collectively for a plurality of color channels, or may be executed individually for each of the plurality of colors.

In CNN, the feature map of the input image is extracted by executing the convolution operation by the filter on the input image. The size of the filter may be any size. In the latter layer, a convolution operation by another filter is executed on the feature map extracted in the previous layer, so that different feature maps are sequentially extracted. In each layer of the intermediate layer, the input signal is multiplied by the weight of the filter representing the strength of the connection, and the bias is further added. By applying the activation function to this calculation result, the output signal in each neuron can be obtained. The weights and biases in each layer are parameters (network parameters), and the values of the parameters are updated by the learning process. The parameters updated by machine learning are the learned parameters. As the activation function, any function such as a sigmoid function or a ReLU function can be applied. The activation function applied in each embodiment will be described as being a Leaky ReLU function represented by the following equation 1. However, as the activation function, any activation function such as a sigmoid function or a tanh function may be applied.

以上の式（１）において、ｍａｘは、引数のうち最大値を出力する関数を表す。

In the above equation (1), max represents a function that outputs the maximum value among the arguments.

As described above, the parameters updated by the learning process (learned parameters) are stored in the parameter storage areas 208-1 to 208-n for each correction condition of the image processing. When the inference processing is performed, the parameters corresponding to the correction conditions of the image processing are acquired from any of the parameter storage areas 208-1 to 208-n. Then, the acquired parameters are applied to the neural network 204, and inference processing is performed. This makes it possible to improve the inference accuracy when performing image processing with different correction conditions.

In the above example, the training process of the neural network 204 is performed using a training image containing noise and a teacher image containing no noise or having low noise. As a result, when an image containing unknown noise is input to the machine-learned neural network 204, the noise-reduced image can be inferred by the inference processing by the neural network 204. The above-mentioned example is a learning process for noise reduction processing, but it can also be applied to learning processing for processes other than noise reduction. For example, when super-resolution inference processing is performed by a neural network, learning processing can be performed using a training image obtained by downsampling a teacher image and a teacher image. At this time, the size of the training image and the teacher image may be adjusted.

Next, the inference process will be described. FIG. 5 is a block diagram showing a function of inference processing of the image processing apparatus 100. The acquisition unit 501 acquires the captured image and the information used for parameter selection from the image pickup device 120. The information used at the time of parameter selection is used to specify the parameter to be selected from each parameter stored in the parameter storage areas 208-1 to 208-n. The information used in parameter selection may be metadata. The metadata may be information added to the captured image (information on setting conditions of the lens unit 121, the image pickup device 120, or the like), or information extracted from the metadata extraction circuit 108. Here, the information used at the time of parameter selection is information related to image processing. When the information used for parameter selection is the above metadata, the acquisition unit 501 does not have to separately acquire the information used for parameter selection.

The parameter selection unit 502 as the selection means acquires the parameter corresponding to the acquired information used at the time of parameter selection from any of the parameter storage areas 208-1 to 208-n. Each parameter stored in the parameter storage areas 208-1 to 208-n is a parameter generated by the above-mentioned learning process, but may be a parameter learned by an external device other than the image processing device 100. .. In this case, the external device is a neural network having the same network structure as the neural network 204, and performs the above-mentioned learning process. Then, the image processing apparatus 100 may acquire the parameters generated by the learning process and store them in the parameter storage areas 208-1 to 208-n.

The trained parameters selected by the parameter selection unit 502 are applied to the neural network 204. Then, the captured image acquired from the imaging device 120 is input to the neural network 204. When the captured image is input, the neural network 204 as an inference means performs an inference process and generates an inference image as an inference result. The inference image output unit 503 outputs the generated inference image to the storage device 201. The control unit 504 performs various controls related to the inference process. The acquisition unit 501, the parameter selection unit 502, the inference image output unit 503, and the control unit 504 are realized by, for example, the CPU 105.

FIG. 6 is a flowchart showing an example of the flow of the inference process of the present embodiment. In S601, the acquisition unit 501 acquires the information used at the time of parameter selection. In S602, the parameter selection unit 502 selects the parameter corresponding to the acquired information from the parameter storage areas 208-1 to 208-n. Each parameter stored in the parameter storage areas 208-1 to 208-n is a parameter updated under the correction conditions of different image processing. The information used when selecting the acquired parameters is information regarding correction conditions for image processing. The parameter selection unit 502 acquires parameters that are the same as the image processing correction conditions represented by the information used at the time of the acquired parameter selection, or that the image processing correction conditions are close to each other.

In S603, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204. The acquisition unit 501 acquires an captured image from the image pickup device 120. The captured image acquired by S604 is assumed to be a RAW image. When the RAW image is coded, the CPU 105 or the image processing circuit 103 performs the decoding process. The acquisition unit 501 may acquire RAW images from the storage device 140, ROM 106, RAM 107, and the like.

In S605, the control unit 504 inputs the acquired captured image to the neural network 204. At this time, the control unit 504 may convert the acquired captured image into an input image for inputting to the neural network 204. The size of the captured image input to the neural network 204 when performing the inference process may be the same as or different from the size of the training image input to the neural network 204 when performing the learning process. .. When the captured image is converted into the input image, the control unit 504 may normalize the signal, separate each color component, and the like.

In S606, the neural network 204 performs inference processing. The inference process by the neural network 204 may be executed by the GPU 109, may be executed by the CPU 105, or may be executed by the cooperative operation of the CPU 105 and the GPU 109. The inference process of S606 generates an inference image as an inference result of the neural network 204. In S607, the inference image output unit 503 outputs the generated inference image to the storage device 140. The inference image output unit 503 may output the generated inference image to the ROM 106, the RAM 107, the display device 130, or the like. When the captured image is converted to the input image in S604, the control unit 504 may perform an inverse conversion of the inferred image to undo the conversion performed in S604.

When the inference processing of another captured image is performed, the image processing apparatus 100 executes each processing of S601 to S607. As described above, the parameters of the neural network 204 learned and processed for each correction condition of the image processing are stored in the parameter storage areas 208-1 to 208-n. Then, the parameter selection unit 502 selects the corresponding parameter from the parameter storage areas 208-1 to 208-n based on the correction conditions of the image processing. This makes it possible to perform inference processing of the captured image using the optimum parameters. As a result, it is possible to obtain an image with reduced noise.

In the above, the image processing apparatus 100 performs both learning processing and inference processing. In this respect, the image processing device that performs learning processing and the image processing device that performs inference processing may be separate devices. In this case, for example, the image processing device (hereinafter, learning device) that performs learning processing has the configuration of FIG. 2A or FIG. 2B, and the image processing device that performs inference processing (hereinafter, inference device). Has the configuration of FIG. In this case, the image processing device that performs learning processing and the image processing device that performs inference processing are communicably connected.

The inference device acquires each learned parameter from the learning device, and stores each acquired parameter in a different parameter storage area 208 in the same manner as the control unit 209 described above. Then, the inference device performs each process of the flowchart of FIG. As described above, even if the image processing device that performs learning processing and the image processing device that performs inference processing are separate devices, the image processing device that performs learning processing and the image processing device that performs inference processing are the same device. The same effect can be obtained as in the case of.

(First Embodiment)
In the first embodiment, ISO sensitivity correction is taken as an example of image processing, and learning processing is performed using a teacher image and a training image in which ISO sensitivity correction is performed with correction values corresponding to each of a plurality of ISO sensitivity corrections. A specific example of the parameters of the training model performed will be described. Further, in the first embodiment, a specific example of the parameters of the learning model in which the learning process is performed in consideration of the temperature condition will be described.

In the first embodiment, the configuration described in the above-described embodiment is used. First, ISO sensitivity correction will be described. ISO sensitivity correction is a correction process that corrects the sensitivity of the analog gain determined by the combination of the column amplifier and the lamp for each combination of the column amplifier and the lamp so that the target ISO brightness is obtained. .. The target ISO brightness is a standard of sensitivity defined by the International Organization for Standardization. Here, there are a plurality of combinations of the column amplifier and the lamp, and the optimum ISO sensitivity correction value differs for each combination. Therefore, when the ISO sensitivity correction is performed by inference processing, if only one type of parameter for which learning processing has been performed is stored, the inference accuracy is lowered. Therefore, the image processing device 100 stores the parameters that have been learned for each combination of the column amplifier and the lamp.

The optimum correction value for analog gain also differs depending on the temperature. That is, when the ISO sensitivity correction is performed by inference processing, the optimum parameters differ depending on the temperature. Therefore, in the present embodiment, the image processing apparatus 100 can also store parameters according to the ISO sensitivity and the temperature.

FIG. 7 is a block diagram showing the functions of the image processing unit 205 of the present embodiment. FIG. 7A is a diagram when the neural network 204 is provided on the input side of the image processing unit 205. FIG. 7B is a diagram when the neural network 204 is provided on the output side of the image processing unit 205. In the case of FIG. 7A, the teacher image is input to the image processing unit 205 from the teacher image holding unit 203, and the output image is input from the neural network 204. In the case of FIG. 7B, the training image is input to the image processing unit 205 from the training image holding unit 202, and the teacher image is input from the teacher image holding unit 203. In FIG. 7A, an output image processed by the neural network 204 with the training image as an input is input to the image processing unit 205. On the other hand, in FIG. 7B, a training image that has not been processed by the neural network 204 is input. The following processing is the same for FIGS. 7 (A) and 7 (B). As shown in FIGS. 7A and 7B, the image processing unit 205 includes an extraction unit 702, a correction value acquisition unit 703, and a correction unit 704.

The extraction unit 702 extracts metadata from the teacher image. The metadata is information added to the teacher image and includes ISO sensitivity information about ISO sensitivity and temperature information about temperature. In the present embodiment, the extraction unit 702 extracts ISO sensitivity information and temperature information as metadata of the teacher image. The metadata may be information added to the image in, for example, an EXIF (Exif Image File Form) format. However, the metadata is not limited to EXIF format data.

The correction value acquisition unit 703 acquires ISO sensitivity correction data corresponding to the extracted ISO sensitivity information and temperature information from the ROM 106. The ROM 106 stores data of correction values corresponding to the combination of ISO sensitivity and temperature. The ROM 106 may store the correction value data corresponding to the ISO sensitivity instead of the above combination. The correction unit 704 uses the ISO sensitivity correction data acquired by the correction value acquisition unit 703 to perform ISO sensitivity correction on the output image and the teacher image output by the neural network 204. The output image and the teacher image to which the ISO sensitivity correction has been performed are output to the error evaluation unit 206.

Next, a first example of ISO sensitivity correction will be described. FIG. 8 is a diagram showing a first example of analog gain. FIG. 8A is a diagram showing an example of an ideal analog gain. FIG. 8B is a diagram showing an example of an actual analog gain. The analog gain is determined by the combination of the column amplifier and the lamp. The analog gain shown in FIG. 8B has a variation with respect to the analog gain shown in FIG. 8A. Therefore, the correction unit 704 corrects the analog gain.

For example, when focusing on ISO 1, as shown in the example of FIG. 8A, the analog gain “0.” when the column amplifier is “0.6 times” and the lamp is “0.85 times”. 51 times (= 0.6 × 0.85) ”is the ideal value. On the other hand, in the example of FIG. 8B, the actual analog gain is "0.59 times" for the column amplifier and "0.84 times" for the lamp, so that the analog gain is "0.50 times". (= 0.59 × 0.84) ”. Therefore, the correction value of the ISO sensitivity correction for ISO1 is "1.02 times (= 0.51 / 0.50)".

Focusing on ISO9, as shown in the example of FIG. 8A, the analog gain “8.00 times” when the column amplifier is “8.0 times” and the lamp is “1.00 times”. (= 8.0 × 1.00) ”is the ideal value. On the other hand, in the example of FIG. 8B, the actual analog gain is "7.88 times" for the column amplifier and "0.97 times" for the lamp, so that the analog gain is "7.64 times". (= 7.88 × 0.97) ”. Therefore, the correction value of the ISO sensitivity correction for ISO1 is "1.047 times (= 8.00 / 7.64)".

As described above, the correction value of the ISO sensitivity correction is different for each ISO sensitivity. Therefore, the correction unit 704 performs ISO sensitivity correction with a correction value corresponding to the ISO sensitivity information of the metadata extracted from the teacher image. The ROM 106 stores the correction value of the ISO sensitivity correction for each ISO sensitivity. The correction value of the ISO sensitivity correction for each ISO sensitivity may be stored in the RAM 107, the storage device 140, or the like. Although the example of FIG. 8B shows an example in which the actual analog gain is lower than the ideal analog gain, the actual analog gain may be higher than the ideal analog gain.

Next, a second example of ISO sensitivity correction will be described. FIG. 9 is a diagram showing a second example of analog gain. The characteristics of the analog gain shown in FIG. 9 vary depending on the temperature. Therefore, the correction value of the ISO sensitivity correction also differs depending on the temperature. This is due to the temperature characteristics of the channel amplifier. Therefore, the ROM 106 may store the correction value for each combination of each ISO sensitivity and each temperature.

Here, when the ROM 106 stores the correction values for all the temperatures, the number of the correction values stored in the ROM 106 also increases. In this case, the number of combinations of each ISO sensitivity and each temperature increases, and the number of parameters stored in the parameter storage area 208 also increases. As a result, the amount of data used for the parameter storage area 208 is large. Therefore, the ROM 106 may collectively store the temperature correction values for each predetermined range.

For example, the ROM 106 has three types of temperature ranges: a first temperature range (low temperature less than 0 ° C.), a second temperature range (temperature of 0 ° C. to 40 ° C.), and a third temperature range (high temperature exceeding 40 ° C.). A common correction value may be stored in. When the correction values in the three types of temperature ranges are used for each of the ISOs as described above, a total of 33 correction values (= 11 × 3) are stored in the ROM 106. As a result, since one common correction value is used for a plurality of temperatures (for example, 0 ° C to 40 ° C), the number of parameters stored in the parameter storage area 208 can be reduced, and the amount of data used can be reduced. be able to. The temperature range setting method, the number of types, and the like are not limited to the above examples. For example, the temperature range may be set variably according to the degree of change in analog gain due to temperature. In this case, the temperature range may be set narrow in the section where the degree of change is equal to or greater than the predetermined degree, and the temperature range may be set wide in the section where the degree of change is less than the predetermined degree.

Next, an example of the flow of the learning process in the first embodiment will be described. FIG. 10 is a flowchart showing an example of the flow of the learning process of the first embodiment. The flowchart of FIG. 10 corresponds to the block diagram of FIG. 7 (A). In S1001, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. The process of S1001 corresponds to the process of S301 of FIG. In S1002, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. The process of S1002 corresponds to the process of S302 in FIG.

In S1003, the extraction unit 702 of the image processing unit 205 extracts metadata from the teacher image. In S1004, the correction value acquisition unit 703 acquires the correction value corresponding to the combination of the ISO sensitivity information and the temperature information of the extracted metadata from the ROM 106. In S1005, the correction unit 704 performs ISO sensitivity correction on the teacher image and the output image output from the neural network 204 with the acquired correction value. The output image output from the neural network 204 is an image obtained by processing the training image with the neural network 204.

In S1006, the error evaluation unit 206 calculates the error between the image-processed output image and the teacher image. The process of S1006 corresponds to the process of S304 of FIG. In S1007, the parameter adjusting unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. The process of S1006 corresponds to the process of S305 of FIG. In S1008, the control unit 209 determines whether or not the predetermined end condition is satisfied. The process of S1008 corresponds to the process of S306 in FIG. When the control unit 209 determines NO in S1008, the control unit 209 returns the flow to S1001 because the predetermined end condition is not satisfied. On the other hand, if the control unit 209 determines YES in S1008, the control unit 209 satisfies the predetermined end condition, so that the flow proceeds to S1009.

In S1009, the control unit 209 stores the updated parameter (learned parameter) in the parameter storage area 208 that is different for each correction condition of the ISO sensitivity correction. The correction condition for the ISO sensitivity correction of the present embodiment is a correction value corresponding to the combination of each ISO sensitivity and each temperature range. When the temperature information is not used, the correction condition for the ISO sensitivity correction of the present embodiment is a correction value corresponding to each ISO sensitivity. The control unit 209 stores the updated parameters in a parameter storage area 208 that is different for each combination of ISO sensitivity and each temperature range. The processing of S1009 corresponds to S307 of FIG. In S1010, the control unit 209 determines whether or not the parameters of all the conditions have been acquired. At this time, the control unit 209 determines S1010 based on whether all combinations of each ISO and each temperature range have been acquired. If the control unit 209 determines YES in S1010, the control unit 209 ends the processing of the flowchart of FIG. On the other hand, when the control unit 209 determines NO in S1010, the control unit 209 advances the flow to S1011. In S1011, the control unit 209 changes the conditions for the combination of ISO and the temperature range. Then, the control unit 209 returns the flow to S1001. As a result, a learning process in which the combination of ISO and the temperature range is changed is newly performed.

Next, another example of the processing flow in the first embodiment will be described. FIG. 11 is a flowchart showing another example of the flow of the learning process of the first embodiment. The flowchart of FIG. 11 corresponds to the block diagram of FIG. 7 (B). In the processing of the flowchart of FIG. 10, ISO sensitivity correction is performed on the output image output from the neural network 204, and learning processing is performed. On the other hand, in the processing of the flowchart of FIG. 11, ISO sensitivity correction is performed on the training image input to the neural network 204, and learning processing is performed.

The process of S1101 corresponds to the process of S1001. In S1102, the extraction unit 702 of the image processing unit 205 extracts metadata from the teacher image. In S1103, the correction value acquisition unit 703 acquires the correction value corresponding to the combination of the ISO sensitivity information and the temperature information of the extracted metadata from the ROM 106. The process of S1102 corresponds to the process of S1003, and the process of S1103 corresponds to the process of S1004. In S1104, the correction unit 704 performs ISO sensitivity correction on the teacher image and the training image. That is, the correction unit 704 performs ISO sensitivity correction on the training image that has not been processed by the neural network 204.

In S1105, the control unit 209 inputs the training image corrected for ISO sensitivity to the neural network 204. An output image is generated by a process performed by the neural network 204 with the training image corrected for ISO sensitivity as an input. In S1106, the error evaluation unit 206 calculates the error between the ISO sensitivity-corrected teacher image and the output image generated in S1105. In S1107, the parameter adjusting unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. Since each process of S1108 to S1111 is the same as each process of S1008 to S1011 of FIG. 10, the description thereof will be omitted.

The flowchart of FIG. 10 and the flowchart of FIG. 11 show the flow of the learning process regarding the combination of the ISO sensitivity information and the temperature information of the first embodiment. In S1004 of FIG. 10 or S1103 of FIG. 11, the correction value acquisition unit 703 acquires the correction value corresponding to the combination of the ISO sensitivity information and the temperature information of the extracted metadata from the ROM 106. At this time, the correction value acquisition unit 703 does not have to acquire the ISO sensitivity information from the ROM 106 and acquire the temperature information. For example, when the ISO sensitivity information is not stored in the ROM 106, the correction value acquisition unit 703 acquires the ISO sensitivity information from the ROM 106 and does not acquire the temperature information. Even in this case, the ISO sensitivity correction can be performed using the respective correction values of each ISO.

Next, the inference processing of the first embodiment will be described. FIG. 12 is a flowchart showing an example of the flow of the inference processing of the first embodiment. In S1201, the acquisition unit 501 acquires ISO sensitivity information and temperature information. The acquisition unit 501 can acquire ISO sensitivity information and temperature information from the image pickup apparatus 120. For example, the acquisition unit 501 can acquire ISO sensitivity information based on the setting conditions of the image pickup apparatus 120. Further, when the image pickup apparatus 120 has a temperature measurement function such as that of a thermistor, the acquisition unit 501 can acquire temperature information from the temperature measured by the thermistor of the image pickup apparatus 120. The acquisition unit 501 may acquire ISO sensitivity information and temperature information from the metadata of the image data stored in the storage device 140.

In S1202, the parameter selection unit 502 selects the parameter corresponding to the combination of the acquired ISO sensitivity information and the temperature information from the parameter storage areas 208-1 to 208-n. In S1203, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204. Since each process of S1204 to S1207 is the same as S604 to S607 of FIG. 6, the description thereof will be omitted.

In the first embodiment, the parameters of the neural network 204 that have been trained according to each ISO sensitivity are stored in the parameter storage areas 208-1 to 208-n. Then, the parameter selection unit 502 selects the corresponding parameter from the parameter storage areas 208-1 to 208-n based on the ISO sensitivity condition. As a result, the inference processing of the captured image can be performed using the parameters that match the ISO sensitivity conditions, so that the inference accuracy is improved. As a result, it is possible to obtain an image with reduced noise. Further, in the first embodiment, the learning process is performed in consideration of the temperature condition. That is, the parameters of the neural network 204 that have been trained according to the combination of each ISO and each temperature range can be stored in the parameter storage areas 208-1 to 208-n. Then, the parameter selection unit 502 selects the corresponding parameter from the parameter storage areas 208-1 to 208-n based on both the ISO sensitivity and the temperature conditions. As a result, it is possible to perform inference processing of the captured image using parameters that match the ISO sensitivity characteristics and the temperature characteristics, so that the inference accuracy can be further improved. As a result, the noise reduction effect of the image can be further improved.

(Second embodiment)
In the second embodiment, F-number correction is taken as an example of image processing, and learning processing is performed using a teacher image and a training image in which F-number correction is performed with correction values corresponding to each of a plurality of F-number corrections. A specific example of the parameters of the training model performed will be described.

F-number correction is a correction that applies gain to an image so that the relationship between the F-number and the amount of light (luminance) approaches linear when the relationship between the F-number, which is the aperture value, and the amount of light (luminance) is not linear. be. FIG. 13 is a graph showing an example of the relationship between the F value and the amount of light and the relationship between the F value and the correction value in the second embodiment. FIG. 13A is a graph showing the relationship between the F value and the amount of light. The horizontal axis is the F value, and the vertical axis shows the amount of light received by the image pickup device 122 of the image pickup device 120. As described above, the F value correction is a process of applying a gain according to the aperture value (F value) of the lens unit 121. Here, when the aperture is changed from the small aperture side to the open side, the relationship between the F value and the amount of light (luminance) is linear on the small aperture side, but the F value and the amount of light (luminance) are on the open side. The relationship is non-linear. One of the factors that causes the relationship between the F value and the amount of light (luminance) to be non-linear on the open side is that the light incident on the image sensor 122 from an angle cannot be sufficiently captured on the open side. Therefore, in the section where the F value is on the open side, F value correction is performed by multiplying the image signal by the gain in order to approach the ideal linearity. In the section on the open side, the F value is corrected, so that noise is generated in the image.

In FIG. 13A, the relationship between the F value and the amount of light in the section of Fx to Fs changes linearly as shown by the solid line. On the other hand, the relationship between the F value and the amount of light in the section from Fe to Fx changes non-linearly as represented by the alternate long and short dash line. Therefore, as described above, a correction value for removing noise is multiplied in the section from Fe to Fx (hereinafter referred to as the non-linear section) so that the relationship between the F value and the amount of light is linear even in the non-linear section. The correction value in the non-linear interval is not a constant value as shown in FIGS. 13 (A) and 13 (B). On the other hand, a correction value of a certain value is multiplied in the section from Fx to Fs (hereinafter, linear section). That is, the correction value of the non-linear section needs to be changed to an appropriate value depending on the F value so that the relationship between the F value and the amount of light becomes linear. In FIG. 13B, the correction value of the non-linear section is shown by the dotted line.

Next, an example of the flow of the learning process in the second embodiment will be described. FIG. 14 is a flowchart showing an example of the flow of the learning process of the second embodiment. The flowchart of FIG. 14 corresponds to the block diagram of FIG. 7A, and the correction unit 704 of the image processing unit 205 corrects the F value. In S1401, the control unit 209 newly sets the F value when performing the learning process. In S1402, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S1403, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. The noise amount and noise pattern of the training image may be different from the noise amount and noise pattern of other training images, and may be the same.

In S1404, the image processing unit 205 corrects the F value with respect to the teacher image and the output image from the neural network 204. At this time, the image processing unit 205 corrects the F value with the correction value corresponding to the F value set in S1401. As described above, the image processing unit 205 may be on the output side of the neural network 204 as shown in FIG. 2A, or may be on the input side as shown in FIG. 2B. That is, the processing of S1401 may be performed before the processing of S1403. If the extraction unit 702 of the image processing unit 205 acquires the F value from the metadata of the teacher image, S1401 can be omitted. In S1405, the error evaluation unit 206 calculates the error between the output image corrected for the F value and the teacher image. In S1406, the parameter adjustment unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S1407, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the control unit 209 determines NO in S1407, the control unit 209 returns the flow to S1402. In this case, the processing of S1402 to S1406 is performed using the learning set of the new training image and the correct answer image. On the other hand, if the control unit 209 determines YES in S1407, the control unit 209 advances the flow to S1408.

In S1408, the control unit 209 determines whether the F value set in S1401 is in the linear section. The linear section will be described later. If the control unit 209 determines YES in S1408, the control unit 209 advances the flow to S1409. In S1409, the control unit 209 stores the updated parameter (learned parameter) in the same parameter storage area 208. On the other hand, if the control unit 209 determines NO in S1408, the control unit 209 advances the flow to S1410. In S1410, the control unit 209 stores the updated parameter (learned parameter) in a different parameter storage area 208 according to the F value set in S1401. In S1409 and S1410, the control unit 209 may store the information related to the parameters updated by machine learning, the model structure of the neural network 204, etc. in the parameter storage area 208 in association with the F value condition set in S1401. .. In S1411, the control unit 209 determines whether or not the parameters for all the F value conditions have been acquired. When the control unit 209 determines NO in S1411, the flow is returned to S1401 and a new F value condition is set. If the control unit 209 determines YES in S1411, the control unit 209 ends the processing of the flowchart of FIG.

In this way, the control unit 209 controls to perform machine learning (deep learning) of the neural network 204 by changing the F value to be corrected. As a result, machine learning is performed reflecting the correction value for each F value, and the parameters are updated. Then, in S1408 of FIG. 14, the control unit 209 determines whether the F value set when performing machine learning is within the linear section. The control unit 209 can determine S1408, for example, based on whether the set F value is equal to or greater than the above-mentioned Fx. The value of Fx (predetermined threshold value) may be set to an arbitrary value, but it is preferably set to the boundary where the relationship between the F value and the amount of light changes from linear to non-linear. However, the value of Fx may be set to a value within a predetermined range from the boundary.

Then, when the control unit 209 determines YES in S1408, the control unit 209 stores the updated learned parameter in the same parameter storage area 208 of the parameter storage areas 208-1 to 208-n. If the control unit 209 determines YES in S1408 even if the updated parameters have different F values, the control unit 209 stores them in the same parameter storage area 208-1. Alternatively, when the control unit 209 determines YES in S1408, if the updated parameter is saved once, the processing of S1409 may be omitted from the second time onward. On the other hand, when the control unit 209 determines NO in S1408, a different F value is set and the updated learned parameters are stored in different parameter storage areas 208. Each time the process of S1410 is executed, the control unit 209 sequentially stores the parameters in different parameter storage areas 208 (for example, parameter storage areas 208-2 to 208-n).

Therefore, in the section where the F value and the amount of light (luminance) have a non-linear relationship, the inference process can be executed by using the parameters of the neural network 204 corresponding to the F value having a fine grain size. This improves the inference accuracy. On the other hand, in the section where the F value and the amount of light (luminance) are linear, the inference process is executed using one parameter of the neural network 204. As a result, the number of parameters to be stored can be reduced, so that the amount of data used can be reduced and the amount of hardware resources such as the storage device 140 and RAM 107 can be reduced. That is, since the image processing apparatus 100 changes the particle size of the parameter to be stored according to the F value, it is possible to reduce the amount of data used while suppressing the deterioration of the inference accuracy.

In the above-mentioned example, the image processing apparatus 100 stores the parameters of the machine-learned neural network 204 for each F value in the non-linear section, but the present invention is not limited to this example. For example, in a non-linear section, the image processing device 100 may store learned parameters for each drive resolution of the diaphragm provided in the lens unit 121 of the image pickup device 120. Further, the image processing apparatus 100 performs both learning processing and inference processing. In this respect, the image processing device that performs learning processing and the image processing device that performs inference processing may be separate devices. This point is the same as that of the first embodiment. An example in which the image processing device that performs learning processing and the image processing device that performs inference processing are separate devices is the same in each of the following embodiments.

In this case, the inference device acquires and stores one trained parameter in the linear section, and acquires and stores the trained parameter corresponding to each of the plurality of aperture values in the non-linear section. Then, the inference device performs each process of the flowchart of FIG. As described above, the same effect can be obtained even if the image processing device that performs the learning process and the image processing device that performs the inference processing are separate devices.

Next, a modified example of the second embodiment will be described. FIG. 15 is a graph showing an example of the relationship between the F value and the correction value in the modified example of the second embodiment. In the first embodiment, the correction value in the section where the F value is Fx to Fs is constant, but in the second embodiment, the correction value in the section where the F value is Fx to Fs changes linearly. The section where the F value is Fx to Fs is a section in which the relationship between the F value and the amount of light (luminance) is linear. In the second embodiment, the control unit 209 sets the F value to Fx and performs the processing of FIG. 14, and sets the parameter (first parameter) obtained by performing the processing of FIG. 14 and the F value to Fs, and performs the processing of FIG. The obtained parameter (second parameter) is stored in the parameter storage area 208.

Then, as shown in FIG. 15, when the inference processing of the captured image is performed when the F value is Fz, the control unit 504 infers using the inference image using the first parameter and the inference using the second parameter. Interpolate using an image. As a result, it is possible to obtain an inferred image for an arbitrary F value in the section of Fx to Fs in which the correction value changes linearly. In this case, there are two parameters (machine-learned parameters) stored in order to obtain an inferred image for an arbitrary F value in the section of Fx to Fs.

Here, even if the control unit 504 performs the processing of FIG. 14 using any two F values in the section of Fx to Fs in which the correction value changes linearly, and saves the first parameter and the second parameter. good. Even in this case, inference about an arbitrary F value in the interval of Fx to Fs in which the correction value changes linearly by interpolating using the inference image using the first parameter and the inference image using the second parameter. You can get an image.

FIG. 16 is a flowchart showing an example of the flow of inference processing in the modification of the second embodiment. In S1601, the acquisition unit 501 acquires F-number information from the image pickup apparatus 120. In S1602, the control unit 504 determines whether the F value indicated by the information acquired in S1601 is within a linear section. As described above, when the F value indicated by the information acquired in S1601 is Fx or more (predetermined threshold value or more), the control unit 504 determines NO in S1602, and the F value is less than Fx (predetermined). If it is less than the threshold value), it may be determined as YES in S1602. When the control unit 504 determines NO in S1602, the parameter selection unit 502 selects the parameter corresponding to the acquired F value information from the parameter storage areas 208-1 to 208-n in S1603. Each parameter stored in the parameter storage areas 208-1 to 208-n is a parameter updated under different F value conditions. The parameter selection unit 502 acquires a parameter for the F value that is the same as the acquired F value information or a parameter for the F value that is the closest to the value.

In S1604, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204. In S1605, the acquisition unit 501 acquires an captured image from the image pickup device 120. This captured image is assumed to be a RAW image. When the RAW image is coded, the CPU 105 or the image processing circuit 103 performs the decoding process. The acquisition unit 501 may acquire RAW images from the storage device 140, ROM 106, RAM 107, and the like.

In S1606, the control unit 504 inputs the acquired captured image to the neural network 204. In S1607, the neural network 204 performs inference processing. The inference process by the neural network 204 may be executed by the GPU 109, may be executed by the CPU 105, or may be executed by the cooperative operation of the CPU 105 and the GPU 109. The inference process of S1607 generates an inference image as an inference result of the neural network 204. In S1608, the inference image output unit 503 outputs the generated inference image to the storage device 140. The inference image output unit 503 may output the generated inference image to the ROM 106, the RAM 107, the display device 130, or the like. At this time, the parameter selected in S1603 is a trained parameter that has been trained using the F value in the non-linear interval.

If the control unit 504 determines NO in S1602, the control unit 504 advances the flow to S1609. In S1609, the parameter selection unit 502 selects the first parameter described above. For example, when the first parameter is stored in the parameter storage area 208-1, the parameter selection unit 502 selects the first parameter from the parameter storage area 208-1. In S1610, the neural network 204 executes an inference process to which the first parameter is applied. The processing of S1610 corresponds to the processing of S1604 to S1607. That is, the inference process using the captured image as an input is executed for the neural network 204 to which the first parameter is applied. As a result, an inference image (first inference image) is generated by inference processing in which the first parameter is applied to the neural network 204.

In S1611, the parameter selection unit 502 selects the second parameter described above. For example, when the second parameter is stored in the parameter storage area 208-2, the parameter selection unit 502 acquires the second parameter from the parameter storage area 208-2. Then, the neural network 204 executes the inference process to which the second parameter is applied. The processing of S1612 corresponds to the processing of S1604 to S1607. That is, the inference process using the captured image as an input is executed for the neural network 204 to which the second parameter is applied. As a result, an inference image (second inference image) is generated by inference processing in which the second parameter is applied to the neural network 204.

In S1613, the control unit 504 executes the interpolation processing of the inferred image. At this time, the control unit 504 synthesizes the first inference image and the second inference image and executes the interpolation process. Further, the control unit 504 executes the interpolation process by weighting the first inference image and the second inference image with reference to the F value indicated by the information acquired in S1601. For example, when the F value indicated by the information acquired in S1601 is close to the F value corresponding to the first parameter, the control unit 504 may execute the interpolation process by increasing the weight of the first inference image. .. On the other hand, when the F value indicated by the information acquired in S1601 is close to the F value corresponding to the second parameter, the control unit 504 may execute the interpolation process by increasing the weight of the second inference image. .. When interpolating the first inference image and the second inference image, the control unit 504 may interpolate for each pixel, or may interpolate for each area in which the entire image is divided in an arbitrary range. Then, in S1608, the control unit 504 outputs an inferred image in which the interpolation process is executed.

As described above, in the second embodiment, the same effect as that of the first embodiment can be obtained even if the correction value is not constant in the section where the correction value is linear. In the second embodiment, two parameters, the first parameter and the second parameter, which have been trained, are used. The F value corresponding to the first parameter and the F value corresponding to the second parameter when the learning process is performed may be selected according to the linearity of the correction value. Further, in the second embodiment, the control unit 504 may further divide a section (a section of Fx to Fs) in which the F value and the amount of light (luminance) are linear into a plurality of sections. In this case, the control unit 504 may use the divided section as one processing unit and execute interpolation processing for two parameters for each processing unit.

(Third embodiment)
In the third embodiment, color suppression processing and color bending correction are taken as examples of image processing, and a teacher who has performed image processing with correction values corresponding to each of a plurality of color suppression processing or a plurality of color bending corrections. A specific example of the parameters of the training model in which the training process is performed using the image and the training image will be described.

When a high-luminance subject is photographed by the image pickup device, any of the RGB signals output from the image pickup element may be saturated. In this case, the ratio of the RGB signals may change from the value representing the original color of the subject. To solve this problem, by applying color suppression processing, which is a processing for reducing the saturation in a region where any of the RGB signals is saturated, the color change in the high-luminance portion caused by signal saturation is alleviated. How to do it is known.

Further, when the image pickup apparatus captures a low-luminance and high-saturation subject, any of the RGB signals may approach the lower limit black level. In this case, when gamma correction with a sharp rise at the black level is applied to the RGB signal, the noise component superimposed on the RGB signal is emphasized by the gamma correction, causing deterioration in image quality. To solve this problem, we know how to increase the signal value of a signal with a small signal value among RGB signals by applying color suppression processing to the lower limit region where any of the RGB signals takes a value close to the lower limit. Has been done. This makes it possible to reduce the degree of enhancement of the noise component by gamma correction.

Since the color suppression process is a process of reducing the saturation, noise in the area to which the process is applied is reduced. Therefore, by performing the color suppression process during learning, a neural network considering the noise reduced by the color suppression process can be obtained. Further, the characteristics of the color suppression processing are changed by the processing of gamma correction and color gamut conversion. Therefore, when the output format input at the time of learning and the output format of the image input at the time of inference are different, the inference accuracy of the inferred image is lowered because the characteristics of the color suppression processing are also different.

Further, the image to be viewed by the user is subjected to gamma correction and color gamut conversion processing according to the output format. At this time, depending on the color of the subject, a color different from the original subject may be output. In order to suppress this color change, the process of correcting by applying a gain for each color is called color bending correction. How the color changes depends on the gamma correction and color gamut conversion processing. That is, the characteristics of color bending correction change depending on the output format. Therefore, when the output format input at the time of learning and the output format of the image input at the time of inference are different, the inference accuracy of the inferred image is lowered because the characteristics of the color bending correction are also different.

Next, an example of the flow of the learning process in the third embodiment will be described. FIG. 17 is a flowchart showing an example of the flow of the learning process of the third embodiment. The flowchart of FIG. 17 can be realized by using, for example, the block diagram of FIG. 2 and FIG. 7 (A). The correction unit 704 of the image processing unit 205 performs the development processing and the color suppression processing described later. Here, color suppression processing will be described as an example of image processing, but the same applies when color bending correction is used instead of color suppression processing.

In S1701, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S1702, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. In S1703, the extraction unit 702 of the image processing unit 205 extracts gamma, color gamut, or metadata related to gamma and color gamut from the teacher image. The gamma may be one that is premised on performing color adjustment such as color grading later, or may be one that is intended for the user to appreciate (PQ, HLG, etc.). The color gamut is BT. 2020, BT. It may be 709 or DCI-P3, or it may be one that matches the display characteristics of the display device 130 (2.2, 2.4, etc.). Moreover, these combinations may be used.

In S1704, the correction value acquisition unit 703 acquires a correction value corresponding to the gamma, color gamut, or combination of gamma and color gamut of the extracted metadata from the ROM 106. In S1705, the correction unit 704 converts the teacher image and the output image output from the neural network 204 into a color space divided into luminance and hue such as YCC or ICtCp by performing development processing. The development process may include a debayer process performed on the images in the Bayer array. Further, the correction unit 704 performs color suppression correction with the acquired correction value for the teacher image and the output image to which the development process is applied.

In S1706, the error evaluation unit 206 calculates the error between the image-processed output image and the teacher image. In S1707, the parameter adjustment unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S1708, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the control unit 209 determines NO in S1708, the control unit 209 returns the flow to S1701 because the predetermined end condition is not satisfied. On the other hand, if the control unit 209 determines YES in S1708, the control unit 209 satisfies the predetermined end condition, so that the flow proceeds to S1709.

In S1709, the control unit 209 stores the updated parameter (learned parameter) in the parameter storage area 208 that is different for each correction condition of the color suppression process. The correction condition of the color suppression process of the present embodiment is a correction value corresponding to the combination of each gamma and each color range. The control unit 209 stores the updated parameters in a parameter storage area 208 that is different for each combination of each gamma and each color range. If there is no change in the color gamut, different parameters for each gamma are stored in the parameter storage area 208. In S1710, the control unit 209 determines whether or not the parameters of all the conditions have been acquired. At this time, the control unit 209 determines S1710 based on whether all combinations of each gamma and each color range have been acquired. If the control unit 209 determines YES in S1710, the control unit 209 ends the processing of the flowchart of FIG. On the other hand, if the control unit 209 determines NO in S1710, the control unit 209 advances the flow to S17111. In S1711, the control unit 209 changes the condition of the combination of the gamma and the color gamut. Then, the control unit 209 returns the flow to S1701. As a result, a new learning process in which the combination of gamma and color gamut is changed is performed. In FIG. 17, the training image is input to the neural network 204, and the output image from the neural network 204 is subjected to development processing and color suppression processing, but the present invention is not limited to this. The training image may be developed and color-suppressed, and then the trained image subjected to these processes may be input to the neural network 204. In this case, the output image from the neural network 204 and the teacher image to which the development processing and the color suppression processing are applied are input to the error evaluation unit 206.

Next, the inference processing of the third embodiment will be described. FIG. 18 is a flowchart showing an example of the flow of inference processing according to the third embodiment. In S1801, the acquisition unit 501 acquires gamma and color gamut information. Alternatively, the captured image of S1804 may be acquired first, and the gamma and color gamut information may be acquired from the metadata of the captured image. In S1802, the parameter selection unit 502 selects the parameter corresponding to the acquired gamma and color gamut information from the parameter storage areas 208-1 to 208-n. In S1803, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204. Since each process of S1804 to S1807 is the same as S604 to S607 of FIG. 6, the description thereof will be omitted.

In addition to gamma correction and color gamut conversion processing, color balance correction that adjusts exposure compensation, gain compensation, saturation correction, etc. also has parameters for neural networks according to the combination of these correction values. The same effect can be obtained by preparing.

(Fourth Embodiment)
In the fourth embodiment, peripheral light falloff correction is taken as an example of image processing, and a teacher image and a training image in which peripheral light falloff correction is performed with correction values corresponding to each of a plurality of peripheral light falloff corrections are used. A specific example of the parameters of the training model in which the training process has been performed will be described.

Peripheral illumination falloff correction is a correction that applies a gain to the image so as to correct the lens characteristics in which the amount of light gradually drops from the center of the lens to the periphery, and the gain becomes larger toward the periphery from the center of the lens. .. In addition, the characteristics of peripheral illumination drop vary depending on the lens type, aperture, focal length, and zoom position conditions, and the optimum gain differs for each region.

19 (a) and 19 (b) are views showing a state of vignetting. The horizontal axis is the image height which is the distance from the center position of the lens, and the image height at the center position of the lens is 0% and the image height at the outer periphery of the image circle of the lens is 100%. The vertical axis is the amount of light, and the amount of light at the center position of the lens is 100%. The lens conditions (aperture, focus, zoom position) are different in FIGS. 19 (a) and 19 (b). In FIG. 19A, the amount of light at the position corresponding to the image height of 15% is close to 100%, and the amount of light at the position corresponding to the image height of 100% is 80%. In FIG. 15B, the amount of light at the position corresponding to the image height of 70% is close to 100%, and the amount of light at the position corresponding to the image height of 100% is 90%. As described above, the starting point at which the peripheral illumination starts to drop and the shape such as the inclination differ depending on the state of the lens.

19 (c) shows a gain curve for correcting the peripheral light falloff shown in FIG. 19 (a), and FIG. 19 (d) shows a gain curve for correcting the peripheral light falloff shown in FIG. 19 (b). The gain curve of is shown. The horizontal axis is the image height, and the vertical axis is the gain. The gain curve of FIG. 19 (c) has a shape for making the amount of light 100% at each image height, and a gain of about 1.25 times is set at the position of 100% image height, and the gain of FIG. 19 (d) is set. The gain curve is set to a gain of about 1.1 times at a position where the image height is 100%. As the gain is increased, the noise component contained in the image is also amplified, so that an upper limit may be set for the gain.

Next, an example of the flow of the learning process in the fourth embodiment will be described. FIG. 20 is a flowchart showing an example of the flow of the learning process of the fourth embodiment. The flowchart of FIG. 20 can be realized by using, for example, the block diagram of FIG. 2 and FIG. 7 (A). The correction unit 704 of the image processing unit 205 performs peripheral light falloff correction, which will be described later.

In S2001, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S2002, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. In S2003, the extraction unit 702 of the image processing unit 205 extracts metadata regarding the lens type, aperture, zoom position, and focal length from the teacher image.

In S2004, the correction value acquisition unit 703 acquires the correction value corresponding to the combination of the extracted lens type, aperture, zoom position, and focal length from the ROM 106. In S2005, the correction unit 704 performs peripheral illumination drop correction with the acquired correction value for the teacher image and the output image output from the neural network 204.

In S2006, the error evaluation unit 206 calculates the error between the image-processed output image and the teacher image. Then, in S2007, the parameter adjusting unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S2008, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the control unit 209 determines NO in S2008, the control unit 209 returns the flow to S1001 because the predetermined end condition is not satisfied. On the other hand, if the control unit 209 determines YES in S2008, the control unit 209 satisfies the predetermined end condition, so that the flow proceeds to S2009.

In S2009, the control unit 209 stores the updated parameters (learned parameters) in the parameter storage area 208, which is different for each correction condition of the peripheral illumination amount drop correction. The correction condition for the peripheral illumination fall correction of the present embodiment is a correction value corresponding to a combination of a lens type, an aperture, a zoom position, and a focal length. The control unit 209 stores the updated parameters in the parameter storage area 208, which is different for each combination of lens type, aperture, zoom position, and focal length. If the lens is not an interchangeable lens but a lens integrated with an image pickup device, it is not necessary to save parameters according to the type of lens. Further, if the lens is a single focus lens without a zoom function, it is not necessary to save the parameters according to the zoom position. In S2010, the control unit 209 determines whether or not the parameters of all the conditions have been acquired. If the control unit 209 determines YES in S2010, the control unit 209 ends the processing of the flowchart of FIG. On the other hand, if the control unit 209 determines NO in S2010, the control unit 209 advances the flow to S2011. In S2011, the control unit 209 changes the conditions for the combination of the lens type, the aperture, the zoom position, and the focal length. Then, the control unit 209 returns the flow to S2001.

FIG. 21 is a diagram showing an example of table data of peripheral light falloff correction data. The table data 2100 is a data group having a focal length of A1. The horizontal axis is the aperture value (S1, S2, S3 ... Sn), and the vertical axis is the zoom position (T1, T2, T3, ... Tn). For example, the peripheral light falloff correction data at the zoom position T1, the aperture value S1, and the focal length A1 is Dt1s1a1. Further, the table data 2101, 1022, 2103, 2104 are data groups having focal lengths A1, A2, A3, and An, respectively. The peripheral light falloff correction data corresponding to each lens condition is stored in the ROM 106. In addition, since the shape of the peripheral light falloff differs for each lens in the interchangeable lens, table data is provided for each lens. The interchangeable lens can be identified by the lens ID assigned to each lens.

Next, the inference processing of the fourth embodiment will be described. FIG. 22 is a flowchart showing an example of the flow of inference processing according to the fourth embodiment. In S2201, the acquisition unit 501 acquires information on the lens type, aperture, zoom position, and focal length. Alternatively, the captured image of S2204 may be acquired first, and the lens type, aperture, zoom position, and focal length information may be acquired from the metadata of the captured image. In S2202, the parameter selection unit 502 selects the parameters corresponding to the acquired lens type, aperture, zoom position, and focal length information from the parameter storage areas 208-1 to 208-n. Then, in S2203, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204. Since each process of S2204 to S2207 is the same as S604 to S607 of FIG. 6, the description thereof will be omitted.

(Fifth Embodiment)
In the fifth embodiment, inter-channel gain correction is taken as an example of image processing, and a teacher image and a training image in which inter-channel gain correction is performed with correction values corresponding to each of a plurality of inter-channel gain corrections are used. A specific example of the parameters of the training model in which the training process has been performed will be described.

The inter-channel gain correction is a process of correcting the gain of the output amplifier arranged in the final stage of the analog output of the sensor. Since this output amplifier is an analog amplifier, each amplifier has a gain difference. There is a difference in gain among multiple output amplifiers in the same sensor. It also has the characteristic that it changes depending on the temperature. Therefore, if only one network parameter is used, the noise difference due to the gain difference of the output amplifier caused by the temperature condition is not taken into consideration, and a sufficient noise reduction effect cannot be obtained. That is, the optimum network parameters suitable for the inter-channel gain correction characteristics are required for each temperature condition.

FIG. 23 is a diagram showing a sensor structure. A vertical line 2303 is connected to the photodiode 2301 that converts the intensity of light into a voltage via a switch 2302. The vertical line 2303 is connected to the output amplifiers 2305 to 2308 via the switch 2304. By switching the switches 2302 and 2304 on and off, the voltage signal obtained from the photodiode 2301 is read from the output amplifiers 2305 to 2308. The inter-channel gain correction in the present embodiment is a function of correcting the gains of the four output amplifiers 2305 to 2308, and the four output amplifiers 2305 to 2308 are referred to as channels. For example, here, the output amplifier 2305 is the channel A, the output amplifier 2306 is the channel B, the output amplifier 2307 is the channel C, and the output amplifier 2308 is the channel D.

FIG. 24 is a diagram showing the output voltage of each channel with respect to the amount of light incident on the photodiode 2301.

In FIG. 24A, graph 2401 shows the output of channel A, graph 2402 shows the output of channel B, graph 2403 shows the output of channel C, and graph 2404 shows the output of channel D. Since the output amplifier is an analog amplifier, there is a difference in output level as shown in FIG. 24. The cause of this difference in output level is the variation in offset between channels and the variation in gain. After the variation in offset is corrected, the variation in gain is corrected. FIG. 24B is a diagram showing the output voltage of each channel in a state where the offset variation is corrected so that the black levels in FIG. 24A match. Graphs 2411, 2412, 2413, and 2414 are graphs in which the black levels of graphs 2401, 2402, 2403, and 2404 are corrected, respectively. In the inter-channel gain correction, the sensor is uniformly irradiated with a constant amount of light indicated by the line 2415 to acquire one image, and each channel is obtained from the area of 400 pixels × 400 pixels at the center position of the acquired pixel output. Calculate the average of the output voltage of. The offset is subtracted from the average value of the output voltage of each channel, and the correction gain for each channel is calculated so that the subtraction result becomes a constant value. This correction gain is calculated in each case where the sensor temperature is 60 degrees and 30 degrees. The correction gain acquired in this way is stored in the ROM 106 as a correction value in association with the sensor number indicating the individual sensor, the channel number (numbers of channels A to D), and the temperature information.

Next, an example of the flow of the learning process in the fifth embodiment will be described. FIG. 25 is a flowchart showing an example of the flow of the learning process of the fifth embodiment. The flowchart of FIG. 25 corresponds to the block diagram of FIG. 7A, and the correction unit 704 of the image processing unit 205 performs interchannel gain correction. In S2501, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S2502, the control unit 209 inputs the training image of one of the acquired learning sets to the neural network 204. The output image is generated by the process performed by the neural network 204. In S2503, the extraction unit 702 of the image processing unit 205 extracts metadata from the teacher image. In S2504, the correction value acquisition unit 703 acquires the correction value corresponding to the combination of the sensor number of the extracted metadata and the temperature information from the ROM 106. In S2505, the correction unit 704 performs interchannel gain correction with the acquired correction value for the teacher image and the output image output from the neural network 204.

In S2506, the error evaluation unit 206 calculates the error between the image-processed output image and the teacher image. In S2507, the parameter adjustment unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S2508, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the control unit 209 determines NO in S2508, the control unit 209 returns the flow to S2501 because the predetermined end condition is not satisfied. On the other hand, if the control unit 209 determines YES in S2508, the control unit 209 satisfies the predetermined end condition, so that the flow proceeds to S2509.

In S2509, the control unit 209 stores the updated parameter (learned parameter) in the parameter storage area 208 which is different for each correction condition of the sensor number and the temperature information. The correction condition for the gain correction between channels in the present embodiment is a correction value corresponding to the combination of the sensor number and the temperature information. The control unit 209 stores the updated parameters in the parameter storage area 208, which is different for each combination of the sensor number and the temperature information. In S2510, the control unit 209 determines whether or not the parameters of all the conditions have been acquired. At this time, the control unit 209 determines S2510 based on whether all combinations of the sensor number and the temperature information have been acquired. If different parameters are prepared for each of the plurality of image pickup devices, it is sufficient to acquire only the parameters corresponding to the sensor numbers corresponding to the image pickup devices. If the control unit 209 determines YES in S2510, the control unit 209 ends the processing of the flowchart of FIG. 25. On the other hand, when the control unit 209 determines NO in S2510, the control unit 209 advances the flow to S2511. In S2511, the control unit 209 changes the condition of the combination of the sensor number and the temperature information. Then, the control unit 209 returns the flow to S2501. As a result, a learning process in which the combination of the sensor number and the temperature information is changed is newly performed.

Next, the inference processing of the fifth embodiment will be described. FIG. 26 is a flowchart showing an example of the flow of the inference processing according to the fifth embodiment. In S2601, the acquisition unit 501 acquires the sensor number and temperature information. The acquisition unit 501 can acquire the sensor number and the temperature information from the image pickup apparatus 120. In S2602, the parameter selection unit 502 selects the parameter corresponding to the acquired sensor number from the parameter storage areas 208-1 to 208-n. In this embodiment, since only the correction value at 30 degrees and the parameter at 60 degrees are stored as the temperature information, both the correction value at 30 degrees and the parameter at 60 degrees corresponding to the sensor number are selected. Of course, not only 30 degrees and 60 degrees but also parameters corresponding to many temperatures may be prepared in advance and the parameters corresponding to the temperature closest to the acquired temperature may be used. In S2603, the control unit 504 applies each selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204.

In S2604, the acquisition unit 501 acquires a captured image. In S2605, the control unit 504 inputs the acquired captured image to the neural network 204. In S2606, the neural network 204 performs inference processing. In the present embodiment, the inference process using the parameters at 30 degrees is performed to generate the first inference image, and the inference process using the parameters at 60 degrees is performed to generate the second inference image. In S2607, the inference image output unit 503 interpolates the first inference image and the second inference image based on the acquired temperature information, and generates an inference image according to the temperature information. Specifically, t is the acquired temperature information, α is the weighted addition coefficient, Z ₁ is the signal level of the coordinates (x, y) in the first inference image, and Z ₂ is the coordinates in the second inference image. Let the signal level be (x, y). In this case, the signal level Z _OUT (x, y) of the coordinates (x, y) in the inference image obtained by interpolating the first inference image and the second inference image uses the following equations 2 and 3. Can be asked.
α = (-1 / (60-30)) × t + 2 ・ ・ ・ (Equation 2)
Z _OUT (x, y) = α × Z ₁ (x, y) + (1-α) × Z ₂ (x, y) ... (Equation 3)

When the temperature information is less than 30 degrees, the temperature information may be regarded as 30 degrees, and when the temperature information exceeds 60 degrees, the temperature information may be regarded as 60 degrees. In S2608, the inference image output unit 503 outputs the inference image generated in S2206 to the storage device 140. The inference image output unit 503 may output the generated inference image to the ROM 106, the RAM 107, the display device 130, or the like.

As described above, in the first to fifth embodiments, as image processing, ISO sensitivity correction, F value correction, color suppression processing, color bending correction, color balance correction, peripheral light falloff correction, and interchannel gain correction are performed. Although a specific example of the parameters of the learning model has been described by taking the above as an example, the present invention is not limited to this. For example, there is flicker correction as an example of image processing different from the image processing described so far. The flicker correction is a process of correcting the difference in the luminance value generated between the lines of the sensor or between the frames due to the blinking of a light source such as a fluorescent lamp. The magnitude of flicker can be detected from the amount of change (amplitude) of the luminance value that changes depending on the line, and the frequency of flicker can be detected from the cycle of the luminance value that changes for each line. There is a feature that the generation method differs depending on the conditions such as flicker, brightness of the light source, blinking frequency, and accumulation time of the sensor. That is, the parameters of the learning model may be prepared for each of these conditions. Even in such a case, the same effect as that of the first to fifth embodiments can be obtained.

(Sixth Embodiment)
In the sixth embodiment, compression processing and decompression processing are taken as examples of image processing, and learning processing is performed using a teacher image and a training image that has been subjected to compression processing and decompression processing at each of a plurality of compression rates. A specific example of the parameters of the learning model obtained will be described. In the present embodiment, since the image input to the neural network 204 is compressed, the circuit scale of the neural network can be suppressed, which is different from the first to fifth embodiments.

In the present embodiment, the image processing unit 205 performs a compression process of reducing the pixel value to 1/m and a compression process of reducing the pixel value to 1/n. The compression processing method is not limited to this method, and any other method may be used as long as it is the same method as the compression processing used in the inference processing step.

FIG. 27 is a block diagram showing a function of learning processing of the image processing apparatus 100 according to the embodiment of the sixth embodiment. In the sixth embodiment, the image processing unit 205 is different from the first embodiment in that the image processing unit 205 is arranged on the input side and the output side of the neural network 204. The training image is input to the image processing unit 205, and the image processing unit 205 performs compression processing on the training image at one of a plurality of compression rates. The image compressed by the image processing unit 205 is input to the neural network 204, and the neural network 204 generates and outputs an output image. Then, the output image is input to the image processing unit 205 again, and the image processing unit 205 performs decompression processing corresponding to the compression rate in the compression processing on the output image. Then, the error evaluation unit 206 calculates the error between the output image stretched by the image processing unit 205 and the teacher image.

The learned parameters that have been compressed at different compression rates are stored in the parameter storage areas 208-1 and 208-2, respectively. Here, an example of two types of compression ratios has been described, but the compression ratios may be three or more. When there are n types of compression ratios, the parameter storage areas 208-1 to 208-n are used.

Next, an example of the flow of the learning process in the sixth embodiment will be described. FIG. 28 is a flowchart showing an example of the flow of the learning process of the first embodiment. In S2801, the control unit 209 acquires a pair (learning set) of the training image and the teacher image from the storage device 301. In S2802, the control unit 209 acquires the compression information. In the present embodiment, the compressed information is information in which a value of 0 is set for each pixel value of the training image when the pixel value is less than a predetermined threshold value and a value of 1 is set when the pixel value is greater than or equal to the threshold value. It is composed of the same number of data as the number of pixels of. The compressed information is not limited to the information in such a data format, and may be information in another data format as long as the information shows the compression processing method of the training image. Further, the control unit 209 may acquire the compression information stored in advance in either the image pickup device 120 or the storage device 140 in step S2802. Further, the image processing unit 205 may directly calculate the compression information from the training image in the following step S2803 without acquiring the compression information in step S2802.

In step S2803, the control unit 209 causes the image processing unit 205 to perform compression processing of the training image based on the compression information acquired in step S202. In the image processing unit 205, when the compression information is 0, the pixel value is reduced to 1/m, and when the compression information is 1, the pixel value is reduced to 1/n. The compression processing method is not limited to this method, and any other method may be used as long as it is the same method as the compression processing used in the inference processing step. Further, the training image acquired in step S2801 may have already been compressed. In this case, since it is not necessary to perform the compression process of the training image, the process directly proceeds from step S2801 to step S2804 described later. Further, when there are a plurality of operation modes of the image pickup apparatus 120 and the processing method of the compression processing is different for each, the compression information acquired in step S2802 may be changed or the presence / absence of the compression processing may be switched according to the operation mode. good.

In S2804, the control unit 209 inputs the training image compressed in step S2803 to the neural network 204 to generate an output image. Subsequently, in step S2805, the image processing unit 205 executes decompression processing on the output image generated in step S2804, and the error evaluation unit 206 calculates the error between the decompressed output image and the correct answer image.

In the present embodiment, the image processing unit 205 performs a decompression process that is the reverse of the compression process performed on the training image in step S2803. Specifically, when the compression information is 0, the pixel value is multiplied by m, and when the compression information is 1, the pixel value is multiplied by n. However, the method of the decompression process is not limited to the method of the present embodiment, and any other method may be used as long as it is the same method as the decompression process used in the step of the inference process. Further, when there are a plurality of operation modes of the image pickup apparatus 120 and the processing methods of the compression processing and the decompression processing are different for each, the processing method of the compression processing and the decompression processing and the presence / absence of the compression processing and the decompression processing are performed according to the operation mode. May be switched. By using the same decompression processing method for the decompression process used when decompressing the inferred image and the decompression process performed on the output image in step S2805, the accuracy is more stable regardless of the amount of noise after the decompression process. Inference is possible. Subsequently, in step S2806, the parameter adjusting unit 207 updates each parameter of the neural network 204 by using the error back propagation method so that the calculated error becomes small. In S2807, the control unit 209 determines whether or not the predetermined end condition is satisfied. When the control unit 209 determines NO in S2807, the control unit 209 returns the flow to S2801. In this case, the processing of S2801 to S2806 is performed using the learning set of the new training image and the correct answer image. On the other hand, if the control unit 209 determines YES in S2807, the control unit 209 advances the flow to S2808.

In step S2808, the control unit 209 stores the updated parameters, the structure of the neural network, and the like in the parameter storage area 208-1 or 208-2. In S2809, the control unit 209 determines whether or not the parameters for all the compression information have been acquired. When the control unit 209 determines NO in S2809, the control unit 209 returns the flow to S2801 and acquires other compression information in S2802. If the control unit 209 determines YES in S2809, the control unit 209 ends the processing of the flowchart of FIG. 28. By the above operation, it is possible to obtain a neural network in which the inference accuracy is not easily affected by the decompressed image without increasing the circuit scale.

FIG. 29 is a block diagram showing a function of inference processing of the image processing apparatus 100. The difference is that the image processing unit 205 is arranged between the acquisition unit 501 and the neural network 204 and between the neural network 204 and the inference image output unit 503 with respect to the block shown in FIG. The image processing unit 205 performs image compression processing and decompression processing.

FIG. 30 is a flowchart showing an example of the flow of inference processing in the sixth embodiment. In S3001, the acquisition unit 501 acquires the compression information. The acquisition unit 501 can acquire compression information from the image pickup apparatus 120. For example, the acquisition unit 501 can acquire compression information based on the setting conditions of the image pickup apparatus 120. This compression information is in the same format as the compression process used in the learning process. Specifically, in this embodiment, the information is set with a value of 0 when the pixel value is less than a predetermined threshold value and 1 when the pixel value is greater than or equal to the threshold value with respect to the pixel value of the captured image. The compressed information is not limited to this format, and may be information in another data format as long as it is the same as the compression process used in the learning process. In S3002, the parameter selection unit 502 selects the parameter corresponding to the acquired compression information from the parameter storage area 208-1 or 208-2. Then, in S3003, the control unit 504 applies the selected parameter to the neural network 204. As a result, it becomes possible to perform inference processing by the neural network 204.

In S3004, the acquisition unit 501 acquires an captured image from the image pickup device 120. This captured image is an uncompressed RAW image, and is composed of the same number of data as the number of pixels of the training image. The compression information acquired at this time is in the same format as the compression process used in the learning process. The acquisition unit 501 may acquire a RAW image from the storage device 140. In S3005, the compression unit 1901 performs a compression process on the acquired image according to the compression information acquired in S3001. The image processing unit 205 performs compression processing by reducing the pixel value to 1/m when the compression information is 0 and reducing the pixel value to 1/n when the compression information is 1. The compression processing method is not limited to this method, and any other method may be used as long as it can reduce the amount of data in the captured image. Further, the captured image acquired in S3004 may have already been compressed. In this case, since it is not necessary to perform the compression process of the captured image, the process proceeds from S3004 to S3006. Further, when there are a plurality of operation modes of the image pickup apparatus 120 and the compression processing method is different for each, the compression information acquired in step S3001 may be changed or the presence / absence of the compression processing may be switched according to the operation mode. ..

In S3006, the control unit 504 inputs the compressed captured image to the neural network 204. In S3007, the neural network 204 performs inference processing and generates an inference image. In S3008, the image processing unit 205 executes decompression processing on the inferred image generated in step S3007. In the present embodiment, the image processing unit 205 performs a decompression process that is the reverse of the compression process performed on the captured image in step S3005. Specifically, when the compression information is 0, the pixel value is multiplied by m, and when the compression information is 1, the pixel value is multiplied by n. However, the method of the decompression process is not limited to the method of the present embodiment, and any other method may be used as long as it is the same method as the decompression process used in the step of the learning process. Further, when the decompression process is separately performed in the subsequent stage, the decompression process may not be performed here. In S3009, the inference image output unit 503 outputs the decompressed inference image to the storage device 140. The inference image output unit 503 may output the generated inference image to the ROM 106, the RAM 107, the display device 130, or the like.

According to this embodiment, since the image input to the neural network 204 is compressed, the circuit scale of the neural network can be suppressed. Further, since each network parameter is updated in the learning process after the expansion process is performed on the output image from the neural network 204, it is possible to suppress the decrease in the noise suppression effect of the neural network for inference due to the compression process. ..

Although the first to sixth embodiments have been described, the parameters of the neural network according to the combination of the correction conditions in each embodiment may be used. That is, ISO sensitivity correction, F value correction, color suppression processing, color bending correction, color balance correction, peripheral light falloff correction, interchannel gain correction, flicker correction, and correction that combines two or more of compression and decompression processing. You may use parameters according to the conditions.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist thereof. The present invention supplies a program that realizes one or more functions of each of the above-described embodiments to a system or device via a network or storage medium, and one or more processors of the computer of the system or device implements the program. It can also be realized by the process of reading and executing. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image processing device 103 Image processing circuit 105 CPU
106 RAM
109 GPU
108 Metadata extraction circuit 120 Imaging device 140 Storage device 204 Neural network 205 Image processing unit 206 Error evaluation unit 207 Parameter adjustment unit 208 Parameter storage area 501 Acquisition unit 502 Parameter selection unit

Claims (27)

An image processing means that performs multiple image processing on training images and teacher images,
A learning means for performing machine learning of a learning model using the training image and the teacher image to which the plurality of image processes have been applied, respectively.
A control means for controlling that a plurality of parameters for the learning model learned in each of the plurality of image processes are saved in association with the correction conditions of the image processing.
An image processing device characterized by comprising. A selection means for selecting one parameter from a plurality of stored parameters according to the information regarding the correction condition of the image processing, and a selection means.
An inference means that applies the selected parameters to the learning model to perform inference processing on the image.
The image processing apparatus according to claim 1, further comprising. The image processing apparatus according to claim 1 or 2, wherein the training image to which the image processing is performed is processed by the learning model. The image processing apparatus according to any one of claims 1 to 3, wherein the training image is an image in which noise is added to the teacher image. The image processing includes sensitivity correction that corrects the analog gain according to the combination of the column amplifier and the lamp, correction that multiplies the image by the gain according to the aperture value so that the relationship between the lens aperture value and the amount of light approaches linear, and color. At least one of color suppression processing to correct the degree, color bending correction to correct color change, color balance correction, peripheral light falloff correction, correction based on the gain difference between sensor channels, flicker correction, and compression processing. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is any one of the above. The image processing includes sensitivity correction that corrects the analog gain according to the combination of the column amplifier and the lamp, correction that multiplies the image by the gain according to the aperture value so that the relationship between the lens aperture value and the amount of light approaches linear, and color. At least one of color suppression processing to correct the degree, color bending correction to correct color change, color balance correction, peripheral light falloff correction, correction based on the gain difference between sensor channels, flicker correction, and compression processing. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is a combination of any two or more. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing is an ISO sensitivity correction that corrects an analog gain according to a combination of a column amplifier and a lamp. The image processing apparatus according to claim 7, wherein the correction condition for the image processing is ISO sensitivity, and the control means stores the plurality of parameters in association with the ISO sensitivity. The image processing according to claim 7, wherein the correction condition of the image processing is a combination of ISO sensitivity and temperature, and the control means stores the plurality of parameters in association with the combination of ISO sensitivity and temperature. Device. The image processing is according to any one of claims 1 to 4, wherein the image processing is a correction for multiplying the image by a gain according to the aperture value so that the relationship between the aperture value of the lens and the amount of light approaches linearly. The image processing device described. The image processing apparatus according to claim 10, wherein the correction condition for the image processing is an aperture value, and the control means stores the plurality of parameters in association with the aperture value. The image processing apparatus according to claim 10, wherein the control means has a different grain size of the parameter to be stored in a section in which the relationship between the aperture value and the gain is linear and a non-linear section. The image according to any one of claims 1 to 4, wherein the image processing is at least one of a color suppression process for correcting saturation and a color bending correction for correcting color change. Processing device. 13. The correction condition of the image processing is at least one of gamma and color gamut, and the control means stores the plurality of parameters in association with at least one of gamma and color gamut according to claim 13. The image processing device described. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing is peripheral light falloff correction. The correction condition of the image processing is at least one of the lens type, the aperture value, the zoom position, and the focal length, and the control means is at least one of the lens type, the aperture value, the zoom position, and the focal length. The image processing apparatus according to claim 15, wherein the plurality of parameters are stored in association with the above. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing is a correction based on a gain difference between the channels of the sensor. The image processing apparatus according to claim 17, wherein the correction condition of the image processing is a sensor and a temperature, and the control means stores the plurality of parameters in association with the individual sensor and the temperature. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing is flicker correction. The correction condition of the image processing is at least one of the brightness of the light source, the frequency of blinking of the light source, and the accumulation time of the sensor, and the control means is the brightness of the light source, the frequency of blinking of the light source, and the sensor. The image processing apparatus according to claim 19, wherein the plurality of parameters are stored in association with at least one of the storage times. The image processing apparatus according to claim 1, wherein the image processing is a compression process. The image processing apparatus according to claim 21, wherein the correction condition for the image processing is a compression rate, and the control means stores the plurality of parameters in association with the compression rate. A selection means for selecting one parameter from a plurality of stored parameters according to the compression ratio, and
An inference means that applies the selected parameters to the learning model and performs inference processing on the image compressed at the compression rate.
The image processing apparatus according to claim 22, further comprising. A storage means that stores multiple parameters for the training model trained in each of the multiple image processes in different areas.
A selection means for selecting one parameter from a plurality of stored parameters according to the information regarding the correction condition of the image processing, and a selection means.
An inference means that applies the selected parameters to the learning model to perform inference processing on the image.
An image processing device characterized by comprising. The process of performing multiple image processing on training images and teacher images,
A process of performing machine learning of a learning model using the training image and the teacher image to which the plurality of image processes have been applied, respectively.
A step of saving a plurality of parameters for the learning model learned in each of the plurality of image processes in association with the correction conditions of the image processing.
A method for controlling an image processing apparatus, which comprises. The process of storing multiple parameters for the training model trained in each of multiple image processes in different areas, and
A step of selecting one parameter from a plurality of stored parameters according to the information regarding the correction condition of the image processing, and a step of selecting one parameter.
A step of applying the selected parameters to the learning model to perform inference processing on the image, and
A method for controlling an image processing apparatus, which comprises. A program for causing a computer to execute each means of the image processing apparatus according to any one of claims 1 to 24.

Images