JPH04200081A - Image pickup device - Google Patents

Abstract

PURPOSE:To allow the correction of lens aberrations by correcting the electric signal detected by a CCD using the learning function of neural networks. CONSTITUTION:Digitalized video signals are inputted to the neural networks 6a to 6n, 7a to 7n, 8a to 8n previously learned by plural training images in such a manner that the light images condensed and inputted by a lens 1 are converted to electric signals by the CCD 2 and a prescribed number of the sample signals are taken out of the digitalized video signals including the aberrations so as to correct the aberrations. The weights of the prescribed digital signals successively inputted from A/D converters 4a to 4n and the threshold are minimized in accordance with the weight previously learned by learning computing means and certain threshold data in the neural networks 6a to 6n, 7a to 7n, 8a to 8n, by which the lens aberrations are corrected. The lens aberrations are corrected and outputted in this way.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to an imaging device, and particularly to an imaging device suitable for correcting lens aberrations using a neural network.

(Prior art) Generally, when light is focused and imaged by a lens, it deviates from perfect image formation, resulting in optical aberration (hereinafter referred to as lens aberration).
In some cases, a deviation called .

Examples of this lens aberration include spherical aberration, comatic aberration, astigmatism, and chromatic aberration, as well as curvature and distortion of the field surface.

The CCD receives the light image focused by the lens.
(charge-coupled device)
There are imaging devices that detect video signals from imaging devices such as
In this imaging device, a correction lens is also used in the front group to correct lens aberrations, and an aspherical lens is used.

(Problem to be Solved by the Invention) However, in correcting lens aberrations using the lens described above, the front lens group is also used, which increases the size of the lens system, and the use of an aspherical lens complicates lens processing. It has become.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging device that corrects lens aberrations by correcting electrical signals detected by a CCD using a learning function of a neural network.

(Means for Solving the Problems) In order to achieve the above object, the present invention includes an optical lens means, a photoelectric conversion means for converting an optical image focused by the optical lens means into an electric signal, and a an A/D conversion means for converting the electrical signal from the photoelectric conversion means into a digital signal; a plurality of two-neelon units to which predetermined digital signals are input from the A/D conversion means; and a plurality of neuron units. A learning calculation means for pre-learning weights and threshold data to be associated with each of the neuron units using a predetermined training image, and the plurality of two-neelon units are connected between adjacent layers in a multi-stage hierarchical structure. a neural network that corrects lens aberrations by minimizing the weights and thresholds of the predetermined digital signals sequentially inputted from the A/D converter based on the weights and threshold data learned by the learning calculation means in advance; An imaging device having the following can be provided.

(Function) According to the imaging device configured as described above, the optical image collected by the lens and inputted is converted into an electrical signal by the CCD,
A predetermined number of sample signals are extracted from a digitized video signal including aberrations. Then, it is possible to provide an imaging device that corrects lens aberrations by inputting the digitized video signal to a neural network trained in advance using a plurality of training pixels to correct aberrations.

(Example) Hereinafter, an example of the present invention will be described in detail with reference to the drawings.

The block diagram in FIG. 1 shows a schematic configuration of an imaging device according to the present invention.

In the configuration of this imaging device, first, an optical signal focused by a lens 1 is incident on a pixel of a CCD 2, and a video signal (electrical signal) 2a is detected from the CCD 2 and sent to a process processing device 3. This electrical signal includes, for example, chromatic aberration due to the lens.

After performing predetermined processing, the process processing device 3 converts the signal into, for example, an RBG signal and sends it to the A/D converters 4a to 4n. The RBG signal is sent to the A/D converters 4a to 4n.
The signals are converted into digital signals and stored in memory elements 5a to 5n. The processed signal here is RBG
If it is a signal, there are three types of signals, and the A/D converter 48
4n and memory elements 5a to 5n can be implemented in three sets. Of course, the processed signal is not limited to the RBG signal, and may be in other forms such as y and c signals. Further, the electric signal from the C0D2 is the lens aberration described above, and is not limited to only the chromatic aberration.

The RBG image signals stored in the memory elements 5a-5n are divided into MXN neural networks 6a-6n, 7a-7n, 8 as shown in FIG.
It is sent to a to 8n. By dividing the image into blocks in this manner, the number of pixels included in one block can be reduced, the neural network can be downsized, and the learning time can be shortened.

The aberration-corrected RBG signal sent from the neural network, which has been trained in advance to correct aberrations using a method that will be described later, is input to the image memory element 9.

Under the control of the host computer 10, the image memory element 9 reads out an aberration-corrected image signal and outputs it for display.

In addition, this series of learning and processing is carried out by the host computer.
Executed under the control of 0.

Here, the conceptual configuration of the neural network used in the example is shown in FIG. 3(a).

The neural network shown in FIG. 3(a) has a three-layer hierarchical structure, and there is no connection between neuron units belonging to the same layer, and only connections between neuron units in adjacent layers. The layers are called an input layer 11, an intermediate layer 12, and an output layer 13 from the input side.

The signal input to the input layer 11 is as shown in FIG.
) corresponds to each pixel of the CCD shown in FIG.

Next, each two-neelon unit performs processing based on the following equation: y-f(ΣW,.Xl-θ) (1). Here, the symbol X1 is the output of the first two-neelon unit belonging to the layer immediately before the layer to which the neuron of interest belongs, and the symbol W is the strength of the connection with the first neuron unit. It is a weight that represents the

The symbol n is 1 to which the neuron unit of interest belongs.
It is the number of all two-neelon units belonging to the previous layer, the symbol θ is the input threshold of the neuron unit of interest, and the symbol y is the output value. Furthermore, the symbol f is expressed by the following formula, f (x) = 1/ (1+e-x) −
This is the Sigmond function shown in 12).

Next, FIG. 4 is a block diagram showing the configuration of the neural network of this embodiment, and FIG. 5 is a block diagram showing the configuration of the two-needle unit making up the neural network.

The configuration of this neural network is A (not shown).
The RBG signals digitized by the /D converter are transferred from the memory element 2o to the four neuron units 22a to 22d via the buffer 21.
A BG signal is input.

:, :Te4;i,4) Although the number of two-knee long units belonging to the interlayer is four, it is not limited to this. Further, the number of neuron units in the output layer is the number of pixels in one block of the image sXt. Further, the memory element 2
o corresponds to the input layer.

That is, the data stored in the memory element 2o corresponds to the outputs of the neuron units 22a to 22d belonging to the input layer.

The first data in the memory device 20 is stored in the buffer 21.
through neuron units 22a to 22d, 23a
~23n.

Each two-neelon unit sends an aberration-corrected output signal to the image memory element 24 based on learning data learned using a method described later. That is, the value stored in the image memory element 24 is an aberration-corrected video signal.

The host computer 25 reads this value from the image memory device 24 and outputs and displays it.

Further, the controller 26 controls the neuron unit 2
A synchronizing signal is generated for operating 2a to 22d and 23a to 23n simultaneously and in parallel.

Then, as shown in FIG. 5, the configuration of each two-neelon unit will be shown, and the creation of learning data for the neural network will be explained.

The N operator 26 in the neuron unit 22 calculates the product of the first data of the connection weights stored in advance in the weight memory element 27 and the first data sent from the memory element 20 (not shown). is calculated and stored in register 28. Next, the product of the second data of the memory element 20 and the second data of the weight memory element 27 is calculated, and the sum of the product and the contents of the register 28 is calculated and stored in the register 28 again. .

Similar processing is repeatedly performed on the data in the memory element 2o, and when the processing of the last data stored in the memory element 2o is completed, the register 28 contains the two-neelon unit of the above-mentioned formula (1). Output x1 and weight w
The sum of products with l up to n is stored. Here, the symbol n is the number of data stored in the memory element 20 and corresponds to the number of input units.

Next, based on the contents of the register 28, the threshold value memory element 2 is set in advance.
The value corresponding to the input threshold θ of the neuron unit in the equation (1) stored in the function table 3o is extracted, and the value f (x) in the equation (2) stored in the function table 3o is extracted.
The output y of the above equation (1) is determined based on the correspondence between
is stored in

Such processing is carried out by the neuron units 22a to 22d.
are executed simultaneously and in parallel.

Furthermore, the same processing as that for the two-needle units 22a-22d described above is performed on the neuron units 23a-23n.

However, regarding the input data, the neuron unit 2
2a to 22d, the input data is input from the memory element 20, but the neuron units 23a to 23n use data sequentially read from the output buffer 31 of the neuron units 22a to 22d as input data.

First, the data of the output buffer of the neuron unit 22a is read out, and each of the two neuron units 23a to
23n, the product of the weight of the connection with the neuron unit 22a stored in advance in the weight memory element 27 is calculated by the arithmetic unit 26 and stored in the register 28.

Next, the product of the contents of the output buffer 31 of the neuron unit 22b and the weight of the connection with the neuron unit 22b stored in the weight memory element 27 of each two-neelon unit 23a to 23n is calculated, and
After adding it to the contents of , it is stored in the register 28 again.

This process is repeated for the second neuron unit 22C122d, and each neuron unit 23a to 23n
The register 28 stores the sum of the products of the output X of the neuron unit in equation (1) and the weight Wl up to "4". Here, "4" is the number of intermediate units.

Next, from the contents of the register 28, the threshold value memory element 2 is set in advance.
Each two kneelon units 238 to 23 stored in 3
The threshold value of n is taken out, and the output y of the equation (1) is obtained from the correspondence between f(x) and X in the equation (2), which is stored in advance in the function table 30, and is sent to the image memory device 24 through the output buffer 31 of the image data.

Such processing is performed on the memory elements 48 to 4n in FIG.
The neural networks 6a to 6n, 7a, to 7n, and 8a to 8n simultaneously and in parallel perform the processing on the R80 images stored in the R80 images.

First, the weight W and threshold value θ in equation (1) of the neuron units belonging to the intermediate layer and the output layer of the neural network are determined using a normal Neumann-type host computer 25.
This is done by the backpropagation algorithm (also known as the generalized delta rule) devised by Art and others (D.E., Rumelhart, J.L., McClelland.

Neural networks 6a to 6n, 7a to 7n were applied to the results obtained by PDP Research Group, translated by Toshi Seri, Chapter 8, Sangyo Tosho, 1989) before normal imaging.

Weight memory element 2 of each neuron unit 8a to 8n
7 and threshold memory element 29, respectively.

A method for determining the weight W and the threshold value θ, which correspond to the creation of learning data for the neural network, will be explained with reference to the components shown in FIG.

Then, training images (for example, color charts, various display images, etc.) are captured by the above-mentioned imaging device shown in FIG.

The captured training images are processed into RG in process processing 3.
It is converted into a B signal and output. The RGB signals are digitized by the A/D converters 4a to 4n and stored in the memory elements 5a to 5n, respectively.

This stored RGB signal is divided into blocks of sXt and read into the host computer 10, and at the same time the original training image is divided into blocks of size sXt to be used as teacher data. This teacher data is registered as learning data together with the data of the previous memory elements 5a to 5n.

Next, learning data is created and registered for a large number of other training images using the same procedure to obtain a plurality of learning data.

The learning data obtained in this way is converted into a vector (ma:, K
H)-<ma:xN, "i to -s)" Here, vector X:k(l-IL...2m1J-1,
..., M, -1, ..., N) is a vector representing data processed by the jk-th neural network for one training image. The number of elements of this vector matches the number of pixels included in the block, and becomes the number of neuron units in the input layer of the neural network.

Also, vector subtraction k(i-1,..., ms j-
1°..., M, and -1,..., N) are vector patterns whose chromatic aberrations are corrected, i.e., consistent with the original image and desired for the output layer of the neural network, and the number of elements is , is consistent with the number of incoming crab nits.

Each neural network based on the pack propagation algorithm is E”−−Σ (””'
y lk p rh) 2 ... (3) 2
The weight and threshold value of each two-knee-long unit are changed so as to minimize +-1. Here, vector V rh(i =
1. ..., m) is the learning data vector ma Ik jk
This is the output of the neural network obtained when inputting to the th neural network.

Also, if m sets of data are input once each and the weight is changed so that Ejk is decreased the most, the amount of change in weight between the two-neelon unit f in the output layer and the neuron unit g in the middle layer is ΔW+ ' is given by, and the amount of change in weight △W+' between the two-neelon unit g in the intermediate layer and the unit h in the input layer is given by ΔW: mg >R'4(yjk''-d; ″) (1-
)”,”) V’,”W;:’×(l yl'l
' ) ylkp Given by yIkp. Here, the symbol ε is a positive constant (out, the symbols ylkp, yIk
p is the output of the output layer unit f and the intermediate layer unit g when the learning data vector T I k is input to the jk-th neural network, respectively. Symbol yI
kl' is the h-th element of vector ma1', and the symbol r is the number of output kaninits. Regarding the threshold value θ, if we consider that the weight of the terminal that always inputs "-1" for each two-neelon unit, the above equation (4) and the above (
5) It can be determined by the formula.

For each connection weight, the above formula (4) and the above (
5) The process is completed by repeatedly executing the equation (3) when Ejk in equation (3) becomes sufficiently small, and the weight value at that time is stored in the weight memory element 27 and threshold memory element 29 of each two-knee-long unit. Store it.

Such processing is performed for all MXN neural networks that process each image of RGB signals.

Therefore, the weight value and the minimum threshold value are repeatedly determined, and when the output Elk becomes sufficiently small, it is considered that the correction of the lens aberration is completed.

In the imaging device of the present invention as described above, the input optical signal is C
Since the aberrations of the lens are corrected using a neural network after the signal is converted into an electrical signal by the CD, there is no need for the complicated design of the lens as in the past.
Since the neural network is trained for each individual lens and aberrations are corrected, there is no need to improve the accuracy of each individual lens.

Furthermore, since aberrations can be corrected regardless of the combination of multiple lenses, the number of lens groups can be reduced, and by dividing the electrical signals from the CCD into multiple blocks, the number of memory elements can be reduced. can be reduced.

Furthermore, the imaging apparatus of the present invention can correct lens aberrations in real time by following digital signals that are sequentially input.

Further, the present invention is not limited to the one embodiment described above, and it goes without saying that various modifications and applications can be made without departing from the gist of the invention.

[Effects of the Invention] As described in detail above, according to the present invention, digital signals obtained by digitizing and predetermined divisions from a focused optical signal are sequentially input to a pre-trained neural network,
It is possible to provide an imaging device that outputs an image with lens aberrations corrected.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing a schematic configuration of an imaging device according to an embodiment of the present invention, and FIG. 2 shows an RBG image signal of MxN
Figure 3 (a) is a configuration diagram showing the conceptual configuration of the neural network, Figure 3 (b)
) is a diagram showing the structure of each pixel of the CCD corresponding to the input signal of the input layer, FIG. 4 is a block diagram showing one structure of a neural network, and FIG. 5 is a block diagram showing the structure of a neuron unit. 1... Lens, 2... CCD, 2a... Video signal (electrical signal), 3... Process processing device, 4a to 4n
...A/D converter, 5a to 5n...Memory element, 6
a to 6n, 7a to 7n, 8a to 8n...-neural network, 9...image memory element, 10...host computer. Applicant's agent Patent attorney Tsuboi Atsube 20 Article 30(a)

Claims (1)

[Scope of Claims] Optical lens means; photoelectric conversion means for converting a light image focused by the optical lens means into an electrical signal; and electrical signals from the photoelectric conversion means are converted into digital signals. a plurality of neuron units into which a predetermined digital signal is input from the A/D converter; and a weight that is associated with each of the neuron units by using a predetermined training image for the plurality of neuron units. a learning calculation means for learning threshold data in advance; the plurality of neuron units are configured by connecting adjacent layers in a multi-stage hierarchical structure; the learning calculation means learns the A/ An imaging apparatus comprising: a neural network that corrects lens aberrations by minimizing weights and thresholds of the predetermined digital signals sequentially input from a D-converting means.