JP3662075B2 - Image processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus that performs image data processing corresponding to an operation of combining a plurality of images by overlapping (joining) them together with overlapping portions. The present invention is effective when applied to obtain a high-definition image based on an image having the number of pixels.
[0002]
[Prior art]
FIG. 6 is a conceptual diagram showing a state in which a plurality of images to be subjected to the bonding process are generated by a plurality of imaging elements. In FIG. 6, the imaging field of view FVA through the optical system LA of the imaging device SA and the imaging field of view FVB through the optical system LB of the imaging device SB are adjacent to each other with an overlapping portion W. Images (image data) and the like corresponding to the imaging fields of view FVA and FVB are examples of images to be subjected to a pasting (joining) process in this type of image processing apparatus.
[0003]
FIG. 7 is a schematic diagram showing a state of one continuous image obtained by pasting two images generated by the imaging method as shown in FIG.
FIG. 7A shows the state of an image obtained by simply pasting the images corresponding to the imaging fields of view FVA and FVB. In the case of (A), when both the images are pasted together, the geometrical characteristics and gradation characteristics of the images are aligned and joined together with a linear boundary. There is a problem that the boundary line of the joint portion becomes conspicuous as shown in the figure due to the difference in the signal (data) representing the image, that is, the video signal level, due to the sensitivity difference between the two image pickup devices SA and SB.
[0004]
In order to alleviate the problem that the boundary line is conspicuous as in the case of (A), the part (B) in FIG. It is a schematic diagram which shows the image obtained by giving (smoothing).
A method for obtaining an image such as (B) is disclosed in, for example, Japanese Patent Application Laid-Open No. 57-9168.
[0005]
FIG. 8 is a block diagram showing the configuration of a conventional apparatus. The apparatus shown in FIG. 8 has substantially the same configuration as the apparatus disclosed in the above publication. An image sensor 10 corresponding to the image sensor SA of FIG. 6 and an image sensor 20 corresponding to the image sensor SB of FIG. 6 are provided, and an output signal of the image sensor 10 is converted into an A / A signal through an analog signal processing unit 11 including an amplification function unit. The digital image data is supplied to the D converter 12 and converted into digital image data, and the digital image data is stored in the first memory 13. Similarly, an output signal of the image pickup device 20 is supplied to an A / D converter 22 through an analog signal processing unit 21 including an amplification function unit and converted into digital image data, and this digital image data is stored in the second memory 23. Is done.
[0006]
The output image data read from the first and second memories 13 and 23 is multiplied by the corresponding weighting coefficients (A) and (B) by the corresponding multipliers 14 and 24, and then the image data (C ) And (D) are supplied to one and the other input terminals of the adder 30, where they are added and synthesized.
[0007]
These image data (C) and (D) are images of respective areas corresponding to the imaging fields of view FVA and FVB of FIG.
[0008]
The multipliers 14 and 24 respectively add the corresponding weighting factors (A) and (B) from the mixing ratio distributor 40 in order to weight both data relating to the addition of the image data (C) and (D) in the adder 30. ) Are respectively supplied. The analog signal processing units 11 and 21 and the mixture ratio distributor 40 are operated under the control of the CPU 50.
[0009]
The output image data of the adder 30 is image data (E) corresponding to the image subjected to the pasting process shown in the part (B) of FIG. The image data (E) is supplied to the CPU 50 and recognized there.
[0010]
FIG. 9 is a timing chart showing the timing relationship of signals (data) of each part in the apparatus of FIG. 9, (A) to (E) are respectively associated with the signals or data (A) to (E) of the respective units described with reference to FIG. The region indicated by F1 in FIG. 9 is an image (image data) relating to the output of the image sensor 10 in FIG. 8 and corresponds to the imaging field of view FVA in FIG. 6, and indicated by F2 by the arrow. The area is an image (image data) relating to the output of the image sensor 20 in FIG. 8 and corresponds to the imaging field FVB in FIG. The regions F1 and F2 have an overlapping portion W corresponding to the overlapping portion W in FIGS. 6 and 7B.
[0011]
As shown in FIG. 9, one scanning line (C) belonging to one image F1 and another scanning line (D) belonging to another image F2 joined to the scanning line (C) are joined. In the region corresponding to the overlapping portion W, the weight of the video signal representing the one scanning line (C) (that is, the weighting coefficient A) is relatively large toward the side closer to the center of the one image F1. The adder 30 performs weighted addition processing by weighting such that the weight (that is, the weighting coefficient B) of the video signal representing the other scanning line (D) becomes relatively closer to the side closer to the center of the other image F2. Thus, a smoothing process is performed so as to make the boundary of the joining portion inconspicuous. In addition, the curve shown with the broken line in the (E) part of FIG. 9 shows the level of the scanning lines (C) and (D) before the weighting coefficients (A) and (B) are multiplied.
[0012]
As shown in the figure, when viewed on the time axis, the start point of the image F1 is t1, the end point is t3, the start point of the image F2 is t2, the end point is t4, and the start point of the overlapping portion W is t2. The end point is t3. The weighting coefficient (A) maintains 1 in the interval from t1 to t2, and gradually decreases from 1 to 0 during the period from the start end time t2 to the end time t3 of the overlapping portion W, while the weighting coefficient (B) overlaps. It gradually increases from 0 to 1 during the period from the start point t2 to the end point t3 of the portion W, and maintains 1 in the section from t3 to t4.
Generally, as the width of the overlapping portion W is increased, the boundary of the joining portion becomes inconspicuous.
[0013]
[Problems to be solved by the invention]
FIG. 10 is a diagram illustrating the characteristics of the smoothing process described above in the apparatus described with reference to FIGS. 8 and 9.
Part (A) of FIG. 10 shows a state in which the image F1 having a relatively low level of the image signal and the image F2 having a relatively high level are joined. Part (B) shows the transition of the image signal level when the smoothing process is applied to the overlapping part and connected. Part (C) shows the characteristics when noise is superimposed on the image signal. As shown in the part (C), in the overlapping part W subjected to the smoothing process, the noise level is suppressed by this process when viewed relative to the other part. As shown in part (D), in the overlapping part W, the noise may be improved by about 3 dB.
In this way, if the noise level is suppressed only in the overlapped portion subjected to the smoothing process, the stitched portion (overlapped portion) becomes more noticeable in the composite image resulting from the pasting process, and the corner smoothing process is performed. The effect of will be reduced.
[0014]
The present invention has been made in view of the circumstances as described above, and when a plurality of images including noise are bonded (joined) to create one image (image signal), smoothing is performed at the overlapping portion of the bonding. It is an object of the present invention to provide an apparatus of this type in which the joined portions (overlapping portions) are not more noticeable by processing.
[0015]
[Means and Actions for Solving the Problems]
In order to solve the above problems, one invention of the present application is:
An image processing apparatus that performs image data processing corresponding to an operation of combining a plurality of images by combining each scan line with overlapping portions,
One scan line belonging to one image and another scan line belonging to another image joined to the scan line are closer to the center of the one image in the region corresponding to the overlapping portion. Weighted addition processing by weighting so that the weight of the video signal representing one scanning line is relatively large and the weight of the video signal representing the other scanning line is relatively large toward the side closer to the center of the other image Smoothing means configured to make the boundary of the joining portion inconspicuous,
When the noise level related to the video signal is relatively low, the width of the overlapping portion is relatively widened, and when the noise level is relatively high, the width of the overlapping portion is relatively narrowed, and weighting by the smoothing means An operation mode changing means for changing the width of the region corresponding to the overlapping portion to be subjected to addition processing ;
An image processing apparatus comprising: ...... (1)
[0016]
Another invention of the present application is:
The image processing apparatus according to (1), wherein when the circuit gain is relatively high, the noise level is determined to be relatively high . ... (2)
[0017]
Yet another invention of the present application is:
The image processing apparatus according to (1), wherein when the video signal level is relatively high, it is determined that the noise level is relatively low . ...... (3)
[0018]
The features seen in various limited aspects of the present invention will be listed below.
The operation mode changing unit is configured to selectively change execution or non-execution of the weighted addition processing by the smoothing unit according to a noise level or a circuit gain as a state relating to the video signal. The image processing apparatus according to (1) above, characterized in that ... (4)
The operation mode changing means is configured to selectively change execution or non-execution of the weighted addition processing by the smoothing means according to the video signal level as a state relating to the video signal. The image processing apparatus described in (1) above, which is characterized ... (5)
[0019]
Embodiment
FIG. 1 is a block diagram showing an image processing apparatus as a first embodiment of the present invention. The embodiment shown in FIG. 1 has substantially the same configuration on the block diagram as the apparatus described with reference to FIG. 8, but an operation mode changing function that performs an operation mode changing function described later in place of the CPU 50 shown in FIG. The CPU 500 provided with the unit 501 is applied. For convenience of later explanation, symbols (different from those in FIG. 8) are used for signals (data) of each part.
[0020]
In FIG. 1, an image sensor 10 corresponding to the image sensor SA of FIG. 6 and an image sensor 20 corresponding to the image sensor SB of FIG. 6 are provided, and an output signal of the image sensor 10 is a known analog signal including an amplification function unit. The digital image data is supplied to the A / D converter 12 through the processing unit 11 and converted into digital image data, and the digital image data is stored in the first memory 13. Similarly, an output signal of the image sensor 20 is supplied to an A / D converter 22 through a known analog signal processing unit 21 including an amplification function unit and converted into digital image data, and the digital image data is converted into the second memory 23. Stored in
[0021]
Image data e1 (E1) corresponding to the output of the image sensor 10 read from the first memory 13 and image data e2 (E2) corresponding to the output of the image sensor 20 read from the second memory 14 ) Is multiplied by the corresponding weighting coefficients C1 (c1) and C2 (c2) by the corresponding multipliers 14 and 24, respectively, and one of the adders 30 as image data e3 (E3) and e4 (E4) and Each is supplied to the other input terminal, where it is added and synthesized.
[0022]
The image data e1 (E1) and e2 (E2), and thus the image data e3 (E3) and e4 (E4), are in the region corresponding to the imaging fields FVA and FVB in FIG. It is an image. Note that the symbols representing the above data indicated by lower case letters and numbers indicate data when the noise level is relatively low, and those indicated by the upper case letters and numbers in parentheses are relative to the noise level. This is a convenient notation showing data in the case of high, and so on.
[0023]
The multipliers 14 and 24 add the corresponding weights from the mixing ratio distributor 40 in order to weight both data with respect to the addition of both input image data e3 (E3) and e4 (E4) by the adder 30 in the subsequent stage. Coefficients C1 (c1) and C2 (c2) are respectively supplied. The analog signal processors 11 and 21 and the mixing ratio distributor 40 are operated under the control of the CPU 500. As described above, the CPU 500 includes the operation mode change function unit 501 that performs the operation mode change function.
[0024]
The output image data of the adder 30 is image data e5 (E5) corresponding to the image subjected to the pasting process, which is an output of the present apparatus. The image data e5 (E5) is supplied to the CPU 500 and recognized there.
[0025]
FIG. 2 is a timing chart showing the timing relationship of signals (data) of each part in the apparatus of FIG. In FIG. 2, (e1), (e2), (c1), (c2), (e3), (e4), and (e5) are data when the circuit gain is relatively low and the noise level is relatively low. (E1), (E2), (C1), (C2), (E3), (E4), and (E5) show data when the circuit gain is relatively high and the noise level is relatively high. Has been. These are associated with the codes of the signals or data of each part described with reference to FIG. Note that e1 and E1 and e2 and E2 are conceptual notations that serve both cases.
[0026]
The area indicated by F1 in FIG. 2 as an arrow F1 is an image (image data) relating to the output of the image sensor 10 in FIG. 1, and corresponds to the imaging field of view FVA in FIG. The area is an image (image data) relating to the output of the image sensor 20 of FIG. 1, and corresponds to the imaging field of view FVB of FIG. The regions F1 and F2 have an overlapping portion W corresponding to the overlapping portion W in FIGS. 6 and 7B or an overlapping portion W ′ narrower than this.
[0027]
As shown in FIG. 2, one scan line e1 (E1) belonging to one image F1 and another scan line e2 (E2) belonging to another image F2 joined to the scan line e1 (E1) When the circuit gain of the apparatus is relatively small and the noise level is relatively low, the operation mode changing function of the CPU 500 is performed so that the smoothing process is performed so that the overlapping portion W is relatively wide. The unit 501 operates to adjust the output timing of the weighting coefficient output from the mixture ratio distributor 40.
[0028]
In the region corresponding to the overlapping portion W, the weight (c1) of the video signal representing the one scanning line e1 is relatively larger toward the side closer to the central portion of the one image F1, and the central portion of the other image F2. The weighting and adding process is performed by the adder 30 with weighting such that the weight (c2) of the video signal representing the other scanning line e2 becomes relatively larger toward the side closer to. Smoothing processing is performed so as to make the boundary of the joining portion inconspicuous with the wide overlapping portion W.
[0029]
As shown in the figure, when viewed on the time axis, the start point of the image F1 is t1, the end point is t3, the start point of the image F2 is t2, the end point is t4, and the start point of the overlapping portion W is t2. The end point is t3. The weighting coefficient c1 maintains 1 in the interval from t1 to t2, and gradually decreases from 1 to 0 during the period from the start end point t2 to the end point t3 of the overlapped portion W, while the weighting coefficient c2 is the start end of the overlapped portion W. It gradually increases from 0 to 1 during the period from the time point t2 to the end time point t3, and maintains 1 in the interval from t3 to t4.
[0030]
As described above, in general, the boundary of the joining portion becomes inconspicuous as the width of the overlapping portion W is increased. However, as described with reference to FIG. 10, when noise is superimposed on the signal, The noise level is suppressed only in the overlapped areas where the smoothing process has been performed, compared to the adjacent area, and the stitched parts (overlapped areas) become more noticeable in the resultant composite image. The effect of processing will be diminished. This tendency becomes more prominent as the width of the overlapping portion subjected to the smoothing process is wider.
[0031]
In the present embodiment, according to the level of the noise level superimposed on the image signal (video signal), when the noise level is relatively low, the width of the overlapped portion is set to a relatively wide W, and the noise level is relative. On the other hand, the problem that the boundary portion becomes conspicuous can be solved by performing the smoothing process when the width of the overlapping portion is relatively narrow as shown in FIG.
[0032]
In the present embodiment, when the amplification factor (gain) in the known analog signal processing units 11 and 21 including the amplification function unit is set low by a command from the CPU 500, the noise level is relatively set. Since the operation mode changing function unit 501 of the CPU 500 gives a command to the mixing ratio distributor 40 and the smoothing process is performed, the width of the overlapping portion to be subjected to the smoothing process is set to a relatively wide W, and the amplification factor (gain) is set high. Since the noise level is relatively high, the operation mode changing function unit 501 of the CPU 500 gives a command to the mixing ratio distribution unit 40 to perform the smoothing process, so that the width of the overlapping portion is relatively narrow. Control is performed as'.
[0033]
As a modified example having the same configuration on the block diagram as the embodiment described with reference to FIGS. 1 and 2, instead of adjusting the width of the overlapping portion to be subjected to the smoothing process according to the circuit gain (noise level), It is also possible to take a form in which the width of the overlapped portion to be subjected to the smoothing process is adjusted according to the signal level.
That is, since the noise level is relatively large in the low signal level region corresponding to the gamma characteristics of the video signal, the width of the overlapping portion where the smoothing process is performed is relatively narrow, and the signal level is sufficiently high. Since the noise level is sometimes relatively small, the width of the overlapping portion subjected to the smoothing process can be made relatively wide so that the effect of making the boundary portion inconspicuous is sufficient.
[0034]
FIG. 3 is a signal waveform diagram for explaining the operation of the embodiment described with reference to FIGS. 1 and 2 or the modification described above. Each represents the output signal (data) E5 in FIG. 1. When the noise level is relatively lowest in the upper part of the figure, the noise level is relatively highest in the lower part, and the middle part is between the two. It shows the time of the target state. In the figure, W1, W2, and W3 represent the widths of the overlapping portions to be smoothed in each case.
[0035]
As another embodiment of the present invention, the configuration on the block diagram is the same as that of the embodiment of FIG. 1, and the operation mode changing function unit 501 of the CPU 500 may be different. FIG. 4 is a timing chart showing the timing relationship of signals (data) of each part representing the operation in such another embodiment. 4, as in FIG. 2, (e1), (e2), (c1), (c2), (e3), (e4), and (e5) have relatively low circuit gains and relatively low noise levels. The low case data is shown. These are associated with the codes of the signals or data of each part described with reference to FIG. (E1 '), (E2'), (C1 '), (C2'), (E3 '), (E4'), and (E5 ') have relatively high circuit gains and relatively high noise levels. Case data is shown. Note that e1 and E1 ′, and e2 and E2 ′ are conceptual notations that serve both cases.
[0036]
As can be understood by comparing (c1), (c2) and (C1 ′), (C2 ′) in FIG. 4, in this embodiment, the circuit gain is relatively low and the noise level is relatively low. As in the above-described embodiment, the smoothing process is executed. However, when the circuit gain is relatively high and the noise level is relatively high, the smoothing process operation is not executed. Therefore, the problem that the noise is suppressed more than the adjacent region only in the overlapping portion where the smoothing process is performed, which occurs when the noise level is relatively high, is solved. The selection of execution / non-execution of the smoothing processing operation as described above is performed based on a command from the operation mode change function unit 501 of the CPU 500.
[0037]
As a modification of the embodiment described with reference to FIG. 4, switching between execution / non-execution of the smoothing processing operation based on whether or not the noise level is relatively high (whether or not the circuit gain is relatively high). Instead of performing the selection, a mode is also possible in which the smoothing operation is switched between execution and non-execution according to whether the level of the video signal (data) is relatively high or low.
[0038]
FIG. 5 is a signal waveform diagram for explaining the operation of the modification of the embodiment described with reference to FIG. FIG. 5A shows a case where an image is synthesized without performing smoothing processing when the level of the video signal is relatively high (noise level is relatively low). This is a case where the level difference of the video signal (data) level is noticeable and the boundary is conspicuous, resulting in poor jointing. In FIG. 5B below, when the level of the video signal is in the same state as in (A), the step of the video signal (data) level becomes inconspicuous at the boundary portion of the stitching by performing the smoothing process. This is a good case.
[0039]
FIG. 5C shows a case where an image is synthesized without performing smoothing processing when the level of the video signal is relatively low (the noise level is relatively high). This is a case where the noise level is substantially the same as the other parts, the boundary is not conspicuous, and the connection is good. In FIG. 5D below, when the level of the video signal is in the same state as in FIG. 5C, the smoothing process is performed, so that the noise level becomes lower than the other part at the boundary part of the joint, and the boundary is conspicuous. This is a case of poor jointing.
[0040]
In the embodiment described above with reference to FIG. 5, the processing as shown in FIG. 5C is performed on a region where the signal level of the gamma characteristic portion of the video signal is low, and the region where the video signal level is high is shown in FIG. By performing the treatment as in (B), good characteristics can be obtained. Such selection of the smoothing processing mode is performed based on a command from the operation mode changing function unit 501 of the CPU 500.
[0041]
Below, the invention seen in the various situation contained in this application is enumerated, and those effects are described together.
[0042]
(1) An image processing apparatus that performs image data processing corresponding to an operation of combining a plurality of images by combining each scan line with overlapping portions.
One scan line belonging to one image and another scan line belonging to another image joined to the scan line are closer to the center of the one image in the region corresponding to the overlapping portion. Weighted addition processing by weighting so that the weight of the video signal representing one scanning line is relatively large and the weight of the video signal representing the other scanning line is relatively large toward the side closer to the center of the other image Smoothing means configured to make the boundary of the joining portion inconspicuous,
When the noise level related to the video signal is relatively low, the width of the overlapping portion is relatively widened, and when the noise level is relatively high, the width of the overlapping portion is relatively narrowed, and weighting by the smoothing means An operation mode changing means for changing the width of the region corresponding to the overlapping portion to be subjected to addition processing ;
An image processing apparatus comprising:
[0043]
In the technology before the invention of (1), when a plurality of images including noise are pasted (joined) to create one image (image signal), the overlapping portion of the pasting is subjected to a smoothing process to connect There is a problem that the combined part (overlapping part) becomes more conspicuous. According to the invention of (1), such a problem can be solved.
[0044]
(2) The image processing apparatus according to (1), wherein when the circuit gain is relatively high, it is determined that the noise level is relatively high .
[0045]
According to the invention of (2) above, in addition to the effect of the invention of (1) above, particularly when the circuit gain is relatively high, it is determined that the noise level is relatively high, and the smoothing process is adaptively performed. It is possible to make the joining portion (overlapping portion) inconspicuous.
[0046]
(3) The image processing apparatus according to (1), wherein when the video signal level is relatively high, it is determined that the noise level is relatively low .
[0047]
According to the invention of (3) above, in addition to the effect of the invention of (1) above, particularly when the video signal level is relatively high, it is determined that the noise level is relatively low, and the smoothing process is adaptively performed. It is possible to make the joining portion (overlapping portion) inconspicuous.
[0048]
(4) The operation mode changing unit is configured to selectively change execution or non-execution of the weighted addition processing by the smoothing unit according to a noise level or a circuit gain as a state relating to the video signal. The image processing apparatus according to (1), wherein:
[0049]
According to the invention of (4) above, in addition to the effects of the invention of (1) above, in particular, a smoothing process is applied adaptively according to the noise level or circuit gain as the state relating to the video signal, (Overlapping part) can be made inconspicuous.
[0050]
(5) The operation mode changing means is configured to selectively change execution or non-execution of the weighted addition processing by the smoothing means in accordance with a video signal level as a state relating to the video signal. The image processing apparatus according to (1), characterized in that:
[0051]
According to the invention of the above (5), in addition to the effect of the invention of the above (1), in particular, a smoothing process is applied adaptively to the video signal level (and hence the influence of noise), and the stitched part (overlapping) Part) can be made inconspicuous.
[0052]
【The invention's effect】
According to the present invention, when a plurality of images including noise are joined (joined) to create one image (image signal), a smoothing process is performed on the overlapped part of the joined parts (overlapping part). It is possible to provide an image processing apparatus that can prevent the image from being conspicuous.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an image processing apparatus as a first embodiment of the present invention.
FIG. 2 is a timing chart showing a timing relationship of signals (data) of respective units in the apparatus of FIG.
FIG. 3 is a signal waveform diagram for explaining the operation of the embodiment described with reference to FIGS. 1 and 2 or a modification thereof;
FIG. 4 is a timing diagram showing a timing relationship of signals (data) of respective parts representing the operation in another embodiment.
FIG. 5 is a signal waveform diagram used for explaining the operation of the modification of the embodiment described with reference to FIG. 4;
FIG. 6 is a conceptual diagram illustrating a state in which a plurality of images to be subjected to a bonding process are generated by a plurality of imaging elements.
7 is a schematic diagram showing a state of one continuous image obtained by pasting two images generated by the imaging method as shown in FIG. 6; FIG.
FIG. 8 is a block diagram showing a configuration of a conventional apparatus.
9 is a timing chart showing a timing relationship of signals (data) of respective units in the apparatus of FIG.
10 is a diagram illustrating the characteristics of the smoothing process described above in the apparatus described with reference to FIGS. 8 and 9. FIG.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 Image sensor 11 Analog signal processor 12 A / D converter 13 Memory 14 Multiplier 20 Image sensor 21 Analog signal processor 22 A / D converter 23 Memory 24 Multiplier 30 Adder 40 Mixing ratio distributor 50 CPU
500 CPU
501 Operation mode change function unit

Claims (3)

An image processing apparatus that performs image data processing corresponding to an operation of combining a plurality of images by combining each scan line with overlapping portions,
One scan line belonging to one image and another scan line belonging to another image joined to the scan line are closer to the center of the one image in the region corresponding to the overlapping portion. Weighted addition processing by weighting so that the weight of the video signal representing one scanning line is relatively large and the weight of the video signal representing the other scanning line is relatively large toward the side closer to the center of the other image Smoothing means configured to make the boundary of the joining portion inconspicuous,
When the noise level related to the video signal is relatively low, the width of the overlapping portion is relatively widened, and when the noise level is relatively high, the width of the overlapping portion is relatively narrowed, and weighting by the smoothing means An operation mode changing means for changing the width of the region corresponding to the overlapping portion to be subjected to addition processing ;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein when the circuit gain is relatively high, the noise level is determined to be relatively high . The image processing apparatus according to claim 1, wherein when the video signal level is relatively high, it is determined that the noise level is relatively low .

Images