JPH04369780A - Edge preservation noise reducing system - Google Patents

Abstract

PURPOSE: To smooth a picture signal by using a cyclic space smoothing system by operating one-dimensional and two-dimensional estimated arithmetic operations to the picture signal of an input picture, and connecting and outputting the arithmetic result. CONSTITUTION: A sub-unit 12 constituting a smoothing operator 11 of a pipe line structure is constituted of basic one-dimensional Kalman estimator based on division smoothing. A sub-unit 16 constitutes a bidirectional mechanism which automatically executes selection in response to an edge signal, fine line or outline signal, or smoothing concentration signal, and calculates a smoothed result for both scan directions. A sub-unit 18 constitutes a two-dimensional smoothing calculating processing for connecting the present bidirectionally smoothed result with a previously scanned and smoothed picture line. A Kalman filter gain factor is preliminarily calculated, and stored in an index table 17. Then, the cascade-connected smoothing operator 11 selects the optimal gain factor, and operates a cyclic arithmetic operation based on a space smoothing algorithm by using this gain factor.

Description

[Detailed description of the invention]

0001

[Industrial Application Field] The present invention relates to an image noise reduction method for restoring digital images, and more specifically, it smoothes an input image signal that has been degraded by noise to quickly and optimally restore the original image signal. This paper relates to an image noise reduction method to be estimated.

0002

2. Description of the Related Art Normally, an input image signal obtained by digitally converting an analog image signal or an image scanning signal from a scanner contains noise. Traditionally, to remove noise in images, the luminance surfaces of the image are gradually changed and smoothed, while at the same time preserving more abrupt luminance changes such as edges, lines, and contours. Is going. Residual distortion in the smoothed image compared to the noisy original image is manifested as an error in the image signal estimation process.

For example, when using a TV camera used for process control in a factory, a portable TV camera for collecting commercial information such as advertising, an image camera for simultaneous broadcast communication, etc., images are often facing low lighting situations. In that case, the signal/noise ratio (S/N
As a result, the image quality deteriorates. In such an image signal, noise due to electronic noise or non-uniformity of the CCD detector is generally emphasized, and is observed as a clear point in the displayed image. This problem also appears with still video cameras. With a still video camera, one snapshot is taken per scene, and the combination of fixed patterns and random noise makes the noise more clearly visible in the image and more visible to the viewer. There was a problem that bothered me.

[0004] Another problem with image-enhanced television cameras used for in-store surveillance, cell surveillance, etc. is that images are often observed with a high degree of noise due to the low level of the light source. was there.

[0005] Image noise can also be caused by medical imaging systems,
For example, it appears in ultrasound imaging devices that show tomographic images of the internal state of the human body, and in photon counting imaging systems, which are special electro-optical devices used under very low lighting conditions.

Furthermore, in the case of image scanning from a fixed pattern using a CCD scanner detector, the quality of the image obtained depends on the conditions of the light source and the scanned data (images or text on paper or film).

There is also noise caused by non-uniformities in the sensitivity of CCD scanner detectors. And it is often necessary to reduce noise in these cases as well.

[0008]

[Problems to be Solved by the Invention] As described above, the problem of image restoration to remove noise mixed in images (images restored for display) is a problem that can be solved by electro-optic scanning contrast imaging, electronic television, graphic display, etc. applied over a wide range of areas.

By the way, this noise can be corrected using a digital image restoration processing scheme. Conventional digital image restoration processing schemes can generally be divided into two types. That is, the first is a spatial smoothing operator that uses only spatial image information, and the second is a time integration operator that sequentially integrates a sequence of digitized images (digital sequence) using a finite response time. This extends the effective exposure duration and improves the signal/noise ratio.

The image processing result using the spatial smoothing operator is
Normally, compared to the real image, that is, the original image, residual distortion such as "blurring" or "smearing" occurs frequently in image contents such as edges, lines, and contours, and the restoration is often dissatisfying for viewers. It becomes an image. However, in the case of TV images, for example, as long as there is "movement" in the image, the image is continuously updated, and combined with the visual correction ability of humans, the blurring of the image becomes less noticeable.

When image processing using a temporal image integral operator is applied to a still image, since the real image signal is constant while the noise signal is random, noise removal is easy and the reconstructed image is Image quality has been significantly improved. However, for moving subjects, the effective exposure of the video camera is extended by integrating successive video signals (video frames), causing inevitable image smearing. Therefore, it turns out that it is not suitable for image processing of moving images.

[0012] As described above, each method has its advantages and disadvantages, and there has been no method that can optimally remove noise from a restored image, regardless of whether it is a moving image or a still image. A first object of the present invention is to realize an image noise reduction method that smoothes an input image signal that is degraded by noise and estimates an original image signal quickly and optimally.

[0013] The second purpose is to automatically reduce each pixel signal of an image in the image noise reduction method.
By selecting the most appropriate signal model: edge model, line or contour model, or smooth surface model,
The goal is to combine spatial smoothing with the noise reduction benefits of time integration.

[0014]

[Means and operations for solving the problem] This spatial smoothing method involves a Kalman filter gain factor and uses, for example, a Wiener filter as an estimation filter for predicting and filtering a stationary time series. In addition, for example, a median filter is used as a filter for edge-preserving smoothing.

Iterative spatial smoothing is then achieved by repeating the process using two-dimensional arithmetic units of the same type in a cascade configuration. The spatial smoothing operator for this purpose is a two-way cyclic two-dimensional operator, which is accompanied by a moderate video line memory for the estimation calculation of the image luminance signal value, and supports real-time processing. Create an appropriate pipeline structure for this purpose.

This operator provides a local signal-to-noise ratio that is a simple and accurate estimate of local image statistics for additive noise in an image. This statistical estimate is computed selectively depending on whether the current pixel (the pixel currently being processed) is an edge pixel, a line or contour pixel, or a smooth bright surface pixel, thereby selecting an appropriate Kalman gain factor. do. The Kalman gain factor for estimating the complete range of the signal/noise ratio is calculated cyclically, aggregated to a constant state, and stored in a look-up table in direct access memory.

Combining the cyclic two-dimensional calculation using the pixel line smoothed before the current pixel (the pixel row immediately above the pixel row in which the current pixel exists) is the combination of the edge real image value and the line or contour. To efficiently provide an input value to an operator without being estimated by confusing it with a real image value.

In addition, this spatial smoothing method uses a smoothed image before the current pixel (
For example, a cyclic three-dimensional computation is performed using a digital sequence of a video image (forming the frame or field immediately preceding the frame or field in which the current pixel is located) to combine related pixel data in the time domain.

In this way, the advantages of the integral method for still images can be realized even for moving images that tend to cause "bleeding" by applying the minimum number of cyclic operations using a simple spatial smoothing method.

The method of the present invention improves image enhancement, encoding and compression, interpolation, electronic zoom ( It can be directly and easily applied to a wide range of image reproduction processing such as enlargement).

[0021]

Embodiments Next, the structure and operation of an embodiment of the present invention will be explained in detail. FIG. 1 is a conceptual block diagram showing the function arrangement and processing procedure of calculation processing performed by the edge-preserving image noise reduction method of the present invention.

In the figure, in the unit 10, first,
The input analog video signal or image scanning signal is digitized. Also included are all means for automatically controlling the gain level for optimal digitization of the input signal.

The unit 11 consists of a complete noise reduction calculation processing space algorithm, and is arranged in parallel in a pipeline form consisting of operators of the same configuration from number 1 to number n. Therefore, iterative computations can be performed without reducing data flow throughput and scan rate.

[0024] Each unit 11 has numbers 12 to 18, 7
It consists of sub-units. The sub-unit 12 consists of a broadly stationary Markov image model, described below, ie a basic one-dimensional Kalman estimator based on piecewise smoothing.

The sub-unit 13 is an intermediate storage with a delay device, ie, a delay line FIFO (First In First Out), for storing input signals, and sub-units 14 and 15 for storing intermediate results of calculation processing.

The sub-unit 16 constitutes a two-way mechanism which automatically selects the appropriate signal mode of the current pixel in response to a set of logic rules, namely an edge signal, a thin line or contour signal, or a smooth density signal. and calculate smoothing results for both scan directions.

The sub-unit 17 has a constant state look-up table, ie for the complete range of expected signal/noise ratio estimates and fixed (assumed and preset) spatial correlation values. with a pre-calculated Kalman filter gain factor.

Subunit 18 constitutes a two-dimensional smoothing calculation process that combines the current two-dimensional smoothing result with the previously scanned and smoothed image line. Unit 19 consists of a manual or automatic noise level input to the noise reduction scheme in the form of an estimate of the variation in image noise. This value is then used as a scale factor for the estimated signal variation and as an input signal for the noise measurement of the Kalman gain factor look-up table in the sub-unit 17.

FIG. 2 shows a characteristic diagram showing the relationship between the Kalman gain factor K(i) and the signal/noise ratio snr. Also,
FIG. 3 shows the basic one-dimensional estimation calculation mechanism and data flow of the present invention.

In FIG. 3, a noisy image signal is generated from adjacent regions modeled as a first (broad sense) Markov process by means, fluctuations, and correlation coefficients associated with each region. This can be modeled as a piecewise stationary process.

In general, image signals are degraded by uncorrelated additive noise that is usually interspersed. The purpose of this configuration is to estimate the input noise-degraded image signal to the most desirable original image signal by smoothing it.

In general, on the one hand, the inherently high amplitude signal content, such as edges, lines, and contours, is preserved while, on the other hand, the varying luminance (density) signal is successively smoothed. That is to say. The residual distortion in the smoothed image compared to the original image signal is then defined as the estimation error.

The minimum mean square error (MMSE) cyclic unbiased estimate of a wide sense Markov process with a non-zero mean value is

[
0034

[Math 1]

It is given by: Here, outside 1 is MMS
represents the E-cycle estimate,

[0036]

[Outside 1]

s(i-1) represents the previous Markov processing output value, ρS represents the correlation coefficient of the processing, and μS represents the average value of the processing.

The usefulness of Markov model processing, which is also Gaussian distribution processing, lies in the fact that the Kalman cyclic estimate theory is directly applied to the problem in the most convenient sense and in a closed form as a reliably valid solution. Image pixel estimates are expressed as state estimates by the well-known discrete Kalman filter equation. The system model is
Markov model of image pixel sequence s(i) = ρS s(i-1) + ψ(i-1)
; ψ˜N(0, бS 2 (1−ρS 2 )). Here, the notation N(μ, б2)
represents an uncorrelated normally distributed (Gaussian) process. The density measurement of a noise degraded pixel is represented by x(i), and the measurement model symbology in this one-dimensional decomposition is x(i)
=s(i) +v(i) ;v〜N(0,σS 2
) is given by

At each sample point i in the estimation process,
The state estimation represented by

[0040]

[Outside 2]

The value extrapolation formula is:

[0042]

[Math 2]

It is given by: Here, outside 3
is the state estimate in the previous processed sample

[0044]

[Outer 3]

[0045] Represents an update. The error covariance extrapolation at sample point i is expressed as P-(i), P- (i) = ρS 2 P+ (i-1) +σS
2 (1-ρS 2 ). Here, P+
(i-1) represents the error covariance update in the previous processed sample. The state estimate update at pixel i is

[0046]

[Math 3]

It is given by: Here, K(i) represents the Kalman gain factor of pixel i, and K(i) = P-(i)/(P-(i)+ρS 2)=
(ρS 2 P+ (i-1) +σS 2 (1-ρ
S 2 ))/(ρS 2 P+ (i-1) +σS
2 (1-ρS 2 )+ρn 2 ). Also, the error covariance update is P+ (i) = [1-K(
i) ]P- (i) = K(i) ρn 2 .

From the above symbolic logic formula, the Kalman gain factor K
(i) depends on K(i-1), K(i)
=(ρS 2 K(i-1) σn 2 +σS 2
(1-ρS 2 ))/(ρS 2 K(i-1)
σn 2 +σS 2 (1−ρS
2)+ρn2). Then, with the signal/noise ratio given by snr=ρS 2 /ρn 2 , we obtain the following simple expression for K(i):

K(i) = (ρS 2 K(i-1) + snr(1
-ρS 2 ))/(ρS 2 K(i-1) +sn
r(1-ρS 2 )+1) In the estimation process, piecewise (local) mean, variation, and correlation estimates must be computed to compute K(i) for a given point. Given these statistics, constant state values for the Kalman gain factor are expected for snr and ρS.
can be precomputed cyclically as a function of the value of , and
Stored in the index table. The cyclic operation uses the initial value K(
0) to the following equation: K(0) = snr/(snr+1).

Additional simplifications are introduced to the estimation process. The value for ρS is preset to one constant. Then, the noise variation ρS 2 of the input signal is well known, which estimates the snr piecewise. snr is related to the degree of uncertainty in partition estimation. A self-discovery rough estimate of the snr for each pixel is given by the following difference formula:

[
0051

[Math 4]

This is proposed by [0052] Further compound sn
The r estimate is computed using a window of fit around the current pixel location. The piecewise mean value μS is expected to correct between adjacent regions. In order to avoid iterative calculation of the segmented area average value, we modify the state estimation update to the following equation:

[0053]

[Outside 4]

[0054]

[0055]

[Math 5]

The Kalman gain factor K(i) in this equation is:

[0057]

[Math 6]

with snr given by This modification simplifies the estimation process to avoid drifting away from the area average value. The one-dimensional Kalman estimation filter (sub-unit 12 in FIG. 1) is the basic calculation configuration of the method of the present invention, and is shown in more detail in FIG. 3.

Two points should be noted regarding the above Kalman estimation procedure resulting in one-dimensional approximation. First, the Kalman gain factor of the fit is compatible with the intuition that the Kalman gain factor is proportional to the degree of uncertainty in the estimate and inversely proportional to the measurement noise. It is. This results in small changes in the current estimate when the measurement noise is large compared to the estimation uncertainty (low SNR).

On the other hand, small measurement noise and large uncertainties (high SNR) in the state estimation pose that the current sample measurements contain information that should be considered. That is, when there is a high degree of uncertainty in the estimation and, at the same time, a small degree of measurement noise, the current measure is considered to have a high degree of certainty and therefore contains information about possible changes in the signal amplitude. It will be done. Therefore, discrepancies between actual and predicted measurements are used as a basis for strong revisions to current estimates.

The second point relates to the main drawback of one-dimensional calculation methods. That is, the degree of noise signal preserved in the image as a result of the calculation increases with the amplitude of the noise. That is, the noise modulation and fluctuations in intensity remain unchanged.

However, this problem is solved in the present invention.
This is solved by combining it with a two-way processing method. There, the Kalman gain factor is selected using a two-way estimation corresponding to a piecewise screening logic of signal enhancement between edge signals, line or contour signals, or smooth density signals.

As shown in FIG. 1, the two-way smoothing mechanism (subunit 16) is computed based on the one-dimensional smoothing process and the smoothing results in the input signals in the temporary storage and the related delay line FIFOs. Because of the high sensitivity of the snr estimate to image noise, the two-way smoothing logic uses the piecewise Kalman gain snr
In order to estimate a more robust measure for each pixel position (see Figure 4), the left-to-right smoothed estimator
(Hereafter, ⌒s→ (i)

[0064]

[Outer 5]

) and the right-to-left smoothed estimator
Outside 6 (hereinafter written as ⌒s← (i))

006
6]

[Outside 6]

They are combined by using both one-dimensional smoothing results. The two-way smoother starts assuming that the one-dimensional smoothing result of the complete left-to-right smoothing estimator ⌒s→ (i) has been precomputed and stored in one FIFO 13. And the following formula

[0068]

[Math 7]

[0069]

[Math. 8]

Two absolute differences s1 (i) given by
and s2(i), calculate the smoothing result of the right-to-left smoothing estimator ⌒s←(i) at each pixel. If s1 (i) > s2 (i), the smooth density signal is identified and the related s
The nr measure is given by snr=s2 2 /σn 2 . Then, the above snr measure selects the Kalman gain factor K(i), and the selected Kalman gain factor K(i) is calculated by the following formula:

[0071]

[Math. 9]

The right-to-left smoothed estimator ⌒s← according to
(i) is used to calculate. Additionally, the same snr measure can be expressed as

[0073]

[Math. 10]

The left-to-right smoothing estimator ⌒s→ (i) given by is used to correct the value of (i). Alternatively, if s1 (i) < s2 (i), an edge signal is hypothesized and the right-to-left smoothing estimator ⌒s← (i) is computed as follows. That is, the expression

[0075]

[Math. 11]

According to [0076] The related snr measure is sn
It is given by r=s1 2 /σn 2 . Now, using the right-to-left smoothed estimator ⌒s← (i) smoothed for the current pixel, the snr measure for the left-to-right smoothed estimator ⌒s→ (i) is Re-estimate according to Eq.

[0077]

[Math. 12]

[0078] And the snr measure related to this is

0
079]

[Math. 13]

[0080] In this way, the two-way smoother gives the right-to-left estimate ⌒s← (i) at each pixel.
A directional smoothing estimate of and left-to-right estimator ⌒s→ (i) is computed from the most likely pixel for the Kalman gain factor using the estimate of that pixel.

The main disadvantages of the two-way smoother are:
This often involves smoothing out fine lines and contours of the image, which are essential image information. This drawback is compensated for by combining a two-dimensional method that utilizes the previous image line smoothed pixels in the estimation of the current pixel (see FIG. 5). of the previous line adjacent to the currently processed pixel

[0082]

[Outside 7]

A one-dimensional Kalman smoothing operator is applied to each two-dimensionally smoothed pixel in [0083] using a gain factor index value given by the following difference formula.

[0084]

[Math. 14]

This extension is easily illustrated to preserve image fine lines in the smoothed output image. Thus, we obtain five intermediate smoothing results. That is, the two bidirectional estimates (⌒s→ (i), ⌒s← (i)) and the three one-dimensional values obtained from the previous line

[0086]

[Outside 8]

[0087] This is an estimated value. These five estimates are then averaged with equal weight, resulting in a two-dimensional smooth pixel estimate

[0088]

[Math. 15]

The following is obtained. The above two-dimensional smoothing method is shown in Figure 1.
can be easily repeated using a computational scheme similar to that shown in . That is, the smoothed output signal of the first iterative calculation is used as an input to the second iterative calculation, and this is repeated sequentially.

[0090] This pipeline structure does not affect the data processing capacity (throttle rate) of the system as a whole. Furthermore, this structure is a convenient design solution for executing various estimation processes that require the sum of differences in repeated calculations.

Only the required control variables are changed according to the corresponding iterative scheme, resulting in a noise-smoothed estimate of the input signal. Moreover, it is calculated in advance from a smoothing simulation based on smoothing and noise density signals.

[0092] Also, each iteration is variably performed directionally in a different direction. For example, the first iteration is from top to bottom, the second from bottom to top, the third from left to right, and the fourth from right to left. When performing such iterations, the image resulting from the first iteration is scanned in the second iteration. The resulting image is scanned in the third iteration. The resulting image of the third iteration is scanned in the fourth iteration. In this way, each iteration uses the image resulting from the previous iteration. Therefore, each iteration occurs after the completion of the previous iteration.

Each iteration performed in a directionally manner assumes that the original noisy image and the smoothed image resulting from each iteration are each stored in a digital image RAM. Therefore, line-sequential scanning in either the horizontal or vertical direction is possible, and any digital image scanning shape can be taken.

In the above scheme, for still images, one image RAM buffer is used for directed repetition. On the other hand, for moving images, a pipeline structure using four image RAM buffers is required. In either case, there is a resulting gap of 4 scan execution periods (
One scan is a line sequential scan of one frame or one field), but this delay does not pose any problem in practice.

The two-dimensional smoothing method described above can be applied to color images. For example, red, green, blue (
RGB) images are smoothed individually using the same calculation method for each color image. The result is a noise-reduced RGB color image.

The two-dimensional smoothing method described above can be applied, for example, to the case of a digital sequence of moving images, by using a similar combination of Kalman estimation to calculate the current pixel from a previously smoothed image (frame or field). It becomes possible to perform an estimation calculation that combines pixels in the vicinity of , and smoothing can be extended to the time domain, thereby making it possible to estimate a three-dimensional region consisting of space and time (see FIG. 6).

This three-dimensional method is extremely effective for iterative calculations on the time axis in a stationary number sequence, and is also effective for area estimation where image movement affects the iterative calculation results and degrades the image quality. .

[0098]

[Effects of the Invention] In this way, regardless of whether the image is a still image or a moving image, the optimal pixel estimation amount can be easily calculated and output using the same operator, so it can be used as an image restoration process to remove mixed noise. It has a wide range of applications such as electro-optical scanning imaging, electronic television, or graphical displays.

In other words, T used for process control in factories, etc.
If applied to the reproduction of images obtained from V cameras, portable TV cameras for collecting commercial information such as advertisements, image cameras for simultaneous broadcast communications, etc., it is possible to reproduce images obtained from low-lighting and image signals.
It is possible to construct a system that can obtain good image quality even when the noise ratio is low.

[0100] This is similarly effective when used for image reproduction of a television camera used for in-store surveillance, in-cell surveillance, and the like. Furthermore, if applied to the reproduction of images obtained from still video cameras, ultrasonic scanners, CCD scanners, etc., high-quality images with noise removed can be reproduced. Even when playing moving images such as video images, images without blurred contours can be obtained.

[Brief explanation of drawings]

FIG. 1 is a conceptual block diagram showing the function arrangement and processing procedure of calculation processing performed by the edge-preserving image noise reduction method of the present invention.

[Figure 2] Kalman gain factor K(i) and signal/noise ratio s
It is a characteristic diagram showing the relationship with nr.

FIG. 3 is a diagram showing the basic one-dimensional estimation calculation mechanism and data flow of the present invention.

FIG. 4 is a diagram illustrating the data flow, logic, and calculations in which the one-dimensional estimation of the present invention is bidirectionally coupled.

FIG. 5 is a diagram illustrating the data flow, logic, and calculations in which the two-way estimation of the present invention is coupled in two dimensions.

FIG. 6 is a diagram illustrating the data flow, logic, and calculations of a three-dimensional estimation in which the two-dimensional estimation of the present invention is coupled in space and time.

Claims (9)

[Claims] Claim 1: Performing one-dimensional, two-directional, and two-dimensional estimation calculations on pixel signals of input still images,
An edge-preserving noise reduction method characterized by combining and outputting the calculation results. [Claim 2] One-dimensional, two-directional, two-dimensional, and three-dimensional estimation calculations are performed on pixel signals of input moving images, and the calculation results are combined and output. Edge-preserving noise reduction method. 3. Piecewise estimating a Kalman gain factor for the estimation operation in response to piecewise signal interpretation between an edge signal, a thin line signal and a contour signal, and a smooth surface density signal;
3. The edge-preserving noise reduction method according to claim 1, wherein edges, thin lines, planes, etc. are automatically determined by calculating the edge-preserving estimated amount of each pixel in the image. 4. The estimation operation has an arrangement of estimation operators that performs repeated operations by using cascaded operators of the same configuration and modifying control variables that participate in the estimation operation. The edge-preserving noise reduction method according to item 1, 2 or 3. 5. The one-dimensional, two-directional estimation calculation is performed by scanning a pixel column on a scanning line in opposite directions from one opposing end to the other end. The edge-preserving noise reduction method according to item 1, 2, 3 or 4. 6. The one-dimensional, two-way estimation calculation calculates the estimated amount of each pixel adjacent to the estimated pixel on the scanning line, and the estimated amount of each pixel adjacent to the estimated pixel on the scan line scanned immediately before. Claims 1, 2, and 3 are characterized in that the estimated amount of each pixel is combined with the estimated amount of the pixel.
, 4 or 5. The edge-preserving noise reduction method according to . 7. The two-dimensional estimation calculation is performed by performing the one-dimensional and two-directional estimation calculation by horizontal/line sequential scanning or vertical/line sequential scanning. , 3, 4, 5 or 6. [Claim 8] Horizontal and
A claim characterized in that line sequential scanning or vertical/line sequential scanning is performed in two directions, from top to bottom and from bottom to top, or from left to right and from right to left. 8. The edge-preserving noise reduction method according to item 1, 2, 3, 4, 5, 6, or 7. 9. The input image pixel signal is an image pixel signal configured individually for each of the three primary colors of a color image. ,6
, 7 and 8.