TW201923704A - Restoration method for blurred image - Google Patents

Abstract

A restoration method for motion-blurred frames is provided. The restoration method for blurred image includes following steps: (1) Point spread function estimation: Firstly, a motion-blurred image is transformed to the cepstrum domain using Fourier transform and both blur angle and blur length of PSF are directly estimated in the cepstrum. (2) Single-image deblurring: Both estimated blur angle and blur length are employed to normalize PSF and then the image restoration is achieved by performing the iterative deconvolution with the normalized PSF. (3) Clear image register: The selected clear images or the motion-deblurred images are stored in the clear image register. (4) Multi-image deblurring: It is first to search the corresponding feature points between the current image and the latest image stored in the image register. These feature points are used for deducing a homography matrix required by the perspective transformation which can align the corresponding pixels of those two images. Then, the weights of pixels among such two images are calculated and compared for generating the restored pixels in such a way that the high-weight pixel replaces the low-weight pixel, where each weight of pixel is generated according to the temporal information.

Description

Restoration method of blurred image

The invention relates to a method for restoring a blurred picture, in particular to a method for restoring a blurred image.

Due to the development of science and technology in recent years, computer vision technology has been widely used in important institutions or traffic monitoring systems for license plate recognition to achieve an intelligent monitoring system for maintaining law and order and preventing crime, but it may be possible while shooting Due to some external factors, motion blur may occur when capturing images. However, these blur problems may also lead to misjudgment of subsequent identification applications. Therefore, it is necessary to repair and compensate the blurred images.

In the field of image blur restoration, most images deblurring a single image (Image Deblurring), but the single blur restoration has a good restoration result for artificial blur images (Artificial Motion-Blurred Image), or it is constant speed The result of motion (Uniform Motion) blurred image is better, but on the restoration of non-uniform motion (Non-Uniform Motion) blurred image, there will be no better results, and the calculation of blur reduction is too large, and the The requirements of related devices and equipment are relatively increased, and the cost is also increased.

In view of the above-mentioned conventional problems, the object of the present invention is to provide a method for restoring blurred images to solve the problems faced in the conventional technologies.

Based on the above objective, the present invention provides a method for restoring a blurred image, which includes the following steps: Fourier transform the blurred image to generate converted values, then obtain the cepstrum of the converted values, and then calculate the point spread from the cepstrum domain The blur angle and blur length of the function; normalize the point spread function according to the blur angle and blur length, and then perform the deconvolution of the single image on the normalized point spread function to generate a restored image; select a clear image or restore the image and save it Clear image register; and search the corresponding feature points between the current image and the restored image stored in the clear image register, use the feature points to derive the homography matrix required for perspective transformation, and then correct the current image and Recover the corresponding pixels of the image, and then calculate and compare the weights of the pixels in the current image and the recovered image based on the time domain information, replace the low-weight pixels with high-weight pixels to generate the recovered pixels, and output another recovered image.

Preferably, when the clear image register inputs the image sequence, the clear image in the image sequence is stored, and the blurred image in the image sequence is restored to the restored image by the restoration method of the blurred image.

Preferably, the point spread function can be expressed by the following formula: Among them, L system is the blur length, and θ system is the blur angle.

Preferably, the definition of the cepstrum domain of the blurred image can be expressed by the following formula: Among them, F represents the Fourier transform, F ^-1 represents the inverse Fourier transform, and g (x, y) is a blurred image.

Preferably, calculating the blur length and blur angle of the point spread function may include the following steps: (1) input the cepstrum to find the center point; (2) from three directions of 90 °, 45 ° and 0 ° of the center point Select the point with the largest cepstrum value as the search point; (3) Move to the search point and use a 5 * 5 mask, the center of the mask corresponds to the search point; (4) Cepstrum in the mask Add all the values and store the coordinates of the search point; (5) Determine whether the direction of the search point is offset, if not, go to step (6), if yes, go to step (7); (6) Calculate the distance between the search point and the center point Whether it is greater than the threshold, if not, go back to step (2) to repeat step (2) to step (5), and if so, go to step (7); (7) Find the turning point of falling and then rising from the trajectory value distribution chart; ( 8) Calculate the blur length and blur angle according to the turning point and the center point; where the coordinates of the turning point are (x ₁ , y ₁ ), the coordinates of the center point are (x ₀ , y ₀ ), L is the blur length, and substitute (x ₁ , y ₁ ) and (x ₀ , y ₀ ) to obtain the fuzzy length are expressed by the following formula: R is the length of the base of the triangle formed by the turning point, the center point and the X axis. Substituting x ₁ and x ₀ to obtain the base length is expressed by the following formula: θ is the blur angle. Substituting the length of the blur and the length of the bottom edge to obtain the blur angle is expressed by the following formula: .

As mentioned above, the blurred image restoration method of the present invention can be applied to the restoration of linear motion blurred pictures. The proposed video deblurring method can be implemented in common image monitoring systems or the shooting of general mobile vehicles The device can be used for real-time deblurring through video streaming. Due to the low computational complexity of the algorithm proposed in this case, it can be embedded in a general security surveillance camera system to increase the added value of related products.

In order to better understand the features, contents and advantages of the present invention and the effects that can be achieved, the present invention is described in detail with the drawings in the form of examples, and the main purpose of the drawings is only The use of illustrations and auxiliary descriptions may not be the actual proportions and precise configurations after the implementation of the present invention, so the accompanying drawings should not be interpreted and limited to the scope of rights of the present invention in actual implementation.

The advantages, features, and technical methods of the present invention will be described in more detail with reference to the exemplary embodiments and the accompanying drawings for easier understanding, and the present invention may be implemented in different forms, so it should not be understood to be limited to this The embodiments described herein, on the contrary, to those having ordinary knowledge in the technical field, the embodiments provided will make this disclosure more thoroughly and comprehensively and completely convey the scope of the invention, and the invention will only be appended As defined by the scope of patent applications.

Please refer to FIG. 1, which is a first flowchart of the method for restoring a blurred image of the present invention. As shown in the figure, the blurred image restoration method 100 of the present invention includes the following steps:

In step S11: Fourier transform the blurred image to generate converted values, then obtain the cepstrum of the converted values, and then calculate the blur angle and blur length of the point spread function from the cepstrum domain.

In step S12, the point spread function is normalized according to the blur angle and the blur length, and then the normalized point spread function is subjected to deconvolution of a single image to generate a restored image.

In step S13: select a clear image or restore the image and store it in the clear image register.

In step S14: search for the corresponding feature points between the current image and the restored image stored in the clear image register, use the feature points to derive the homography matrix required for perspective transformation, and then correct the current image and the restored image accordingly Corresponding pixels, and then calculate and compare the weights of the pixels in the current image and the restored image according to the time domain information, replace the low-weight pixels with high-weight pixels to generate restored pixels, and output another restored image.

Please also refer to Figures 2 to 13; Figure 2 is the cepstrum of the blurred image of constant velocity motion; Figure 3 is the cepstrum of the blurred image of non-constant motion; Figure 4 is the blurred image of the present invention The second flow chart of the restoration method; Figure 5 is the search trajectory of the cepstrum; Figure 6 is the 3D map of the cepstrum domain; Figure 7 is the search trajectory of the non-constant velocity motion cepstrum Figure 8 is a trajectory value distribution diagram; Figure 9 is a schematic diagram of the result of a single image restoration; Figure 10 is a schematic diagram of the homography matrix performing transformation between two images; Figure 11 is a corner matching Schematic diagram of the results: (a) part is the t-1th picture; (b) part is the tth picture; (c) is the matching result of the corner points of the t-1 picture and the tth picture; the twelfth picture is The schematic diagram of the result of perspective transformation; Figure 13 is a schematic diagram of the result of the restoration of multiple images. The above steps will be described in detail in conjunction with the drawings.

1. Point spread function estimation

The point spread function can be regarded as the pulse function in the optical system. Mathematically, the point light source (input) can be represented by the point pulse function, and the output light field distribution can be called the impulse response (Impulse Response), and this represents the pulse received by the image response. If the imaging system produces an inverted image, you can simply reverse the image plane coordinate axis from the object plane coordinate axis. When there is no distortion, calculating the image plane convolution integral is just a simple process.

1.1, image conversion

Most of the diffusion function estimation methods are processed in the spatial domain. The main method is to estimate the local feature points of the image such as points, edges, and lines. In these methods, if you want to estimate the point diffusion function, you must Knowing the type of blur in advance, and the accuracy of these methods will decrease when the blur is more serious, so this system is based on the frequency domain method to calculate the length ( L ) and angle (θ) of the blur.

Among them, the point spread function of linear motion blur is approximated by equation (1).

(1)

Among them, L system is the blur length, and θ system is the blur angle.

1.2, Cepstrum domain conversion

The cepstrum domain was first proposed in the field of one-dimensional signal processing, and was later introduced into image processing. The cepstrum domain of the blurred image g ( x, y ) is defined as formula (2), where F and F ^-1 represent Fourier Transform, inverse Fourier transform, from the equation we can see that the cepstrum domain of the image is the inverse Fourier transform of the logarithm of the original image power spectrum.

(2)

1.3, calculate the fuzzy length and angle

In the experiments of the present invention, it was found that if the constant velocity motion is blurred, there will be two obvious black points in the cepstrum, as shown by the white circle in Figure 2, and we only need to calculate the difference between the center point and this point. The distance is the blur length of the image, and then the angle θ formed by this point and the X axis is calculated, and this θ is also the blur angle; but if it is non-constant motion blur, there will be no obvious black spots, as shown in Figure 5. As shown in the figure, therefore, the present invention proposes an algorithm that can calculate the length and angle of the blur no matter whether it is constant-speed or non-constant-speed motion blur. As shown in Figure 4, the algorithm flow includes the following steps:

In step S61: input the cepstrum and find the center point.

In step S62: the point with the largest cepstrum value is selected from the three directions of 90 °, 45 ° and 0 ° of the center point to use it as the search point.

In step S63: move to the search point and use a 5 * 5 mask, the center of the mask corresponds to the search point.

In step S64: add all the cepstrum values in the mask and store the coordinates of the search point.

In step S65: determine whether the direction of the search point is shifted, if not, proceed to step S66, if yes, proceed to step S67.

In step S66: calculate whether the distance between the search point and the center point is greater than the threshold, if not, return to step S62 to repeat steps S62 to S65, and if so, proceed to step S67.

In step S67: find the turning point of falling and rising from the trajectory value distribution chart.

In step S68: calculate the blur length and blur angle according to the turning point and the center point.

As mentioned above, assuming that a non-constant velocity blurred cepstrum is input, the first step is to find the center point from it, and the second step is to select the three directions of the center point from 90 °, 45 °, and 0 °. The point of the maximum cepstrum value is used as the search point. The third step moves to the search point, and then uses a 5 * 5 mask, the center of the mask corresponds to the search point, and the fourth step adds all the cepstrum values in the mask. And store the coordinates of the search point, the fifth step judges whether the direction of the search point is offset, if not, it is calculated whether the distance between the search point and the center point is greater than the threshold, if not, go back to the second step and continue to repeat the above steps, if it is in the The search will stop in five steps, as shown in the track in the white circle in Figure 5.

The reason for the deflection is because the value at the descent point is mostly negative, so it will deflect along the surrounding positive value when searching. The depression can be clearly seen from the 3D diagram in Figure 6, but Non-constant motion blur usually does not have a significant descent point, so no deflection occurs during the fifth step of the search, as shown in Figure 7, then enter the sixth step to determine if the distance between the search point and the center point exceeds the set The threshold value stops searching, and enters the seventh step. Through the points stored in each mask, you can calculate the turning point when the value gradually decreases and then rises. It can be clearly seen from the trajectory value distribution diagram in Figure 8 , And finally calculate the distance between the turning point and the center point, assuming that the turning point is ( x ₁ , y ₁ ) and the center point is ( x ₀ , y ₀ ), then apply the Euclidean distance formula (3) to calculate two points The distance between them is the fuzzy length L , and the two points and the x-axis can be regarded as a triangle. L is the hypotenuse and the base is R. The length of the base can be calculated by formula (4), and then by the formula ( 5) The angle can be calculated by the inverse cosine formula.

(3)

(4)

(5)

2. Single image deblurring

This step is mainly divided into two parts, namely the normalization of the point spread function and the deconvolution of a single image. Since the point spread function (PSF) is unknown before the restoration, iterative operation requires an initialization seed point, Therefore, the point spread function can be initialized by the blur length L and the angle θ calculated above, and then calculated by the Richardson-Lucy iterative restoration algorithm, and the Richardson-Lucy calculation is performed by formula (6). In formula (6), B represents a Blurred image, clear image at different positions according to point spread function The average results, CM I _m is the total number of clear images. And the Lucy-Richardson algorithm pair The updated formula can be derived as formula (7), where Represents the updated image in the iterative operation, For the pre-update image, Represents the Convolution Operation, and the Richardson-Lucy algorithm assumes that the image noise is based on the Poisson Distribution, and the method of the present invention assumes that the image noise is Gaussian Distribution, then the formula (7) It can be expressed by formula (8).

(6)

(7)

(8)

2.1, point spread function normalization

It is known from the above formula that to convolute the blurred image, the blur length and angle obtained by the point spread function estimation can be regarded as a two-dimensional matrix. Using formula (9), the values in the two-dimensional matrix are normalized between 0 and 1, and the sum is 1, where the denominator is the sum of the values of the elements in the matrix.

(9)

2.2 Deconvolution of a single image

Through the formula (8) to deconvolute the blurred image, due to the beginning of the iterative algorithm of Richardson-Lucy method Unknown, so the original blurred image is initially Then, the image is iterated through the above formula and a clear image is restored, as shown in Figure 9 for the restored result.

3. Clear image register

At the beginning, when inputting the image sequence, it will first determine whether it is a blurred image. If the image is a clear image, it is stored in the clear image register and the clear image is output. If the input is a blurred image and the degree of blur is large, then A single image will be deblurred first, and the processed result will be transferred to the clear image register for storage. The register will store K clear images or restored images, and shaved when a new frame is entered. The oldest picture, besides, before the image enters the scratchpad, there will be a counter to accumulate. When the value in the counter exceeds K before entering the scratchpad, it means that it is in the current scratchpad. None of the pictures have been restored to a single image, so a single image will be forced to be restored and then stored in the clear image register, so as to prevent the accumulation of blurred images.

4. Deblur multiple images

In the actual image sequence, it usually contains clear images and blurred images. If each blurred image frame is deblurred with a single image, it will be very time-consuming and costly, and it is not easy to apply in real time. Therefore, the method of the present invention uses The concept of time domain (Temporal) information to achieve real-time video repair, mainly includes image alignment (Alignment) and video restoration. First, the K images in the image register include the original clear image and the restored image. Then, in order to obtain the data of the time-adjacent domain, the feature points of the current image and the previous image are searched to calculate the perspective transformation (Perspective Transformation ) To obtain the transformation between the two images by using the homography matrix (Homography Matrix). After applying perspective transformation, the pixel alignment between the two images can be corrected; after obtaining the corrected image, the first K sheets are used for alignment After the video screen, weighted calculation of the time neighborhood is performed, so that each pixel of the current image obtains a new calculation result.

4.1, image correction

Using perspective transformation image matching method for the continuous correction, as shown in FIG. 10, two consecutive images to calculate the energy through a homography matrix H is performed transform image, figure 10 by H t -1 This picture screen The shooting angle becomes the same as the t-th sheet. The transformation process may include translation, rotation and scaling, and then use Shi-Tomasi corners corner detection to calculate the corners in the image, using this corner as a feature point, the most subsequent homography The characteristic points of the sex matrix, as shown in Figure 11, the perspective transformation of the image through the homography matrix, the main purpose is to correct the images in the clear image register to the same position as the current blurred image to be restored, so In order to calculate the image weight value in the subsequent video repair process.

In the imaging of the image, basically the camera follows the perspective transformation model of small hole imaging, assuming any point in three-dimensional space , Its corresponding point , Where the homogeneous coordinates are with , According to the perspective transformation model, the three-dimensional point x and its corresponding point m should satisfy the formula (10) where Is a constant, versus Is the focal length of the camera, s is the tilt coefficient of the corresponding point coordinate, which depends on the angle between the X and Y axes of the corresponding point coordinate system, R is a 3 × 3 rotation matrix, and T is a 3 × 1 translation vector. Selecting a three-dimensional coordinate system with the two-dimensional plane XY plane overlap, the three-dimensional coordinate value Z can be expressed as 0, if the first rotation matrix R Column elements are Means that equation (10) can be rewritten as equation (11), where Represents any point on the two-dimensional plane Homogeneous coordinates of , So we can produce equation (12) where H is the homography matrix between the three-dimensional plane and the corresponding two-dimensional plane. Constant, so the mapping between 3D and 2D is transformed into a set of homography matrices. It takes four or more known feature points to obtain the homography matrix.

, (10)

(11)

, (12)

As mentioned above, when building a homography matrix, at least four sets of feature points are needed. Each set of feature points ( X, Y ) and their corresponding feature points ( u, v ) are brought into the formula (12) λm '= Hx ' Four sets of formulas (13) are generated, and each set of corresponding points can provide two linear equations of the homography matrix H. The four sets of formulas (13) are used to calculate the values of h ₁₁ ~ h _{33 in} the perspective transformation matrix H and Find the perspective transformation matrix H. If the parameter λ is set to 0, the above function can use Least Squares method to calculate its homography. However, not all points can adapt to this transmission transformation. If an abnormal value is calculated, it will cause a large deviation in its homography estimate. Therefore, the RANSAC (RANdom SAmple Consensus) method is used here to deal with this situation. A group of four points is randomly selected from the feature point group in the picture. The set and the least square method are used to estimate the homography matrix H. By minimizing the error The sum of squares finds the best function match of the data to satisfy the best function match of the current transformation matrix, that is, the sum of squares of the error between the data and the actual data is the smallest, and returns the data in the consistent set, and finally calculates the quality ratio of the homography matrix , Use the best set of the set as the value of the homography matrix. As shown in Figure 12, the result after perspective transformation, in the figure First The homography matrix of the picture, First The homography matrix of the picture The picture and the first After the pictures are calculated by the respective homography matrix, the pictures can be normalized and the same as the t-th picture.

(13)

4.2 Video repair

Image correction is helpful for pixel-to-pixel matching during pixel-to-pixel matching, which makes multiple image restoration more effective. By matching pixels to pixels, you can compare weight values between pixels , The weight value is calculated as in formula (14), l _{i, x} is to find the pixel point to replace the current blurred image, use the CK clear images in the image register to calculate the weight value, and finally select the pixel point with the highest weight value, To replace the pixel values of the blurred image, first calculate the weighted average of the surrounding pixels for each sequential pixel q of the image CB _{ti, q in} the register, as shown in formula (15), where I _{t, p} is the desired replacement Fuzzy reduction current pixel value p of the picture, t represents the time the picture frame number, here make calculation frame in the scratchpad CK, W is the weight of ownership ԝ (t, p, ti, q) of the sum, and ԝ ( t, p, ti, q ) The weight value is calculated as formula (16), H ( CB _ti , _q ) is CB _ti , _q The image processed by perspective transformation H is calculated by calculating H ( CB _ti , _q ) and the current between the Euclidean blurry frame (B _{t, p)} from the exponential function for calculation, This weight value is normalized between [0,1], if this both H (CB _{ti, q),} farther (B _{t, p)} of the distance, the exponentiation result is close to zero, This means that this point does not have any reference significance. Here, the constant σ = 0.5 is set, which is mainly to adjust the Euclidean distance calculated by H ( CB _ti , _q ) and ( B _t , _p ) to make The larger the distance, the more exponentially the value approaches 0. Finally, the pixel value with the largest weight value is selected to replace the pixel value of the current blurred image. As shown in Figure 13 after the restoration of multiple images, the left part of the figure is the original blurred image, and the right part is the result of restoration. It can be seen that the edge of the content of the restored image becomes clearer.

(14)

(15)

(16)

As mentioned above, the blurred image restoration method of the present invention can be applied to the restoration of linear motion blurred pictures. The proposed video deblurring method can be implemented in common image monitoring systems or the shooting of general mobile vehicles The device can be used for real-time deblurring through video streaming. Due to the low computational complexity of the algorithm proposed in this case, it can be embedded in a general security surveillance camera system to increase the added value of related products.

The above-mentioned embodiments are only for explaining the technical ideas and characteristics of the present invention. The purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. When the scope of the patent of the present invention cannot be limited, That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention should still be covered by the patent scope of the present invention.

S11 to S14, S61 to S68

FIG. 1 is a first flowchart of the method for restoring a blurred image of the present invention. Figure 2 is a cepstrum of a blurred image with constant velocity. Figure 3 is a cepstrum of a non-constant motion blurred image. FIG. 4 is a second flowchart of the blurred image restoration method of the present invention. Figure 5 is the search trajectory of the cepstrum. Figure 6 is a 3D image of the cepstrum domain. Figure 7 is a search trajectory diagram of a non-constant velocity fuzzy cepstrum. Figure 8 is the trajectory value distribution diagram. Figure 9 is a schematic diagram of the results of a single image restoration. Figure 10 is a schematic diagram of the homography matrix performing transformation between two images. Figure 11 is a schematic diagram of the results of corner matching: (a) part t-1 picture; (b) part t picture; (c) part t-1 picture and t picture corner Match result. Figure 12 is a schematic diagram of the results of perspective transformation. Figure 13 is a schematic diagram of the results of multiple image restoration processes.

Claims (5)

A method for restoring a blurred image includes the following steps: Fourier transform the blurred image to generate a converted value, then obtain the cepstrum of the converted value, and then calculate the blur angle and blur length of the point spread function from the cepstrum domain ; Normalize the point spread function according to the blur angle and the blur length, and then perform the deconvolution of the single image on the normalized point spread function to generate a restored image; select a clear image or store the restored image in a clear image Temporary register; and search for the corresponding feature points between the current image and the restored image stored in the clear image register, use the feature points to derive the homography matrix required for perspective transformation, and correct the current image accordingly And the corresponding pixels of the restored image, and then calculate the weights of the pixels in the current image and the restored image according to the time domain information and compare them, replace the low-weight pixels with high-weight pixels to generate restored pixels, and output another Restore the image. The method for restoring a blurred image as described in item 1 of the patent scope, wherein the clear image register stores the clear image in the image sequence when inputting the image sequence, and the blurred image in the image sequence is The restored image is restored by the restoration method of the blurred image. The method for restoring a blurred image as described in item 1 of the patent application scope, in which the point spread function is expressed by the following formula: Among them, L system is the blur length, and θ system is the blur angle. The method for restoring a blurred image as described in item 3 of the patent application scope, wherein the definition of the cepstrum domain of the blurred image is expressed by the following formula: Among them, F represents the Fourier transform, F ^-1 represents the inverse Fourier transform, and g (x, y) is a blurred image. The method for restoring a blurred image as described in item 4 of the patent application scope, wherein the calculation of the blur length and the blur angle of the point spread function includes the following steps: (1) Input the cepstrum to find the center point; (2 ) Select the point with the largest cepstrum value in the three directions of 90 °, 45 ° and 0 ° of the center point to use it as the search point; (3) Move to the search point and use a 5 * 5 mask Mask, the center of the mask corresponds to the search point; (4) add all the cepstrum values in the mask and store the coordinates of the search point; (5) determine whether the direction of the search point is offset, otherwise enter Step (6), if yes, go to step (7); (6) Calculate whether the distance between the search point and the center point is greater than the threshold, otherwise return to step (2) to repeat step (2) to step ( 5) If yes, proceed to step (7); (7) Find the turning point of falling and rising from the trajectory value distribution diagram; and (8) calculate the blur length and the blur angle based on the turning point and the center point; , The coordinate of the turning point is (x ₁ , y ₁ ), the coordinate of the center point is (x ₀ , y ₀ ), and L is the fuzzy length , Substituting (x ₁ , y ₁ ) and (x ₀ , y ₀ ) to obtain the fuzzy length is expressed by the following formula: R is the length of the base of the triangle formed by the turning point, the center point and the X axis. Substituting x ₁ and x ₀ to obtain the base length is expressed by the following formula: θ is the blur angle. Substituting the blur length and the bottom edge length to obtain the blur angle is expressed by the following formula: .