CN106960414B - Method for generating high-resolution HDR image from multi-view LDR image - Google Patents

Abstract

The invention discloses a method for generating a high-resolution HDR image from a multi-view LDR image. In step (1), two images with different viewing angles on the same straight line are acquired, and the content of the field of view of one image is required to be the same as that of the left and right adjacent images. There are overlapping parts; step (2), identify and match the feature points of two adjacent images, and extract the common part in the two images; step (3), the adjacent two images have different exposure lengths. The overlapping part uses contrast, saturation and moderate exposure as three measurement factors to generate a weight map, and performs multi-resolution pyramid fusion to generate an image with HDR effect; step (4), obtain two images with similar HDR effect; step (5) Integrate two images with similar HDR effects with the HDR image generated by the overlapping part to generate a high-resolution HDR image. The invention combines the process of fusion and splicing, and improves the real-time performance of multiple splicing and the frame rate and visual effect in the video system.

Description

A method for generating high-resolution HDR images from multi-view LDR images

technical field

The present invention relates to the field of image processing, in particular to a method for generating a high-resolution HDR image.

Background technique

The generation algorithm of HDR images and the stitching method of panoramic images have been researched a long time ago, and some of the more classic algorithms have been developed relatively maturely. The research on the generation method of HDR panoramic image or video that combines the two has just emerged in recent years, mainly because it conforms to the trend of social development and has many application values and prospects. The generation method of HDR panoramic image involves many steps, including image acquisition, image fusion, and image stitching. Each part can be realized by different methods considering different situations. It seems to be relatively independent, but the integrated method of each part, the overall design, is also a part of the researchers' focus.

The generation of high-resolution HDR and panoramic HDR images has become the focus of attention, and many scholars and institutions have carried out research on this. For example, Fumio Okura et al. applied HDR panorama to aerial photography, and introduced a complete system for obtaining HDR panorama images from the air. By fixing a stereo camera on the top and bottom of the drone, each camera obtained LDR through automatic exposure control. The image sequence is then obtained through a specific fusion algorithm to obtain an HDR image, and finally the HDR images obtained by the upper and lower cameras are fused to obtain the final HDR panoramic image.

Vladan Popovic et al. designed an FPGA-based image processing system, which can realize real-time HDR panoramic frame fusion by using the annular image acquisition device they designed. For every two adjacent cameras, one has a long exposure time and the other has a short exposure time. The overlapping parts of the adjacent images are used for image fusion, and a panoramic HDR image frame is finally generated. The designed system method is highly dependent on hardware. , it is not convenient to process the image directly.

Kvale Stensland等人通过不同角度的五个相机实现对足球场比赛现场的全面覆盖，通过拼接形成对整个足球场的全景影像，并且在每个角度，通过自动曝光获得长短曝光不同的两帧，对其融合生成HDR，再对各角度的HDR图像进行拼接形成最后的HDR全景图像帧。

Kvale Stensland et al. achieved comprehensive coverage of the football field through five cameras at different angles, formed a panoramic image of the entire football field by splicing, and obtained two frames with different length and short exposure through automatic exposure at each angle. It fuses to generate HDR, and then stitches the HDR images from various angles to form the final HDR panoramic image frame.

Fundamentally speaking, most of the current methods are to first obtain several frames of low dynamic range (LDR) images with different exposure times from different perspectives, fuse them to generate HDR images from different perspectives, and then analyze these HDR images from different perspectives. For splicing (as shown in Figure 1), this method takes a long time to obtain images, and it is cumbersome to perform fusion and splicing in steps.

SUMMARY OF THE INVENTION

Based on the prior art, the present invention proposes a method for generating a high-resolution HDR image from a multi-view LDR image. By using the overlapping parts of the obtained LDR images with different viewing angles, the fusion and splicing are combined, and a method based on histogram matching is proposed. high-resolution HDR image generation method.

The present invention provides a method for generating a high-resolution HDR image from a multi-view LDR image, the method comprising the following steps:

Step (1): Acquire two images with different viewing angles on the same straight line, and the two adjacent cameras are set to different exposure times. There are two types of exposure times, long and short, which alternate in adjacent cameras, and require an image to have different exposure times. The content of the field of view and the two adjacent images on the left and right have overlapping parts;

Step (2), using the surf operator to identify and match the feature points of two adjacent images, and obtain the specific value of the translational parallax by averaging the matched feature points, thereby extracting the common part in the two images; The specific method is: first, use the Hessian matrix to find the eigenvalues of each pixel, and the Hessian matrix is

Among them, L _xx (x,σ) is the second derivative of the image g(σ) in the x direction obtained by Gaussian filtering of the original image I, and L _xy (x, σ) and L _yy (x, σ) are also The second derivative of g(σ) in each direction.

The formula for calculating the eigenvalue is

det(H _approx )=D _xx D _yy -(0.9D _xy ) ²

Among them, D _xx , D _yy , and D _xy are respectively the second derivative of the Hessian matrix approximation template in the corresponding direction.

If the eigenvalue of a point is the largest among the 27 points in its field, the point can be considered as a feature point. Part of the feature points in the extracted image as shown in accompanying drawing 5;

To obtain the feature vector of the feature point, first calculate the main direction of the feature point. The specific process is as follows:

1) The statistics are centered on the feature point, proportional to a certain digital radius of the feature point scale, and sumX=(wavelet transform response in the y direction)*(Gaussian function), sumY= (X-direction wavelet transform response)*(Gaussian function), calculate the composite vector angle θ=arctan(sumY/sumX), and the modulus length sqrt(sumy*sumy+sumx*sumx).

2) Rotate the sector counterclockwise, and calculate the resultant vector in the same way.

3) Find the maximum value of the composite vector modulus length of the sector in each direction, and the corresponding angle is the main direction of the feature point.

The specific process of obtaining the feature vector is as follows:

1) Select a square area centered on the feature point, and align its rotation with the main direction.

2) Divide the square into 16 sub-regions of 4×4, and perform wavelet transform on each region to obtain 4 coefficients.

3) From the above two steps, a 4×4×4=64-dimensional vector is generated.

Calculate the inner product of the two vectors, the maximum value is the point that best matches the point, and a specific threshold is set. Only when the maximum value is greater than this threshold value can be considered to match the two feature points.

Step (3), fuse to generate HDR image: generate a weight map for the overlapping parts of two adjacent images with different exposures of different lengths and lengths, using contrast, saturation and moderate exposure as three measurement factors to generate a weight map, perform multi-resolution pyramid fusion, and generate HDR effect image; the specific formula for fusion is as follows:

为归一化后的权重值。where R _ij is the pixel value of the result image generated by fusion at the position (i, j), L _yy (x, σ) is the pixel value of the corresponding position of the kth input image,

is the normalized weight value.

The formula for calculating contrast is:

C=|h*I|

Among them, C represents the contrast, I is the image to be contrasted, and h is the Laplacian filter.

Saturation is obtained by calculating the standard deviation of the three chrominance channels. The specific calculation formula is as follows:

Among them, S represents saturation, I _R , I _G , and I _B are the pixel values of the three color channels R, G, and B, respectively, and μ is the average of the three.

Moderate exposure is estimated by a Gaussian curve, and the specific calculation formula is:

E＝E _R ×E _G ×E _B

Where E is the moderate exposure of the overall image, E _R , _EG , and _EB are the moderate exposure of each channel, here we specify σ=0.2.

The weight map calculation formula is:

w _C =w _S =w _E =1

Among them, C _ij,k , S _ij,k , E _ij,k are the contrast, saturation and moderate exposure of the pixel at the position (i, j) in the k-th image, respectively. The final weight value is obtained by multiplying the three measurement factors. w _C , w _S , and w _E are used to represent the "influence" of the three measurement factors in the generated weight map.

Step (4), obtain the reference image, that is, the histogram of each color channel of the HDR image fused and generated in step (3), take the HDR image generated by the fusion of the overlapping part as a reference, and perform a histogram on the obtained two original LDR images. Image matching, adjust the pixel value size of the original image by mapping so that the histogram of the adjusted image is approximately equal to the histogram of the reference image, so that the non-overlapping part of the image can obtain a tone similar to that of the HDR image, that is, to obtain a similar HDR effect two images of ;

Step (5), the HDR image generated by the fusion of the two images with similar HDR effect after histogram matching and the overlapping part, for the chromaticity value Pixel of each channel of the pixel point in the overlapping area of the image is corresponding to the two images. The gray value Pixel_L and Pixel_R of the point are weighted and averaged, namely:

Pixel=k×Pixel_L+(1-k)×Pixel_R

Among them, k=h/R, which is used to indicate the proportion of the distance h between the current pixel and the left border of the overlapping area to the total width R of the overlapping area; the larger the value of k, the larger the pixel value of the left image in the fusion. proportion. That is, in the overlapping area, along the direction of the left image to the right image, k gradually changes from 1 to 0, so as to realize the smooth splicing of the overlapping area; the splicing is performed through the weighted fusion splicing algorithm, and the adjacent places are smoothly transitioned to generate High resolution HDR images.

Compared with the prior art, the present invention effectively utilizes the overlapping parts of images from different viewing angles, skillfully combines the fusion and splicing processes, reduces the number of image frames required to obtain a high-resolution HDR image; improves the efficiency, The algorithm complexity is reduced; the real-time performance of multiple stitching, the frame rate in the HDR video system and the visual effect of improving the image quality are improved.

Description of drawings

Figure 1 is a schematic diagram of a commonly used HDR high-resolution image generation;

2 is a schematic diagram of an image acquisition process of the present invention;

3 is a schematic overall flowchart of a method for generating a high-resolution HDR image from a multi-view LDR image according to the present invention;

Fig. 4 is the original image obtained;

Figure 5 is the result of feature point identification and matching in the image by surf operator;

FIG. 6 is an HDR image generated by fusion of overlapping parts in two images;

Figure 7 shows the histogram matching of the initial LDR image with reference to the fusion result in the above figure, to obtain an image with a tone similar to that of the generated HDR image;

Figure 8 shows the final generated high-resolution HDR image obtained by stitching several images together through weighted fusion.

Detailed ways

The present invention will be described in further detail below with reference to the accompanying drawings.

Step (1): Acquire two images with different viewing angles on the same straight line, and the two adjacent cameras are set to different exposure times. There are two types of exposure times, long and short, which alternate in adjacent cameras, and require an image to have different exposure times. The content of the field of view and the two adjacent images on the left and right have overlapping parts;

Step (2), using the surf operator to identify and match the feature points of two adjacent images, and obtain the specific value of the translational parallax by averaging the matched feature points, thereby extracting the common part in the two images; The specific method is: first, use the Hessian matrix to find the eigenvalues of each pixel, and the Hessian matrix is

Among them, L _xx (x,σ) is the second derivative of the image g(σ) in the x direction obtained by Gaussian filtering of the original image I, and L _xy (x, σ) and L _yy (x, σ) are also The second derivative of g(σ) in each direction.

The formula for calculating the eigenvalue is

det(H _approx )=D _xx D _yy -(0.9D _xy ) ²

Among them, D _xx , D _yy , and D _xy are respectively the second derivative of the Hessian matrix approximation template in the corresponding direction.

If the eigenvalue of a point is the largest among the 27 points in its field, the point can be considered as a feature point. Part of the feature points in the extracted image as shown in accompanying drawing 5;

To obtain the feature vector of the feature point, first calculate the main direction of the feature point. The specific process is as follows:

1) The statistics are centered on the feature point, proportional to a certain digital radius of the feature point scale, and sumX=(wavelet transform response in the y direction)*(Gaussian function), sumY= (X-direction wavelet transform response)*(Gaussian function), calculate the composite vector angle θ=arctan(sumY/sumX), and the modulus length sqrt(sumy*sumy+sumx*sumx).

2) Rotate the fan shape counterclockwise (usually the step size is 0.1 radians), and calculate the composite vector in the same way.

3) Find the maximum value of the composite vector modulus length of the sector in each direction, and the corresponding angle is the main direction of the feature point.

The specific process of obtaining the feature vector is as follows:

1) Select a square area centered on the feature point, and align its rotation with the main direction.

2) Divide the square into 16 sub-regions of 4×4, and perform wavelet transform on each region to obtain 4 coefficients.

3) From the above two steps, a 4×4×4=64-dimensional vector is generated.

Calculate the inner product of the two vectors, the maximum value is the point that best matches the point, and a specific threshold is set. Only when the maximum value is greater than this threshold value can be considered to match the two feature points.

Step (3), fuse to generate HDR image: generate a weight map for the overlapping parts of two adjacent images with different exposures of different lengths and lengths, using contrast, saturation and moderate exposure as three measurement factors to generate a weight map, perform multi-resolution pyramid fusion, and generate HDR effect image; the specific formula for fusion is as follows:

为归一化后的权重值。Among them, R _ij is the pixel value of the result image generated by fusion at the position (i, j), L _yy (x, σ) is the pixel value of the corresponding position of the kth input image,

is the normalized weight value.

The formula for calculating contrast is:

C=|h*I|

Among them, C represents the contrast, I is the image to be contrasted, and h is the Laplacian filter.

Saturation is obtained by calculating the standard deviation of the three chrominance channels. The specific calculation formula is as follows:

Among them, S represents saturation, I _R , I _G , and I _B are the pixel values of the three color channels R, G, and B, respectively, and μ is the average of the three.

Moderate exposure is estimated by a Gaussian curve, and the specific calculation formula is:

E＝E _R ×E _G ×E _B

_Among them, _E is the moderate exposure of the overall image, _ER , EG, EB are the moderate exposure of each channel, here we specify σ=0.2.

The weight map calculation formula is:

w _C =w _S =w _E =1

Among them, C _ij,k , S _ij,k , E _ij,k are the contrast, saturation and moderate exposure of the pixel at the position (i, j) in the k-th image, respectively. The final weight value is obtained by multiplying the three measurement factors. w _C , w _S , and w _E are used to represent the "influence" of the three measurement factors in the generated weight map.

Step (4), obtain the reference image, that is, the histogram of each color channel of the HDR image fused and generated in step (3), take the HDR image generated by the fusion of the overlapping part as a reference, and perform a histogram on the obtained two original LDR images. Image matching, adjust the pixel value size of the original image by mapping so that the histogram of the adjusted image is approximately equal to the histogram of the reference image, so that the non-overlapping part of the image can obtain a tone similar to that of the HDR image, that is, to obtain a similar HDR effect two images of ;

Step (5), the HDR image generated by the fusion of the two images with similar HDR effect after histogram matching and the overlapping part, for the chromaticity value Pixel of each channel of the pixel point in the overlapping area of the image is corresponding to the two images. The gray value Pixel_L and Pixel_R of the point are weighted and averaged, namely:

Pixel=k×Pixel_L+(1-k)×Pixel_R

Among them, k=h/R, which is used to indicate the proportion of the distance h between the current pixel and the left border of the overlapping area to the total width R of the overlapping area; the larger the value of k, the larger the pixel value of the left image in the fusion. proportion. That is, in the overlapping area, along the direction of the left image to the right image, k gradually changes from 1 to 0, so as to realize the smooth splicing of the overlapping area; the splicing is performed through the weighted fusion splicing algorithm, and the adjacent places are smoothly transitioned to generate High resolution HDR images.

A specific embodiment of the present invention is shown in Figures 4-8. The Laplacian pyramid is generated from each chrominance channel of the original multi-exposure LDR image, so that the multi-resolution information of the original image is obtained; The exposure amount is used as an indicator to calculate the weight map of the original image, so that the proportion of the fused image is measured by the amount of information contained in this pixel; the weight map of different exposure images is normalized, that is The sum of the weights of different images of the same pixel is 1; then the obtained weight map is generated into a Gaussian pyramid; finally, the Laplacian pyramid of the same image is multiplied by the Gaussian pyramid generated by their weight map, and the pyramid obtained by different images is multiplied. Add the fused Laplacian pyramid, and restore the Laplacian pyramid to obtain the final HDR image generated by fusion.

Claims (1)

1. A method for generating a high resolution HDR image from a multi-view LDR image, the method comprising:

the method comprises the following steps that (1) two images with different visual angles on the same straight line are obtained, two adjacent cameras are set to have different exposure times, the two exposure times are different in total, the two exposure times are alternated in the adjacent cameras, and the view content of one image and the two left and right adjacent images are required to have overlapped parts;

identifying and matching feature points of two adjacent images by using surf operators, and obtaining a specific value of translational parallax by carrying out average operation on the matched feature points so as to extract a common part in the two images; the specific method comprises the following steps: firstly, the characteristic value of each pixel point is solved by using a Hessian matrix, wherein the Hessian matrix is

Wherein L is_xx(x, σ) is the second derivative of the original image I in the x direction of the image g (σ) obtained by Gaussian filtering, L_xy(x,σ)、L_yy(x, σ) are also the second derivatives of g (σ) in each direction;

the formula for calculating the characteristic value is

det(H_approx)＝D_xxD_yy-(0.9D_xy)²

Wherein D is_xx、D_yy、D_xyRespectively approximating second derivatives of the templates in the corresponding directions for the sea plug matrix;

if the characteristic value of a certain point is the largest of 27 points in the field, the point can be considered as the characteristic point;

acquiring a feature vector of a feature point, and firstly calculating the main direction of the feature point, wherein the specific process comprises the following steps:

1) counting a certain digit radius which takes the feature point as a center and is proportional to the dimension of the feature point, wherein the sum X of all pixel points in a fan-shaped area with an opening angle of 60 degrees is (y-direction wavelet transformation response) multiplied by a gaussian function, sum Y is (x-direction wavelet transformation response) multiplied by a gaussian function, and calculating a synthetic vector angle theta which is arctan (sum Y/sum X) and a modular length sqrt (sum my + sum x) multiplied by a gaussian;

2) rotating the sector along the counterclockwise direction, and calculating a synthetic vector in the same way; 3) the maximum value of the module length of the synthetic vector of each direction sector is obtained, the corresponding angle is the main direction of the characteristic point, and the specific process of obtaining the characteristic vector is as follows:

31) selecting a square area with the characteristic point as the center, and aligning the rotation of the square area with the main direction;

32) dividing the square into 16 sub-regions of 4 multiplied by 4, and performing wavelet transformation on each region to obtain 4 coefficients;

33) generating a 4 × 4 × 4 ═ 64-dimensional vector by the two steps;

calculating the inner product of the two vectors, setting a specific threshold value when the maximum value is the point which is most matched with the point, and considering that the two feature points are matched only when the maximum value is greater than the threshold value;

and (3) fusing to generate an HDR image: generating a weight map for the overlapped part of two adjacent images with different lengths of exposure by taking contrast, saturation and proper exposure as three measure factors, and performing multi-resolution pyramid fusion to generate an image with HDR effect; the specific formula of fusion is as follows:

wherein R is_ijThe pixel value, L, of the resulting image at the (i, j) position generated for the fusion_yy(x, sigma) is the pixel value of the corresponding position of the kth input image,

the normalized weighted value is obtained;

the contrast ratio calculation formula is:

C＝|h*I|

wherein, C represents contrast, I is an image with the contrast to be obtained, and h is a Laplace filter;

the saturation is obtained by calculating the standard deviation of three chrominance channels, and the specific calculation formula is as follows:

wherein S represents saturation, I_R、I_G、I_BR, G, B pixel values of three color channels, mu is the average value of the three;

the moderate exposure is estimated by a Gaussian curve, and the specific calculation formula is as follows:

E＝E_R×E_G×E_B

where E is the overall moderate exposure of the image, E_R、E_G、E_GThe exposure dose for each channel is appropriate, where σ is defined as 0.2;

the weight graph has the following calculation formula:

w_C＝w_S＝w_E＝1

wherein, C_ij,k、S_ij,k、E_ij,kRespectively representing the contrast, saturation and proper exposure of a pixel point at the (i, j) position in the k image; the final weight value, w, is obtained by multiplying the three measurement factors_C、w_S、w_EIs used for representing the 'influence' magnitude of the three measurement factors in generating the weight graph;

step (4), a reference image, namely a histogram of each color channel of the HDR image generated by fusion in the step (3), is solved, the HDR image generated by fusion in the overlapped part is taken as a reference, histogram matching is carried out on the two obtained original LDR images, the size of the pixel value of the original image is adjusted through mapping so that the histogram of the adjusted image is approximately equal to that of the reference image, so that the part which is not overlapped in the image obtains a tone similar to the HDR image, namely the two images with similar HDR effect are obtained;

and (5) fusing the two images which are subjected to histogram matching and have similar HDR effects with the overlapped part to generate an HDR image, wherein as for the chromatic value Pixel of each channel of the Pixel point in the image overlapped area, the chromatic value Pixel is obtained by weighted average of the gray value Pixel _ L and the Pixel _ R of the corresponding point in the two images, namely:

Pixel＝k×Pixel_L+(1-k)×Pixel_R

wherein, k is h/R, which is used to represent the proportion of the distance h between the current pixel point and the left boundary of the overlap region to the total width R of the overlap region; the larger the k value is, the larger the proportion of the pixel value of the left image in the fusion is; namely in the overlapping area, in the direction from the left image to the right image, k gradually changes from 1 to 0, so that the smooth splicing of the overlapping area is realized; and (4) carrying out splicing through a weighted fusion splicing algorithm, and smoothly transiting at the adjacent positions so as to generate a high-resolution HDR image.

Images