CN103226830B - The Auto-matching bearing calibration of video texture projection in three-dimensional virtual reality fusion environment - Google Patents

Abstract

The invention relates to an automatic matching correction method for video texture projection in a three-dimensional virtual-real fusion environment and a real video image and virtual scene fusion method. The steps of the automatic matching correction method include: building a virtual scene, acquiring video data, video texture fusion, and projector correction. Using the real video shot, through the way of texture projection, the virtual scene fusion is carried out on the complex ground surface and the scene surface of the building, which improves the expression and display ability of the dynamic information of the scene in the virtual reality environment, and also enhances the sense of layering of the scene. By increasing the number of videos taken from different shooting angles, the dynamic video texture coverage effect of a large-scale virtual scene can be realized, thereby realizing the dynamic realistic effect of virtual reality fusion of the virtual reality environment and the displayed scene. Through the pre-color consistency processing of the video frame, the obvious color jump is eliminated and the visual effect is improved. Through the automatic correction algorithm proposed by the present invention, the fusion of virtual scene and real video is more accurate.

Description

Automatic Matching and Correction Method for Video Texture Projection in 3D Virtual Reality Fusion Environment

technical field

This article relates to virtual reality, in particular to a method for fusing and correcting real video images and virtual scenes, and belongs to the technical fields of virtual reality, computer graphics, computer vision, and human-computer interaction.

Background technique

In a virtual reality system, it is the most common method to use static pictures to express the details of buildings or ground surfaces, which are usually realized by texture mapping. The disadvantage of this method is that the texture of the scene surface will not change once it is set, and the neglect of the changing elements of the scene model surface reduces the sense of reality of the virtual environment and cannot give people an immersive feeling. In order to eliminate the lack of realism caused by static pictures, it is an intuitive idea to use video instead of pictures. At this stage, there are also some systems that add video elements, but most of them use the form of pop-up windows and use existing video players to play videos, which only achieves the effect of global monitoring, but does not achieve the real integration of video and scenes. Some research work has improved on this basis, by constructing an additional plane in space and playing video on this plane to enhance the sense of reality (see K.Kim, S.Oh, J.Lee, I.Essa.AugmentingAerialEarthMapswithDynamicInformation . IEEE international Symposium on Mixed and Augmented Reality, Science and Technology Proceedings. 19-22 Oct, 2009, Orlando, Florida, USA. and Y. Wang, D. Bowman, D. Krum, E. Coelho, T. Smith-Jackson, D. Bailey, S. Peck, S. Anand, T.Kennedy, and Y.Abdrazakov.Effects ofVideoPlacementandSpatialContextPresentationonPathReconstructionTaskswithContextualizedVideos.IEEETransactionsonvisualizationandcomputergraphics, Vol.14,No.6,November/December2008.), although the above method has added the video to the virtual environment, but the use of the environment is very limited and can only be attached to On some large building planes or flat ground, for slightly complex scene conditions, such as building corners or uneven ground, their geometric shapes cannot be approximated by planes, and these methods of playing video on planes are not applicable.

On the other hand, due to the development of graphics and vision fields, there are already many mature algorithms, such as color-based matching, texture matching, and feature matching (EdgeDirection, SIFT, HOG). However, these methods are all applied to two-dimensional images, and have relatively large limitations when used in three-dimensional space. Moreover, the algorithms related to projector correction at this stage are mostly algorithms for trapezoidal correction of the projection area under the "projector-screen" system, such as: multi-projector image correction method and equipment, application number 201010500209.6, and the correction method is limited to two-dimensional space In this method, the correction parameters corresponding to the independent image information of the non-overlapping areas collected by the cameras are respectively obtained, and the correction process is performed according to the video data of the cameras corresponding to the correction parameters. The correction is only for images with overlapping or small overlapping areas. Real 3D display method based on multi-projector rotating screen 3D images that can be touched, application number: 200810114457.X, by obtaining 3D space description to obtain cross-sectional images at different angles, so that the hands can directly touch the 3D images, and at the same time improve the 3D images contrast, but this application mainly relies on the rotating screen to solve the problem that the 3D image can be touched, which is different from the application scenario of this work.

The above patent application or the feature matching method in the prior art does not have much reference significance for the calibration of the three-dimensional space projector.

Contents of the invention

The purpose of the present invention is to use the captured real video to perform virtual scene fusion on complex ground surfaces and buildings and other scene surfaces by means of texture projection, so as to improve the expression and display capabilities of scene dynamic information in the virtual reality environment, and also enhance The layering of the scene can be achieved, and by increasing the number of videos from different shooting angles, the dynamic video texture coverage effect of a large-scale virtual scene can be realized, so as to realize the dynamic realistic effect of the virtual reality fusion of the virtual reality environment and the displayed scene.

In order to realize technical purpose, the present invention adopts following technical scheme:

An automatic matching and correction method for video texture projection in a three-dimensional virtual-real fusion environment, the steps of which include:

1) Based on pre-acquired remote sensing data images, establish a surface model with static texture images on the surface and a virtual scene composed of multiple models containing 3D geometry and texture; obtain multiple real shooting video streams and record the camera pose information when shooting ;

2) Add a virtual projector model and a projector’s viewing volume corresponding to the camera parameters in the virtual scene according to the camera pose information at the time of shooting, and set the virtual projector model according to the camera pose information The initial pose value in the virtual scene;

3) Perform video frame preprocessing on the image of the real shot video stream to obtain dynamic video texture, and project the preprocessed video data into the virtual environment by using projection texture technology;

4) Fusing the static texture of the model surface in the virtual environment and/or the original remote sensing image texture of the ground surface with the dynamic video texture to obtain the final texture value of the surface coverage of the scene;

5) According to the final texture value, the virtual projector is obtained from the virtual projector model by means of rendering as an image under the viewpoint, and matched with the corresponding image in the real shot video stream to construct an energy function;

6) Using the optimal solution in the energy function to reset the initial pose value of the projector in the virtual scene to complete the calibration of the virtual projector.

Further, the texture fusion method in step 4) is as follows:

1) Reset the model view matrix and projection matrix to transform the virtual viewpoint to the projector viewpoint, draw the virtual scene, and obtain the depth value under the current projector viewpoint (use Z-Buffer to realize depth buffer);

2) Reset the model view matrix and projection matrix to change the viewpoint back to the virtual viewpoint, redraw the virtual scene, and obtain the real depth value corresponding to each point in the scene;

3) Draw the virtual scene sequentially under each projector viewpoint, obtain the projected texture coordinates of each point in the scene through automatic texture generation, and compare the real depth value obtained in the above steps 1) and 2) with the depth Comparison of values (using Z-Buffer to implement depth buffer);

4) If the two are equal, use the video texture of the projector; if not, use the texture of the scene model itself, and iterate by setting the texture combiner function until all the projectors in the scene are traversed to obtain each point in the scene The final texture value.

Further, the step 5) is matched with the corresponding image in the real shooting video stream, and the construction method of the energy function with the pose information as the independent variable is established as follows:

The first step is to reset the model-view matrix and projection matrix, adjust the viewpoint in the virtual scene to the projector, draw the scene to get an image in the virtual environment, use the mean-shift algorithm to segment the image, and then binary the image treatment;

The second step is to extract a key frame from the real shooting video stream, and use the method of the first step to do binarization;

The third step is to calculate the contour error in the viewing volume area formed by the projector, perform XOR processing pixel by pixel on the images obtained in the first two steps, and count the number of pixels whose result is 1, which is the first part of the energy function;

The fourth step is to use the SIFT consistency operator to add the features of local information, collect the matching point pairs in the image that has not been binarized obtained in the first and second steps, and obtain the key point constraint (Key-point constraint) process The error value of the matching point pair, which is the second part of the energy function;

The fifth step is to assign different weights to the two parts of the energy function;

The sixth step is to solve the optimal value of the energy function,

The seventh step is to replace the initial pose value of the projector with the optimal solution.

Furthermore, the optimal value of the energy function is solved according to the following method:

First, the simulated annealing algorithm is applied to the energy function to reduce the solution space of the function to the approximate range of the optimal solution, and then the downhillsimplex algorithm is used to compress the approximate solution space to obtain the optimal solution.

Furthermore, the first and second steps use the mean-shift algorithm to segment the image by utilizing the color features of buildings and roads, setting the pixel values of non-buildings or road areas to white, and retaining the corresponding areas of building models or roads, Then binarize the image, and set the building and road-related areas to black.

Further, the method for performing video frame preprocessing on the image of the real shot video stream is as follows:

The video data is decoded to obtain a single video image frame, and a sample frame is extracted from each video stream, and the SIFT operator is used to find the matching of feature points in the sample frame, and the color consistency is processed.

Further, the color consistency processing is:

1) Extract a sample frame from each of the two matching videos, construct a color histogram formed by all pixels in the frame, and make the two video frames have the same color histogram through color histogram equalization and specification processing distributed;

2) Do the same histogram equalization and specification processing as the corresponding sample frame for each frame in the same video stream, thereby completing the consistent processing for the entire video stream;

3) Create a cache for the video frame, which can accommodate about 50 video frames (the video frame resolution is 1920*1080);

4) Load the frame data using the list structure of first-in-first-out (FIFO).

Furthermore, the real shooting video stream is obtained through the http protocol, the video data is decoded locally, and the video frame is saved in Jepg format.

Furthermore, multi-resolution processing is performed on the video image frame, and video frames of different resolutions are loaded into the same image according to different situations, and a bilinear interpolation operation is performed pixel by pixel to extract the image into the original image One or more of 1/4, 1/16, 1/64.

Furthermore, an Alpha channel is added to the Jepg format video image.

The present invention also proposes a real video image and a virtual scene fusion method, the steps of which are:

1) Establish a static texture image model and a virtual scene based on pre-acquired remote sensing data images; the spatial position of the model in the virtual scene and the relative position, orientation, and size of the models are consistent with the real scene;

2) Obtain multiple real shooting video streams and record the pose information of the camera where the shooting takes place;

3) The realization of the method of the present invention can be established on a virtual reality platform based on digital earth, and each virtual projector has geographic positioning information and Cartesian coordinate representation of virtual reality. The longitude and latitude coordinates of the earth’s surface where the shooting is located are converted to the world coordinates represented by the Cartesian coordinates where the virtual scene is located, combined with adding a virtual projector model and the corresponding viewing volume of the projector model to the virtual scene, and at the same time according to the camera The pose information sets the initial pose value in the virtual scene of the virtual projector model in the world coordinate system;

4) Perform video frame preprocessing on the image of the real shot video stream to obtain dynamic video texture, and project the preprocessed video data into the virtual environment by using projection texture technology;

5) Fusing the static texture of the model in the virtual environment and/or the original remote sensing image texture of the ground surface with the dynamic video texture;

6) Texture fusion is used for the intersecting coverage areas of different projectors in the virtual projector model.

Beneficial effects of the present invention

(a) Overcoming complex scene conditions, the integration of video and virtual scenes is realized, and the original terrain remote sensing texture and the inherent rough static image texture of the model are replaced by video texture, which adds dynamic information to the virtual scene texture and improves visual effects. And expand the scope of influence by increasing the number of videos.

(b) A cache structure is provided for the video, and a data pyramid is built to improve the display efficiency, and the data replacement of two adjacent layers is possible.

(c) An automatic correction algorithm is provided to adjust the initial pose of the virtual projector to make the fusion of the virtual scene and the real video more accurate. Compared with the initial position, it is more accurately reflected in the value of the energy function, the position The more precise, the closer the value of the energy function is to zero.

(d) Perform pre-color consistency processing on video frames to eliminate obvious color jumps and improve visual effects.

Description of drawings

Fig. 1 is a schematic diagram of the specific operation implementation process in an embodiment of the automatic matching and correction method for video texture projection in a three-dimensional virtual-real fusion environment of the present invention;

Fig. 2a and Fig. 2b are schematic diagrams of scenes without adding projection texture in an embodiment of the automatic matching and correction method for video texture projection in a three-dimensional virtual-real fusion environment of the present invention;

Fig. 3a and Fig. 3b are schematic diagrams of scenes with projection texture added in an embodiment of the automatic matching and correction method for video texture projection in a three-dimensional virtual-real fusion environment of the present invention;

Fig. 4 is a schematic diagram of a scene where the projector has not been corrected in an embodiment of the automatic matching and correction method for video texture projection in a three-dimensional virtual-real fusion environment;

Fig. 5 is a schematic diagram of a corrected scene of a projector in an embodiment of an automatic matching correction method for video texture projection in a three-dimensional virtual-real fusion environment according to the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the described embodiments are only some of the embodiments of the present invention, not all of them. example. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

(1) Construct a virtual scene. Use pre-acquired remote sensing images to set terrain textures, construct models with static texture images on the surface and virtual scenes composed of models in virtual space, the spatial position of models in the scene and the relative position, orientation, size and other elements between models should be as close as possible It may be consistent with the real scene.

(2) Acquire video data. The data source can be a surveillance camera, or a video image shot by a mobile device. At the same time, the parameter information of the camera at the time of shooting is obtained, which is used for the initial positioning of the projector in the virtual space. Extract the video stream, obtain a single frame image, and perform multi-resolution processing and color and light consistency processing on the image.

(3) Video texture fusion. According to the latitude and longitude information of the camera obtained in step (2), add the virtual projector model and its view volume in step (1), and set the orientation of the virtual projector model through the camera pose information obtained in step (2). At present, due to the limitation of the implementation method, in the viewing volume space of a single virtual viewpoint, at most 32 virtual projector models can be loaded at the same time. The video data is projected into the virtual environment by using projection texture technology, and the original remote sensing image texture of the model or the ground is fused with the dynamic video texture. If different projectors have intersecting coverage areas, the blending operation also needs to be applied to this area.

(4) Projector calibration. Obtain the image under the virtual projector viewpoint and match it with the image in the related real video. Through the algorithm in the present invention, the range of differences between buildings or roads in the virtual image and the real image, as well as the difference in local features are calculated, an energy function is constructed, and an optimal solution in the energy function is obtained. Use the optimal solution to reset the projector in the virtual scene to complete the calibration process and improve the effect.

The method of the present invention is described in detail below from several aspects.

First, let's explain some concepts in detail:

Camera: A video source in real space, used to acquire video data.

Projector: A virtual model in a virtual scene, used to project video textures in a virtual scene.

Texture fusion: A model can apply several textures from different sources, so it is necessary to fuse different texture color values at the same point to obtain the final color value.

Depth value: The value obtained after perspective transformation at any point in space represents the distance from the virtual viewpoint in the Z direction.

Depth buffer (Z-Buffer): A buffer of the same size as the color buffer saved after the scene is rendered, each element in the buffer stores a depth value in the scene, indicating that the 3D scene corresponding to the element is closest to the viewpoint The depth value that the surface of the object has.

Projection texture technology: Different from the traditional four-point texture mapping method, the texture is applied to the virtual scene in the form of projection, and it is integrated with the building and/or terrain in the virtual scene as the final texture of the building and/or terrain.

Technical solution (2) The specific implementation plan is as follows: For video data, obtain it through the traditional http protocol, decode the video data locally, obtain a single video frame, save the video in Jepg format, and extract a sample from each video stream Frame, use the SIFT operator to find the matching of feature points in the sample frame (here, the pre-classification process can be performed on the video source, and the video source of the same building can be divided together, which can reduce the time-consuming matching process and improve the pre-processing efficiency) , and perform color consistency processing.

The specific operation of color consistency processing is to extract a sample frame from each of the two videos for matching, construct a color histogram formed by all pixels in the frame, and equalize and prescribe the color histogram to make the two video frames Has the same color histogram distribution, all frames in the same video stream have approximately the same color histogram distribution as the sample frame, so for each frame in the same video stream, do the same histogram equalization as the corresponding sample frame Standardization and prescriptive processing, thereby completing consistent processing for the entire video stream. The purpose of this processing is to make the video textures in overlapping areas have the same texture color, improve the fusion effect between videos, and avoid obvious jumping effects. Since the parsed video frames are relatively large and memory resources are precious, a cache is created for the video frames, with a size of 50 video frames. The reason for choosing 50 here is that in the process of obtaining and parsing the video stream, the maximum resolution of a single video frame is 1920*1080, and each pixel has 4 bytes, namely RGB and Alpha channels, where Alpha determines The degree of translucency of the image, its value ranges from 0 to 255 (0 means opaque, 255 means fully transparent), then reading a frame of video will consume 1MB of space, and 30 channels of video will consume 30MB of memory. If the memory is 1G It is estimated that if you do more, you can cache 50 frames. Adopt the list structure of FIFO. If the cache space is full, pause incoming entry waiting. If the display speed is faster than the loading speed due to network problems, return to the previous frame of video until a new frame data is incoming. An optimization solution provided by the present invention: Another space-saving strategy is to perform multi-resolution processing on video frames, build a pyramid for the same image, and load video frames with different resolutions in different situations to save memory overhead. The specific method is to perform bilinear interpolation operation pixel by pixel, and extract the image into 1/4, 1/16, 1/64 of the original image. In addition, an optimization solution provided by the present invention: in order to improve the efficiency and enable the video frame to be directly used in the texture projection algorithm, it is necessary to add an Alpha channel to the video frame image as a fusion parameter between the video and the scene or between the videos.

The specific implementation plan of technical solution (3) is as follows:

First, transform the virtual view point to the projector view point by resetting the model view matrix and projection matrix, clear the depth buffer under the virtual view point, and set the polygon offset and color mask, and draw the corresponding virtual view in technical solution (1). Scene, obtain the depth buffer under the current projector as the viewpoint, and form a depth texture;

Secondly, change the view point back to the virtual view point by resetting the model view matrix and projection matrix, clear the color and depth buffers, and draw the corresponding virtual scene in the technical solution (1), including its surface texture, thus obtaining each scene in the scene The real depth value corresponding to the point.

Finally, the scene is drawn sequentially under each projector viewpoint by resetting the modelview matrix and projection matrix. The projected texture coordinates of each point in the scene are obtained by automatic texture generation, and the final texture value of each point in the scene is determined by comparing the real depth value obtained in the first and second steps with the Z-Buffer value. If both If they are equal, use the projector video texture, if not, use the scene model's own texture. Iterate this process until all projectors in the scene have been traversed.

For the fusion between different videos, it is realized by setting the texture combiner function. Because the color consistency correction has been completed in the video frame preprocessing process, the Replace method is used here, that is, the original value is replaced by the subsequent texture fragment.

The specific implementation of the technical solution (4) is as follows: the correction of the pose of the projector is the correction of the three-dimensional coordinates x, y, z of the projector in the virtual scene and the deflection angles φ, θ, γ in three directions.

The pose value obtained during video acquisition is used as the initial value in the virtual scene of the virtual projector, but due to the influence of equipment precision, this value cannot completely integrate the projected texture with the virtual space. Therefore, an additional correction process is required. The present invention corrects the virtual projector by constructing an energy function with pose information as an independent variable and solving the optimal value of the energy function.

First, adjust the viewpoint in the virtual scene to the projector by resetting the model-view matrix and projection matrix, draw the scene to obtain an image in the virtual environment, use the mean-shift algorithm to segment the image, and use the Color features, set the pixel values of non-building or road areas to white, and only keep the area corresponding to the building model or road, and then perform binary processing on the image, and set the building and road related areas to black.

The second step is to extract a key frame from the video, and use a method similar to the first step to retain the area corresponding to the building model or road for binarization.

The third step is to calculate the contour error in the viewing volume area of the projector

The image obtained in the first two steps is XORed pixel by pixel, and finally the number of pixels with a result of 1 is counted, and the result is used as the first part of the energy function.

In the fourth step, for buildings with axisymmetric appearance, if only shape matching is performed, wrong results may occur, so some local information features need to be added. Use the SIFT consistency operator to collect matching point pairs in the image that has not been binarized in the first two steps. Calculate the error value of the matching point pair through the Key-point constraint process, and use this value as the second part of the energy function.

The fifth step is to assign different weights to the two parts of the energy function. The present invention assigns more weights to the global contour error, and the energy function is constructed so far.

The sixth step is to solve the optimal value of the energy function. Firstly, the simulated annealing algorithm is applied to the energy function to reduce the solution space of the function to the approximate range of the optimal solution. Then the downhillsimplex algorithm is used to further compress the approximate solution space to obtain the optimal solution.

The seventh step is to use the optimal solution replacement technology to realize the initial projector pose value in (3).

According to the process of video fusion and correction generation, this embodiment can be divided into the following steps for implementation:

1Construct a virtual scene

Taking the virtual campus as an example, a data pyramid is constructed through the existing terrain remote sensing data, and textures of different levels are bound to the terrain at different viewpoints. Create models of landmark buildings on campus, and manually add the models to corresponding positions based on terrain remote sensing data, so that the relative positional relationship between buildings is as consistent as possible with reality. See Figure 1 step (2) Remote sensing terrain input data → (3) LOD processing → (4) Terrain texture value → (6) Model data → (7) Spatial position calibration → (8) Model texture.

2 Get video data

Get unprocessed video streams from different video sources, such as campus surveillance cameras or cameras, mobile phones. Extract the video stream into a single frame image, and perform multi-resolution processing on the image to create images with different resolutions, and add an Alpha channel to the image to facilitate the fusion between subsequent videos and between videos and scenes. In addition, the latitude and longitude, viewing angle and direction information of the video source are saved for initial positioning of the projector in the virtual space. See Figure 1 step (11) for the extraction of video frames, build multi-resolution, add alpha channel for the effect → (13) video stream → (14) projector texture.

3 video texture fusion

Blend video images with terrain and model textures. It is realized by three scene renderings. The object is drawn at the perspective of the projector for the first time to obtain the corresponding Z-Buffer, and the scene is drawn at the virtual viewpoint for the second time to obtain the real depth value of each point in the scene. In the third drawing, the depth value obtained by the first two drawing is compared to determine the texture value of each point in the scene. The fusion process is realized by setting different texture fusion devices. See Figure 1 Step (1) Automatic generation of texture coordinates → (5) Comparison of Z-buffer value and real depth value → (9) Multi-pass rendering of virtual scene → (10) Texture combiner function → (12) Final image.

4 Projector Calibration

Place the viewpoint at the projector, draw the scene, and obtain the virtual scene image. Find its video stream in the real scene according to the projector, and extract one of the key frames. For image segmentation processing of the above two images, there are many algorithms that can be selected, such as mean-shift, normalized cut, JSEG, pixelaffinity, and the present invention uses the mean-shift algorithm. Divide the image into different areas, extract the ground or building parts according to the color features, and remove the irrelevant parts. The separated image is normalized, the part where the model is not is set to white, and the part where the model is is set to black. Then perform a pixel-by-pixel XOR operation on the two images. If the resolutions of the two images are different, a consistency process needs to be added. Count the pixels with a result of one as the first part of the energy function. Contour matching is a global comparison method that requires some local feature matching as a supplement. Here, the SIFT feature matching operator is used to select several groups of feature points, calculate their key-pointerror values, and superimpose this value as the first energy function. two parts. At this time, the calibration problem of the projector is transformed into the problem of finding the optimal solution of the energy function of the multivariate independent variables. The present invention uses a combination of the simulated annealing algorithm and the downhillsimplex algorithm. The approximate optimal solution is found by the simulated annealing algorithm, and then the projector position is optimized in a small range by the downhillsimplex algorithm. After finding the optimal solution, replace the value with the pose of the virtual projector to complete the calibration process. See Figure 1 step (15) Image segmentation based on Mean-shift → (16) SIFT operator extracts local feature matching points → (17) Virtual projector pose calibration value → (18) Building or road extraction, and do different Or operation → (19) Key-Point error → (20) Downhillsimplex → (21) Simulated annealing → (22) Construct energy function.

Claims (10)

1. An automatic matching correction method for video texture projection in a three-dimensional virtual-real fusion environment, the steps comprising: 1) Establish a surface model and a virtual scene with static texture images on the surface according to the pre-acquired remote sensing data images; obtain multiple real shooting video streams and record the camera pose information when shooting; 2) Add a virtual projector model and a projector’s viewing volume corresponding to the camera parameters in the virtual scene according to the camera pose information at the time of shooting, and set the virtual projector model according to the camera pose information The initial pose value in the virtual scene; 3) performing video frame preprocessing on the images of the real captured video streams to obtain dynamic video textures, and projecting the preprocessed video data into the virtual environment using projection texture technology; 4) Fusing the static surface texture of the surface model in the virtual environment and/or the original remote sensing image texture of the surface with the dynamic video texture to obtain the final texture value of the scene surface coverage, including: 4-1) Resetting the model-view matrix and the projection matrix to transform the virtual viewpoint to the projector viewpoint, draw the virtual scene, and obtain the depth value under the current projector viewpoint; 4-2) Resetting the model-view matrix and the projection matrix to change the viewpoint back to the virtual viewpoint, redraw the virtual scene, and obtain the real depth value corresponding to each point in the scene; 4-3) Draw the virtual scene sequentially under each projector viewpoint, obtain the projected texture coordinates of each point in the scene through automatic texture generation, and perform the above steps 4-1), 4-2) to obtain the real a comparison of a depth value with said depth value; 4-4) If the two are equal, use the video texture of the projector; if they are not equal, use the texture of the scene model itself, and iterate by setting the texture combiner function until all projectors in the scene are traversed, and each image in the scene is obtained. The final texture value of points; 5) Acquiring the virtual projector from the virtual projector model as an image under the viewpoint according to the final texture value, and matching with the corresponding image in the real shooting video stream to construct an energy function; 6) Using the optimal solution in the energy function to reset the initial pose value of the projector in the virtual scene to complete the calibration of the virtual projector. 2. the automatic matching correction method of video texture projection in the three-dimensional virtual and real fusion environment as claimed in claim 1, it is characterized in that, described step 5) matches with the corresponding image in the real shooting video stream, and establishes the pose information as an independent variable The energy function construction method of is as follows: The first step is to reset the model-view matrix and projection matrix, adjust the viewpoint in the virtual scene to the projector, draw the scene to get an image in the virtual environment, use the mean-shift algorithm to segment the image, and then binary the image treatment; The second step is to extract a key frame from the real shooting video stream, and use the method of the first step to do binarization; The third step is to calculate the contour error in the viewing volume area formed by the projector, perform XOR processing pixel by pixel on the images obtained in the first two steps, and count the number of pixels whose result is 1, which is the first part of the energy function; The fourth step is to use the SIFT consistency operator to add the features of local information, collect the matching point pairs in the images that have not been binarized in the first and second steps, and obtain the matching points through the Key-point constraint process The error value of the pair, the error value is the second part of the energy function; The fifth step is to assign different weights to the two parts of the energy function; The sixth step is to solve the optimal value of the energy function, The seventh step is to replace the initial pose value of the projector with the optimal solution. 3. The automatic matching correction method of video texture projection in the three-dimensional virtual-real fusion environment as claimed in claim 1 or 2, is characterized in that, described energy function optimal value is solved according to the following method: First, the simulated annealing algorithm is applied to the energy function to reduce the solution space of the function to the approximate range of the optimal solution, and then the downhillsimplex algorithm is used to compress the approximate solution space to obtain the optimal solution. 4. the automatic matching correction method of video texture projection in the three-dimensional virtual and real fusion environment as claimed in claim 2, is characterized in that, described first, second step utilizes the color of building and road when image is segmented by mean-shift algorithm Features, set the pixel values of non-building or road areas to white, retain the area corresponding to the building model or road, and then perform binarization on the image, and set the building and road related areas to black. 5. the automatic matching correction method of video texture projection in the three-dimensional virtual and real fusion environment as claimed in claim 1, is characterized in that, the method for carrying out video frame preprocessing to the image of described real shooting video stream is as follows: The video data is decoded to obtain a single video image frame, and a sample frame is extracted from each video stream, and the SIFT operator is used to find the matching of feature points in the sample frame, and the color consistency is processed. 6. the automatic matching correction method of video texture projection in the three-dimensional virtual-real fusion environment as claimed in claim 5, is characterized in that, described color consistency is processed as: 1) Extract a sample frame from each of the two videos for matching, construct a color histogram formed by all pixels in the frame, and make the two video frames have the same color histogram through color histogram equalization and prescriptive processing distributed; 2) Perform the same histogram equalization and prescriptive processing on each frame in the same video stream as the corresponding sample frame, thereby completing the consistent processing on the entire video stream; 3) Create a cache for video frames, the size of which can accommodate 50 video frames; 4) The video frame data is loaded using the list structure of the first-in-first-out FIFO. 7. the automatic matching correction method of video texture projection in the three-dimensional virtual-real fusion environment as claimed in claim 5, it is characterized in that, described real shot video stream is obtained by http protocol, video data decoding is carried out locally, and video frame is saved It is in Jepg format. 8. The method for automatic matching and correction of video texture projection in a three-dimensional virtual-real fusion environment as claimed in claim 5, wherein the video image frame is subjected to multi-resolution processing, and the same image is loaded with different resolutions according to different situations. High-rate video frame, adopts bilinear interpolation operation pixel by pixel, and extracts the image into one or more of 1/4, 1/16, 1/64 of the original image. 9. the automatic matching correction method of video texture projection in the three-dimensional virtual-real fusion environment as claimed in claim 7, is characterized in that, increases Alpha channel to described Jepg format video image. 10. A real video image and virtual scene fusion method, the steps of which are: 1) Establishing a static texture image model and a virtual scene on the surface according to pre-acquired remote sensing data images; the spatial position of the model in the virtual scene and the relative position, orientation, and size between the models are consistent with the real scene; 2) Obtain multiple segments of real shooting video streams and record the pose information of the camera where the shooting takes place; 3) Convert the latitude and longitude coordinates of the earth surface where the shooting is located to the world coordinates represented by Cartesian coordinates where the virtual scene is located and combine them, and add a virtual projector model and a visual volume corresponding to the projector model to the virtual scene , and set the initial pose value in the virtual scene of the virtual projector model in the world coordinate system according to the camera pose information; 4) performing video frame preprocessing on the images of the real shooting video stream to obtain dynamic video textures, and projecting the preprocessed video data into a virtual environment using projection texture technology; 5) Fusing the static texture of the model in the virtual environment and/or the original remote sensing image texture of the ground surface with the dynamic video texture, including: 5-1) Resetting the model-view matrix and the projection matrix to transform the virtual viewpoint to the projector viewpoint, draw the virtual scene, and obtain the depth value under the current projector viewpoint; 5-2) Resetting the model-view matrix and the projection matrix to change the viewpoint back to the virtual viewpoint, redraw the virtual scene, and obtain the real depth value corresponding to each point in the scene; 5-3) Draw the virtual scene sequentially under each projector viewpoint, obtain the projected texture coordinates of each point in the scene through automatic texture generation, and perform the above steps 5-1) and 5-2) to obtain the real a comparison of a depth value with said depth value; 5-4) If the two are equal, use the video texture of the projector; if they are not equal, use the texture of the scene model itself, and iterate by setting the texture combiner function until all projectors in the scene are traversed, and each image in the scene is obtained. The final texture value of points; 6) Texture fusion is used for overlapping coverage areas of different projectors in the virtual projector model.