WO2019136950A1 - 虚拟图像处理方法、图像处理系统及存储介质 - Google Patents

虚拟图像处理方法、图像处理系统及存储介质 Download PDF

Info

Publication number
WO2019136950A1
WO2019136950A1 PCT/CN2018/095890 CN2018095890W WO2019136950A1 WO 2019136950 A1 WO2019136950 A1 WO 2019136950A1 CN 2018095890 W CN2018095890 W CN 2018095890W WO 2019136950 A1 WO2019136950 A1 WO 2019136950A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
parameter
camera device
scene
image processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/095890
Other languages
English (en)
French (fr)
Inventor
严振航
吴振志
谢明璞
刘宇
李选中
孙兴红
吴涵渠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Aoto Electronics Co Ltd
Original Assignee
Shenzhen Aoto Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Aoto Electronics Co Ltd filed Critical Shenzhen Aoto Electronics Co Ltd
Priority to ES18899867T priority Critical patent/ES3035820T3/es
Priority to EP18899867.8A priority patent/EP3745700B1/en
Priority to JP2020511180A priority patent/JP2020534724A/ja
Priority to US16/646,017 priority patent/US11354816B2/en
Publication of WO2019136950A1 publication Critical patent/WO2019136950A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/812—Monomedia components thereof involving advertisement data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H04N5/2723—Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement

Definitions

  • the present invention relates to the field of image processing technologies, and in particular, to a virtual image processing method, an image processing system, and a storage medium.
  • the real scene captured by the camera is combined with the virtual three-dimensional scene rendered by the graphic workstation to obtain the image that the final viewer sees.
  • This method can be applied in a virtual studio to seamlessly integrate the host's live activities with the virtual background generated by the graphics workstation. It can also be applied during the broadcast of sports games such as football and basketball, and can be replaced by virtual image processing. Off the advertisements around the stadium, the audience sees and the TV viewers can see two different advertisements, and the same sports competition can also advertise different advertisements for different broadcast countries to improve the advertiser's revenue. .
  • the current virtual image processing method requires that the camera pan/tilt is fixed on the tripod and cannot be moved at random.
  • the graphics workstation can calculate the synthesized virtual image relative to the original image by acquiring the pan, tilt, focus, and focus parameters of the camera.
  • the perspective relationship thus achieves a virtual image.
  • the fixed position of the camera limits the flexibility of video shooting. It can only be shot from a fixed position. For example, if the player runs fast or the high-speed motion of the car is difficult to be completely tracked, and once the camera position is After the move, the system will be recalibrated, which is time consuming and laborious. Therefore, the prior art has drawbacks and needs improvement.
  • a first aspect of the present invention provides a virtual image processing method, which is used to implement virtualization of a captured picture, and the method includes the following steps:
  • a gyroscope is disposed in the imaging device, and a moving parameter of the real-time position of the imaging device relative to the reference position in three directions of X, Y, and Z is calculated in real time by the gyroscope.
  • the virtualizing the position and posture of the virtual camera according to the movement parameter includes:
  • the resting parameter includes at least one of a panning of the camera, a pitch of the camera, a focus of the lens, and a zoom of the lens.
  • the calculating the position and posture of the virtual camera device according to the preset algorithm by the reference position, the movement parameter and the rest parameter includes:
  • the attitude of the virtual camera device is calculated based on the static parameter calculation.
  • the calculating the position and posture of the virtual camera device according to the preset algorithm by the reference position, the movement parameter and the rest parameter includes:
  • the attitude of the virtual camera device is calculated based on the static parameter calculation.
  • the calculating the perspective parameters of the scene to be replaced relative to the virtual camera includes:
  • the perspective parameter comprises at least one of position, size, and distortion.
  • the position and posture of the virtual camera device are virtualized by the acquired motion parameters, thereby calculating a perspective parameter of the scene to be replaced relative to the virtual camera device, and encoding the virtual scene according to the perspective parameter, and realizing the coded virtual scene and shooting Seamless integration of the picture. It can be applied to the virtual display in the movement of the camera.
  • a second aspect of the present invention provides an image processing system, including:
  • a position acquisition component configured to acquire a real-time position and a reference position of the camera device
  • a calculation component for calculating a movement parameter of the real-time position of the camera relative to the reference position
  • a processor configured to virtualize a position and a posture of the virtual camera device according to the movement parameter, and calculate a perspective parameter of the scene to be replaced relative to the virtual camera device;
  • a replacement component is used to replace the scene to be replaced with a virtual scene.
  • a gyroscope is disposed in the imaging device, and the computing component is a gyro chip.
  • the method further includes:
  • a parameter acquisition component is configured to acquire a resting parameter of the camera.
  • a third aspect of the invention provides a machine readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the virtual image processing method of any of the above.
  • the position and posture of the virtual camera device are virtualized by the acquired motion parameters, thereby calculating a perspective parameter of the scene to be replaced relative to the virtual camera device, and encoding the virtual scene and the captured image by encoding the virtual scene according to the perspective parameter.
  • Seamless integration It can be applied to the virtual display in the movement of the camera.
  • FIG. 1 is a flowchart of a virtual image processing method according to an embodiment of the present invention.
  • FIG. 2 is a partial flow exploded view of a virtual image processing method according to an embodiment of the present invention
  • FIG. 3 is a partial flow exploded view of a virtual image processing method according to still another embodiment of the present invention.
  • Figure 4 is a coding process of the display screen of the LED display screen
  • FIG. 5 is a flowchart of a virtual image processing method according to still another embodiment of the present invention.
  • FIG. 6 is a structural block diagram of an image processing system according to an embodiment of the present invention.
  • FIG. 1 is a flowchart of a virtual image processing method according to some embodiments of the present invention, and the method for processing the virtual image is applicable to live broadcast, broadcast, and virtual studio of a television. It should be noted that the virtual image processing method of the embodiment of the present invention is not limited to the steps and the sequence in the flowchart shown in FIG. 1, and the steps in the flowchart may add, remove, or change the order according to different requirements.
  • the virtual image processing method takes a captured image of the imaging device as a processing target, and the imaging device may be statically placed at a specific position during the shooting process, or may be dynamically moved to capture a virtual image.
  • the image processing method can virtually replace the display screen on the specific carrier in the shooting area according to the specific shooting picture, and replace the display picture with a virtual scene, so that different virtual scenes can be seen in different regions, for example, on the television broadcast.
  • different advertisements are broadcasted to different regions, and the broadcast revenue is increased.
  • the particular carrier is typically a billboard or advertising machine (such as an LED display), and the display may be a video being played on the screen.
  • the virtual image processing method includes the following steps:
  • Step S102 Acquire a reference position and a real-time position of the imaging device.
  • the shape and size of the specific carrier of the display screen in the shooting screen of the imaging device are constantly changing. Therefore, in order to realize the virtual replacement of the display image in the shooting screen during the movement of the imaging device, the imaging is performed.
  • the device requires a predetermined reference position, and the shape, size, and distortion of the carrier are evaluated according to the reference position during the shooting of the camera.
  • Step S104 Calculate the movement parameter of the real-time position relative to the reference position.
  • a computing component is disposed within the camera device, and the computing component calculates a motion parameter of the camera device based on the real-time position of the camera device relative to the reference position to provide a reference for the virtual camera virtual.
  • a gyroscope is disposed in the imaging device, and the movement parameters of the real-time position of the imaging device relative to the reference position in the X, Y, and Z directions are calculated in real time by the gyroscope.
  • Step S106 virtualize the position and posture of the virtual camera device according to the movement parameter, and calculate a perspective parameter of the virtual camera device that is to be replaced by the scene to be replaced.
  • the scene to be replaced refers to a display screen displayed on a specific carrier captured by the camera.
  • the position and posture of the virtual imaging device is a shooting angle of the camera calculated from a specific image of one frame.
  • the photographing screen of the image capturing apparatus is transmitted in units of frames, and the image forming angle of the image frame currently being processed is the initial position of the virtual image capturing apparatus before the arrival of the next frame image, based on the initial position of the virtual image capturing apparatus and the movement of the image capturing apparatus.
  • the parameter can virtualize the position of the virtual camera device in the next frame image, thereby virtualizing the imaging angle of view of the next frame image, so that the perspective parameter of the scene to be replaced relative to the virtual camera device can be calculated according to the change of the imaging angle of view.
  • the virtual position and posture of the virtual camera device may have a certain delay, which may be based on the position and posture of the virtual camera device of the next frame or frames according to the current frame, and the delay is Can be adjusted. It can be understood by those skilled in the art that, on the basis that the processing efficiency of the processor satisfies the condition, the position and posture of the virtual camera device of the current frame can be directly calculated, and the perspective parameter of the virtual camera device that is relatively virtual to be replaced is calculated. And use it as a standard for adjusting virtual scenes.
  • the virtualizing the position and posture of the virtual camera according to the movement parameter includes:
  • Step S1061 Acquire a positioning parameter of the imaging device
  • Step S1063 Calculate the position and posture of the virtual camera device according to a preset algorithm from the reference position, the movement parameter, and the rest parameter.
  • the rest parameters of the camera include at least one of panning of the camera, tilt of the camera, focusing of the lens, and zooming of the lens.
  • the imaging device performs moving shooting, it is usually necessary to add a camera pan/tilt for the image stabilization, so that the camera pan and tilt can be acquired by the camera pan/tilt.
  • the focus and zoom of the lens will directly determine the size and sharpness of the captured image, which is usually controlled by the control chip of the camera. Therefore, the focus and zoom of the lens can be read directly from the control chip of the camera.
  • the posture change of the camera device at a specific position can be determined by the static parameter, and the moving position of the camera device can be determined by the reference position and the movement parameter, so that the position and posture of the virtual camera device can be virtualized according to the position of the virtual camera device of the current frame. .
  • the calculating the position and posture of the virtual camera device according to the preset algorithm by the reference position, the movement parameter, and the rest parameter includes:
  • the attitude of the virtual camera device is calculated based on the static parameter calculation.
  • the virtual change of the position and posture of the virtual camera in the front and rear frames is small and substantially the same, so that the position and posture of the virtual camera of the previous frame image can be replaced by the latter one.
  • the position and posture of the virtual camera device of the frame or multi-frame image, as the position and posture of the virtual camera device, the error can usually be controlled within the allowable range without being visually captured by the observer.
  • the calculating the position and posture of the virtual camera device according to the preset algorithm by the reference position, the movement parameter, and the rest parameter includes:
  • the attitude of the virtual camera device is calculated based on the static parameter calculation.
  • the movement of the imaging device in the three directions of X, Y, and Z can respectively calculate the position of the imaging device after the time T in the three directions of X, Y, and Z.
  • the position, and then according to the position of the camera device after the time T and the position of the current camera device, can predict the moving tendency of the camera device, and the predicted result can be expressed as a trend vector for characterizing the moving direction of the camera device.
  • the time T is less than or equal to the shutter open time of the image pickup device.
  • the calculating the perspective parameters of the virtual camera device that is to be replaced by the scene to be replaced includes:
  • Step S1065 Identify the position of the scene to be replaced in the shooting screen
  • Step S1067 Calculate the perspective parameter of the scene to be replaced relative to the virtual camera device according to the position and posture of the virtual camera device.
  • the identification of the scene to be replaced can be implemented by encoding the display screen of the carrier.
  • the encoding of the display screen of the carrier may be: inserting a virtual frame between normal display frames, so that the normal display is performed.
  • the frame and the virtual frame are periodically displayed on the carrier (LED display screen), and by controlling the time slot ratio of the display frame and the virtual frame, the image displayed by the LED display screen in the observer's eyes is only the display frame, and the virtual frame is only the display frame. Because the occupied time slot is too short to be captured by the human eye.
  • the shutter opening time of the camera device is synchronized with the display time of the virtual frame, that is, the shutter of the camera device is only turned on when the LED display screen displays the virtual frame, and thus the display screen of the LED display screen captured by the camera device (ie, the scene to be replaced) is only Is a virtual frame.
  • the virtual frame may be a red, green, or blue solid color image.
  • FIG. 4 exemplarily shows the encoding process of the display screen of the carrier (taking the LED display screen as an example).
  • the processor is provided with a processor, and the display frame 100 is displayed on the LED display after being processed by the processor.
  • a frame of virtual frames 200 is inserted between the two display frames 100, and the virtual frames 200 may be externally transmitted to the processor or automatically generated by the processor.
  • the processor adjusts the time slot ratio of the display frame 100 to the virtual frame 200
  • the data stream is sent to the LED for display, so that the display frame 100 and the virtual frame 200 are alternately displayed on the LED display screen.
  • the display frame 100 is observed by the naked eye, and the virtual frame 200 is captured by the image capturing device to form a scene to be replaced, and thus the recognition of the scene to be replaced is only required to identify the solid color image area.
  • the imaging angle of the virtual camera device can be virtualized.
  • the scene to be replaced can be calculated according to the change of the imaging angle of view.
  • a perspective parameter of the camera device including at least one of position, size, and distortion.
  • Step S108 Encoding the virtual scene according to the perspective parameter.
  • the virtual scene In order to make the virtual scene coincide with the position of the scene to be replaced of the next frame image, the virtual scene needs to be encoded according to the perspective parameter to conform to the perspective relationship between the scene to be replaced and the virtual camera device, thereby achieving seamless integration with the captured image.
  • Step S110 Replace the scene to be replaced with the encoded virtual scene.
  • the scene to be replaced is imaged and replaced with a virtual scene, that is, the seamless integration of the virtual scene and the captured image is completed.
  • the fused image is transmitted to a display device for display.
  • a virtual scene is displayed on the display on the display screen on the display device, and a display video composed of the display frame 100 is displayed on the carrier of the shooting scene, that is, The virtualization of the shooting picture is realized.
  • the virtual image processing method further includes:
  • Step S112 Updating the real-time position of the imaging device to the reference position of the imaging device.
  • the accuracy of the calculation can be improved by updating the reference position of the camera.
  • the reference position of the camera device may also be constant, maintaining a fixed position and using the position as a zero point of the coordinate system, and the movement of the camera device may be regarded as movement within the coordinate system.
  • the position and posture of the virtual camera device are virtualized by the acquired motion parameters, thereby calculating a perspective parameter of the scene to be replaced relative to the virtual camera device, and encoding the virtual scene according to the perspective parameter, and realizing the coded virtual scene and shooting Seamless integration of the picture. It can be applied to the virtual display in the movement of the camera.
  • the present invention further provides an image processing system 10 .
  • the image processing system 10 includes an imaging device 110 , a location acquisition component 120 , a computing component 130 , a processor 140 , and an encoding component 150 . And replacing component 160.
  • the image processing system 10 may include some or all of the components or devices shown in FIG. 5, and the functions of the components or devices will be as follows. Specifically, it should be noted that the same noun related terms and their specific explanations in the embodiments of the above virtual image processing method can also be applied to the following functions for each component or device, in order to avoid duplication, This is not repeated here.
  • the imaging device 110 is configured to capture a picture.
  • the location acquiring component 120 is configured to acquire a real-time location and a reference location of the camera 110.
  • the calculating component 130 is configured to calculate a moving parameter of the real-time position of the camera relative to the reference position.
  • a gyroscope can be disposed in the camera device 110.
  • the chip of the gyroscope can be used as the computing component 130 to calculate the moving parameter of the real-time position of the camera relative to the reference position in real time through the chip of the gyroscope.
  • the processor 140 is configured to virtualize the position and posture of the virtual camera device according to the movement parameter, and calculate a perspective parameter of the scene to be replaced relative to the virtual camera device;
  • the scene to be replaced refers to a display screen displayed on the specific carrier captured by the camera device 110.
  • a parameter acquisition component 170 is also included, which is used to acquire a rest parameter of the camera 110.
  • the rest parameters of the camera device 110 include at least one of panning of the camera, tilting of the camera, focusing of the lens, and zooming of the lens.
  • the parameter acquisition component 170 can be disposed within the camera device 110.
  • the processor 140 is further configured to:
  • the position where the imaging device 110 is actually located is calculated based on the reference position and the movement parameter of the imaging device 110 as the position of the virtual imaging device;
  • the attitude of the virtual camera device is calculated based on the static parameter calculation.
  • the processor 140 is further configured to:
  • the perspective parameter of the scene to be replaced relative to the virtual camera device is calculated according to the position and posture of the virtual camera device.
  • the encoding component 150 is configured to encode a virtual scene
  • the replacing component 160 is configured to replace the scene to be replaced with a virtual scene.
  • a location update component 180 is also included, which is used to update the reference location of the camera device 110.
  • An embodiment of the present invention further provides a machine readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the virtual image processing method of any of the above embodiments.
  • the components/modules/units of the image system/computer device integration if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium.
  • the present invention implements all or part of the processes in the above-described embodiments, and may also be completed by a computer program to instruct related hardware.
  • the computer program may be stored in a computer readable storage medium.
  • the steps of the various method embodiments described above may be implemented when the computer program is executed by the processor.
  • the computer program comprises computer program code, which may be in the form of source code, object code form, executable file or some intermediate form.
  • the computer readable storage medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read only memory (ROM, Read-Only) Memory), random access memory (RAM, Random) Access Memory), electrical carrier signals, telecommunications signals, and software distribution media.
  • ROM Read Only memory
  • RAM Random Access Memory
  • electrical carrier signals telecommunications signals
  • telecommunications signals and software distribution media. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, computer readable media Does not include electrical carrier signals and telecommunication signals.
  • each functional module/component in various embodiments of the present invention may be integrated in the same processing module/component, or each module/component may exist physically separately, or two or more modules/components may be integrated in the same In the module/component.
  • the above integrated modules/components can be implemented in the form of hardware or in the form of hardware plus software function modules/components.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Vascular Medicine (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Studio Devices (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Analysis (AREA)

Abstract

本发明涉及一种虚拟图像处理方法、图像处理系统及存储介质，所述方法用于实现拍摄画面的虚拟，所述方法包括以下步骤：获取摄像装置的基准位置及实时位置；计算实时位置相对基准位置的移动参数；根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数；根据透视参数编码虚拟场景；用编码后的虚拟场景替换待替换场景。本发明提供的虚拟图像处理方法，通过获取的移动参数虚拟出虚拟摄像装置的位置及姿态，从而计算出待替换场景相对虚拟摄像装置的透视参数，在根据透视参数编码虚拟场景，可以实现编码虚拟场景与拍摄画面的无缝融合，可以适用于摄像装置移动中的虚拟显示。

Description

虚拟图像处理方法、图像处理系统及存储介质 技术领域
本发明涉及图像处理技术领域，特别是涉及一种虚拟图像处理方法、图像处理系统及存储介质。
 背景技术
像机拍摄的真实场景与图型工作站渲染出的虚拟三维场景合成，得到最终观众看到的画面。这一方法可应用在虚拟演播室中，将主持人在现场的活动与图形工作站产生的虚拟背景无缝融合，也可应用在足球、篮球等体育比赛的转播过程中，可以通过虚拟图像处理替换掉球场周边的广告，现场观众看到的和电视机观众看到的可以是两个不同的广告，同时同一个体育比赛针对不同的转播国家也可以虚拟出不同的广告，以提高广告商的收益。
目前的虚拟图像处理方法要求摄像机云台固定在三脚架上不能随意移动，此时图形工作站通过获取摄像机的平摇、俯仰、聚焦、对焦这几个参数就可以计算出所合成的虚拟画面相对于原始画面的透视关系，从而实现虚拟图像。但是摄像机的固定机位就限制了视频拍摄的灵活性，只能从某一固定位置进行拍摄，例如球员快速跑动或者赛车高速运动的拍摄就很难被完整的跟踪拍摄，另外一旦摄像机位置被移动后就要对系统进行重新的标定工作，费时费力。因此，现有技术存在缺陷，需要改进。
 技术问题
基于此，有必要针对摄像机云台不能随意移动的问题，提供一种虚拟图像处理方法、图像处理系统及存储介质。
 技术解决方案
本发明第一方面提供一种虚拟图像处理方法，所述方法用于实现拍摄画面的虚拟，所述方法包括以下步骤：
获取摄像装置的基准位置及实时位置；
计算实时位置相对基准位置的移动参数；
根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数；
根据透视参数编码虚拟场景；
用编码后的虚拟场景替换待替换场景。
在其中一个实施例中，所述摄像装置内设置有陀螺仪，并通过陀螺仪实时计算摄像装置的实时位置相对基准位置在X、Y、Z三个方向的移动参数。
在其中一个实施例中，所述根据移动参数虚拟出虚拟摄像装置的位置及姿态包括：
获取摄像装置的静置参数；
由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态；
其中，所述静置参数包括摄像机的平摇、摄像机的俯仰、镜头的聚焦、镜头的变焦中的至少一种。
在其中一个实施例中，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
基于摄像装置的基准位置、移动参数综合计算得到摄像装置实际所处的位置，作为虚拟摄像装置的位置；
基于静置参数计算得到虚拟摄像装置的姿态。
在其中一个实施例中，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
基于摄像装置的基准位置、移动参数综合计算后得到摄像装置拍摄时实际所处的位置；
基于移动参数计算预测摄像装置的移动趋势；
将摄像装置实际所处的位置与移动趋势计算所述虚拟摄像装置的位置；
基于静置参数计算得到虚拟摄像装置的姿态。
在其中一个实施例中，所述计算出待替换场景相对虚拟摄像装置的透视参数包括：
识别待替换场景在拍摄画面中的位置；
依据虚拟摄像装置的位置及姿态计算待替换场景的透视参数；
其中，所述透视参数包括位置、大小、畸变中的至少一种。
上述虚拟图像处理方法，通过获取的移动参数虚拟出虚拟摄像装置的位置及姿态，从而计算出待替换场景相对虚拟摄像装置的透视参数，在根据透视参数编码虚拟场景，可以实现编码虚拟场景与拍摄画面的无缝融合。可以适用于摄像装置移动中的虚拟显示。
本发明第二方面提供一种图像处理系统，包括：
摄像装置，用于拍摄画面；
位置获取部件，用于获取摄像装置的实时位置和基准位置；
计算部件，用于计算摄像机的实时位置相对基准位置的移动参数；
处理器，用于根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟摄像装置的透视参数；
编码部件，用于编码虚拟场景；及
替换部件，用于将待替换场景替换为虚拟场景。
在其中一个实施例中，所述摄像装置内设置有陀螺仪，所述计算部件为陀螺仪芯片。
在其中一个实施例中，还包括：
参数获取部件，用于获取摄像装置的静置参数。
本发明第三方面提供一种机器可读存储介质，其上存储有计算机程序，其中所述计算机程序在由处理器执行时实现上述任一项所述的虚拟图像处理方法。
 有益效果
上述图像处理系统，通过获取的移动参数虚拟出虚拟摄像装置的位置及姿态，从而计算出待替换场景相对虚拟摄像装置的透视参数，在根据透视参数编码虚拟场景，可以实现编码虚拟场景与拍摄画面的无缝融合。可以适用于摄像装置移动中的虚拟显示。
 附图说明
图1为本发明一实施例的虚拟图像处理方法的流程图；
图2为本发明一实施例的虚拟图像处理方法的部分流程分解图；
图3为本发明又一实施例的虚拟图像处理方法的部分流程分解图；
图4为LED显示屏的显示画面的编码过程；
图5为本发明又一实施例的虚拟图像处理方法的流程图；
图6为本发明一实施例的图像处理系统的结构框图。
 本发明的实施方式
为了能够更清楚地理解本发明的上述目的、特征和优点，下面结合附图和具体实 施方式对本发明进行详细描述。需要说明的是，在不冲突的情况下，本申请的实施方式及实 施方式中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本发明，所描述的实施方式 仅仅是本发明一部分实施方式，而不是全部的实施方式。基于本发明中的实施方式，本领域 普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施方式，都属于本发明保 护的范围。
除非另有定义，本文所使用的所有的技术和科学术语与属于本发明的技术领域的 技术人员通常理解的含义相同。本文中在本发明的说明书中所使用的术语只是为了描述具 体的实施方式的目的，不是旨在于限制本发明。
图1是本发明一些实施例的虚拟图像处理方法的流程图，所述虚拟图像的处理方法可应用于电视的现场直播、转播，以及虚拟演播室。应该说明的是，本发明实施方式的虚拟图像处理方法并不限于图1所示的流程图中的步骤及顺序，根据不同的需求，流程图中的步骤可以增加、移除或者改变顺序。
本发明实施方式的虚拟图像处理方法以摄像装置的拍摄画面为处理对象，所述摄像装置在拍摄过程中，可以是静态的置于某个特定的位置，也可以是进行动态的移动拍摄，虚拟图像处理方法可以根据具体的拍摄画面对处于拍摄区域内的特定载体上显示画面进行虚拟替换，以一虚拟场景替换显示画面，从而可以使得不同地区看见不同的虚拟场景，示例性的，在电视转播过程中，通过广告的替换，实现向不同地区转播不同的广告，提升转播收入。所述特定载体通常是广告牌或广告机（如LED显示屏），所述显示画面可以是屏幕上正在播放的视频。
如图1所示，所述虚拟图像处理方法包括以下步骤：
步骤S102：获取摄像装置的基准位置及实时位置。
摄像装置在移动拍摄过程中，显示画面的特定载体在摄像装置的拍摄画面中的形状、大小处于不断变化中，因而，为了实现对摄像装置在移动过程中拍摄画面中显示画面的虚拟替换，摄像装置需要预先确定的一基准位置，并在摄像装置的拍摄过程中根据该基准位置对载体的形状、大小、畸变进行评估。
步骤S104：计算实时位置相对基准位置的移动参数。
在一些实施例中，摄像装置内设置有计算部件，计算部件根据摄像装置的实时位置相对基准位置计算出摄像装置的移动参数，以为虚拟摄像机的虚拟提供参考。在具体的实施例中，摄像装置内设置有陀螺仪，并通过陀螺仪实时计算摄像装置的实时位置相对基准位置在X、Y、Z三个方向的移动参数。
步骤S106：根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数。
所述待替换场景是指摄像装置所捕获的显示在特定载体上的显示画面。所述虚拟摄像装置的位置及姿态为根据一帧特定图像计算出的摄像机的拍摄视角。摄像装置的拍摄画面以帧为单位进行传送，在下一帧图像到达之前，以当前正在处理的图像帧的成像视角为虚拟摄像装置的初始位置，则根据虚拟摄像装置的初始位置以及摄像装置的移动参数，可以虚拟出下一帧图像中虚拟摄像装置的位置，进而虚拟出下一帧图像的成像视角，从而可以根据成像视角的变化计算待替换场景相对虚拟摄像装置的透视参数。根据不同处理器的处理效率，虚拟摄像装置的位置及姿态的虚拟可以具有一定的延迟，可以是根据当前帧虚拟出后面一帧或多帧的虚拟摄像装置的位置及姿态，并且这种延迟是可以调节的。本领域技术人员可以理解的是，在处理器的处理效率满足条件的基础上，可以直接计算当前帧的虚拟摄像装置的位置及姿态，计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数，并将其作为调整虚拟场景的标准。
请参阅图2，在一些实施例中，所述根据移动参数虚拟出虚拟摄像装置的位置及姿态包括：
步骤S1061：获取摄像装置的静置参数；
步骤S1063：由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态。
摄像装置的静置参数包括摄像机的平摇、摄像机的俯仰、镜头的聚焦、镜头的变焦中的至少一种。摄像装置在进行移动拍摄时，为了画面的防抖，通常需要加装摄像云台，因而摄像装置的平遥和俯仰可以通过摄像云台进行获取。而镜头的聚焦和变焦将直接决定拍摄画面的大小和清晰度，通常由摄像装置的控制芯片进行控制，因此，镜头的聚焦和变焦可以直接从摄像装置的控制芯片读取。
通过静置参数可以确定摄像装置在特定位置的姿态变化，通过基准位置和移动参数可以确定摄像装置的移动方位，从而可以根据当前帧的虚拟摄像装置的位置，虚拟出虚拟摄像装置的位置及姿态。
在一实施例中，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
基于摄像装置的基准位置、移动参数综合计算得到摄像装置实际所处的位置，作为虚拟摄像装置的位置；
基于静置参数计算得到虚拟摄像装置的姿态。
由于相机的快门通常时间极短，因而，在前后帧的虚拟摄像装置的位置和姿态的虚拟变化较小，基本相同，因而可以采用前一帧图像的虚拟摄像装置的位置和姿态替代为后一帧或多帧图像的虚拟摄像装置的位置和姿态，作为虚拟摄像装置的位置和姿态，误差通常可以控制在允许的范围内，不会被观察者的视觉捕捉到。
在另一些实施例中，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
基于摄像装置的基准位置、移动参数综合计算后得到摄像装置拍摄时实际所处的位置；
基于移动参数计算预测摄像装置的移动趋势；
将摄像装置实际所处的位置与移动趋势计算所述虚拟摄像装置的位置；
基于静置参数计算得到虚拟摄像装置的姿态。
当摄像装置内设置有陀螺仪时，依据摄像装置在X、Y、Z三个方向上的运动可以分别计算出摄像装置在X、Y、Z三个方向上在时间T后摄像装置所处的位置，进而依据时间T后摄像装置所处的位置与当前摄像装置的位置可以对摄像装置的移动趋势进行预测，预测的结果可以表示为一趋势向量，用以表征摄像装置的移动方向。一般地，时间T小于等于摄像装置的快门打开时间。
请参阅图3，在一些实施例中，所述计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数包括：
步骤S1065：识别待替换场景在拍摄画面中的位置；
步骤S1067：依据虚拟摄像装置的位置及姿态计算待替换场景相对虚拟摄像装置的透视参数。
待替换场景的识别可以通过对载体的显示画面进行编码实现，在一具体的实施例中，所述载体的显示画面的编码可以是：在正常的显示帧之间插入虚拟帧，使得正常的显示帧与虚拟帧周期性的显示在载体（LED显示屏）上，通过控制显示帧与虚拟帧的时隙比，使得LED显示屏在观察者眼中所呈现的画面仅为显示帧，而虚拟帧则由于所占时隙过短无法被人眼捕捉。摄像装置的快门开启时间与虚拟帧的显示时间同步，即摄像装置的快门仅在LED显示屏显示虚拟帧时开启，因而，摄像装置捕捉到的LED显示屏的显示画面（即待替换场景）仅为虚拟帧。所述虚拟帧可以是红、绿、蓝纯色图像。因而，只要识别出虚拟帧，即可识别出待替换场景。
请继续参阅图4，图4示例性的示出了载体（以LED显示屏为例）的显示画面的编码过程，载体内设置有处理器，显示帧100经过处理器的处理后显示在LED显示屏上，多个连续的显示帧100经过处理器后，两个显示帧100之间被插入一帧虚拟帧200，虚拟帧200可以是由外接传送至处理器，也可以由处理器自动生成，处理器调节显示帧100与虚拟帧200的时隙比后，将数据流送至LED进行显示，使得显示帧100与虚拟帧200交替的显示在LED显示屏上。最终，显示帧100被肉眼所观察到，虚拟帧200被摄像装置捕获形成待替换场景，因而待替换场景的识别只要识别出纯色图像区域即可。
虚拟出虚拟摄像装置的位置及姿态后，即可虚拟出虚拟摄像装置的成像视角，根据虚拟摄像装置的当前成像视角及虚拟出的成像视角，即可根据成像视角的变化计算待替换场景相对虚拟摄像装置的透视参数，所述透视参数包括位置、大小、畸变中的至少一种。
步骤S108：根据透视参数编码虚拟场景。
为了使得虚拟场景与下一帧图像的待替换场景位置重合，需要根据透视参数对虚拟场景进行编码，使其符合待替换场景与虚拟摄像装置的透视关系，进而实现与拍摄画面的无缝融合。
步骤S110：用编码后的虚拟场景替换待替换场景。
虚拟场景编码后，将待替换场景区域进行抠像，并替换为虚拟场景，即完成虚拟场景与拍摄画面的无缝融合。将融合后的画面传送至一显示装置上进行显示，此时，显示装置上的显示画面中载体上显示的为虚拟场景，而拍摄现场的载体上显示的为显示帧100组成的显示视频，即实现了拍摄画面的虚拟。
请参阅图5，在一些实施例中，所述虚拟图像处理方法还包括：
步骤S112：将摄像装置的实时位置更新为摄像装置的基准位置。
通过更新摄像装置的基准位置，可以提升计算的精确度。
当然，在一些实施例中，摄像装置的基准位置也可以是不变的，维持一个固定的位置，并将该位置作为坐标系零点，摄像装置的移动可以看成在坐标系内的移动。
上述虚拟图像处理方法，通过获取的移动参数虚拟出虚拟摄像装置的位置及姿态，从而计算出待替换场景相对虚拟摄像装置的透视参数，在根据透视参数编码虚拟场景，可以实现编码虚拟场景与拍摄画面的无缝融合。可以适用于摄像装置移动中的虚拟显示。
请从参阅图6，本发明还提供一种图像处理系统10，在一些实施例中，所述图像处理系统10包括摄像装置110、位置获取部件120、计算部件130、处理器140、编码部件150及替换部件160。可以理解的是，对应于上述虚拟图像处理方法中的各实施例，所述图像处理系统10可以包括图5中所示的各部件或装置的部分或全部，各部件或装置的功能将在以下具体介绍，需要说明的是，以上的虚拟图像处理方法的各实施例中相同的名词相关的名词及其具体的解释说明也可适用于以下对各部件或装置的功能介绍，为避免重复，在此不再重复赘述。
所述摄像装置110，用于拍摄画面。
所述位置获取部件120，用于获取摄像装置110的实时位置和基准位置。
所述计算部件130，用于计算摄像机的实时位置相对基准位置的移动参数。
在一些实施例中，可以在摄像装置110内设置陀螺仪，陀螺仪的芯片可以作为计算部件130，通过陀螺仪的芯片实时计算摄像机的实时位置相对基准位置的移动参数。
所述处理器140，用于根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟摄像装置的透视参数；
所述待替换场景是指摄像装置110所捕获的显示在特定载体上的显示画面。
在一些实施例中，还包括参数获取部件170，所述参数获取部件170用于获取摄像装置110的静置参数。
摄像装置110的静置参数包括摄像机的平摇、摄像机的俯仰、镜头的聚焦、镜头的变焦中的至少一种。所述参数获取部件170可以设置于所述摄像装置110内。
在一实施例中，所述处理器140还用于：
基于摄像装置110的基准位置、移动参数综合计算得到摄像装置110实际所处的位置，作为虚拟摄像装置的位置；及
基于静置参数计算得到虚拟摄像装置的姿态。
在一些实施例中，所述处理器140还用于：
识别待替换场景在拍摄画面中的位置；以及
依据虚拟摄像装置的位置及姿态计算待替换场景相对虚拟摄像装置的透视参数。
所述编码部件150，用于编码虚拟场景；
所述替换部件160，用于将待替换场景替换为虚拟场景。
在一些实施例中，还包括位置更新部件180，所述位置更新部件180用于更新摄像装置110的基准位置。
本发明一实施例还提供一种机器可读存储介质，其上存储有计算机程序，其中所述计算机程序在由处理器执行时实现上述任一实施例所述的虚拟图像处理方法。
所述图像系统/计算机装置集成的部件/模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实现上述实施方式方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，所述计算机程序在被处理器执行时，可实现上述各个方法实施方式的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读存储介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器 (ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。
在本发明所提供的几个具体实施方式中，应该理解到，所揭露的系统和方法，可以通过其它的方式实现。例如，以上所描述的系统实施方式仅仅是示意性的，例如，所述部件的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
另外，在本发明各个实施例中的各功能模块/部件可以集成在相同处理模块/部件中，也可以是各个模块/部件单独物理存在，也可以两个或两个以上模块/部件集成在相同模块/部件中。上述集成的模块/部件既可以采用硬件的形式实现，也可以采用硬件加软件功能模块/部件的形式实现。
对于本领域技术人员而言，显然本发明实施例不限于上述示范性实施例的细节， 而且在不背离本发明实施例的精神或基本特征的情况下，能够以其他的具体形式实现本发明实施例。因此，无论从哪一点来看，均应将实施例看作是示范性的，而且是非限制性的，本发明实施例的范围由所附权利要求而不是上述说明限定，因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本发明实施例内。不应将权利要求中的任何附图标记视为限制所涉及的权利要求。此外，显然“包括”一词不排除其他单元或步骤，单数不排除复数。系统、装置或终端权利要求中陈述的多个单元、模块或装置也可以由同一个单元、模块或装置通过软件或者硬件来实现。第一，第二等词语用来表示名称，而并不表示任何特定的顺序。
以上所述实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本发明的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。

Claims (10)

  1. 一种虚拟图像处理方法，所述方法用于实现拍摄画面的虚拟，其特征在于，所述方法包括以下步骤：
    获取摄像装置的基准位置及实时位置；
    计算实时位置相对基准位置的移动参数；
    根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟出的虚拟摄像装置的透视参数；
    根据透视参数编码虚拟场景；
    用编码后的虚拟场景替换待替换场景。
  2. 根据权利要求1所述的虚拟图像处理方法，其特征在于，所述摄像装置内设置有陀螺仪，并通过陀螺仪实时计算摄像装置的实时位置相对基准位置在X、Y、Z三个方向的移动参数。
  3. 根据权利要求1或2所述的虚拟图像处理方法，其特征在于，所述根据移动参数虚拟出虚拟摄像装置的位置及姿态包括：
    获取摄像装置的静置参数；
    由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态；
    其中，所述静置参数包括摄像机的平摇、摄像机的俯仰、镜头的聚焦、镜头的变焦中的至少一种。
  4. 根据权利要求3所述的虚拟图像处理方法，其特征在于，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
    基于摄像装置的基准位置、移动参数综合计算得到摄像装置实际所处的位置，作为虚拟摄像装置的位置；
    基于静置参数计算得到虚拟摄像装置的姿态。
  5. 根据权利要求3所述的虚拟图像处理方法，其特征在于，所述由基准位置、移动参数及静置参数根据预设算法计算出虚拟摄像装置的位置及姿态包括：
    基于摄像装置的基准位置、移动参数综合计算后得到摄像装置拍摄时实际所处的位置；
    基于移动参数计算预测摄像装置的移动趋势；
    将摄像装置实际所处的位置与移动趋势计算所述虚拟摄像装置的位置；
    基于静置参数计算得到虚拟摄像装置的姿态。
  6. 根据权利要求3-5任一项所述的虚拟图像处理方法，其特征在于，所述计算出待替换场景相对虚拟摄像装置的透视参数包括：
    识别待替换场景在拍摄画面中的位置；
    依据虚拟摄像装置的位置及姿态计算待替换场景的透视参数；
    其中，所述透视参数包括位置、大小、畸变中的至少一种。
  7. 一种图像处理系统，其特征在于，包括：
    摄像装置，用于拍摄画面；
    位置获取部件，用于获取摄像装置的实时位置和基准位置；
    计算部件，用于计算摄像机的实时位置相对基准位置的移动参数；
    处理器，用于根据移动参数虚拟出虚拟摄像装置的位置及姿态，并计算出待替换场景相对虚拟摄像装置的透视参数；
    编码部件，用于编码虚拟场景；及
    替换部件，用于将待替换场景替换为虚拟场景。
  8. 根据权利要求7所述的图像处理系统，其特征在于，所述摄像装置内设置有陀螺仪，所述计算部件为陀螺仪芯片。
  9. 根据权利要求7或8所述的图像处理系统，其特征在于，还包括：
    参数获取部件，用于获取摄像装置的静置参数。
  10. 一种机器可读存储介质，其上存储有计算机程序，其中所述计算机程序在由处理器执行时实现如权利要求1至6任一项所述的虚拟图像处理方法。
PCT/CN2018/095890 2018-01-15 2018-07-17 虚拟图像处理方法、图像处理系统及存储介质 Ceased WO2019136950A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
ES18899867T ES3035820T3 (en) 2018-01-15 2018-07-17 Virtual image processing method, image processing system, and storage medium
EP18899867.8A EP3745700B1 (en) 2018-01-15 2018-07-17 Virtual image processing method, image processing system, and storage medium
JP2020511180A JP2020534724A (ja) 2018-01-15 2018-07-17 仮想画像処理方法、画像処理システムおよび記憶媒体
US16/646,017 US11354816B2 (en) 2018-01-15 2018-07-17 Virtual image processing method, image processing system, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810033430.1A CN108289220B (zh) 2018-01-15 2018-01-15 虚拟图像处理方法、图像处理系统及存储介质
CN201810033430.1 2018-01-15

Publications (1)

Publication Number Publication Date
WO2019136950A1 true WO2019136950A1 (zh) 2019-07-18

Family

ID=62835330

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/095890 Ceased WO2019136950A1 (zh) 2018-01-15 2018-07-17 虚拟图像处理方法、图像处理系统及存储介质

Country Status (6)

Country Link
US (1) US11354816B2 (zh)
EP (1) EP3745700B1 (zh)
JP (1) JP2020534724A (zh)
CN (1) CN108289220B (zh)
ES (1) ES3035820T3 (zh)
WO (1) WO2019136950A1 (zh)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7170441B2 (ja) * 2018-07-12 2022-11-14 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
CN109045694B (zh) * 2018-08-17 2020-01-10 腾讯科技（深圳）有限公司 虚拟场景显示方法、装置、终端及存储介质
CN109040528B (zh) * 2018-09-30 2021-07-23 深圳市励得数码科技有限公司 演播室中摄像机的运动状态确定方法、装置及演播室系统
CN110377861B (zh) * 2019-07-23 2023-11-07 腾讯科技（深圳）有限公司 场景间元素互动方法、装置、存储介质和计算机设备
CN111737518B (zh) * 2020-06-16 2024-07-02 浙江大华技术股份有限公司 基于三维场景模型的图像展示方法、装置及电子设备
CN112312041B (zh) * 2020-10-22 2023-07-25 北京虚拟动点科技有限公司 基于拍摄的图像校正方法、装置、电子设备及存储介质
CN114286142B (zh) * 2021-01-18 2023-03-28 海信视像科技股份有限公司 一种虚拟现实设备及vr场景截屏方法
CN113766119B (zh) * 2021-05-11 2023-12-05 腾讯科技（深圳）有限公司 虚拟形象显示方法、装置、终端及存储介质
CN113674172B (zh) 2021-08-17 2023-11-28 上海交通大学 一种图像处理方法、系统、装置及存储介质
CN113923354B (zh) * 2021-09-30 2023-08-01 卡莱特云科技股份有限公司 基于多帧图像的视频处理方法、设备及虚拟背景拍摄系统
CN116546304B (zh) * 2022-05-13 2025-11-07 腾讯数码（深圳）有限公司 一种参数配置方法、装置、设备、存储介质及产品
CN115761114B (zh) * 2022-10-28 2024-04-30 如你所视(北京)科技有限公司 视频生成方法、装置及计算机可读存储介质
CN117119294B (zh) * 2023-08-24 2024-08-20 腾讯科技（深圳）有限公司 虚拟场景的拍摄方法、装置、设备、介质和程序
CN117097878B (zh) * 2023-10-16 2024-07-09 杭州穿石物联科技有限责任公司 一种基于超低延迟视频传输技术的云控交互系统
CN118172471B (zh) * 2024-03-15 2025-11-04 中国传媒大学 虚拟相机行为生成方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103226838A (zh) * 2013-04-10 2013-07-31 福州林景行信息技术有限公司 地理场景中移动监控目标的实时空间定位方法
US20140002580A1 (en) * 2012-06-29 2014-01-02 Monkeymedia, Inc. Portable proprioceptive peripatetic polylinear video player
CN103841299A (zh) * 2012-11-23 2014-06-04 西安天动数字科技有限公司 虚拟演播室系统
CN107158695A (zh) * 2017-06-16 2017-09-15 苏州蜗牛数字科技股份有限公司 一种现实与虚拟场景混合展现的方法及系统
CN107231531A (zh) * 2017-05-23 2017-10-03 青岛大学 一种网络虚拟现实技术与实拍结合影视制作系统

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5533348A (en) * 1994-08-26 1996-07-09 American Standard Inc. Average air temperature sensor
GB2305051B (en) * 1995-09-08 2000-06-21 Orad Hi Tec Systems Ltd Method and apparatus for automatic electronic replacement of billboards in a video image
GB9601101D0 (en) * 1995-09-08 1996-03-20 Orad Hi Tech Systems Limited Method and apparatus for automatic electronic replacement of billboards in a video image
US6327004B1 (en) * 1998-03-18 2001-12-04 Nippon Television Network Corp. Picture replacement system and method of the same
JP2004227332A (ja) 2003-01-23 2004-08-12 Hitachi Ltd 情報表示方法
GB2437575A (en) * 2006-04-27 2007-10-31 Andrew Peter Phelan Audience-specific image display
SG10201507715WA (en) * 2010-09-20 2015-10-29 Fraunhofer Ges Forschung Method for differentiating between background and foreground of scenery and also method for replacing a background in images of a scenery
GB201208088D0 (en) 2012-05-09 2012-06-20 Ncam Sollutions Ltd Ncam
CN105023294B (zh) * 2015-07-13 2018-01-19 中国传媒大学 结合传感器与Unity3D的定点移动增强现实方法
CN105472399B (zh) * 2015-11-20 2019-04-05 深圳市奥拓电子股份有限公司 一种转播视频与现场显示视频分离方法及装置
CN106296683A (zh) 2016-08-09 2017-01-04 深圳迪乐普数码科技有限公司 一种虚拟屏幕墙的生成方法及终端
CN106412459B (zh) * 2016-09-30 2019-03-08 佛山市青松科技股份有限公司 一种图像显示系统及方法
CN107170047A (zh) * 2017-04-13 2017-09-15 北京小鸟看看科技有限公司 虚拟现实场景的更新方法、设备及虚拟现实设备
US10914957B1 (en) * 2017-05-30 2021-02-09 Apple Inc. Video compression methods and apparatus
CN107229335A (zh) * 2017-06-01 2017-10-03 北京小鸟看看科技有限公司 一种虚拟现实场景的更换方法、装置和头戴显示设备
CN107229355A (zh) 2017-06-20 2017-10-03 柳州易农科技有限公司 一种用于家庭使用的无线鼠标

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140002580A1 (en) * 2012-06-29 2014-01-02 Monkeymedia, Inc. Portable proprioceptive peripatetic polylinear video player
CN103841299A (zh) * 2012-11-23 2014-06-04 西安天动数字科技有限公司 虚拟演播室系统
CN103226838A (zh) * 2013-04-10 2013-07-31 福州林景行信息技术有限公司 地理场景中移动监控目标的实时空间定位方法
CN107231531A (zh) * 2017-05-23 2017-10-03 青岛大学 一种网络虚拟现实技术与实拍结合影视制作系统
CN107158695A (zh) * 2017-06-16 2017-09-15 苏州蜗牛数字科技股份有限公司 一种现实与虚拟场景混合展现的方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3745700A4 *

Also Published As

Publication number Publication date
EP3745700A1 (en) 2020-12-02
US11354816B2 (en) 2022-06-07
CN108289220A (zh) 2018-07-17
US20200349734A1 (en) 2020-11-05
ES3035820T3 (en) 2025-09-09
EP3745700A4 (en) 2021-10-13
CN108289220B (zh) 2020-11-27
EP3745700B1 (en) 2025-04-23
JP2020534724A (ja) 2020-11-26

Similar Documents

Publication Publication Date Title
CN108289220B (zh) 虚拟图像处理方法、图像处理系统及存储介质
US20250166312A1 (en) Immersive content production system with multiple targets
CN112330736B (zh) 一种场景画面的拍摄方法、装置、电子设备及存储介质
US7118228B2 (en) Image display system
US9679369B2 (en) Depth key compositing for video and holographic projection
US20070247518A1 (en) System and method for video processing and display
CN110300316A (zh) 视频中植入推送信息的方法、装置、电子设备及存储介质
US11792511B2 (en) Camera system utilizing auxiliary image sensors
CN105898139A (zh) 全景视频制作方法及装置、全景视频播放方法及装置
CN113692734A (zh) 用于采集和投影图像的系统和方法，以及该系统的应用
JP7676209B2 (ja) 情報処理装置、情報処理方法、及び、プログラム
WO2021031210A1 (zh) 视频处理方法和装置、存储介质和电子设备
WO2026016666A1 (zh) 一种虚拟拍摄方法、视频处理方法及装置
CN108509173A (zh) 图像展示系统及方法、存储介质、处理器
WO2023189079A1 (ja) 画像処理装置、および画像処理方法、並びにプログラム
WO2026040556A1 (zh) 一种虚拟拍摄方法及装置
CN114979689A (zh) 多机位直播导播方法、设备以及介质
CN107016728A (zh) 一种在虚拟现实场景中模拟摄影的方法及系统
US11615755B1 (en) Increasing resolution and luminance of a display
CN105915972A (zh) 一种虚拟现实中4k视频优化方法和装置
CN115514921B (zh) 一种多机位led显示拍摄方法、系统、装置及存储介质
US20210297649A1 (en) Image data output device, content creation device, content reproduction device, image data output method, content creation method, and content reproduction method
CN107911686B (zh) 控制方法以及摄像终端
US20250247611A1 (en) Device, computer program and method
CN112433599A (zh) 显示方法、装置和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18899867

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020511180

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018899867

Country of ref document: EP

Effective date: 20200817

WWG Wipo information: grant in national office

Ref document number: 2018899867

Country of ref document: EP