WO2009125988A2 - Fast multi-view three-dimensinonal image synthesis apparatus and method - Google Patents

Fast multi-view three-dimensinonal image synthesis apparatus and method Download PDF

Info

Publication number
WO2009125988A2
WO2009125988A2 PCT/KR2009/001834 KR2009001834W WO2009125988A2 WO 2009125988 A2 WO2009125988 A2 WO 2009125988A2 KR 2009001834 W KR2009001834 W KR 2009001834W WO 2009125988 A2 WO2009125988 A2 WO 2009125988A2
Authority
WO
WIPO (PCT)
Prior art keywords
view
pixel data
disparity map
image
right image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2009/001834
Other languages
French (fr)
Other versions
WO2009125988A3 (en
Inventor
Hong Jeong
Jang Myoung Kim
Sung Chan Park
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
POSTECH Academy Industry Foundation
Original Assignee
POSTECH Academy Industry Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by POSTECH Academy Industry Foundation filed Critical POSTECH Academy Industry Foundation
Priority to JP2011503910A priority Critical patent/JP2011519209A/en
Priority to EP09731429.8A priority patent/EP2263383A4/en
Publication of WO2009125988A2 publication Critical patent/WO2009125988A2/en
Priority to US12/923,820 priority patent/US20110026809A1/en
Anticipated expiration legal-status Critical
Publication of WO2009125988A3 publication Critical patent/WO2009125988A3/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Definitions

  • the present invention relates to a fast multi-view 3D (three-dimensional) image synthesis apparatus and method; and, more particularly, to a fast multi-view 3D image synthesis apparatus and method using a disparity map for, e.g., autostereoscopic 3D TV (television) displays.
  • Stereo image matching is a technique for re-creating 3D spatial information from a pair of 2D (two-dimensional) images.
  • Fig. 1 illustrates an explanatory view of stereo image matching.
  • First found in the stereo image matching are left and right pixels 10 and 11, corresponding to an identical point (X,Y,Z) in a 3D space, on image lines on a left image epipolar line and a right image epipolar line, respectively.
  • a disparity for a conjugate pixel pair i.e., the left and right pixels
  • the disparity has distance information, and a geometrical distance calculated from the disparity is referred to as a depth.
  • a disparity map is a set of disparities obtained by the stereo image matching. From the disparity map of an input image, 3D distance and shape information on an observation space can be measured. Hence, the disparity map is used in a multiple image synthesis, which is necessary for the autostereoscopic 3D TV displays.
  • the present invention provides a fast multi-view 3D image synthesis apparatus and method using a disparity map for, e.g., autostereoscopic 3D TV displays.
  • a fast multi-view three-dimensional image synthesis apparatus including:
  • a disparity map generation module for generating a left image disparity map by using left and right image pixel data
  • intermediate-view generation modules for generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map
  • a multi-view three-dimensional image generation module for generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data.
  • the left and right image pixel data are on an identical epipolar line.
  • the disparity map generation module generates the left image disparity map based on belief propagation based algorithm.
  • each of the intermediate-view generation module includes:
  • a right image disparity map generation unit for generating a rough right image disparity map by using the left image disparity map
  • an occluded region compensation unit for generating a right image disparity map by removing occluded regions from the rough right image disparity map
  • an intermediate-view generation unit for generating the intermediate-view pixel data from the different view points by using the right image disparity map generated by the occluded region compensation unit.
  • the multi-view three-dimensional image generation module generates the multi-view three-dimensional image pixel data by interweaving the intermediate-view pixel data from the different view points.
  • a fast multi-view three-dimensional image synthesis method including:
  • said generating the intermediate-view pixel data includes:
  • said determining the right image disparity includes:
  • said compensating the occluded region includes:
  • said generating the intermediate-view pixel data from the different viewpoints is performed in parallel.
  • the multi-view 3D image synthesis apparatus can perform a fast multi-view 3D image synthesis via linear parallel processing, and also can be implemented with a small-sized chip. Further, multi-view 3D images having a low error rate can be generated.
  • the present invention can be competitively applied to not only autostereoscopic 3D TV displays but also various autostereoscopic 3D displays, e.g., autostereoscopic 3D mobile phone displays, autostereoscopic 3D medical instruments displays and the like, due to the high-speed and high-quality multi-view 3D image synthesis thereof.
  • various autostereoscopic 3D displays e.g., autostereoscopic 3D mobile phone displays, autostereoscopic 3D medical instruments displays and the like, due to the high-speed and high-quality multi-view 3D image synthesis thereof.
  • Fig. 1 illustrates an explanatory view of stereo image matching
  • Fig. 2 illustrates an explanatory view of generating an intermediate-view in accordance with an embodiment of the present invention
  • Fig. 3 illustrates a block diagram of a fast multi-view 3D image synthesis apparatus in accordance with the embodiment of the present invention
  • Fig. 4 illustrates a parallel processing mechanism in the intermediate-view image generation unit of Fig. 3;
  • Fig. 5 illustrates a flowchart of intermediate-view generation procedure performed in the intermediate-view image generation unit shown in Fig. 3;
  • Fig. 6 illustrates a flowchart of right image disparity map d RL generation procedure performed in the intermediate-view generation module in Fig. 3;
  • FIG. 7 illustrates a flowchart of occluded region compensation procedure in Fig. 6;
  • Fig. 8 illustrates a flowchart of multi-view 3D image generation procedure performed in the multi-view 3D image generation module in Fig. 3;
  • Fig. 9 illustrates a block diagram of a parallel processing mechanism for a fast multi-view image synthesis using the apparatus in Fig. 3.
  • FIG. 2 illustrates an explanatory view of generating an intermediate-view in accordance with an embodiment of the present invention.
  • an intermediate-view image 20 is a re-projected image from a left image 21 and a right image 22.
  • Fig. 3 illustrates a block diagram of a fast multi-view 3D image synthesis apparatus in accordance with the embodiment of the present invention.
  • a fast multi-view 3D image synthesis apparatus of this embodiment includes a disparity map generation module 100, an intermediate-view image generation module 200 and a multi-view 3D image generation module 300.
  • the disparity map generation module 100 receives left and right images to produce a left image disparity map d LR .
  • the intermediate-view image generation module 200 receives the left image disparity map d LR from the disparity map generation module 100 and the left and right images to produce intermediate-view images from different viewpoints, i.e., a 1 st to N th intermediate-view images, wherein N is an integer.
  • the multi-view 3D image generation module 300 receives the 1 st to N th intermediate-view images from the intermediate-view image generation module 200 to produce a multi-view 3D image for, e.g., autostereoscopic 3D TV displays, which gives 3D perception to viewers.
  • the intermediate-view image generation module 200 includes a right image disparity map ( d RL ) generation unit 210, an occluded region compensation unit 220 and an intermediate-view image generation unit 230.
  • the right image disparity map generation unit 210 receives the left image disparity map d LR from the disparity map generation module 100 to produce a rough right image disparity map having therein occluded regions.
  • the occluded region compensation unit 220 receives the rough right image disparity map from the right image disparity map generation unit 210 and removes the occluded regions therefrom to produce a precise right image disparity map d RL .
  • the intermediate-view image generation unit 230 receives the precise right image disparity map d RL from the occluded region compensation unit 220 and the left and right images to produce the 1 st to N th intermediate-view images from different viewpoints.
  • the multi-view 3D image generation module 300 receives the 1 st to N th intermediate-view images from the intermediate-view image generation unit 230 and calculates multi-view image pixel data to produce the multi-view 3D image.
  • the disparity map generation module 100, the intermediate-view image generation module 200 and the multi-view 3D image generation module 300 repeatedly perform the above-described processes by epipolar line basis to complete the multi-view 3D image.
  • Fig. 4 illustrates a parallel processing mechanism in the intermediate-view image generation unit 230 of Fig. 3.
  • the input data of the intermediate-view image generation unit 230 includes one line pixel data 411 of left image I L , one line pixel data 413 of right image I R on the same epipolar line of a pair of images and one line pixel data 412 of right image disparity d RL for the stereo image pair.
  • the intermediate data of the intermediate-view image generation unit 230 includes reprojected intermediate image 421 from the left image I L , reprojected intermediate image 424 from the right image I R .
  • reference numeral 422 indicates multiplication of the left image disparity d LR and a coefficient (0 ⁇ ⁇ 1), which represents a relative position of the intermediate image between the left and right image
  • reference numeral 423 indicates multiplication of the right image disparity d RL and a coefficient 1- (0 ⁇ ⁇ 1), which also represents a relative position of the intermediate image between the left and right image.
  • the intermediate image 421 is projected from one line pixel data 411 of the left image I L by using * d LR
  • the intermediate image 424 is projected from one line pixel data 413 of the right image 413 by using (1- )* d RL .
  • the output data of the intermediate-view image generation unit 230 includes one line pixel data 430 of the N th intermediate-view.
  • the one line pixel data 430 of the N th intermediate-view is produced by combining the reprojected intermediate image 421 from the left image pixel data 411 and the reprojected intermediate image 424 from the right image pixel data 413.
  • Fig. 5 illustrates a flowchart of intermediate-view generation procedure performed in the intermediate-view image generation unit 230.
  • Equation 1 Equation 1
  • M is a width of the image
  • N is the number of viewpoints
  • (X,Y) is a plane coordinate in the reprojected intermediate image.
  • Occluded region occurs when reprojected image from left to intermediate or from right to intermediate has no information.
  • the region occluded in the orginal left image is exposed in the reprojected intermediate image, because viewpoints therebetween are different.
  • the initial values of I L (X IL ,Y) and I R (X IR ,Y) are set to 0. Accordingly, by projecting from the left image to the intermediate image and from the right image to the intermediate image, a point in unoccluded region may differ from a point in the occluded region by a pixel value thereof.
  • d LR is the disparity map from the left image I L to the right image I R
  • d RL is the disparity map from the right image I R to the left image I L .
  • a disparity with respect to a desired intermediate position is required to be assigned. This process is performed by projecting the disparity maps d LR and d RL onto the intermediate image. For a position (X L ,Y) on the left image, the projected position in the intermediate image is (X L + *d LR (X,Y),Y) . For a position (X R ,Y) on the right image, the projected position in the intermediate image is (X R +(1- )*d RL (X,Y),Y) .
  • step S530 it is determined whether a final desired intermediate image is near to the left image.
  • the intermediate image I IP is used for compensating the occluded region of the intermediate image I IL (step S541). Meanwhile, if it is determined in the step S530 that the final desired intermediate image is near to right image I R , the intermediate image I IL is used for compensating the occluded region of the intermediate image I IR (step S542).
  • I IR (X IR ,Y) is set as I IL (X IL ,Y) and the final desired intermediate image I I is assigned with I IL (X IL ,Y) in the step S541, or, I IL (X IL ,Y) is set as I IR (X IR ,Y) and the final desired intermediate image I I is assigned with I IR (X IR ,Y) in the step S542, as in Equation 3 or 4.
  • Fig. 6 illustrates a flowchart of right image disparity map d RL generation procedure performed in the intermediate-view generation module 200.
  • d RL is the disparity map from the right image I R to the left image I L
  • d LR is the disparity map from the left image I L to the right image I R .
  • the disparity map generation module 100 produces the disparity map d LR only.
  • the disparity map d RL is also needed.
  • the intermediate-view generation module 200 calculates the disparity map d RL by mapping point from d RL to d LR .
  • An initial value of the disparity map d RL (X,Y) is set as in Equation 5 for the occluded region detection (step S610):
  • d RL is the disparity map from the right image I R to the left image I L
  • (X,Y) is the plane coordinate in the disparity map.
  • the occluded region occurs when the reprojected disparity map d RL from d LR has no information.
  • the region occluded in the orginal disparity map d LR is exposed in the reprojected disparity map d RL because the viewpoints therebetween are different.
  • the initial value of the reprojected disparity map d RL is set to 0.
  • points in the unoccluded region and in the occluded region may differ by pixel values thereof.
  • the intensity value of the disparity map d RL is assigned by projecting (mapping) from the left image I L and the right image I R (step S620) as in Equation 6:
  • d LR is the disparity map from the left image I L to the right image I R
  • d RL is the disparity map from the right image I R to the left image I L .
  • the intensity value with respect to the original disparity map d LR location is required to be assigned. This process is performed by projecting the disparity map d LR onto the disparity map d RL .
  • the projected position on the disparity map d RL is (X+d LR ,Y) .
  • the intensity value d LR (X,Y) is assigned in the step S620 to synthesize the virtual view.
  • the occluded region of the disparity map d RL still exist.
  • the occluded region is compensated by neighbor pixel values (step S630).
  • the generation of the disparity map d RL is finished. The compensation of the occluded region compensation will be described later.
  • FIG. 7 illustrates a flowchart of occluded region compensation procedure in Fig. 6.
  • the disparity map d RL having therein the occluded region is synthesized.
  • forward and backward neighbor pixel values of the occluded region are used.
  • conflicts can occur if both of the forward and backward neighbor pixel values are in the occluded region.
  • the smaller one among the neighbor pixel values is used.
  • Equation 7 Equation 7
  • d RL is the disparity map from the right image I R to the left image I L
  • (X-1,Y-1) stands for the forward neighbor pixel value of the occluded region
  • d F RL indicating the intensity value of the occluded region after compensation is set as the forward neighbor pixel value.
  • the intensity value of the occluded region of d RL is filled with the backward neighbor pixel value as in Equation 8 (step S720):
  • d RL is the disparity map from the right image I R to the left image I L
  • (X+1,Y+1) stands for the backward neighbor pixel value of the occluded region
  • d B RL indicating the intensity value of the occluded region after compensation is set as the backward neighbor pixel value.
  • d F RL (X,Y) If d F RL (X,Y) ⁇ d B RL (X,Y) , the forward neighbor pixel value d F RL is selected to compensate the occluded region of d RL (X,Y) (step S741). If d F RL (X,Y)>d B RL (X,Y) , the backward neighbor pixel value d B RL is selected to compensate the occluded region d RL (X,Y) (step S742).
  • the intensity value of the occluded region d RL (X,Y) is determined through the steps S730, S741 and S742.
  • the occluded region of d RL (X,Y) is a background object in the stereo image pairs. Since a disparity value of a background object is always smaller than that of a foreground object, the disparity value of the occluded region of d RL (X,Y) is given with the smaller value between forward and backward neighbor pixel values.
  • Fig. 8 illustrates a flowchart of multi-view 3D image generation procedure performed in the multi-view 3D image generation module 300 in Fig. 3.
  • An autostereoscopic multi-view display with n views requires n-2 intermediate-view from various viewpoints between original left and right images. If the intermediate-view synthesis in Fig. 5 is applied, intermediate-views from various viewpoints can be created.
  • a multi-view 3D image for the autostereoscopic 3D TV displays is made by interweaving the columns from n views of various viewpoints.
  • the n views are arranged so that the left eye is allowed to see strips from left eye images only and the right eye is allowed to see strips from right eye images only, which gives a viewer a 3D perception (depth of a 3D scene).
  • I AutostereoView stands for the pixel value of a multi-view 3D image
  • I 0 to I n-1 stand for the pixel values of n sub-images from various viewpoints to form a multi-view 3D image for the autostereoscopic 3D TV displays.
  • I 0 is the original left image
  • I n-1 is the original right image
  • I 1 to I n-2 stand for n-2 intermediate-views from various viewpoints between the original left and right images.
  • X%n represents a remainder of division of the sub-images n by the horizontal axis X in steps S810 to S860,
  • Multi-view 3D image content observed by the viewer depends upon a position of the viewer with respect to the autostereoscopic 3D TV displays screen. Due to the autostereoscopic 3D TV displays screen (lenticular or Parallax barrier), the left eye of the viewer receives a column pixels that is different from what the right eye thereof receives, which gives the viewer a 3D perception (depth of a 3D scene).
  • Fig. 9 illustrates a block diagram of a parallel processing mechanism for a fast multi-view image synthesis using the apparatus in Fig. 3.
  • the disparity map generation module 100 outputs one line pixel value of a left image disparity map d LR . Further, one line pixel values of the left and right images at the same time are also produced.
  • the number of the intermediate-view generation module 200 is N-2 .
  • each of the N-2 intermediate-view generation modules After receiving one line pixel values of the left image, the right image and the left image disparity map d LR , each of the N-2 intermediate-view generation modules outputs one line pixel value from a viewpoint thereof.
  • the 1 st intermediate-view is the left image and the N th intermediate-view is the right image.
  • the multi-view 3D image generation module 300 receives one line pixel values of the 1 st to N th intermediate-views from the intermediate-view generation modules 200, and outputs a multi-view 3D image for autostereoscopic 3D TV displays to give user a 3D perception. Since the respective 1 st to N th intermediate-views are produced line by line, the fast multi-view image synthesis method using disparity map can be processed in parallel. That is, the left and right images can be synthesized a multi-view 3D image for the autostereoscopic 3D TV displays in parallel.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)

Abstract

A fast multi-view three-dimensional image synthesis apparatus includes: a disparity map generation module (100) for generating a left image disparity map (dLR) by using left and right image pixel data; intermediate-view generation modules (200) for generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map (dLR); and a multi-view three-dimensional image generation module (300) for generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data. Each of the intermediate-view generation module (200) includes: a right image disparity map generation unit (210) for generating a rough right image disparity map; an occluded region compensation unit (220) for generating a right image disparity map (dRL) by removing occluded regions from the rough right image disparity map; and an intermediate-view generation unit (230) for generating the intermediate-view pixel data from the different view points.

Description

FAST MULTI-VIEW THREE-DIMENSINONAL IMAGE SYNTHESIS APPARATUS AND METHOD
The present invention relates to a fast multi-view 3D (three-dimensional) image synthesis apparatus and method; and, more particularly, to a fast multi-view 3D image synthesis apparatus and method using a disparity map for, e.g., autostereoscopic 3D TV (television) displays.
Stereo image matching is a technique for re-creating 3D spatial information from a pair of 2D (two-dimensional) images.
Fig. 1 illustrates an explanatory view of stereo image matching. First found in the stereo image matching are left and right pixels 10 and 11, corresponding to an identical point (X,Y,Z) in a 3D space, on image lines on a left image epipolar line and a right image epipolar line, respectively.
Next, a disparity for a conjugate pixel pair, i.e., the left and right pixels, is obtained. Referring to Fig. 1, a disparity d is defined as d=x l -x r . The disparity has distance information, and a geometrical distance calculated from the disparity is referred to as a depth.
A disparity map is a set of disparities obtained by the stereo image matching. From the disparity map of an input image, 3D distance and shape information on an observation space can be measured. Hence, the disparity map is used in a multiple image synthesis, which is necessary for the autostereoscopic 3D TV displays.
However, the image synthesis speed and the quality of a synthesized image are still remained to be improved.
In view of the above, the present invention provides a fast multi-view 3D image synthesis apparatus and method using a disparity map for, e.g., autostereoscopic 3D TV displays.
In accordance with an aspect of the present invention, there is provided a fast multi-view three-dimensional image synthesis apparatus, including:
a disparity map generation module for generating a left image disparity map by using left and right image pixel data;
intermediate-view generation modules for generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map; and
a multi-view three-dimensional image generation module for generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data.
Preferably, the left and right image pixel data are on an identical epipolar line.
Preferably, the disparity map generation module generates the left image disparity map based on belief propagation based algorithm.
Preferably, each of the intermediate-view generation module includes:
a right image disparity map generation unit for generating a rough right image disparity map by using the left image disparity map;
an occluded region compensation unit for generating a right image disparity map by removing occluded regions from the rough right image disparity map; and
an intermediate-view generation unit for generating the intermediate-view pixel data from the different view points by using the right image disparity map generated by the occluded region compensation unit.
Preferably, the multi-view three-dimensional image generation module generates the multi-view three-dimensional image pixel data by interweaving the intermediate-view pixel data from the different view points.
In accordance with another aspect of the present invention, there is provided a fast multi-view three-dimensional image synthesis method, including:
generating a left image disparity map by using left and right image pixel data;
generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map; and
generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data.
Preferably, said generating the intermediate-view pixel data includes:
initializing the intermediate-view pixel data;
determining a first intermediate-view pixel data mapped from the left image pixel data by using the left image disparity map;
determining a right image disparity map mapped from the left image pixel data by using the left image disparity map;
determining a second intermediate-view pixel data mapped from the right image pixel data by using the right image disparity map;
determining whether a desired intermediate-view is near to the left image pixel data or near to the right image pixel data; and
combining the first and second intermediate-view pixel data to generate the intermediate-view pixel data.
Preferably, said determining the right image disparity includes:
initializing the right image disparity map;
determining pixel values of the right image disparity map mapped from the left image disparity map by using the left image disparity map; and
compensating an occluded region of the right image disparity map by using pixel values of pixels neighboring pixels whose pixel values have been determined.
Preferably, said compensating the occluded region includes:
storing a forward neighbor pixel value of a pixel forwardly neighboring a pixel whose pixel value has been determined;
storing a backward neighbor pixel value of a pixel backwardly neighboring a pixel whose pixel value has been determined;
comparing the forward and backward neighbor pixel value;
selecting a smaller pixel value between the forward and backward neighbor pixel values; and
filling a pixel value of a pixel in the occluded region with the selected pixel value.
Preferably, said generating the intermediate-view pixel data from the different viewpoints is performed in parallel.
According to the present invention, the multi-view 3D image synthesis apparatus can perform a fast multi-view 3D image synthesis via linear parallel processing, and also can be implemented with a small-sized chip. Further, multi-view 3D images having a low error rate can be generated.
The present invention can be competitively applied to not only autostereoscopic 3D TV displays but also various autostereoscopic 3D displays, e.g., autostereoscopic 3D mobile phone displays, autostereoscopic 3D medical instruments displays and the like, due to the high-speed and high-quality multi-view 3D image synthesis thereof.
Fig. 1 illustrates an explanatory view of stereo image matching;
Fig. 2 illustrates an explanatory view of generating an intermediate-view in accordance with an embodiment of the present invention;
Fig. 3 illustrates a block diagram of a fast multi-view 3D image synthesis apparatus in accordance with the embodiment of the present invention;
Fig. 4 illustrates a parallel processing mechanism in the intermediate-view image generation unit of Fig. 3;
Fig. 5 illustrates a flowchart of intermediate-view generation procedure performed in the intermediate-view image generation unit shown in Fig. 3;
Fig. 6 illustrates a flowchart of right image disparity map d RL generation procedure performed in the intermediate-view generation module in Fig. 3;
FIG. 7 illustrates a flowchart of occluded region compensation procedure in Fig. 6;
Fig. 8 illustrates a flowchart of multi-view 3D image generation procedure performed in the multi-view 3D image generation module in Fig. 3; and
Fig. 9 illustrates a block diagram of a parallel processing mechanism for a fast multi-view image synthesis using the apparatus in Fig. 3.
Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings, which form a part hereof.
Fig. 2 illustrates an explanatory view of generating an intermediate-view in accordance with an embodiment of the present invention. In Fig. 2, an intermediate-view image 20 is a re-projected image from a left image 21 and a right image 22.
Fig. 3 illustrates a block diagram of a fast multi-view 3D image synthesis apparatus in accordance with the embodiment of the present invention. As shown in Fig. 3, a fast multi-view 3D image synthesis apparatus of this embodiment includes a disparity map generation module 100, an intermediate-view image generation module 200 and a multi-view 3D image generation module 300.
The disparity map generation module 100 receives left and right images to produce a left image disparity map d LR . The intermediate-view image generation module 200 receives the left image disparity map d LR from the disparity map generation module 100 and the left and right images to produce intermediate-view images from different viewpoints, i.e., a 1 st to N th intermediate-view images, wherein N is an integer. The multi-view 3D image generation module 300 receives the 1 st to N th intermediate-view images from the intermediate-view image generation module 200 to produce a multi-view 3D image for, e.g., autostereoscopic 3D TV displays, which gives 3D perception to viewers.
As shown in Fig. 3, the intermediate-view image generation module 200 includes a right image disparity map (d RL ) generation unit 210, an occluded region compensation unit 220 and an intermediate-view image generation unit 230. The right image disparity map generation unit 210 receives the left image disparity map d LR from the disparity map generation module 100 to produce a rough right image disparity map having therein occluded regions. The occluded region compensation unit 220 receives the rough right image disparity map from the right image disparity map generation unit 210 and removes the occluded regions therefrom to produce a precise right image disparity map d RL . The intermediate-view image generation unit 230 receives the precise right image disparity map d RL from the occluded region compensation unit 220 and the left and right images to produce the 1 st to N th intermediate-view images from different viewpoints.
The multi-view 3D image generation module 300 receives the 1 st to N th intermediate-view images from the intermediate-view image generation unit 230 and calculates multi-view image pixel data to produce the multi-view 3D image.
The disparity map generation module 100, the intermediate-view image generation module 200 and the multi-view 3D image generation module 300 repeatedly perform the above-described processes by epipolar line basis to complete the multi-view 3D image.
Fig. 4 illustrates a parallel processing mechanism in the intermediate-view image generation unit 230 of Fig. 3.
Referring to Fig. 4, the input data of the intermediate-view image generation unit 230 includes one line pixel data 411 of left image I L , one line pixel data 413 of right image I R on the same epipolar line of a pair of images and one line pixel data 412 of right image disparity d RL for the stereo image pair. The intermediate data of the intermediate-view image generation unit 230 includes reprojected intermediate image 421 from the left image I L , reprojected intermediate image 424 from the right image I R . Further, reference numeral 422 indicates multiplication of the left image disparity d LR and a coefficient
Figure PCTKR2009001834-appb-I000001
(0<
Figure PCTKR2009001834-appb-I000002
<1), which represents a relative position of the intermediate image between the left and right image, and reference numeral 423 indicates multiplication of the right image disparity d RL and a coefficient 1-
Figure PCTKR2009001834-appb-I000003
(0<
Figure PCTKR2009001834-appb-I000004
<1), which also represents a relative position of the intermediate image between the left and right image. The intermediate image 421 is projected from one line pixel data 411 of the left image I L by using
Figure PCTKR2009001834-appb-I000005
*d LR , and the intermediate image 424 is projected from one line pixel data 413 of the right image 413 by using (1-
Figure PCTKR2009001834-appb-I000006
)*d RL . The output data of the intermediate-view image generation unit 230 includes one line pixel data 430 of the N th intermediate-view. The one line pixel data 430 of the N th intermediate-view is produced by combining the reprojected intermediate image 421 from the left image pixel data 411 and the reprojected intermediate image 424 from the right image pixel data 413.
Fig. 5 illustrates a flowchart of intermediate-view generation procedure performed in the intermediate-view image generation unit 230.
Referring to Fig. 5, initial values of reprojected intermediate images I L (X IL ,Y) and I R (X IR ,Y) from the left image and from the right image, respectively, are as in Equation 1 (step S510):
MathFigure 1
Figure PCTKR2009001834-appb-M000001
wherein M is a width of the image, N is the number of viewpoints, and (X,Y) is a plane coordinate in the reprojected intermediate image.
The initial values are given for occluded region detection (steps S541 and S542). Occluded region occurs when reprojected image from left to intermediate or from right to intermediate has no information. The region occluded in the orginal left image is exposed in the reprojected intermediate image, because viewpoints therebetween are different. In order to detect the occluded region, the initial values of I L (X IL ,Y) and I R (X IR ,Y) are set to 0. Accordingly, by projecting from the left image to the intermediate image and from the right image to the intermediate image, a point in unoccluded region may differ from a point in the occluded region by a pixel value thereof.
The intermediate images I IL and I IR projected from the left image I L and the right image I R , respectively, are assigned with intensity values as in Equation 2 (step S520):
MathFigure 2
Figure PCTKR2009001834-appb-M000002
wherein
Figure PCTKR2009001834-appb-I000007
is in a range 0<
Figure PCTKR2009001834-appb-I000008
<1,
Figure PCTKR2009001834-appb-I000009
and 1-
Figure PCTKR2009001834-appb-I000010
stand for normalized relative distances of the desired intermediate images I IL and I IR from the left and right images I L and I R respectively, d LR is the disparity map from the left image I L to the right image I R , and d RL is the disparity map from the right image I R to the left image I L .
If
Figure PCTKR2009001834-appb-I000011
=0 and
Figure PCTKR2009001834-appb-I000012
=1 indicate positions of the left and right images, respectively, 0<
Figure PCTKR2009001834-appb-I000013
<1 indicates a valid position of an intermediate image. In order to generate an intermediate image, a disparity with respect to a desired intermediate position is required to be assigned. This process is performed by projecting the disparity maps d LR and d RL onto the intermediate image. For a position (X L ,Y) on the left image, the projected position in the intermediate image is (X L +
Figure PCTKR2009001834-appb-I000014
*d LR (X,Y),Y). For a position (X R ,Y) on the right image, the projected position in the intermediate image is (X R +(1-
Figure PCTKR2009001834-appb-I000015
)*d RL (X,Y),Y). For each position (X I ,Y) on the intermediate image, two corresponding positions (X L ,Y) on the left image and (X R ,Y) on the right image can be easily found through the two disparity maps d LR and d RL . For a position (X I ,Y) on the intermediate images I IL andI IR , virtual views can be synthesized by assigning the intensity values I IL (X I ,Y)=I IL (X L +
Figure PCTKR2009001834-appb-I000016
*d LR (X,Y),Y)=I L (X L ,Y) and I IR (X I ,Y)=I IR (X R +(1-
Figure PCTKR2009001834-appb-I000017
)*d RL (X,Y),Y)=I R (X R ,Y) (step S520).
Referring to Fig. 5, it is determined whether a final desired intermediate image is near to the left image (step S530).
According to the determination result in the step S530, compensation of the occluded region is performed by using Equation 3 or 4:
MathFigure 3
MathFigure 4
Figure PCTKR2009001834-appb-M000004
wherein (X IL ,Y) stands for the occluded region of the intermediate image I IL when I IL (X IL ,Y)=0, and (X IR ,Y) stands for the occluded region of the intermediate image I IR when I IR (X IR ,Y)=0.
Referencing back to Fig. 5, if it is determined in the step S530 that the final desired intermediate image I I is near to left image I L , the intermediate image I IP is used for compensating the occluded region of the intermediate image I IL (step S541). Meanwhile, if it is determined in the step S530 that the final desired intermediate image is near to right image I R , the intermediate image I IL is used for compensating the occluded region of the intermediate image I IR (step S542). Through the compensation, I IR (X IR ,Y) is set as I IL (X IL ,Y) and the final desired intermediate image I I is assigned with I IL (X IL ,Y) in the step S541, or, I IL (X IL ,Y) is set as I IR (X IR ,Y) and the final desired intermediate image I I is assigned with I IR (X IR ,Y) in the step S542, as in Equation 3 or 4.
Fig. 6 illustrates a flowchart of right image disparity map d RL generation procedure performed in the intermediate-view generation module 200. As described above, d RL is the disparity map from the right image I R to the left image I L , and d LR is the disparity map from the left image I L to the right image I R . The disparity map generation module 100 produces the disparity map d LR only. However, in the intermediate-view generation procedure performed in the intermediate-view generation module 200, the disparity map d RL is also needed. Thus, the intermediate-view generation module 200 calculates the disparity map d RL by mapping point from d RL to d LR .
An initial value of the disparity map d RL (X,Y) is set as in Equation 5 for the occluded region detection (step S610):
MathFigure 5
Figure PCTKR2009001834-appb-M000005
wherein d RL is the disparity map from the right image I R to the left image I L , and (X,Y) is the plane coordinate in the disparity map.
The occluded region occurs when the reprojected disparity map d RL from d LR has no information. The region occluded in the orginal disparity map d LR is exposed in the reprojected disparity map d RL because the viewpoints therebetween are different. In order to detect the occluded region, the initial value of the reprojected disparity map d RL is set to 0. Thus, after projecting (mapping) from d LR to d RL , points in the unoccluded region and in the occluded region may differ by pixel values thereof.
The intensity value of the disparity map d RL is assigned by projecting (mapping) from the left image I L and the right image I R (step S620) as in Equation 6:
MathFigure 6
Figure PCTKR2009001834-appb-M000006
wherein d LR is the disparity map from the left image I L to the right image I R , and d RL is the disparity map from the right image I R to the left image I L .
In order to generate the disparity map d RL , the intensity value with respect to the original disparity map d LR location is required to be assigned. This process is performed by projecting the disparity map d LR onto the disparity map d RL . For a position (X,Y) on the disparity map d LR , the projected position on the disparity map d RL is (X+d LR ,Y). For each position (X+d LR ,Y) on the disparity map d RL , the corresponding position (X,Y) on the disparity map d LR can be easily found through the disparity map d LR . For a position (X+d LR ,Y) on the disparity map d RL , the intensity value d LR (X,Y) is assigned in the step S620 to synthesize the virtual view.
After the step S620, though the disparity map d RL is synthesized by assigning the intensity value as in Equation 6, the occluded region of the disparity map d RL still exist. To solve this problem, the occluded region is compensated by neighbor pixel values (step S630). When the occluded region of the disparity map d RL has been compensated, the generation of the disparity map d RL is finished. The compensation of the occluded region compensation will be described later.
FIG. 7 illustrates a flowchart of occluded region compensation procedure in Fig. 6. In the step S620 in Fig. 6, the disparity map d RL having therein the occluded region is synthesized. In order to compensate the occluded region in d RL , forward and backward neighbor pixel values of the occluded region are used. Here, conflicts can occur if both of the forward and backward neighbor pixel values are in the occluded region. Hence, by comparing the forward and backward neighbor pixel values, the smaller one among the neighbor pixel values is used.
Referencing to Fig. 7, the intensity value of the occluded region of d RL is filled with the forward neighbor pixel value as in Equation 7 (step S710):
MathFigure 7
Figure PCTKR2009001834-appb-M000007
wherein d RL is the disparity map from the right image I R to the left image I L , (X,Y) stands for the occluded region of the disparity map d RL when d RL (X,Y)=0, (X-1,Y-1) stands for the forward neighbor pixel value of the occluded region, and d F RL indicating the intensity value of the occluded region after compensation is set as the forward neighbor pixel value.
Referring to Fig. 7, the intensity value of the occluded region of d RL is filled with the backward neighbor pixel value as in Equation 8 (step S720):
MathFigure 8
Figure PCTKR2009001834-appb-M000008
where d RL is the disparity map from the right image I R to the left image I L , (X,Y) stands for the occluded region of the disparity map d RL when d RL (X,Y)=0, (X+1,Y+1) stands for the backward neighbor pixel value of the occluded region, and d B RL indicating the intensity value of the occluded region after compensation is set as the backward neighbor pixel value.
The intensity values of the occluded region d F RL and d B RL are compared as in Equation 9 (step S730):
MathFigure 9
Figure PCTKR2009001834-appb-M000009
If d F RL (X,Y)<d B RL (X,Y), the forward neighbor pixel value d F RL is selected to compensate the occluded region of d RL (X,Y) (step S741). If d F RL (X,Y)>d B RL (X,Y), the backward neighbor pixel value d B RL is selected to compensate the occluded region d RL (X,Y) (step S742).
The intensity value of the occluded region d RL (X,Y) is determined through the steps S730, S741 and S742. The occluded region of d RL (X,Y) is a background object in the stereo image pairs. Since a disparity value of a background object is always smaller than that of a foreground object, the disparity value of the occluded region of d RL (X,Y) is given with the smaller value between forward and backward neighbor pixel values.
Fig. 8 illustrates a flowchart of multi-view 3D image generation procedure performed in the multi-view 3D image generation module 300 in Fig. 3.
An autostereoscopic multi-view display with n views requires n-2 intermediate-view from various viewpoints between original left and right images. If the intermediate-view synthesis in Fig. 5 is applied, intermediate-views from various viewpoints can be created.
A multi-view 3D image for the autostereoscopic 3D TV displays is made by interweaving the columns from n views of various viewpoints. The n views are arranged so that the left eye is allowed to see strips from left eye images only and the right eye is allowed to see strips from right eye images only, which gives a viewer a 3D perception (depth of a 3D scene).
Individual images from various viewpoints are interleaved as in Equation 10 to form a multi-view 3D image for the autostereoscopic 3D TV displays:
MathFigure 10
Figure PCTKR2009001834-appb-M000010
wherein I AutostereoView stands for the pixel value of a multi-view 3D image, I 0 to I n-1 stand for the pixel values of n sub-images from various viewpoints to form a multi-view 3D image for the autostereoscopic 3D TV displays. To be specific, I 0 is the original left image, I n-1 is the original right image and I 1 to I n-2 stand for n-2 intermediate-views from various viewpoints between the original left and right images. In Equation 10, X%n represents a remainder of division of the sub-images n by the horizontal axis X in steps S810 to S860,
Referring to Fig. 8, in order to form a multi-view 3D image for the autostereoscopic 3D TV displays, column pixels of n views from various viewpoints are interleaved in sequence from I 0 to I n-1 . Multi-view 3D image content observed by the viewer depends upon a position of the viewer with respect to the autostereoscopic 3D TV displays screen. Due to the autostereoscopic 3D TV displays screen (lenticular or Parallax barrier), the left eye of the viewer receives a column pixels that is different from what the right eye thereof receives, which gives the viewer a 3D perception (depth of a 3D scene).
Fig. 9 illustrates a block diagram of a parallel processing mechanism for a fast multi-view image synthesis using the apparatus in Fig. 3. As shown in Fig. 9, the disparity map generation module 100 outputs one line pixel value of a left image disparity map d LR . Further, one line pixel values of the left and right images at the same time are also produced. For parallel processing to generate 1 st to N th views from various viewpoints, the number of the intermediate-view generation module 200 is N-2. After receiving one line pixel values of the left image, the right image and the left image disparity map d LR , each of the N-2 intermediate-view generation modules outputs one line pixel value from a viewpoint thereof. Here, the 1 st intermediate-view is the left image and the N th intermediate-view is the right image.
The multi-view 3D image generation module 300 receives one line pixel values of the 1 st to N th intermediate-views from the intermediate-view generation modules 200, and outputs a multi-view 3D image for autostereoscopic 3D TV displays to give user a 3D perception. Since the respective 1 st to N th intermediate-views are produced line by line, the fast multi-view image synthesis method using disparity map can be processed in parallel. That is, the left and right images can be synthesized a multi-view 3D image for the autostereoscopic 3D TV displays in parallel.
While the invention has been shown and described with respect to the embodiments, it will be understood by those skilled in the art that various changes and modification may be made without departing from the scope of the invention as defined in the following claims.

Claims (10)

  1. A fast multi-view three-dimensional image synthesis apparatus, comprising:
    a disparity map generation module for generating a left image disparity map by using left and right image pixel data;
    intermediate-view generation modules for generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map; and
    a multi-view three-dimensional image generation module for generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data.
  2. The apparatus of claim 1, wherein the left and right image pixel data are on an identical epipolar line.
  3. The apparatus of claim 1, wherein the disparity map generation module generates the left image disparity map based on belief propagation based algorithm.
  4. The apparatus of claim 1, wherein each of the intermediate-view generation module includes:
    a right image disparity map generation unit for generating a rough right image disparity map by using the left image disparity map;
    an occluded region compensation unit for generating a right image disparity map by removing occluded regions from the rough right image disparity map; and
    an intermediate-view generation unit for generating the intermediate-view pixel data from the different view points by using the right image disparity map generated by the occluded region compensation unit.
  5. The apparatus of claim 1, wherein the multi-view three-dimensional image generation module generates the multi-view three-dimensional image pixel data by interweaving the intermediate-view pixel data from the different view points.
  6. A fast multi-view three-dimensional image synthesis method, comprising:
    generating a left image disparity map by using left and right image pixel data;
    generating intermediate-view pixel data from different view points by using the left and right image pixel data and the left image disparity map; and
    generating multi-view three-dimensional image pixel data by using the left image pixel data, the right image pixel data and intermediate-view pixel data.
  7. The method of claim 6, wherein said generating the intermediate-view pixel data includes:
    initializing the intermediate-view pixel data;
    determining a first intermediate-view pixel data mapped from the left image pixel data by using the left image disparity map;
    determining a right image disparity map mapped from the left image pixel data by using the left image disparity map;
    determining a second intermediate-view pixel data mapped from the right image pixel data by using the right image disparity map;
    determining whether a desired intermediate-view is near to the left image pixel data or near to the right image pixel data; and
    combining the first and second intermediate-view pixel data to generate the intermediate-view pixel data.
  8. The method of claim 7, wherein said determining the right image disparity includes:
    initializing the right image disparity map;
    determining pixel values of the right image disparity map mapped from the left image disparity map by using the left image disparity map; and
    compensating an occluded region of the right image disparity map by using pixel values of pixels neighboring pixels whose pixel values have been determined.
  9. The method of claim 8, wherein said compensating the occluded region includes:
    storing a forward neighbor pixel value of a pixel forwardly neighboring a pixel whose pixel value has been determined;
    storing a backward neighbor pixel value of a pixel backwardly neighboring a pixel whose pixel value has been determined;
    comparing the forward and backward neighbor pixel value;
    selecting a smaller pixel value between the forward and backward neighbor pixel values; and
    filling a pixel value of a pixel in the occluded region with the selected pixel value.
  10. The method of claim 6, wherein said generating the intermediate-view pixel data from the different viewpoints is performed in parallel.
PCT/KR2009/001834 2008-04-10 2009-04-09 Fast multi-view three-dimensinonal image synthesis apparatus and method Ceased WO2009125988A2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2011503910A JP2011519209A (en) 2008-04-10 2009-04-09 Apparatus and method for synthesizing high-speed multi-view 3D stereoscopic video for 3D stereoscopic television without glasses
EP09731429.8A EP2263383A4 (en) 2008-04-10 2009-04-09 APPARATUS AND METHOD FOR THREE-DIMENSIONAL IMAGE SYNTHESIS WITH MULTIPLE QUICK VIEWS
US12/923,820 US20110026809A1 (en) 2008-04-10 2010-10-08 Fast multi-view three-dimensional image synthesis apparatus and method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2008-0033176 2008-04-10
KR1020080033176A KR100950046B1 (en) 2008-04-10 2008-04-10 High Speed Multiview 3D Stereoscopic Image Synthesis Apparatus and Method for Autostereoscopic 3D Stereo TV

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US12/923,820 Continuation US20110026809A1 (en) 2008-04-10 2010-10-08 Fast multi-view three-dimensional image synthesis apparatus and method

Publications (2)

Publication Number Publication Date
WO2009125988A2 true WO2009125988A2 (en) 2009-10-15
WO2009125988A3 WO2009125988A3 (en) 2011-03-24

Family

ID=41162401

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2009/001834 Ceased WO2009125988A2 (en) 2008-04-10 2009-04-09 Fast multi-view three-dimensinonal image synthesis apparatus and method

Country Status (5)

Country Link
US (1) US20110026809A1 (en)
EP (1) EP2263383A4 (en)
JP (1) JP2011519209A (en)
KR (1) KR100950046B1 (en)
WO (1) WO2009125988A2 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011109898A1 (en) * 2010-03-09 2011-09-15 Berfort Management Inc. Generating 3d multi-view interweaved image(s) from stereoscopic pairs
WO2011123174A1 (en) * 2010-04-01 2011-10-06 Thomson Licensing Disparity value indications
EP2393298A1 (en) * 2010-06-03 2011-12-07 Zoltan Korcsok Method and apparatus for generating multiple image views for a multiview autostereoscopic display device
WO2013025149A1 (en) * 2011-08-15 2013-02-21 Telefonaktiebolaget L M Ericsson (Publ) Encoder, method in an encoder, decoder and method in a decoder for providing information concerning a spatial validity range
EP2765775A1 (en) * 2013-02-06 2014-08-13 Koninklijke Philips N.V. System for generating intermediate view images
EP2765774A1 (en) * 2013-02-06 2014-08-13 Koninklijke Philips N.V. System for generating an intermediate view image
EP2822279B1 (en) * 2013-07-05 2019-10-16 Dolby Laboratories Licensing Corporation Autostereo tapestry representation

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2475730A (en) * 2009-11-27 2011-06-01 Sony Corp Transformation of occluding objects in 2D to 3D image generation
KR101666019B1 (en) 2010-08-03 2016-10-14 삼성전자주식회사 Apparatus and method for generating extrapolated view
KR20120063984A (en) * 2010-12-08 2012-06-18 한국전자통신연구원 Multi-viewpoint image generating method and apparatus thereof
KR101088634B1 (en) 2011-04-11 2011-12-06 이종오 Stereoscopic display panel, stereoscopic display device and stereoscopic display method
JP2013026826A (en) * 2011-07-21 2013-02-04 Sony Corp Image processing method, image processing device and display device
US8817073B2 (en) * 2011-08-12 2014-08-26 Himax Technologies Limited System and method of processing 3D stereoscopic image
US9451232B2 (en) 2011-09-29 2016-09-20 Dolby Laboratories Licensing Corporation Representation and coding of multi-view images using tapestry encoding
AU2012318854B2 (en) * 2011-10-05 2016-01-28 Bitanimate, Inc. Resolution enhanced 3D video rendering systems and methods
BR112013017322A2 (en) 2011-11-11 2017-03-01 Sony Corp device and method of transmission, and method of reception
TWI478095B (en) 2012-02-07 2015-03-21 Nat Univ Chung Cheng Check the depth of mismatch and compensation depth error of the
HUE045628T2 (en) * 2012-04-20 2020-01-28 Affirmation Llc Systems and methods for real-time conversion of video into three-dimensions
US9076249B2 (en) 2012-05-31 2015-07-07 Industrial Technology Research Institute Hole filling method for multi-view disparity maps
RU2012138174A (en) * 2012-09-06 2014-03-27 Сисвел Текнолоджи С.Р.Л. 3DZ TILE FORMAT DIGITAL STEREOSCOPIC VIDEO FLOW FORMAT METHOD
KR20140039649A (en) 2012-09-24 2014-04-02 삼성전자주식회사 Multi view image generating method and multi view image display apparatus
KR101956353B1 (en) * 2012-12-05 2019-03-08 삼성전자주식회사 Image processing Appratus and method for generating 3D image thereof
US9171373B2 (en) * 2012-12-26 2015-10-27 Ncku Research And Development Foundation System of image stereo matching
US9565414B2 (en) * 2013-05-24 2017-02-07 Disney Enterprises, Inc. Efficient stereo to multiview rendering using interleaved rendering
WO2015013851A1 (en) * 2013-07-29 2015-02-05 北京大学深圳研究生院 Virtual viewpoint synthesis method and system
KR20150041225A (en) * 2013-10-04 2015-04-16 삼성전자주식회사 Method and apparatus for image processing
US9407896B2 (en) 2014-03-24 2016-08-02 Hong Kong Applied Science and Technology Research Institute Company, Limited Multi-view synthesis in real-time with fallback to 2D from 3D to reduce flicker in low or unstable stereo-matching image regions
KR102240564B1 (en) * 2014-07-29 2021-04-15 삼성전자주식회사 Apparatus and method for rendering image
US9412034B1 (en) * 2015-01-29 2016-08-09 Qualcomm Incorporated Occlusion handling for computer vision
US11315328B2 (en) * 2019-03-18 2022-04-26 Facebook Technologies, Llc Systems and methods of rendering real world objects using depth information
JP7218801B2 (en) * 2019-05-31 2023-02-07 日本電信電話株式会社 Image generation device, image generation method, program
JP7329795B2 (en) * 2019-10-18 2023-08-21 日本電信電話株式会社 Image supply device, image supply method, display system and program
JP7329794B2 (en) * 2019-10-18 2023-08-21 日本電信電話株式会社 Image supply device, image supply method, display system and program
CN113132706A (en) * 2021-03-05 2021-07-16 北京邮电大学 Controllable position virtual viewpoint generation method and device based on reverse mapping

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070296721A1 (en) 2004-11-08 2007-12-27 Electronics And Telecommunications Research Institute Apparatus and Method for Producting Multi-View Contents
KR20080000149A (en) 2006-06-26 2008-01-02 광주과학기술원 Method and apparatus for processing multi-image

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5530774A (en) * 1994-03-25 1996-06-25 Eastman Kodak Company Generation of depth image through interpolation and extrapolation of intermediate images derived from stereo image pair using disparity vector fields
JP3157384B2 (en) * 1994-06-20 2001-04-16 三洋電機株式会社 3D image device
JP3826236B2 (en) * 1995-05-08 2006-09-27 松下電器産業株式会社 Intermediate image generation method, intermediate image generation device, parallax estimation method, and image transmission display device
JPH0962844A (en) * 1995-08-29 1997-03-07 Oki Electric Ind Co Ltd Phase difference detection device for three-dimensional picture and detection of phase difference in three-dimensional picture
JP3769850B2 (en) * 1996-12-26 2006-04-26 松下電器産業株式会社 Intermediate viewpoint image generation method, parallax estimation method, and image transmission method
MY124160A (en) * 1997-12-05 2006-06-30 Dynamic Digital Depth Res Pty Improved image conversion and encoding techniques
JP2001346226A (en) * 2000-06-02 2001-12-14 Canon Inc Image processing apparatus, three-dimensional photo print system, image processing method, three-dimensional photo print method, and medium recording processing program
JP2003044880A (en) * 2001-07-31 2003-02-14 Canon Inc Three-dimensional image forming apparatus, three-dimensional image forming method, program, and storage medium
KR100433625B1 (en) * 2001-11-17 2004-06-02 학교법인 포항공과대학교 Apparatus for reconstructing multiview image using stereo image and depth map
JP4238586B2 (en) * 2003-01-30 2009-03-18 ソニー株式会社 Calibration processing apparatus, calibration processing method, and computer program
JP2004282217A (en) * 2003-03-13 2004-10-07 Sanyo Electric Co Ltd Multiple-lens stereoscopic video image display apparatus
JP4164670B2 (en) * 2003-09-29 2008-10-15 独立行政法人産業技術総合研究所 Model creation device, information analysis device, model creation method, information analysis method, and program
KR100517517B1 (en) * 2004-02-20 2005-09-28 삼성전자주식회사 Method for reconstructing intermediate video and 3D display using thereof
KR100672925B1 (en) * 2004-04-20 2007-01-24 주식회사 후후 Multi-view Image Generation Method Using Adaptive Parallax Estimation
JP4539203B2 (en) * 2004-07-15 2010-09-08 ソニー株式会社 Image processing method and image processing apparatus
ES2323287T3 (en) * 2005-01-12 2009-07-10 Koninklijke Philips Electronics N.V. PERCEPTION OF DEPTH.
KR100636785B1 (en) * 2005-05-31 2006-10-20 삼성전자주식회사 Multi-view stereoscopic imaging system and compression and decompression method applied to it
EP2120209A1 (en) * 2008-05-14 2009-11-18 Thomson Licensing Apparatus for evaluating images from a multi camera cystem, multi camera system and process for evaluating
WO2010048632A1 (en) * 2008-10-24 2010-04-29 Real D Stereoscopic image format with depth information

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070296721A1 (en) 2004-11-08 2007-12-27 Electronics And Telecommunications Research Institute Apparatus and Method for Producting Multi-View Contents
KR20080000149A (en) 2006-06-26 2008-01-02 광주과학기술원 Method and apparatus for processing multi-image

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP2263383A4

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011109898A1 (en) * 2010-03-09 2011-09-15 Berfort Management Inc. Generating 3d multi-view interweaved image(s) from stereoscopic pairs
WO2011123174A1 (en) * 2010-04-01 2011-10-06 Thomson Licensing Disparity value indications
CN102823260A (en) * 2010-04-01 2012-12-12 汤姆森特许公司 Parallax value indication
CN102823260B (en) * 2010-04-01 2016-08-10 汤姆森特许公司 Parallax value indication
EP2393298A1 (en) * 2010-06-03 2011-12-07 Zoltan Korcsok Method and apparatus for generating multiple image views for a multiview autostereoscopic display device
WO2013025149A1 (en) * 2011-08-15 2013-02-21 Telefonaktiebolaget L M Ericsson (Publ) Encoder, method in an encoder, decoder and method in a decoder for providing information concerning a spatial validity range
US9497435B2 (en) 2011-08-15 2016-11-15 Telefonaktiebolaget Lm Ericsson (Publ) Encoder, method in an encoder, decoder and method in a decoder for providing information concerning a spatial validity range
WO2014121860A1 (en) * 2013-02-06 2014-08-14 Koninklijke Philips N.V. System for generating an intermediate view image
WO2014122012A1 (en) * 2013-02-06 2014-08-14 Koninklijke Philips N.V. System for generating intermediate view images
CN104982033A (en) * 2013-02-06 2015-10-14 皇家飞利浦有限公司 System for generating intermediate view images
EP2765774A1 (en) * 2013-02-06 2014-08-13 Koninklijke Philips N.V. System for generating an intermediate view image
EP2765775A1 (en) * 2013-02-06 2014-08-13 Koninklijke Philips N.V. System for generating intermediate view images
EP2954674B1 (en) 2013-02-06 2017-03-08 Koninklijke Philips N.V. System for generating an intermediate view image
CN104982033B (en) * 2013-02-06 2017-11-24 皇家飞利浦有限公司 System for generating medial view image
US9967537B2 (en) 2013-02-06 2018-05-08 Koninklijke Philips N.V. System for generating intermediate view images
TWI630814B (en) * 2013-02-06 2018-07-21 皇家飛利浦有限公司 System for generating an intermediate view image
EP2822279B1 (en) * 2013-07-05 2019-10-16 Dolby Laboratories Licensing Corporation Autostereo tapestry representation

Also Published As

Publication number Publication date
KR100950046B1 (en) 2010-03-29
US20110026809A1 (en) 2011-02-03
EP2263383A2 (en) 2010-12-22
JP2011519209A (en) 2011-06-30
KR20090107748A (en) 2009-10-14
WO2009125988A3 (en) 2011-03-24
EP2263383A4 (en) 2013-08-21

Similar Documents

Publication Publication Date Title
WO2009125988A2 (en) Fast multi-view three-dimensinonal image synthesis apparatus and method
WO2017065517A1 (en) 3d display apparatus and control method thereof
WO2016021925A1 (en) Multiview image display apparatus and control method thereof
WO2012064010A4 (en) Image conversion apparatus and display apparatus and methods using the same
US20090315982A1 (en) Arrangement and method for the recording and display of images of a scene and/or an object
US20070165942A1 (en) Method for rectifying stereoscopic display systems
WO2015037796A1 (en) Display device and method of controlling the same
WO2013081429A1 (en) Image processing apparatus and method for subpixel rendering
WO2013081435A1 (en) 3d image display device and method
KR20110124473A (en) 3D image generating device and method for multi-view image
WO2011004976A2 (en) Method and apparatus for the high-speed calibration and rectification of a stereo camera
WO2010140332A1 (en) Stereoscopic image display apparatus
CN1706201A (en) Stereoscopic video providing method and stereoscopic video display
WO2011005056A2 (en) Image output method for a display device which outputs three-dimensional contents, and a display device employing the method
WO2011129488A1 (en) Parallel axis stereoscopic camera
WO2022045779A1 (en) Restoration of the fov of images for stereoscopic rendering
WO2014051309A1 (en) Stereo matching apparatus using image property
KR101377960B1 (en) Device and method for processing image signal
WO2012086912A1 (en) Method for converting two-dimensional image into stereo image
WO2016195167A1 (en) Content conversion method, content conversion apparatus, and program for multi-layered hologram
TWI462569B (en) 3d video camera and associated control method
WO2019103240A1 (en) System and method for multi-camera video distribution and combination through ip
WO2012165717A1 (en) Apparatus for generating stereo 3d image using asymmetric, two-camera module and method for same
WO2012169698A1 (en) Device and method for generating a stereoscopic augmented reality image by using a stereoscopic 3d image and gpu
WO2012074294A2 (en) Image processing device using an energy value, and image processing method and display method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09731429

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2009731429

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2011503910

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE