Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20092—Interactive image processing based on input by user
  • G06T2207/20101—Interactive definition of point of interest, landmark or seed

Definitions

• the invention relates to a device for generating a depth map based on a two- dimensional [2D] image for enabling three-dimensional [3D] rendering of the 2D image.
• the invention further relates to a method for generating a depth map based on a 2D image for enabling 3D rendering of the 2D image, and to a computer program product comprising instructions for causing a processor system to perform the method.
• display devices such as televisions, digital photo frames, tablets and smartphones comprise 3D displays to provide a user with a perception of depth when viewing content on such a device.
• 3D displays may, either by themselves or together with glasses worn by the user, provide the user with different images in each eye so as to provide the user a perception of depth based on stereoscopy.
• 3D displays typically require content which contains depth information.
• the depth information may be provided implicitly in the 3D content.
• the depth information is provided by the differences between a left image and a right image of the stereo content.
• the depth information may also be provided explicitly in the 3D content.
• the depth information is provided by a depth map which may comprise depth values, disparity values and/or parallactic shift values, with all of said values being indicative of the distance that objects within the image have towards the camera.
• Conversion to 3D may comprise generating a depth map for a 2D image, e.g., for each 2D image of a 2D video.
• the depth map may be generated automatically.
• a device may estimate the distance that objects within the 2D image have towards the camera and, based thereon, generate a depth map for the 2D image.
• the user may also manually generate the depth map.
• a digital photo frame may offer a consumer the possibility to draw a depth map using a touch sensitive surface of the digital photo frame.
• a software tool running on a workstation may offer a professional user the possibility to add depth to 2D images by drawing depth maps using a digital pen.
• the depth map may also be generated semi- automatically, in that the device generates the depth map based on user input.
• the user provides depth information to the device, such as a relative depth-order between objects. Based on the depth information, the device then generates the depth map.
• a semi-automatically generated depth map may be of higher quality than a fully automatically generated depth map.
• the user may be inconvenienced by having to provide depth information to the device.
• a publication titled "Depth Director: A System for Adding Depth to Movies" by B. Ward et al., Computer Graphics and Applications, IEEE, Vol. 31 , Issue 1 , pp. 36-48, Jan.- Feb. 201 1 describes an interactive system for converting existing 2D footage to stereo.
• the system is said to provide automated depth assignment for scenes with a moving camera using structure from motion. It is said that the user is free to visually emphasize or de- emphasize parts of the scene. This is accomplished through a stroke-based interface along with more complex depth templates to iteratively refine the computer vision results.
• a depth template consists of a triangle mesh, which can be translated, scaled and rotated in 3D to fit a selected region. Examples are a planar surface, sphere, cylinder, box, car and face templates. It is said that the planar templates can be used to create a ground plane.
• a drawback of the above interactive system is that it is too difficult for a novice user to convert existing 2D footage to stereo using said interactive system.
• One of the objects of the invention is to provide a device and/or method for enabling a novice user to more easily generate a depth map for a 2D image.
• a first aspect of the invention provides a device for generating a depth map based on a two-dimensional [2D] image for enabling three-dimensional [3D] rendering of the 2D image, comprising:
• a display output for displaying a marker in the 2D image, the marker corresponding to a component of a geometric template, the geometric template comprising depth information for defining a global depth profile of a reference scene;
• a user input for enabling a user to position the marker in the 2D image so as to enable the user to match the geometric template to an actual scene shown in the 2D image by positioning the component with respect to the 2D image;
• a depth map generator for establishing the global depth profile as the depth map based on a position of the marker in the 2D image.
• a further aspect of the invention provides a method for generating a depth map based on a two-dimensional [2D] image for enabling three-dimensional [3D] rendering of the 2D image, comprising: displaying a marker in the 2D image, the marker corresponding to a component of a geometric template, the geometric template comprising depth information for defining a global depth profile of a reference scene;
• a further aspect of the invention provides a computer program product comprising instructions for causing a processor system to perform the method.
• a depth map generally comprises depth values which correspond to the distances that portions of the 2D image have, or estimated to have, towards the camera.
• the term camera refers to a real or a virtual camera used in capturing or generating the 2D image.
• a corresponding portion of the depth map exists which reflects its distance towards the camera.
• the depth map may be used for 3D rendering of the 2D image, e.g., using techniques generally known as view rendering.
• one or more additional 2D images are generated by displacing portions of the 2D image based on the depth map.
• the 2D image is used in the generation of the depth map as follows.
• the device comprises a display output which enables a marker to be displayed to the user.
• the marker is displayed within the 2D image.
• the marker may be overlaid over the 2D image.
• the user is thus presented with a display of the 2D image and the marker.
• the display output may also enable the 2D image to be displayed.
• the 2D image may be displayed by a different device or component of the device.
• the display output may be connectable to a display or may comprise a display.
• the marker corresponds to a component of a geometric template, i.e., a mathematically defined shape made up of one or more components.
• the geometry of the geometric template is defined by the relative 2D positions of the one or more components.
• the position of the geometric template is defined by the absolute 2D positions of the one or more components.
• the geometric template comprises depth information. For example, one or more depth values may be assigned to the one or more components of the geometric template.
• the geometric template together with the depth information defines a global depth profile of a reference scene.
• the depth information provides the depth of the global depth profile and the shape and/or position of the geometric template provides the shape and/or the position of the global depth profile.
• the scene is a reference scene in that it constitutes an instance of a certain type of scene, e.g., a landscape scene or an indoor scene.
• the term scene refers to the combination of a space and the objects therein, e.g., people. Examples of spaces are, e.g., a room, a narrow alley, a flat landscape, etc.
• the global depth profile is indicative how depth is globally distributed within the scene.
• the term global refers to the depth profile giving a general overview of the depth of the scene, i.e., an initial approximation, omitting many or all details.
• the global depth profile corresponds to the depth profile of the space rather than the objects therein since this typically provides the general overview of the depth of the scene.
• the device comprises a user input which enables the user to position the marker in the 2D image, e.g., by receiving appropriate instructions from a user input means which is operated by the user.
• the positioning of the marker is in 2D, i.e., yields a 2D position.
• the user establishes a position of the component of the geometric template with respect to the 2D image.
• the positioning of the component is also in 2D, i.e., yields a 2D position.
• the position of the component of the geometric template affects the position and/or shape of the geometric template as a whole. The user can therefore affect the position and/or shape of the geometric template with respect to the 2D image by appropriately positioning the marker in the 2D image.
• the user can therefore adjust the global depth profile by adjusting the position and/or shape of the geometric template.
• the user adjusts the global depth profile.
• the user may adjust the marker such that the geometric template, and thus the global depth profile defined by the geometric template, matches an actual scene shown in the 2D image.
• the user may thus compensate for differences between the reference scene and the actual scene by appropriately positioning the marker.
• the user may compensate for differences in composition, such as a height of the horizon in both scenes, a location of a vanishing point, etc.
• the device comprises a depth map generator.
• the depth map generator generates the depth map for the 2D image based on the position of the component of the geometric template in the 2D image. Said position is provided by the position of the marker.
• the depth map generator generates the depth map by establishing the global depth profile defined by the geometric template. The global depth profile may have been adjusted by the user by appropriately positioning the marker in the 2D image. By taking the position of the marker into account, the depth map generator takes into account such adjustments, i.e., establishes a global depth profile which is adjusted for the actual scene.
• the present invention is partially based on the recognition that it is difficult for a novice user to establish a depth map for a 2D image, even when using an interactive system.
• a reason for this is that known interactive systems consider all elements of a scene as separate objects to which the user has to assign a depth. Disadvantageously, the user may be unwilling to assign a depth to all of the elements of the scene.
• the inventors have recognized that it is of particular importance to correctly establish the global depth of a scene. A reason for this the following.
• a user may be primarily focused on an object within the scene. In the depth perception of the object, its absolute depth plays an important role. However, the depth of an object is frequently judged with respect to the space the object is placed in.
• the so-termed relative depth of the object also plays an important role.
• the global depth plays an important role in establishing a global impression of the depth of a scene.
• the present invention enables the user to conveniently establish a global depth profile for the 2D image by the device providing a global depth profile of a reference scene and enabling the user to match the global depth profile to an actual scene shown in the 2D image by appropriately positioning a marker.
• the device Upon positioning the marker, the device then automatically generates a depth map by establishing the possibly adjusted global depth profile as the depth map.
• the user does not need to assign a depth to the individual elements of the space in which the object is placed, e.g., a floor, walls and ceiling. Rather, the device enables the user to generate the depth map for all of the space in a single step.
• the user does not need to position the marker in 3D so as to establish a depth of the 2D image. Rather, the depth information is provided separately.
• the term map refers to data arranged in rows and columns.
• the adjective depth is to be understood as being indicative of the depth of portions of the 2D image to the camera. Therefore, the depth map may be constituted by depth values, but also by, e.g., disparity values or parallactic shift values. Essentially, the depth map may therefore constitute a disparity map or a parallactic shift map.
• disparity refers to a difference in position of an object when perceived with a left eye or a right eye of the user.
• parallactic shift refers to a displacement of the object between two views so as to provide said disparity to the user. Disparity and parallactic shift are generally negatively correlated with distance or depth. Device and methods for conversion between all of the above types of maps and/or values are known.
• the geometric template defines a depth gradient
• the depth map generator is arranged for establishing the depth gradient in the depth map based on the position of the marker.
• a depth gradient is well suited as a global depth profile since the depth in many scenes is globally distributed in a gradient-like manner.
• the depth in many landscape scenes may be represented as a gradient towards the horizon, and the sky having a depth corresponding to a depth at an end of the gradient.
• the position of the marker constitutes a starting point or an end point of the depth gradient.
• a component of the geometric template defines a starting point or an end point of the depth gradient, and the marker corresponds to said component. The user can therefore conveniently adjust a position and/or shape of the depth gradient by appropriately positioning the marker in the 2D image.
• the position of the marker constitutes a vanishing point in the global depth profile.
• a component of the geometric template defines a vanishing point, and the marker corresponds to said component.
• a vanishing point is well suited as a component of the global depth profile since the depth in many scenes converges towards a vanishing point in the scene.
• the global depth profile is well applicable to many scenes. The user can conveniently adjust the position of the vanishing point by appropriately positioning the marker in the 2D image.
• the marker is one of a plurality of markers
• the plurality of markers correspond to a plurality of components together defining the global depth profile
• the user input is arranged for enabling the user to individually position each of the plurality of markers for adjusting the global depth profile to fit the actual scene in the 2D image.
• the user can therefore adjust multiple components of the global depth profile by appropriately positioning multiple markers. By adjusting said components individually, the user can better match the global depth profile to the actual scene shown in the 2D image.
• the display output is arranged for displaying one or more connections between the plurality of markers
• the depth map generator is arranged for establishing one or more edges of the global depth profile based on a position of the one or more connections in the 2D image.
• the user is thus shown one or more connections between the markers, e.g., as lines.
• the one or more connections correspond to edges of the global depth profile and thus result in depth edges or depth transitions being generated in the depth map. By showing these one or more connections, the user can better match the global depth profile to the actual scene shown in the 2D image.
• a reason for this is that the user can, in addition to or as an alternative for positioning the markers with respect to characteristic points in the 2D image, also position the markers so as to match the one or more connections to characteristic structures in the 2D image.
• characteristic structures may be edges or transitions in the 2D image.
• the user can more accurately match the global depth profile to the actual scene shown in the 2D image.
• the plurality of markers and the one or more connections correspond to a horizon of the global depth profile.
• a horizon typically is a characteristic structure in a scene.
• a horizon is well suited as a component of the global depth profile as a horizon constitutes an edge or a transition between the depth gradient of a ground plane and the homogenous depth of the sky.
• the user can thus match the global depth profile to the actual scene shown in the 2D image by matching the markers and the one or more connections to the horizon.
• the user can more conveniently match the global depth profile to the actual scene shown in the 2D image.
• the plurality of markers and the one or more connections form at least one of: a line segment, a polyline or a polygon.
• the user input is arranged for enabling the user to add and/or remove markers to and/or from the plurality of markers.
• the user can thus change the number of markers.
• the user can adapt the number of markers to the actual scene at hand, i.e., add markers if this allows better matching the geometric template to the actual scene shown in the 2D image, or remove markers if one or more markers are not needed to match the geometric template to the actual scene shown in the 2D image.
• the user input is arranged for enabling the user to select a foreground object in the 2D image, the foreground object constituting an object in the 2D image having a foreground depth which is insufficiently represented by the global depth profile;
• the depth map generator is arranged for i) establishing the foreground depth of the foreground object, and ii) based on the foreground depth, including a depth representation of the foreground object in the depth map.
• the user input enables the user to select an object in the 2D image, e.g., by pointing to the object or clicking on the object.
• the object constitutes a foreground object.
• the background is typically provided by the space in which the objects in the scene are placed, with the foreground object covering or occluding a part of the space.
• the foreground object is insufficiently represented by the global depth profile in that the global depth profile contains depth values at the corresponding portion of the depth map which do not, or not accurately, reflect those of the foreground object, and may rather reflect those of the surrounding space if the foreground object would be absent.
• the depth map generator establishes a depth of the foreground object, and uses the depth to include a depth representation of the foreground object in the corresponding portion of the depth map.
• the user can, in addition to conveniently establishing the global depth of a scene, also assign depth to a foreground object of the scene.
• the user can conveniently and quickly obtain a suitable depth map for the 2D image by first establishing the global depth profile as the depth map, thereby obtaining a depth map for the space, and then selecting one or more foreground objects within the scene that are to be represented in the depth map.
• the depth map generator is arranged for establishing the foreground depth based on a depth of the global depth profile at a boundary with the depth representation of the foreground object.
• a foreground object is frequently connected with its surrounding space, e.g., by standing on a floor or hanging from a ceiling.
• the global depth profile is likely to represent the depth of the space.
• the depth of the foreground object can be determined at least to a large extend from the depth of the global depth profile at a boundary with the portion of the depth map corresponding to the foreground object, since the depth of the space equals the depth of the foreground object here.
• the foreground depth can be automatically determined.
• the user input is arranged for enabling the user to establish that the foreground object is standing or hanging within the actual scene shown in the 2D image
• the depth map generator is arranged for establishing a point on the boundary based on said standing or hanging.
• a hanging object is typically connected to its surrounding space at its top.
• a standing object is typically connected to the space at its bottom.
• the foreground object is a person
• the feet of the person form the person's connection to the ground.
• the user input is arranged for enabling the user to provide the foreground depth to the depth map generator.
• the user can manually determine the depth of the foreground object by providing the foreground depth to the depth map generator, e.g., by typing a value into a text box or moving a depth slider to a certain position.
• Fig. 1 shows a device for generating a depth map
• Fig. 2 shows a method for generating the depth map
• Fig. 3 shows a computer program product for performing the method
• Fig. 4a shows a two-dimensional [2D] image
• Fig. 4b shows a geometric template comprising depth information
• Fig. 4c shows a global depth profile for a reference scene, the global depth profile being defined by the geometric template
• Fig. 4d shows a plurality of markers being displayed in the 2D image, the plurality of markers corresponding to the components of the geometric template
• Fig. 4e shows the user positioning the plurality of markers in the 2D image to match the geometric template to an actual scene shown in the 2D image;
• Fig. 4f shows the global depth profile being established as a depth map based on the positioning of the plurality of markers by the user
• Fig. 5a shows another geometric template comprising depth information
• Fig. 5b shows the user positioning a plurality of markers in a 2D image to match the geometric template to an actual scene shown in the 2D image;
• Fig. 5c shows a global depth profile being established as a depth map based on the positioning of the plurality of markers by the user;
• Fig. 5d shows the user selecting a foreground object in the 2D image
• Fig. 5e shows a depth representation of the foreground object being included in the depth map based on the foreground object standing in the actual scene
• Fig. 5f shows a depth representation of another foreground object being included in the depth map based on said foreground object hanging in the actual scene
• Fig. 6a shows a geometric template comprising depth information, the geometric template defining a global depth profile comprising a vanishing point;
• Fig. 6b shows the user positioning a marker in the 2D image to match the vanishing point to an actual scene shown in the 2D image.
• Fig. 1 shows a device 100 for generating a depth map 142 based on a two- dimensional [2D] image 144.
• the device 100 comprises a display output 160 for displaying a marker 320 in the 2D image 144.
• the display output 160 is shown to provide display data 162 to a display 1 10, the display 1 10 being connected to the display output 160 of the device 100.
• the display data 162 may comprise the 2D image 144 itself.
• the device 100 may comprise the display 1 10.
• the device 100 further comprises a user input 120 for enabling a user to position the marker 320 in the 2D image 144.
• the user input 120 is shown to receive position data 132 from a user interface device 130 such as a computer mouse, keyboard, touch screen, etc.
• the device 100 further comprises a depth map generator 140 for generating the depth map 142 based on a position 122 of the marker in the 2D image 144.
• the depth map generator 140 is shown to receive the position 122 of the marker 320 from the user input 120.
• the marker 320 corresponds to a component of a geometric template.
• the geometric template comprises depth information for defining a global depth profile of a reference scene.
• the user positions the marker 320 so as to match the geometric template to an actual scene shown in the 2D image 144. This is enabled since the marker 320 corresponds to a component of the geometric template, and therefore by positioning the marker 320 the user also positions said component with respect to the 2D image 144. As a result of the positioning, a position 122 of the marker 320 is obtained.
• the depth map generator 140 establishes the global depth profile as the depth map 142 based on the position 122 of the marker 320 in the 2D image 144.
• a depth map 142 is obtained for the 2D image 144 which enables three-dimensional [3D] rendering of the 2D image 144.
• the depth map 142 may be together with the 2D image 144 be sent to a 3D display which performs the 3D rendering using techniques such as view rendering, which are known per se from the technical field of 3D display.
• the display 1 10 may be a 3D display to provide a preview of the 3D rendering.
• the display 1 10 may also be a 2D display.
• the geometric template may be chosen by the user from a plurality of geometric templates.
• the choice of the user may be determined by which of the plurality of geometric templates best matches the actual scene shown in the 2D image 144.
• the device 100 may indicate to which reference scene a geometric template corresponds.
• the user may select a geometric template which also corresponds to a landscape scene.
• Fig. 2 shows a method 200 for generating a depth map based on a two- dimensional [2D] image for enabling three-dimensional [3D] rendering of the 2D image.
• the method 200 comprises, in a step titled "DISPLAYING MARKER IN 2D IMAGE", displaying 210 a marker in the 2D image, the marker corresponding to a component of a geometric template, the geometric template comprising depth information for defining a global depth profile of a reference scene.
• the method 200 further comprises, in a step titled "ENABLING USER TO POSITION MARKER", enabling 220 a user to position the marker in the 2D image so as to enable the user to match the geometric template to an actual scene shown in the 2D image by positioning the component with respect to the 2D image.
• the method 200 further comprises, in a step titled "ESTABLISHING DEPTH MAP BASED ON POSITION OF MARKER", establishing 230 the global depth profile as the depth map based on a position of the marker in the 2D image.
• the method 200 may correspond to an operation of the device 100. However, the method 200 may also be performed in separation of the device 100.
• Fig. 3 shows a computer readable medium 250 comprising a computer program product 260 for causing a processor system to perform the method according to the present invention.
• the computer program product 260 comprises instructions for the processor system, which, upon execution, cause the processor system to perform the method.
• the computer program product 260 may be comprised on the computer readable medium 250 as a series of machine readable physical marks and/or as a series of elements having different electrical, e.g., magnetic, or optical properties or values.
• Fig. 4a shows a two-dimensional [2D] image 144.
• the 2D image 144 depicts a scene comprising a sailing boat 190 sailing through the sea, with the sea extending towards a horizon and the horizon constituting a boundary between the sea and the sky.
• the sailing boat 190 constitutes an object 190 within a surrounding space of the sea and the sky.
• the sailing boat 190 further constitutes a foreground object 190 in that it covers or occludes part of the space, i.e., part of the sea and the sky is located behind and thus covered by the foreground object 190.
• Fig. 4b shows a geometric template 300 comprising depth information.
• Fig. 4b Changes in the depth information are schematically indicated in Fig. 4b by a change in density of lines, where a high density denotes a large depth and a low density denotes a small depth or close proximity.
• the geometric template 300 may be represented by data which allows a schematic representation as shown in Fig. 4b to be generated, but which rather takes a different form, e.g., a list of coordinates and associated depth information.
• Fig. 4b further shows a plurality of markers 320, 322, i.e., a first marker 320 and a second marker 322, which define a line within the geometric template 300.
• Fig. 4c shows a global depth profile 360 as defined by the geometric template
• the intensities within the global depth profile 360 indicate a degree of depth, i.e., dark areas correspond to a large depth or large distance, and bright areas correspond to a small depth or close proximity.
• the global depth profile 360 is of a reference scene.
• the reference scene can be recognized from the global depth profile 360 in that the reference scene comprises a ground plane slanting towards a horizon, and a sky plane extending upwards from the horizon.
• Fig. 4c also shows the plurality of markers 320, 322 and a connection 340 between said markers, the connection 340 being shown in the form of a dashed line.
• connection 340 corresponds to the horizon in the reference scene in that it constitutes a boundary in the global depth profile 360 between a depth gradient of the ground plane which gradually increases in depth towards the horizon, and the flat depth plane of the sky extending above the horizon.
• the geometric template 300 as shown in Fig. 4b defines the global depth profile 360 as shown in Fig. 4c in that a position of the plurality of markers 320, 322 and the connection 340 indicate where the horizon is located in the global depth profile 360.
• the geometric template 300 may further define the shape of the depth gradient associated with the ground plane, the degrees of depth within the global depth profile 360, etc.
• Fig. 4d shows the plurality of markers 320, 322 being displayed in the 2D image 144.
• the plurality of markers 320, 322 are overlaid on top of the 2D image 144.
• the connection 340 between the plurality of markers 320, 322 is displayed in the 2D image 144.
• Fig. 4d shows the positions of the plurality of markers 320, 322 and the connection 340 initially corresponding to those as shown in Figs 4b and 4c, i.e., their positions are such as if the 2D image 144 were to show the reference scene.
• Fig. 4d further shows a cursor 164 being displayed.
• the cursor 164 may be provided by the display output 160, i.e., may be part of the display data 162.
• the user may move the cursor 164 onscreen by suitably operating the user interface device 130.
• the display output 160 may receive the position data 132 from the user input 120.
• the user may use the cursor 164 to position one or more of the plurality of markers in the 2D image 144, i.e., to reposition the one or more of the plurality of markers with respect to their initial positions.
• the user may reposition the first marker 320 by dragging and dropping the first marker 320 to a new position in the 2D image 144.
• the user may position the plurality of markers 320, 322 such that the geometric template 300 is matched to the actual scene shown in the 2D image.
• the user may be made aware that the plurality of markers 320, 322 are components of the geometric template which represent a horizon in the global depth profile 360.
• the concept of the geometric template may also be hidden to the user. Rather, the user may be made aware that the plurality of markers 320, 322 represents a horizon. Accordingly, the user may position the plurality of markers 320, 322 so as to match the horizon shown in the 2D image 144, i.e., that of the actual scene.
• the user may only need to position one or more of the plurality of markers 320, 322, with the connection 340 being automatically drawn by the display output 160. Hence, it may not be needed to manually position the connection 340.
• alternative means of positioning the plurality of markers 320, 322 may be used.
• the device 100 may be arranged for directly establishing a new marker at a location indicated by the user. Hence, it may not be needed to reposition a marker.
• the user may rather directly establish the initial position of the marker in the 2D image 144.
• the display 1 10 may comprise a touch sensitive surface and the user may establish the initial position of the marker by touching an appropriate location on the touch sensitive surface.
• Fig. 4e shows a result of the positioning of plurality of markers 320, 322.
• Fig. 4f shows the global depth profile 360 being established as a depth map 142 for the 2D image 144 based on the positioning of the plurality of markers 320, 322 by the user.
• the boundary between the depth gradient of the ground plane and the flat depth plane of the sky extending above the horizon is now positioned at a different vertical position, i.e., lower within the global depth profile, as well as at an angle, with the position and angle corresponding to that of the horizon shown in the 2D image 144.
• the depth map 142 may be used for 3D rendering of the 2D image 144.
• the user will perceive the sea slanting towards the horizon, and the sky extending upwards from the horizon.
• the sailing boat 190 since no depth representation of the foreground object, i.e., the sailing boat 190, was explicitly included in the depth map 142, the user will, based on stereoscopy alone, perceive the sailing boat 190 as having a depth of that of the surrounding space. Consequently, the hull of the sailing boat 190 will be perceived as slanting towards the horizon, whereas the sails of the sailing boat 190 will be perceived as extending upwards from the horizon.
• the sailing boat 190 is established at a depth as if it were part of the space of the sea and the sky.
• Figs. 5a through 5f illustrate an optional feature of the present invention which allows a depth representation of a foreground object to be included in the depth map 142.
• Fig. 5a shows another example of a geometric template 300 comprising depth information.
• a change in depth information is schematically indicated by a change in density of lines, i.e., a high density denotes a large depth and a low density denotes a small depth or close proximity.
• Fig. 5a further shows a plurality of markers 320-326 which define a rectangle within the geometric template 300. From Fig. 5a, it can be seen that the geometric template 300 may define a global depth profile of a, e.g., a room, an inside of a box, etc.
• Fig. 5b shows a 2D image 144 depicting a scene of a person standing in a room with a door in a wall at a far side of the room and a lamp hanging from the ceiling.
• the person and the lamp constitute objects within the space provided by the room.
• the person and the lamp further constitute foreground objects in that they each cover or occlude part of the space, i.e., part of the room forms a background with respect to the person and the lamp.
• Fig. 5b further shows the user positioning the plurality of markers 320- 326 in the 2D image 144 to match the geometric template 300 of Fig. 5a to the actual scene shown in the 2D image 144. This may be done using, e.g., drag and drop.
• Fig. 5c shows a result of the user appropriately positioning each of the plurality of markers 320-326 in the 2D image 144, and the device subsequently establishing the global depth profile 300 as the depth map 142 for the 2D image 144 based on the positioning of the plurality of markers 320-326 by the user. It will be appreciated that the floor, ceiling, side walls and far wall of the room may be recognized in the depth map 142.
• the user input 120 of the device 100 may be arranged for enabling the user to select a foreground object 192, 194 in the 2D image 144, the foreground object constituting an object in the 2D image having a foreground depth which is insufficiently represented by the global depth profile 360, and the depth map generator 140 may be arranged for i) establishing the foreground depth of the foreground object, and ii) based on the foreground depth, including a depth representation 196, 198 of the foreground object in the depth map 142.
• Fig. 5d shows the user selecting a foreground object in the 2D image, the foreground object being the person 192.
• the user may select the person 192 by, e.g., moving the cursor 164 over the person 192 in the 2D image 144 and subsequently clicking on the person 192 by suitably operating the user interface device 130.
• the clicking may indicate to the device 100 that the foreground object is located at the position of the cursor 164.
• the device 100 may establish an outline of the foreground object in the 2D image 144 by segmenting the foreground object.
• the depth map generator 140 may therefore segment the person 192 by, e.g., placing a seed for a seed-based segmentation algorithm at a position in the 2D image 144 corresponding to the position of the cursor 164.
• alternative segmentation means or in general alternative ways of establishing the outline of the foreground object, may be used as well.
• the user may manually delineate the foreground object using the cursor 164.
• the depth map generator 140 obtains a position and an outline of the person 192 in the 2D image 144.
• Fig. 5e shows a result of the depth map generator 140 including a depth representation 196 of the person 192 in the depth map 142 which has substantially the position and outline as the person 192 in the 2D image 144, yet comprises depth values instead of luminance and/or chrominance values.
• the depth map generator 140 may obtain the foreground depth from the user.
• the user input 120 may be arranged for enabling the user to provide the foreground depth to the depth map generator 140.
• the user may provide the foreground depth using a slider, dial or similar user interface element which may be shown on the display 1 10.
• the user input 120 may be arranged for enabling the user to indicate a point in the space surrounding the foreground object which has a same or similar depth as that of the foreground object, and the depth map generator 140 may be arranged for establishing the foreground depth based on the depth of the space at the point.
• the depth map generator 140 may be arranged for establishing the foreground depth based on a depth of the global depth profile 360 at a boundary 197 with the depth representation of the foreground object 196.
• the depth map generator 140 may automatically determine which point on said boundary corresponds to a point where the foreground object is in contact with the space. Per default, the depth map generator 140 may assume that such a point is located on a lower portion of the boundary of the foreground object, e.g., based on an assumption that most objects are standing within a scene.
• the depth map generator 140 may also be arranged for estimating whether the foreground object is standing, floating or hanging within the actual scene shown in the 2D image 144, and based on said estimate, appropriately establish a point on the boundary of the foreground object.
• the user input 120 may be arranged for enabling the user to manually indicate the point where the foreground object is in contact with its surrounding space, e.g., by clicking on said point in the 2D image 144.
• hanging is understood as including both hanging from a ceiling and hanging from a wall, e.g., being wall-mounted. Therefore, when establishing that the foreground object is hanging, the depth map generator 140 may additionally establish whether the foreground object is hanging from a ceiling or hanging from a wall.
• the user input 120 may be arranged for enabling the user to establish that the foreground object is standing or hanging within the actual scene shown in the 2D image 144, and the depth map generator 140 may be arranged for establishing a point on the boundary based on said standing or hanging.
• the user may indicate that the person 192 is standing in the room, and as a result, the depth map generator 140 may establish a point on the lower boundary 197 of the depth representation 196 of the person 192, and subsequently use the depth of the global depth profile 360 at said position as the foreground depth.
• the depth map generator 140 may propagate the depth at the point on the lower boundary 197 of the foreground object upwards so as to fill the depth representation of the foreground object with the foreground depth. It is noted that, since in previous steps, the global depth profile 360 has been established as the depth map 142, the foreground depth may be equally established from the corresponding point in the depth map 142.
• Fig. 5f shows an example of the user additionally selecting the lamp 194 in the 2D image 144.
• the user may further indicate that the foreground object, i.e., the lamp 194, is hanging in the room. Consequently, the depth map generator 140 may establish a point on the upper boundary 199 of the depth representation 198 of the lamp 194, and propagate the depth at said point on the upper boundary 199 downwards in the
• the depth map generator 140 may also automatically establish that the foreground object is hanging or standing, e.g., based on whether the top or bottom of the foreground object is located in an upper half or lower half of the 2D image 144.
• the depth map generator 140 may also take into account the depth of the space surrounding the foreground object to establish whether the foreground object is hanging or standing. For example, the depth map generator 140 may establish that the person 192 shown in Fig. 5d constitutes an object standing in its surrounding space based on its bottom being located on a lower depth gradient which likely constitutes a floor within the global depth profile.
• the device 100 therefore enables the user to iteratively establish a depth map 142 for a 2D image 144 by firstly establishing a global depth profile as the initial depth map 144, and subsequently selecting one or more foreground objects which are to be included in the depth map 144. As a result, an iterative refinement of the depth map 142 is obtained.
• Fig. 6a shows another example of a geometric template 300 comprising depth information.
• a change in depth information is schematically indicated by a change in density of lines, i.e., a high density denotes a large depth and a low density denotes a small depth or close proximity.
• Fig. 6a further shows a marker 320 which constitutes a vanishing point in the global depth profile 360 as defined by the geometric template 300.
• Fig. 6b shows an example of a 2D image 144 comprising an actual scene of which the global depth may be well established by the global depth profile of the geometric template 300 of Fig. 6a.
• the user may position the marker 320 in the 2D image 144 so as to match the vanishing point of the global depth profile to an actual vanishing point shown within the 2D image 144.
• the user input 120 may be arranged for enabling the user to add and/or remove markers to and/or from the plurality of markers.
• the user may add markers by clicking on a connection 340 between a first maker 320 and a second marker 322, with a third marker being added at said position, and the connection 340 being divided into two connections, the first connection connecting the first marker 320 with the third marker, and the second connection connecting the third marker with the second marker 322.
• alternative means of adding and/or removing markers may be used as well.
• the plurality of markers and the one or more connections may form a line segment, i.e., comprising two markers and one connection between the two markers.
• the plurality of markers and the one or more connections may also form a polyline, i.e., comprising multiple connected line segments.
• the plurality of markers and the one or more connections may also form a polygon, i.e., comprising multiple connected line segments which form a closed shape.
• a user interface may be provided that allows a user to select amongst, e.g., the aforementioned line segment, polyline or polygon so as to enable the user to select between thereto associated geometric templates.
• the geometric templates may be indicated with names, upon selection of which, automatically either a line segment, polyline or polygon is provided. For example, when the user selects 'horizon', a line segment may be provide, when the user selects 'vanishing point', a single marker may be provided, and when the user selects 'indoor', a rectangular polygon may be provided
• any reference signs placed between parentheses shall not be construed as limiting the claim.
• Use of the verb "comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim.
• the article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements.
• the invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Processing Or Creating Images (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=48607245

Family Applications (1)

Country Status (2)

Cited By (3)

Families Citing this family (1)

Family Cites Families (4)

• 2013
  • 2013-06-07 WO PCT/EP2013/061798 patent/WO2014001062A2/fr not_active Ceased
  • 2013-06-10 TW TW102120585A patent/TW201412094A/zh unknown

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events