WO2015019208A1 - Appareil et procédé pour corriger des distorsions de perspective d'images - Google Patents

Appareil et procédé pour corriger des distorsions de perspective d'images Download PDF

Info

Publication number
WO2015019208A1
WO2015019208A1 PCT/IB2014/062727 IB2014062727W WO2015019208A1 WO 2015019208 A1 WO2015019208 A1 WO 2015019208A1 IB 2014062727 W IB2014062727 W IB 2014062727W WO 2015019208 A1 WO2015019208 A1 WO 2015019208A1
Authority
WO
WIPO (PCT)
Prior art keywords
point
projection
image
center
plane
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IB2014/062727
Other languages
English (en)
Other versions
WO2015019208A9 (fr
Inventor
Pietro Porzio Giusto
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sisvel Technology SRL
Original Assignee
Sisvel Technology SRL
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sisvel Technology SRL filed Critical Sisvel Technology SRL
Publication of WO2015019208A1 publication Critical patent/WO2015019208A1/fr
Publication of WO2015019208A9 publication Critical patent/WO2015019208A9/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence

Definitions

  • the present invention relates to an apparatus and a method for correcting images, so as to reduce the deformations that appear, in both bidimensional and stereoscopic vision, when the images are viewed from a point of view not corresponding to the center of projection of the perspective according to which they have been produced.
  • Linear perspective also called simply perspective
  • Fig. 1 The basic criterion of perspective construction, shown in Fig. 1, consists in projecting onto a plane 101, referred to as “projection plane” or “projection frame” or simply “frame”, the points of the three-dimensional space as viewed from a "center of projection” C.
  • the straight line extending from the center of projection in the direction towards which the viewer's sight, or the camera lens, is oriented is called “optical axis”.
  • a Cartesian reference is defined, such as the one shown in Fig.
  • the axes originate from the center of projection C, the z-axis coinciding with the optical axis, the j-axis being vertical, oriented upwards, and the -axis being horizontal, oriented from left to right for the viewer.
  • the following will generally refer to the case wherein the frame is perpendicular to the optical axis (this is the case of the perspective referred to as "vertical frame perspective"), but the man skilled in the art will understand that the method of the present invention is not limited to such a case, but is also applicable to cases wherein the frame is not perpendicular to the optical axis.
  • the z-axis is also called “depth axis", since the "depth" of a point of the three-dimensional space is defined as the distance of that given point from the y-plane.
  • the projection plane 101 is at a distance / from the center of projection C and is perpendicular to the optical axis, which intersects the projection plane (101) at the point Ic.
  • the projection Q of a point of the space d results from the intersection between the projection plane 101 and the "projective straight line", i.e. the straight line that passes through the point to be projected A and the center of projection C.
  • Photo and video cameras produce images that are theoretically compliant with linear perspective, but in fact real lenses often introduce more or less visible distortions, e.g. like the so-called “barrel” and “cushion” distortions.
  • the present invention will not deal with such kinds of distortions, which are the subjects of many studies and correction techniques (see for example European Patent EP1333498 Bl to Agilent Technologies Inc. and International patent application WO 98/57292 Al by Apple Computer).
  • the present invention will not even tackle those image deformations which are caused by errors in the positioning of the lenses with respect to the desired position and orientation, such as tapering of images of tall buildings taken from below and deformation of images of documents photographed obliquely or from point of views offset from the document axis.
  • image deformations have been amply discussed in the literature as well (see for example documents US 7,990,412 Al, US 2009/0103808 A1, BR PI0802865 A2, US 2004/0022451 Al, US 2006/0210192 Al,
  • the correction technique of the present invention processes the images as if they were perfectly compliant with linear perspective. Any distortions with respect to the linear perspective, and in particular those mentioned above, will not be taken into account and will be present in the processing results.
  • the present invention relates to deformations that appear in perspective images when said images are viewed from a point of view not corresponding to the center of projection, e.g. if the image of the frame 101 (Fig. 1) is viewed from the point V, not from the center of projection C.
  • FIG. 2 An example of such deformations is shown in Fig. 2.
  • This shows the image of a solid 202 drawn by a calculation program rigorously in accordance with the linear perspective rules.
  • the image of the solid 202 is located in the lower right corner of a frame, of which the figure only shows the lower right quadrant (also called fourth quadrant) 201 in order to enlarge the elements of interest (i.e. the solid 202 and the center Ic of the projection plane) compared to the dimensions that they would have if the entire frame were reproduced.
  • the lower right quadrant also called fourth quadrant
  • Fig. 2 is observed from a point close to the corresponding center of projection, which is located on the straight line passing through Ic, orthogonal to the figure plane, at a distance from the sheet (i.e. the image plane) equal to 25 times the cube edge, then the solid 202 will appear correctly with a cube shape, not with the deformed parallelepiped shape.
  • Fig. 3 represents the horizon plane of a perspective, i.e. the z-plane of Fig. 1, in which a square ABDE is drawn, with its front side adjacent to the projection plane.
  • the element 301 represents the projection plane 101 of Fig. 1, as viewed from the direction of the j-axis.
  • the projection plane and the image plane coincide, although in reality the image is normally reproduced on a support which is distinct from the projection plane. If the two planes are distinct, they can be transferred one over the other, with all their respective geometric elements, by means of a homothetic transformation known to those skilled in the art.
  • the linear perspective reproduces the reality well if the images are viewed from the point corresponding to the center of projection, but the images will turn out to be deformed if the viewer moves away from such point.
  • Pannini painted suggestive architectural views with wide angles of view, with no visible perspective deformations (v. Thomas K. Sharpless, Bruno Postle, and Daniel M. German, “Pannini: A New Projection for Rendering Wide Angle Perspective Images”, Computational Aesthetics in Graphics, Visualization, and Imaging (2010), The Eurographics Association 2010, http://vedutismo.net/Pannini/panini.pdf).
  • Denis Zorin, Alan H. Barr see Zorin D., Barr A. H., "Correction of geometric perceptual distortions in pictures", SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (1995), pp.
  • Robert Carroll et al. (Carroll R., Agrawal M., Agarwala A., "Optimizing content-preserving projections for wide-angle images", SIGGRAPH '09: ACM SIGGRAPH 2009 papers (New York, NY, USA, 2009), ACM, pp. 1-9) have proposed to minimize deformations by adapting the projection to the contents.
  • a man- machine interface is provided, through which the user can characterize the areas and elements of the images to be corrected in particular ways, such as straight lines that must remain as such, people's faces, etc. This method is however time-consuming and uncomfortable, and requires specific calibrations for various types of elements.
  • Image recognition and correction are made easier by determining the orientation of the video cameras by means of sensors (accelerometers, gyroscopes). As can be guessed, this method is very complex and is only applicable in particular circumstances. Furthermore, it does not solve the problem of deformations that arise, in general, when a perspective image is viewed from a point of view not corresponding to the center of projection.
  • the present invention provides an adequate solution to the above-described problem by disclosing a method, and the associated apparatus, for correcting the deformations that appear on images when the latter are viewed from a point not corresponding to the center of projection of the perspective.
  • the apparatus and the associated method are applicable to both the bidimensional reproduction of a single image and the reproduction of a pair of stereoscopic images.
  • Said apparatus comprises suitable means for acquiring a bidimensional image or a pair of stereoscopic images, with sufficient data to determine the coordinates of the point corresponding to the center of projection of the perspective (e.g. image center and focal length) and with the associated depth map.
  • the latter is defined as the set of depths of the points of the scene represented in the image, that is, with reference to Fig. 1, the set of ⁇ -coordinates of the points of the three-dimensional scene.
  • data may be provided from which it can then be obtained.
  • this apparatus comprises storage means and processing means (e.g. a processor executing a suitable software code) configured to correct the position of the points of the acquired images, according to a technique called "Partial Perspective Gradient" (PPG).
  • PPG Partial Perspective Gradient
  • the corrections made by this technique tend to represent, in the image plane, the position of each point as if it had been captured with the lens pointed at it.
  • Such technique numerous variants of which can be defined, is based on the calculation of a gradient of the position of the points in the image plane. By integrating the components of this gradient, the functions can be found according to which the points will be positioned in order to make said correction.
  • each pixel represents, in an approximate manner, a small area of the image; for simplicity, however, in some parts of this description a geometric point will be identified as a pixel, accepting the approximation according to which the pixel's discrete coordinates are assumed to correspond to those of the geometric point it is intended to represent.
  • said gradient is calculated by taking into account a generic point A of the three-dimensional space, corresponding, according to linear perspective, to the point Q of the image plane 401, and to the point P resulting from the intersection between the projective straight line of A and the plane 402, which will be called ⁇ (or auxiliary ⁇ - plane), and which, in the preferred embodiment of the invention, is orthogonal to the projective straight line of A.
  • An incremental displacement of A corresponds, in the ⁇ -plane 402, to an incremental displacement of P (hereafter, an incremental displacement of A is meant to be an infinitesimal virtual increment, whether positive or negative, of one or more coordinates of the point A; such increment is hypothetically applied, according to the mathematical method of infinitesimal calculation, to determine the mathematical functions that bind the position of the points in the neighbourhood of A in the three- dimensional space to the positions of the points in the neighbourhood of the projection of A on the projection planes, as will be explained below). From the components of the incremental displacement the gradient components, i.e. the partial derivatives, of the functions are calculated, with which to represent, in the image plane 401, the correct coordinates of the image of the point A.
  • Fig. 1 geometrically illustrates the operation of linear perspective
  • Fig. 2 illustrates the perspective representation of a cube in an offset position relative to the optical axis
  • Fig. 3 qualitatively illustrates the deformations inherent in linear perspective when viewing images from a point not corresponding to the center of projection
  • Fig. 4 geometrically illustrates one embodiment of the invention
  • Fig. 5 geometrically illustrates the operation of linear perspective in the stereoscopic case
  • Fig. 6 illustrates a plan view of a part of Fig. 4
  • Fig. 7 illustrates a representation of Cartesian references
  • Fig. 8 illustrates the trend of the raised cosine function and its complement that are used in the method and apparatus according to the invention
  • Fig. 9 illustrates stereoscopic images of the cube of Fig. 2;
  • Fig. 10 illustrates a block diagram of an apparatus according to the invention
  • Fig. 11 illustrates a flow chart of a process wherein the perspective correction of the present invention is applied.
  • the present invention relates to the correction of single bidimensional images or pairs of stereoscopic images, aimed at reducing the deformations that appear in perspective images when they are viewed from a point not corresponding to the center of projection of the perspective.
  • the following will describe an apparatus, and the method it implements, with reference to a preferred embodiment and some exemplary but non- limiting variants thereof.
  • the apparatus of the present invention processes images by using a technique called "Partial Perspective Gradient” (PPG), which corrects the position of the points (pixels) of the images in such a way as to locate them as if each point of the scene represented in the image had been captured by a lens pointed at it.
  • PPG Partial Perspective Gradient
  • said technique uses the coordinates of the point the representation of which has to be corrected, and the data defining the geometry of the perspective according to which the image has been generated.
  • the coordinates of the point A taken into account are obtained from the image, i.e. from the x, y coordinates of the point Q, and from the distance of the point s from the y-plane.
  • this distance is called “depth”
  • the set of distances of the points of the three-dimensional space from the y-plane is called “depth map”.
  • the geometry of the perspective according to which the image has been generated is essentially defined by the focal length and by the frame dimensions.
  • the interoptical distance b i.e. the distance between the centers of projection from which the two stereoscopic images have been generated, is also considered in addition to the focal length and the frame dimensions.
  • the data sufficient to determine the depth of the points represented in the image also known as depth data, may comprise the depth map, and can be obtained by using various methods known to those skilled in the art, whether for an image intended for bidimensional vision or for a pair of stereoscopic images. In the case of drawings or paintings, such data are implicit in the artist's project.
  • the depth map can be obtained from the disparity map, which represents the difference between the horizontal coordinates of homologous points of the two images, as shown in Fig. 5.
  • the two centers of projection are horizontally aligned, and that the horizontal axes of the Cartesian references lie in the horizon plane, i.e. the horizontal plane that contains the centers of projection.
  • Fig. 5 shows one example wherein a point A of the three-dimensional space is projected from two distinct centers of projection CL and CR, onto two distinct projection planes, i.e. plane 501 and plane 503, whereon the coordinates are referred to the Cartesian references Ic L x y L and Ic R x R y R , respectively.
  • the optical axes starting off from the centers of projection, i.e. z L from C L and z R from C R are parallel to each other and are located at a distance b (interoptical distance) from each other.
  • the planes 501 and 503 are orthogonal to said optical axes and equidistant from the respective centers of projection by a distance /, whereas their horizontal axes, respectively xL and R , lie on one same straight line, which is parallel to the line joining the two centers of projection C L and C R , and intersects the optical axes at the points Ic L and Ic R , respectively.
  • the homologous points Q L and Q R resulting from the projection of A onto the planes x y L and R y R , respectively, have the same vertical coordinate, which for simplicity is not indicated in Fig. 5, and have the horizontal coordinates X QL and X QR , respectively.
  • z A "depth" of the point taken into account i.e. the z coordinate of the point A of Fig. 4 or the coordinates of the point A of Fig. 5 on the axes z L and z R ;
  • interoptical distance i.e. the distance between the two centers of projection C L and C R , f focal length, i.e. the distance between the projection center C L or C R and the respective projection plane 501 or 503; disp disparity, i.e. the difference between the horizontal coordinate of one point of the left image (Q L ) and the corresponding coordinate of the homologous point of the right image (Q R ), the coordinates being referred to the centers of the respective images (Ic L and Ic R ) or to other homologous reference points.
  • the equation (2) is defined with reference to the centers of projection and the projection plane, but, as is known to the man skilled in the art, the same equation, mutatis mutandis, can also be used for relating the depth of a point represented on the two images of a stereoscopy with the disparity measured on the pair of stereoscopic images and with the value of the equivalent interoptical distance of the geometrical configuration according to which the stereoscopy has been generated.
  • the center of projection of the perspective according to which an image has been generated can be located by providing the center of the projection plane and the distance between the center of projection and the projection plane (i.e. the focal length in the case of video cameras), or by providing the dimensions of the projection plane and the angle of view, since the angle of view, the image diagonal and the distance of the center of projection from the projection plane are bound by the equation (1).
  • the apparatus claimed by the present invention comprises suitable means for acquiring, in numerical form, a bidimensional image or a pair of stereoscopic images, along with depth or disparity data and other data sufficient to determine the coordinates of the center of projection C corresponding to the center of projection of the perspective, as mentioned above.
  • the apparatus claimed by the present invention applies the method ("Partial Perspective Gradient") according to the invention, wherein said method has been developed on the basis of the principle of representing the neighbourhood of each point of an image as if it had been pointed at while shooting or drawing. This principle takes into account that, in such conditions of sight orientation, small displacements of the point in question (e.g. the point A in the annexed figures) would be perceived.
  • the plane 402 lies at the same distance from the center of projection C as the projection plane 401. This configuration should however only be considered as a non-limiting explanatory example of the preferred embodiment. As the man skilled in the art will guess, and as will be explained below, the plane 402 can be set at any distance from the center of projection and with any orientation.
  • this projection Ap can be defined by using the C ⁇ y Cartesian reference.
  • This reference has its origin at C, which is coincident with the origin of the Cxyz reference, and its y -axis develops along a direction that coincides with that of the straight line passing through C and through A.
  • the ⁇ -plane is orthogonal to the y -axis.
  • the ⁇ -axis is defined by the intersection between the ⁇ -plane and the xz-plane (the ⁇ -axis lies in the xz-plane), whereas the ⁇ -axis, also passing through C, is orthogonal to both the ⁇ -axis and the y -axis.
  • a displacement parallel to the -axis of the point A implies, in the plane 402, a displacement of P with components in both the direction of the a-axis and the direction of the ⁇ -axis.
  • the component in the direction of the ⁇ -axis is null, and hence only the component in the direction of the ⁇ -axis, referred to as , will remain to be treated. Since with we want to determine the partial derivative that the component along the x-axis of the point Q, also referred to as x Q , must take, is related with .
  • the calculation can be made by using the common rules of geometry and mathematics, which are known to the man skilled in the art. They essentially provide for changing the reference system, switching the representation of the displacement from the Cxyz Cartesian reference to the C ⁇ y Cartesian reference.
  • the formulae for changing the reference systems can be found in school books and on various Internet sites, among which, for example, the following:
  • the expression (3 ⁇ y) indicates that the incremental displacement of the point P in the direction of the ⁇ -axis does not depend on the displacement component of the point Q in the direction of the y-axis, because the y-axis is orthogonal to the plane in which the ⁇ -axis lies, as aforesaid when commenting on Fig. 7 (the ⁇ -axis lies in the xz- plane).
  • the formula (4 ⁇ xa) provides results that are only slightly different from the integral calculation (4 ⁇ x). For example, in the case of application to the cube shown in Fig. 2, in which there are points corresponding to angle of views of 90°, the calculations made with the formula (4 ⁇ xa) will differ by less than 1% from those made with the formula (4 ⁇ x). As can be noticed, the use of the formula (4 ⁇ xa) does not require, unlike that of the formula (4 ⁇ x), any steps of numerical integration, thus advantageously reducing the processing time and load.
  • the formulae (4..) constitute the first embodiment of the "Partial Perspective Gradient” technique of the present invention, consisting of representing, in terms approximated in the image plane, what would appear from the center of the perspective, in the neighbourhood of each point of the scene to be reproduced, if the optical axis passed through that point.
  • a certain number of variants of the above technique can be taken into consideration.
  • formulae may be used which represent dependence from the "disparity" between homologous points of stereoscopic images, coherently with the formula (2).
  • a second variant uses formulae determined by assuming that the point P (Fig. 4, Fig. 6, Fig. 7), instead of being at a fixed distance of f from the center of projection C, is located at a distance from C that depends on some parameter. In particular, it can be imposed that , so that P will coincide with Q.
  • the formulae are determined by imposing that the point P is located on a segment VA, instead of the segment CA, with V distinct from C (Fig. 1, Fig. 3, Fig. 4).
  • the point V may preferably be located on the optical axis passing through the points C and I c , or away from said optical axis.
  • the distance of P from V may either be preset to a constant value or be variable depending on some parameter. For example, it can be imposed that .
  • a second embodiment of the idea consists of applying partly the formulae (4..) and partly the linear perspective formulae.
  • linear perspective reproduces images well within certain limits, and therefore within such limits it may be profitable to maintain the reproduction provided by linear perspective, combining the formulae (4..) with the linear perspective ones.
  • such combination may be made in such a way as to gradually switch from exclusive application of the formulae (4..) to exclusive application of the linear perspective formulae, but it may also be carried out in other manners that the man skilled in the art will be able to imagine.
  • One way to make such combination is to multiply the results of the formulae (4..) by a first factor, preferably comprised between the unitary value and the null value, and to multiply the results of the linear perspective formulae by a second factor, preferably complementary to the first factor; the results of the products thus obtained are then added up.
  • one example of a function suitable for creating the multiplicative factors is the raised cosine function represented by the curve 801, together with its complement 802.
  • the curve 801 stays at the unitary value for abscissa values between zero and a limit t s ; afterwards, in the interval from t s to t f , it gradually decreases to zero, and then it stays at the null value. Instead, its complementary function 802 has the opposite trend.
  • the abscissas of Fig. 8, and hence the limits t s to t f , can be related with the offset angle at which the point to be represented is seen from the center of projection, or with the distance of the point from the center of projection, or with other parameters or combinations of parameters allowing the man skilled in the art to meet the requirements of a specific application of the method according to the invention.
  • Said second embodiment of the invention is also liable to all variations that may be conceived for the first embodiment.
  • a F is the enlarged image of the cube 202 (cube projected from the center of projection Ic);
  • a E is the image of the same cube 202 projected from a center of projection with an abscissa equal to twice the cube side;
  • b F is the image obtained by processing the image a F with the "Partial
  • the figure In order to obtain the stereoscopic vision of the images of Fig. 9, the figure needs to be reproduced in such a way that the distance between the vertical dashes hanging from the upper horizontal line is about equal to, or slightly shorter than, the viewer's interpupillary distance.
  • An adequate dimension is normally obtained by reproducing the sheet containing Fig. 9 in the A4 format (21cm wide). After making sure that the straight line joining the centers of the viewer's pupils is parallel to the straight lines that delimit the figure at the top and at the bottom, it is then necessary to look fixedly at the figure, so as to obtain the merging of the right and left images.
  • Such merging can be facilitated by initially looking fixedly at the arrows running from the lower images to the upper images, or, better still, by placing a card near the viewer's forehead in a position orthogonal to said delimiting horizontal straight lines, so that the right eye will not see the left image, or at least most of it, and vice versa.
  • the image "b" (obtained from the merging of the images b F and b E ) well represents the shapes of a cube
  • the image "a” does not even look like a parallelepipedon, because the dimensions of the rear face appear to be bigger than those of the front face.
  • the horizontal and vertical lines maintain their own directions; in particular, the segments that lie in the projection plane, such as the edges of the front face of the cube reproduced in Fig. 9, maintain their length and their orientation, so that the front face of the cube will appear perfectly square.
  • the result of the processing carried out by using the PPG method instead, shows a cube which is seen obliquely, coherently with the fact that it is offset from the optical axis by 21° horizontally and by -15° vertically.
  • the PPG technique can be applied to stereoscopic images with even more advantage than to monoscopic images.
  • the PPG method can be applied to each one of the images of the stereoscopic pair, with all the possible variants mentioned above, but in the stereoscopy case there exist additional variants and expedients.
  • An image processing apparatus 1 like for example a photo camera or a video camera or the like, comprises image acquisition means 1001, input/output means 1002, a central processing unit (CPU) 1003, a read-only memory 1004, a random access memory (RAM) 1005, and means for producing processed images 1006. All of these elements of the apparatus 1 are in signal communication with one another, so that data can be mutually exchanged between any two of such elements.
  • CPU central processing unit
  • RAM random access memory
  • the image acquisition means 1001 can acquire both bidimensional images and pairs of stereoscopic images. If the images are equipped with the respective depth map and with data allowing to go back to the geometry with which they have been generated (e.g. focal length, angle of view, sensor inclination, or the like), the image acquisition means 1001 will acquire such data as well. Otherwise, some data can be set through a user interface (not shown in the drawings) in signal communication with the input/output means 1002, whereas for stereoscopic images the depth map may even be produced by the apparatus 1 itself, as will be explained below.
  • the user interface also allows the user to set options and variants, along with their parameters, that he/she may prefer to use in the specific case. For example, one can choose from the following settings:
  • the point of view relative to which the processing of the method of the invention is applied may be different from the center of projection C (see Fig. 1) relative to which the image to be processed has been generated.
  • the coordinates of Q and the depth of A being known, and having calculated with them the position of A in the three- dimensional space, one can in fact determine the projection of A onto any plane and from any point of view. For example, assuming as a center of projection a point K (the point corresponding, for example, to the point of view from which the viewer is supposed to be looking at the image) different from the center of projection C, one can determine the projection of A onto a plane perpendicular to the straight line that joins A to V, or onto a different plane. Processing the image from a point of view other than the center of projection C and onto various planes allows to make particular corrections to the image and to produce useful artificial images, such as those which can be used in stereoscopy for filling blanks, as will be discussed below.
  • the central processing unit (CPU) 1003 is that part of the apparatus which executes the calculation algorithms, including complementary operations that, after the application of the PPG technique, are useful for completing the correction of the images to be returned. These operations will be discussed while commenting on Fig. 11.
  • the central processing unit 1003 may actually comprise specially developed integrated circuits, one or more microprocessors, programmable logic circuits (e.g. CPLD, FPGA), and the like. These and other implementation possibilities neither anticipate nor make obvious the teachings of the present invention.
  • the read-only memory 1004 is preferably used for permanently storing some instructions for managing the apparatus and the instructions that implement the calculation algorithms, while the random access memory (RAM) 1005 is typically used for temporarily storing the images and the intermediate processing results.
  • RAM random access memory
  • the means for producing processed images 1006 return the processed images, e.g. by transferring them from the RAM memory 1005 to the input/output means 1002, so that said means 1002 can save said processed images into a permanent memory (e.g. a hard disk or a type of Flash memory, such as Secure Digital, MMC or the like), display them on one or more screens or other display means (not shown in the annexed drawings), print them, and the like.
  • a permanent memory e.g. a hard disk or a type of Flash memory, such as Secure Digital, MMC or the like
  • display them e.g. a screen or other display means (not shown in the annexed drawings), print them, and the like.
  • the process that implements the invention may comprise the following steps:
  • - start step 1101 during which the apparatus 1 is configured for processing at least one image
  • - setting step 1102 for acquiring the settings and the data that the user intends to manually provide for processing at least said image
  • step 1 103 during which the image to be processed is acquired by the image acquisition means 1001 and is preferably transferred into the RAM memory 1005 (for simplicity, it is assumed that, downstream of step 1103, the images are in numerical form, and that, in the event that they should be available in another form, the man skilled in the art will know how to convert them into numerical form); it is understood that the image acquired at step 1103 is equipped with data defining the geometry according to which the image has been generated, and that it is preferably also equipped with the depth map, or with data allowing to create it; the subsequent steps 1104 and 1105 will consider the case wherein the depth map is not provided at this step 1103;
  • step 1 104 depth map presence verification step 1 104, during which it is determined whether the depth map is available or not;
  • step 1106 during which the apparatus 1, by using the depth map acquired at step 1103 or calculated at step 1105, determines the position in the three- dimensional space of the points corresponding to the pixels of the image acquired at step 1103 and applies thereto the position correction algorithm according to the invention;
  • the result of step 1106 is a matrix indicating the position that the pixels of the image acquired at step 1103 must take after said processing; at this stage no pixel shifting occurs yet, in order to avoid having to repeat this kind of operation after the additional processing carried out in the next steps;
  • the resizing process consists of recalculating, starting from the result of step 1106, the position that the pixels of the processed image must take after the resizing applied at this step 1107;
  • - processed image returning step 1110 during which the image is preferably stored into an area of the RAM memory 1005, or another memory, and is made available for display, printing, transfer to other apparatuses, and other operations;
  • the apparatus 1 When the apparatus 1 is in an operating condition, after the start step 1101 said apparatus 1 enters the setting step 1102, and then, simultaneously or afterwards, the image acquisition step 1103. At the end of step 1103, the apparatus 1 verifies the availability of a depth map (step 1104) for what has been acquired at step 1103; if the depth map is available, the apparatus 1 enters the processing step 1106; otherwise, if the map is not present, the apparatus enters the depth map calculation step 1105 prior to proceeding with the processing step 1106. After step 1106, the apparatus may optionally carry out the resizing step 1107. Then the apparatus 1 carries out the overlap elimination step 1108, followed by the pixel shifting and blank filling step 1109 and by the processed image returning step 1110. The process ends at the final step 1111. At step 1106 the PPG technique of the present invention is applied in one of the above- described embodiments thereof.
  • the processing means 1003 and the storage means 1004 and 1005 of the apparatus 1 are configured to correct the image represented in the plane 401, which has been generated in compliance with the linear perspective rules relative to the center of projection C and comprises at least one first point Q, wherein said first point Q is the result of the perspective projection of the second point A of a region of the three- dimensional space from the center of projection C; to do so, the processing means 1003 and the storage means 1004 and 1005 execute the method according to the invention, which comprises the following steps: a) calculating the position of the second point A in the three-dimensional space;
  • some pixels of this area may overlap the pixels of areas that have not been shifted, or that have been shifted less, or that have been shifted in different directions.
  • one may, for example, have the pixel with less depth occupy the contended position (i.e. the pixel located at the smaller distance from the xy- plane will prevent seeing the farther one).
  • stereoscopy uses two images of the same scene viewed from two different points of view.
  • the corrective shift of a point of the left image is generally different from the corrective shift of the homologous point of the right image.
  • the blanks in either one of the two processed images can be at least partially filled by appropriately processing the other image.
  • one way of filling the blanks in the left (right) image is to superimpose it on an "artificial" image obtained by processing the right (left) image, assuming as a center of projection the same center of projection of the left (right) image.
  • step 1109 It is advantageous to carry out the pixel shifting and blank filling operations during step 1109 on already resized images and after having resolved any conflicts, i.e. downstream of step 1108, because resizing may also cause overlaps and blanks.
  • the application of the PPG method as described in the present invention will improve the vision of the images of objects located in offset positions relative to the pointing direction of the capturing device. Such improvements are evident in monoscopic vision, and are even more evident in stereoscopic vision.
  • the PPG technique which can be applied in real time while shooting or afterwards on acquired images, allows to capture scenes with angles of view exceeding the currently recommended limits, leading to significant advantages for both stereoscopic and monoscopic shooting.
  • the PPG technique turns out to be very versatile and can be optimized for different types of applications and apparatuses, which may even be characterized by very different processing capabilities.
  • the PPG technique can be used for correcting deformations in video streams, by applying it to every image it is composed of. This applies to both 2D and 3D video streams; in the latter case, the technique will have to be applied to each image of the stereoscopic pairs forming the 3D stream.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)
  • Analysing Materials By The Use Of Radiation (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

La présente invention concerne un appareil et un procédé qui permettent de corriger les déformations qui apparaissent, à la fois dans une vision bidimensionnelle et stéréoscopique, lorsque des images générées conformément aux règles de perspective linéaire sont visualisées d'un point de vue ne correspondant pas au centre de projection de la perspective. Les corrections sont déterminées par une technique qui tend à représenter les images comme si chaque point les constituant était capturé par la lentille pointant vers ce dernier. Dans une telle hypothèse d'orientation de lentille, le gradient de position des points à représenter est calculé, et les coordonnées des positions que les points doivent prendre dans le plan image sont obtenues par intégration des composantes d'un tel gradient. La plage d'application du procédé est très large, étant donné qu'il permet différents modes de réalisation et de nombreuses variantes.
PCT/IB2014/062727 2013-08-08 2014-06-30 Appareil et procédé pour corriger des distorsions de perspective d'images Ceased WO2015019208A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IT000683A ITTO20130683A1 (it) 2013-08-08 2013-08-08 Apparato e metodo per la correzione delle deformazioni prospettiche delle immagini
ITTO2013A000683 2013-08-08

Publications (2)

Publication Number Publication Date
WO2015019208A1 true WO2015019208A1 (fr) 2015-02-12
WO2015019208A9 WO2015019208A9 (fr) 2015-07-16

Family

ID=49354843

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2014/062727 Ceased WO2015019208A1 (fr) 2013-08-08 2014-06-30 Appareil et procédé pour corriger des distorsions de perspective d'images

Country Status (2)

Country Link
IT (1) ITTO20130683A1 (fr)
WO (1) WO2015019208A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3072812A1 (fr) 2015-03-27 2016-09-28 Airbus Helicopters Procede et dispositif pour signaler au sol un aeronef en vol, et aeronef muni de ce dispositif
CN110246169A (zh) * 2019-05-30 2019-09-17 华中科技大学 一种基于梯度的窗口自适应立体匹配方法及系统
DE102021103323A1 (de) 2021-02-12 2022-08-18 Carl Zeiss Ag Pannini-Objektiv und abbildendes optisches Gerät
CN115170726A (zh) * 2022-01-21 2022-10-11 中国人民解放军战略支援部队信息工程大学 一种动态沙盘的三维投影方法

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0583060A2 (fr) * 1992-07-24 1994-02-16 The Walt Disney Company Procédé et système pour simuler un effet tri-dimensionnel
WO1998057292A1 (fr) 1997-06-12 1998-12-17 Apple Computer, Inc. Procede et systeme de creation d'un environnement de realite virtuelle a partir d'une image et au moyen d'un oeil-de-poisson
WO2000035200A1 (fr) * 1998-12-07 2000-06-15 Universal City Studios, Inc. Procede de correction d'images destine a compenser la distorsion de celles-ci en fonction du point de vue
US20040022451A1 (en) 2002-07-02 2004-02-05 Fujitsu Limited Image distortion correcting method and apparatus, and storage medium
US20060210192A1 (en) 2005-03-17 2006-09-21 Symagery Microsystems Inc. Automatic perspective distortion detection and correction for document imaging
EP1333498B1 (fr) 2002-01-31 2007-08-29 Agilent Technologies, Inc. Réseau de capteurs d'image pour la correction de la distorsion curviligne d'un système photographique de lentilles et sa procédé de fabrication
US20090103808A1 (en) 2007-10-22 2009-04-23 Prasenjit Dey Correction of distortion in captured images
US20110090303A1 (en) 2009-10-16 2011-04-21 Apple Inc. Facial Pose Improvement with Perspective Distortion Correction
US20110149094A1 (en) 2009-12-22 2011-06-23 Apple Inc. Image capture device having tilt and/or perspective correction
US7990412B2 (en) 2004-11-01 2011-08-02 Hewlett-Packard Development Company, L.P. Systems and methods for correcting image perspective
BRPI0802865A2 (pt) 2008-08-14 2011-11-22 Audaces Automacao E Informatica Ind Ltda processo de geração de arquivos de imagens com método automatizado de correção de distorções óticas

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0583060A2 (fr) * 1992-07-24 1994-02-16 The Walt Disney Company Procédé et système pour simuler un effet tri-dimensionnel
WO1998057292A1 (fr) 1997-06-12 1998-12-17 Apple Computer, Inc. Procede et systeme de creation d'un environnement de realite virtuelle a partir d'une image et au moyen d'un oeil-de-poisson
WO2000035200A1 (fr) * 1998-12-07 2000-06-15 Universal City Studios, Inc. Procede de correction d'images destine a compenser la distorsion de celles-ci en fonction du point de vue
EP1333498B1 (fr) 2002-01-31 2007-08-29 Agilent Technologies, Inc. Réseau de capteurs d'image pour la correction de la distorsion curviligne d'un système photographique de lentilles et sa procédé de fabrication
US20040022451A1 (en) 2002-07-02 2004-02-05 Fujitsu Limited Image distortion correcting method and apparatus, and storage medium
US7990412B2 (en) 2004-11-01 2011-08-02 Hewlett-Packard Development Company, L.P. Systems and methods for correcting image perspective
US20060210192A1 (en) 2005-03-17 2006-09-21 Symagery Microsystems Inc. Automatic perspective distortion detection and correction for document imaging
US20090103808A1 (en) 2007-10-22 2009-04-23 Prasenjit Dey Correction of distortion in captured images
BRPI0802865A2 (pt) 2008-08-14 2011-11-22 Audaces Automacao E Informatica Ind Ltda processo de geração de arquivos de imagens com método automatizado de correção de distorções óticas
US20110090303A1 (en) 2009-10-16 2011-04-21 Apple Inc. Facial Pose Improvement with Perspective Distortion Correction
US20110149094A1 (en) 2009-12-22 2011-06-23 Apple Inc. Image capture device having tilt and/or perspective correction

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
"Rendering Techniques '98", 1 January 2000, SPRINGER VIENNA, Vienna, ISBN: 978-3-70-916453-2, ISSN: 0946-2767, article MANEESH AGRAWALA ET AL: "Artistic Multiprojection Rendering", pages: 125 - 136, XP055119830, DOI: 10.1007/978-3-7091-6303-0_12 *
CARLO PEDRETTI: "Leonardo da Vinci on Painting - A lost book (Libro A", 1964, UNIVERSITY OF CALIFORNIA PRESS
CARROLL R.; AGRAWAL M.; AGARWALA A.: "SIGGRAPH '09: ACM SIGGRAPH 2009 papers", 2009, ACM, article "Optimizing content-preserving projections for wide-angle images", pages: 1 - 9
THOMAS K. SHARPLESS; BRUNO POSTLE; DANIEL M. GERMAN: "Pannini: A New Projection for Rendering Wide Angle Perspective Images", COMPUTATIONAL AESTHETICS IN GRAPHICS, VISUALIZATION, AND IMAGING, 2010, Retrieved from the Internet <URL:http://vedutismo.net/Pannini/panini.pdf>
ZORIN D.; BARR A. H.: "Correction of geometric perceptual distortions in pictures", SIGGRAPH '95: PROCEEDINGS OF THE 22ND ANNUAL CONFERENCE ON COMPUTER GRAPHICS AND INTERACTIVE TECHNIQUES, 1995, pages 257 - 264, XP000546235, DOI: doi:10.1145/218380.218449

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3072812A1 (fr) 2015-03-27 2016-09-28 Airbus Helicopters Procede et dispositif pour signaler au sol un aeronef en vol, et aeronef muni de ce dispositif
KR101825571B1 (ko) 2015-03-27 2018-02-05 에어버스 헬리콥터스 비행중인 항공기를 위해 지면에 마킹하기 위한 방법 및 장치와, 그러한 장치를 포함하는 항공기
US9944405B2 (en) 2015-03-27 2018-04-17 Airbus Helicopters Method and a device for marking the ground for an aircraft in flight, and an aircraft including the device
CN110246169A (zh) * 2019-05-30 2019-09-17 华中科技大学 一种基于梯度的窗口自适应立体匹配方法及系统
DE102021103323A1 (de) 2021-02-12 2022-08-18 Carl Zeiss Ag Pannini-Objektiv und abbildendes optisches Gerät
CN115170726A (zh) * 2022-01-21 2022-10-11 中国人民解放军战略支援部队信息工程大学 一种动态沙盘的三维投影方法

Also Published As

Publication number Publication date
WO2015019208A9 (fr) 2015-07-16
ITTO20130683A1 (it) 2015-02-09

Similar Documents

Publication Publication Date Title
EP3163535B1 (fr) Procédé et dispositif d&#39;acquisition d&#39;image étendue
US9774837B2 (en) System for performing distortion correction and calibration using pattern projection, and method using the same
CA3018965C (fr) Determination efficace de flux optique entre des images
EP2549762B1 (fr) Appareil d&#39;adaptation de position d&#39;image en stéréovision, procédé d&#39;adaptation de position d&#39;image en stéréovision et programme correspondant
US6204876B1 (en) Stereoscopic computer graphics moving image generating apparatus
US20180063513A1 (en) Stitching frames into a panoramic frame
US20160301868A1 (en) Automated generation of panning shots
WO2015048694A2 (fr) Systèmes et procédés destinés à la correction de la distorsion de la perspective utilisant la profondeur
JP6810873B2 (ja) 電子ディスプレイの前方または上に投影されて見える仮想３次元画像を作製するためのシステム、方法、およびソフトウェア
EP3189493B1 (fr) Correction de la perspective des photos numériques avec l&#39;aide d&#39;une carte de profondeur
US20180213215A1 (en) Method and device for displaying a three-dimensional scene on display surface having an arbitrary non-planar shape
WO2006100805A1 (fr) Unite et procede d’affichage d’images stereoscopiques et programme informatique
EP3526639A1 (fr) Affichage de données visuelles avec un casque de réalité virtuelle
CA2540538C (fr) Imagerie stereoscopique
WO2015019208A1 (fr) Appareil et procédé pour corriger des distorsions de perspective d&#39;images
US12519914B2 (en) Viewer-adjusted stereoscopic image display from cameras sharing a common portion of their field of view
US20120081520A1 (en) Apparatus and method for attenuating stereoscopic sense of stereoscopic image
JP2011211551A (ja) 画像処理装置および画像処理方法
KR20110025083A (ko) 입체 영상 시스템에서 입체 영상 디스플레이 장치 및 방법
CN107798703B (zh) 一种用于增强现实的实时图像叠加方法以及装置
WO2012176526A1 (fr) Dispositif de traitement d&#39;images stéréoscopiques, procédé de traitement d&#39;images stéréoscopiques et programme associé
US9602708B2 (en) Rectified stereoscopic 3D panoramic picture
CN117528236B (zh) 虚拟摄像机的调整方法和装置
US12511843B2 (en) Method for generating pass-through view with better scale and host
CN120807314A (zh) 一种图像拼接方法、装置及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14752400

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14752400

Country of ref document: EP

Kind code of ref document: A1