EP1735749A2 - Procede de compensation de rotation d'images spheriques - Google Patents

Procede de compensation de rotation d'images spheriques

Info

Publication number
EP1735749A2
EP1735749A2 EP05735021A EP05735021A EP1735749A2 EP 1735749 A2 EP1735749 A2 EP 1735749A2 EP 05735021 A EP05735021 A EP 05735021A EP 05735021 A EP05735021 A EP 05735021A EP 1735749 A2 EP1735749 A2 EP 1735749A2
Authority
EP
European Patent Office
Prior art keywords
camera
rotation
pixel
matrices
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP05735021A
Other languages
German (de)
English (en)
Inventor
Reinhard Koch
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Christian Albrechts Universitaet Kiel
Original Assignee
Christian Albrechts Universitaet Kiel
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Christian Albrechts Universitaet Kiel filed Critical Christian Albrechts Universitaet Kiel
Publication of EP1735749A2 publication Critical patent/EP1735749A2/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/60—Rotation of whole images or parts thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture

Definitions

  • the invention relates to a method for computer-aided stabilization of images, in particular of electronically recorded images (for example with a Nideo camera), in relation to rotations of the camera about an arbitrary axis by arbitrarily large angles.
  • the invention also relates to image recording systems with a fisheye lens for (semi) spherical images, in particular also cameras with a 360 ° all-round view.
  • Examples of devices of the first category are described in the publications DE 43 42 717 AI or DE 196 18 979 AI. These are also particularly suitable for compensating for camera shake when shooting on film material.
  • a camera it is known to equip a camera with a permanently mounted acceleration gyro sensor in order to record the rotation of the camera in addition to images (for example Suya You, Ulrich Neumann, and Ronald Azuma: Orientation Tracking for Outdoor Augmenten Reality Registration. IEEE Computer Graphics and Applications 19, 6 (Nov / Dec 1999), 36-42).
  • the rotation sensor uses an accelerometer (gyroscope principle) to measure the relative change in the 3D orientation between different recording times.
  • other absolute measurement parameters such as magnetic north pole and gravitation (plumb) can be used to generate an absolute orientation reference.
  • An absolute 3D orientation is determined by integrating the relative change over time.
  • the rotation of the sensor can be described by three angles (Rx, Ry, Rz) around the three spatial axes (X, Y, Z) (Euler angle, see Fig. 1 a) or by rotation around an axis of rotation in space ( 1b, axis-angle representation (A, ⁇ ), also: quaternion representation).
  • a hemispherical image recording system is also known, in particular a camera with a fisheye lens.
  • This maps a hemisphere of the room onto a circular disk in the image plane.
  • the 2D image representation takes place in the angular coordinates above the standard hemisphere.
  • the center of the picture shows the pole of the hemisphere.
  • the sphere surface is parameterized by two angles, which run along the longitude ( ⁇ ) and latitude ( ⁇ ) of the sphere (cf. Fig. 2 a).
  • a point ( ⁇ , ⁇ ) on the spherical surface is defined by the spherical mical image on the image plane (x, y), where (xo, yo) depicts the pole (optical axis) (cf. FIG. 2 b).
  • a spherical image recording system is also known, for example, from US Pat. No. 6,002,430, which images the entire surrounding space in that two hemispherical cameras mounted with their backs to one another each record a half space. Due to the non-zero distance between the two camera centers, it makes sense to use fisheye lenses with e.g. Use a 190 ° angle of view to seamlessly combine the two images into a complete all-round image. Nevertheless, there is always (i) a "dead zone" of the device in the immediate vicinity of the double camera, which cannot be viewed, and (ii) a "near zone" in which the parallax shift of the two cameras is noticeable. For scene points that are sufficiently far away, the center of the cara is practically irrelevant.
  • augmented vision In the area of so-called "augmented vision", efforts are made to display additional information in their field of vision, for example for complex work steps.
  • people are equipped with transparent display glasses that can display text or graphics in a real view, comparable to that well-known head-up display (HUD) from military aircraft, in order to automatically determine the required precise knowledge of the position of the glasses in 3D space, all-round view cameras can be rigidly connected to the glasses Marking points in the recorded image make it possible to determine the position of the glasses in the work area, but in order to be able to do without such markings, for example for outdoor use, much more complex image analyzes are required.
  • HUD head-up display
  • FIG. 1 shows possible representations of the camera rotations, FIG. 1 a) with Euler angles, FIG. 1 b) in an axis-angle representation,
  • FIG. 2a shows a sketch of the hemispherical geometry
  • 3 shows a schematic course of the calibration function k ( ⁇ ) of a fisheye lens.
  • a basic requirement for carrying out the method according to the invention is the rigid coupling of a spherical camera with a 3 DoF (Degree of Freedom) rotation sensor.
  • a 3 DoF (Degree of Freedom) rotation sensor To simplify the explanation, only a single fisheye camera is depicted below, which 1 depicts a hemisphere.
  • the method is used simultaneously for both cameras, and the images stabilized according to the invention are then combined in a manner known per se.
  • the pixel plane of the camera requires a two-dimensional coordinate system in 3D space.
  • the orientation of the pixel plane is arbitrary.
  • Technical CCD chips show usually rectangular pixels in a rectangular arrangement, so the Cartesian coordinate system (x, y) is the natural choice. Its origin typically lies in a corner pixel.
  • the spherical projection of the fisheye shows object points on a hemisphere (e.g. with a radius of one) around the camera center on a circular disk in the pixel plane.
  • the natural coordinates of the image are then plane polar coordinates (r, ⁇ ).
  • the optical axis of the camera should run perpendicular to the pixel plane through a central pixel with coordinates (x 0 , y 0 ).
  • Each object point on the hemisphere appears at an angle ⁇ to the optical axis and is also characterized by its azimuth position ⁇ .
  • Fig. 3 shows an example of the calibration function k ( ⁇ ), which specifies the respective conversion factor from angle (e.g. Radian) to pixel coordinates and must be determined once for each lens.
  • the spherical image can be described by the polar coordinates ( ⁇ , ⁇ ) because of the linearity mentioned above. ;
  • an object point on the hemisphere has the pixel coordinates at the coordinates ( ⁇ , ⁇ ) in the pixel plane:
  • the angular coordinates of the object change, for example from
  • the principle of the image stabilization according to the invention consists in the direct calculation of the coordinate shifts (xp, yp) - »(xp ', yp') for all pixels from additional rotation data and in their reversal by copying the color values from (xp ', yp') back to (xp, y P ). This copying is now implemented in all common graphics cards and optimized for real-time applications, for example for rotating a virtual camera in a computer-generated room.
  • the invention therefore deals solely with the mapping (xp, yp) - »(xp ', yp'), which is to be determined quickly from the data of a rotation sensor. Since the camera rotation can be described in a natural way by means of a 3x3 rotation matrix, it makes sense to formulate the problem three-dimensionally, for which a z-axis along the optical axis, ie perpendicular to the pixel plane, is additionally defined. Likewise, a z 'axis is introduced in the rotated camera coordinate system and one can write
  • M M ( ⁇ '. ⁇ ', ⁇ , ⁇ ) denotes a 3x3 matrix that depends on the new and old angular coordinates of the object point on the hemisphere after and before the camera rotation.
  • the invention now shows a way to get this matrix quickly, which is essential for real-time stabilization of the image.
  • Equations (1) can be summarized as a matrix equation
  • K is a 3x3 calibration matrix (with calibration function k ( ⁇ )) and p t is the standard representation of the pixel point (xp, yp) in the polar coordinates ( ⁇ , ⁇ ).
  • the matrix K has to be set up and saved only once for all ⁇ . It is always invertible, and the standard representation
  • a 360 ° all-round view could be depicted on a single circular disk, but this would have disadvantages because of the associated distortions.
  • the rotation sensor When the camera is rotated, the rotation sensor immediately delivers measurement data for the angles rotated around each axis with typical clock rates of 100-200 measurements per second, an angular resolution of typically 0.05 degrees and a short-term accuracy of 0.1-0.3 degrees. From this, a 3x3 rotation matrix can be set up immediately in a manner known per se, which allows the coordinate transformation of any point in space into the now rotated coordinate system which is connected to the camera. This also applies to every object point of the unit sphere, especially for P. It has the representation in the rotated coordinate system
  • the image stabilization is to take place particularly quickly or if very large images are to be stabilized, it is an advantageous embodiment of the method according to the invention not to carry out the mapping for all pixels but only for a selection from these. If, in particular, one selects the nodes of a triangular mesh network (“triangular mesh”), only these network points are initially copied back, but modern graphi-d types also have a so-called “fragment Shader ", which can interpolate color values between known support points. This is typically used in the texturing of computer graphics. In many applications (especially in the field of human vision), pixel-precise stabilization of the image is not important, so that the reduction of the Procedure can bring high speed advantages to a few network nodes.>

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne un procédé de production d'une séquence d'images compensée en rotation permettant la reconstruction simplifiée de la translation d'une caméra, par compensation de rotation en temps réel des images enregistrées séquentiellement par une caméra électronique pourvue d'un capteur de rotation, déplacée aléatoirement dans une scène, lesdites images étant produites par affectation de valeurs de couleur et/ou de luminosité des pixels d'un champ de pixels de la caméra, indexés par des coordonnées rectangulaires. Le procédé selon l'invention consiste à se munir d'un objectif sphérique (oeil de poisson); à déterminer des matrices d'étalonnage pour l'objectif sphérique; à calculer les matrices de rotation décrivant la rotation relative de la caméra déterminée à partir des données de mesure du capteur de rotation; à calculer l'affectation entre les coordonnées de pixels d'un point d'image dans des images consécutives, à partir des matrices d'étalonnage et des matrices de rotation; et à calculer les coordonnées de pixels en fonction de l'affectation calculée, avec compensation de la rotation de caméra.
EP05735021A 2004-04-10 2005-04-07 Procede de compensation de rotation d'images spheriques Withdrawn EP1735749A2 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102004017730A DE102004017730B4 (de) 2004-04-10 2004-04-10 Verfahren zur Rotationskompensation sphärischer Bilder
PCT/DE2005/000613 WO2005101308A2 (fr) 2004-04-10 2005-04-07 Procede de compensation de rotation d'images spheriques

Publications (1)

Publication Number Publication Date
EP1735749A2 true EP1735749A2 (fr) 2006-12-27

Family

ID=34981174

Family Applications (1)

Application Number Title Priority Date Filing Date
EP05735021A Withdrawn EP1735749A2 (fr) 2004-04-10 2005-04-07 Procede de compensation de rotation d'images spheriques

Country Status (4)

Country Link
US (1) US7747105B2 (fr)
EP (1) EP1735749A2 (fr)
DE (2) DE102004017730B4 (fr)
WO (1) WO2005101308A2 (fr)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB0701300D0 (en) * 2007-01-24 2007-03-07 Dreampact Ltd An inspection device which may contain a payload device
WO2008114264A2 (fr) * 2007-03-21 2008-09-25 Mantis Vision Ltd Procédé et appareil de stabilisation d'images vidéo
KR101014572B1 (ko) * 2007-08-27 2011-02-16 주식회사 코아로직 영상 왜곡 보정 방법 및 그 보정 방법을 채용한 영상처리장치
US8217995B2 (en) * 2008-01-18 2012-07-10 Lockheed Martin Corporation Providing a collaborative immersive environment using a spherical camera and motion capture
DE102008023439B4 (de) 2008-05-14 2011-02-17 Christian-Albrechts-Universität Zu Kiel Augmented Reality Fernglas zur Navigationsunterstützung
GB2483224A (en) * 2010-08-26 2012-03-07 Dreampact Ltd Imaging device with measurement and processing means compensating for device motion
WO2015152923A1 (fr) * 2014-04-03 2015-10-08 Empire Technology Development Llc Correction du maculage couleur utilisant des mesures d'inertie
US9883101B1 (en) * 2014-07-23 2018-01-30 Hoyos Integrity Corporation Providing a real-time via a wireless communication channel associated with a panoramic video capture device
FR3041135B1 (fr) * 2015-09-10 2017-09-29 Parrot Drone avec camera a visee frontale avec segmentation de l'image du ciel pour le controle de l'autoexposition
US10097759B1 (en) * 2015-09-30 2018-10-09 Apple Inc. 360 degree image presentation
EP3378221B1 (fr) 2015-11-16 2022-01-12 Google LLC Stabilisation basée sur des données d'accéléromètre
US11477382B2 (en) * 2016-02-19 2022-10-18 Fotonation Limited Method of stabilizing a sequence of images
US10404915B1 (en) * 2016-04-07 2019-09-03 Scott Zhihao Chen Method and system for panoramic video image stabilization
KR102697559B1 (ko) 2016-12-22 2024-08-23 삼성전자주식회사 영상 표시 방법, 저장 매체 및 전자 장치
US10388077B2 (en) 2017-04-25 2019-08-20 Microsoft Technology Licensing, Llc Three-dimensional environment authoring and generation
TWI668541B (zh) * 2017-09-29 2019-08-11 財團法人工業技術研究院 機器人工具中心點校正系統及其方法
CN108038820B (zh) * 2017-11-14 2021-02-02 影石创新科技股份有限公司 一种实现子弹时间拍摄效果的方法、装置及全景相机
US10832377B2 (en) * 2019-01-04 2020-11-10 Aspeed Technology Inc. Spherical coordinates calibration method for linking spherical coordinates to texture coordinates

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4736436A (en) * 1984-04-13 1988-04-05 Fujitsu Limited Information extraction by mapping
US6002430A (en) * 1994-01-31 1999-12-14 Interactive Pictures Corporation Method and apparatus for simultaneous capture of a spherical image
CH687108A5 (de) * 1992-12-24 1996-09-13 Anschuetz & Co Gmbh Bildstabilisierungseinrichtung.
KR0163916B1 (ko) * 1995-08-30 1999-01-15 김광호 캠코더의 손떨림 영상의 보정장치
US5844602A (en) * 1996-05-07 1998-12-01 Recon/Optical, Inc. Electro-optical imaging array and camera system with pitch rate image motion compensation which can be used in an airplane in a dive bomb maneuver
US5798786A (en) * 1996-05-07 1998-08-25 Recon/Optical, Inc. Electro-optical imaging detector array for a moving vehicle which includes two axis image motion compensation and transfers pixels in row directions and column directions
WO2000060853A1 (fr) * 1999-04-08 2000-10-12 Internet Pictures Corporation Procede et dispositif servant a produire des effets de traitement virtuels pour des images video grand angle
US7042508B2 (en) * 2002-07-26 2006-05-09 Appro Technology Inc. Method for presenting fisheye-camera images

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2005101308A3 *

Also Published As

Publication number Publication date
WO2005101308A3 (fr) 2006-03-02
DE102004017730B4 (de) 2006-05-24
US7747105B2 (en) 2010-06-29
DE112005001396A5 (de) 2007-05-24
US20070036460A1 (en) 2007-02-15
WO2005101308A2 (fr) 2005-10-27
DE102004017730A1 (de) 2005-11-10

Similar Documents

Publication Publication Date Title
EP1735749A2 (fr) Procede de compensation de rotation d'images spheriques
DE69624550T2 (de) Gerät und Verfahren zur Extraktion von dreidimensionalen Formen
DE202017105894U1 (de) Headset-Entfernung in virtueller, erweiterter und gemischter Realität unter Verwendung einer Blick-Datenbank
EP3427474B1 (fr) Procédé de traitement d'images, moyen de traitement d'images et dispositif de traitement d'images pour générer des reproductions d'une partie d'un espace tridimensionnel
EP2880853B1 (fr) Dispositif et procédé destinés à déterminer la situation d'une caméra de prise de vue
DE112016004216T5 (de) Allgemeine Sphärische Erfassungsverfahren
DE102012108567A1 (de) Verfahren und Vorrichtung zum Erlangen von Tiefeninformationen unter Verwendung eines Lichtmusters
DE3219032A1 (de) Stereophotogrammetrisches aufnahme- und auswerteverfahren sowie auswertevorrichtung
DE102019213199A1 (de) Virtuelle räumlich registrierte video überlagerungsanzeige
EP3104330B1 (fr) Procede de suivi d'au moins un objet et procede de remplacement d'au moins un objet par un objet virtuel dans un signal d'image animee enregistre par une camera
EP3711289A1 (fr) Procédé de remise automatique en état d'un état mesuré d'un système de projection
DE112017003426T5 (de) Verfahren und System zum Rekonstruieren einer dreidimensionalen Darstellung
DE102008023439B4 (de) Augmented Reality Fernglas zur Navigationsunterstützung
DE202010017889U1 (de) Anordnung zur Aufnahme geometrischer und photometrischer Objektdaten im Raum
DE102011082280A1 (de) Bildmessvorrichtung und Bildmessverfahren
DE102007030378A1 (de) System zur Bestimmung der Lage eines Kamerasystems
DE102006055746A1 (de) Verfahren zur Korrektur einer Volumenabbildungsgleichung zur Bestimmung eines Geschwindigkeitsfeldes von Teilchen in einem Volumen
EP2859531B1 (fr) Procédé permettant de détecter des modifications à partir d'images
DE102014106718A1 (de) Verfahren und System zur Bestimmung einer gegenständlichen Lage
EP3539085B1 (fr) 3d localisation
DE202010017899U1 (de) Anordnung zur Aufnahme geometrischer und photometrischer Objektdaten im Raum
WO2007014966A1 (fr) Procede et dispositif pour determiner la position relative d'un premier objet par rapport a un second objet, programme informatique correspondant, et support d'enregistrement correspondant, lisible par ordinateur
DE102007032471A1 (de) Verfahren und Vorrichtung zur Bestimmung der Lage eines Kamerasystems relativ zu einem Objekt
DE102010004233B3 (de) Verfahren zur Bestimmung der Lage eines Kamerasystems bezüglich eines Objekts
DE102011055967A1 (de) Messverfahren und Vorrichtung zur Durchführung des Messverfahrens

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20060926

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20121101