JP2020042575A - Information processing apparatus, positioning method, and program - Google Patents

Abstract

To provide an information processing apparatus, a positioning method, and a program which can easily position even an image having less remarkable feature.SOLUTION: The present invention is directed to an information processing apparatus having: an inertial sensor for outputting an acceleration data of the information processing apparatus; a gravity direction estimating unit for estimating a first gravity direction in a world coordinate system based on the acceleration data; a model position estimating unit for calculating a first conversion matrix for changing a position and a posture of a model in a camera coordinate system in response to the first gravity direction relative to the camera coordinate system, converting the position and the posture of the model in the camera coordinate system into a position and a posture of the model in a model coordinate system, and converting the position and the posture of the model in the model coordinate system into a position and a posture of the model in the world coordinate system; a model drawing unit for converting the position and the posture of the model in the camera coordinate system into a position of the model in an image coordinate system to draw an input image and the model in the image coordinate system; and a display unit for displaying the input image and the model in response to the drawing result of the model drawing unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing device, a positioning method, and a program.

  In recent years, AR (Augmented Reality) technology has been receiving attention. AR is used as a term that refers to, for example, a technology of extending a real environment perceived by humans with a computer, and a real environment itself extended by a computer. In AR, for example, a virtual object can be projected in the real world based on the real world, and an image in which a part of the real world is extended can be projected. The AR may be used, for example, in comparison with a VR (Virtual Reality) that is based on a virtual space and does not enter the real world.

  In the AR technology, a virtual object is three-dimensionally superimposed on the real world according to the movement of a camera. For example, in an information processing apparatus to which the AR technology is applied, in each image frame, the position of a virtual object to be superimposed on the real world is determined according to the position (or direction) and attitude (or rotation) of the camera. When projecting the position of the virtual object onto the image coordinate system according to the position and orientation of the camera, obtaining information for converting the coordinate system from the real world coordinate system to the image coordinate system may be performed, for example, by using the `` position It may be called "matching".

  As an alignment method, for example, there is a marker-based alignment method. In the marker-based alignment method, for example, a binary (for example, black and white) rectangular marker is prepared in advance, the marker is imaged in the information processing apparatus, and a straight line of the marker is extracted from the captured marker image. This is a method of performing position alignment by detecting. In the marker-based alignment method, for example, a known marker is used, so that the information processing apparatus can easily detect a straight line or the like from an image and can easily perform alignment.

  However, in the marker-based alignment method, a marker is separately prepared. For this reason, the marker-based positioning method may have a higher installation cost than other methods that do not use a marker.

  Therefore, as a positioning method, there is a technique of a markerless positioning method. For example, three-dimensional 6D (Degree of freedom) is obtained from a single viewpoint RGB (Red, Green, Blue) image obtained by a camera by observing objects in various postures, acquiring learning data, storing the learning data in a memory, or the like. There is a technique for estimating the position and orientation of a subject on a learning basis.

  Also, by moving the virtual image from the virtual camera fixed in the virtual coordinate system to a position substantially matching the real video image of the object in the real space, the position of the virtual model in the virtual coordinate system is moved. Is mapped to the position of the object in the real coordinate system.

  According to this technique, an improved system and method for co-registration of a virtual image of an object to an actual position of the object can be provided.

Wadim Kehl et al., “SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again”, ICCV (IEEE (The Institute of Electrical and Electronics Engineers, Inc.) International Conference on Computer Vision), 2017

JP-T-2009-501609

  However, in the technology of estimating the position and orientation of a subject from a single viewpoint RGB image on a learning basis, it may be difficult to estimate the position and orientation of the object when the image having poor appearance characteristics is the subject. . The “image with poor appearance characteristics” is, for example, an image in which the number of corner points and edge points having strong contrast is smaller than the threshold value, or the number of patterns on the surface having strong contrast is smaller than the threshold value. . In such an image of the object, the number of features that can be extracted by image processing tends to be smaller than a threshold. In this technique, the position and orientation of the subject are estimated based on the learning data stored in the memory. If the number of features is smaller than the threshold, the learning data corresponding to the RGB image is read from the memory. The accuracy of the learning data may decrease. Therefore, in such a technique, the accuracy of the estimated position and orientation of the subject may be reduced with respect to the “image having poor appearance characteristics”.

  In addition, a technique for substantially matching a virtual image from a fixed virtual camera with a real video image is, for example, a method in which a virtual image is fixedly displayed, so that a user can substantially match a real video image. However, the camera is moved to one determined position and posture. Therefore, in such a technique, the movement of the camera is limited at the time of the alignment, and the user may not be able to easily perform the alignment.

  Therefore, an object of the present invention is to provide an information processing apparatus, a positioning method, and a program that can easily perform positioning even for an image having poor appearance characteristics.

  One disclosure is an inertial sensor that outputs acceleration data of an information processing device, a gravitational direction estimating unit that estimates a first gravitational direction in a world coordinate system from the acceleration data, and a position and orientation of a model in a camera coordinate system. Changing the position and orientation of the model in the camera coordinate system into the position and orientation of the model in the model coordinate system, respectively, by changing the position and orientation of the model in the camera coordinate system, And a model position estimating unit for calculating a first transformation matrix for converting the position and orientation into the position and orientation of the model in the world coordinate system, and the position and orientation of the model in the camera coordinate system and the position of the model in the image coordinate system. And a model drawing unit that draws the input image and the model in the image coordinate system, according to the drawing result of the model drawing unit. Te, an information processing apparatus and a display unit for displaying said model and the input image.

  According to an embodiment of the present disclosure, it is possible to easily perform positioning even for an image having poor appearance characteristics.

FIG. 1 is a diagram illustrating a configuration example of an information processing apparatus. FIG. 2 is a diagram illustrating an example of a coordinate system. Figure 3 is a diagram illustrating an example of the relationship between the viewpoint position and the gravity vector g _c of the camera. FIG. 4 is a flowchart showing an operation example. FIG. 5 is a flowchart illustrating an example of the initialization processing. FIG. 6 is a diagram illustrating an example of the relationship between the world coordinate system and the virtual camera coordinate system. FIGS. 7A and 7B are diagrams illustrating an example of the relationship between the coordinate systems. FIG. 8 is a flowchart illustrating an example of the calculation processing of the 3D model. FIGS. 9A and 9B are diagrams illustrating an example of the initialization processing. FIG. 10A shows an example of the relationship between the viewpoint position of the camera and the two points P _{1, c} , P _{2, c} , and FIG. 10B shows the two points P _{1, c} , P _2,2 when the camera viewpoint position moves _. It is a figure showing the example of a relationship of _c , respectively. FIGS. 11A and 11B are diagrams illustrating display examples of a target object and a 3D model. FIG. 12 is a diagram illustrating an example of the relationship between the viewpoint position of the camera and two points P1 _{, c} , P2 _{, c} . FIGS. 13A and 13B are diagrams illustrating a relationship example of each coordinate system. FIGS. 14A and 14B are diagrams illustrating display examples of a target object and a 3D model. FIGS. 15A and 15B are diagrams illustrating display examples of a target object and a 3D model. FIGS. 16A and 16B are diagrams illustrating display examples of a target object and a 3D model. FIG. 17 is a diagram illustrating a hardware configuration example of the information processing apparatus. FIG. 18 is a diagram illustrating a configuration example of an information processing system.

  Hereinafter, embodiments for carrying out the present invention will be described. The following embodiments do not limit the disclosed technology. The embodiments can be appropriately combined within a range that does not contradict processing contents.

[First Embodiment]
<Configuration example of information processing device>
FIG. 1 is a diagram illustrating a configuration example of the information processing apparatus 100.

  The information processing device 100 is, for example, a smartphone, a game device, an equipment inspection and management device, a navigation device, or the like.

  The information processing apparatus 100 according to the first embodiment uses an AR technique to create a model image (or a virtual model or a 3D (3-Dimension) model in the real world; hereinafter, may be referred to as a “3D model”. ) Is displayed. The information processing device 100 performs alignment using the 3D model. In this case, the information processing apparatus 100 can perform positioning with respect to the target object by using a 3D model from various camera viewpoints even when the target object has a poor appearance characteristic.

In the first embodiment, “positioning” means, for example, calculating information for linking two coordinate systems. “Positioning” in the first embodiment is, for example, calculating a matrix T _wm for converting a model coordinate system into a world coordinate system. "Positioning" may be referred to as, for example, registration. Details including the coordinate system will be described later.

  The 3D model after the positioning is displayed on the display unit 109 in a state where the 3D model matches the target object, as shown in, for example, FIG. 14B and FIG. 16B. Therefore, the user can check and manage the equipment by confirming that the target object has changed based on the positional relationship between the 3D model displayed on the display unit 109 and the target object. Alternatively, in the information processing apparatus 100, after the positioning, it is possible to display a three-dimensional navigation image on the display unit 109 or to display a three-dimensional character of the game.

  As shown in FIG. 1, the information processing apparatus 100 includes an imaging unit 101, an inertial sensor 102, a storage unit 103, a self-position estimation unit 104, a gravity direction estimation unit 105, an initialization processing unit 106, a model position estimation unit 107, a model It includes a drawing unit 108, a display unit 109, a recognition start determination unit 110, and an object position recognition unit 111.

  The imaging unit 101 captures an image including a target object, uses the captured image as an input image, and outputs image data of the input image. The imaging unit 101 stores the image data in the storage unit 103. The image data is, for example, RGB image data having each plane of RGB (Red Green Blue).

  Inertial sensor 102 measures the acceleration of information processing device 100 and outputs the measured acceleration as acceleration data. The inertial sensor 102 stores the acceleration data in the storage unit 103. The inertial sensor 102 may be, for example, an acceleration sensor, a gyro sensor, or the like.

The storage unit 103 is, for example, a memory, and stores RGB image data, acceleration data, 3D model data, and various setting values. The 3D model data includes, for example, position information indicating a position of the 3D model in the world coordinate system and RGB data at the position. In addition, the various setting values include, for example, arbitrary two points P _{1, m} , P _{2, m} of the 3D model in the virtual camera coordinate system. Note that coordinate systems such as the world coordinate system and the virtual camera coordinate system will be described later.

The self-position estimating unit 104 estimates the position and orientation of a real camera (for example, the imaging unit 101) in the world coordinate system based on the RGB image data read from the storage unit 103. For example, the self-position estimating unit 104 calculates camera parameters representing the position and orientation of a real camera from RGB image data of a plurality of image frames using Simultaneous Localization and Mapping (SLAM). Further, the self-position estimating unit 104 calculates, for example, a matrix including camera parameters. The matrix including the camera parameters can be, for example, a transformation matrix T _cw from the world coordinate system to the real camera coordinate system. Details will be described in an operation example. Self-position estimating section 104 outputs transformation matrix T _{cw and the} like to model drawing section 108 and model position estimating section 107.

  The gravity direction estimation unit 105 estimates the gravity direction in the world coordinate system based on the acceleration data read from the storage unit 103. For example, the gravitational direction estimating unit 105 estimates the gravitational direction from the acceleration data using an equation stored in the internal memory. Details will be described in an operation example. The gravity direction estimation unit 105 outputs the estimated gravity direction to the model position estimation unit 107.

The initialization processing unit 106 sets a virtual camera coordinate system using the 3D model data read from the storage unit 103 and various setting values. As various setting values, for example, an arbitrary two points P _{1, m} , P _{2, m} on the 3D model, a Euclidean distance L between the two points P _{1, m} , P _{2, m} and a two point P _{1, m} , there is a normal vector _{n c,} and the vertically downward direction vector _{g m} of a plane including the _{P 2, m.} Then, the initialization processing unit 106 calculates unit line-of-sight vectors r ₁ and r ₂ in the virtual camera coordinate system. The initialization processing unit 106 outputs the calculated unit line-of-sight vectors r ₁ and r ₂ and various numerical values in the virtual camera coordinate system to the model position estimation unit 107. Details will be described in an operation example.

The model position estimating unit 107 uses the 3D model data read from the storage unit 103, the gravitational direction, the unit line-of-sight vectors r ₁ and r ₂ , the transformation matrix T _cw from the world coordinate system to the real camera coordinate system, and the like, to obtain model coordinates. Estimate the position and orientation of the 3D model in the system. Specifically, for example, the model position estimating unit 107 calculates a conversion matrix T _wm that converts the position and orientation of the 3D model in the model coordinate system into the position and orientation of the 3D model in the world coordinate system. The model position estimating unit 107 converts, for example, the position and orientation of the 3D model in the world coordinate system into the position and orientation of a 3D model in the camera coordinate system, and converts the position and orientation of the 3D model in the model coordinate system. Or perform a process. Details will be described in an operation example. The model position estimating unit 107 outputs the position and orientation of the 3D model in the camera coordinate system to the model drawing unit 108, and outputs the position and orientation of the 3D model in the model coordinate system, the transformation matrix _{Twm, and the} like to the object position recognizing unit 111. Output to

  The conversion of the position and orientation of the 3D model into another coordinate system is performed using, for example, a conversion matrix. Numerical values for conversion to a position and orientation in another coordinate system are included in each component of the conversion matrix. Hereinafter, for example, a description will be given on the assumption that the position and orientation of a 3D model in a certain coordinate system can be converted into the position and orientation of a 3D model in another coordinate system by using one transformation matrix. The transformation matrix converts the position and orientation of the 3D model from one coordinate system to another coordinate system. For example, this is because the 3D model data in one coordinate system is converted to the 3D model data in another coordinate system. May be described as being converted to The position and orientation of the 3D model may be referred to as, for example, 3D model data.

The model drawing unit 108 draws a 3D model on the input image. Specifically, the model drawing unit 108 performs the following processing. That is, the model rendering unit 108, the 3D model data of the camera coordinate system, using the projection matrix T _p, converted into 3D model data of the image coordinate system. The model drawing unit 108 draws the RGB image data and the 3D model data in an image coordinate system. At this time, the model drawing unit 108 changes the RGB image data at the position of the 3D model in the image coordinate system to the image data of the 3D model. The model drawing unit 108 outputs the RGB image data and the 3D model image data in the image coordinate system to the display unit 109 and the recognition start determination unit 110.

  The display unit 109 displays the input image and the 3D model based on the RGB image data output from the model drawing unit 108 and the image data of the 3D model. The position of the 3D model displayed on the display unit 109 changes according to the direction of gravity in the camera coordinate system. The user moves the information processing device 100 (or the imaging unit 101) while watching the 3D model that changes in this way on the display unit 109, matches the target object included in the input image with the 3D model, and sets the “position Matching ". When such a match occurs, examples of the target object and the 3D model displayed on the display unit 109 are, for example, FIGS. 14B and 16B.

  Returning to FIG. 1, the recognition start determination unit 110 determines whether to start positioning the target object based on, for example, whether or not the user has pressed the operation button of the information processing apparatus 100. When the recognition start determining unit 110 determines that the user has performed the positioning start determination, the recognition start determining unit 110 notifies the object position recognizing unit 111 of the determination.

The object position recognizing unit 111, for example, upon receiving a notification indicating that the start of alignment has been determined, converts the transformation matrix T _wm from the model coordinate system to the world coordinate system at the time of receiving the notification, from the model position Received from the estimation unit 107. The object position recognition unit 111 uses the conversion matrix _Twm to convert 3D model data in the model image system into 3D model data in the world coordinate system or convert 3D model data in the world coordinate system into the camera coordinate system. To the 3D model data of. The object position recognition unit 111 outputs the 3D model data of the camera coordinate system to the model drawing 108. The model rendering unit 108, with respect to the 3D model data, converts the 3D model data to the image coordinate system using the projection matrix T _p, to draw the 3D model data to the image coordinate system. The display unit 109 displays the input image and the 3D model according to the drawing result.

<About each coordinate system>
FIG. 2 is a diagram illustrating an example of each coordinate system. In the first embodiment, the world coordinate system (X, Y, Z), the virtual camera coordinate system ( _Xv , _Yv , _Zv ), the real camera coordinate system ( _Xc , _Yc , _Zc ), model coordinate system _{(X m, Y m, Z} m) there are five coordinate system, and image coordinate system (x, y).

World coordinate system (X, Y, Z) are present origin O _w in the world coordinate system at an arbitrary position. The target object may be located at a fixed position in the world coordinate system (X, Y, Z). Further, the 3D model is also located at a specific position in the world coordinate system (X, Y, Z).

Further, the virtual camera coordinate system at an arbitrary position in the world coordinate system _{(X v, Y v, Z} v) origin _{O v} is present in the origin _{O v} virtual camera coordinate system with respect to the _{(X v, Y} v, Z _v ). The origin _Ov is, for example, the viewpoint position of the virtual camera.

Actual camera coordinate system _{(X c, Y c, Z} c) for also, the origin _{O c} is present at any position in the world coordinate system, the actual camera coordinate system around the origin _{O c (X c, Y c} , Z _c ). The origin _Oc is, for example, the viewpoint position of the real camera. Hereinafter, the origin O _v, the O _c, may be referred to each and viewpoint position of the virtual camera and the real cameras.

The position and orientation of the 3D model in the virtual camera coordinate system ( _Xv , _Yv , _Zv ) and the position and orientation of the 3D model in the real camera coordinate system ( _Xc , _Yc , _Zc ) are the viewpoint positions. Are different and therefore different. Moreover, the actual camera coordinate system _{(X c, Y c, Z} c) the viewpoint position _{O c} of is movable in the world coordinate system (X, Y, Z).

Actual camera coordinate system _{(X c, Y c, Z} c) and the origin _{O c} of, on a line connecting the center of the 3D model, the origin o of the image coordinate system (x, y) is present. The image coordinate system (x, y) is located between the real camera coordinate system ( _Xc , _Yc , _Zc ) and the 3D model in the world coordinate system (X, Y, Z). A viewpoint position O _c of the real camera, the distance between the origin o of the image coordinate system (x, y), for example, it called the focal length f, to maintain a constant distance in the world coordinate system.

Further, there is a model coordinate system (X _m , Y _m , Z _m ) having an origin O _m at a specific position on the 3D model (in the example of FIG. 2, the upper right corner in the drawing).

As shown in FIG. 2, the target object, the world coordinate system (X, Y, Z) is the gravity vector g _w in work. In FIG. 2, since the gravity vector g _w acts in the negative direction in the Y-axis direction, it is represented as −g _w . The gravity vector g _w also works for the 3D model.

In the first embodiment, the actual camera coordinate system _{(X c, Y c, Z} c) in, there is a gravity vector _{g c.}

Figure 3 is a diagram representing the viewpoint position _{O c} of the real camera, real camera coordinate system _{(X c, Y c, Z} c) an example of the relationship between the gravity vector _{g c} in. As shown in FIG. 3, the viewpoint position _{O c,} is moved in a direction to look up a 3D model, the gravity vector (-g _c) are inclined in the direction of the viewpoint position _{O c.} On the other hand, in FIG. 2, the viewpoint position O _c, be moved in a direction closer to a 3D model, the direction of the gravity vector (-g _c) hardly changes.

That is, the gravity vector g _c in the real camera coordinate system (X _c , Y _c , Z _c ) is, for example, the viewpoint position of the real camera while maintaining the same direction with respect to the direction of the gravity vector g _w (gravity direction). When _Oc moves in the world coordinate system (X, Y, Z), its direction does not change. On the other hand, when the viewpoint position O _c of the real camera moves in the direction of changing its direction with respect to the direction of the gravity vector g _w (the direction of gravity), the direction of the gravity vector g _c changes. Thus, the direction of the gravity vector g _c is the movement of the viewpoint position O _c of the real camera, it may vary, with the feature that. In the first embodiment, by the movement of such a viewpoint position O _c, if the direction of the gravity vector g _c varies, depending on the change, the position and orientation of the real camera coordinate system of the 3D model I try to change it. Details will be described in an operation example.

  In the following, the actual camera coordinate system may be referred to as a camera coordinate system, for example.

<Operation example>
FIG. 4 is a flowchart illustrating an operation example of the information processing apparatus 100.

  When the processing is started (S10), the information processing apparatus 100 performs an initialization processing (S11).

FIG. 5 is a flowchart illustrating an operation example of the initialization processing. The initialization processing is performed by, for example, the initialization processing unit 106, and the unit line-of-sight vectors r ₁ and r ₂ in the virtual camera coordinate system are calculated.

When the initialization processing unit 106 starts the initialization processing (S110), any two points P1 _{, v} , P2 _{, v} (P1 _, vｖP2 _{, v} ) of the 3D model in the virtual camera coordinate system are determined. It is determined (S111).

FIG. 6 is a diagram illustrating a relationship example between the world coordinate system (X, Y, Z) and the virtual camera coordinate system ( _Xv , _Yv , _Zv ). As shown in FIG. 6, arbitrary two points P1 _{, v} , P2 _{, v} represent arbitrary two points on the 3D model in the virtual camera coordinate system ( _Xv , _Yv , _Zv ). For example, the initialization processing unit 106 reads the position coordinates of the two points P _{1, v} , P _{2, v} stored in the storage unit 103 from the storage unit 103, and thus, the arbitrary two points P _{1, v} , P2 _{, v} may be determined.

Returning to FIG. 5, the initialization processing unit 106 determines a Euclidean distance L between the two points P _{1, v} , P _{2, v} (S111). For example, the initialization processing unit 106 may calculate L by calculating the distance based on the position coordinates of the two points P _{1, v} , P _{2, v} , or may be stored in the storage unit 103. Alternatively, it may be determined by reading out the L.

Furthermore, the initialization processing unit 106, the two points _P 1, _v, determines the normal vector _{n v} of suitable plane containing _{P 2, v} (S111). As shown in FIG. 6, the normal vector n _v may be the normal vector to the surface with a 3D model. For example, the initialization processing unit 106, the point _P 1, _v, based on the position coordinates of _{P 2, v,} may be calculated normal vector _{n v,} information from the storage unit 103 of the normal vector _{n v} by reading, it may determine the normal vector n _v.

Returning to FIG. 5, further, the initialization processing unit 106, the virtual camera coordinate system _{(X v, Y v, Z} v) determining the vertically downward vector _{g v} in (S 111). The vertical downward vector _gv can be, for example, a gravity vector in a virtual camera coordinate system ( _Xv , _Yv , _Zv ), as shown in FIG. For example, the initialization processing unit 106, by reading the information of the vertically downward direction vector g _v from the storage unit 103 may be determined. The virtual camera coordinate system _{(X v, Y v, Z} v) is also in the vertically downward vector _{g v,} similarly to the gravity vector _{g c,} for example, as shown in FIG. 3, in accordance with the viewpoint position _{O v} of the virtual camera , May change.

Returning to FIG. 5, then the initialization processing unit 106 is located and the two points from the orientation _P 1, _v, defines the virtual camera to observe the _{P 2, v} (S112). For example, the initialization processing unit 106, as shown in FIG. 6, the world coordinate system (X, Y, Z) any position O _v of reads from the storage unit 103, setting a viewpoint position of the virtual camera Defines a virtual camera.

Returning to FIG. 5, next, the initialization processing unit 106 calculates the unit line-of-sight vectors r ₁ , r ₂ (r ₁ ≠ r ₂ ) in the virtual camera coordinate system (X _v , Y _v , Z _v ) (S113). ). For example, the initialization processing unit 106 connects the position coordinate of the virtual camera viewpoint position _{O v} (0,0,0), 2 points _P 1, v virtual model coordinate _system, the position coordinates of _{P 2, v} The position coordinates of two points where the length of the line segment is “1” may be calculated, and the position coordinates may be used as the components of the unit line-of-sight vectors r ₁ and r ₂ .

Then, the initialization processing unit 106 ends the initialization processing (S114). The initialization processing unit 106 outputs the calculated unit line-of-sight vectors r ₁ and r ₂ , the determined normal vector n _v , and the vertical downward vector g _v to the model position estimation unit 107.

  Returning to FIG. 4, next, the information processing apparatus 100 acquires RGB image data and acceleration data (S12). For example, the imaging unit 101 stores RGB image data of a captured input image in the storage unit 103, and the inertial sensor 102 stores acceleration data measured when the input image is captured in the storage unit 103.

  Note that the information processing apparatus 100 performs the processing from S12 to S17 for each image frame of the image captured by the imaging unit 101. Therefore, the information processing apparatus 100 acquires RGB image data for each image frame, or acquires acceleration data from the inertial sensor 102 for each image frame.

Next, the information processing apparatus 100 estimates the position and orientation of the camera (S13). For example, the self-position estimating unit 104 obtains camera parameters using SLAM, estimates the position and orientation of the real camera, and calculates a transformation matrix T _cw including the obtained camera parameters.

  Here, SLAM will be described. SLAM is, for example, based on a plurality of images (two-dimensional) captured by the same camera, extracting and tracking feature points of the image, thereby recognizing a three-dimensional structure around the camera and recognizing the position of the camera. And the calculation of the posture at the same time.

  As the SLAM processing, the self-position estimating unit 104 performs, for example, the following processing.

  That is, first, the self-position estimating unit 104 reads the RGB image data from the storage unit 103 and extracts feature points from a plurality of images (or image frames) indicated by the RGB image data. For example, the self-position estimating unit 104 extracts a feature point for each image using a known method such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Feature).

  Next, the self-position estimating unit 104 performs matching in each image of feature points extracted in each image. At this time, the self-position estimating unit 104 may perform the matching by a known method used for feature point extraction.

Then, the self-position estimating unit 104 calculates three-dimensional coordinates of the feature points based on the matching result, and calculates camera parameters corresponding to each image from the calculated three-dimensional coordinates. The camera parameters include, for example, the position coordinates of the camera and the rotation angle of the coordinate axes. The self-position estimating unit 104 calculates a transformation matrix T _cw including the camera parameters. Since the transformation matrix T _cw includes, for example, the position coordinates (or position) of the camera and the rotation angle (or posture) of the coordinate axes, an arbitrary position and posture in the world coordinate system (X, Y, Z) can be calculated. It can be a conversion matrix for converting into a position and orientation in a camera coordinate system ( _Xc , _Yc , _Zc ) with the viewpoint position as the origin.

FIG. 7A is a diagram illustrating a relationship example of each coordinate system. The self-position estimating unit 104 calculates the position and orientation of the camera by using the SLAM, thereby converting the world coordinate system (X, Y, Z) to the real camera coordinate system ( _Xc , _Yc , _Zc ). The transformation matrix T _cw is calculated.

  The example of the SLAM processing has been described above. Examples of the SLAM include an EKF-based SLAM using an EKF (Extended Kalman Filter) and a SLAM using a particle filter. In the first embodiment, for example, any method of SLAM may be used.

  Returning to FIG. 4, next, the information processing apparatus 100 calculates the direction of gravity (S14). For example, the gravity direction estimation unit 105 performs the following processing.

That is, the gravity direction estimation section 105, read from the storage unit 103, the camera coordinate system _{(X c, Y c, Z} c) acceleration data in respective axis directions of the (Ax, Ay, Az) from using the following equation Then, the inclination (θ, ψ, φ) with respect to each axis direction is calculated.

Then, the gravity direction estimation unit 105 estimates the gravity direction based on the inclination (θ, ψ, φ). For example, the gravity direction estimation unit 105, the gradient is (0, -1,0) when the camera coordinate system _{(X c, Y c, Z} c) and the -Y _c-axis direction is the direction of gravity estimated and, slope, the camera coordinate system _{(X c, Y c, Z} c) estimates that there is a direction of gravity _{X c-axis} direction when the (1,0,0). The gravitational direction estimating unit 105 determines that the estimated gravitational direction is the gravitational direction in the world coordinate system (X, Y, Z), and determines the direction of the gravitational vector g _{w in} the world coordinate system (X, Y, Z) (or the gravitational direction). Direction).

For example, the gravity direction estimating unit 105 stores the formulas (1) to (3) in the internal memory, reads out the data at the time of this processing, and reads the acceleration data read from the storage unit 103 from the formulas (1) to (3). ) To calculate the slope. Then, the gravity direction estimation unit 105 estimates the direction of the gravity vector g _w based on the inclination.

  The above is the calculation method of the direction of gravity.

  Next, the information processing apparatus 100 performs calculation processing of the position and orientation of the 3D model (hereinafter, may be referred to as “3D model calculation processing”) (S15).

  FIG. 8 is a flowchart illustrating an operation example of the calculation processing of the 3D model.

Model position estimation unit 107 starts the calculation process of the 3D model (S150), the direction of the gravity vector g _c in the camera coordinate system, the law so that the direction of the vertically downward vector g _v in the virtual model coordinate system is coincident The line _nm is rotated (S151).

FIGS. 9A and 9B are diagrams for explaining the processing of S151. In this process, calculates the virtual camera coordinate system _{(X v, Y v, Z} v) by rotating the normal vector _{n v} in the camera coordinate system _{(X c, Y c, Z} c) the normal vector _{n c} in I do. At that time, the model position estimating section 107 is calculated using a camera coordinate system _{(X c, Y c, Z} c) and the gravity vector _{g c} of the vertically downward direction vector _{g v} of the virtual camera coordinate system. The model position estimating unit 107 performs, for example, the following calculation.

により、回転軸ｖと法線ベクトルｎ_ｃとを計算する。 That is, the model position estimating unit 107, the rotation axis v, when the rotation angle to rotate about the axis of rotation v of the normal vector n _v to the normal vector n _c and theta,

Accordingly, to calculate the rotation axis v and the normal vector n _c.

  Next, the model position estimating unit 107 calculates a rotation matrix R for rotating around the rotation axis v by the angle θ using the following equation.

The model position estimation unit 107 uses the rotation matrix R, using the following formula, the camera coordinate system _{(X c, Y c, Z} c) calculating a normal vector _{n c} in.

Note that the model position estimating unit 107 uses the transformation matrix T _cw obtained in the camera position estimating process (S13 in FIG. 4) and the gravity vector g _w obtained in the gravity direction calculating process (S14) to The gravitational vector g _c is calculated by the equation.

For example, the model position estimating unit 107 stores the formulas (4) to (8) in the internal memory, reads out the formulas (8) to (8) from the internal memory at the time of this processing, and sets the vertical downward vector g _v etc. and gravity vector _{g c,} such as by substituting the equation (4) into equation (8), to calculate the normal vector _{n c.}

Returning to FIG. 8, next, the model position estimating unit 107 calculates the scales t ₁ and t ₂ of the fixed line of sight (S152).

FIG. 10A is a diagram illustrating an example of the scales t ₁ and t ₂ . The scales t ₁ and t ₂ are obtained by extending the unit line-of-sight vectors r ₁ and r ₂ obtained by the initialization process from the viewpoint position O _{c in} the camera coordinate system (X _c , Y _c , Z _c ), for example, to obtain two points. It represents the scale for the unit line-of-sight vectors r ₁ , r ₂ when extending to two points P _{1, c} , P _{2, c} on the 3D model in which the distance between them is L.

The model position estimating unit 107 calculates the scales t ₁ and t ₂ using, for example, the following equations.

  Here, α is represented by the following equation.

For example, the model position estimating unit 107 stores the expression (11) from equation (9) in the internal memory, is read from the internal memory during processing, such as an expression normal vector _{n c} obtained in S151 (9) By substituting into Equation (11), scales t ₁ and t ₂ are obtained.

Returning to FIG. 8, next, the model position estimating unit 107 calculates two points P _{1, c} , P _{2, c} of the 3D model in the camera coordinate system using the following formula (S153).

_{P 1, c = t 1 r} 1, P 2, c = t 2 r 2 ··· (12)
For example, the model position estimating unit 107 stores the formula (12) in the internal memory, reads out the formula from the internal memory at the time of processing, and substitutes the scales t ₁ and t ₂ calculated in S152 into the formula (12). , Two points P _{1, c} and P _{2, c} are obtained.

Here, the camera coordinate system _{(X c, Y c, Z} c) any two points _P 1, c _in, and _{P 2, c,} the camera coordinate system _{(X c, Y c, Z} c) viewpoint in the position _{O c} Will be described.

FIG. 10B is a diagram illustrating an example of the relationship. As described above, the gravitational vector g _c in the camera coordinate system (X _c , Y _c , Z _c ) is obtained when the viewpoint position O _{c of the} camera changes its direction with respect to the direction of the gravitational vector g _w (gravity direction). The direction changes.

For example, as shown in FIG. 10 (B), consider a case where the viewpoint position of the camera is moved from O _c to O _'c. Just viewpoint position _{O c} is, with respect to the 3D model, the world coordinate system (X, Y, Z) in the Y-axis direction (the sky direction of the 3D model), a case where the movement.

In this case, as in the case of FIG. 3, the gravity vector g _c in the camera coordinate system (X _c , Y _c , Z _c ) changes its direction with respect to the direction of the gravity vector g _w. Change. Therefore, different from the gravity vector g _c when the gravity vector g _c when the viewpoint position of the camera is O _c, the viewpoint position of the camera is in the O _'c. This difference, as shown in equation (4) and (5), when the rotation axis v and the rotational angle θ of when the viewpoint position of the camera is O _c, the viewpoint position of the camera is in the O _'c The rotation axis v and the rotation angle θ are different. The difference in the point of view of the camera position, when the rotation axis v and the rotational angle θ are different, also becomes different rotation matrix R shown in equation (6), as a result, also be different normal vector n _c. Scale _t 1, _{t 2,} as shown from equation (9) into equation (11), because it contains the normal vector _{n c,} the difference in the viewpoint position of the camera described above, different scales _t 1, _{t 2} also It will be. Due to the difference between the scales t ₁ and t ₂ , as shown in FIG. 10B, when the viewpoint position of the camera moves from O _c to O ′ _c , the camera coordinate system (X _c , Y _c , Z _c ) The two points P1 _{, c} , P2 _{, c} move to the two points P'1 _{, c} , P'2 _{, c} , respectively. Thus, for example, from the view point O _'c, as compared with the case of the viewpoint position O _c, a state in which the upper surface of the 3D model appear larger.

Figure 11 (A) and FIG. 11 (B) when the viewpoint position of the camera is changed from _{O c} to O _'c, is a diagram illustrating a display example of a 3D model. FIG. 11B shows that the top surface of the 3D model is displayed larger than in FIG. 11A.

As described above, in the first embodiment, the position and the posture (for example, two points P _{1, c} , P _{2, c} ) of the 3D model in the camera coordinate system (X _c , Y _c , Z _c ). Is changed in accordance with the direction of gravity from the camera coordinate system (for example, the gravity vector g _c ). Such relationships, the information processing apparatus 100 includes, as shown in FIG. 11 (A) and FIG. 11 (B), the position and orientation of the 3D model are changed in accordance with the position and orientation of the camera viewpoint position O _c .

Referring back to FIG. 8, the model position estimating unit 107 next calculates the camera vector based on the gravity vector g _{c in} the camera coordinate system (X _c , Y _c , Z _c ), the normal vector n _c , and the model width W. coordinate system _{(X c, Y c, Z} c) model coordinate system _{(X m, Y m, Z} m) to, to calculate the transform matrix _{T mc} of converting the coordinate system (S154). The position of the transformation matrix T _{mc in} the entire coordinate system is, for example, as shown in FIG. Incidentally, the model the width W is, for example, as shown in FIG. 12, from the point P _{1, c} in the camera coordinate system, represents the distance to the origin O _m of the model coordinate system, 3D in X _m-axis direction of the model coordinate system Represents the length of the model.

For example, the model position estimating unit 107 may calculate a matrix including all or a part of the gravity vector g _c , the normal vector n _c , and the model width W in all components of the transformation matrix T _mc . Alternatively, for example, the model position estimating unit 107 determines that some components of the transformation matrix T _mc are numerical values, and that the other components include all or some of the gravity vector g _c , the normal vector n _c , and the model width W May be calculated. Alternatively, the model position estimating unit 107 stores the transformation matrix T _mc in, for example, an internal memory, and converts the gravity vector g _c , the normal vector n _c , and the model width W into each component of the transformation matrix T _mc . by substituting all or part of, may be obtained a transformation matrix T _mc.

Note that the model width W is stored in, for example, the internal memory of the storage unit 103 or the model position estimating unit 107, and is read from the storage unit 103 or the internal memory from the model position estimating unit 107 to calculate the transformation matrix _Tmc . You may do so.

Returning to FIG. 8, the model position estimating unit 107 next calculates the model coordinate system (X _m , P _m) based on the camera position in the world coordinate system and the two points P _{1, c} , P _{2, c} on the 3D model. Y _m, the world coordinate system _{Z m)} (X, Y, to Z), calculates a transformation matrix _{T wm} for converting the coordinate system (S155).

Figure 12 is a diagram showing the camera coordinate system _{(X c, Y c, Z} c) a model coordinate system _{(X m, Y m, Z} m) an example of the relationship. In this process, the two points _P 1, c on the 3D _{model, P 2, c,} the camera coordinate system _{(X c, Y c, Z} c) in terms of the model coordinate system _{(X m, Y m, Z} m ) _Is calculated. Just from the point of view of the camera position _{O c,} the 3D model to a view point _{O m,} when changing the viewpoint position, the two points on the 3D model, two points of a model coordinate system _P 1, _{m, P 2, m} The conversion matrix T _wm in the case of conversion to is calculated.

As shown in FIG. 7 (B), the processing of S154, the camera coordinate system _{(X c, Y c, Z} c) model coordinate system _{(X m, Y m, Z} m) computing a transformation matrix _{T wm} to did. Further, a transformation matrix T _cw from the world coordinate system (X, Y, Z) to the camera coordinate system (X _c , Y _c , Z _c ) was calculated by the self-position estimation processing (S13 in FIG. 4). In this process, by utilizing this relationship, the model coordinate system _{(X m, Y m, Z} m) to compute the transformation matrix _{T wm} from the world coordinate system (X, Y, Z) to.

That is, the model position estimating unit 107 calculates the transformation matrix _Twm using the following equation.

For example, the model position estimating unit 107 reads expression (13) from the internal memory, and substitutes the conversion matrix T _cw calculated in S13 and the conversion matrix T _mc calculated in S154 into expression (13). , To obtain a transformation matrix T _wm .

  Returning to FIG. 8, when the process of S155 ends, the model position estimating unit 107 ends the 3D model calculation process (S156).

  The 3D model calculation processing (S15 in FIG. 4) has been described above.

The model position estimating unit 107 uses the transformation matrix T _cw received from the self-position estimating unit 104 to convert the 3D model data in the world coordinate system (X, Y, Z) read from the storage unit 103 into the camera coordinate system. Conversion to 3D model data in ( _Xc , _Yc , _Zc ). The model position estimating unit 107 outputs the 3D model data in the camera coordinate system ( _Xc , _Yc , _Zc ) to the model drawing unit 108.

Further, the model position estimating unit 107, by using the transformation matrix _{T mc} calculated in S154, the camera coordinate system _{(X c, Y c, Z} c) the 3D model data in the model coordinate system _(X m, _{Y m} , Z _m ). Model position estimation unit 107, a model coordinate system _{(X m, Y m, Z} m) and 3D model data in, and a transformation matrix _{T wm} calculated in S155, and outputs it to the object position recognizing unit 111.

  Returning to FIG. 4, next, the information processing apparatus 100 draws a 3D model on the camera video (S16). For example, the model drawing unit 108 performs the following processing.

That is, as shown in FIG. 13A, the model drawing unit 108 calculates the projection matrix of the 3D model data in the camera coordinate system ( _Xc , _Yc , _Zc ) received from the model position estimating unit 107. using T _p, converted into 3D model data of the image coordinate system (x, y). Then, the model drawing unit 108 draws the RGB data (or the camera image) read from the storage unit 103 and the converted 3D model data in the image coordinate system (x, y). At this time, the model drawing unit 108 draws the 3D model by changing the RGB image data of the input image at the position of the 3D model in the image coordinate system (x, y) to the image data of the 3D model. The model drawing unit 108 outputs the drawing result to the display unit 109 and the recognition start determination unit 110. The display unit 109 displays an image in which the 3D model is captured in the camera video according to the drawing result.

  For example, FIG. 11A, FIG. 11B, and FIG. 14A show examples of a camera image and a 3D model displayed on the display unit 109. The camera video includes a target object (in the example of FIG. 11A, a tissue box), and the user operates the imaging unit 101 (or the information processing apparatus 100) so that the 3D model matches the target object. By moving, "positioning" is performed.

  Returning to FIG. 4, next, the information processing apparatus 100 determines whether or not the user's determination operation has been performed (S17). The deciding operation is, for example, an operation of pressing an operation button or the like of the information processing apparatus 100 when the user determines that the target object and the 3D model match in the video displayed on the display unit 109. For example, FIG. 14B illustrates an example of the 3D model displayed on the display unit 109. When the user determines that the 3D model matches the target object, the user presses a predetermined operation button. When receiving a signal indicating that the operation button has been pressed from the operation button, the recognition start determination unit 110 determines that the determination operation has been performed (Yes in S17), and when the signal has not been received, the determination operation is performed. It is determined that it has not been performed (No in S17). When the determination operation is not performed, as illustrated in FIG. 4, the information processing apparatus 100 proceeds to S12 and repeats the processing from S12 to S17. Instead of the operation buttons, for example, the determination operation may be performed by operating the touch panel displayed on the display unit 109.

When determining that the determination operation has been performed (Yes in S17), the information processing apparatus 100 calculates the position and orientation of the target object (S18). Specifically, the object position recognizing unit 111 performs “alignment” by acquiring the transformation matrix _Twm received from the model position estimating unit 107 when the determination operation is performed, for example. The transformation matrix _Twm when the determination operation is performed is, for example, when the position and orientation of the target object in the world coordinate system and the position and orientation of the 3D model in the world coordinate system have a certain correspondence (or mapping). ). As the correspondence relationship, for example, there is a relationship where the target object and the 3D model match in the world coordinate system. It can be said that the object position recognizing unit 111 has obtained the transformation matrix _Twm from the model position estimating unit 107 when there is such a correspondence.

Then, as shown in FIG. 13B, the object position recognition unit 111 converts the transformation matrix T _wm acquired by “positioning”, the transformation matrix T _cw received from the model position estimation unit 107, and the projection matrix T _p. Is used to perform coordinate conversion in the order of (1) to (3).

Specifically, the information processing apparatus 100 performs, for example, the following processing. That is, the object position identification unit 111, received from the model position estimation unit 107, a model coordinate system _{(X m, Y m, Z} m) of the 3D model data in, received from the model position estimating unit 107 at the timing of determining operation Using the conversion matrix _Twm , the data is converted into 3D model data in the world coordinate system (X, Y, Z) ((1) in FIG. 13B). Further, the object position recognizing unit 111 converts the 3D model data of the world coordinate system (X, Y, Z) into the camera coordinate system (X _c , Y _c ) using the transformation matrix T _cw received from the self-position estimating unit 104. , _Zc ) into 3D model data ((2) in FIG. 13B). The object position recognition unit 111 outputs the 3D model data in the camera coordinate system ( _Xc , _Yc , _Zc ) to the model drawing unit 108. The model rendering unit 108, using the projection matrix _{T p,} received from the object position recognizing unit 111, the camera coordinate system _{(X c, Y c, Z} c) the 3D model data in the image coordinate system (x, y) (3) in FIG. 13 (B). Then, the model drawing unit 108 draws the converted 3D model data and the RGB image data read from the storage unit 103 in the image coordinate system (x, y).

Incidentally, the model rendering unit 108, by using the projection matrix _{T p, x = -fX c /} Z c, by _y = -fY _c / _Z c, the point of the camera coordinate system _{(X c, Y c, Z} c ) Is converted to a point (x, y) in the image coordinate system. Model rendering unit 108, for example, stores the projection matrix _{T p} in the internal memory, reads the time of processing, the camera coordinate system _{(X c, Y c, Z} c) By applying the 3D model data in , 3D model data in the image coordinate system (x, y) is obtained.

As shown in FIG. 13B, coordinate conversion is performed in the order of (1) to (3), and the object position recognition unit 111 generates 3D model data in the world coordinate system (X, Y, Z). . By the subsequent coordinate conversion, the information processing apparatus 100 converts the 3D model data from the world coordinate system (X, Y, Z) to the image coordinate system ( _Xc , _Yc , _Zc ) via the camera coordinate system ( _Xc , _Yc , _Zc ). x, y). As described above, after “alignment”, the 3D model data is converted from the 3D model data to the image coordinate system (x, y) via the world coordinate system (X, Y, Z). Therefore, the 3D model data is displayed on the display unit 109 in correspondence with the world coordinate system (X, Y, Z) after “alignment”. For example, after “alignment” is performed in FIG. 14B, the target object “tissue box” and the 3D model data are displayed on the display unit 109 in a state where they match. Even when the position and orientation of the camera are changed, the 3D model is displayed in a state in which it matches the target object.

FIGS. 15A to 16B show examples of 3D models having different shapes. The target object also represents an example of the “server device”. As shown in FIG. 16 (B) from FIG. 15 (A), 3D model, depending on the gravity vector _{g c,} is changing its position and orientation.

Thus, 3D model in the first embodiment, depending on the gravity vector g _c, the position and orientation changes. Therefore, compared to the case where the 3D model is fixed on the display unit 109, the degree of freedom for the user to move the camera is increased, and “positioning” can be easily performed according to the position and orientation of the camera. it can.

In addition, in the “positioning” in the information processing apparatus 100, the process of detecting the feature point of the image data of the 3D model or the process of detecting the feature point of the target object is not performed in the processes from S14 to S18. . Therefore, even if the image of the target object is “poor in appearance”, such as the “tissue box” shown in FIG. 11A or the “server device” shown in FIG. 15A, the gravity vector g _c Since the 3D model that changes according to the position is used, it is possible to perform “alignment” with high accuracy.

[Other embodiments]
FIG. 17 is a diagram illustrating a hardware configuration example of the information processing apparatus 100.

  The information processing apparatus 100 further includes a camera 120, a memory 121, a CPU (Central Processing Unit) 122, a ROM (Read Only Memory) 123, and a RAM (Random Access Memory) 124.

  The memory 121 corresponds to, for example, the storage unit 103 in the first embodiment.

  Further, the CPU 122 reads the program stored in the ROM 123, loads the program into the RAM 124, and executes the loaded program. Thereby, the CPU 122 determines that the self-position estimating unit 104, the gravity direction estimating unit 105, the initialization processing unit 106, the model position estimating unit 107, the model drawing unit 108, the display unit 109, the recognition start determining unit 110, and the object position recognizing unit 111 functions are realized. The CPU 122 includes, for example, the self-position estimating unit 104, the gravity direction estimating unit 105, the initialization processing unit 106, the model position estimating unit 107, the model drawing unit 108, the display unit 109, the recognition start determining unit 110, and the object position recognizing unit 111. Corresponding to

  Note that, instead of the CPU 122, a processor or a controller such as an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) may be used.

  FIG. 18 is a diagram illustrating another configuration example of the information processing apparatus 100. The example illustrated in FIG. 18 illustrates an example in which the imaging device 130 is provided outside the information processing device 100 and the imaging device 130 captures an image including the target object. The imaging device 130 is, for example, a camera, and includes the imaging unit 101 and the inertial sensor 102. The imaging device 130 is movable, and can be moved to various places by the user. The RGB data captured by the imaging device 130 and the measured acceleration data can be transmitted to the information processing device 100 by wire or wirelessly. As shown in FIG. 18, the information processing system 10 includes an information processing device 100 and an imaging device 130.

  The summary is as follows.

(Appendix 1)
An inertial sensor that outputs acceleration data of the information processing device;
A gravitational direction estimating unit that estimates a first gravitational direction in a world coordinate system from the acceleration data;
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. A model position estimating unit that calculates a first conversion matrix that converts and converts the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
A model drawing unit that converts a position and orientation of the model in a camera coordinate system into a position of the model in an image coordinate system, and draws an input image and the model in an image coordinate system.
An information processing apparatus, comprising: a display unit that displays the input image and the model according to a drawing result of the model drawing unit.

(Appendix 2)
Further, an object position recognizing unit that acquires the first transformation matrix from the model position estimating unit when the model and the target object included in the input image are in a correspondence relationship in a world coordinate system, The information processing apparatus according to Supplementary Note 1.

(Appendix 3)
The information processing apparatus according to claim 2, wherein the object position recognizing unit acquires the first transformation matrix when the model and the target object match in the world coordinate system from the model position estimating unit. .

(Appendix 4)
The information processing apparatus according to claim 2, wherein the object position recognition unit acquires the first transformation matrix from the model position estimation unit when receiving a signal indicating a user's determination operation.

(Appendix 5)
The object position recognition unit converts the position and orientation of the model in a model coordinate system into the position and orientation of the model in a camera coordinate system via a world coordinate system, using the first transformation matrix. ,
3. The model drawing unit according to claim 2, wherein the model drawing unit draws the model and the input image in an image coordinate system based on the position and orientation of the model converted into a camera coordinate system by the object position recognition unit. Information processing device.

(Appendix 6)
The model position estimating unit converts the first gravitational direction into a second gravitational direction in a camera coordinate system, and sets the first gravitational direction as the first gravitational direction in a camera coordinate system.
When the viewpoint position in the camera coordinate system moves in a direction in which the first gravity direction changes, the second gravity direction changes, and the viewpoint position in the camera coordinate system moves in a direction in which the first gravity direction does not change. The information processing apparatus according to claim 1, wherein the second gravity direction does not change when moving.

(Appendix 7)
Further, based on the image data of the input image, the position and orientation of the imaging unit or the imaging device in the world coordinate system are estimated, and based on the estimated position and orientation, a second coordinate conversion is performed from the world coordinate system to the camera coordinate system. A self-position estimating unit that calculates a transformation matrix of
The information processing apparatus according to claim 6, wherein the model position estimating unit converts the first gravitational direction into the second gravitational direction using the second conversion matrix.

(Appendix 8)
Further, a virtual model coordinate system is set based on any two points on the model, a Euclidean distance between the two points, a first normal vector on a plane including the two points, and a vertical downward vector. The information processing apparatus according to claim 1, further comprising an initialization processing unit that calculates a unit line-of-sight vector from the viewpoint position of the virtual camera in the virtual model coordinate system to the two points.

(Appendix 9)
Further, self-position estimation for estimating the position and orientation of the imaging unit or the imaging device in the world coordinate system based on the image data of the input image and calculating a second transformation matrix for converting the world coordinate system to the camera coordinate system. Part,
The model position setting unit converts the first gravitational direction to the second gravitational direction using the second conversion matrix, and based on the second gravitational direction and the unit line-of-sight vector, Calculating a third transformation matrix for transforming the position and orientation of the model in the camera coordinate system into the position and orientation of the model in the model coordinate system, and using the third transformation matrix to calculate the model in the camera coordinate system; 9. The information processing apparatus according to claim 8, wherein the position and the posture of the model are respectively converted into the position and the posture of the model in a model coordinate system.

(Appendix 10)
The model position estimating unit converts the first normal vector into a second normal vector in a camera coordinate system based on the gravity vector having the first gravity direction and the vertical downward vector. Calculating two points of the model in a camera coordinate system based on the second normal vector and the unit line-of-sight vector, and calculating the two points of the model in a camera coordinate system, and the second normal vector; 10. The information processing apparatus according to claim 9, wherein the third transformation matrix is calculated based on a length of the model in the X-axis direction of the model coordinate system.

(Appendix 11)
_Assuming that the second transformation matrix is T _cw and the third transformation matrix is T _mc , the model position estimating unit calculates the first transformation matrix by using the following equation (14) read from the internal memory. The information processing device according to claim 10, wherein _Twm is calculated.

(Appendix 12)
Outputting acceleration data of the information processing device,
Estimating a first direction of gravity in the world coordinate system from the acceleration data,
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. Calculating a first transformation matrix for transforming the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
Convert the position and orientation of the model in the camera coordinate system to the position of the model in the image coordinate system, draw the input image and the model in the image coordinate system,
A positioning method, comprising: displaying the input image and the model according to a drawing result.

(Appendix 13)
A program to be executed by a computer of the information processing apparatus,
Outputting acceleration data of the information processing device;
Estimating a first direction of gravity in the world coordinate system from the acceleration data,
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. Calculating a first transformation matrix for transforming the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
Convert the position and orientation of the model in the camera coordinate system to the position of the model in the image coordinate system, draw the input image and the model in the image coordinate system,
A program for causing the computer to execute a process of displaying the input image and the model according to a drawing result.

10: Information processing system 100: Information processing apparatus 101: Imaging unit 102: Inertial sensor 103: Storage unit 104: Self-position estimation unit 105: Gravity direction estimation unit 106: Initialization processing unit 107: Model position estimation unit 108: Model drawing Unit 109: display unit 110: recognition start determination unit 111: object position recognition unit 120: camera 122: CPU 130: imaging device

Claims (6)

An inertial sensor that outputs acceleration data of the information processing device;
A gravitational direction estimating unit that estimates a first gravitational direction in a world coordinate system from the acceleration data;
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. A model position estimating unit that calculates a first conversion matrix that converts and converts the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
A model drawing unit that converts a position and orientation of the model in a camera coordinate system into a position of the model in an image coordinate system, and draws an input image and the model in an image coordinate system.
An information processing apparatus, comprising: a display unit that displays the input image and the model according to a drawing result of the model drawing unit.   Further, an object position recognizing unit that acquires the first transformation matrix from the model position estimating unit when the model and the target object included in the input image are in a correspondence relationship in a world coordinate system, The information processing apparatus according to claim 1, wherein   The information processing apparatus according to claim 2, wherein the object position recognition unit acquires the first transformation matrix when the model and the target object match in a world coordinate system from the model position estimation unit. apparatus. The model position estimating unit converts the first gravitational direction into a second gravitational direction in a camera coordinate system, and sets the first gravitational direction as the first gravitational direction in a camera coordinate system.
When the viewpoint position in the camera coordinate system moves in a direction in which the first gravity direction changes, the second gravity direction changes, and the viewpoint position in the camera coordinate system moves in a direction in which the first gravity direction does not change. The information processing apparatus according to claim 1, wherein the second gravity direction does not change when moving. Outputting acceleration data of the information processing device,
Estimating a first direction of gravity in the world coordinate system from the acceleration data,
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. Calculating a first transformation matrix for transforming the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
Convert the position and orientation of the model in the camera coordinate system to the position of the model in the image coordinate system, draw the input image and the model in the image coordinate system,
A positioning method, comprising: displaying the input image and the model according to a drawing result. A program to be executed by a computer of the information processing apparatus,
Outputting acceleration data of the information processing device;
Estimating a first direction of gravity in the world coordinate system from the acceleration data,
The position and orientation of the model in the camera coordinate system are changed according to the first direction of gravity with respect to the camera coordinate system, and the position and orientation of the model in the camera coordinate system are respectively changed to the position and orientation of the model in the model coordinate system. Calculating a first transformation matrix for transforming the position and orientation of the model in the model coordinate system to the position and orientation of the model in the world coordinate system, respectively.
Convert the position and orientation of the model in the camera coordinate system to the position of the model in the image coordinate system, draw the input image and the model in the image coordinate system,
A program for causing the computer to execute a process of displaying the input image and the model according to a drawing result.

Images