JPH0944630A - Image representation method and device - Google Patents

Abstract

(57) 【Abstract】 PROBLEM TO BE SOLVED: To temporally and especially spatially relate to a plurality of videos obtained by shooting a plurality of videos obtained by photographing the same object, and each video content. (EN) Provided are an expression method and a device that enable intuitive grasp of. A shooting state detection unit detects a shooting state when an input image is shot. Next, the fragment image creation unit 103
At, for each of the plurality of videos, the videos are converted so that the size and position of the object to be photographed and static are the same in each frame to create a fragment image. Next, the fragment image matching unit 104
At, the correspondence between the fragment images is calculated. Next, the fragment image combining unit 105 combines the fragment images using the correspondence relationship. In this way, by reconstructing the spatiotemporal images captured from multiple videos, temporal and spatial relationships between multiple videos and each video content can be used when supporting easy browsing of multiple videos. Realize an expression that enables intuitive grasp of.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for displaying a plurality of images obtained by shooting the same object, and more particularly, to a method and an apparatus for displaying the plurality of images. The present invention relates to an image representation method and device for reconstructing a spatiotemporal space.

[0002]

2. Description of the Related Art Conventionally, many methods have been reported in the field of computer vision as a method for expressing and reconstructing a space photographed from an image photographed by a camera. Among them, "Panoramic expression of environment" (Journal of the Institute of Electronics, Information and Communication Engineers, Vo
l. 74, no. 4, pp. 354-359, 1991
(April, 2010) is a method to create a 360-degree omnidirectional panoramic image by connecting images in the slit that are continuously taken while rotating the camera, and a video taken by the camera installed on the robot by moving the robot. This is a method of monitoring the environment in which the robot moves around by converting it into a panoramic representation using spatial spatial analysis and matching it with the map stored in the robot using dynamic programming.

Recently, there have been reported some methods for automatically extracting a camera operation from an image and expressing the photographed space as a still image using the information. Among them, "Salient Video Stills: C
ontent and Context Presenter
ved ”(Proceedings of ACMMu
ltimedi 93, pp. 39-46, CA, Ju
n. , 1993) is a method of creating a still image having a high resolution from a video, which calculates a motion vector, estimates an affine transform coefficient between each frame of the video, and creates a still image using this transform coefficient. is there.

A method for estimating a projective transformation and constructing a shooting space from each frame of a video or a plurality of images has also been reported. "Virtual Bellow: Cons"
tracing High Quality Sti
"lls from Video" (Proceedin
gs of ICIP-94, Vol. I, pp. 36
3-367, Austin, Nov. , 1994).

[0005]

SUMMARY OF THE INVENTION The above prior arts include:
A technique aimed at monitoring the environment and a technique aiming at creating a higher-definition still image from video are disclosed. However, although it is possible to spatially reconstruct (express) the shooting space using the above-mentioned conventional technique, the temporal information of the video and the content of the video (hereinafter referred to as video content) in the expression are temporally reconstructed. Since continuity is not taken into consideration, it is not possible to easily browse a plurality of pieces of video information both temporally and spatially as long as the video is represented using these conventional techniques. In order to support easy browsing of multiple videos, it is necessary for the expression to support the intuitive understanding of the relationship between the videos and the content of each video, both temporally and spatially.

An object of the present invention is to provide an expression method that enables intuitive understanding of the relationship between a plurality of videos and each video content in terms of time, especially spatially, when supporting easy browsing of a plurality of videos. And to provide a device.

[0007]

In order to solve the above-mentioned problems, an image expression apparatus of the present invention is provided with a plurality of images for browsing a plurality of images obtained by photographing the same object. And a shooting state detection unit that detects a shooting state at the time of shooting the image, and a still object to be shot for each image in the plurality of images based on the detected shooting state. A fragment image creating unit that creates a fragment image by converting the plurality of videos so that the sizes and positions of the fragment images are the same in each frame, and the correspondence relationship between each fragment image and each created fragment image. And a fragment image synthesizing unit for synthesizing the fragment images using the calculated correspondence relationship, and reconstructing the photographed spatiotemporal space from the synthesized video. It is characterized by

Similarly, the image representation method of the present invention is a method of representing a plurality of images obtained by photographing the same object so as to browse the plurality of images. An image conversion process of converting the plurality of images so that the size and position of the object to be photographed and stationary are the same for each image in each frame, and a step of integrating each of the converted images in the time direction. A weighted integration process of 1 and the first
A step of calculating a correspondence relationship for each image calculated as a result of the weighted integration step of, and a second weighted integration step of performing a weighted integration on each image using the calculated correspondence relationship, And reconstructing the captured space-time from the image resulting from the second weighted integration process.

As described above, according to the image representation method and apparatus of the present invention, the photographing state at the time of photographing the image is detected, and the size and position of the still object to be photographed is determined for each frame of the plurality of images. Converting the video so that they are the same between each other, creating fragment images, calculating the correspondence between each fragment image for each created fragment image, and combining the fragment images using the calculated correspondence. In the case of supporting the easy browsing of multiple videos by reconstructing the spatiotemporal images taken from multiple videos, the relationship between multiple videos in terms of both time and space and each video Realize expressions that allow intuitive understanding of content.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram of an apparatus according to an embodiment of the present invention, and FIG. 2 is a flow chart for explaining a method according to the embodiment of the present invention. Hereinafter, each component and method will be described with reference to FIGS. 1 and 2.

The input unit 101 shown in FIG. 1 receives an image taken by a camera or the like in the image signal input step 201 shown in FIG. The image from the input unit 101 is converted by the image capturing state detecting unit 102 and the fragment image creating unit 103 so that the size and position of the imaged still target object in the input image become equal in each frame. In this process, the global movement included in the movement in the image such as the camera operation is canceled. For example, FIG. 3 shows an example in which a moving car is photographed by shaking a camera. Figure 3 taken that way
When the image 301 in (a) is observed from the line-of-sight direction (observation direction) indicated by 302, an image 303 (hereinafter referred to as a panoramic image) in the shooting space as shown in FIG. 3B is obtained. The panoramic images respectively created from a plurality of videos of the same target object represent the target object in a fragmentary manner, and hereinafter, each of them will be referred to as a panoramic fragmentary image.

Next, the process of extracting the shooting state in the shooting state detecting unit 102 of FIG. 1 in the spatiotemporal projection image creating process 202 and the shooting state detecting process 203 of FIG. 2 will be described.

A method using a spatiotemporal projection image is disclosed as an extraction method, "Examination of a global motion extraction method using a spatiotemporal image" (1992 Autumn Meeting of the Society of Shinkai, D
-254, pp. 6-256, 1992). The features of this spatiotemporal projection image are that the temporal behavior of motion is expressed as a flow, and that the processing is fast and robust.

Here, a global motion extraction method using a spatiotemporal projection image will be described. FIG. 4 shows the flow of the processing algorithm.

First, the process in step 401 is a process of rearranging video frames on the time axis, and creates a spatiotemporal image from an input video. The spatio-temporal image here refers to an object in which images captured in time series are arranged in the time axis direction.

Next, the process in step 402 is a process of cutting the spatiotemporal image on a plane including the screen normal, and creates a spatiotemporal cross-sectional image. The spatio-temporal cross-sectional image referred to here is an image obtained by cutting the video in the time axis direction, and as an example of this spatio-temporal cross-sectional image, the advancing direction of the camera used in the field of computer vision and the normal line of the screen are included. Cut plane when the spatiotemporal image is cut by a plane (Epipolar Plane Image
e)). The three-dimensional position of the subject is estimated from the spatiotemporal sectional image. This is because the trajectory of the feature points of the object becomes a straight line on this epipolar plane image, and the inclination of this straight line becomes the size of the object feature point.
ipolar-Plane image analyses
is: Anaproach to detail
g structure from motion "
(IJCV, 1,1, pp7-55, jun 198
9). As the spatiotemporal sectional image, an xt spatiotemporal image sequence and a yt spatiotemporal image sequence are calculated. Here, the x-t spatio-temporal image sequence is a plurality of x-t spatio-temporal images, and the y-t spatio-temporal image sequence is also the same. Here, the xt spatiotemporal image / yt spatiotemporal image refers to a cut surface obtained by cutting the spatiotemporal image along a plane including the normal line of the screen.

Next, the process in step 403 is a process of detecting edges and lines by a filter process, which is a first derivative for each xt space-time image with respect to the xt space-time image sequence. The edge and the line are detected by performing a filtering process such as the second derivative. The processing result of step 403 corresponds to the result of detecting the locus related to the movement of the feature points such as the object and the background described above.

After the edge and line intensities are calculated in step 403, next, in step 404, edge and line statistical analysis processing is performed. The statistical processing mentioned here is to perform integration processing in the y-axis direction on the x-t spatiotemporal image sequence, and similarly, perform integration processing on the y-t spatio-temporal image sequence in the x direction. That is. In this way, the spatiotemporal projection image can be calculated from the edge intensity information.

The spatiotemporal projection image visualizes the motion in the video as a flow. The process of visualizing this movement as a flow is the spatiotemporal projection image creation process 202 in FIG.
This flow is tracked in the flow tracking process of step 405, and the shooting state (hereinafter referred to as camera operation) is extracted.
An example of flow tracing is shown in FIG. Reference numeral 501 in FIG. 5 denotes an xt space-time projection image, and this xt space-time projection image 501.
The flow shown in is assuming that the camera is swung to the right. 502 shows the distribution of the integrated value of time T, and 503
Indicates the distribution of the integrated value at time T-1. 502 and 503
From the distribution of the cross-correlation method, Hough transform method, and dynamic programming method
g for solving various
"problems in vision" (IEEE
Trans. Pattern Anal. & Mac
h. Intel. , PAMI-12, Vol. 9, p
p. 855-867 (1990))) is used to calculate the change amount, and the calculated change amount is the camera operation parameter.

The distribution 502 in FIG. 5 is represented by a function F (x, t) in time t and space x, and when the motion of the image is a motion by camera operation, the distribution 502 and the distribution 503 are as follows. The relational expression holds. The function F (x, t) is an integrated value of the edge strength in the y-axis direction in the x-coordinate, and has a large value when there are many edges in the y-axis direction.
That is, it is assumed that, when the motion due to the camera operation is included in the image, the motion (expressed as an edge by the above processing) is present in a large amount in the frame.

F (x, t) = F (ax + b, t-1). Here, the coefficients a and b are parameters representing the camera operation, a is a zoom parameter, and b is a pan parameter. These parameters are calculated by the above methods. Similarly, the tilt parameter c can be calculated from the y-t spatiotemporal projection image.

Only when it is determined that the camera operation parameters calculated by the above method are detected in the photographing state presence / absence determining step 204 (yes), the conversion step 205 in FIG. 2 in the fragment image creating unit 103 in FIG. 1 is performed. In, each frame is converted using the following relational expression. When it is determined that there is no camera operation (no), a = −1, b =
This is equivalent to performing the following conversion with parameters of 0 and c = 0.

[0024]

[Equation 1]

Here, (x, y) indicates the image frame coordinates at time T-1, and (x ', y') indicates the image frame coordinates at time T.

Next, the weighting factor calculation process 206 in FIG. 2 performed by the fragment image creating unit 103 in FIG. 1 will be described. Here, a weighting coefficient used for the next time integration process performed by the fragment image creating unit 103 is calculated. It is considered that this weighting coefficient is suitable as a coefficient that reflects the local movement of an object or the like. In this sense, only the local motion of the object or the like is included in the converted image converted earlier. A method of calculating a coefficient that reflects a local motion using this converted image will be described. This state is shown in FIG.

In FIG. 6, xy showing the coordinates in the frame is shown.
There is an XYT coordinate system that indicates between the coordinate system and the frame. In FIG. 6, 601 is the spatiotemporal image after conversion. 6
02 is a frame at time t, and 603 is a frame at time t + n. The spatiotemporal image 601 can be represented by a function F (X, Y, T) of X, Y, T in the XYT coordinate system.
The weighting coefficient to be calculated is the pixel (X,
It is calculated for each of Y, T) and expressed as w (X, Y, T). Reference numeral 604 denotes an arbitrary pixel f (x0,
y0) and f (X0, Y0, t). Reference numeral 606 denotes an arbitrary pixel f (x1, y1) in the frame 603, which is F
(X1, Y1, t + n). However, X1 = X0, Y
There is a relation of 1 = Y0. 605 is a pixel 604 and a pixel 6
06 is a straight line, and is a function of time L (X0, Y0, T) represented by the values (luminance, R, G, B values, y, cb, cr values, etc.) of the pixel values on this straight line. is there. When there is no local motion of an object or the like in the spatiotemporal image 601, L (X0, Y0, t) = L (X0, Y0, t + 1) = L
(X0, Y0, t + 2) ... = L (X0, Y0, t +
n) and are established. The change in the value of L (X0, Y0, T) is due to the local movement, and the weight coefficient w (X, Y, T) is defined using this function to calculate the coefficient reflecting the local movement. It becomes possible to do. The weighting factor can be generally expressed as follows using the function L (X0, Y0, T).

W = AB. Here, A represents a modified group of functions of the function L (X0, Y0, T), and is, for example, as follows.

A = L (X0, Y0, T), A = dL (X0, Y0, T) / dT, A = dL (X0, Y0, T) ² / d ² T, ... And B is a constant And a group of functions with time as a variable, such as the following.

B = 1, B = T, B = T ⁻¹ , B = exp (T), ... A is a term reflecting local motion, and B is a term reflecting time. By B, the light and shade of the brightness and the color of the still image obtained as a result of the integration process described below change, and the time can be expressed by these changes. For example,
It is also possible to make the past sepia by using B.

Next, in the fragment image creating unit 103, the conversion image is integrated in the time direction by using the weighting coefficient previously calculated in the weighted integration process 207. The weighting factor is defined as follows for convenience so that the integration result can be represented as an image on a CRT and a hard copy. Here, the time width for performing the integration process is n.

[0032]

[Equation 2]

W'represents a normalized form.

The converted image is represented by F (X, Y, Ti, i = t to t
+ N), weighting factor w '(X, Y, Ti, i = t to t +
n), the image G (X, X,
Y) is

[0035]

(Equation 3)

It can be expressed as A panoramic image G (X, Y) created when a plurality of videos are processed by the above means is represented by p _i , _{i = 1-n} .

Next, the fragment image collating unit 104 in FIG. 1 will be described. The fragment image matching unit 104 performs a matching matching process 208 between the created panoramic fragment plane images. The state of this association is shown in FIG. 7, 701, 70
Reference numeral 2 represents a collation target panoramic fragment image. This is represented by p _i and p _j , respectively. Reference numeral 703 denotes an overlapping portion between the images of p _i and p _j . Considering the coordinate system uv, the arrangement relationship between p _i and p _j is represented by the value of (704, 705).
Here, 704 is a panorama fragment image correspondence value in the v direction, and 705 is a panorama fragment image correspondence value in the u direction. For the overlapping portion (703) between the images of p _i and p _j, the cross-correlation coefficients ρ _pi and ρ _pj shown in the following equations are calculated.

Ρ _pi , ρ _pj = ρ _pi , ρ _pj / (ρ _pi ² · ρ _pj ² ) ^1/2 · M, where

[0039]

(Equation 4)

Where M is the number of pixels in the overlapping portion. Also, ρ _pi ² and ρ _pj ² represent the variance.

The arrangement (704, 705) is changed within an appropriate range to calculate the cross-correlation coefficient according to each value, and the arrangement (704, 70) is such that the calculated cross-correlation coefficient becomes the maximum value.
5) is the correspondence.

Using the above method, the relations (704, 705) between a plurality of panoramic fragment images are calculated.

Next, in the fragment image synthesizing section 105 in FIG. 1, the integration performed in the processing in the next weighted integration step 210 is first performed from the number of panorama fragment images collated in the weighting factor calculation step 209 in FIG. The weight of each panorama fragment image used for processing (superimposition) is calculated. The weight here is the transmittance or the like of each panoramic image. For example, an expression such as emphasizing a portion overlapping a plurality of panoramic fragment images can be performed.

Then, in the fragment image synthesizing unit 105,
In the weighted integration step 210, the panoramic images are integrated (overlaid) from the weighting arrangement relationships (704, 705) calculated in the weighting factor calculation step 209 to represent a plurality of panoramic images.

Next, the output unit 106 in FIG. 1 outputs the combined panoramic image in the output step 211 in FIG. In the image output here, temporal information (movement information of an object) is expressed in the panoramic image space as described above, and a new image is displayed in which a large space is reconstructed in each panoramic fragment image. Is possible. It is an expression that makes it possible to access time information by accessing (pointing) this reconstruction space.

The present invention has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Needless to say.

[0047]

As described above, the image representation method and apparatus of the present invention is similar to each image in the image so that the size and position of the imaged stationary target object are equal in each frame. Transforms to create fragment images, calculates the correspondence relationship for each image by performing weighted integration that integrates each converted video in the time direction, and weights integration for each image using the calculated correspondence relationship. It is characterized by reconstructing (representing) the spatiotemporal shot from the video by synthesizing fragmentary images by, for example, so that it is possible to easily browse multiple videos in time and space. Also, it becomes possible to express the relationship between a plurality of videos and intuitive understanding of each video content (contents of the video).

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an example of an embodiment of an apparatus of the present invention.

FIG. 2 is a flow chart showing an example of an embodiment of the method of the present invention.

3A and 3B are explanatory diagrams of a panoramic image in the above embodiment.

FIG. 4 is a flowchart of a camera operation parameter extraction process in the above embodiment.

FIG. 5 is an explanatory diagram of a flow tracking process in the above embodiment.

FIG. 6 is an explanatory diagram of weighting coefficient processing in the above embodiment.

FIG. 7 is an explanatory diagram of a panorama fragment image matching process in the above embodiment.

[Explanation of symbols]

101 ... Input unit 102 ... Shooting state detection unit 103 ... Fragment image creation unit 104 ... Fragment image collation unit 105 ... Fragment image synthesis unit 106 ... Output unit 201 ... Video signal input process 202 ... Spatio-temporal projection image creation process 203 ... Shooting State detection process 204 ... Shooting state presence / absence determination process 205 ... Conversion process 206 ... Weight count calculation process 207 ... Weighted integration process 208 ... Collation process 209 ... Weight count calculation process 210 ... Weighted integration process 211 ... Output process 301 ... Spatio-temporal Image 302 ... Line-of-sight direction (observation direction) 303 ... Panorama image 401 ... Rearrangement of video frames to time axis 402 ... Cutting of spatio-temporal image on plane including screen normal 403 ... Edge and line detection by filtering 404 ... Statistical analysis of edges and lines 405 ... Flow tracking 501 ... xt spatiotemporal projection image 50 ... integrated value of the distribution (F (x, t)) of the time T 503 ... distribution of time-integrated values of T-1 (F (ax + b, t
-1)) 601 ... Converted image 602 ... Frame at time t 603 ... Frame at time t + n 604 ... Arbitrary pixel on frame at time t 605 ... Arbitrary pixel on frame at time t and frame at time t A straight line 606 connecting the pixels on the frame of the corresponding time t + n 606 ... A pixel on the frame of the time t + n corresponding to the frame of the time t 701 ... A panorama fragment image 702 ... A panorama fragment image 703 ... An overlapping portion between panorama fragment images 704 ... v Correspondence value between panorama fragment images in the direction 705 ... Correspondence value between panorama fragment images in the u direction

Claims (2)

[Claims] 1. An apparatus for expressing a plurality of images for browsing a plurality of images obtained by photographing the same object, the photographing being for detecting a photographing state at the time of photographing the images. A state detection unit, and based on the detected shooting state, converts the plurality of images so that the size and position of the object to be photographed and stationary is the same for each frame among the plurality of images. A fragment image creating unit that creates a fragment image, a fragment image inter-matching unit that calculates a correspondence relationship between the created fragment images and the fragment images, and the fragment using the calculated correspondence relationship. And a fragment image synthesizing unit for synthesizing images, and reconstructing the captured space-time from the synthesized video. 2. A method of representing a plurality of images obtained by photographing the same object for browsing the plurality of images, wherein each of the images in the plurality of images is represented. An image conversion process for converting the plurality of images so that the size and position of the imaged stationary target object are equal in each frame; and a first weighted integration process for integrating each of the converted images in the time direction. , A process of calculating a correspondence relationship for each image calculated as a result of the first weighted integration process, and a second weighting process for performing a weighted integration on each image using the calculated correspondence relationship. An image integration method, and reconstructing the captured space-time from the image of the result of the second weighted integration process.