WO2024252585A1 - Information processing system and information processing method - Google Patents

Abstract

[Problem] To provide an information processing system or the like that can easily manage an image captured by a moving body in association with a position-to-be-captured. [Solution] According to an embodiment of the present invention, provided is an information processing system comprising: an actual image-capturing position information acquisition unit 110 that acquires actual image-capturing position information indicating an actual image-capturing position in a real space in association with acquired captured image data; a virtual distance generation unit 120 that generates virtual distance information indicating a virtual distance from a virtual image-capturing position corresponding to the actual image-capturing position to a position-to-be-captured in a virtual space; and a size identification unit 130 that identifies the size of the image data at the position-to-be-captured on the basis of the angle-of-view information and the focal distance information when the captured image data has been captured, and the virtual distance information.

Description

Information processing system and information processing method

The present invention relates to an information processing system and an information processing method.

Traditionally, autonomously controllable mobile objects, such as flying objects (hereinafter collectively referred to as "flying objects") such as drones and unmanned aerial vehicles (UAVs) and mobile objects such as unmanned ground vehicles (UGVs), have begun to be used in industry, and in particular, such mobile objects are used to inspect the interior and exterior of buildings. Patent Document 1 discloses a system that inspects indoors using mobile objects and manages the images.

JP 2019-179422 A

However, with the technology disclosed in Patent Document 1, it is unclear which position on a building the image captured by the moving body was captured from and from which direction, and it is also difficult to know from the image the actual size of the object being photographed at the site. As a result, it is difficult to manage images captured by a moving body in association with the position of the object being photographed.

It is also possible to generate a virtual space that mimics an actual site by pasting a captured image onto a three-dimensional model, but if the size of the object being photographed at the actual site is unknown as described above, it is unclear at what size the captured image should be pasted onto the three-dimensional model. This requires the worker to manually adjust the size of the captured image and the pasting position on the three-dimensional model using an image editing system or the like, which places a heavy burden on the worker, especially when pasting a large number of images onto a three-dimensional model.

The present invention was made in consideration of this background, and aims to provide an information processing system etc. that can easily manage images captured by a moving object by associating them with the position and size of the object being captured.

The main invention of the present invention for solving the above problem is an information processing system that includes an actual shooting position information acquisition unit that acquires actual shooting position information indicating an actual shooting position in real space linked to the acquired captured image data, a virtual distance generation unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space, and a size determination unit that determines the size of the image data at the shooting target position based on the angle of view information, focal length information, and virtual distance information when the captured image data was captured.

The present invention can provide an information processing system that can easily manage images captured by a moving object by associating them with the location and size of the object being captured.

1 is a diagram showing a configuration of an information processing system according to an embodiment of the present invention; FIG. 2 is a block diagram showing a hardware configuration of the management server of FIG. 1 . FIG. 2 is a block diagram showing a hardware configuration of the user terminal of FIG. 1 . FIG. 2 is a block diagram showing a hardware configuration of the moving body of FIG. 1 . FIG. 2 is a block diagram showing the functions of the management server of FIG. 1 . 11 is a diagram for explaining an example of a process in which a size specifying unit specifies the size of image data at a shooting target position; FIG. 1 is a flowchart illustrating an information processing method according to the present embodiment.

The contents of the embodiment of the present invention will be listed and described below. An information processing system according to the embodiment of the present invention has the following configuration.
[Item 1]
an actual photography position information acquisition unit that acquires actual photography position information indicating an actual photography position in a real space associated with the acquired photographed image data;
a virtual distance generating unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space;
a size determination unit that determines a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
An information processing system comprising:
[Item 2]
An image projection unit projects the image data of the specified size onto a shooting target model present at the shooting target position in the virtual space.
Item 2. An information processing system according to item 1.
[Item 3]
Further comprising a difference detection unit that detects a difference in shape between the object to be photographed shown in the projected image data and the object to be photographed model.
3. The information processing system according to item 2.
[Item 4]
and a shooting target specifying unit that specifies at least one of a shooting target model present at the shooting target position in the virtual space and a shooting target space based on a center position of the angle of view when the captured image data is captured.
Item 2. An information processing system according to item 1.
[Item 5]
An image linking unit that links the image data to at least one of the shooting target model or the shooting target space identified by the shooting target identification unit and stores the image data.
5. An information processing system according to item 4.
[Item 6]
acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data;
generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space;
specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
2. An information processing method implemented by a computer, comprising:
[Item 7]
acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data;
generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space;
specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
A program that causes a computer to execute the above.

<Details of the embodiment>
Hereinafter, an information processing system according to an embodiment of the present invention will be described. In the accompanying drawings, identical or similar elements are given identical or similar reference symbols and names, and duplicated descriptions of identical or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment can be applied to other embodiments as long as they are not mutually inconsistent.

<Configuration>
As shown in Fig. 1, the information processing system in this embodiment has a management server 1, one or more user terminals 2, one or more moving bodies 4 (e.g., flying bodies, running bodies, etc.), and one or more moving body storage devices 5. The management server 1, the user terminals 2, the moving bodies 4, and the moving body storage devices 5 are communicatively connected to each other via a network. Note that the illustrated configuration is an example, and is not limited thereto. For example, the configuration may be one in which the moving body storage device 5 is not included and the moving body is carried by the user.

<Management Server 1>
2 is a diagram showing a hardware configuration of the management server 1. Note that the illustrated configuration is an example, and the management server 1 may have other configurations.

As shown in the figure, the management server 1 is connected to a user terminal 2, a mobile object 4, and a mobile object storage device 5, and constitutes part of the system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission/reception unit 13, an input/output unit 14, etc., which are electrically connected to each other via a bus 15.

The processor 10 is a computing device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes programs for this system stored in the storage 12 and deployed in the memory 11 to perform various information processing operations.

Memory 11 includes a main memory consisting of a volatile storage device such as DRAM (Dynamic Random Access Memory) and an auxiliary memory consisting of a non-volatile storage device such as a flash memory or HDD (Hard Disc Drive). Memory 11 is used as a work area for processor 10, and also stores the BIOS (Basic Input/Output System) that is executed when management server 1 is started, various setting information, etc.

Storage 12 stores various programs such as application programs. A database that stores data used for each process may be constructed in storage 12.

The transmission/reception unit 13 connects the management server 1 to the network. The transmission/reception unit 13 may be equipped with a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).

The input/output unit 14 includes information input devices such as a keyboard and mouse, and output devices such as a display.

The bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

<User Terminal 2>
3 also includes a processor 20, a memory 21, a storage 22, a transmission/reception unit 23, an input/output unit 24, etc., which are electrically connected to each other via a bus 25. The functions of each element can be configured in the same way as the management server 1 described above, so detailed explanations of each element will be omitted.

<Mobile object 4>
The moving body 4 is a known moving body including an air vehicle such as a drone or an unmanned aerial vehicle, a running body such as an unmanned ground vehicle, etc., and is particularly a moving body that can be autonomously controlled. As a specific example of such a moving body, the moving body 4 will be described below. FIG. 4 is a block diagram showing a hardware configuration of the moving body 4. The flight controller 41 can have one or more processors such as a programmable processor (e.g., a central processing unit (CPU)).

Flight controller 41 also has and can access memory 411. Memory 411 stores logic, code, and/or program instructions that the flight controller can execute to perform one or more steps. Flight controller 41 may also include sensors 412, such as inertial sensors (accelerometers, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), etc.

The memory 411 may include, for example, a separable medium such as an SD card or random access memory (RAM) or an external storage device. Data acquired from the camera/sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image/video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited to this, and may be recorded in at least one of the management server 1, the user terminal 2, or the mobile body storage device 5 from the camera/sensors 42 or the internal memory via the network NW. The camera 42 is installed on the mobile body 4 via a gimbal 43.

The flight controller 41 includes a control module (not shown) configured to control the state of the moving object. For example, the control module controls the propulsion mechanism (motor 45, etc.) of the moving object via an ESC 44 (Electric Speed Controller) in order to adjust the spatial arrangement, speed, and/or acceleration of the moving object having six degrees of freedom (translational motion x, y, and _z , and rotational motion _{θ x} , θ _y , and θ z ). The propeller 46 rotates by the motor 45 powered by a battery 48, thereby generating lift for the moving object. The control module can control one or more of the states of the mounted parts and sensors.

The flight controller 41 can communicate with a transceiver 47 configured to transmit and/or receive data from one or more external devices (e.g., a transceiver 49, a terminal, a display device, or other remote control). The transceiver 49 can use any suitable communication means, such as wired or wireless communication.

For example, the transceiver unit 47 can utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, etc.

The transmitter/receiver 47 can transmit and/or receive one or more of the following: data acquired by the sensors 42, processing results generated by the flight controller 41, specific control data, user commands from a terminal or a remote controller, etc.

The sensors 42 according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (e.g., LiDAR (Light Detection And Ranging), etc.), or a vision/image sensor (e.g., a camera). Hereinafter, the sensors 42 may also be referred to as "camera 42". The sensors 42 according to this embodiment are configured to acquire, when an image is captured by the camera 42 as a vision/image sensor, information on the actual capturing position of the camera 42 in real space at the time the image was captured, the actual capturing direction which is the direction in real space to which the optical center axis of the camera 42 faces, the focal length and angle of view of the camera 42, and other capturing condition information in association with the image. Therefore, the image data of an image captured by the camera 42 according to this embodiment includes not only the image data, but also the above-mentioned capturing condition information linked thereto.

<Management server functions>
5 is a block diagram illustrating functions implemented in the management server 1. In the embodiment of the present invention, the management server 1 has various functional units for executing a process of arranging a captured image in a corresponding three-dimensional model based on an image captured by a camera included in the sensors 42 of the mobile object 4 and image capture condition information associated therewith, and three-dimensional model data indicating the arrangement of components (e.g., walls, columns, stairs, equipment, etc.) within a structure (e.g., a building such as a building).

In this embodiment, the management server 1 includes a processor 10, a transmission/reception unit 13, and a storage unit 200. The processor 10 includes an actual shooting position information acquisition unit 110, a virtual distance generation unit 120, a size identification unit 130, an image projection unit 140, a difference detection unit 150, a shooting target identification unit 160, an image linking unit 170, and a movement execution unit 180. The storage unit 200 includes various databases, such as a three-dimensional data storage unit 210, a shooting image data storage unit 220, and a movement information storage unit 230.

First, the various databases of the storage unit 200 are described. The three-dimensional data storage unit 210 stores three-dimensional model data showing the components inside and outside the structure and their positions. The three-dimensional model data may be any data as long as it is three-dimensional model data (more preferably three-dimensional model data having dimensional information) showing the arrangement of the components inside the structure, which is created based on data created with CAD (Computer-Aided Design) design software. For example, it may be three-dimensional model data reconstructed from BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, BIM data, etc., or it may be three-dimensional model data obtained by generating a component having a predetermined height based on two-dimensional design drawing data. The generation of the three-dimensional model data such as reconstruction may be executed by the processor 10 of the management server 1, or the management server 1 may acquire the three-dimensional model data generated by executing the process on an external device different from the management server 1. In addition, the three-dimensional model data showing the structure may be linked with information showing the type, size, scale, etc. of the structure. In addition, as information on the three-dimensional model data, for example, information showing a predetermined spatial area based on the three-dimensional model data (for example, a spatial area based on a structure such as the spatial area between pillar A and pillar B, or a spatial area defined by a three-dimensional coordinate system in which the three-dimensional model data is placed (particularly, a spatial area based on coordinates specified by a user operation)) may be stored in the three-dimensional data storage unit 210.

The processor 10 of the management server 1 generates a virtual space represented by the three-dimensional model data in a three-dimensional coordinate system based on the three-dimensional model data stored in the three-dimensional data storage unit 210. The placement information of each three-dimensional model in the virtual space thus generated, such as its placement position and orientation, is already known.

The photographed image data storage unit 220 stores photographed image data captured by the sensors 42 of the moving object 4 and transmitted from the moving object 4 to the management server 1. As described above, the image data captured and acquired by the sensors 42 of the moving object 4 includes not only the image data, but also photographing condition information related to the actual photographing position in real space of the camera 42 at the time the image was captured, the actual photographing direction which is the direction in real space in which the optical center axis of the camera 42 faces, and the focal length of the camera 42, which are associated (linked) with the image. Therefore, the photographed image data stored in the photographed image data storage unit 220 includes image data and photographing condition information linked thereto. In the photographed image data, the photographing condition information may be metadata associated with the image data.

The photographed image data stored in the photographed image data storage unit 220 may further include information on at least one of the photographed subject model present at the photographed subject position in the virtual space, identified by the photographed subject identification unit 160 of the processor 10 described below and linked to the image linking unit 170, or the photographed subject space. Details of the above-mentioned identification process and linking process by the photographed subject identification unit 160 and the image linking unit 170 will be described later. Furthermore, the photographed image data stored in the photographed image data storage unit 220 may be associated with information on the size of the image data at the photographed subject position, which is the position of the photographed subject in the virtual space, identified by the size identification unit 130 as described below.

The movement information storage unit 230 stores movement information used for movement for the purpose of taking actual images of various objects inside and outside buildings, for example. The movement information includes, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (imaging direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.), information acquired during movement (for example, image data and imaging condition information associated therewith, etc.), etc.

The movement information can be generated, for example, by setting the parameters of various information included in the movement information on the management server 1 or the user terminal 2. The movement route may be generated by, for example, setting the position of the mobile body storage device 5 as the movement start position and the movement end position, and passing through each waypoint. Conversely, it may be configured such that, without the mobile body storage device 5, the position where the user carries the mobile body 4 is set as the movement start position (so-called home point), and the user retrieves the mobile body 4 at the movement end position (which may be back to the home point). Alternatively, it may be configured to generate a movement route including the position of the mobile body storage device 5 selected as the movement start position or movement end position based on the information of the mobile body storage device 5 managed by the management server 1 (for example, position information, storage state information, storage machine information, etc.).

Next, the functional units of the processor 10 will be described. The actual shooting position information acquisition unit 110 acquires, as shooting condition information linked to the captured image data captured by the camera 42 of the moving body 4 and transmitted from the moving body 4 to the management server 1, in particular, actual shooting position information indicating the actual shooting position of the camera 42 in real space at the time the image was captured. The actual shooting position information acquisition unit 110 may further acquire information regarding the actual shooting direction, which is the direction in real space in which the optical center axis of the camera 42 faces, and the focal length and angle of view of the camera 42, from the above-mentioned shooting condition information linked to the acquired image data.

The actual photographing position of the camera (sensors) 42 in real space is a position expressed in a three-dimensional coordinate system of real space. If the camera 42 of the moving body 4 includes a GNSS sensor (GPS sensor, etc.) and latitude and longitude information can be acquired by the GNSS sensor as position information of the sensors 42, the actual photographing position of the camera (sensors) 42 in real space may be expressed in latitude and longitude. Alternatively, if the position information of the sensors 42 is acquired by a non-GNSS sensor such as an inertial sensor (acceleration sensor, gyro sensor), the actual photographing position of the camera (sensors) 42 in real space may be expressed as a position in a three-dimensional coordinate system with the origin being a reference position in real space (for example, the flight start position (home point) of the moving body 4).

When the processor 10 of the management server 1 generates a virtual space represented by the three-dimensional model data in a three-dimensional coordinate system based on the three-dimensional model data stored in the three-dimensional data storage unit 210, the processor 10 performs a process of constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, for example by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space. As an example, the processor 10 aligns and associates a reference position in the three-dimensional coordinate system of the real space (for example, the flight start position of the moving body 4) with a reference position in the three-dimensional coordinate system of the virtual space corresponding to that position, thereby constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space. This allows the processor 10 to convert and express the position in the three-dimensional coordinate system of the real space of the moving body 4 flying in the real space into the position in the three-dimensional coordinate system of the virtual moving body (a virtual object corresponding to the moving body 4 in the real space) in the virtual space. If the scale of the three-dimensional model data in the virtual space differs from the scale of the corresponding structure (composition) in the real space, the scale of the three-dimensional model data may be adjusted during alignment.

In this way, by constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, it becomes possible to specify positions such as generating a movement path based on the three-dimensional coordinate system of the moving object 4 in the real space, even when a waypoint is set based on the three-dimensional coordinate system of the virtual space displayed on the user terminal 2. When the three-dimensional model data includes dimensional information, it becomes possible to generate a movement path using a real-world scale, such as flying straight for 10 m and then turning right, based on the three-dimensional coordinate system of the virtual space.

The virtual distance generation unit 120 performs processing to generate virtual distance information indicating the virtual distance from a virtual shooting position corresponding to the real shooting position of the camera 42 in the real space to a shooting target position, which is the position of the shooting target in the virtual space, in the virtual space generated by the processor 10. The shooting target position in the virtual space may be, for example, coordinate information in the virtual space, the position of a three-dimensional model of the shooting target in the virtual space (virtual components such as walls, pillars, stairs, and equipment in a building in the virtual space), or any spatial region position in the virtual space.

The virtual distance generation unit 120 performs a process of calculating a virtual distance, which is the distance from the virtual shooting position to the shooting target position, based on a virtual shooting position in the virtual space in the three-dimensional coordinate system of the virtual space and the shooting target position (the position of the three-dimensional model corresponding to the component of the shooting target), and generating virtual distance information. As an example, the virtual distance, which is the distance from the virtual shooting position to the shooting target position, can be obtained by determining the length of a line segment connecting the virtual shooting position and the shooting target position in the three-dimensional coordinate system of the virtual space along the shooting direction (particularly the optical axis direction).

The size determination unit 130 performs processing to determine the size of the image data at the position of the subject to be photographed, which is the position of the subject to be photographed in the virtual space, based on the angle of view information and focal length information of the camera 42, which is the photographing condition information included in the photographed image data acquired when the camera 42 photographs an image, and the virtual distance information generated by the virtual distance generation unit 120.

Now, with reference to FIG. 6, an example of the process in which the size determination unit 130 determines the size of the image data at the position of the subject to be photographed will be described.

First, the size determination unit 130 uses the angle of view information and focal length information of the camera 42 linked to the captured image data to calculate the size of a first plane (the "near plane" of the viewing frustum shown in FIG. 6) that is a focal length n away from the virtual capture position C in the virtual space that corresponds to the actual capture position of the camera 42. The size of the first plane corresponds to the size of the captured image data of the subject captured by the camera 42.

The size determination unit 130 then uses the size of the first plane calculated as described above, the focal length information, and the virtual distance information generated by the virtual distance generation unit 120 to calculate the size of the image data in a second plane (the "far plane" of the viewing frustum shown in FIG. 6) that is a virtual distance f away from the virtual shooting position C. The size of the second plane is the size of the first plane enlarged according to the ratio of the focal length n and the virtual distance f, and is the size of the image data at the shooting target position. When the three-dimensional model data includes dimensional information and scale information, the size determination unit 130 calculates the size of the image data at the shooting target position as described above, making it possible to obtain the actual size of the component in real space corresponding to the three-dimensional model at the shooting target position from the size of that image data.

The image projection unit 140 performs a process of projecting image data of the size specified by the size specification unit 130 onto the subject model, which is a three-dimensional model of the subject that exists at the subject position in the virtual space. Through this process, an image of a real-space component that corresponds to the subject model is pasted at a size that matches the size of the subject model at the corresponding position on the subject model, which is a three-dimensional model of the subject that exists at the subject position, and the texture of the real-space component (subject) is added to the surface of the subject model.

The difference detection unit 150 performs processing to detect the difference in shape between the subject shown in the image data projected by the image projection unit 140 into the virtual space and the subject model corresponding to that subject.

As an example of the difference detection process, let us take an example in which the object to be photographed is a cupboard having a substantially rectangular parallelepiped shape. The difference detection unit 150 extracts characteristic parts of the object to be photographed that appear in the projected image data (e.g., the outline of the cupboard that is the object to be photographed and the corners where the outlines intersect) and characteristic parts of the object to be photographed that appear in the object to be photographed onto which the image data is projected that correspond to the characteristic parts of the object to be photographed (e.g., the ridges of the object to be photographed that is a three-dimensional model of the cupboard and the corners where the ridges intersect), and detects the difference in shape or position between the object to be photographed that appears in the projected image data and the object to be photographed model based on the positional relationship between these characteristic parts. The difference information obtained as a result of the detection process by the difference detection unit 150 can be used, for example, to correct the position, shape, size, etc. of the image data that the image projection unit 140 projects in the virtual space, and to position the image data so that it is appropriately superimposed on the object to be photographed model.

The subject identification unit 160 performs processing to identify at least either the subject model present at the subject position in the virtual space or the subject space based on the center position of the angle of view of the camera 42 when the camera 42 captured the captured image data.

More specifically, the shooting target identification unit 160 first obtains a virtual shooting direction in the virtual space corresponding to the actual shooting direction of the camera 42 based on information on the actual shooting direction, which is the direction in real space to which the optical center axis of the camera 42 faces at the time the shooting image data was captured, among the shooting condition information acquired when the camera 42 captured the shooting image data and linked to the shooting image data. The virtual shooting direction in the virtual space can be expressed as a virtual straight line extending from a virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 to the virtual shooting direction corresponding to the actual shooting direction of the camera 42. The center position of the angle of view of the camera 42 can correspond to the virtual shooting position in the virtual space. The shooting target identification unit 160 then identifies a shooting target model or shooting target space that exists at a shooting target position in the virtual space, where a virtual straight line extending in the virtual shooting direction from a virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 intersects.

By performing this type of processing, the photographing subject identification unit 160 identifies at least one of the photographing subject model or the photographing subject space that exists at the photographing subject position in the virtual space, based on the center position of the angle of view of the camera 42 when the camera 42 captured the photographed image data.

The image linking unit 170 links the image data to at least one of the subject model or the subject space identified by the subject identification unit 160 and stores the image data in the storage unit 200. More specifically, based on the identification information acquired as a result of the identification process by the subject identification unit 160, the image linking unit 170 associates the photographed image data captured by the camera 42 with the subject model or the subject space in the virtual space that corresponds to the subject in real space reflected in the photographed image data, and stores the photographed image data linked to the corresponding subject model or subject space in the virtual space in the photographed image data storage unit 220 of the storage unit 200.

The movement execution unit 180 executes the movement of the moving body 4 for the purpose of taking real images of various shooting subjects inside and outside the building, etc., based on various movement information stored in the movement information storage unit 230.

<An example of an information processing method according to this embodiment>
Next, an information processing method according to this embodiment (particularly, a method for specifying the size of image data on a three-dimensional model) will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the information processing method according to this embodiment.

First, the actual shooting position information acquisition unit 110 in the information processing system of this embodiment acquires shooting condition information linked to the captured image data from the captured image data captured by the camera 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1 (step S101).

The shooting condition information is acquired when the camera (sensors) 42 serving as a vision/image sensor captures an image, and includes information about the actual shooting position of the camera 42 in real space at the time the image was captured, the actual shooting direction which is the direction in real space to which the optical center axis of the camera 42 faces, the focal length and angle of view of the camera 42, etc., and is associated with the captured image data which is the data of the captured image. In the processing of step S101, the actual shooting position information acquisition unit 110 particularly acquires information about the actual shooting position of the camera 42 in real space at the time the image was captured from the captured image data.

Next, the virtual distance generation unit 120 in the information processing system generates virtual distance information indicating the virtual distance from the virtual shooting position corresponding to the real shooting position of the camera 42 in the real space to the shooting target position, which is the position of the shooting target in the virtual space, in the virtual space generated by the processor 10 (step S102). As an example, the virtual distance generation unit 120 can obtain the virtual distance to the shooting target position, which is the position of the shooting target in the virtual space, by calculating the length of the line segment connecting the virtual shooting position in the virtual space and the shooting target position (the position of the three-dimensional model corresponding to the component of the shooting target) in the three-dimensional coordinate system of the virtual space.

Next, the size determination unit 130 in the information processing system determines the size of the image data at the position of the subject to be photographed in the virtual space based on the angle of view information and focal length information of the camera 42, which are included in the photographing condition information contained in the photographed image data acquired when the camera 42 photographed the image, and the virtual distance information generated by the virtual distance generation unit 120 (step S103).

As explained with reference to FIG. 6 as an example, the size determination unit 130 calculates the size of a first plane (the "near plane" of the viewing frustum shown in FIG. 6) that is a focal distance away from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42, and then uses the size of the first plane, the focal distance, and the virtual distance to calculate the size of a second plane that is a virtual distance away from the virtual shooting position, thereby determining the size of the image data at the shooting target position in the virtual space.

In this way, by determining the size of the image data at the position of the subject in the virtual space, when the captured image data is projected at the position of the subject in later processing (the position of the three-dimensional model corresponding to the component of the subject), it is possible to make the image data a size appropriate for the position of the subject. Furthermore, if the three-dimensional model data includes dimensional information and scale information, it is possible to obtain the actual size of the component in real space corresponding to the three-dimensional model at the position of the subject from the size of the image data.

Next, the image projection unit 140 in the information processing system projects image data of the size specified by the size specification unit 130 onto the subject model, which is a three-dimensional model of the subject that exists at the subject position in the virtual space (step S104). As a result, an image of the real-space component that corresponds to the subject model of the subject that exists at the subject position is pasted onto the subject model of the subject that exists at the subject position at a size that matches the size of the subject model, and the texture of the real-space component (subject) is added to the surface of the subject model.

Next, the difference detection unit 150 in the information processing system detects the difference in shape between the subject shown in the image data projected by the image projection unit 140 into the virtual space and the subject model corresponding to that subject (step S105).
As an example, the difference detection unit 150 extracts characteristic parts of the object to be photographed that are reflected in the projected image data and corresponding characteristic parts of the object to be photographed that are projected onto the image data, and detects the difference in shape or position between the object to be photographed that are reflected in the projected image data and the object to be photographed model based on the positional relationship between these characteristic parts. By using the difference information thus obtained, for example, it becomes possible to correct the position, shape, size, etc. of the image data that the image projection unit 140 projects in the virtual space, and to arrange the image data so that it is appropriately superimposed on the object to be photographed model.

Next, the imaging subject identification unit 160 in the information processing system identifies at least one of the imaging subject model present at the imaging subject position in the virtual space or the imaging subject space based on the center position of the angle of view of the camera 42 when the camera 42 captured the captured image data (step S106).

In the processing of step S106, the shooting subject identification unit 160 first determines a virtual shooting direction in the virtual space corresponding to the actual shooting direction of the camera 42 based on information about the actual shooting direction, which is the direction in real space in which the optical center axis of the camera 42 faces at the time the captured image data was captured, and then identifies the shooting subject model or shooting subject space that exists at the shooting subject position in the virtual space, where a virtual straight line extending from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 in the virtual space intersects with the virtual shooting direction.

Finally, the image linking unit 170 in the information processing system links the image data to at least one of the subject model or subject space identified by the subject identification unit 160 and stores the image data in the storage unit 200 (step S107). This makes it possible, for example, to search for the subject model or subject space corresponding to the image data stored in the photographed image data storage unit 220, or to search for image data corresponding to the subject model or subject space stored in the three-dimensional data storage unit 210, and enables the image projection unit 140 to automatically identify the image data to be projected at the subject position in step S104.

In this way, according to this embodiment, an information processing system or the like is provided that can easily manage images captured by a moving body 4 in association with the position and size of the subject to be captured. In particular, the information processing method according to this embodiment includes identifying the size of image data at the position of the subject to be captured in virtual space, so that when the captured image data is projected onto the position of the subject to be captured (the position of the three-dimensional model corresponding to the constituent feature of the subject to be captured), the image data can be sized to suit the position of the subject to be captured, and further, if the three-dimensional model data includes dimensional information and scale information, it is possible to obtain the actual size of the constituent feature in real space corresponding to the three-dimensional model at the position of the subject to be captured from the size of the image data.

The mobile object 4 may further include devices, equipment, etc. that are used to inspect the interior and/or exterior walls of the structure for the presence or absence of specific events. More specifically, any device necessary to know the state of the structure being inspected may be employed, such as an imaging device (visible light camera, infrared camera, metal detector, ultrasonic measuring device, etc.), a keystroke device, etc., a detection device (metal detector), a sound collection device, an odor measuring device, a gas detector, an air pollution measuring device, a detection device (a device for detecting cosmic rays, radiation, electromagnetic waves, etc.), etc.

The information processing method according to this embodiment may be executed, for example, during security or surveillance within a structure, and may further include devices, equipment, etc. used for security or surveillance. More specifically, any device necessary for capturing images and detecting abnormalities or intruders in the structure to be guarded or monitored, such as an imaging device (visible light camera, infrared camera, night vision camera, metal detector, ultrasonic measuring device, etc.) or a sensor device (motion sensor, infrared sensor, etc.), may be employed.

Furthermore, the mobile object 4 can be suitably used as a mobile object for photography equipped with a camera or the like, and can also be used in various industries such as security, infrastructure monitoring, surveying, inspection of buildings and structures such as sports venues, factories, and warehouses, and disaster response.

The above-described embodiment is merely an example to facilitate understanding of the present invention, and is not intended to limit the interpretation of the present invention. The present invention can be modified and improved without departing from the spirit of the invention, and it goes without saying that the present invention includes equivalents.

Reference Signs List 1 Management server 2 User terminal 4 Mobile object 5 Mobile object storage device

Claims (7)

an actual photography position information acquisition unit that acquires actual photography position information indicating an actual photography position in a real space associated with the acquired photographed image data;
a virtual distance generating unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space;
a size determination unit that determines a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
An information processing system comprising: An image projection unit projects the image data of the specified size onto a shooting target model present at the shooting target position in the virtual space.
The information processing system according to claim 1 . Further comprising a difference detection unit that detects a difference in shape between the object to be photographed shown in the projected image data and the object to be photographed model.
The information processing system according to claim 2 . and a shooting target specifying unit that specifies at least one of a shooting target model present at the shooting target position in the virtual space and a shooting target space based on a center position of the angle of view when the captured image data is captured.
The information processing system according to claim 1 . An image linking unit that links the image data to at least one of the shooting target model or the shooting target space identified by the shooting target identification unit and stores the image data.
5. The information processing system according to claim 4. acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data;
generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space;
specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
2. An information processing method implemented by a computer, comprising: acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data;
generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space;
specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information;
A program that causes a computer to execute the above.

Images