WO2016135447A1 - Système et procédé d'informations - Google Patents

Système et procédé d'informations Download PDF

Info

Publication number
WO2016135447A1
WO2016135447A1 PCT/GB2016/050364 GB2016050364W WO2016135447A1 WO 2016135447 A1 WO2016135447 A1 WO 2016135447A1 GB 2016050364 W GB2016050364 W GB 2016050364W WO 2016135447 A1 WO2016135447 A1 WO 2016135447A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
reality environment
mixed reality
building
screen
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/GB2016/050364
Other languages
English (en)
Inventor
Christopher James WHITEFORD
Nicholas Giacomo Robert Colosimo
Julian David WRIGHT
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BAE Systems PLC
Original Assignee
BAE Systems PLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from EP15275047.7A external-priority patent/EP3062220A1/fr
Priority claimed from GB1503114.9A external-priority patent/GB2535728A/en
Application filed by BAE Systems PLC filed Critical BAE Systems PLC
Publication of WO2016135447A1 publication Critical patent/WO2016135447A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/14—Display of multiple viewports
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/16—Use of wireless transmission of display information

Definitions

  • This invention relates generally to an information system and method and, more particularly but not necessarily exclusively, to a visual information and maintenance system and method for use in passive housekeeping and/or maintenance of a building or other structure.
  • servicing for the purposes of housekeeping and maintenance of some buildings and structures can be time-consuming, costly and may require down-time.
  • Structural analysis of high load environments, such as submarines and the like, is a very specialised activity, which particularly time consuming and often complex. Therefore, servicing is performed at distinct intervals, whereas critical damage could occur between services which may then go undetected, either until an associated emergency occurs, or until the next service time.
  • a mixed reality monitoring system for use within a building or other structure, the system comprising a headset for placing over a user's eyes, in use, said headset including a screen, image capture means for capturing images of the real world in the vicinity of the user, and a processor configured to generate a three-dimensional virtual reality environment and blend images of said real world environment into said three-dimensional virtual reality environment to create a mixed reality environment representative of a user's field of view and display said mixed reality environment on said screen, said system further comprising a storage module having stored therein a three-dimensional virtual model of the layout and at least one required condition of said building or structure, a comparison module for comparing said images of the real world in the vicinity of the user with a corresponding portion of said virtual model to identify differences in said layout and/or at least one condition, and an image processing module configured to generate image data representative of any identified differences and display said image data within said mixed reality environment on said screen.
  • the image data may comprise a coloured mask and the image processing module is configured to overlay said mask onto a corresponding area of said mixed reality environment displayed on said screen to indicate therein the location of an identified difference.
  • the image data may, additionally or alternatively, include verbal data indicative of an identified difference and/or recommended user action.
  • the image data generated to represent the identified difference can take any one of a number of different forms, dependent on the design of the system and/or the nature of the identified difference, and the present invention is not necessarily intended to be in any way limited in this regard.
  • the comparison module may be configured to receive data from at least one external sensor indicative of a condition of said building or other structure at the location of said sensor, compare said data with corresponding data stored in said virtual model to identify any differences in said condition and outputting data representative of an identified difference, said image processing module being configured to receive said signal and generate image data representative of said identified difference and display said image data within said mixed reality environment on said screen.
  • a system may further comprise a positioning module for determining a current location, within said building or structure, of said user.
  • the positioning module may be mounted in or on said headset.
  • the comparison module may be configured to receive, from remote sensors, data representative of the health or structural or environmental status of the building or other structure and/or equipment located therein, compare said data with corresponding data stored in said virtual model to identify any differences in said status and outputting data representative of an identified difference, said image processing module being configured to receive said signal and generate image data representative of said identified difference and display said image data within said mixed reality environment on said screen.
  • the headset may include at least one external sensor mounted therein or thereon.
  • control apparatus for a mixed reality information system according to any of the preceding claims, said control apparatus comprising a storage module having stored therein a three-dimensional virtual model of a building or other structure together with data representative of ambient conditions and status of said building or other structure at respective locations therein, a comparison module configured to receive image data representative of the real world in the vicinity of the user with a corresponding portion of said virtual model to identify differences in said layout and/or at least one condition or status, and an image processing module configured to generate image data representative of any identified differences for display on a user's screen.
  • the apparatus described above may be configured to update said stored three-dimensional virtual model using image data representative of a user's real world environment.
  • a mixed reality environment monitoring system for use within a building or other structure, for monitoring the condition and structural status of said building or other structure, the system comprising at least one headset for placing over a user's eyes, in use, image capture means for capturing images of the real world environment in the vicinity of the user, and a processor configured to generate a three- dimensional virtual reality environment and blend images of said real world environment into said three-dimensional virtual reality environment to create a mixed reality environment representative of a respective user's field of view and display said mixed reality environment on said screen, the system further comprising control apparatus as described above.
  • Yet another aspect of the present invention extends still further to a method of providing an environment monitoring system for a building or other structure, the method comprising providing a mixed reality system comprising a headset for placing over a user's eyes, in use, said headset including a screen, image capture means for capturing images of the real world environment in the vicinity of the user, and a processor configured to generate a three-dimensional virtual reality environment and blend images of said real world environment into said three-dimensional virtual reality environment to create a mixed reality environment representative of a user's field of view and display said mixed reality environment on said screen, the method further comprising providing a storage module and storing therein a three-dimensional virtual model of a building or other structure together with data representative of ambient conditions and status of said building or other structure at respective locations therein, providing a comparison module configured to receive image data representative of the real world in the vicinity of the user with a corresponding portion of said virtual model to identify differences in said layout and/or at least one condition or status, and configuring an image processing module to generate image data representative of any identified differences
  • Figure 1 is a front perspective view of a headset for use in a mixed reality system in respect of which a method and apparatus according to an exemplary embodiment of the present invention can be provided;
  • Figure 2 is a schematic block diagram illustrating the configuration of some principal elements of a mixed reality system for use in an exemplary embodiment of the present invention.
  • Figure 3 is a schematic illustration of a single image frame displayed on the screen of a mixed reality system according to an exemplary embodiment of the present invention.
  • Virtual reality systems comprising a headset which, when placed over a user's eyes, creates and displays a three dimensional virtual environment in which a user feels immersed and with which the user can interact in a manner dependent on the application.
  • the virtual environment may comprise a game zone, within which the user can play a game.
  • augmented and mixed reality systems have been developed, wherein image data captured in respect of a user's real world environment can be captured, rendered and placed within a 3D virtual reality environment.
  • the user views their real world environment as a three dimensional virtual world generated images captured from their surroundings.
  • a mixed reality display system may include a headset 100 comprising a visor 10 having a pair of arms 12 hingedly attached at opposing sides thereof in order to allow the visor to be secured onto a user's head, over their eyes, in use, by placing the curved ends of the arms 12 over and behind the user's ears, in a manner similar to conventional spectacles.
  • the headset is illustrated herein in the form of a visor, it may alternatively comprise a helmet for placing over a user's head, or even a pair of contact lenses or the like, and the present invention is not intended to be in any way limited in this regard.
  • a pair of image capture devices 14 for capturing images of the environment, such image capture devices being mounted as closely as possible aligned with the user's eyes, in use.
  • a typical mixed reality system further comprises a processor, which is communicably connected in some way to a screen which is provided inside the visor 10.
  • Such communicable connection may be a hard wired electrical connection, in which case the processor and associated circuitry will be mounted on the headset.
  • the processor may be configured to wirelessly communicate with the visor, for example, by means of Bluetooth or similar wireless communication protocol, in which case, the processor need not be mounted on the headset but can instead be located remotely from the headset, with the relative allowable distance between them being dictated, and limited only by, the wireless communication protocol being employed.
  • the processor could be mounted on, or formed integrally with, the user's clothing, or instead located remotely from the user, either as a stand-alone unit or as an integral part of a larger control unit, for example.
  • the processor receives image data from the image capture devices, and renders and blends such image data, in real time, into a displayed three dimensional virtual environment.
  • the concept of real time image blending for augmented or mixed reality is known, and several different techniques have been proposed.
  • the present invention is not necessarily intended to be limited in this regard.
  • one exemplary method for image blending will be briefly described.
  • a threshold function may be applied in order to extract that object from the background image. Its relative location and orientation may also be extracted and preserved by means of marker data.
  • the image and marker data is converted to a binary image, possibly by means of adaptive thresholding (although other methods are known).
  • the marker data and binary image are then transformed into a set of coordinates that match the location within the virtual environment in which they will be blended.
  • Such blending is usually performed using black and white image data.
  • colour data sampled from the source image can be backward warped, using homography, to each pixel in the resultant virtual scene. All of these computational steps require minimal processing and time and can, therefore, be performed quickly and in real (or near real) time.
  • image data within the mixed reality environment can be updated in real time.
  • an emergency guidance system comprises at least one mixed reality headset 100 and an information processing system 104, which may be integrated in or mounted on the headset 100 and/or provided at a fixed location within the building or structure and configured for wireless communication with the headset 100. It is envisaged that some exemplary embodiments may comprise a central information processing system for communication with a single headset, or in parallel with a plurality of headsets.
  • the processing functionality of the information processing system may be distributed, partly or fully, amongst individual processing units provided on or on the headsets, which may or may not be the same processing units used to provide on the screen mixed reality images of the wearer's environment derived from image data captured by the image capture devices on the headset, and the present invention is not necessarily intended to be in any way limited in this regard.
  • the processing system 104 is configured to receive, from one or more external sources 108, 1 10, data representative of, for example, the structural status of the building or structure, and the health and/or status or condition of (at least) key equipment therein.
  • the processing system 104 generally includes an interface to enable data to be transmitted therefrom and received thereby, in order that data that could potentially be changing dynamically is updated in real (or near real) time. Furthermore, the processing system 104 may be configured to receive, or have stored therein, a three dimensional, virtual model 1 12 of the building or structure.
  • This virtual model 1 12 includes, not only the internal structural layout of the building (or other structure, such as ship or submarine), but also essential equipment, such as fire extinguishers and other emergency equipment, and their expected locations within the building or structure, together with data representative of walkways and escape routes that are required to be kept clear of obstructions at all times.
  • model 1 12 may include, for example, fire doors that must be kept closed, and the like, and also correct temperatures, air quality and similar environmental characteristics of the various sectors of the building or structure.
  • the virtual model 1 12 is intended to represent precisely the correct layout and overall condition of the building or other structure.
  • the or each headset 100 may include an internal geo-location system for generating data representative of the relative location, within the building or structure, of the wearer of the respective headset and transmit such data, continuously or otherwise, to the processing system 104.
  • the or each user places a respective headset 100 over their eyes, and performs their normal duties around the building or structure, viewing their environment through the mixed reality environment displayed on the screen.
  • the image capture devices on their headset 100 captures images of their real world environment, which are not only used to generate the mixed reality display on the user's screen, but also transmitted to the information processor 104.
  • the information processor 104 is configured to compare the real world images against the corresponding portions of the virtual model 1 12 in order to identify anomalies. It will be appreciated that image recognition and matching techniques may be used to identify the area of the building or structure (and, therefore, the correct portion of the virtual model 1 12) corresponding to the location and viewpoint of the user. However, geo-location sensors on the user's headset may additionally or alternatively be used to facilitate the identification of the current location of the user such that a meaningful comparison between the real world images and the virtual model 1 12 can be made.
  • image data representative of the anomaly is generated by the processor 104 and inserted into the mixed reality environment displayed on the respective user's screen.
  • image data may take any convenient form, provided that it is immediately visually informative to the user.
  • a coloured (e.g. red) mask indicating to the user that the obstruction should be removed, or other action is required to be taken.
  • a fire extinguisher is determined to be missing from its required location, a similar visual indication could be inserted into the user's display to alert them that action is required.
  • written information giving further user guidance, could additionally or alternatively be inserted into the mixed reality display seen by the user.
  • real world image data could additionally (or alternatively) be received from static cameras mounted in and around the building or structure (e.g. CCTV cameras and the like), and image data from such external sources can be fed to the information processor 104 for comparison against the virtual model 1 12 in a similar manner, such that any anomalies detected therein can be visually represented and displayed on one or more users' screens, within their mixed reality display, thereby enabling potentially hazardous situations to be remotely identified and reported to appropriate occupants of the building or structure.
  • static cameras mounted in and around the building or structure e.g. CCTV cameras and the like
  • the processing system 104 is further configured to receive data from external sources 108, 1 10, such as heat sensors and structural sensors located at known points around the building or structure, and such data can be compared against respective 'normal' data for those locations stored within the virtual model 1 12.
  • sources of heat may be mapped to appropriate areas of the virtual model during normal conditions.
  • any anomalies i.e. changes from the stored model
  • these can be classified and representative image data thereof can be generated by the processing system 104 for insertion into one or more users' mixed reality displays, thereby enabling potentially hazardous structural or equipment malfunctions to be identified early and, therefore, addressed.
  • the headsets could be provided with spectral cameras that collect multi-spectral data from the real world environment in the vicinity of the user, and such data could then be employed to identify sources of heat within the user's field of view, compared against the stored model, and any variations from the model reported visually on the user's screen.
  • the processing system 104 may be configured to compare the real world images captured by the image captures devices on a user's headset, compare such images against the stored virtual model of the internal structure of the building, and identify areas of, for example, deformation (perhaps due to excess stress), which can then be visually reported on the user's screen as a potential fault or hazard and/or to maintenance teams for action.
  • the visual representation of a potential hazard/maintenance issue may be inserted into a user's mixed reality environment displayed on the screen, at a location within the image indicative of the location of the potential hazard or maintenance issue.
  • a schematic still frame of an image that might be presented to a user indicates that the duct 300 running along the ceiling of the structure is hot (and may also be coloured in red, for example), and that a fire extinguisher 302 is missing from the walkway.
  • the virtual model 1 12 could be constructed over time, using real world images from users' headsets, and updated thereby as required, such that an initial completely accurate model is not initially required to be constructed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un système de surveillance de réalité mixte destiné à être utilisé dans un bâtiment ou une autre structure, le système comprenant un casque d'écoute (100) destiné à être placé sur les yeux d'un utilisateur, en utilisation, ledit casque d'écoute comprenant un écran, un moyen de capture d'image pour capturer des images du monde réel au voisinage de l'utilisateur, et un processeur (104) configuré pour générer un environnement de réalité virtuelle tridimensionnel et mélanger des images dudit environnement du monde réel dans ledit environnement de réalité virtuelle tridimensionnel pour créer un environnement de réalité mixte représentatif du champ de vision d'un utilisateur et afficher ledit environnement de réalité mixte sur ledit écran, ledit système comprenant en outre un module de stockage (112) ayant, stocké dans ce dernier, un modèle virtuel tridimensionnel de la disposition et au moins une condition requise dudit bâtiment ou de ladite structure, un module de comparaison (104) pour comparer lesdites images du monde réel au voisinage de l'utilisateur à une partie correspondante dudit modèle virtuel pour identifier des différences dans ladite disposition et/ou au moins une condition, et un module de traitement d'image (104) configuré pour générer des données d'image représentatives de différences identifiées et afficher lesdites données d'image dans ledit environnement de réalité mixte sur ledit écran. Des données provenant de capteurs externes (108, 110) peuvent également être reçues par le module de comparaison (104) et comparées à des données correspondantes dans le modèle virtuel (112) pour identifier des différences.
PCT/GB2016/050364 2015-02-25 2016-02-15 Système et procédé d'informations Ceased WO2016135447A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP15275047.7A EP3062220A1 (fr) 2015-02-25 2015-02-25 Système et procédé d'informations
GB1503114.9A GB2535728A (en) 2015-02-25 2015-02-25 Information system and method
EP15275047.7 2015-02-25
GB1503114.9 2015-02-25

Publications (1)

Publication Number Publication Date
WO2016135447A1 true WO2016135447A1 (fr) 2016-09-01

Family

ID=55361894

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2016/050364 Ceased WO2016135447A1 (fr) 2015-02-25 2016-02-15 Système et procédé d'informations

Country Status (1)

Country Link
WO (1) WO2016135447A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107239141A (zh) * 2017-06-01 2017-10-10 成都积木创艺科技有限公司 一种基于沉浸式vr的建筑效果预演系统及方法
GB2618596A (en) * 2022-05-12 2023-11-15 Sony Interactive Entertainment Inc Auxiliary sensing system and method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100226535A1 (en) * 2009-03-05 2010-09-09 Microsoft Corporation Augmenting a field of view in connection with vision-tracking
US20110115816A1 (en) * 2009-11-16 2011-05-19 Alliance For Sustainable Energy, Llc. Augmented reality building operations tool
US20130169681A1 (en) * 2011-06-29 2013-07-04 Honeywell International Inc. Systems and methods for presenting building information
US20130321245A1 (en) * 2012-06-04 2013-12-05 Fluor Technologies Corporation Mobile device for monitoring and controlling facility systems

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100226535A1 (en) * 2009-03-05 2010-09-09 Microsoft Corporation Augmenting a field of view in connection with vision-tracking
US20110115816A1 (en) * 2009-11-16 2011-05-19 Alliance For Sustainable Energy, Llc. Augmented reality building operations tool
US20130169681A1 (en) * 2011-06-29 2013-07-04 Honeywell International Inc. Systems and methods for presenting building information
US20130321245A1 (en) * 2012-06-04 2013-12-05 Fluor Technologies Corporation Mobile device for monitoring and controlling facility systems

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107239141A (zh) * 2017-06-01 2017-10-10 成都积木创艺科技有限公司 一种基于沉浸式vr的建筑效果预演系统及方法
GB2618596A (en) * 2022-05-12 2023-11-15 Sony Interactive Entertainment Inc Auxiliary sensing system and method

Similar Documents

Publication Publication Date Title
GB2535728A (en) Information system and method
US10147234B2 (en) System and method for augmented reality display of electrical system information
US20220198730A1 (en) Information display by overlay on an object
JP5553405B2 (ja) 閉ざされた構造内の人員およびセンサの位置を示し、状況認識の増進を提供する、拡張現実ベース型システムおよび方法
US9538131B2 (en) Remote behavior navigation system and processing method thereof
US20170123747A1 (en) System and Method for Alerting VR Headset User to Real-World Objects
EP3163407B1 (fr) Procédé et appareil permettant d'alerter un utilisateur de visiocasque
US20150371444A1 (en) Image processing system and control method for the same
GB2535723A (en) Emergency guidance system and method
US12243210B2 (en) Maintenance support system, maintenance support method, program, method for generating processed image, and processed image
KR102650169B1 (ko) 열화상 카메라로 산업현장의 이상 온도를 감지 및 표현하고 경보를 발생시켜 알리는 시스템 및 이의 동작 방법
EP3062297A1 (fr) Système et procédé de guidage d'urgence
CN106292719B (zh) 地面站融合系统及地面站视频数据融合方法
TWI755834B (zh) 視覺圖資定位系統
US20190251726A1 (en) Superimposition of situation expression onto captured image
JP7395137B2 (ja) ヘッドマウント型温度分布認識装置
EP3062220A1 (fr) Système et procédé d'informations
WO2016135447A1 (fr) Système et procédé d'informations
WO2016135448A1 (fr) Système et procédé de guidage d'urgence
KR102054175B1 (ko) 열화상 감지 디바이스
US20250148612A1 (en) Information processing apparatus and program
JP7692443B2 (ja) 状況把握支援装置、状況把握支援方法、及びプログラム
TWI834390B (zh) 工作場域安全管理系統及方法
JP7059770B2 (ja) 警告画像提供システム
KR102134419B1 (ko) 열화상 감지 디바이스

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16704930

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16704930

Country of ref document: EP

Kind code of ref document: A1