IL314340A - Pilot debriefing station - Google Patents

Description

PILOT DEBRIEFING STATION TECHNICAL FIELD The presently disclosed subject matter relates to training of combat pilots, and in particular to implementation of mission debriefing systems.

BACKGROUND Problems of implementation in systems of flight simulation systems for pilot training have been recognized in the conventional art and various techniques have been developed to provide solutions.

GENERAL DESCRIPTION According to one aspect of the presently disclosed subject matter there is provided a system of post-mission tactical pilot debriefing, the system comprising a processing circuitry (PC) configured to: a) construct a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: i. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or ii. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generate, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality-based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generate an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and d) present the generated IFS on one or more respective 2D and/or VR and/or MR devices.

In addition to the above features, the system according to this aspect of the presently disclosed subject matter can comprise one or more of features (i) to (xiii) listed below, in any desired combination or permutation which is technically possible: (i) wherein the sensor data collected during the flight comprises data descriptive of one or more of: a) cockpit audio; b) infrared sensor events; c) radar targets; d) radar warning receiver threats; e) hands-on throttle-and-stick (HOTAS) events; f) Up Front Control Panel (UFCP) events; g) onboard weapons type and status; h) weapon release events; i) missile seeker status; j) missile envelope; k) aircraft yaw, pitch, roll, altitude, and/or velocity; 30 l) global positioning system (GPS), inertial navigation system (INS), or attitude and heading reference system (AHRS) data; m) pulse-per-second (1PPS) or coordinated universal time (UTC) time; n) mission load data; o) flight plan; and p) local in-aircraft events or malfunctions. (ii) the video data collected during the flight comprises one or more of: a) Head Up Display (HUD) video; b) mounted display video; c) multifunction color display (MFCD) video; d) enslaved pod video; and e) radar video. (iii) the PC is configured generate VR-based data, such that, responsive to presenting the VR-based data on a first VR or MR device: reconstructed video based on the constructed flight replay data segment is displayable at a first virtual location within the display on the first MR device. (iv) the PC is configured to to generate additional VR-based data such that, responsive to presenting the additional data on a second MR device of one or more additional MR devices: reconstructed video based on the constructed flight replay data segment is displayable at the first virtual location within the display on the second MR device, and 30 an avatar representing the user of the first VR or MR device is displayable within the display on the second MR device, thereby enabling users of two or more VR or MR devices to view reconstructed video at the first virtual location. (v) the PC is configured to configured to generate MR-based data, such that, responsive to presenting the MR-based data on a first MR device: a) a view of a first user’s surroundings, based on data from a camera associated with the first MR device, is displayable on a first portion of a respective view on the first MR device; and b) reconstructed video based on the constructed flight replay data segment is displayable on a second portion of the respective view on the first MR device, the reconstructed video being displayed at a first virtual location within the respective view on the first MR device. (vi) the PC is further configured to generate additional MR-based data such that, responsive to presenting the additional data on a second MR device of one or more additional MR devices: a) a view of a respective user’s surroundings, based on data from a camera associated with the second MR device, is displayable on a first portion of a respective view on the second MR device; and b) reconstructed video based on the constructed flight replay data segment is displayable on a second portion of the respective view on the second MR device, the reconstructed video being displayed at the first virtual location within the respective view on the second MR device, (vii) the PC is configured to perform the generating of the IFS in accordance with user input indicative of one or more of degrees of flight manipulation, the degrees of flight manipulation selected from: a. yaw, b. pitch, c. roll, d. position, e. velocity, and f. acceleration. (viii) the PC is further configured to, responsive to an operator command, modify one or more scenario parameters of the IFS, the one or more scenario parameters being selected from a list consisting of: a) number of peer and/or enemy aircraft; and b) one or more characteristics of peer and/or enemy aircraft. (ix) at least one of the one or more characteristics of the peer and/or enemy aircraft are selected from: a. aircraft motion characteristics; b. weapon types; and c. weapon events. (x) the PC is configured to generate the IFS utilizing one or more machine-learning-based aircraft emulations. (xi) the PC is configured to generate the IFS based on, for at least a first time-interval, one or more degrees of flight manipulation associated with an operator-selected predefined flight scenario. (xii) the PC is configured to generate IFS data such that, responsive to presenting the IFS data on a suitable display device, the display includes instructions indicative of a recommended pilot behavior. (xiii) the PC is further configured to: e) repeat a) – c) for one or more additional aircraft of the mission.

According to another aspect of the presently disclosed subject matter there is provided a processing circuitry-based method of facilitating post-flight tactical pilot debriefing, the method comprising: a) constructing a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: iii. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or iv. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generating, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality-based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generating an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and d) presenting the generated IFS on one or more respective 2D and/or VR and/or MR devices.

This aspect of the disclosed subject matter can further optionally comprise one or more of features (i) to (xiii) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.

According to another aspect of the presently disclosed subject matter there is provided a computer program product comprising a computer readable non-transitory storage medium containing program instructions, which program instructions when read by a processor, cause the processing circuitry to perform a method of facilitating post-flight tactical pilot debriefing, the method comprising: a) constructing a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: v. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or vi. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generating, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality-based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generating an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and d) presenting the generated IFS on one or more respective 2D and/or VR and/or MR devices.

This aspect of the disclosed subject matter can further optionally comprise one or more of features (i) to (xiii) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.

BRIEF DESCRIPTION OF THE DRAWINGS In order to understand the invention and to see how it can be carried out in practice, embodiments will be described, by way of non-limiting examples, with reference to the accompanying drawings, in which: Fig. 1 illustrates a logical block diagram of an example post-flight pilot debriefing station, in accordance with some embodiments of the presently described subject matter; Fig. 2A illustrates an example logical layout of a reconstructed cockpit and its instrumentation, in accordance with some embodiments of the presently described subject matter; Fig. 2B illustrates a display variation of the example logical layout of a reconstructed cockpit and its instrumentation, in accordance with some embodiments of the presently described subject matter; Fig. 3A illustrates an example view of a mixed reality view of a user utilizing a mixed reality headset, in accordance with some embodiments of the presently described subject matter; Fig. 3B illustrates an example view of a virtual debriefing room as seen by a user utilizing a mixed reality headset, in accordance with some embodiments of the presently described subject matter; and Fig. 4 illustrates a flow chart of an example method presenting, to a user, a modifiable reconstruction of a completed flight, in accordance with some embodiments of the presently described subject matter.

DETAILED DESCRIPTION In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the presently disclosed subject matter may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the presently disclosed subject matter.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing", "computing", "comparing", "encrypting", “decrypting”, "determining", "calculating", “receiving”, “providing”, “obtaining”, “emulating” or the like, refer to the action(s) and/or process(es) of a computer that manipulate and/or transform data into other data, said data represented as physical, such as electronic, quantities and/or said data representing the physical objects. The term “computer” should be expansively construed to cover any kind of hardware-based electronic device with data processing capabilities including, by way of non-limiting example, the processor, mitigation unit, and inspection unit therein disclosed in the present application.

The terms "non-transitory memory" and “non-transitory storage medium” used herein should be expansively construed to cover any volatile or non-volatile computer memory suitable to the presently disclosed subject matter.

The operations in accordance with the teachings herein may be performed by a computer specially constructed for the desired purposes or by a general-purpose computer specially configured for the desired purpose by a computer program stored in a non- transitory computer-readable storage medium.

Embodiments of the presently disclosed subject matter are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the presently disclosed subject matter as described herein.

Attention is directed to Fig. 1 , which illustrates a logical block diagram of an example post-flight pilot debriefing station, in accordance with some embodiments of the presently disclosed subject matter.

Debriefing platform 100 can include: - Completed flight data store 102 , which can be a processor-based server that stores data recorded by real or simulated aircraft during flight. The stored flight data can be used for digital reconstruction of the flight experience, which in turn can be utilized for pilot debriefing (e.g. directing pilot attention to events and eliciting explanations of pilot behavior) and training. - Flight playback/simulation subsystem 104 , which can be a processor-based system that utilizes completed flight data (e.g. from completed flight data store 102 ) to reconstruct pilot experiences of completed flights (e.g. via virtual reality display devices), as well as flight simulations that are modifications of completed flights, as described in detail hereinbelow. Flight playback/simulation subsystem 104 can output reconstruction/simulation data in multiple formats for presentation on multiple devices in a synchronized manner. - Presentation platform 106 , which can include display devices such as dimensional screens as well as headsets enabling presentation of virtual reality and/or mixed reality content. Presentation platform 106 can receive data for different devices from e.g. flight playback/simulation subsystem 104, and present these on single or multiple devices (e.g. in a synchronized manner).

Completed flight data store 102 can include processor 118A and memory 120A , as well as recorded flights database 116 .

Completed flight data store 102 can include processor-implemented functional units, including flight data preparation unit 112 , which in turn can include aircraft-recorder specific translation unit 114 .

Flight data preparation unit 112 can receive completed flight data from e.g. a storage medium such as a thumb drive, disk drive, or optical media, or a legacy medium such as a magnetic cassette. Flight data preparation unit 112 can then process the completed flight data (e.g. as described below) and store the processed data to recorded flights database 116 .

By way of non-limiting example, completed flight data can include video from various sources, including: - Cockpit video (e.g. from a pilot point of view, overhead point of view, or other perspective) Completed flight data can further include data from aircraft sensors, such as: - Cockpit audio - infrared sensor events - radar targets - radar warning receiver (RWR) threats and target data - hands-on throttle-and-stick (HOTAS) events - Up Front Control Panel (UFCP) events - onboard weapons type and status - weapon release events - missile seeker status - missile envelope - aircraft yaw, pitch, roll, altitude, and/or velocity - global positioning system (GPS), inertial navigation system (INS), or attitude and heading reference system (AHRS) data - pulse-per-second (1PPS) or coordinated universal time (UTC) time - mission load data - flight plan - local in-aircraft events (e.g. locking on a target, firing a weapon etc.) or malfunctions Completed flight data can include a sequence of measurements taken over a period of time (e.g. the entire flight time of the aircraft). In some examples, the components of completed flight data can be maintained in time-based frames with a certain resolution e.g. 25 milliseconds.

Recorded flights database 116can maintain data in various data formats. Aircraft-recorder specific translation unit 114 can, for example, recognize, detect, or receive operator input indicative of particular formats, and then convert the data to e.g. a format usable by flight playback/simulation subsystem 104 .

It is noted that completed flight data can include data from multiple actual or simulated aircraft flying simultaneously (e.g. in a multi-pilot mission (exercise)).

Flight playback/simulation subsystem 104 can include a processing circuitry (not explicitly shown) which includes processor 118B and memory 120B.

Processor 118B can be a suitable hardware-based electronic device with data processing capabilities, such as, for example, a general purpose processor, digital signal processor (DSP), a specialized Application Specific Integrated Circuit (ASIC), one or more cores in a multicore processor, etc. Processor 118B can also consist, for example, of multiple processors, multiple ASICs, virtual processors, combinations thereof etc.

Memory 120B can be, for example, a suitable kind of volatile and/or non-volatile storage, and can include, for example, a single physical memory component or a plurality of physical memory components. Memory 120B can also include virtual memory. Memory 230 can be configured to, for example, store various data used in computation.

Processing circuitry of flight playback/simulation subsystem 104 can be configured to execute several functional modules in accordance with computer-readable instructions implemented on a non-transitory computer-readable storage medium. Such functional modules are referred to hereinafter as comprised in the processing circuitry. These modules can include, for example, VR / 2D / 3D reconstruction unit 108 and “What If?” unit 110 , and their subunits.

Flight playback/simulation subsystem 104 can be operably connected to completed flight data store 102 (for example: via a computer network connection).

VR / 2D / 3D reconstruction unit 108 can receive completed flight data (e.g. data from completed flight data store 102 ).

In some embodiments, in response to e. g. an instruction from a system operator of flight playback/simulation subsystem 104 , VR / 2D / 3D reconstruction unit 108 can receive completed flight data, and generate one or more datastreams – each of which, when presented on appropriate devices, can reconstruct the completed flight in a particular view.

More specifically: VR / 2D / 3D reconstruction unit 108 can generate, for example, one or more of the following datastreams for some or all of the duration of the flight: - a “pilot view” of the cockpit (including instruments such as radar, altimeter etc.) viewable on a two-dimensional display screen - an overhead view of the cockpit (viewable on a two-dimensional display screen) - an external view of the aircraft and its surroundings (viewable on a display screen, such as a standard two-dimensional display screen, or a three-dimensional convex or concave screen using a suitable projector). In some examples, this external aircraft view can provide an interactive ability to change the viewpoint (i.e. perspective) from which the aircraft is shown – thus providing a kind of “three-dimensional” view on a two-dimensional screen. In some examples, this external aircraft view is constructed from a momentary attitude of the flying platform. - a virtual reality (VR) presentation of a “pilot view” of the cockpit (including instruments such as radar display, data link entities , radar warning receiver (RWR) threats, tactical situation awareness display (TSD) etc.) viewable on a VR headset - a mixed reality (MR) presentation of a pilot debriefing space, viewable on a MR headset. For example: o in some examples, a pilot utilizing an MR headset can be located in a physical debriefing room with e.g. an instructor and/or other pilots who may also be utilizing MR headsets. A camera that is e.g. mounted on the headset device can provide a view of the physical debriefing room, which the MR headset(s) can then display to the pilot(s)/trainer.

The MR headset(s) can then also display one or more “virtual screens” at one or more respective virtual locations in the debriefing room, and a reconstruction of the completed flight can be shown on the virtual screens. In this manner the participants in the debriefing can view, point to, and discuss the one or more “virtual screens” presented in multiple headsets. o In some other examples, the entire debriefing room can be virtual. In this case the other participants may be located in different physical locations. The other participants (who can be MR/VR headset users or 2D screens) can appear as avatars. Again the MR/VR headset(s) or 2D screen(s) can then display one or more “virtual screens” at a respective virtual location(s) in the debriefing room, and a reconstruction of the completed flight can be shown on the virtual screen.

VR / 2D / 3D reconstruction unit 108 can simultaneously reconstruct flight views for different actual or simulated aircraft flying simultaneously (e.g. in a multi-aircraft mission (exercise)), for synchronized presentation on different devices.

Terrain database 136 can include terrain data (elevation, obstacles, visual wavelength, infrared wavelength etc.). Terrain database 136 can include ground coverage information. In some examples, terrain database 136 can be used in flight reconstruction (e.g. in an external aircraft view with configurable viewpoint, as described above).

Synchronization unit 134 can receive different types of completed flight data (video, sensors etc.), and perform synchronization of multiple inputs that pertain to the same time interval. Synchronization unit 134 can then generate a data segment describing the completed flight at a particular time interval, which can then be used to generate 30 respective frames in presentation-specific datastreams, as described below with reference to Fig. 4 .

Virtual space unit 130 can be a component that maintains objects of a virtual space etc. as described above with reference to VR/2D/3D reconstruction unit 108 . Mixed reality unit 132 can be a component that maintains mixed reality space objects etc. as described above with reference to with reference to VR/2D/3D reconstruction unit 108 .

In some embodiments, flight playback/simulation subsystem 104 can, from a particular instant in a presentation of a reconstruction of a completed flight, begin an interactive flight simulation continuous from that point. That is to say: the interactive flight simulation can begin from the topology, flight parameters, peer/enemy/target configurations etc. of the completed flight. This functionality is referred to herein as “what if?” functionality. Flight playback/simulation subsystem 104 can then accordingly generate interactive flight simulation data for display on a 2D screen and/or VR device and/or MR device.

In some such embodiments, flight playback/simulation subsystem 104 can additionally provide an operator with capability to modify particular parameters of the flight, as will be described below.

Accordingly, “What if?” unit 110 can receive flight data from VR / 2D / 3D reconstruction unit 108 . Interactive flight simulation unit 124can then supply datastreams of the simulation (based on the received flight data) in appropriate formats (eg. as described above) for presentation by e.g. display units of presentation platform 106 .

“What if” unit 110 can generate data of an interactive flight simulation which is based on modification of the behavior of entities of the recorded flight from a given instant onward. For example, “What if” unit 110 can employ AI agents that emulate behavior of peer or enemy aircraft or other entities (e.g. new threats and/or new targets), and mission flight results can change as a consequence.

“What if” unit 110 can generate data of an interactive flight simulation for one pilot (i.e. post-mission) while generating data for pilots of other aircraft of the mission so that the other pilots view (or “experience”) respective recorded events of their own aircraft - which have synchronized with and integrated together with the new events generated by “what if” unit 110 .

The AI agents employed in “What if” unit 110can utilize deep neural networks and can be implemented using reinforcement learning, and/or using classification models and state machines.

Changes in flight behavior performed by “What if” unit 110can include : 1. Providing the pilots with capability change the (debriefed) flight path using six degrees of freedom and stick to maneuver the aircraft 2. Enabling an operator to enable AI Agents which simulate enemy targets in one or more scenarios 3. Enabling an operator to change a type of missile discharged - for example: replacing the infrared (IR) missile actually used, with a radar-guided (RDR) RDR missile. In this case, “What if” unit 110can employ artificial intelligence (AI) or other methods to calculate the trajectory of the missile. 4. Enabling an operator to change release of an air-to-ground (AG) weapon from a different aircraft weapon station or to utilize a different weapon type 5. Enabling an operator to change a weapon delivery maneuver or weapon delivery mode 6. Enabling an operator to change parameters such as: radar ghost target type, velocity, heading, and flight attitude 7. Enabling an operator to designate targets from another sensor Interactive flight simulations can provide e.g. 6 degrees of flight manipulation to the simulation pilot i.e.: a) yaw, b) pitch, c) roll, d) x, e) y, and f) z.

Flight playback/simulation subsystem 104 can generate interactive flight simulations (e.g. “what if?” functionality) in modes such as the following: a) Continuous flight mode: the interactive flight simulation continues the topology, flight parameters, peer/enemy/target configurations etc. of the completed flight b) Predefined scenario mode: in some embodiments, flight playback/simulation subsystem 104 enables operator selection of a predefined flight scenario (e.g. stored in a database or prepared scenarios/controlled new events 128 . Responsive to an operator selection of given predefined flight scenario, “What if?” unit 110 can generate, for certain amount of time (or beginning/ending in response to certain events) one or more respective values of one or more degrees of flight manipulation that are associated with the selected predefined flight scenario. Thus, for a particular interval, the simulation pilot has less than the full number of degrees of manipulation/control of the aircraft. c) Scenario modification: in some embodiments, flight playback/simulation subsystem 104 enables operator modification of the current scenario, for example: - modifying the number of peer or enemy aircraft - modifying one or more motion characteristic of one or more peer or enemy aircrafts - modifying a weapon event of a peer or enemy aircraft In some examples, the pilot (directly, or in response to operator instruction) is able to take control in any of the modes to return again to the replayed scenario.

Flight playback/simulation subsystem 104 can provide an “AI superpilot” function, which provides on-display direction of suggested pilot actions and/or provide 30 AI-based textual and/or audio evaluations of the pilot actions. The functionality can be implemented in AI superpilot unit 122 . In some examples, AI superpilot unit 122can perform the entire flight scenario - without pilot input – for instructional purposes.

Presentation platform 106 can be a physical or logical grouping of display devices. Display devices of presentation platform 106 can be operably connected to flight playback/simulation subsystem 104and can present reconstruction and simulation data on different display devices in a synchronized manner. In the example of Fig. 2B , 2D display unit 140A can display an “external view” of an aircraft, while Pilot B and Pilot C observe reconstructions of their flight on respective 2D display units 140B 140C. Pilot A views a reconstruction of his/her flight on VR/Mixed reality display unit (headset) 150 . Presentation platform 106 can include any suitable number and combination of 2D display units, 3D display units (eg. concave or convex screen), MR headsets, and VR headsets.

It is noted that in some embodiments, functions of the flight playback/simulation subsystem 104 can be located within presentation platform 106 . By way of non-limiting example: functionality of VR/2D/3D reconstruction unit 108 can be partially or fully implemented within VR/mixed reality display unit 150 .

It is noted that the teachings of the presently disclosed subject matter are not bound by the system described with reference to FIG. 1 . Equivalent and/or modified functionality can be consolidated or divided in another manner and can be implemented in any appropriate combination of software with firmware and/or hardware and executed on a suitable device. For example, flight playback/simulation subsystem 104can be a standalone entity, or integrated, fully or partly, with other entities.

Attention is directed to Fig. 2A , which illustrates an example logical layout of a reconstructed cockpit and its instrumentation, in accordance with some embodiments of the presently disclosed subject matter.

Fig. 2A illustrates a viewing space 205 of a user. The viewing space 205 can be, for example, a 2-dimensional screen displaying a pilot view from the cockpit, a virtual reality pilot view from the cockpit, or - in a mixed reality debriefing room - a virtual 2-dimensional screen showing a pilot view of the cockpit.

The instruments shown in viewing space 205 can be replayed from video, or can be reconstructed from retrieved data.

Viewing space 205 can include a recorded or reconstructed up-front control panel (UFCP) 225 , which can display UFCP events, navigation waypoints and routes.

Viewing space 205 can include a recorded or reconstructed “head up display” (HUD) 210 . The HUD 210can display for example: - Navigation information e.g flight path vector, waypoint information, course deviation indicators etc. - Air-to-air (AA) information e.g. symbols indicating the position, heading, and aspect angle of detected or designated airborne threats; weapons status; visual cues alerting the pilot to the presence of nearby enemy aircraft, such as warning symbols or threat rings etc. - Air-to-ground (AG) information e.g. symbols indicating the location, type, and status of ground targets, information about the availability, selection, and readiness of air-to-ground weapons; weapon release cues etc.

Viewing space 205 can include a recorded or reconstructed Multi-Function Color Display (MFCD) 240 . The MFCD 240can be a high-resolution color screen that can be configured to show different types of information essential for mission execution.

Among the data that can be displayed on MFCD 240is: - Radar: providing real-time information about nearby aircraft, terrain, and weather conditions, and aiding in navigation and threat detection) - Terrain Display System/Moving Map (TDS/MAPS): providing detailed terrain maps and situational awareness, crucial for navigation and avoiding obstacles - Stores Management System (SMS): controls and monitors the status of weapons and external stores, including missiles, bombs, and fuel tanks, enabling pilots to select, arm, and deploy ordnance as required for mission objectives. - Information from external Pods (such as Targeting Pods): can include sensor feeds, target information, and controls for surveillance, reconnaissance, and target acquisition.

Viewing space 205 can include a recorded or reconstructed view of a “hands on throttle and stick” (HOTAS) 215 which the pilot can use to manipulate aircraft functions such as weapon release in addition to controlling navigation.

Viewing space 205 can include timeline bar 245 . Timeline bar 245 can be a graphical or textual user interface element which can: a) display progress of a flight reconstruction, showing – for example – time elapsed and/or time remaining and/or percentage completed in the reconstructed flight b) enable an operator to move forward or backward to an earlier or later point in the flight reconstruction Fig. 2B , which illustrates a display variation of the example layout of a reconstructed cockpit and its instrumentation, in accordance with some embodiments of the presently disclosed subject matter.

In Fig. 2B , 4 different flight reconstructions (e.g. from 4 simultaneously flown aircraft) are displayed, in a synchronized manner, within a single 2D screen. In the example of Fig. 2B . a single timeline bar 245B , controls/displays timing data for all the simulations. It is noted that the display of Fig. 2B can include any number of flight reconstructions.

Attention is directed to Figs. 3A-3B , which illustrate example logical layouts of a mixed reality/virtual reality user view of a debriefing room, in accordance with some embodiments of the presently disclosed subject matter.

Fig. 3A illustrates an example viewing space 305A of a user. The viewing space 205 can be, for example, a view from within a mixed reality headset.

Viewing space 305A can include a view of the surroundings 305A (from e.g. a camera associated with or affixed to an MR headset). This view of the surroundings 305Acan show the room the user is located in, and e.g. a person 320A .

Viewing space 305A can include a virtual screen 315A . Virtual screen 315Acan be located in a particular virtual location in the debriefing room, so that its position in the debriefing room is perceived by the user as fixed (e.g. while he/she is moving), so that multiple MR users will see virtual screen 315Aat the same place in the debriefing room. The virtual screen 315A can display a view of the reconstructed flight, as described in detail above.

In Fig. 3B, the debriefing room is itself virtual i.e. the participants need not be located in the same physical space. Thus, the viewing space of the VR (or MR) user 305B includes a view of the virtual space, and another user can appear as an avatar 320B . As in Fig. 3A , viewing space of the VR (or MR) user 305Bincludes the virtual screen 315B at a virtual location.

Fig. 4 illustrates a flow chart of an example method presenting, to a user, a modifiable reconstruction of a completed flight, in a manner suitable for post-flight tactical debriefing, according to some embodiments of the presently disclosed subject matter.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can – in some embodiments - receive completed flight data, and can synchronize 405 among the different sources (or types) of completed flight data to construct a flight replay data segment.

By way of non-limiting example: flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can, for a particular time offset within the completed flight, identify video data (e.g. as received from completed flight data store 102 ) of the particular time offset, and for a particular segment duration. Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can, then identify a value or series of values of different sensors (e.g. as received from completed flight data store 102 ) for the time offset and duration corresponding to the video. In some examples, flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) may convert the sensor data or video data from a different format, perform interpolation or other operations to construct the video segment and obtain the corresponding sensor measurements. The term "flight replay data segment" is herein interpreted to include data resulting from such an assembly process.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can next, from the constructed flight replay data segment, generate 410 a 2D screen video segment corresponding to (i.e. in time) to the flight replay data segment.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can generate 415 one or more of: - an in-cockpit view of the flight (e.g. pilot point-of-view) - an overhead view of the cockpit - an external view of the aircraft and its surroundings (optionally: with the ability to select the perspective of the view, thereby providing a kind of “three-dimensional” external view for e.g. the duration of the flight replay data segment.

In some embodiments, the video display of aircraft instruments is derivative of video recorded during the flight. In some other embodiments, the video display of aircraft instruments is a reconstruction based on the received sensor data. In some other embodiments, the video display of aircraft instruments is a combination of a derivation of video recorded and video reconstructed from sensor data.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can then present the one or more 2D screen video segment, for example, on one or more display devices 140A 140B 140C .

In some embodiments, flight playback/simulation subsystem 104does not generate or present a 2D video segment corresponding to the flight replay data segment.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can next - in some embodiments - from the constructed flight replay data segment, generate a virtual reality and or mixed reality view of the flight, corresponding (i.e. in time) to the flight replay data segment.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can generate one or more of: - a virtual reality (VR) presentation of a “pilot view” of the cockpit (including instruments such as radar, altimeter etc.) - a mixed reality (MR) presentation of a pilot debriefing space, viewable on a MR headset, e.g. viewing a real debriefing room or virtual debriefing room, as described above, with reference to Fig. 3A. - a VR presentation of a pilot debriefing space, viewable on a MR or VR headset, e.g. viewing a virtual debriefing room, as described above with reference to Fig. 3B.

In some embodiments, the VR/MR display of aircraft instruments is derivative of video recorded during the flight. In some other embodiments, the VR/MR display of aircraft instruments is a reconstruction based on the received sensor data. In some other embodiments, the VR/MR display of aircraft instruments is a combination of a derivation of video recorded and VR/MR display reconstructed from sensor data.

Flight playback/simulation subsystem 104 (for example: reconstruction unit 108 ) can present the one or more generated VR or MR views on a suitable device 150 .

In some embodiments, flight playback/simulation subsystem 104does not generate a virtual reality or mixed reality view of the flight.

Responsive to (for example) an operator command or other trigger, flight playback/simulation subsystem 104 (for example: interactive flight simulation unit 124 ) can generate 420 data, which when presented on a suitable display unit, presents an interactive flight simulation continuous with the constructed segment, based on the completed flight.

By way of non-limiting example: flight playback/simulation subsystem 104 (for example: interactive flight simulation unit 124 ) can generate a stream of data segments, based on the characteristics (e.g. aircraft location, aircraft motion parameters, location and motion characteristics of peer and/or enemy aircraft etc.) of the completed flight at the time offset of the operator command.

The interactive flight simulation can provide a user with degrees of control of the simulated flight as described above. Consequently, flight playback/simulation subsystem 104 (for example: interactive flight simulation unit 124 ) can utilize the user’s input in conjunction with the flight state in generation of the stream of data segments.

Additionally, the operator (for example) can specify various predefined scenarios, or changes to the completed flight scenarios, as described above with reference to Fig. 1 . Consequently, flight playback/simulation subsystem 104 (for example: interactive flight simulation unit 124 ) can utilize the specified scenario changes in generation of the stream of data segments.

It is noted that the teachings of the presently disclosed subject matter are not bound by the flow diagram illustrated in Fig. 4 , and that in some cases the illustrated operations may occur concurrently or out of the illustrated order (for example: operations 410 and 415 can be reversed). It is also noted that whilst the flow chart is described with reference to elements of the system of Fig. 1 , this is by no means binding, and the operations can be performed by elements other than those described herein.

It is to be understood that the invention is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Hence, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the presently disclosed subject matter.

It will also be understood that the system according to the invention may be, at least partly, implemented on a suitably programmed computer. Likewise, the invention contemplates a computer program being readable by a computer for executing the method of the invention. The invention further contemplates a non-transitory computer-readable memory tangibly embodying a program of instructions executable by the computer for executing the method of the invention.

Those skilled in the art will readily appreciate that various modifications and changes can be applied to the embodiments of the invention as hereinbefore described without departing from its scope, defined in and by the appended claims.

Claims (16)

1. A system of post-mission tactical pilot debriefing, the system comprising a processing circuitry (PC) configured to: a) construct a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: i. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or ii. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generate, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality-based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generate an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and d) present the generated IFS on one or more respective 2D and/or VR and/or MR devices.

2. The system of claim 1, wherein the sensor data collected during the flight comprises data descriptive of one or more of: a. cockpit audio; b. infrared sensor events; c. radar targets; d. radar warning receiver threats; e. hands-on throttle-and-stick (HOTAS) events; f. Up Front Control Panel (UFCP) events; g. onboard weapons type and status; h. weapon release events; i. missile seeker status; j. missile envelope; k. aircraft yaw, pitch, roll, altitude, and/or velocity; l. global positioning system (GPS), inertial navigation system (INS), or attitude and heading reference system (AHRS) data; m. pulse-per-second (1PPS) or coordinated universal time (UTC) time; n. mission load data; o. flight plan; and p. local in-aircraft events or malfunctions.

3. The system of any of claim 1, wherein the video data collected during the flight comprises one or more of: a) Head Up Display (HUD) video; b) mounted display video; c) multifunction color display (MFCD) video; d) enslaved pod video; and e) radar video.

4. The system of claim 1, wherein the PC is configured to generate VR-based data, such that, responsive to presenting the VR-based data on a first VR or MR device: reconstructed video based on the constructed flight replay data segment is displayable at a first virtual location within the display on the first MR device.

5. The system of claim 4, wherein the PC is further configured to generate additional VR-based data such that, responsive to presenting the additional data on a second MR device of one or more additional MR devices: reconstructed video based on the constructed flight replay data segment is displayable at the first virtual location within the display on the second MR device, and an avatar representing the user of the first VR or MR device is displayable within the display on the second MR device, thereby enabling users of two or more VR or MR devices to view reconstructed video at the first virtual location.

6. The system of claim 1, wherein the PC is configured to generate MR-based data, such that, responsive to presenting the MR-based data on a first MR device: a. a view of a first user’s surroundings, based on data from a camera associated with the first MR device, is displayable on a first portion of a respective view on the first MR device; and b. reconstructed video based on the constructed flight replay data segment is displayable on a second portion of the respective view on the first MR device, the reconstructed video being displayed at a first virtual location within the respective view on the first MR device.

7. The system of claim 6, wherein the PC is further configured to generate additional MR-based data such that, responsive to presenting the additional data on a second MR device of one or more additional MR devices: a. a view of a respective user’s surroundings, based on data from a camera associated with the second MR device, is displayable on a first portion of a respective view on the second MR device; and b. reconstructed video based on the constructed flight replay data segment is displayable on a second portion of the respective view on the second MR device, the reconstructed video being displayed at the first virtual location within the respective view on the second MR device, thereby enabling users of two or more MR devices to view reconstructed video at the first virtual location.

8. The system of claim 1, wherein the PC is configured to perform the generating of the IFS in accordance with user input indicative of one or more of degrees of flight manipulation, the degrees of flight manipulation selected from: a. yaw, b. pitch, c. roll, d. position, e. velocity, and f. acceleration.

9. The system of claim 1, wherein the PC is further configured to, responsive to an operator command, modify one or more scenario parameters of the IFS, the one or more scenario parameters being selected from a list consisting of: a. number of peer and/or enemy aircraft; and b. one or more characteristics of peer and/or enemy aircraft.

10. The system of claim 9, wherein at least one of the one or more characteristics of the peer and/or enemy aircraft are selected from: a. aircraft motion characteristics; b. weapon types; and c. weapon events.

11. The system of claim 1, wherein the PC is configured to generate the IFS utilizing one or more machine-learning-based aircraft emulations.

12. The system of claim 1, wherein the PC is configured to generate the IFS based on, for at least a first time-interval, one or more degrees of flight manipulation associated with an operator-selected predefined flight scenario.

13. The system of claim 1, wherein the PC is configured to generate IFS data such that, responsive to presenting the IFS data on a suitable display device, the display includes instructions indicative of a recommended pilot behavior.

14. The system of claim 1, wherein the PC is further configured to: e) repeat a) – c) for one or more additional aircraft of the mission.

15. A processing circuitry-based method of facilitating post-flight tactical pilot debriefing, the method comprising: a) constructing a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: i. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or ii. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generating, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality-based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generating an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and d) presenting the generated IFS on one or more respective 2D and/or VR and/or MR devices.

16. A computer program product comprising a computer readable non-transitory storage medium containing program instructions, which program instructions when read by a processor, cause the processing circuitry to perform a method of facilitating post-flight tactical pilot debriefing, the method comprising: a) constructing a segment of flight replay data, based on, at least, synchronizing data of two or more data sources selected from: i. video data, recorded during the mission from one or more respective video sources of a first aircraft, and/or ii. sensor data, collected during the mission from one or more respective sensors of the first aircraft, b) generating, based on, at least, the constructed flight replay data segment, one or more of: i. data indicative of a two-dimensional (2D) pilot's view corresponding to the constructed data segment, ii. data indicative of a three-dimensional (3D) external view, from a first view perspective, corresponding to the constructed data segment, and iii. data indicative of a virtual reality-based and/or a mixed reality- based pilot’s view corresponding to the constructed data segment, and present the one or more generated data on one or more respective 2D and/or virtual reality (VR) and/or mixed reality (MR) devices; c) responsive to an operator command, generating an interactive flight simulation (IFS) continuous with the constructed flight replay data segment; and 30 d) presenting the generated IFS on one or more respective 2D and/or VR and/or MR devices. For the Applicants, REINHOLD COHN AND PARTNERS By: