WO2025117501A1 - Lunettes de réalité virtuelle à champ de vision complet - Google Patents

Lunettes de réalité virtuelle à champ de vision complet Download PDF

Info

Publication number
WO2025117501A1
WO2025117501A1 PCT/US2024/057391 US2024057391W WO2025117501A1 WO 2025117501 A1 WO2025117501 A1 WO 2025117501A1 US 2024057391 W US2024057391 W US 2024057391W WO 2025117501 A1 WO2025117501 A1 WO 2025117501A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual reality
eye
reality system
mouse
lens
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/057391
Other languages
English (en)
Inventor
Daniel A. Dombeck
Domonkos Peter Pinke
John Bassam ISSA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwestern University
Original Assignee
Northwestern University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwestern University filed Critical Northwestern University
Publication of WO2025117501A1 publication Critical patent/WO2025117501A1/fr
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0176—Head mounted characterised by mechanical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/0004—Microscopes specially adapted for specific applications
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B2207/00—Coding scheme for general features or characteristics of optical elements and systems of subclass G02B, but not including elements and systems which would be classified in G02B6/00 and subgroups
    • G02B2207/114—Two photon or multiphoton effect

Definitions

  • the present invention generally relates to virtual reality, particularly to full field of view virtual reality goggles.
  • Each mouse eye accepts a FOV of about 140 degrees (both in the azimuthal and vertical elevation directions), with about 40 degrees of binocular overlap in the azimuthal plane and more at larger vertical elevations at the resting eye position, leading to a full azimuthal FOV of about 240 degrees and vertical elevation FOV greater than about 200 degrees (panels A-B of FIG. 1).
  • Current virtual reality (VR) systems typically consist of a head-fixed mouse running on a treadmill with a surrounding visual display consisting of either a large, curved screen illuminated by a projector or multiple computer monitors assembled side-by-side.
  • Projection systems typically illuminate about 270 and 160 degrees of the azimuthal and vertical elevation, and monitor based systems about 220 and 140 degrees, leaving portions of the mouse FOV un-illuminated, particularly in the vertical elevation direction in the critical overhead region (panel C of FIG. 1). Additionally, in the distance between the mouse and VR screens in current systems (0.5-1 m), objects in the lab frame are visible (head-fixation bars, optical table-top, lick tube, screen bezels, cameras, etc.). Importantly, the microscope itself is within (and blocking) the overhead field-of-view of the mouse. These immobile objects do not move with the virtual simulation and therefore provide cue-conflicts between the virtual and lab reference frames while also partially blocking views of the virtual world.
  • Another important and unique feature of the mouse visual system is the large binocular region that mice maintain both to the front, and even more prominently, above their head (panel A of FIG. 1). Because of the separation of the eyes on the animal’s head, real world objects in the binocular FOV region are viewed by each eye at different angles (binocular disparity).
  • the overhead visual region is particularly important for rodent behavior and survival, as mice continually monitor binocular overlap for threats coming from above (panel B of FIG. 1).
  • Current VR systems generate a single rendering of the virtual world, so that each eye sees the same view of objects in the binocular region, eliminating stereo depth information that may be present (panel C of FIG. 1).
  • recording components such as an upright microscope
  • head fixation bars occlude the overhead visual region.
  • this invention relates to a virtual reality (VR) system, comprising a pair of concave lenses; and a pair of screens, arranged in relation to the pair of concave lenses and eyes of a subject, for fully illuminating the visual field of view (FOV) of the subject.
  • VR virtual reality
  • the virtual reality system is configured to image objects displayed on the screen onto the eye retinas of the subject through the concave lens.
  • the virtual reality system is configured to illuminate each eye with an about 180-degree field of view in all directions.
  • the virtual reality system is configured to separately illuminate each eye for stereo illumination of the binocular zone, thereby excluding a lab frame from view while also accommodating saccades.
  • the about 180-degree field of view includes about 140 degrees for each eye FOV and +/- 20 degrees for saccades.
  • each concave lens is a positive-meniscus lens having an inner surface operably facing an eye of the subject.
  • each concave lens is arranged such that each eye is centered at a predetermined distance from an inner surface of each positive-meniscus lens.
  • each screen is a curved illumination display.
  • each screen is a flexible light-emitting diode (LED) display.
  • LED flexible light-emitting diode
  • the virtual reality system provides a mean resolution of about 2.2 pixel s/degree, or better, across the about 180-degree FOV.
  • each screen is a high resolution organic light-emitting diode (OLED) screen configured to increase the pixels/degree to further exceed the acuity of mice or by incorporating other sensory modalities including olfactory auditory and tactile into the virtual simulation, so as to further increase the immersiveness of the VR experience for mice.
  • OLED organic light-emitting diode
  • the virtual reality system further comprises a pair of screen holders, each screen holder with a curvature matching that of the screen, to which the screen is affixed; and a pair of lens holders, each lens holder with the lens attached to one end and the other end mated to the screen holder such that the lens is centered at the desired distance from the screen.
  • each lens holder is mated to the screen holder with magnets.
  • an optical axis is the virtual reality system is aligned with the optical axis of the mouse eye at its resting position.
  • the lens holder when aligned, is in a desired position with the eye centered at an about 1 mm distance from an inner surface of each positive-meniscus lens.
  • the virtual reality system is compatible with two-photon functional microscopy.
  • the virtual reality system allows one to place it under an upright two-photon microscope, providing a full FOV, including the overhead visual region, while imaging.
  • a custom shielding system is provided to fit around the objective and connected to a ring on the head of the subject, in order to block light from the illuminated screens from being detected by the microscope’s photodetectors.
  • the virtual reality system is usable to establish the existence of large populations of place cells in the hippocampus during virtual navigation, global remapping during an environment change, and the first descriptions of the response of place cells ensembles to overhead looming stimulation.
  • the virtual reality system is usable for studying neural coding properties of visual behaviors that utilize the large overhead binocular region thought to play a critical role in many rodent behaviors.
  • the virtual reality system further comprises at least one separate, but compatible, optical path for a camera to monitor eye movements and pupil size.
  • the virtual reality system further comprises prism mirrors for reorienting the placement of the screens relative to the head of the subject, thereby affording more access to certain brain regions.
  • the virtual reality system further comprises a data acquisition module for electrical recordings of data and data processing.
  • the virtual reality system is compatible with other VR approaches that rotate the animal in conjunction with movements through the virtual space to activate the vestibular system.
  • the virtual reality system is usable for augmented visual reality paradigms in which the other senses, as well as self-motion cues, are preserved.
  • the virtual reality system is a miniature rodent stereo illumination VR (iMRSIV) system that is at least about 10 times smaller than existing VR systems.
  • iMRSIV miniature rodent stereo illumination VR
  • the virtual reality system further comprises an eye tracking module configured to determine position and orientation of the eye of the subject for aligning each eye to a proper location with respect to the virtual reality system and measuring the orientation of the eye, once aligned.
  • the eye tracking module comprises an infrared (IR) illumination, an IR mirror, and an IR camera positioned in an eye tracking path in relation to the eye of the subject.
  • IR infrared
  • the IR camera comprises CCD (charge-coupled device) and/or CMOS (complementary metal-oxide-semiconductor) sensors.
  • CCD charge-coupled device
  • CMOS complementary metal-oxide-semiconductor
  • the IR illumination comprises one or more IR LEDs emitting IR light with a wavelength of about 850 nm.
  • a dichroic film is provided to cover the display to act as the IR mirror in the eye tracking path while transmitting the VR visible scene.
  • the dichroic film is an IR reflective and visible passing film.
  • the eye tracking module further comprises a visible light filter positioned in the eye tracking path in the front of the camera to block the visible light from the camera and passing the IR light.
  • FIG. 1 shows the mouse visual system and a new concept for mouse virtual reality goggles.
  • Panel A Mouse visual field of view with monocular (green) and binocular (red) regions shown at resting eye gaze position from top-down and front perspectives.
  • Panel B Simulated mouse in a cue rich environment, including overhead owl (left), with simulated 140-degree field of view from the two eyes (right). Note the different perspective from each eye of the cheese and owl objects in the binocular overlap region (highlighted in red).
  • Panel C Simulation of mouse field of view in a computer monitor based VR system (left), with simulated 140-degree field of view from the two eyes (right), binocular overlap region highlighted in red, and representation of overhead microscope (black rectangle above mouse).
  • Panel D Simulation of the mouse field of view using the new concept presented here using goggles (left), with simulated 140-degree field of view from the two eyes (right), and binocular overlap region highlighted in red. Note the different perspective from each eye of the cheese and owl objects in this region, and also note that the full visual field of view is illuminated in each eye. Note that the overhead microscope from C is not visible to the mouse in this setup.
  • FIG. 2 shows iMRSIV goggle device design and validation.
  • Panel A Zemax simulated mouse eye retina at a distance of 200 mm from the checkerboard in panel B. Rays for 3 different object points are shown (red, green blue).
  • Panel B The 482*261 mm checkerboard used as the object in Zemax simulations.
  • Panel C Real world reproduction of simulated Zemax arrangement, from side and top views. 482*261 mm checkerboard shown on computer monitor 200 mm from an extracted mouse eye. Camera used to view the back of the retina.
  • Panel D Resulting image of the checkerboard object on the Zemax simulated eye retina, view from the back of the retina.
  • Panel E Image of the computer monitor checkerboard object on the retina of an extracted mouse eye, as viewed from the camera.
  • Panel F Our optical system to achieve a 180-degree FOV using a custom designed positive-meniscus lens and a small curved illumination display, shown with mouse eye at correct location.
  • Panel G Zemax simulation of rays from different screen points traveling through mouse eye to the retina; blue, center of optical axis; red and green, edges of 140-degree eye field of view imaged onto retina; pink and yellow, edges of 180-degree field of view not imaged onto retina, but illuminated on screen for additional FOV for eye saccades.
  • Panel H Same as panel I, but zoomed in on eye.
  • Panel I Illustration of the checkerboard arrangement from panels B-C, but in a virtual world using Unity3D. 180-degree FOV of this scene was generated using a single Unity3D camera and a custom fish-eye shader. 140-degree FOV highlighted in red. Schematic shows 140-degree FOV and full 180-degree FOV to accommodate 20-degree gazes.
  • Panel J Eye model (as in panel H) and simulated recreation of checkerboard using custom fish-eye shader (as in panel I) after 20- degree saccade (gaze rotation).
  • Panel K Real optical iMRSIV system composed of curved screen and custom lens, along with experimental setup shown underneath.
  • Panel L The 180- degree FOV from panel I was transferred to the small curved display in Zemax and used as the object, which was imaged onto the mouse eye retina through the positive-meniscus lens; the resulting image of the checkerboard object on the Zemax simulated eye retina is shown here (140-degree eye FOV), view from the back of the retina.
  • Panel M Checkerboard scene from J was used to illuminate the real OLED screen; the resulting image (through the real positivemeniscus lens) on the retina of an extracted mouse eye is shown, as viewed from a camera at the back of the retina.
  • Panels N-O Same as panels L-M, but with eye rotated 20-degrees with respect to screen center (as in ray diagram in panel J, left) to simulate a 20-degree saccade.
  • FIG. 3 shows iMRSIV behavior apparatus and device-eye alignment procedures.
  • Panel A Left, iMRSIV system connected to 3D micropositioners with metal bars, and incorporated into a head-fixed behavior apparatus with treadmill and reward delivery system. Right, photo of mouse in iMRSIV system.
  • Panel B Zoomed in view from A showing iMRSIV system and headplate positions with respect to mouse.
  • Panel C Schematic of electronics connections for control and reading from iMRSIV system, treadmill and reward delivery systems.
  • Panel D Left, 3D printed frame used during surgery to position the head-plate at the same location with respect to the eyes across different mice. Middle, view of frame on mouse and aligned to eyes; right, zoomed in view.
  • Panel E Left, 3D printed frame with pointed target used to position each half of the iMRSIV system with respect to each eye before each session. Middle, view of frame on mount and target aligned to mouse eye; right, back view. Panel F : Left, separated iMRSIV system components. Middle, iMRSIV system aligned to correct location with respect to mouse eyes (only one side is shown for clarity); right, back view.
  • FIG. 4 shows iMRSIV spatial behaviors: linear track and looming stimulation.
  • Panel A Linear track used for behavior, with tunnels (brown) and reward (blue) locations shown.
  • Panel B Trials/min over training days (sessions) for the conventional 5-panel VR group (left) and the iMRSIV group (right). Light grey lines show data for individual mice. Thick line and shading represent mean +/- SEM across mice. Dashed line reproduces mean for 5-panel group.
  • Panel C Top, prelicking index over training days for the 5-panel VR group (left) and the iMRSIV group (right). Bottom, mean licking rate vs. position (reward position, blue) over all mice in each group for days 1, 2 and 3 of training.
  • Panel D Linear track used for looming behavior, with tunnels (brown), reward (blue) and looming stimulation (black discs) locations shown.
  • Panel E Top, three examples of behavioral responses to the looming stimulus (dashed line) showing no change in running velocity for a 5-panel group mouse (left) and rapid freezing for one (middle) and fleeing followed by freezing in the other (right) iMRSIV group mice. Bottom, plots of mean velocity vs.
  • FIG. 5 shows two-photon calcium imaging during iMRSIV spatial behaviors.
  • Panel A iMRSIV+2P.
  • Left place field firing patterns in familiar track even trials, sorted based on familiar track peak odd trial locations, scatter plot of place field peak locations familiar even laps vs familiar odd laps, and histogram of place field peak locations; middle, place field firing patterns in novel track trials, sorted based on familiar track peak locations, scatter plot of place field peak locations familiar laps vs novel laps, and spatial correlations between place fields — familiar odd vs familiar even, familiar vs novel, and novel odd vs novel even; right; place field firing patterns in novel track even trials, sorted based on novel track peak odd trial locations, scatter plot of place field peak locations novel even laps vs novel odd laps, and histogram of place field peak locations.
  • Panel E Bayesian decoding of mouse location based on CAI neuron firing patterns. Top, example session showing actual mouse track location vs decoded position, where the encoding model was built with some pre-loom trials and decoding was applied to the remaining pre-loom trials (left) or applied to the post-loom trials (right). Bottom, decoding position error vs track position for pre- encoding/pre-decoding (left) and pre-encoding/post-decoding (right) — pre-pre reproduced in grey for comparison.
  • F Bayesian decoding of mouse location based on CAI neuron firing patterns. Top, example session showing actual mouse track location vs decoded position, where the encoding model was built with some pre-loom trials and decoding was applied to the remaining pre-loom trials (left) or applied to the post-loom trials (right). Bottom, decoding position error vs track position for pre- encoding/pre-decoding (left) and pre-encoding/post-decoding (right) — pre-pre reproduced in grey for comparison
  • decoded position probability vs time heat maps, top
  • FIG. 6 shows the mouse visual system and a new concept for mouse virtual reality goggles (20-degree saccade).
  • Panel A Mouse visual field of view with monocular (green) and binocular (red) regions shown at resting eye gaze position from top-down and angled perspectives.
  • Panel B Same as A, but with 20-degree forward saccade in both eyes; note expanded binocular zone.
  • Panels C-E Columns 1, 2, 3 reproduced from panels B-D of FIG. 1. Columns 4,5 same as 2,3, but with 20-degree forward saccade in both eyes.
  • FIG. 7 shows fisheye shader in Unity used to generate large FOV and compensate for spherical aberrations of the iMRSIV lens.
  • the iMRSIV lens that we used introduced a pincushion distortion (top row), as simulated using Zemax.
  • a fisheye distortion to the input image (bottom left); when that image is passed through the lens, as simulated in Zemax, the output image (represented with red in the overlay image) is now largely undistorted (bottom right) and highly similar to the original checkerboard input image (represented with cyan in the overlay image).
  • FIG. 8 shows quantification of similarity between Zemax and real mouse retinal projections for monitor and iMRSIV displays.
  • Panel A Resulting image of the checkerboard object on the Zemax simulated eye retina with monitor at a distance of 200 mm, view from the back of the retina (same as panel D of FIG. 2). Edges of the checkerboard were detected and overlaid in red (‘edge detection’).
  • Panel B Resulting image of the checkerboard object on the Zemax simulated eye retina with iMRSIV, view from the back of the retina (same as panel L of FIG. 2). Edges of the checkerboard were detected and overlaid in cyan (‘edge detection’).
  • Panel C Vertex points were selected from the checkerboard on the Zemax-simulated retina images (monitor from panel A, red dots; iMRSIV from panel B, blue dots) and superimposed on the Zemax-simulated retina image with the monitor (from panel A).
  • Panel D Deviation between vertices shown in panel C. The Cartesian distance between pairs of points is calculated and then normalized to the total diameter of the eye used in the model. These distances are then averaged over columns or rows of the checkerboard to attain deviation distance as a function of the x-axis or y-axis, respectively.
  • a 1% deviation corresponds to about 0.03 mm (eye diameter about 3 mm) or to about 1.4 degrees (eye diameter about 140 degrees), which is less than the mouse visual acuity of 0.375 cycles/degree (or 2.6 degrees/cycle).
  • Panel E Superposition of Zemax-simulated retina images or detected edges from monitor (panel A) and from iMRSIV (panel B). Scaling the iMRSIV image by 5% (right) corrects for the slight magnification difference between the two optical systems.
  • Panel F Image of the real world computer monitor checkerboard object on the retina of an extracted mouse eye (same as panel E of FIG. 2, but now shown across 4 separate eye experiments).
  • vertex points were selected from the checkerboard images on the extracted eyes. Detected edges from Zemax simulated image superimposed as well to aid comparison.
  • Panel G Vertex points (selected from real eye images in panel F) superimposed on the Zemax-simulated retina image with monitor.
  • Panel H Deviations calculated for each of the 4 eye experiments. Each dot represents data from one eye; line and shading represent mean +/- SEM across the 4 eyes.
  • Panels I-K Same as panels F-H but using iMRSIV (as in panels L-M of FIG. 2).
  • Panels L-Q Same as panels F-K but with 20-degree gaze deviation (as in panels J, N and O of FIG. 2).
  • FIG. 9 shows optical distortions incurred by misalignment of iMRSIV system.
  • Panels A- C We tested the distortions incurred by misalignment of the iMRSIV system relative to the eye using Zemax simulations.
  • the default configuration is 1 mm of distance from the inner curve of the iMRSIV lens to the lens of the eye with no displacement or rotation. In each case, we altered the alignment along one dimension and acquired the Zemax-simulated retina image of the checkerboard pattern.
  • a 1% deviation corresponds to about 0.03 mm (eye diameter about 3 mm) or to about 1.4 degrees (eye diameter about 140 degrees), which is less than the mouse visual acuity of 0.375 cycles/degree (or 2.6 degrees/cycle).
  • Panel A Lens-eye displacement (axial). The lens-eye distance was increased by +1 mm, +2 mm, or +3 mm from the default distance of 1 mm.
  • Panel B Lens displacement (lateral). The iMRSIV lens was displaced relative to the eye position by 1 mm, 2 mm, or 3 mm.
  • Panel C Lens rotation. The iMRSIV lens and display were together rotated relative to the axis of the eye and retina.
  • FIG. 10 shows curved vs non-curved side comparison. Due to mechanical limitations, we could only curve the screen along one axis (azimuthal). Here we used Zemax to simulate the images formed on the retina with and without curvature of the screen. Optically, distortions between the two axes were practically identical. However, along the curved axis, we achieved a slightly larger FOV. We chose the azimuthal axis because the mouse makes more frequent and larger saccades along this axis, but the curvature could easily be switched to the vertical direction if desired. Panel A: Resulting image of the checkerboard object on the Zemax simulated eye retina with iMRSIV (with and without curvature), view from the back of the retina.
  • iMRSIV with and without curvature
  • Edges of the checkerboard were detected and overlaid in red or cyan (‘edge detection’), respectively, and superimposed (‘overlay of detected edges’).
  • Panel B Vertex points (selected from ‘Curved’ and ‘Flat’ images in panel A) superimposed on the Zemax-simulated retina image with the curved iMRSIV (‘Curved’ from panel A).
  • Panel C Deviation between vertices shown in panel B. The Cartesian distance between pairs of points is calculated and then normalized to the total diameter of the eye used in the model. These distances are then averaged over columns or rows of the checkerboard to attain distance as a function of the x-axis or y-axis, respectively.
  • Panel D Same as panel A but with 20-degree gaze deviation and a full square checkerboard as the display pattern. Edges detected from checkerboard are shown superimposed on the retina images and also each other (‘overlay of detected edges’).
  • the ‘Curved’ screen provides a slightly larger field of view; this is visualized by the vertical straight red and cyan lines, which delineate the edge of the image formed for the ‘Curved’ and ‘Flat’ configurations, respectively.
  • FIG. 11 shows verification of freezing response to looming stimulus.
  • Panel A Along with the treadmill velocity, we also took video of the mouse during presentation of a looming stimulus. We quantified any movement by measuring the energy averaged over pixels within an ROI (sum of the square of the time derivative at each pixel). This measure provided a sensitive means of detecting any motion of the mouse, even if the treadmill was not moved. Shown here is a single frame from the movie and three ROIs tested: ‘Mouse’ (red), which selected the whole body; ‘Face’ (blue), which selected the head/neck; and “Paws” (green), which selected the forelimbs.
  • Panel B For the exemplar mouse (‘C4’) we plotted the treadmill velocity and the energy in each ROI over the course of the entire behavior session during which the loom was presented. Underneath each trace, we also plot a threshold indicator function that detects when the corresponding trace is different from zero. As can be appreciated from the plot, all measures are highly correlated. Importantly, in the time from the looming stimulus until the first movement, all channels show zero motion, verifying that the zero treadmill velocity reflects what is likely true freezing by the mouse (and not simply immobility). Panel C: For four individual experiments, we show the treadmill velocity and the energy in the ‘Mouse’ whole-body ROI for a 2-minute span around the time of the loom.
  • FIG. 12 shows cortical regions that are accessible with an overhead microscope and 10X objective.
  • Panel A 3D model of the 2p imaging configuration, showing the mouse skull and eyes, head plate, iMRSIV lens and displays, and the position of the 10X objective and the cone of light centered above the position of CAI.
  • Panel B Accessible (green) and inaccessible (red) regions of the dorsal surface of cortex using a standard overhead microscope and 10X objective with iMRSIV.
  • Panel C Overlap of accessible-inaccessible regions along with a mouse brain atlas.
  • FIG. 13 shows comparison of CAI place cells in iMRSIV system to traditional 5-panel virtual reality.
  • Panel A Lap-by-lap activity of three exemplar CAI neurons during navigation in iMRSIV. Mean traces are shown underneath. Reliability score, defined as the fraction of laps with significant firing within the respective place field of each neuron, is indicated. Histogram (right inset) shows the distribution of reliability scores for all place cells across 7 imaging sessions using iMRSIV.
  • Panel B Aggregate place cell data for all imaging sessions on the linear track (including familiar sessions and first part of track switch sessions when the track was familiar; the subset of these for only familiar sessions is shown in panel B of FIG. 5), for both traditional 5-monitor VR and iMRSIV. Mean transient rate vs.
  • Fraction place cells fraction of cells in a session that are place cells (see Methods).
  • Spatial field width length of track over which lap-averaged cell firing is greater than 30% of the max, applied to place cells only.
  • Mean spatial information spatial information score, applied to all cells. Reliability: fraction of laps with significant firing within the place field of that cell, applied to place cells only (see Methods).
  • Each point represents the mean for all cells from one imaging session; black cross represents mean ⁇ SEM across sessions.
  • Panel D Population response to looming stimulation. The mean transient rate for a given imaging session was triggered on the time of the onset of the looming stimulus.
  • FIG. 14 shows IMRSIV goggle design with eye tracking IR optical path (left) and CAD design (right).
  • the IMRSIV visible OLED display is covered with a dichroic film (IR reflective and visible passing) so that it acts as an IR mirror for the eye tracking path while transmitting the VR visible scene. Visible light is blocked from the IR camera by a visible light filter (IR passing).
  • IR passing visible light filter
  • FIG. 15 shows 3D model of the prototype with light path traced from eye pupil to camera shown in red.
  • FIG. 16 shows example of eye position and pupil size tracking using the (prototype) eye tracking version of IMRSIV (shown in FIG. 14 in a head-fixed mouse running on a treadmill. Only one side of the IMRSIV system was used for this demonstration.
  • first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, or section without departing from the invention's teachings.
  • relative terms such as “lower” or “bottom” and “upper” or “top,” may be used herein to describe one element’s relationship to another element as illustrated in the figures. It will be understood that relative terms are intended to encompass different orientations of the device in addition to the orientation depicted in the figures. For example, if the device in one of the figures, is turned over, elements described as being on the “lower” side of other elements would then be oriented on “upper” sides of the other elements. The exemplary term “lower”, can, therefore, encompasses both an orientation of “lower” and “upper,” depending on the particular orientation of the figure.
  • “around”, “about”, “approximately” or “substantially” shall generally mean within 20 percent, preferably within 10 percent, and more preferably within 5 percent of a given value or range. Numerical quantities given herein are approximate, meaning that the term “around”, “about”, “approximately” or “substantially” can be inferred if not expressly stated.
  • the phrase “at least one of A, B, and C” should be construed to mean a logical (A or B or C), using a non-exclusive logical OR.
  • the term “and/or” includes any and all combinations of one or more of the associated listed items.
  • VR Visual virtual reality
  • head-fixed mice offer the ability to investigate spatial navigation behaviors and perform experiments that are difficult in real world studies.
  • VR also offers the ability to dissect the neural circuitry underlying spatial behaviors using recording techniques that require large platforms and a high degree of mechanical stability.
  • current VR approaches using curved projection screens or multiple monitors are physically large and have limitations that reduce immersion in the virtual environment. For example, these systems do not fully illuminate the visual field of view of mice, do not stereoscopically illuminate the binocular zone, and leave visible many elements of the lab -frame.
  • this invention discloses a virtual reality goggle for mice, i.e., an iMRSIV (Miniature Rodent Stereo Illumination VR) system, which is about 10x smaller than existing VR systems.
  • the iMRSIV system separately illuminates each eye for stereo illumination of the binocular zone and illuminates each eye with an about 180- degree field of view, thus excluding the lab frame from view while also accommodating saccades.
  • the virtual reality system comprises a pair of concave lenses; and a pair of curved screens, arranged in relation to the pair of concave lenses and eyes of a subject, for fully illuminating the visual field of view (FOV) of the subject.
  • the subject can be an animal or a human being.
  • the virtual reality system is configured to image objects displayed on the curved screen onto the eye retinas of the subject through the concave lens.
  • the virtual reality system is configured to illuminate each eye with an about 180-degree field of view in all directions.
  • the virtual reality system is configured to separately illuminate each eye for stereo illumination of the binocular zone, thereby excluding a lab frame from view while also accommodating saccades.
  • the about 180-degree field of view includes about 140 degrees for each eye FOV and +/- 20 degrees for saccades.
  • each concave lens is a positive-meniscus lens having an inner surface operably facing an eye of the subject.
  • each concave lens is arranged such that each eye is centered at a predetermined distance from an inner surface of each positive-meniscus lens.
  • each curved screen is a curved illumination display.
  • each curved screen is a flexible light-emitting diode (LED) display.
  • LED light-emitting diode
  • the virtual reality system provides a mean resolution of about 2.2 pixel s/degree, or better, across the about 180-degree FOV.
  • each curved screen is a high resolution organic light-emitting diode (OLED) screen configured to increase the pixels/degree to further exceed the acuity of mice or by incorporating other sensory modalities including olfactory auditory and tactile into the virtual simulation, so as to further increase the immersiveness of the VR experience for mice.
  • OLED organic light-emitting diode
  • the virtual reality system further comprises a pair of screen holders, each screen holder with a curvature matching that of the screen, to which the curved screen is affixed; and a pair of lens holders, each lens holder with the lens attached to one end and the other end mated to the screen holder such that the lens is centered at the desired distance from the curved screen.
  • each lens holder is mated to the screen holder with magnets.
  • an optical axis is the virtual reality system is aligned with the optical axis of the mouse eye at its resting position.
  • the lens holder when aligned, is in a desired position with the eye centered at an about 1 mm distance from an inner surface of each positive-meniscus lens.
  • the virtual reality system is compatible with two-photon functional microscopy.
  • the virtual reality system allows one to place it under an upright two-photon microscope, providing a full FOV, including the overhead visual region, while imaging.
  • a custom shielding system is provided to fit around the objective and connected to a ring on the head of the subject, in order to block light from the illuminated screens from being detected by the microscope’s photodetectors.
  • the virtual reality system is usable to establish the existence of large populations of place cells in the hippocampus during virtual navigation, global remapping during an environment change, and the first descriptions of the response of place cells ensembles to overhead looming stimulation.
  • the virtual reality system is usable for studying neural coding properties of visual behaviors that utilize the large overhead binocular region thought to play a critical role in many rodent behaviors.
  • the virtual reality system further comprises at least one separate, but compatible, optical path for a camera to monitor eye movements and pupil size.
  • the virtual reality system further comprises prism mirrors for reorienting the placement of the curved screens relative to the head of the subject, thereby affording more access to certain brain regions.
  • the virtual reality system further comprises a data acquisition module for electrical recordings of data and data processing.
  • the virtual reality system is compatible with other VR approaches that rotate the animal in conjunction with movements through the virtual space to activate the vestibular system.
  • the virtual reality system is wearable by a freely moving subject.
  • the virtual reality system is usable for augmented visual reality paradigms in which the other senses, as well as self-motion cues, are preserved.
  • the virtual reality system is a miniature rodent stereo illumination VR (iMRSIV) system that is at least about 10 times smaller than existing VR systems.
  • iMRSIV miniature rodent stereo illumination VR
  • the virtual reality system further comprises an eye tracking module configured to determine position and orientation of the eye of the subject for aligning each eye to a proper location with respect to the virtual reality system and measuring the orientation of the eye, once aligned.
  • the eye tracking module comprises an infrared (IR) illumination, an IR mirror, and an IR camera positioned in an eye tracking path in relation to the eye of the subject.
  • IR infrared
  • the IR camera comprises CCD (charge-coupled device) and/or CMOS (complementary metal-oxide-semiconductor) sensors.
  • CCD charge-coupled device
  • CMOS complementary metal-oxide-semiconductor
  • the IR illumination comprises one or more IR LEDs emitting IR light with a wavelength of about 850 nm.
  • a dichroic film is provided to cover the display to act as the IR mirror in the eye tracking path while transmitting the VR visible scene.
  • the dichroic film is an IR reflective and visible passing film.
  • the eye tracking module further comprises a visible light filter positioned in the eye tracking path in the front of the camera to block the visible light from the camera and passing the IR light.
  • the novel VR system can operably illuminate the full field of view of mouse, with custom lens design, making the system fully immersive (no external frame visible).
  • the novel VR system can also provide additional field of view for saccades (eye movements).
  • the novel VR system can provide stereo illumination of the visual system of mouse, thereby providing depth information, and solutions for alignment to eyes.
  • the invention may have widespread applications in rodent neuroscience including, but is not limited to, memory research, fear research, visual processing, sensory integration; rodent behavior training, and immersive human VR goggles. These and other aspects of the invention are further described below. Without intent to limit the scope of the invention, exemplary instruments, apparatus, methods, and their related results according to the embodiments of the invention are given below. Note that titles or subtitles may be used in the examples for convenience of a reader, which in no way should limit the scope of the invention. Moreover, certain theories are proposed and disclosed herein; however, in no way they, whether they are right or wrong, should limit the scope of the invention so long as the invention is practiced according to the invention without regard for any particular theory or scheme of action.
  • mice virtual reality goggle system panel D of FIG. 1 where the lab frame is not visible and each mouse eye is separately illuminated, providing a full field of view (with additional field of view for saccades, FIG. 6) and stereo illumination to the binocular visual zone, including the critical overhead region.
  • This 180-degree FOV was displayed on the small curved display in Zemax and used as the object so we could examine the resulting image (through the positive-meniscus lens) on the simulated mouse eye retina, which was centered at a 1mm distance from the inner surface of the lens.
  • the resulting simulated retina image of the checkerboard pattern panel L of FIG. 2; 140 of the available 180-degree FOV
  • panel D of FIG. 2 the Zemax model of the checkerboard object
  • panels A-E of FIG. 8 panels
  • the small OLED screen, custom positive-meniscus lens and 3D printed parts make up one half (one eye) of our mouse virtual reality goggle system; a duplicate assembly was therefore made for the other eye.
  • Unity3D cameras were then used to generate the about 180-degree FOV for each display for each eye.
  • mice were head fixed using the mounting posts and then a 3D printed frame was used to position each half of the iMRSIV system with respect to each eye (panel E of FIG. 3).
  • the conical lens holder was removed from each half (pulled off from magnetic attachment) and replaced with a frame with an eye target, which was aligned to each eye using the micropositioners. Once aligned, the target was again replaced with the conical lens holder, which was now in the correct position with the eye centered at a 1 mm distance from the lens (panel F of FIG. 3).
  • iMRSIV spatial behaviors linear track and looming stimulation
  • mice navigate a linear track to a fixed reward location; trained mice in such tasks develop behaviors indicative of anticipation of the reward location.
  • mice in the iMRSIV group ran more than 0.5 trials per minute over the first about 45 minute sessions (iMRSIV group mean of 0.79 +/- 1.40 trial/minute in first session), and on average mice in this group reached expert levels (3.21 +/- 1.80 trials/minute) after about 6 days of training (panel B of FIG. 4).
  • mice in this group reached expert levels at a similar number of days of training as the iMRSIV group (5-panel group mean of 0.13 +/- 0.16 trials/minute on day 1 and 2.75 +/- 1.88 trials/minute on day 6; no significant difference in trials/minute between groups, p > 0.05, 2- sample t-test).
  • mice in both groups displayed similar prelicking indices, with most non-consumptive licking occurring just before the reward location. Therefore, mice engaged in a virtual navigation behavior more quickly using the iMRSIV system compared to the existing monitor based VR system, and refined their licking behavior to become highly precise and location specific after several days of training.
  • mice were trained on the first linear track (panel A of FIG. 4) for at least 6 days (about 2-3 rewards/minute), they were switched to a new linear track with the same tunnels at the ends, but an open field in the middle with few cues (panel D of FIG. 4).
  • this track became familiar (2-3 sessions)
  • we introduced a single, sudden overhead looming stimulus overhead increasing size sphere, with shadow over the mouse
  • the mice were in the center of the open field (panel D of FIG. 4).
  • mice Four out of the seven iMRSIV behavior group mice were injected with a virus to induce expression of jGCaMP8m in CAI of the dorsal hippocampus and were implanted with a chronic hippocampal imaging window (panel A of FIG. 5); these four mice were used for the subsequent imaging experiments.
  • mice performed the linear track task (8 imaging sessions from 4 mice, 4 familiar track sessions, 4 environment switch sessions; 450x450 pm field size, 30.28 frames/sec, 28.7 minutes/imaging session, 297 +/- 95 neurons segmented/field).
  • a familiar linear track we identified a large number of place cells (204 +/- 61 place cells per field, 69.4 +/- 8.8% of active cells had significant place fields), and these place cells were highly reliable, with most cells active on the majority of trials (mean reliability: 0.51 +/- 0.14; panel A of FIG. 13).
  • iMRSIV virtual reality goggles for mice in a system we refer to as iMRSIV.
  • Our system separately illuminates each eye to achieve stereo illumination of the binocular zone and illuminates each eye with a about 180-degree field of view (140-degree eye view +/- 20-degrees for saccades in every direction), thus excluding the lab frame from view.
  • This FOV is larger than the FOV achieved using conventional rodent VR systems and further provides stereo illumination of the binocular region in a way not currently possible with existing VR systems.
  • mice engaged (performed anticipatory licking) more quickly in a virtual linear track task in the iMRSIV system compared to a conventional monitor based mouse VR system and were able to achieve expert level performance within several days of training, similar to conventional VR. It is unclear exactly why mice were able to engage in the task in the iMRSIV system more quickly, but we hypothesize that it is because their full FOV was illuminated and the conflicting lab frame was not visible. This advantage, combined with the potential depth information provided by the stereoscopic illumination of the binocular region, may combine to provide a more immersive experience, facilitating increased task engagement and spatial awareness.
  • a potential future advantage of our iMRSIV system is the significant size reduction compared to existing rodent VR systems (about lOx smaller). This could allow for the iMRSIV system to be more easily combined with microscope or other recording systems that do not have sufficient space, or are of an unusual geometry, and thus are not compatible with larger current VR systems. Further, miniaturizing virtual reality systems, as we have done here, is likely to facilitate the building and use of large scale training arrays where dozens of mice can be trained in parallel.
  • Another limitation of our iMRSIV system is optical access to certain brain areas for imaging. Due to steric hindrances between the display screens and the microscope objective, not all brain regions can be readily accessed, especially those that are far rostral and lateral (FIG. 12). Further, some microscope objectives, especially those with high NA and low working distance, will have more severe steric hindrances. In most cases, these issues may be physically impossible to avoid due to the large view angles of the mouse eye and its proximity to rostral and lateral brain regions. Indeed, most microscope objectives are probably within the mouse’s visual fields when imaging rostral and lateral brain regions in a conventional setup. Certain approaches could help circumvent these issues, however.
  • a long working distance objective (as used here) or a thin GRIN lens can be used to provide additional clearance.
  • removing the outer casing on the objective (as done here) can help add additional clearance.
  • Tilting the head of the mouse could also be used to add some clearance for the system.
  • Yet another approach, if the experiment allows for it, is to use electrical recordings instead of imaging.
  • neuropixels are compatible with iMRSIV, though the size of the headstage (about 6x7x2mm) will also need to be taken into account to work around the steric constraints.
  • the iMRSIV system itself it may be possible to reduce the physical size in future versions, which would make it easier to access rostral and lateral brain region. This could be accomplished using a different lens design combined with a smaller screen.
  • other optical designs, such as those that incorporate a prism mirror could reorient the placement of the display screens relative to the mouse’s head, affording more access to certain brain regions.
  • mice All animal procedures were approved by the Northwestern University Institutional Animal Care and Use Committee. All mice were housed in a vivarium with a reversed light/dark cycle (12 hours light during the night) and all experiments were performed during the day (during their dark cycle). For behavior and CAI imaging experiments, about 12 week old adult C57BL/6J male mice (The Jackson Laboratory, strain #000664) were used. For extracted eye experiments, 10-14 week old adult BALB/c mice (Charles River) of both sexes were used.
  • mice were aligned and attached to adult C57BL/6J male mice as detailed below.
  • mice 4 iMRSIV and 5 control mice
  • CAI cannulation and virus injection was also performed to allow for imaging.
  • mice (1-2% isoflurane in 0.5 L/min 02) were head-fixed to a stereotaxic apparatus (Model 1900, David Kopf Instruments). The skull was leveled and aligned to bregma. We then positioned the eyes relative to the headplate holder by using a custom 3D-printed alignment tool (FIG. 3D). This tool has two prongs that approximate the position of the center of each eyeball. Once centered, the tool was replaced with a custom titanium headplate (1 mm thick, eMachineShop). This headplate is the same size and shape as the alignment tool but without the centering prongs. Further details on our alignment procedures are provided below under “iMRSIV alignment procedure”. Dental cement (Metabond, Parkell) was used to adhere the headplate to the skull. Mice were monitored closely for 24 hours and given 3-5 days to recover before water restriction and behavioral training were begun.
  • mice used for CAI imaging before attaching the headplate we performed a small craniotomy (0.5 mm) and, using a beveled glass micropipette, injected about 60 nL of AAV1- syn-jGCaMP8m-WPREl (Addgene catalog #162375, diluted about 8x from 2.5el3 GC/ml stock into phosphate buffered solution) into right CAI (2.3 mm caudal, 1.8 mm lateral, 1.3 mm beneath dura). Then a stainless steel cannula with an attached 2.5 mm No. 1 coverslip (Potomac Photonics) was implanted over CAI.2
  • mice To measure the image formed on the mouse retina, we used explanted eyes from BALB/c mice. We chose to use albino mice because the retinal epithelium is not pigmented and thus images formed on the retina using the visible spectrum can be observed by photographing the back of the explanted eye. We chose this particular strain (BALB/c) because the size of the eye and the optical parameters are highly similar to the strain of mice used for our behavior and imaging experiments (C57BL/6J).3
  • mice were deeply anesthetized with isoflurane (2% in 0.5 L/min 02). The eye was then removed, the optic nerve transected, and any connective tissue cleared. The eye was then placed on a custom 3D printed mount that centered the eye relative to the rest of the setup.
  • a camera Basler acA5472 with a 25 mm f/1.4 lens, HR978NCNH1198
  • the display either large flat-panel display or the iMRSIV lens and miniature display
  • Using rotation and translation stages we could control the distance from the displays to the eye and we could adjust the rotation of the eye relative to the display.
  • the mouse eye has a large angle field of view that spans 140 degrees plus another +/- 20 degrees for saccades4-6. Not only does this large angle occupy a large space, it also requires solutions that account for the Petzval field curvature.
  • the binocular region requires a solution that can deliver different perspectives of the same object to each eye (thus transmitting binocular disparity information). Because these views physically overlap, either the two eyes need to receive different images from the same position (such as is accomplished when viewing 3D televisions through polarized lenses) or the optical field needs to be separated so that the physical space illuminating the medial portions of each eye are different.
  • the display itself (6.3 mm from the front surface of the lens) needed to have some curvature as well to reduce distortions introduced when the display-to-lens distance varies across different angles.
  • Virtual reality environments were rendered in Unity3D. The same computer was also used to synchronize behavior and two-photon imaging data during execution of VR simulations.
  • This fisheye projection corresponds to about 180 degrees FOV, projected onto one circular display.
  • Our custom lens also provides a strong anti-fisheye effect (see https://www.mathworks.com/help/vision/ug/camera- calibration.html); we compared the fish-eye vs. anti-fisheye effect, and we found that they are approximately the opposite effect (inverse transforms), so there was no need to further correct the lens distortion (FIG. 7).
  • Each lens was then paired with a small, flexible, round OLED screen (1.39 in diameter, 400x400 pixel, Innolux).
  • Refresh rate for both systems was 60 Hz, which were driven by a video card (Nvidia RT3070).
  • Monitor brightness per unit area was higher for the round OLED screens of iMRSIV than for the large monitors we used for the traditional 5-panel display. This brightness was measured by collecting light over a 5-mm diameter region of the display using a fiber optic cable pressed against the screen and light collected on the other side using a photodetector (DET-100A, Thorlabs). For a given uniform display (either 50% gray or 100% white), the voltage measured from the photodetector was about 10 fold higher for the OLED screens.
  • Custom scripts were written in C# to enable communication with a data acquisition card (PCIe-6323, National Instruments) from within the Unity runtime environment.
  • PCIe-6323 National Instruments
  • the data acquisition card (DAQ) was used to output timed digital output to control the opening of a water reward solenoid.
  • the timing was calibrated to provide a volume of 3 pL of water.
  • Inputs to the DAQ included a quadrature encoder and digital signals.
  • the quadrature encoder was used to read running velocity from an optical encoder (E2-5000, US Digital) attached to the axis of the treadmill.
  • the iMSRIV displays To position the iMSRIV displays relative to each mouse eye, we developed the following alignment procedure that minimized mouse-to-mouse variability while also permitting adjustments to be made for each mouse.
  • the grooves were positioned exactly 30 mm apart, thus allowing precise and reliable mounting using off-the-shelf parts (such as the Thorlabs 30 mm cage system).
  • off-the-shelf parts such as the Thorlabs 30 mm cage system.
  • the headbar is aligned to the eyes of each mouse.
  • This alignment is accomplished by first using a custom 3D-printed alignment tool (panel D of FIG. 3).
  • This tool has two prongs situated for positioning to the center of each eyeball. Once the tool is aligned (prongs centered on each eye), the stereotax micromanipulator is fixed while the tool is replaced with a headplate and cemented in place. Thus, the relative position of the headplate mount to the eyes of the individual mouse is fixed (within experimental measurement error).
  • the headplate is attached to the headbars.
  • the goal was to position the display assembly (consisting of the lens holder attached to the display holder) at the desired position relative to the mouse eye lens, (panels F-G of FIG. 2, FIG. 3).
  • the lens holder and display holder are attached using a set of 3 magnets, allowing us to attach and detach the lens holder in a reproducible manner.
  • the assembly is attached to a 3-axis stage (3x MS IS, Thorlabs), allowing precise control of x-y-z position, along with a rotation stage (RP005, Thorlabs).
  • an alignment tool (panel E of FIG. 3) was attached in place of the lens holder.
  • This tool is similar to the lens holder but instead of the lens has a probe at the desired location of the center of the front of the mouse eye lens.
  • Any final fine adjustments are then performed using micromanipulators for each iMRSIV display. In practice, however, we found that little to no adjustments were needed between mice.
  • mice were restricted to receiving 0.8-1.0 mL of water each day. Mice were weighed daily and training was begun once weights fell to about 80% of baseline.
  • iMRSIV alignment procedure For iMRSIV mice, once the mouse was head fixed, an alignment procedure was performed as detailed above (“iMRSIV alignment procedure”) and in FIG. 3. Note that it was not possible to perform truly blinded experiments when comparing iMRSIV mice to the 5-monitor control mice. We however matched training conditions in every aspect that we could by using mice of the same age, water restricting for the same duration with the same target weight, matching the duration of training sessions, etc. We also practiced the iMRSIV alignment procedures beforehand so as to minimize the time and potential discomfort incurred while positioning the screens around the mouse. Once proficient, we were able to perform this alignment within a couple minutes.
  • mice In the subset of cannulated mice, we performed two-photon imaging of populations of neurons in CAI of the hippocampus during behavior sessions as described above, either with iMRSIV (4 mice) or with the traditional 5-panel display (5 mice). Imaging was performed using a customized upright microscope. A mode-locked Ti: Sapphire laser (Chameleon Ultra II, Coherent) tuned to 920 nm was raster scanned using a resonant scanning module (Sutter Instruments). Emission light was filtered (FF01-510/84, Semrock) before being collected by a GaAsP PMT (H10770PA-40, Hamamatsu Photonics). Scanimage software (Vidrio) was used to control the microscope and acquire images. A TTL frame sync signal was output to the DAQ of the VR computer to allow for synchronization of two-photon imaging times to the behavior data acquired by Unity. All imaging was performed at 512x512 pixels and 30 Hz using bidirectional scanning.
  • a 10X objective (UPLFLN, Olympus), with outer housing removed to fit within the geometric constraints, was used for imaging. We removed the outer housing of the objective (unscrewing it) to increase clearance between the objective and the iMRSIV lens holder.
  • FIG. 12 we delineate regions of the cortex that are accessible using this objective without physically colliding with the iMRSIV lens mounts. As the placement of our headplate (and the iMRSIV system) is relative to the eyes of the animal, the exact relative location of bregma can vary across mice (and correspondingly the position of brain structures relative to bregma will vary as well).
  • deconvolution is performed to infer firing events.14
  • firing events after smoothing with a 170-ms Gaussian filter.
  • the “transient rate” refers to the amplitude and frequency of these detected events in a given time window or spatial bin.
  • Prelicking index This measure quantified whether mice were licking near to the reward during reward approach, indicative of learning of the reward location and anticipation of the reward.
  • the prelicking ratio was then calculated as lickl/(lickl+lick2).
  • the minimum possible value of 0 indicates no licks in the pre-reward zone while the maximum possible value of 1 indicates all the licks were in the pre-reward zone.
  • Loom freezing period For mice that did freeze, we measured the time when mice resumed running. Such running was found by looking for the first moment the running velocity reached half of the maximum running velocity, which was calculated for each mouse as the 98th percentile of running velocity over the entire session. We ignored the first 10 seconds immediately after the loom since some mice initially and transiently increased their running velocity (fleeing) before freezing. We also measured the freezing time until first detected movement since it was possible the mouse resumed movement but without running. To ensure that the treadmill velocity faithfully reported any movements (and not just running), we recorded video of the mouse’s body during the loom sessions. We quantified the energy in a region-of- interest around the body of the mouse (mean across pixels of the square of the time derivative of individual pixels in the region) and found a high correspondence to the treadmill velocity (FIG. H).
  • Criteria for place cells For each neuron, spatial information was calculated using binned position (5 pm bins, periods of immobility and reward consumption excluded).15 The calculation was repeated using shuffled data. Neurons with spatial information of at least 0.75 bits/event and that was also larger than 98% or more of shuffles were categorized as place cells.
  • Reliability score For each place cells, we calculated the fraction of laps at which significant firing occurred within the dominant place field of that neuron2.
  • Cross-validation procedure Spatial firing maps and other within-environment calculations used cross-validated data. In these cases, data was separated by even and odd laps.
  • Bayesian decoding For a given imaging session, population neural activity was used to decode the position of the track. This procedure was performed in two ways. First, for assessing the ability of pre-loom activity to decode post-loom position, we trained the Bayesian decoderl6 using the pre-loom data after binning the data using position along the track. This information was then used to decode the post-loom data, again after already binning for position along the track. For comparison, we also decoded pre-loom position using pre-loom data by splitting the data into odd laps (training set) and even laps (test set). Second, we assessed the decoded position during the freezing period in response to the loom stimulus. To perform this calculation, we trained the Bayesian decoder using all the pre-loom data. This decoder was then applied to the neural activity during the time that the mouse froze in response to the loom stimulus.
  • the iMRSiV goggle system fully covers the eyes of mice, making it difficult to determine the position and orientation of the eyes.
  • the alignment procedure to align each eye to the proper location with respect to the goggles was time-consuming as it had to be repeated with each eye for each mouse.
  • This exemplary embodiment of the virtual reality goggle has achieved both requirements by engineering eye tracking capability into our goggle system for rapid alignment and to monitor eye positions.
  • the IMRSIV uses a miniature CMOS camera (120 Hz frame rate, 800x600 pixels, Raspberry Pi Camera Module 3) and 850 nm IR illumination. Pupil center and diameter was tracked using custom MATLAB software based on a published algorithm (https://doi.org/10.1126/science.aav7893) (FIG. 16). The visible OLED display was illuminated during the acquisition of the eye tracking data in FIG. 16, and it did not interfere with the IR eye tracking path. The current capabilities suffice to extract eye position and pupil diameter and can be used for goggle alignment.
  • FIG. 16 The visible OLED display was illuminated during the acquisition of the eye tracking data in FIG. 16, and it did not interfere with the IR eye tracking path. The current capabilities suffice to extract eye position and pupil diameter and can be used for goggle alignment.
  • FIG. 14 shows an IMRSIV goggle design with eye tracking IR optical path (left) and CAD design (right).
  • the IMRSIV visible OLED display is covered with a dichroic fdm (IR reflective and visible passing) so that it acts as an IR mirror for the eye tracking path while transmitting the VR visible scene. Visible light is blocked from the IR camera by a visible light filter (IR passing).
  • FIG. 15 shows 3D model of the prototype with light path traced from eye pupil to camera shown in red.
  • FIG. 16 shows example of eye position and pupil size tracking using the (prototype) eye tracking version of IMRSIV (shown in FIG. 14 in a head-fixed mouse running on a treadmill. Only one side of the IMRSIV system was used for this demonstration, which shows the ability to perform eye tracking with this system.
  • Hippocampal CA3 NMDA receptors are crucial for memory acquisition of onetime experience. Neuron 38, 305-315. 10.1016/s0896-6273(03)00165-x.
  • Neuropixels probes enable high-yield recordings in freely moving mice. Elife 8. 10.7554/eLife.47188. [46], Zong, W., Obenhaus, H.A., Skytoen, E.R., Eneqvist, EL, de Jong, N.L., Vale, R., Jorge,

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne un système de réalité virtuelle (RV), comprenant une paire de lentilles concaves; et une paire d'écrans, agencés par rapport à la paire de lentilles concaves et aux yeux d'un sujet, pour éclairer complètement le champ de vision visuel (FOV, « field of view ») du sujet et conçus pour éclairer chaque œil avec un champ de vision à environ 180 degrés dans toutes les directions.
PCT/US2024/057391 2023-11-28 2024-11-26 Lunettes de réalité virtuelle à champ de vision complet Pending WO2025117501A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363603208P 2023-11-28 2023-11-28
US63/603,208 2023-11-28

Publications (1)

Publication Number Publication Date
WO2025117501A1 true WO2025117501A1 (fr) 2025-06-05

Family

ID=95898153

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/057391 Pending WO2025117501A1 (fr) 2023-11-28 2024-11-26 Lunettes de réalité virtuelle à champ de vision complet

Country Status (1)

Country Link
WO (1) WO2025117501A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160341966A1 (en) * 2015-05-19 2016-11-24 Samsung Electronics Co., Ltd. Packaging box as inbuilt virtual reality display
KR102032980B1 (ko) * 2017-12-19 2019-10-16 주식회사 브이알이지이노베이션 접이식 가상현실 뷰어
US20210208357A1 (en) * 2019-08-28 2021-07-08 Lg Electronics Inc. Electronic device
US20220014224A1 (en) * 2018-11-13 2022-01-13 Vr Coaster Gmh & Co. Kg Underwater vr headset
US20230004008A1 (en) * 2015-03-16 2023-01-05 Magic Leap, Inc. Augmented and virtual reality display systems and methods for diagnosing health conditions based on visual fields

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230004008A1 (en) * 2015-03-16 2023-01-05 Magic Leap, Inc. Augmented and virtual reality display systems and methods for diagnosing health conditions based on visual fields
US20160341966A1 (en) * 2015-05-19 2016-11-24 Samsung Electronics Co., Ltd. Packaging box as inbuilt virtual reality display
KR102032980B1 (ko) * 2017-12-19 2019-10-16 주식회사 브이알이지이노베이션 접이식 가상현실 뷰어
US20220014224A1 (en) * 2018-11-13 2022-01-13 Vr Coaster Gmh & Co. Kg Underwater vr headset
US20210208357A1 (en) * 2019-08-28 2021-07-08 Lg Electronics Inc. Electronic device

Similar Documents

Publication Publication Date Title
US20090295683A1 (en) Head mounted display with variable focal length lens
Yang et al. Visual search: psychophysical models and practical applications
EP2805671B1 (fr) Système de vidéographie oculaire
US10368743B2 (en) System and method for visualization of ocular anatomy
US6830336B2 (en) Automated generation of fundus images based on processing of acquired images
US20140226128A1 (en) Methods and Apparatus for Retinal Imaging
Wu et al. High-resolution eye-tracking via digital imaging of Purkinje reflections
Pinke et al. Full field-of-view virtual reality goggles for mice
US12239378B2 (en) Systems, methods, and apparatuses for eye imaging, screening, monitoring, and diagnosis
Zoccolan et al. A self-calibrating, camera-based eye tracker for the recording of rodent eye movements
JP2023026672A (ja) 視覚検査装置及び視覚検査方法
Dagnelie et al. Toward an artificial eye
van Rheede et al. Simulating prosthetic vision: Optimizing the information content of a limited visual display
CN105431076A (zh) 视线引导装置
TWI819654B (zh) 改善視網膜損傷患者視力的系統
Fortenbaugh et al. Losing sight of the bigger picture: Peripheral field loss compresses representations of space
US20240049963A1 (en) Cover-Uncover Test in a VR/AR Headset
CN109303544B (zh) 一种多尺度混合视力障碍分析仪及其分析方法
CN113080836A (zh) 非中心注视的视觉检测与视觉训练设备
Isaacson et al. MouseGoggles: an immersive virtual reality headset for mouse neuroscience and behavior
Johnson Development of the visual system
US20060209257A1 (en) Integral viewing and eye imaging system for visual projection systems
WO2025117501A1 (fr) Lunettes de réalité virtuelle à champ de vision complet
WO2020240867A1 (fr) Procédé de traitement d'image, dispositif de traitement d'image, et programme de traitement d'image
Rizzo et al. The effect of bilateral visual cortex lesions on the development of eye movements and perception

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24898615

Country of ref document: EP

Kind code of ref document: A1