US12372958B2 - Spatial teleoperation of legged vehicles - Google Patents

Spatial teleoperation of legged vehicles

Info

Publication number
US12372958B2
US12372958B2 US17/417,206 US201917417206A US12372958B2 US 12372958 B2 US12372958 B2 US 12372958B2 US 201917417206 A US201917417206 A US 201917417206A US 12372958 B2 US12372958 B2 US 12372958B2
Authority
US
United States
Prior art keywords
vehicle
target
velocity
spatial controller
reference frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/417,206
Other languages
English (en)
Other versions
US20220075364A1 (en
Inventor
Matthew D. Summer
William S. Bowman
Andrew D. Falendysz
Kevin M. Makovy
Daniel R. Hedman
Bradley D. Truesdell
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tomahawk Robotics Inc
Original Assignee
Tomahawk Robotics Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tomahawk Robotics Inc filed Critical Tomahawk Robotics Inc
Priority to US17/417,206 priority Critical patent/US12372958B2/en
Assigned to TOMAHAWK ROBOTICS reassignment TOMAHAWK ROBOTICS ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BOWMAN, WILLIAM S., FALENDYSZ, Andrew D., HEDMAN, Daniel R., MAKOVY, Kevin M., SUMMER, MATTHEW D., TRUESDELL, Bradley D.
Publication of US20220075364A1 publication Critical patent/US20220075364A1/en
Assigned to BANK OF AMERICA, N.A., AS THE ADMINISTRATIVE AGENT reassignment BANK OF AMERICA, N.A., AS THE ADMINISTRATIVE AGENT NOTICE OF GRANT OF SECURITY INTEREST IN PATENTS Assignors: TOMAHAWK ROBOTICS, INC.
Assigned to TOMAHAWK ROBOTICS, INC. reassignment TOMAHAWK ROBOTICS, INC. CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED ON REEL 058137 FRAME 0561. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: MAKOVY, Kevin M., BOWMAN, WILLIAM S., FALENDYSZ, Andrew D., SUMMER, MATTHEW D., HEDMAN, Daniel R., TRUESDELL, Bradley D.
Application granted granted Critical
Publication of US12372958B2 publication Critical patent/US12372958B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62D—MOTOR VEHICLES; TRAILERS
    • B62D57/00—Vehicles characterised by having other propulsion or other ground- engaging means than wheels or endless track, alone or in addition to wheels or endless track
    • B62D57/02—Vehicles characterised by having other propulsion or other ground- engaging means than wheels or endless track, alone or in addition to wheels or endless track with ground-engaging propulsion means, e.g. walking members
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F41—WEAPONS
    • F41H—ARMOUR; ARMOURED TURRETS; ARMOURED OR ARMED VEHICLES; MEANS OF ATTACK OR DEFENCE, e.g. CAMOUFLAGE, IN GENERAL
    • F41H11/00—Defence installations; Defence devices
    • F41H11/12—Means for clearing land minefields; Systems specially adapted for detection of landmines
    • F41H11/16—Self-propelled mine-clearing vehicles; Mine-clearing devices attachable to vehicles
    • F41H11/28—Self-propelled mine-clearing vehicles; Mine-clearing devices attachable to vehicles using brushing or sweeping means or dozers to push mines lying on a surface aside; using means for removing mines intact from a surface
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F41—WEAPONS
    • F41H—ARMOUR; ARMOURED TURRETS; ARMOURED OR ARMED VEHICLES; MEANS OF ATTACK OR DEFENCE, e.g. CAMOUFLAGE, IN GENERAL
    • F41H7/00—Armoured or armed vehicles
    • F41H7/005—Unmanned ground vehicles, i.e. robotic, remote controlled or autonomous, mobile platforms carrying equipment for performing a military or police role, e.g. weapon systems or reconnaissance sensors
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0016—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement characterised by the operator's input device
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0033—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0223—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving speed control of the vehicle
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G05D1/223—Command input arrangements on the remote controller, e.g. joysticks or touch screens
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G05D1/2235—Remote-control arrangements operated by humans involving the operator tracking the vehicle by direct line of sight
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G05D1/224—Output arrangements on the remote controller, e.g. displays, haptics or speakers
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G05D1/224—Output arrangements on the remote controller, e.g. displays, haptics or speakers
    • G05D1/2244—Optic
    • G05D1/2247—Optic providing the operator with simple or augmented images from one or more cameras
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/222—Remote-control arrangements operated by humans
    • G05D1/224—Output arrangements on the remote controller, e.g. displays, haptics or speakers
    • G05D1/2244—Optic
    • G05D1/2247—Optic providing the operator with simple or augmented images from one or more cameras
    • G05D1/2248—Optic providing the operator with simple or augmented images from one or more cameras the one or more cameras located remotely from the vehicle
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/24—Arrangements for determining position or orientation
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/60—Intended control result
    • G05D1/65—Following a desired speed profile
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Definitions

  • This disclosure is related to robotic solutions. More particularly, embodiments disclosed herein are directed at robotic control systems and their associated interaction workspaces.
  • FIG. 2 D is an example flowchart of a process for detecting an intended regime of a velocity command from a spatial controller.
  • FIGS. 3 A- 3 B show representative examples of application hardware configured to run the disclosed robotic control system.
  • FIG. 4 is an example flowchart of a process for accurately estimating a change in an inertial property of signals measured by an inertial sensor.
  • FIGS. 10 A and 10 B show an example use case of a single-controller coordinated manipulation.
  • FIG. 12 shows a conceptual diagram illustrating examples of teleoperation for different types of vehicles.
  • FIGS. 15 A- 15 D are example Illustrations showing a reorient mode of teleoperating a legged vehicle.
  • FIG. 18 is an example flowchart of a process for teleoperating a legged vehicle in accordance with either a reorient mode or a manipulate mode.
  • one patentable advantage of the disclosed control system is that it is ruggedized for deployment in the real world where sensors (i.e., hardware and/or software) such as IoT sensors are subject to dynamic environmental challenges.
  • Another advantage of the disclosed control system is that provides highly Intuitive (i.e., easily understandable by humans) control of robotic manipulation systems (e.g., robotic arms).
  • Yet another advantage of the disclosed robotic control system is that it is platform agnostic (i.e. it can be mounted or is otherwise associated with any suitable robotic platform).
  • another advantage of the disclosed robotic control system is that it can be integrated to exchange (i.e., send to and/or receive from) information with a wide variety of sensors, from different vendors, and for different applications.
  • the disclosed control system can be a stand-alone app running on a mobile device (e.g., on a mobile phone or a tablet having a graphical user interface (GUI)).
  • GUI graphical user interface
  • the disclosed control system can be integrated into third party apps running on a mobile device.
  • the disclosed control system can be integrated into a robot (or, otherwise suitable application hardware) to remotely control the operation and/or movement of the robot via a desktop app or a mobile app. Examples of a robot include manipulation systems with one or more robotic arms and remotely operated vehicles that move underwater, in the air, or on land using propellers, tracks, wheels, legs, or serpentine motion.
  • robotic arm(s) can have multiple degrees of freedom, a gripper for interaction with the environment and a camera sensor that relays image/video feedback to the disclosed control system.
  • the robotic arm(s) can be mounted on a ground vehicle having wheels, tracks, legs, or other suitable mechanisms to enable the “body” of the vehicle to be controlled by the robotic control system in 1 (one) or more degrees of freedom.
  • the disclosed control system can send commands or instructions to a robot or a vehicle, causing the robot to move or maneuver.
  • the commands or instructions can be directed to move or maneuver a part of a robot without movement of the other body.
  • the commands or instructions can cause movement of the body of the vehicle without movement of the ends of legs of the vehicle.
  • movement of a robot can be an outcome of moving a spatial controller, a joystick, or a user-provided gesture indicated on a user interface.
  • the spatial controller can be connected via a wired or a wireless connection to a computing device that is configured to run the disclosed robotic control system.
  • Information e.g., linear and/or angular displacement, velocity, etc.
  • compute parameters e.g., target velocities, direction of motion, etc.
  • a user can view movement of the robot or the vehicle on a display screen of the computing device.
  • the disclosed robotic control system can be configured to operate in two modes: a position or “mimic” mode and a velocity or “fly” mode.
  • the robotic control system mimics the velocity of the spatial controller.
  • the robotic control system maps the velocity of the spatial controller to the velocity of the robot.
  • the robotic control system operates the robot or vehicle to move with a non-zero velocity as the operator continues to hold the spatial controller at a position away from the rest position of the spatial controller while holding down the “move” button.
  • a robot, vehicle, or spatial controller can have up to 6 degrees-of-freedom (DOF), including up to 3 linear DOF and up to 3 angular DOF.
  • DOF degrees-of-freedom
  • the linear and angular DOFs can each be expressed as vectors in a coordinate system with up to 3 dimensions.
  • a coordinate system is referred to herein as a “coordinate frame” or “reference frame”.
  • FIG. 1 A shows an overview of operation of a representative robot.
  • the robotic system (which can be added as a “bolt-on” attachment) includes a robotic manipulator arm 152 coupled to a gripper 156 .
  • Gripper 156 includes video camera and lights (collectively denoted as 154 ). Operation of gripper 156 can be viewed remotely (in the form of live video) on the screen of an electronic device 158 configured to run the disclosed control system.
  • the disclosed embodiments allow use of intuitive movements or gestures associated with the mobile device 158 to remotely operate gripper 156 and/or robotic manipulator arm 152 by interpreting the operator's intentions from the measurements of the inertial sensors in the mobile device 158 .
  • the disclosed system can detect the gesture indicating the operator's intent and move the gripper 156 through a pure rotation about its axis.
  • the disclosed control system software can detect and execute the operator's intended motion.
  • control system software is able to resolve the intent of the operator's intended motion, the control system software is termed as a “direction intent arbiter.”
  • Region 170 in FIG. 1 A shows that the gripper and the direction intent arbiter have identical the principal axes.
  • a user sitting on wheelchair 150 would possibly be able to control the operation of gripper 156 and/or robotic manipulator arm 152 only by manipulating joystick 160 .
  • the present technology eliminates such a need.
  • the user sitting on a wheelchair can operate gripper 156 and/or robotic manipulator arm 152 using intuitive motion of mobile device 158 and the robotic manipulator arm 152 does not receive any command signal or telemetry signal from knob 160 , or otherwise from the wheelchair.
  • one advantage of the disclosed technology is that it increases user independence and quality of life by maximizing reachable space while minimizing user fatigue and effort. This can be of benefit in reducing the cognitive burden of users with limited motor functions.
  • camera and lights 156 extend a user's sight and reaching abilities. For example, the user can use the disclosed technology to reach items on a shelf that is a few feet vertically above wheel chair 150 . Consequently, a user on a wheelchair would require less assistive care for mundane and repetitive tasks.
  • FIG. 1 B shows an overview of operation of another representative robot controlled in accordance with the disclosed technology.
  • the disclosed robotic control system can recognize, select, and interact with objects from the perspective of different robotic agents or platforms.
  • FIG. 1 B shows a Squad Assist Robotic System (including an air/flying robotic system 182 and a ground robotic system 184 ) for addressing threats in highly unstructured terrain, such as in explosive ordnance disposal (EOD), intelligence, surveillance and reconnaissance (ISR) and special weapons and tactics (SWAT) missions.
  • EOD explosive ordnance disposal
  • ISR intelligence, surveillance and reconnaissance
  • SWAT special weapons and tactics
  • the operator selects the object of interest on a user interface associated with the control system, and commands flying robot 182 to automatically track the object of interest with a different camera for a better view.
  • the robotic system can be used as a force multiplier for the operator via seamless UAS/UGV teaming and commanded via an intuitive GUI configured to run on a portable controller.
  • the operator uses a portable controller that relies on intuitive inertial inputs to command both air and ground robotic systems, minimizing training time and cognitive burden while enabling novel coordinated capabilities.
  • the operator could control the ground robot from the perspective of a camera mounted on the aerial robot that provides an unobstructed view of its surroundings, while the ground vehicle is in the field of view of the camera on the aerial vehicle.
  • the robotic system complies with the military's interoperability (IOP) architecture requirements to ensure future-proof capability expansion through upgrades and new payloads without costly platform replacements.
  • IOP interoperability
  • FIG. 1 C shows a diagrammatic representation of a distributed robotic system for remote monitoring using multiple robots.
  • the distributed robotic system is a collaborative, one-to-many control system for multi-domain unmanned systems.
  • multi-domain unmanned systems include slaved UAS, smart sensor pods, facility monitoring UASs, and the like. These unmanned systems can be used to monitor hazardous worksites and transfer of hazardous materials.
  • the control system can be configured to be accessible via a mobile device app.
  • FIG. 1 D shows a diagrammatic representation of a robotic control system operating in an example environment.
  • a spatial input device (SID) 102 provides inputs to electronic devices 104 A, 104 B, 104 C running the disclosed robotic control system.
  • the robotic control system is an application—mission planner software, configured to run on devices 104 A, 104 B, 104 C.
  • the control system application software can be written using a software environment called the Robot Operating System (ROS).
  • ROS provides a collection of tools, libraries, and conventions to create robot capabilities 112 across a wide variety of platforms 110 .
  • Devices 104 A, 104 B, 1040 are configured to display user dashboard 106 with an easy-to-operate, user-friendly format.
  • Devices 104 A, 104 B, 104 C communicate over a wireless communications network 108 (e.g., 4G, LTE, or mesh networks).
  • a wireless communications network 108 e.g., 4G, LTE, or mesh networks.
  • the disclosed robotic control system can be used to control unmanned vehicles in the air, on the ground, on the water surface, or underwater.
  • the disclosed control system enables collaboration of multi-domain (air/ground/maritime) unmanned systems.
  • one advantage of the control system is that it can solve problems larger than the sum of robotic “parts” with multi-platform teaming.
  • Examples of spatial input device 102 include (but are not limited to) a mobile phone, a tablet computer, a stylus for a touch screen, a light beam scanner, a motion controller, a gaming controller, or a wearable device worn on any part of the human body.
  • Examples of platforms 110 include (but are not limited to) mobile aerial vehicles (multi-rotor and fixed-wing), ground vehicles (tracked, wheeled, or legged), and maritime vehicles (surface and subsurface), or stationary (fixed or steerable) platforms.
  • Capabilities 112 can be associated with performing autonomous behaviors, sensor-related tasks, or object manipulation tasks (e.g., using robotic arm manipulators).
  • the disclosed control system provides supports for tele-operation (e.g., allowing users to remotely “drive” a robot) integrated with obstacle detection and avoidance functionalities.
  • the disclosed control system is agnostic to the underlying communications network 108 .
  • the robotic control system can integrate deep learning, artificial intelligence, or other suitable machine learning methodologies.
  • the robotic control system provides cloud-based access that augments edge processing to unlock deep learning and custom analytics.
  • FIG. 2 A shows an example use case of application of detecting an intended direction of a velocity vector from a spatial controller.
  • a user may wish to generate pristine or ideal movement of a gripper or a spatial controller. But the generated movement may not be ideal.
  • the disclosed control system may detect that the initially generated movement is not “notably ideal” and accordingly rectify such movement.
  • a user may wish to generate movement such that a gripper moves along an intended or desired straight line, but the resulting movement may be generated along a different straight line.
  • FIG. 2 A shows such a scenario.
  • a user may wish to generate movement of the gripper along the Z axis, but the resulting movement occurs at an angle with respect to the Z axis.
  • FIG. 1 A user may wish to generate movement of the gripper along the Z axis, but the resulting movement occurs at an angle with respect to the Z axis.
  • the disclosed robotic control system can detect that the Z axis of the control reference frame is the nearest principal axis direction (i.e., among X, Y, and Z principal axes) of the control reference frame.
  • the disclosed control system can also compute the angle (e.g., a misalignment representing the deviation of the initial desired velocity vector from the direction of the nearest principal axis) with respect to the Z axis of the control reference frame.
  • a user may wish to generate a linear movement and/or an angular movement (e.g., a twist, a rotation, or a revolution with respect to an axis) of a spatial controller or a gripper.
  • the disclosed robotic control system can detect whether the intended movement is a linear-only movement, an angular-only movement, or a combination of a linear and an angular movement.
  • the disclosed control system can perform regime arbitration to determine whether the intended movement is exclusively in the linear regime, exclusively in the angular regime, or in a combination of both linear and angular regimes.
  • the regime arbitration can be performed on the basis of a linear ratio and an angular ratio computed from movement data of the spatial controller or joystick.
  • FIG. 2 B shows a diagrammatic representation of regime arbitration.
  • Regime arbitration is a technique of determining (using a linear ratio and an angular ratio) whether a user intended to generate linear motion, angular motion, or both linear and angular motion of a spatial controller.
  • FIG. 2 B graphically illustrates regime arbitration using five points denoted a, b, c, d, and e. Each of these points is represented as a coordinate point expressed as (linear ratio, angular ratio). These five points represent possible combinations of linear and angular ratios computed using the magnitudes of linear and angular velocities obtained from spatial motion data of a spatial controller.
  • Point e is an example of a combined linear and angular regime (i.e., motion data corresponds to both linear and angular motion) because the linear ratio and the angular ratio both exceed 1.
  • the point (1, 1) in FIG. 2 B is an inflection point FIG. 2 B because it marks a boundary between small (ratio less than 1) for large (ratio greater than 1) for linear and angular motion of a spatial controller. If either ratio is less than 1, it is ignored unless the other ratio is smaller (i.e. a user probably intended the more dominate regime of motion). If both ratios are greater than 1, then neither regime is ignored (i.e. the user probably intended both linear and angular motion).
  • FIG. 2 C is an example flowchart of a process for detecting an intended direction of a velocity command from a spatial controller.
  • the process can be one of multiple processes associated with the disclosed robotic control system. For example, a user or an operator can move a spatial controller which causes movement or maneuvering of a robot, a vehicle, a gripper, or a robotic arm, causing the vehicle, the gripper, or the robotic arm to go along a direction relatively “close” to the original direction intended by the user.
  • the movement can be related to a portion or part of a robot, a vehicle, a gripper, a robotic arm.
  • the spatial controller can be connected via a wired or a wireless connection to a computing device (e.g., a mobile device or a tablet computer) that is configured to run the process. Based on the movement data from the spatial controller, the computing device can send commands or movement parameters to a remote robot, vehicle, or robotic arm electronically coupled to the computing device.
  • the process receives spatial motion data generated from movement of the spatial controller with respect to a global reference frame (e.g., that of the Earth), wherein the spatial motion data is representative of a desired (or, intended) motion of at least a portion of a vehicle.
  • the spatial motion data can be one or more of: displacement parameter(s), velocity parameter(s), acceleration parameter(s) or any combination thereof.
  • the process computes an initial desired velocity vector representing a desired linear velocity or a desired angular velocity of at least the portion of the vehicle.
  • the process transforms the initial desired velocity vector (e.g., expressed with respect to a frame of the spatial controller) into a control reference frame.
  • the control reference frame can be defined with respect to any one of: the spatial controller, a frame with respect to a user interface on a mobile device or a tablet computer on which the process is configured to run, the robot, vehicle, gripper, robotic arm, or a portion thereof.
  • the transformation in step 214 ) may not be necessary if the control reference frame is the same as the frame of the spatial controller.
  • the control reference frame can be composed with a set of principal axes (e.g., X, Y, Z axes).
  • the process identifies a nearest principal axis direction based on comparing the direction of the initial desired velocity vector with directions of the set of principal axes that are parallel to the control reference frame.
  • the process computes a misalignment angle representing a deviation in direction of the initial desired velocity vector from the nearest principal axis direction from the set of principal axes of the control reference frame.
  • the process determines whether the misalignment angle is less than or within a pre-specified axis-snapping tolerance value (e.g. 5 degrees).
  • the axis-snapping tolerance value can be a tunable parameter.
  • the axis-snapping tolerance can be adjusted based on a metric associated with the accuracy of the desired velocity vector such as its magnitude. For example, the axis-snapping tolerance can be larger (e.g. 7 degrees) when the magnitude of the velocity is smaller (e.g. less than or equal to 0.1 m/s) and be smaller (e.g. 3 degrees) when the magnitude of the velocity is larger (e.g. greater than 0.1 m/s).
  • the process Upon determining that the misalignment angle (computed in step 218 ) is greater than the axis-snapping tolerance value, the process defines (at step 222 ) a final desired velocity vector equal to the initial desired velocity vector. However, upon determining that the misalignment angle (computed in step 218 ) is less than or equal to the axis-snapping tolerance value, the process defines (at step 224 ) a final desired velocity vector based on rotating the initial desired velocity vector such that the final desired velocity vector is parallel to the nearest principal axis of the control reference frame.
  • the axis-snapping tolerance value defines an extent of a match or closeness between the direction of the initial desired velocity vector and the direction of the nearest principal axis.
  • the process transforms the final desired velocity vector (e.g., defined either from step 222 or step 224 ) into a vehicle reference frame of at least the portion of the vehicle.
  • the process sends Information indicating the final desired velocity to at least the portion of the vehicle.
  • the initial desired velocity and the final desired velocity can be linear velocities.
  • the initial desired velocity and the final desired velocity can be angular velocities. It will be understood that several steps of the process can occur in a parallel or near parallel manner. Also, some steps can be optional depending on the specific use case to which the process is directed.
  • a rule can be to determine whether the linear ratio is less than 1 and also less than the angular ratio. Another rule can be to determine whether the angular ratio is less than 1 and also less than a linear ratio. If the process determines that rules associated with the desired linear velocity vector are satisfied, then at step 242 , the process ignores the desired linear velocity vector. If the process determines that rules associated with the desired angular velocity vector are satisfied, then at step 244 , the process ignores the desired angular velocity vector. Ignoring a vector can imply setting the vector to a null vector having a zero magnitude. The purpose of ignoring the linear or angular regimes of motion is to retain only the motion that was intended and ignore any motion that was not intended (or, determined to be not notable).
  • FIG. 6 is an example flowchart of a process for computing a sliding velocity limit boundary for a spatial controller. Diagrammatic representations of the velocity limit boundary and associated concepts are described in connection with FIGS. 8 A- 8 C .
  • a user or an operator expresses a desired velocity of a robot, a vehicle, a gripper, or a robotic arm by moving a spatial controller within a “virtual boundary” that represents the maximum magnitude of the desired velocity.
  • the desired velocity can be related to a portion/part of a robot, a vehicle, a gripper, a robotic arm.
  • the spatial controller can be connected via a wired or a wireless connection to a computing device (e.g., a mobile device or a tablet computer) that is configured to run the process. Based on the movement data from the spatial controller, the computing device can send commands or movement parameters to a remote robot, vehicle, or robotic arm electronically coupled to the computing device.
  • the process receives information describing a first position boundary of a spatial controller (e.g., a circle with a center corresponding to the spatial controller's initial rest position).
  • the first position boundary can correspond to a maximum velocity of the connected vehicle or portion thereof.
  • the maximum velocity of the connected vehicle can be a user-defined quantity or based on manufacturer's specifications.
  • the maximum velocity can be a pre-calculated upper limit of the vehicle's velocity.
  • the maximum velocity can be a velocity calculated to avoid a collision between the vehicle and an obstacle.
  • the maximum velocity can be a velocity calculated to avoid a restricted airspace or geographical zone.
  • the process receives a current position of the spatial controller.
  • the process determines whether or not the spatial controller is outside the first position boundary. Upon determining that the spatial controller is not outside (i.e., within) the first position boundary, the process computes (at step 614 ) a target velocity of the vehicle based on a difference between the current position of the spatial controller and a point lying at the center of the first position boundary. Because the spatial controller is within the first position boundary, the process does not change the first position boundary of the spatial controller.
  • the process sends the computed target velocity to the vehicle, which causes the vehicle to move in accordance with the target velocity.
  • the current position of the spatial controller, the first position boundary, and the second position boundary can be expressed in 1-dimensional, 2-dimensional, or 3-dimensional space.
  • the calculation of target velocity can be based on the difference between the current position of the spatial controller and the point at the center of the first position boundary by considering a deadband boundary and setting the target velocity to zero if the position of the spatial controller is within the deadband boundary. If, after the first position boundary is updated (step 612 ), the spatial controller is moved back inside the first position boundary, then the first position boundary is not moved (i.e., the condition in step 606 will be false).
  • the robotic manipulator arm can be used to first open the door and keep holding the door open.
  • the inertial hold can be applied by means of the inertial hold filter described in FIG. 5 .
  • the manipulator receives no direct control signals or telemetry signals from the wheelchair. This implies that the manipulator can cancel out movements at its base such that it can hold or stabilize itself or an object.
  • the process can utilize inaccurate or coarse signal measurements from inexpensive, commercial-grade inertial sensors to calculate the (final) velocity of the wheel chair (and the robotic arm) and provide control instructions such that the gripper stays “fixed” in space while the wheelchair moves through the doorway. This allows the operator to lock specific degree(s)-of-freedom (DOF) in the inertial frame.
  • DOE degree(s)-of-freedom
  • FIG. 7 B is an example use case of application of the inertial lock process described in FIG. 5 .
  • FIG. 7 B shows the process applying an “automatic level control” by applying an “orientation lock” in the horizontal plane on the angular (rotational) degrees of freedom of the robotic arm to prevent the open container from spilling the liquid.
  • the robotic arm manipulator can be controlled by a mobile device operated by a user on a wheelchair.
  • the wheelchair can be the wheelchair arrangement described in connection with FIG. 1 A .
  • the robotic manipulator arm is autonomously controlled (without manual involvement).
  • the “orientation lock” is enabled by sensing gravity at the base of the robotic manipulator arm.
  • FIGS. 8 A- 8 C also show a deadband boundary defining an outer extent of a deadband region.
  • the deadband region is a region within which the robot does not move (i.e., the robot has zero velocity) even though the spatial controller is moved.
  • the size of the deadband region depends on the accuracy of the inertial sensors. In some embodiments, the size of the deadband region is small.
  • the current position of the spatial controller is shown as a dot moving towards the velocity limit boundary from the rest position at the center of the circle (termed velocity center in FIG. 8 A ). Anywhere outside the deadband region, the spatial movement/displacement of the spatial controller produces a corresponding velocity command to be sent to the robot or the vehicle.
  • FIGS. 9 A and 9 B show example use cases of applications of increasing the workspace of the manipulation system.
  • the disclosed control system can use the mobility of the platform with the manipulator arm(s) to increase the workspace of the manipulation system.
  • a manipulator arm mounted on a legged ground robot grasps an object that would normally be out of reach by automatically using the mobility of the robot to raise its body.
  • the use of the mobility of the platform with the manipulator arm(s) is managed by the disclosed robotic control system.
  • the robotic control system associated with a multi-rotor flying robot can use the mobility of the 3 degree-of-freedom (DOF) manipulator arm to allow full 6-DOF control of an object attached to the manipulator arm.
  • DOF 3 degree-of-freedom
  • FIGS. 10 A and 10 B show an example use case of coordinated manipulation by a single-controller.
  • the controller such as a spatial controller
  • the controller is able to singly control the end-effectors of two or more manipulators mounted on separate robotic platforms by sending commands to a common point (e.g. the midpoint between two grippers on the manipulators).
  • a single spatial controller can be used to command two ground robots with manipulator arms to lift an object that would otherwise be too heavy for each individual robot to lift.
  • FIGS. 11 A- 11 D Illustrate example use cases of teleoperation (also referred to herein as third person teleoperation) using a remote camera.
  • the disclosed robotic control system can operate/control a first vehicle (or a portion thereof) from the perspective of a camera that keeps the vehicle in view.
  • FIG. 11 A shows a generic overview of teleoperation of the vehicle.
  • vehicle (or a portion thereof) 1106 can either move or remain stationary.
  • Vehicle 1106 is in the field of view of camera 1102 .
  • the position of the vehicle and the position of the camera can be with reference to a global reference frame, e.g., the Earth's reference frame using altitude, latitude, and longitude.
  • Direction 1107 is a direction of motion of vehicle 1106 with respect to the global reference frame.
  • a user can view vehicle 1106 (and its movements) in real time or near real time on the user interface displayed on computing device 1104 .
  • Desired direction 1105 (displayed on computing device 1104 ) corresponds to direction 1107 of the movement of vehicle 1106 .
  • desired direction 1105 is represented with respect to a control frame of the user interface (or, alternatively the user interface reference frame).
  • camera 1102 can provide (via a first network connection) information of its position to computing device 1104 and a live video feed of vehicle 1106 .
  • vehicle 1106 provides information of its location to computing device 1104 via a second network connection.
  • FIG. 11 B shows a first use case of teleoperation.
  • camera 1110 is shown as a stationary camera that is mounted on pole 1111 and vehicle 1112 is shown as a ground vehicle located on Earth's surface 1113 .
  • Camera 1110 provides information of its position to computing device 1108 and a live video feed of vehicle 1112 via a first network connection.
  • the desired direction corresponds to direction of the movement of vehicle 1112 .
  • Vehicle 1112 provides information of its location to computing device 1108 via a second network connection.
  • Commands (e.g., for movement) of vehicle 1112 can be provided using a graphical user interface (GUI) displayed on computing device 1108 .
  • GUI graphical user interface
  • FIG. 11 C shows a second use case of teleoperation.
  • camera 1124 is shown as included on robot 1123 which is an unmanned aerial vehicle.
  • Robot 1123 can hover and/or fly in a manner such that legged robot 1126 is automatically in the field of view of camera 1124 .
  • Camera 1124 provides information of its position to computing device 1122 and a live feed of legged robot 1126 via a first network connection.
  • the desired direction (displayed on computing device 1122 ) corresponds to direction of the movement of legged robot 1126 .
  • Legged robot 1126 located on Earth's surface 1121 provides information of its location to computing device 1122 via a second network connection.
  • a user can move joystick 1125 electronically attached to computing device 1122 to specify movement of vehicle 1112 .
  • the movements made via joystick 1125 are received at computing device 1122 and electronically converted into commands or instructions at computing device 1122 .
  • the commands or instructions are sent to legged robot 1126 via the second network connection. Accordingly, legged robot 1126 can move based on the commands.
  • the disclosed robotic control system that allows teleoperation can be configured to run on computing device 1122 .
  • FIG. 11 D shows a third use case of teleoperation.
  • camera 1134 is shown as included on robot 1133 which is an unmanned aerial vehicle.
  • Robot 1133 can hover and/or fly in a manner such that unmanned vehicle 1136 is automatically in the field of view of camera 1134 .
  • Camera 1134 provides information of its position to computing device 1132 and a live video feed of unmanned vehicle 1136 via a first network connection.
  • the desired direction corresponds to direction of the movement of legged robot 1136 .
  • Unmanned vehicle 1136 can be located on the ground or in air.
  • Unmanned vehicle 1136 can provide information of its location to computing device 1132 via a second network connection.
  • FIG. 12 shows a conceptual diagram illustrating examples of teleoperation for different types of vehicles.
  • One or more of the vehicles can be independently controlled by a spatial controller.
  • spatial controller 1210 (having a reference frame denoted 1210 a ) can independently control one or more of: a multi-rotor vehicle 1220 , an unmanned aerial vehicle 1230 , a ground vehicle 1240 , or a legged robot 1250 .
  • the vehicles shown in FIG. 12 can have a camera mounted on the body of the vehicle.
  • the spatial controller can independently control movement of the camera and/or the body of a given vehicle.
  • the movement of the spatial controller can be converted into movement of the camera mounted on a vehicle and also a movement of the body of the vehicle.
  • Multi-rotor vehicle 1220 includes a camera 1220 b and a body 1220 c .
  • a reference frame of camera 1220 a is denoted as 1220 b .
  • a reference frame of body 1220 c of multi-rotor vehicle 1220 is denoted as 1220 d .
  • Unmanned aerial vehicle 1230 includes a camera 1230 a (having a frame of reference denoted 1230 b ) and a body 1230 c (having a frame of reference denoted 1230 d ).
  • Ground vehicle 1240 includes a first body part 1240 c (having a frame of reference denoted 1240 d ) and a second body part 1240 e (having a frame of reference denoted 1240 f ).
  • Camera 1240 a (having a frame of reference denoted 1240 b ) is mounted on ground vehicle 1240 .
  • spatial controller 1210 can independently control movement of camera 1240 a , movement of first body part 1240 c (such as a manipulator arm attached to vehicle 1240 ), and/or movement of second body part 1240 f .
  • Legged robot 1250 includes a body 1250 (having a frame of reference denoted 1250 b ).
  • the disclosed robotic control system (e.g., configured to run on an input device connected to spatial controller 1210 ) can perform appropriate transformations of commands and translations of reference frames associated with movement of a camera, a body of a vehicle, or a part of a body of a vehicle.
  • FIG. 13 is an example flowchart of a process for teleoperating a vehicle.
  • the process receives, from an input device coupled to a user interface device, a desired direction of motion of at least one portion of a vehicle in a user interface reference frame.
  • the process receives, via a first network connection coupled to a remote camera, a position of the camera in a global reference frame, wherein the vehicle is within a field of view of the camera and the camera provides visual data (e.g., images and/or video in a suitable digital format) for projection into the user interface reference frame.
  • the position of the camera can be defined using 6 degrees-of-freedom such as latitude, longitude, altitude, roll, pitch, and heading.
  • spatial teleoperation of a vehicle can be implemented using various modes.
  • a vehicle's velocity can be expressed as moving in accordance with a traverse mode, a reorient mode, or a manipulate mode.
  • a traverse mode the planar velocity of the body of the vehicle with respect to the ground (e.g., including a translation over XY plane on the ground and a rotation about a vertical Z axis) is proportional to angular displacement of the spatial controller controlling the vehicle.
  • an operator can select a traverse mode to move a vehicle on the ground along a curving roadway.
  • FIGS. 15 A- 15 D are example illustrations showing a reorient mode of teleoperating a legged vehicle.
  • the ends of the legs of the vehicle (e.g., vehicle 1500 ) in FIG. 15 A do not move (i.e., they are stationary).
  • the upper part of the body of the vehicle can have a linear motion and/or a angular motion.
  • the linear motion and the angular motion can be independent of one another.
  • these motions can be commanded using linear movement and/or angular movement of a spatial controller (e.g., spatial controller 1506 ) shown in FIG. 15 B .
  • Spatial controller generates linear velocity 1504 and angular velocity 1508 (e.g. both represented in global reference frame 1502 ) which is mapped into linear and/or angular velocity of vehicle 1500 .
  • FIGS. 16 A- 16 D are example illustrations showing a manipulate mode of teleoperating a vehicle.
  • movement of at least one leg of the vehicle is caused without movement of ends of other legs of the vehicle.
  • the teleoperated leg can have up to 6 degrees-of-freedom (DOF).
  • these motions can be commanded using linear movement and/or angular movement of a spatial controller (e.g., spatial controller 1606 ) shown in FIG. 16 B .
  • Spatial controller generates linear velocity 1604 and angular velocity 1608 (e.g. both represented in global reference frame 1602 ) which will be mapped into linear and/or angular velocity of the teleoperated leg of vehicle 1600 .
  • the process computes a target linear velocity of the part of the vehicle in the spatial controller reference frame, wherein the target linear velocity of the part of the vehicle is derived from the linear velocity data of the spatial controller in the spatial controller reference frame.
  • the process computes a target angular velocity of the part of the vehicle in the spatial controller reference frame, wherein the target linear velocity of the part of the vehicle is derived from the angular velocity data of the spatial controller in the spatial controller reference frame.
  • the process transforms the target linear velocity and the target angular velocity into a vehicle reference frame defined with respect to the vehicle.
  • inventions or portions thereof of the system and method of the present invention may be implemented in computer hardware, firmware, and/or computer programs executing on programmable computers or servers that each includes a processor and a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements).
  • Any computer program may be implemented in a high-level procedural or object-oriented programming language to communicate within and outside of computer-based systems.
  • Any computer program may be stored on an article of manufacture, such as a storage medium (e.g., CD-ROM, hard disk, or magnetic diskette) or device (e.g., computer peripheral), that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the functions of the embodiments.
  • a storage medium e.g., CD-ROM, hard disk, or magnetic diskette
  • device e.g., computer peripheral
  • the embodiments, or portions thereof may also be implemented as a machine-readable storage medium, configured with a computer program, where, upon execution, instructions in the computer program cause a machine to operate to perform the functions of the embodiments described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Automation & Control Theory (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Human Computer Interaction (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Manipulator (AREA)
  • Operation Control Of Excavators (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Studio Devices (AREA)
US17/417,206 2018-12-31 2019-12-31 Spatial teleoperation of legged vehicles Active 2041-02-11 US12372958B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/417,206 US12372958B2 (en) 2018-12-31 2019-12-31 Spatial teleoperation of legged vehicles

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201862786888P 2018-12-31 2018-12-31
US17/417,206 US12372958B2 (en) 2018-12-31 2019-12-31 Spatial teleoperation of legged vehicles
PCT/US2019/069159 WO2020142561A1 (en) 2018-12-31 2019-12-31 Spatial teleoperation of legged vehicles

Publications (2)

Publication Number Publication Date
US20220075364A1 US20220075364A1 (en) 2022-03-10
US12372958B2 true US12372958B2 (en) 2025-07-29

Family

ID=71407134

Family Applications (5)

Application Number Title Priority Date Filing Date
US17/417,206 Active 2041-02-11 US12372958B2 (en) 2018-12-31 2019-12-31 Spatial teleoperation of legged vehicles
US17/417,176 Active 2040-10-17 US11886182B2 (en) 2018-12-31 2019-12-31 Systems and methods of detecting intent of spatial control
US17/417,194 Active 2040-09-08 US12124256B2 (en) 2018-12-31 2019-12-31 Systems and methods of remote teleoperation of robotic vehicles
US18/540,632 Active US12189386B2 (en) 2018-12-31 2023-12-14 Systems and methods of detecting intent of spatial control
US18/908,639 Pending US20250028317A1 (en) 2018-12-31 2024-10-07 Systems and methods of detecting intent of spatial control

Family Applications After (4)

Application Number Title Priority Date Filing Date
US17/417,176 Active 2040-10-17 US11886182B2 (en) 2018-12-31 2019-12-31 Systems and methods of detecting intent of spatial control
US17/417,194 Active 2040-09-08 US12124256B2 (en) 2018-12-31 2019-12-31 Systems and methods of remote teleoperation of robotic vehicles
US18/540,632 Active US12189386B2 (en) 2018-12-31 2023-12-14 Systems and methods of detecting intent of spatial control
US18/908,639 Pending US20250028317A1 (en) 2018-12-31 2024-10-07 Systems and methods of detecting intent of spatial control

Country Status (4)

Country Link
US (5) US12372958B2 (es)
EP (6) EP3880547B1 (es)
ES (3) ES2995021T3 (es)
WO (3) WO2020142561A1 (es)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102339002B1 (ko) 2020-07-30 2021-12-14 네이버랩스 주식회사 로봇 관제 방법 및 시스템
JP7274157B1 (ja) * 2022-08-01 2023-05-16 株式会社アルヴィオン 操作対象機器の操作用のプログラム

Citations (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5617515A (en) 1994-07-11 1997-04-01 Dynetics, Inc. Method and apparatus for controlling and programming a robot or other moveable object
EP0664040B1 (en) 1993-08-02 2000-02-23 Motorola, Inc. Map-matching with competing sensory positions
EP0679974B1 (en) 1989-12-11 2002-05-22 Caterpillar Inc. Integrated vehicle positioning and navigation system, apparatus and method
US20060028548A1 (en) 2004-08-06 2006-02-09 Salivar William M System and method for correlating camera views
EP1193168B1 (en) 2000-09-26 2006-05-10 Rafael Armament Development Authority Ltd. Unmanned mobile device
EP1870670A1 (en) 2006-06-21 2007-12-26 Microinfinity, Inc. Method and apparatus for space recognition according to the movement of an input device
US20090153349A1 (en) * 2007-12-17 2009-06-18 Omni Motion Technology Corporation Handheld controller and method of controlling a controlled object by detecting a movement of a handheld controller
US20090234499A1 (en) 2008-03-13 2009-09-17 Battelle Energy Alliance, Llc System and method for seamless task-directed autonomy for robots
US7702442B2 (en) 2004-08-06 2010-04-20 Honda Motor Co., Ltd. Control device for vehicle
US20100103096A1 (en) * 2007-07-06 2010-04-29 Sony Corporation Input apparatus, control apparatus, control system, control method, and handheld apparatus
US20100228406A1 (en) 2009-03-03 2010-09-09 Honeywell International Inc. UAV Flight Control Method And System
US20110071789A1 (en) 2009-09-22 2011-03-24 Samsung Electronics Co., Ltd. Real-time motion recognition system and method
US20110098860A1 (en) * 2009-10-28 2011-04-28 Honda Motor Co., Ltd. Control device for legged mobile robot
US20110106339A1 (en) 2006-07-14 2011-05-05 Emilie Phillips Autonomous Behaviors for a Remote Vehicle
US20110231050A1 (en) * 2010-03-22 2011-09-22 Goulding John R In-Line Legged Robot Vehicle and Method for Operating
US20120185098A1 (en) 2011-01-19 2012-07-19 Harris Corporation Telematic interface with directional translation
US20120203487A1 (en) 2011-01-06 2012-08-09 The University Of Utah Systems, methods, and apparatus for calibration of and three-dimensional tracking of intermittent motion with an inertial measurement unit
US8267786B2 (en) 2005-08-24 2012-09-18 Nintendo Co., Ltd. Game controller and game system
US8355818B2 (en) 2009-09-03 2013-01-15 Battelle Energy Alliance, Llc Robots, systems, and methods for hazard evaluation and visualization
US20140046589A1 (en) 2011-04-14 2014-02-13 Hexagon Technology Center Gmbh Measuring system for determining 3d coordinates of an object surface
US20140210663A1 (en) 2011-04-14 2014-07-31 Hexagon Technology Center Gmbh Measuring system and method for determining new points
US20140336848A1 (en) 2013-05-10 2014-11-13 Palo Alto Research Center Incorporated System and method for detecting, tracking and estimating the speed of vehicles from a mobile platform
US8924021B2 (en) 2006-04-27 2014-12-30 Honda Motor Co., Ltd. Control of robots from human motion descriptors
US8965620B2 (en) 2013-02-07 2015-02-24 Harris Corporation Systems and methods for controlling movement of unmanned vehicles
US20150220085A1 (en) 2014-01-31 2015-08-06 Kabushiki Kaisha Topcon Measuring System
US20150217449A1 (en) * 2014-02-03 2015-08-06 Brain Corporation Apparatus and methods for control of robot actions based on corrective user inputs
WO2016089442A1 (en) 2014-12-06 2016-06-09 Horsemoon Llc Hand gesture recognition system for controlling electronically controlled devices
US20160291593A1 (en) 2015-03-03 2016-10-06 PreNav, Inc. Scanning environments and tracking unmanned aerial vehicles
US9498883B2 (en) 2011-12-15 2016-11-22 Korea Institute Of Ocean Science & Technology Multi-joint underwater robot having complex movement functions of walking and swimming and underwater exploration system using same
US20160371983A1 (en) 2015-06-22 2016-12-22 Garmin Switzerland Gmbh Parking assist system and method
US20170068242A1 (en) 2014-05-21 2017-03-09 SZ DJI Technology Co., Ltd. Remote control device, control system and method of controlling
US20170095382A1 (en) 2014-03-21 2017-04-06 Rensselaer Polytechnic Institute Mobile human-friendly assistive robot
US20170185081A1 (en) * 2015-12-27 2017-06-29 Spin Master Ltd. Remotely controlled motile device system
WO2017204826A1 (en) 2016-05-27 2017-11-30 Abb Schweiz Ag User interface for a teleoperated robot
US9969086B1 (en) * 2015-11-03 2018-05-15 Boston Dynamics, Inc. Achieving a target gait in a legged robot based on steering commands
US20180154518A1 (en) 2015-05-29 2018-06-07 Abb Schweiz Ag User interface for a teleoperated robot
US10081104B1 (en) 2015-09-15 2018-09-25 Boston Dynamics, Inc. Determination of robotic step path
US20180364048A1 (en) 2017-06-20 2018-12-20 Idhl Holdings, Inc. Methods, architectures, apparatuses, systems directed to device position tracking
US11036217B2 (en) 2018-05-03 2021-06-15 King Abdullah University Of Science And Technology Controlling a vehicle using a remotely located laser and an on-board camera
US11079751B2 (en) 2018-01-12 2021-08-03 Superior Marine Products Llc Gesturing for control input for a vehicle

Patent Citations (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0679974B1 (en) 1989-12-11 2002-05-22 Caterpillar Inc. Integrated vehicle positioning and navigation system, apparatus and method
EP0664040B1 (en) 1993-08-02 2000-02-23 Motorola, Inc. Map-matching with competing sensory positions
US5617515A (en) 1994-07-11 1997-04-01 Dynetics, Inc. Method and apparatus for controlling and programming a robot or other moveable object
EP1193168B1 (en) 2000-09-26 2006-05-10 Rafael Armament Development Authority Ltd. Unmanned mobile device
US7702442B2 (en) 2004-08-06 2010-04-20 Honda Motor Co., Ltd. Control device for vehicle
US20060028548A1 (en) 2004-08-06 2006-02-09 Salivar William M System and method for correlating camera views
US8267786B2 (en) 2005-08-24 2012-09-18 Nintendo Co., Ltd. Game controller and game system
US8924021B2 (en) 2006-04-27 2014-12-30 Honda Motor Co., Ltd. Control of robots from human motion descriptors
EP1870670A1 (en) 2006-06-21 2007-12-26 Microinfinity, Inc. Method and apparatus for space recognition according to the movement of an input device
US20110106339A1 (en) 2006-07-14 2011-05-05 Emilie Phillips Autonomous Behaviors for a Remote Vehicle
US20100103096A1 (en) * 2007-07-06 2010-04-29 Sony Corporation Input apparatus, control apparatus, control system, control method, and handheld apparatus
US20090153349A1 (en) * 2007-12-17 2009-06-18 Omni Motion Technology Corporation Handheld controller and method of controlling a controlled object by detecting a movement of a handheld controller
US20090234499A1 (en) 2008-03-13 2009-09-17 Battelle Energy Alliance, Llc System and method for seamless task-directed autonomy for robots
US20100228406A1 (en) 2009-03-03 2010-09-09 Honeywell International Inc. UAV Flight Control Method And System
US8355818B2 (en) 2009-09-03 2013-01-15 Battelle Energy Alliance, Llc Robots, systems, and methods for hazard evaluation and visualization
US20110071789A1 (en) 2009-09-22 2011-03-24 Samsung Electronics Co., Ltd. Real-time motion recognition system and method
US20110098860A1 (en) * 2009-10-28 2011-04-28 Honda Motor Co., Ltd. Control device for legged mobile robot
US20170045893A1 (en) 2010-03-22 2017-02-16 John R. Goulding Single-track legged vehicle
US20110231050A1 (en) * 2010-03-22 2011-09-22 Goulding John R In-Line Legged Robot Vehicle and Method for Operating
US20120203487A1 (en) 2011-01-06 2012-08-09 The University Of Utah Systems, methods, and apparatus for calibration of and three-dimensional tracking of intermittent motion with an inertial measurement unit
US20120185098A1 (en) 2011-01-19 2012-07-19 Harris Corporation Telematic interface with directional translation
US20140046589A1 (en) 2011-04-14 2014-02-13 Hexagon Technology Center Gmbh Measuring system for determining 3d coordinates of an object surface
US20140210663A1 (en) 2011-04-14 2014-07-31 Hexagon Technology Center Gmbh Measuring system and method for determining new points
US9498883B2 (en) 2011-12-15 2016-11-22 Korea Institute Of Ocean Science & Technology Multi-joint underwater robot having complex movement functions of walking and swimming and underwater exploration system using same
US8965620B2 (en) 2013-02-07 2015-02-24 Harris Corporation Systems and methods for controlling movement of unmanned vehicles
US20140336848A1 (en) 2013-05-10 2014-11-13 Palo Alto Research Center Incorporated System and method for detecting, tracking and estimating the speed of vehicles from a mobile platform
US20150220085A1 (en) 2014-01-31 2015-08-06 Kabushiki Kaisha Topcon Measuring System
US20150217449A1 (en) * 2014-02-03 2015-08-06 Brain Corporation Apparatus and methods for control of robot actions based on corrective user inputs
US20170095382A1 (en) 2014-03-21 2017-04-06 Rensselaer Polytechnic Institute Mobile human-friendly assistive robot
US20170068242A1 (en) 2014-05-21 2017-03-09 SZ DJI Technology Co., Ltd. Remote control device, control system and method of controlling
WO2016089442A1 (en) 2014-12-06 2016-06-09 Horsemoon Llc Hand gesture recognition system for controlling electronically controlled devices
US20170329403A1 (en) 2014-12-06 2017-11-16 Horsemoon Llc Hand gesture recognition system for controlling electronically controlled devices
US20160291593A1 (en) 2015-03-03 2016-10-06 PreNav, Inc. Scanning environments and tracking unmanned aerial vehicles
US20180154518A1 (en) 2015-05-29 2018-06-07 Abb Schweiz Ag User interface for a teleoperated robot
US20160371983A1 (en) 2015-06-22 2016-12-22 Garmin Switzerland Gmbh Parking assist system and method
US10081104B1 (en) 2015-09-15 2018-09-25 Boston Dynamics, Inc. Determination of robotic step path
US9969086B1 (en) * 2015-11-03 2018-05-15 Boston Dynamics, Inc. Achieving a target gait in a legged robot based on steering commands
US20170185081A1 (en) * 2015-12-27 2017-06-29 Spin Master Ltd. Remotely controlled motile device system
WO2017204826A1 (en) 2016-05-27 2017-11-30 Abb Schweiz Ag User interface for a teleoperated robot
US20180364048A1 (en) 2017-06-20 2018-12-20 Idhl Holdings, Inc. Methods, architectures, apparatuses, systems directed to device position tracking
US11079751B2 (en) 2018-01-12 2021-08-03 Superior Marine Products Llc Gesturing for control input for a vehicle
US11036217B2 (en) 2018-05-03 2021-06-15 King Abdullah University Of Science And Technology Controlling a vehicle using a remotely located laser and an on-board camera

Non-Patent Citations (17)

* Cited by examiner, † Cited by third party
Title
Extended European Search Report issued in corresponding European Patent Application No. 19906801.6 on Dec. 13, 2022 (9 pages).
Extended European Search Report issued in corresponding European Patent Application No. 19906876.8 on Jul. 8, 2022 (10 pages).
Extended European Search Report issued in corresponding European Patent Application No. 19907050.9 on Dec. 20, 2022 (18 pages).
International Search Report and Written Opinion for International Application No. PCT/US19/69141; Date of Mailing: Apr. 29, 2020; 14 pages.
International Search Report and Written Opinion mailed Mar. 18, 2020 in International Application No. PCT/US2019/69159 (11 pages).
International Search Report and Written Opinion of International Application No. PCT/US19/69148; Date of Mailing: Mar. 24, 2020; 8 pages.
Josep-Arnau Claret et al., "Teleoperating a Mobile Manipulator and a Free-Flying Camera from a Single Haptic Device," 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, Oct. 23, 2016, pp. 291-296, XP0330222240, DOI: 10.1109/SSRR.2016.7784318.
Joseph-Arnau Claret et al., "Teleoperating a mobile manipulator and a free-flying camera from a single haptic device," 2016, IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, Oct. 23, 2016, pp. 291-296, XP033022240, DOI: 10.1109/SSRR.2016.7784318 [retrieved on Dec. 14, 2016].
Notice of Allowance issued in related U.S. Appl. No. 17/417,176, filed Sep. 15, 2023 (21 pages).
Notice of Allowance issued May 22, 2024, in related application, U.S. Appl. No. 17/417,194 (10 pages).
Office Action issued Jul. 17, 2024, in related application, U.S. Appl. No. 18/540,632 (8 pages).
Pongrac, Helena & Peer, Angelika & Faerber, Berthold & Buss, Martin. (2008). Effects of Varied Human Movement Control on Task Performance and Feeling of Telepresence. 755-765. 10.1007/978-3-540-69057-3_97. (Year: 2008). *
Replacement Extended European Search Report issued in corresponding European Patent Application No. 19906801.6 on Dec. 20, 2022 (13 pages).
U.S. Appl. No. 17/417,176 for Summer et al. filed on Jun. 22, 2021.
U.S. Appl. No. 17/417,194 for Summer et al. filed on Jun. 22, 2021.
Wieber, P. B., Tedrake, R., & Kuindersma, S. (2016). Modeling and control of legged robots. In Springer handbook of robotics (pp. 1203-1234). Cham: Springer International Publishing. (Year: 2016). *
Xuesu Xiao et al., "Visual Servoing for Teleoperation using a Tethered UAV," 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR), IEEE, Oct. 11, 2017, pp. 147-152, XP033242142, DOI: 10.1109/SSRR.2017.8088155.

Also Published As

Publication number Publication date
EP4685037A3 (en) 2026-03-18
EP4468125A2 (en) 2024-11-27
EP3880547A1 (en) 2021-09-22
US20220083069A1 (en) 2022-03-17
EP3880547C0 (en) 2024-10-09
EP3881155A4 (en) 2023-01-18
EP4685037A2 (en) 2026-01-28
WO2020142550A1 (en) 2020-07-09
EP3881155B1 (en) 2024-10-23
ES3054960T3 (en) 2026-02-09
WO2020142543A1 (en) 2020-07-09
EP3881025C0 (en) 2025-09-10
US11886182B2 (en) 2024-01-30
US20250028317A1 (en) 2025-01-23
EP3881025A4 (en) 2022-08-10
EP3880547A4 (en) 2023-01-11
EP3880547B1 (en) 2024-10-09
EP3881025B1 (en) 2025-09-10
EP3881025A1 (en) 2021-09-22
EP3881155C0 (en) 2024-10-23
EP4471549A3 (en) 2025-03-05
EP3881155A1 (en) 2021-09-22
US20240111303A1 (en) 2024-04-04
US20220083054A1 (en) 2022-03-17
EP4468125A3 (en) 2025-01-22
US12189386B2 (en) 2025-01-07
WO2020142561A1 (en) 2020-07-09
US12124256B2 (en) 2024-10-22
ES2997207T3 (en) 2025-02-14
EP4471549A2 (en) 2024-12-04
ES2995021T3 (en) 2025-02-05
US20220075364A1 (en) 2022-03-10

Similar Documents

Publication Publication Date Title
US8271132B2 (en) System and method for seamless task-directed autonomy for robots
US20250028317A1 (en) Systems and methods of detecting intent of spatial control
US8355818B2 (en) Robots, systems, and methods for hazard evaluation and visualization
US7211980B1 (en) Robotic follow system and method
US7584020B2 (en) Occupancy change detection system and method
US7801644B2 (en) Generic robot architecture
US7587260B2 (en) Autonomous navigation system and method
US8073564B2 (en) Multi-robot control interface
US7668621B2 (en) Robotic guarded motion system and method
US20080009967A1 (en) Robotic Intelligence Kernel
WO2008005660A2 (en) Robotics virtual rail system and method
Galeone Design and test of a multi-agent robotic system for inspection: coordination and integration of a drone and a robotic rover.

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO MICRO (ORIGINAL EVENT CODE: MICR); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: TOMAHAWK ROBOTICS, FLORIDA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUMMER, MATTHEW D.;BOWMAN, WILLIAM S.;FALENDYSZ, ANDREW D.;AND OTHERS;REEL/FRAME:058137/0561

Effective date: 20190102

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

AS Assignment

Owner name: BANK OF AMERICA, N.A., AS THE ADMINISTRATIVE AGENT, NORTH CAROLINA

Free format text: NOTICE OF GRANT OF SECURITY INTEREST IN PATENTS;ASSIGNOR:TOMAHAWK ROBOTICS, INC.;REEL/FRAME:065394/0847

Effective date: 20210219

AS Assignment

Owner name: TOMAHAWK ROBOTICS, INC., FLORIDA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED ON REEL 058137 FRAME 0561. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:SUMMER, MATTHEW D.;BOWMAN, WILLIAM S.;FALENDYSZ, ANDREW D.;AND OTHERS;SIGNING DATES FROM 20231102 TO 20231121;REEL/FRAME:066072/0260

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCF Information on status: patent grant

Free format text: PATENTED CASE