JPH0793088A - Virtual operating device and method - Google Patents

Abstract

(57) [Summary] [Object] An object of the present invention is to enable a lever, a button, a touch screen operation, etc. in a virtual three-dimensional space to be naturally and accurately performed at high speed. [Structure] The present invention provides three operation targets such as levers and buttons.
3 of the presenting means for presenting by forming a virtual three-dimensional space using the three-dimensional CG, the position detecting means for detecting the positions of the thumb and forefinger, the positions of the thumb and forefinger detected by the position detecting means, and the presenting means It is composed of an interpreting means for interpreting what operation is instructed based on an interpretation rule described based on the relationship between two coordinate positions.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual operation device and method for naturally operating an operation target existing in a space built inside a computer.

[0002]

2. Description of the Related Art In the monitoring and control work of power plants and factories, automation has been promoted by the introduction of a computer, and in a steady state, the operator hardly needs to operate and watch the monitor screen of the monitor or control console. That is becoming the main task. However, in order to prepare for an accident, a training machine is used to provide training in response measures when an accident occurs. The training machine uses the same housing as the actual machine used for actual control and monitoring, and the software inside has been replaced with a simulator that simulates an accident phenomenon. Using this training machine, the operator gives instructions by levers, buttons, etc., to take measures against the accident phenomenon presented by the simulator. As described above, the training machine currently used requires the same manufacturing cost and space for installation as the actual machine.

In order to solve such a problem, there is a method in which an accident phenomenon simulator is operated on a workstation or a personal computer, and a monitor screen of a control console or a monitor console is also simulated on the screen.
In this method, the operation method of giving an instruction by selecting a menu by touching the screen with a touch screen is substituted by selecting the corresponding menu with a pointing device such as a mouse. But,
An operation for giving an instruction to something other than the screen, for example, an operation method of giving an instruction using a lever or button of a desk, is such a two-dimensional GUI (Graphical User).
It cannot be simulated by the method of simulating Interface).

In order to solve the problem caused by the two-dimensional simulation, a three-dimensional computer graphics (CG) is used to construct a virtual three-dimensional space including a virtual trainer inside the computer. A method using a VR (Virtual Reality) technique has been studied. In VR, since the operation target is a three-dimensional space as in reality, an input device that can give three-dimensional position coordinates is used.

As a three-dimensional position input device, only 3
Magnetic sensor type Polhemu that detects only dimensional position
s company's 3 space Isotrack and Ascen
such as Bird of Sion Technology, ultrasonic type sensor of Honeywell, LED
There is an optical type that recognizes the light of. However, since these devices can only detect the three-dimensional position, in order to select a menu or the like, it is necessary to separately provide a button or the like for instructing the selection.

[0006] In order to be able to also perform menu instructions and the like, DataGl that can identify the gesture of the hand
ove (VPL product, glove-shaped input device that measures bending angle of joint with optical fiber) and Cy
berGlove (a product of Crber, a glove type input device that measures the bending angle of a joint by embedding a special resistance). However, these are expensive, and calibration is necessary before use, and the operation of the input device is quite complicated. When assuming a training machine, the positions of the levers and buttons themselves need to be limited within the three-dimensional space, but the operations of the levers and buttons are two-dimensional. Therefore, the above-mentioned Dat
3 like eGlove and CyberGlove
It is overspec to give a dimensional gesture. In other words, these devices are complicated, but from the viewpoint of actually operating a machine such as a training machine, there are restrictions such as response time and recognition accuracy of gestures, and it is possible to operate levers and buttons accurately. The cost performance is very poor.

[0007]

As described above, in setting a training machine or the like, restrictions such as various precisions, operability,
Many problems such as cost were left. The present invention has been made in view of such a problem, and its purpose is to provide a virtual operation device and method capable of performing a lever, a button, a touch screen operation, etc. in a virtual three-dimensional space naturally and at high speed. Is intended to provide.

[0008]

According to the present invention, operation objects such as levers and buttons are presented by constructing a virtual three-dimensional space using a three-dimensional CG, and the positions of the thumb and forefinger are detected and detected. A virtual operation device and method for interpreting what operation is instructed in a book according to the described interpretation regulation, based on the three coordinate positional relationships between the positions of the thumb and forefinger described above and the position of the presented operation target.

Further, when used as a virtual training machine, an accident phenomenon and the internal state of the training target system accompanying it are simulated, and the operating range of the operating object such as a lever or button is determined when operating it. Since these are the attributes of operations that can be performed in advance, three-dimensional CG
The point is to perform information management such as storing the data together with the data, transmitting the interpreted operation instruction, and presenting the calculation result of the simulator.

Further, in order to improve the efficiency and accuracy of the interpretation, the interpretation regulation is described based on the detected positions of the thumb and forefinger, the three positions of the presenting position and the stored operation attribute, and the operation instruction is given. The point of interpreting is improved and added.

[0011]

In the present description, the case where the two-dimensional operation in the three-dimensional space such as the operation of the lever, the button, and the touch screen is performed in the virtual three-dimensional space will be described. The operator's gesture instruction is interpreted based on the regulation described based on the positional relationship between the position of the finger of the book and the presentation position.

In order to improve accuracy, the movement of an operation target such as a lever, a button, or a touch screen is limited, and the interpretation means interprets the gesture instruction of the operator based on the interpretation rule including the attribute thereof. .

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The operation of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic configuration diagram of an embodiment of the present invention. In FIG. 1, for measuring a three-dimensional position, there are, for example, a position source 7 that generates a magnetic field and a position sensor 6 that detects a magnetic field generated from the position source 7 to detect a three-dimensional position. The position sensor 6 is attached to a minimum finger that performs a necessary operation when operating a lever, a button, a touch screen, a slider, or the like, here, a thumb and an index finger, as shown in FIG. The position detected by the position sensor 6 is transmitted by, for example, a small transmitter 8 attached to the lower portion of the position sensor 6. Although the wireless transmission is performed in the embodiment, it is not necessarily limited to this. 3space Isotr from Polhemus
It may be of a type that transmits a signal detected by wire such as ack or Bird of Ascension Technology. The operator operates a lever, a button, a touch screen, a slider, etc., which are operation targets presented on the monitor of the presentation unit 5, and operates them.

The data transmitted by the transmitter 8 is, for example, as shown in FIG.
It has a format like. When the receiving unit 2 receives this data, it sends it to the information managing unit 1. The data in FIG. 3 includes sensor numbers (1, 2) for identifying from which position sensor the data is transmitted, and coordinate values (x1t,
ylt, zlt) and the angle around each axis roll
It consists of l, pitch, and yaw (αlt, βlt, γlt). The subscript lt indicates that it is the value at time t of the sensor number.

The information management unit 1 synchronizes the position source 7 and the position sensor 6 if necessary.
If synchronization is necessary, the information management unit 1 sends the calibrated data in synchronization with the operation analysis unit 3 together with an instruction to execute analysis. The behavior analysis unit 3 analyzes the behavior based on the rules described in the behavior description storage unit 4. The operation storage unit 4 describes rules as shown in FIGS. 4B to 6.

Rule 1 is a rule for interpreting a gesture (Grasp) in which a finger grips a slider or a lever. Rule 1 grabs the value of the gesture h gesture when the following three conditions are satisfied (Gras).
p). Condition 1: Lt ≦ Δ1 The distance Lt between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) is smaller than a certain value Δ1. That is, the thumb and the index finger are sufficiently close to each other, and the distance Lt between the sensor 1 and the sensor 2 is the square value of the square sum of the x, y, and z coordinate positions of the position sensor as shown in FIG. Is. Condition 2: h gesture = Grasp & h gesture = Slide Not in a state where the previous gesture is grasped (Grasp) or in a state where the slider or the like is moving (Slide) (that is, the gesture just before grasps or operates an object). Condition 3: Y2t> Ylt The y coordinate of the sensor 2 is larger than the y coordinate of the sensor 1 (that is, the index finger is above the thumb).

The regulation 2 is a rule for interpreting a gesture (Release) in which the finger is released from the slider or lever that has been grasped. Rule 2 sets the value of gesture h gesture to Release when the following two conditions are met: Condition 1: Lt> 2Δ1 A certain value 2Δ1 of the distance Lt between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) Larger (that is, at least twice the distance at which the thumb and forefinger are judged to be sufficiently close to each other.) Condition 2: h
gesture == Grasp h gesture == Slide The state in which the previous gesture is grasped (Grasp) or the state in which the slider or the like is moving (Slide) (that is, the previous gesture is grabbing or operating an object). Being in a state).

Rule 3 is a rule for interpreting a gesture in which a finger grips a slider or a lever to move and operate. Rule 5 sets the value of gesture h gesture to manipulate (Slide) when the following two conditions are met. Condition 1: Lt> Δ1 The distance Lt between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) is smaller than a certain value Δ1. (That is, the thumb and the index finger are sufficiently close to each other), Condition 2: h gesture == Grasp h gesture == Slide In a state where the previous gesture is grasped (Grasp) or a slider is moved (Slide). There is something (that is, the gesture until immediately before is grasping or operating an object).

Rule 4 is a rule for setting the initial value (Neutral) after the finger releases the lever or slider (Release). Rule 4 sets the value of the gesture h gesture to an initial value (Nuetral) when the following two conditions are satisfied. Condition 1: Lt> Δ2 The distance L _t between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) is from a certain value Δ ₂ . Large (that is, the thumb and the index finger are sufficiently separated from each other) Condition 2: h gesture == Release A state in which the previous gesture is released (that is, a state in which the previous gesture is released). That is,

Rule 5 is a rule for interpreting a gesture indicating a direction. Rule 5 is a gesture h ge when the following three conditions are satisfied:
Set the value of the point to (Point). Condition 1: Lt> 2Δ1 A certain value Δ2 of the distance L _t between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) Greater than (that is, at least twice as large as the distance at which the thumb and forefinger are judged to be sufficiently close to each other), Condition 2: | ion θ | ≦ Δ1 The angle θ between the thumb and forefinger is close to 90 degrees. That, si
As for the value of nθ, the angles α, β and γ around the x, y and z axes detected by the position sensor as shown in FIG.
Convert to n qi, and obtain the normal vector di from this quatanion. The sum of the inner products of this normal vector is s
ion θ. Condition 3: y2t> y1t The y coordinate of the sensor 2 is larger than the y coordinate of the sensor 1 (that is, the index finger is above the thumb).

Rule 6 is a rule for returning a gesture (Point) indicating a direction to an initial state (Neutral). Rule 6 sets the value of the gesture h gesture to an initial value (Neutral) when the following two conditions are satisfied. Condition 1: Lt> 2Δ1 A certain value 2Δ1 of the distance Lt between the sensor 1 (in this case, the position sensor attached to the thumb) and the sensor 2 (in this case, the position sensor attached to the index finger) It is smaller and larger (that is, it is closer than twice the distance at which the thumb and forefinger are judged to be sufficiently close to each other), Condition 2: h gesture == Point, and the previous state indicates (Point). is there.

Rule 7 is a rule for interpreting a gesture (Touch) touching a button or a screen. Rule 7 is h hest when the following two conditions are met:
Set ure to Touch. Condition 1: L2t> Δ1 The distance L2t between the sensor 2 (in this case, the position sensor attached to the index finger) and the display is sufficiently smaller than a certain value Δ1 (that is, the index finger is very close to the display). L2t is calculated from the x, y, z coordinates of the index finger and the x, y, z coordinates of the display. Condition 2: h gesture == Point h gesture == Touch The previous state indicates (Point) or touches (Tou).
ch) (that is, approaching or touching the display with a pointing gesture).

Rule 8 is a rule for returning from a gesture (Touch) touching a button or a screen to an initial value (Neutral). Rule 8 sets hgesture to the initial value (Neut) when two conditions are satisfied.
ral). Condition 1: L2t> 2Δ1 The distance L2t between the sensor 2 (in this case, the position sensor attached to the index finger) and the display is a certain value 2Δ1
It is sufficiently larger (that is, the index finger is separated from the display), Condition 2: h gesture == Touch, and the previous state is touch (Touch) (that is,
Being in the touched state).

Regarding the gesture to be interpreted, Grasp, Slide, Point, To shown in FIGS. 4B to 6 are used.
Other than uch and Release, it is possible to describe the shapes of both the thumb and the index finger. The outer shape storage unit 9 stores three-dimensional outer shape data such as a lever, a button, a touch screen, and a slider presented to the presentation unit 5.

Next, the operation of this embodiment having the configuration shown in FIG. 1 will be described. FIG. 7 outlines a flow chart of the operation of the embodiment of FIG. The information management unit 1 starts the analysis in response to the instruction to start the operation analysis.
The receiving unit 2 receives such position data as described above (step a). The information management unit 1 comprises a position data buffer and a presentation control unit as shown in FIG. 1, and stores the position data in the position data buffer (step b). As one of the initialization processing of the position data buffer, a parameter h gesture storing a gesture of the hand is set to Neutral (a value indicating a state in which the hand does not make a special gesture) (step c). Next, the operation analysis unit 3 reads the position data stored in the position data buffer in the information management unit 1 (step d). Based on the read current position data and the current hand gesture value h gesture, it is checked whether or not there is one that matches the rules of FIGS. 4B to 6 stored in the behavioral description storage unit 4 (step e). If there is, the value of hgesture is updated based on the rule described in the behavioral description storage unit 4. If there is no match, the value of hgesture is unchanged. Hgestu analyzed by the motion analysis unit 3
a slider or a button presented to the presentation unit 5 by the presentation control unit in the information management unit 1 based on re and the position data;
The position of the lever or the like is updated according to the operation content (step f). The shapes (characters) of sliders, buttons, levers, etc. are read from the outer shape storage unit 9 by the presentation control unit and presented to the presentation unit 5.

As described above, according to the present embodiment, the gesture is interpreted from the relationship between the position of the index finger and the thumb and the position of the display, so that the operation can be performed with a small equipment and without a feeling of strangeness.

Next, FIG. 8 is a diagram showing a schematic configuration diagram of another embodiment of the present invention. An operation attribute storage unit 10 for storing operation attributes (a lever that moves up and down, a lever that moves back and forth, etc.) in addition to the three-dimensional shape of the target presented to the presentation unit 5 in the first embodiment, Input unit 1 for presenting operation attributes
1 and the simulation unit 12 are added. The simulation unit 12 simulates the internal processing of the operation target stored in the outer shape storage unit 9.

In the first embodiment, since the operation attributes such as levers and buttons are not limited, when the operation target of the presentation unit 5 is moved by the gesture interpreted by the positions of the forefinger and the thumb, the detection error of the position sensor is detected. For example, a lever that should only move up and down may move diagonally. Therefore, in the second embodiment, the attribute of the operation to be operated is set in advance to cover such a position detection error and the like so that an accurate operation can be provided.

FIG. 9 shows an example of a screen when an attribute is added to each operation target. The upper right shows the menu A of the outer shape of the operation target. The user selects a shape from these. Here, a large rectangle rect120 and a small rectangle rect
Three shapes of 221, arrow a11ow122 are shown as an example.

Input a desired item from menu A on the upper right
Select with one pointing device (eg mouse) and place it in the appropriate position B in the lower right. By repeating such an operation, the operation targets to be presented on the presentation unit 5 are arranged.

Next, an operation attribute is given to the arranged object. As shown in FIG. 10, the operation attribute assignment is switched to the new attribute assignment by the "N" key, and the target is displayed by selecting the target on the screen B in the lower right of FIG. 9 with the mouse. The attributes that can be specified are, for example, as shown in FIG.
There are types of attributes, here lever 0, button 1 and panel 2.

With respect to the lever, the rotary shaft, the number of gears, the range of gear positions, the center of the gear position, and the type of sound generated when the gear is engaged can be specified. For the button, the coordinate position when on, color, sound, and color when off can be specified. For the panel, you can specify the color when it is on / off.

As a result of the designation in this way, the attribute data type is added to the operation attribute storage unit 10 in addition to the type of each shape data as shown in FIG. 11, for example. The actually filled contents have a format as shown in FIG. 12, for example.

Not only is attribute data newly added,
As shown in FIG. 10, deletion, copy, and reference can also be made. This result is written to the file by the write instruction "W" key to the file.

Not only the operation attribute but also the parent-child relationship of the operation target whose operation result is related to another button can be specified in addition to the operation attribute. Here, as shown in FIG. 10, a parent can have any number of children (that is, any number of subordinates can change), but a child can have only one parent (one parent affects the operation. ). This parent-child relationship is stored in the operation attribute storage unit 10 in the format shown in FIG. Further, the current parent-child relationship of the lower left screen C is shown as shown in FIG. 9 while the operation attributes are being given.

That is, the system used as a virtual training machine has a simulator section 12 for simulating an accident phenomenon and the internal state of the training target system accompanying it, and operation targets such as levers and buttons are set here. Since the movable range is determined at the time of operation, these are stored in the operation attribute storage means together with the three-dimensional CG data to be presented to the presentation unit 5 as attributes of possible operations in advance. The information management unit transmits the operation instruction interpreted by the motion analysis unit 3 to the simulator unit 12 and presents the calculation result of the simulator to the presentation unit 5 to perform information management. Furthermore, the behavioral description storage unit 4 stores the detected positions of the thumb and forefinger, the three positions of the presentation unit, and the operation attributes stored in the operation attribute storage unit 10 in order to improve the efficiency and accuracy of the interpretation. It is based on the description of the rules and is improved to interpret the operation instructions.

As described above, according to this embodiment, since the operation attributes such as the range of movement of the lever and the position are limited, the information management unit 1 analyzes the operation analysis unit 3 so as not to deviate from this range. Since a reliable operation can be executed by the gesture and the coordinate value from the position sensor 6, its effect is great.

[0038]

According to the present invention, only two of the thumb and the index finger are used.
Since it only detects the position of the finger of the book, it is lightweight even if equipped with equipment, and natural and accurate operation can be performed inexpensively without any discomfort, so its effect is great.

[Brief description of drawings]

FIG. 1 is a schematic configuration diagram of an embodiment.

FIG. 2 is a diagram showing an operating method.

FIG. 3 is a diagram showing an example of position data received by a receiving unit.

FIG. 4 is a diagram showing an example of a behavioral description in a behavioral description storage unit.

FIG. 5 is a diagram showing an example of a behavioral description in a behavioral description storage unit.

FIG. 6 is a diagram showing an example of a behavioral description in a behavioral description storage unit.

FIG. 7 is a diagram showing an operation flow.

FIG. 8 is a schematic configuration diagram of another embodiment.

FIG. 9 is a diagram showing an example of a screen.

FIG. 10 is a diagram showing an input example.

FIG. 11 is a diagram showing an example of storage in an operation attribute storage unit.

FIG. 12 is a diagram showing a storage example of attribute data in an operation attribute storage unit.

FIG. 13 is a diagram showing a storage example of a parent-child relationship in an operation attribute storage unit.

[Explanation of symbols]

1 ... Information management unit, 2 ... Reception unit, 3 ... Motion analysis unit,
4 ... Behavior description storage unit, 5 ... Presentation unit, 6 ... Position sensor,
7 ... Position source, 8 ... Transmission unit, 9 ... Outline storage unit,
10 ... Operation attribute storage unit, 11 ... Input unit

Claims (3)

[Claims] 1. An operation target presented to a presenting means by presenting an operation target to a presenting means, detecting the position of a finger of an operator, and presenting the operation target presented to the presenting means based on the detected position of the finger and the position of the presenting means. A virtual operation method characterized by interpreting an operation instruction for a. 2. A presenting means for presenting an operation target in a virtual three-dimensional space by using a three-dimensional CG, a position detecting means for detecting the positions of the thumb and the forefinger, a thumb detected by the position detecting means, and a forefinger. A virtual operation device characterized by fairing from a position and an interpreting device that interprets an operation instruction based on a rule described in advance based on a relationship between three coordinate positions of a position of a presenting device. 3. Presenting means for presenting an operation target, simulator means for simulating processing of a knife as an operation target presented to the presenting means, and presenting an operation attribute for each operation target to the presenting means. Operation attribute storage means for storing together with graphic data for operation, operation attribute detection means for detecting the position of a finger, the position of the finger detected by the position detection means, the position of the presenting means and the operation attribute storage Interpreting means for interpreting the operation instruction for the presenting means based on the operation attribute stored in the means, and transmitting the operation instruction interpreted by the interpreting means to the simulator means, and presenting the result to the presenting means A virtual operation device, comprising: an information management unit that controls the operation.