JPH11288342A - Interface device and method for multimodal input / output device - Google Patents

Abstract

(57) [Summary] [Problem] To provide an interface device of a multi-modal input / output device that allows a user to freely input and is adaptable to the surrounding environment. A gaze detection engine 101 based on image input, a voice recognition engine 102 based on voice input, an operation input unit 103 including a mouse / keyboard, etc., and inputs from the above 101 to 103 are integrated to detect a user's intention. Input integration unit 104
A feedback generation unit 105 that outputs to a user based on the intention detection result, and detects a user's intention to select an icon based on the information of 101 to 103, In order to inform the user of the recognition of the selection instruction for the corresponding icon based on the detection result, and to return the required presentation to the user, the intention is detected using causal relationship information between input information. At the same time, the causal relationship information is statistically learned through actual input / output with the user.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimodal input / output that estimates a user's intention through at least one input or output of visual information, voice information, and operation information that is input, and that returns feedback to the user based on the input or output. Regarding the interface.

[0002]

2. Description of the Related Art In recent years, in a computer system including a personal computer, multimedia information such as voice information and image information has been input and output in addition to the conventional input by a keyboard and a mouse and the output of characters and image information by a display. It is becoming possible to do so. In addition to these circumstances, natural language analysis, natural language generation, and advances in speech recognition, speech synthesis, and dialog processing technologies have led to a growing demand for speech dialogue systems that allow users to interact with speech input and output. , "TOSBURG, a dialogue system that can be used by voice input using free speech
-II "(IEICE, Vol.J77-D-II, No.8, pp 1417-1428, 1994)
Various speech dialogue systems have been developed.

Further, in addition to the voice input / output described above, for example, visual information input using a camera is used,
Alternatively, information that can be exchanged with the user through various input / output devices such as a touch panel, a pen, a tablet, a data glove, a foot switch, an interpersonal sensor, a head-mounted display, a force display (force-providing device), and the like. There is an increasing demand for a multi-modal dialogue system for interacting with a user by using the Internet.

This multi-modal interface does not use only one medium (channel), such as voice, for communication in human-to-human communication, but exchanges information through various media such as gestures, hand gestures, and facial expressions. It interacts naturally with non-verbal messages to create a natural and smooth interaction ("Intelligent Multimedia
Interfaces ", MayburyM.T, Eds., The AAAI Press / The MI
(T Press, 1993) Considering this, expectations are growing as one of the leading methods to realize a natural and easy-to-use human interface.

[0005]

However, since the analysis accuracy of the input from each medium and the characteristics of each input / output medium have not been clarified,
A multi-modal interface that efficiently uses each newly available input / output medium or a plurality of input / output media, is highly efficient, effective, and reduces the burden on the user has not been realized.

[0006] In particular, when integrating inputs from respective recognizing devices, such as images and sounds, the likelihood of information sent from predetermined recognizing means is assumed in advance, and a predetermined input order is assumed. Had been done. Therefore, when the accuracy of a certain recognition device is reduced due to a change in a certain surrounding environment, the interpreting unit is stagnated without performing the corresponding input interpretation process,
There is a problem that malfunctions easily occur. In addition, it is not possible to cope with the input order peculiar to the user, and it is necessary for the user to learn input means that can be accepted on the system side, which greatly reduces convenience.

Accordingly, an object of the present invention is to provide a multimodal input / output device interface device and a method thereof which allow a user more free input and can adapt to changes in the surrounding environment. To provide an output method.

[0008]

According to the first aspect of the present invention, various operation objects having predetermined processing contents are prepared in advance,
A user's selection instruction for these operation targets is obtained by recognizing at least one or more of the user's gaze input information, voice input information, operation input information, image input information, and operation input information. Based on the result, the user is required to detect the intention and, based on the detection result, return a required presentation to the user in order to notify that the user has recognized the selection instruction for the corresponding operation target. An interface device for a modal input / output device, wherein the intention is detected using causal relationship information between input information, and the causal relationship information is statistically learned through actual input / output with a user. An interface device for a multi-modal input / output device, wherein the interface device is configured to be implemented by integrating means.

According to a second aspect of the present invention, in the interface device of the multimodal input / output device according to the first aspect, the integrating means holds time management means for managing time and input information at present and past times. And an interface device for a multi-modal input / output device, wherein the causal relationship information uses current and past input information held by the holding device.

According to a third aspect of the present invention, in the interface device of the multimodal input / output device according to any one of the first and second aspects, the integrating means includes a causal relationship between input information or between input information and input time. An interface device of a multi-modal input / output device characterized by learning information statistically through actual input / output with a user.

According to a fourth aspect of the present invention, various operation objects having predetermined processing contents are prepared in advance, and a user's selection instruction for these operation objects is provided by the user's line-of-sight input information, voice input information, and operation information. Based on information obtained by recognizing at least one or more of the input information, the image input information, and the operation input information, the intention of the user is detected,
In the interface device of the multi-modal input / output device configured to return a required presentation to a user in order to notify that the selection instruction for the corresponding operation target has been recognized based on the detection result, An interface device for a multimodal input / output device, which is provided independently for each target.

According to a fifth aspect of the present invention, in the interface device for a multi-modal input / output device according to the third aspect, the integrating means determines the intention detection result in the specific operation target by using a positional / An interface device of a multi-modal input / output device, which is determined based on a shape, color, and language relationship.

According to a sixth aspect of the present invention, in the interface device of the multi-modal input / output device according to the third aspect, the integrating means detects, with respect to the specific operation target, its own intention detection result and the intention detection of another operation target group. An interface device of a multimodal input / output device characterized by changing a positional, a shape, a color, and a linguistic relationship with another operation target group based on a result.

According to a seventh aspect of the present invention, in the interface device of the multimodal input / output device according to any one of the first and second aspects, the integrating means converts the intention information into the intention information at a past time or using the intention information. An interface device for a multi-modal input / output device, which is obtained from a required presentation result returned to a user.

According to the invention of claim 8, in the interface device of the multimodal input / output device according to claim 1,
An interface device for a multi-modal input / output device, characterized in that learning of causal relationship information is started or ended based on intention information obtained by the integration means.

According to a ninth aspect of the present invention, in the interface device of the multimodal input / output device according to the first aspect,
The output for returning the required presentation to the user by the operation target is performed at the time of the learning, and is an interface device of a multimodal input / output device.

According to a tenth aspect of the present invention, in the interface device of the multimodal input / output device according to the first aspect, the integrating means confirms or cancels at least one of the various input information with an intention detection result. An interface device of a multi-modal input / output device, which is used as information for a multi-modal input / output device.

According to an eleventh aspect of the present invention, various operation objects having predetermined processing contents are prepared in advance, and a user's selection instruction for these operation objects is provided by user's line-of-sight input information,
Based on information obtained by recognizing at least one of the voice input information, the operation input information, the image input information, and the operation input information, a user's intention is detected, and the detection result is included in the detection result. An interface method for a multi-modal input / output device configured to return a required presentation to a user in order to notify that a selection instruction for the corresponding operation target has been recognized based on the input information. An interface method for a multi-modal input / output device, wherein the method is implemented by using causal relationship information between the devices and learning the causal relationship information statistically through actual input / output with a user.

According to a twelfth aspect of the present invention, in the interface method of the multimodal input / output device according to the seventh aspect,
An interface method for a multi-modal input / output device, wherein the learning is performed by statistically performing causal relationship information between input information or between input information and input time through actual input / output with a user. It is.

[0020]

Embodiments of the present invention will be described below with reference to the drawings.

[0021] First Embodiment FIG. 1 is is a general block diagram of a system according to the first embodiment of the present invention.

For each operation target of this system, a gaze detection engine 101 based on image input, a voice recognition engine 102 based on voice input, an operation input unit 103 including a mouse / keyboard, etc. It has an input integration unit 104 that integrates inputs and detects the intention of the user, and a feedback generation unit 105 that outputs to the user based on the result of intention detection.

In this embodiment, the window system is targeted, and the operation target is one of the icons shown in FIG. The intention in the present embodiment is a user's intention to select an operation target (icon).

(Input unit) The input units 101 to 103 send information obtained by converting the operation target and input information into similarity to the input integration unit 104.

For example, the line-of-sight detection engine 101 analyzes the input face image of the user, expresses the degree to which the line of sight is directed to the icon of the user by a similarity of 0 to 1, and displays the input integration unit 104
Send to

Similarly, for the speech recognition engine, the degree to which the speech recognition engine refers to itself (the name, shape, color, positional relationship, etc. of the icon) is set to 0 based on the similarity between the input speech and the vocabulary set holding it. Input similarity
Send to 104. As for the operation information, information of the similarity 1 is sent to the input integration unit 104 when the operation is performed (selected), and otherwise, information of the similarity 0 is sent.

The eye-gaze detection engine 101 includes, for example, an eye tracker device for observing the user's eye movements, a head tracker device for detecting the movement of the user's head, a seating sensor, and, for example, Japanese Patent Application Laid-Open No. 08-059071. Gaze point estimation device and its method '', by processing the image information obtained from the camera that observes the user or the camera worn by the user to detect the user's line of sight, etc. The degree of gaze at each operation target is represented as similarity information.

The speech recognition engine has means for receiving the user's speech information as input and outputting the degree of conformity with the vocabulary to be recognized. For this purpose, for example, "TOSBURG-II" (IEEE, Vol.
D-II, No. 8, pp 1417-1428, 1994). From this matching degree information, the degree of similarity to the selection target can be obtained, for example, by the following equation.

[0029]

(Equation 1) By defining a vocabulary indicating a name, a shape, a color, and a position for each selection target icon, similarity information can be obtained by using the above equation (1).

(Operation Input) With respect to the operation input 103, the destination of the event given by the keyboard / mouse is sent to the input integration unit 104 as the input of the degree of similarity 1 to the operation target.

(Input Integration Unit) Next, the operation of the input integration unit 104 will be described.

The input integration unit 104 obtains the probability of being selected from each input information. In addition, through learning, more accurate intention selection is performed in accordance with the user and the environment.

For example, the following method can be used.

As shown in FIG. 3, a causal relationship between input information is expressed using a table.

In FIG. 3, frequency information indicates the causal relationship between the user's selection intention and the similarity information (gaze similarity) obtained from the gaze detection engine and the similarity information (speech similarity) obtained from the speech recognition engine. As hold.

In this embodiment, the degree of similarity is divided between 0 and 1 (this is not always necessary).

FIG. 3 expresses the relationship between the two based on a model in which the selection intention is expressed as the cause of the gaze similarity.

[0038] Positive,
A case where there is no selection intention is called Negative, and the distribution of gaze similarity in each case is expressed as frequency information.

FIG. 4 shows the relationship between the three persons based on a model in which the line-of-sight similarity and the selection intention cause the voice similarity, and shows the voice similarity when the selection intention and the line-of-sight similarity are given. Degree distribution information.

The input integration unit 104 receives each input information at fixed time intervals, and obtains a probability when there is a selection intention using the table of FIG. For that purpose, the following equation 1 is used. Here, G and S indicate the values of the visual line similarity and the voice similarity obtained from the input information, respectively.

[0041]

(Equation 2) Since the values of G and S are obtained from the input information in the above equation 2, FIG.
The selection probability can be obtained by using the table. For example,
In the case of gaze similarity 0.85 and audio similarity 0.73,

(Equation 3) And the probability of the selection intention is determined to be 1.

Next, a method of learning causal relationship information will be described.

When the user gives operation information with a similarity of 1 using a keyboard / mouse or the like, the information integration unit immediately sets the probability of the selection intention to 1 for the operation target and sets the line-of-sight similarity at that time. The similarity distribution information in FIGS. 3 and 4 is updated based on the values of the degrees and the voice similarities.

That is, 1 is added to the corresponding item in FIGS.

For example, assuming that the line-of-sight similarity and the voice similarity when the selection is made with the mouse are 0.65 and 0.87, respectively, in FIG. 3, the line-of-sight similarity = 0.6 to 0.7 and the selection = Posi
In the column of “active”, 1 is added to the columns of gaze similarity = 0.6 to 0.7, selection = Positive, and voice similarity = 0.8 to 0.9 in FIG.

For the operation object to which the similarity 1 is not given at that time, the probability of the selection intention is immediately set to 0, and based on the values of the visual line similarity and the voice similarity at that time, The similarity distribution information in FIGS. 3 and 4 is similarly updated. However, for the corresponding gaze similarity and voice similarity, selection = N
1 will be added to the column of “egative”.

As described above, it is possible to dynamically perform learning so as to make the selection intention clear, adapting to the user / environment.

(Feedback Generation Unit) Next, the operation of the feedback generation unit 105 will be described.

The feedback generation unit 105 determines the feedback that the selection target icon gives to the user based on the selection intention probability sent from the input integration unit 104. This can be performed, for example, by referring to the utility table shown in FIG.

In the present embodiment, two-stage feedback is assumed (more feedback may be assumed), but FIG. 5 describes the utility value of the feedback operation.

The expected utility value of each feedback operation is calculated based on this table. The formula for calculating the expected utility value is
Assuming that the selection will probability obtained from the input integration unit 104 is x, it can be expressed by the following equation 2.

[0052]

(Equation 4) Based on the above equation 3, the feedback operation n having the largest expected utility value is obtained and executed. For example, the selection intention probability is 0.6
Is obtained, the expected utility value of each feedback operation is calculated as follows using the table of FIG.

Expected utility value (selection feedback) = 0.6 ×
1.0+ (1-0.6) × 0 = 0.6 Expected utility value (selection candidate feedback) = 0.6 × 0.7 + (1-
0.6) × 0.6 = 0.66 Expected utility value (no feedback) = 0.6 × 0 + (1-0.6) ×
1.0 = 0.4 In this case, the selection candidate feedback that maximizes the expected utility is determined.

In the feedback operation shown in FIG. 5, selection feedback and selection candidate feedback are prepared. The actual feedback operation is realized by changing the brightness, size, shape of the icon on the window system, or outputting sound.

Further, the feedback generation unit 105 can send prediction information to the gaze detection engine / speech recognition engine prior to generation of feedback. The prediction information in this case is that the gaze is directed toward the selection target when the selection target is feedback, or that the selection target is referred to (name, location, etc.). Each input unit performs processing such as switching the processing contents and the processing data set during each recognition processing based on the prediction information.

In the system configured as described above, the causal relationship between the modalities such as the line of sight and the voice is learned, and the intention is detected based on the learned result. Each input unit also performs a process according to the predicted behavior of the user. This makes it possible to easily configure an interface that dynamically adapts to the user and the environment.

(Modification 1) In this embodiment, the selection of the icon is set as the intention of the user. However, the present invention is not limited to this, and the selection of all the objects and the execution of the command are respectively performed. However, it can be similarly realized by adopting a system configuration as shown in FIG.

(Modification 2) In FIGS. 3 and 4, a table is used for similarity distribution information. However, the present invention is not limited to this, and a distribution having continuous values like a function expression may be used. It may be assumed.

(Modification 3) In the present embodiment, the learning of the causal relationship information is realized by updating the similarity distribution information shown in FIGS. 3 and 4. However, the learning is not limited to this. A means such as another table may be used for storing the learning result.

In this case, the selection intention is calculated based on the conventional similarity distribution information and the similarity distribution information obtained as a result of learning.

This can be realized, for example, by calculating the selection intention probability based on the result of adding the same item with the similarity distribution information of both. Such a learning result may be stored for each user.

(Fourth Modification) In this embodiment, visual line detection, voice recognition, and operation input (using a mouse / keyboard) are used as input information. However, the present invention is not limited to this.

The other input information can be processed by constructing a table relating to the causal relationship between the input information as shown in FIGS.

(Modification 5) In the present embodiment,
We use the principle of maximum expected utility to determine the feedback method, but this is not necessarily so. Other decision rules, such as the maximin criterion, may be used.

(Modification 6) In this embodiment, the input integration unit 104 uses the input information of the current time. However, the present invention is not limited to this.

The input information at the past time may be used. In this case, the gaze similarity / speech similarity at the past time is stored, and the similarity at the past time stored in the tables of FIGS. FIGS. 3 and 4 show the causal relationship between the similarity between the current time and the past time.
Can be expressed in the form of a table.

Second Embodiment Next, a second embodiment will be described.

FIG. 6 is an overall block diagram of the system according to the second embodiment.

The first operation target of this system is:
A gaze detection engine 5001 based on image input, a voice recognition engine 5002 based on voice input, an operation input unit 5003 including a mouse and a keyboard, and an input integration unit that integrates inputs from the above 5001 to 5003 and detects a user's intention. 5004, and a feedback generation unit 5005 that outputs to the user based on the intention detection result.

The second and subsequent operation objects are 5101 to 511, respectively.
05, 5201-5205 have similar units. The input integration unit and the feedback generation unit of each operation object are connected to each other, and can exchange information.

In this embodiment, the window system is targeted, and the operation target is one of the icons shown in FIG. The intention in the present embodiment is a user's intention to select an operation target.

(Input unit) Input units 5001 to 5003, 5101 to 5103
In this case, information obtained by converting the operation target and the input information into the similarity is sent to the input integration units 5004 and 5104 in the same manner as in the first embodiment.

(Input Integration Unit) The input integration units 5004, 5104, etc., determine the probability that they are selected from each input information. In addition, through learning, more accurate intention selection is performed in accordance with the user and the environment.

For this, the following method can be used, for example.

As shown in FIGS. 7 to 10, a causal relationship between input information is expressed using a table.

7 to 10 show the user's intention to select, similarity information (gaze similarity) obtained from the gaze detection engine, similarity information (speech similarity) obtained from the speech recognition engine, and the average between the icons. The causal relationship of the distance is stored as frequency information. In this embodiment, the similarity is 0
This is divided into 1 (this is not always necessary).

FIGS. 7 and 9 show the relationship between the two based on a model in which the selection intention is expressed as the cause of the gaze similarity. Positive,
The case where there is no gaze is called Negative, and the distribution of the gaze similarity in each case is expressed as frequency information.

FIGS. 8 and 11 show the relationship between the three persons based on a model in which the line-of-sight similarity and the selection intention cause the voice similarity. In the case where the selection intention and the line-of-sight similarity are given. Is distribution information of speech similarity.

In each of FIGS. 7, 8, 9 and 10, distribution information is given for each case of 5, 10 pixels based on the model that the average distance between icons is the cause.

The input integration unit receives each input information at a fixed time. Further, an average distance between the icons to be selected is obtained by exchanging information between the objects to be selected, and a closer value among 5, 10 is adopted.

From these values, the probability in the case where there is a selection intention is obtained using the tables of FIGS. Equation 4 below is used for that purpose. Here, G and S indicate the values of the line-of-sight similarity and the voice similarity obtained from the input information, respectively, as in Expression 1. D indicates the distance between icons.

[0082]

(Equation 5) Next, a method of learning causal relationship information will be described.

When the user gives operation information with a similarity of 1 using a keyboard / mouse or the like, the information integration unit immediately sets the probability of the selection intention to 1 for the operation target and sets the line-of-sight similarity at that time. The similarity distribution information in FIGS. 7 to 10 is updated based on the values of the degree, the voice similarity, and the distance between the icons. That is, 1 is added to the corresponding item in FIGS. For example, assuming that the line-of-sight similarity and the voice similarity at the time of selection by the mouse are 0.65 and 0.87, respectively, and the distance between the icons is 5 pixels, the line-of-sight similarity = 0.6 to 0 in FIG.
7, 1 is added to the column of selection = Positive, and in FIG. 8, 1 is added to the column of gaze similarity = 0.6 to 0.7, selection = Positive, and voice similarity = 0.8 to 0.9.

For the operation object to which the similarity 1 was not given at that time, the probability of the selection intention is immediately set to 0, and the values of the visual line similarity, the voice similarity, and the distance between the icons at that time are changed. Based on this, the similarity distribution information in FIGS. 3 and 4 is similarly updated. However, 1 is added to the column of selection = Negative for the corresponding gaze similarity and voice similarity.

As described above, it is possible to perform learning dynamically so as to be able to confirm the intention of selection in accordance with the user / environment.

(Feedback Generation Unit) Next, the operation of the feedback generation units 5005 and 5105 will be described.

In the feedback generation section, the input integration section 50
Based on the selection intention probability transmitted from 04, 5104, or the like, the feedback that the selection target icon gives to the user is determined. This can be performed using, for example, the utility table shown in FIG. 5 and Equation 2 as in the first embodiment.

Further, the feedback generation unit can change the value of the distance between icons based on the obtained expected utility value of each selection target, and can provide feedback to the user. In the present embodiment, the expected utility value of each selection target is obtained assuming a case where the distance between icons is 5, 10 pixels. The average of the expected utility values at 5 and 10 is taken for all the selected objects, and the larger the average is, the higher the expected utility value is, the higher is the relationship between the selected objects. The distance between icons is changed based on the result. At this time, information is exchanged between the feedback generation units to be selected, and the distance between the icons is adjusted to a value close to the specified value. For this, for example, a constraint satisfaction programming technique (see the Artificial Intelligence Society, Vol. 10, No. 3) can be used.

Further, the feedback generation section can send prediction information to the gaze detection engine / speech recognition engine prior to the feedback generation. The prediction information in this case is that the gaze is directed toward the selection target when the selection target is feedback, or that the selection target is referred to (name, location, etc.). Each input unit performs processing such as switching the processing contents and the processing data set during each recognition processing based on the prediction information.

(Modification 1) In this embodiment, the distance between the icons is set as the relationship between the selection targets. However, this is merely an example. A linguistic relationship may be set. Further, the relationship between the selection targets is not limited to the discrete value as in the present embodiment, and may be set to take a continuous value like a function expression.

(Modification 2) The feedback generation unit averages the expected utility values of all the selection targets in order to obtain the optimum relationship between the selection targets. However, the present invention is not limited to this. The expected utility value between the selection targets may be used.

(Modification 3) In the present embodiment, the learning of the causal relationship information is realized by updating the similarity distribution information shown in FIGS. 7 to 10, but is not limited to this. A means such as another table may be used for storing the learning result.

In this case, the selection intention is calculated based on the conventional similarity distribution information and the similarity distribution information obtained as a result of learning.

This can be realized, for example, by calculating the selection intention probability based on the result of adding the same item between the two similarity distribution information items. Such a learning result may be stored for each user or each usage environment. As described above, in the present invention, various modifications can be made without departing from the spirit of the present invention.

Third Embodiment Next, a third embodiment will be described.

FIG. 6 is used for the entire block diagram similarly to the second embodiment. FIG. 6 is an overall block diagram of the system.

The first operation target of this system is:
A gaze detection engine 5001 based on image input, a voice recognition engine 5002 based on voice input, an operation input unit 5003 including a mouse and a keyboard, and an input integration unit that integrates inputs from the 5001 to 5003 and detects an intention of the user 5004, and a feedback generation unit 5005 that outputs to the user based on the intention detection result.

The second and subsequent operation objects are 5101 to 511, respectively.
05, 5201-5205 have similar units.

The input integration unit and the feedback generation unit of each operation object are connected to each other, and can exchange information. In this embodiment, the window system is targeted, and the operation target is one of the icons shown in FIG. The intention in the present embodiment is a user's intention to select an operation target.

(Input unit) Input units 5001 to 5003, 5101 to 5103
In this case, information obtained by converting the operation target and the input information into the similarity is sent to the input integration units 5004 and 5104 in the same manner as in the first embodiment.

(Input Integration Unit) The input integration units 5004, 5104, etc., determine the probability that they are selected from each input information. In addition, through learning, more accurate intention selection is performed in accordance with the user and the environment.

For this, the following method can be used, for example. As in the first embodiment, as shown in FIGS.
The causal relationship between input information is expressed using a table.

In FIGS. 3 and 4, the user's intention of selection, similarity information obtained from the gaze detection engine (gaze similarity), and similarity information obtained from the speech recognition engine (speech similarity) are shown.
Is held as frequency information. In this embodiment, the degree of similarity is divided between 0 and 1 (this is not always necessary).

In FIG. 3, the relationship between the two is expressed based on a model in which the selection intention is expressed as the cause of the gaze similarity. The case where there is a selection intention is called Positive, and the case where there is no intention of selection is called Negative, and the distribution of gaze similarity in each case is expressed as frequency information.

In FIG. 4, the relationship between the three is expressed based on a model in which the line-of-sight similarity and the selection intention cause the voice similarity. Degree distribution information.

The input integration unit receives each input information at fixed time intervals. Further, an average distance between the icons to be selected is obtained by exchanging information between the objects to be selected, and a closer value among 5, 10 is adopted. From these values, the probability in the case where there is a selection intention is obtained using the tables of FIGS. Equation 4 below is used for that purpose. Here, G and S indicate the values of the line-of-sight similarity and the voice similarity obtained from the input information, respectively, as in Expression 1. D indicates the distance between icons.

[0107]

(Equation 6) Here, δ is a real number greater than or equal to 0 and less than 1, and P ₋₁ is a selection intention probability obtained one unit time ago. By using Expression 5, the probability of the selection intention reflects the past selection intention, and it is possible to detect the intention information more smoothly.

In the input integrating units 5004 and 5104, learning similar to that in the first embodiment is performed. At that time, the result of the intention information obtained is used as a trigger for learning start and end.
For example, a threshold value X is provided for the selection intention probability obtained by Expression 5, and P (selection intention = Positive |
The learning is started when (G, S, D) ≧ X, and the selection is ended when P (selection intention = Positive | G, S, D) <X. Accordingly, learning can be performed even when an explicit learning start / end signal using a mouse / keyboard or the like cannot be obtained, and there is an advantage that causal relationship information is more easily obtained.

(Feedback Generation Unit) In the feedback generation units 5005, 5105, etc., feedback is determined in the same manner as in the first embodiment, but this is also performed during learning in the input integration unit. To do.

This is done by obtaining the selection intention probability from the input integration unit based on Equation 5 even during learning, and sequentially sending it to the feedback generation unit. The feedback generation unit determines and implements the feedback upon receiving the selection intention probability. It is feasible. Thus, even during learning, the user can confirm whether his or her intention is correctly reflected in the learning result, and there is an advantage that subsequent operations can be performed more smoothly.

(Modification 1) Here, the input integration units 5004 and 5104
For example, at least one of information obtained from a specific input unit may be used for confirming or canceling the intention detection result.

This can be realized, for example, by the following procedure.

1) A table as shown in FIG. 11 is prepared, and when input information arrives, it is checked whether or not the confirmation similarity condition or the cancellation similarity condition is met.

Here, "mouse right" is input information for setting the similarity to 1 if the right mouse button is clicked with the mouse cursor placed on the operation target, and "mouse left" is the operation target If the left mouse button is clicked with the mouse cursor placed on top, the similarity is set to 1
Is the input information set in.

The confirmation voice is assumed to be the maximum similarity among the recognition results with a specific voice input such as "Yes" or "OK" as input information, and the cancellation voice is a specific voice such as "No" or "No". It is assumed that the maximum similarity among the recognition results with the voice input is input information.

2) If the above conditions are met, a confirmation or cancellation signal is sent to the feedback generator.

3) The feedback generation section performs processing according to the confirmation or cancellation signal.

For example, when the confirmation signal is received, the operation object having the maximum expected utility is determined as the intention of the user, and selection feedback is performed. When the cancellation signal is received, all the operation objects are expected. This can be realized by setting the utility value to 0 and not performing feedback.

With the above-mentioned expansion, the user can directly reflect his / her own intention in the intention detection on the system side, and a more convenient interface can be configured.

(Modification 2) In the present embodiment, the learning in the input integration unit is performed using the similarity information at the current time. However, the present invention is not limited to this, and the past similarity information is used. May be. This can be achieved by providing a buffer in the input integration unit and storing past similarity information.

Here, for example, it is determined whether or not there is a selection intention at a past time based on mouse / keyboard operation information obtained from an input unit or intention information obtained from an input integration unit, and the accumulated past similarity information is determined. It is more possible to reflect them in the causal relationship tables shown in FIGS.

As described above, in the present invention, various modifications can be made without departing from the spirit of the present invention.

[0123]

According to the present invention, free multi-modal input by the user is allowed by dynamic learning of the causal relationship information between the input information from the user, and the environment can be freed. The system can be adapted so that the detection of the user's intention becomes reliable even when it changes. Further, by changing the relationship between the objects to be used, it is possible to configure an environment in which the intention can be more reliably detected.

[Brief description of the drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention.

FIG. 2 is an example of a task used in the first, second, and third embodiments.

FIG. 3 is an example of an input information integration table used in the first and third embodiments.

FIG. 4 is an example of an input information integration table used in the first and third embodiments.

FIG. 5 is an example of a feedback generation table used in the first, second, and third embodiments.

FIG. 6 is a block diagram of the second and third embodiments.

FIG. 7 is an example of an input information integration table used in the second embodiment.

FIG. 8 is an example of an input information integration table used in the second embodiment.

FIG. 9 is an example of an input information integration table used in the second embodiment.

FIG. 10 is an example of an input information integration table used in the second embodiment.

FIG. 11 is an example of a table used for intention confirmation / cancellation processing used in the third embodiment.

[Explanation of symbols]

 101 Eye-gaze detection engine 102 Voice recognition engine 103 Operation input unit 104 Input integration unit 105 Feedback generation unit

Claims (12)

[Claims] 1. Various operation objects having predetermined processing contents are prepared in advance, and a user's selection instruction for these operation objects is provided by user's line-of-sight input information, voice input information, operation input information, image input information. Based on the information obtained by recognizing at least one or more of the information and the operation input information, the intention of the user is detected, and the operation target corresponding to the operation target is detected based on the detection result. An interface device of a multi-modal input / output device configured to return a required presentation to a user in order to notify that the selection instruction has been recognized, wherein the intention is detected using causal relationship information between input information. A multimodal input / output device characterized in that the causal relationship information is statistically learned through actual input / output with a user. Interface device. 2. An interface device for a multi-modal input / output device according to claim 1, wherein said integrating means has time managing means for managing time and holding means for holding input information at present and past times. An interface device for a multimodal input / output device, wherein the causal relationship information uses current and past input information held by the holding means. 3. The interface device for a multi-modal input / output device according to claim 1, wherein said integrating means transmits the causal relationship information between the input information or between the input information and the input time to the user. An interface device for a multi-modal input / output device characterized by learning statistically through actual input / output. 4. Various operation objects whose processing contents are determined are prepared in advance, and a user's selection instruction for these operation objects is input by the user's gaze input information, voice input information, operation input information, image input information. Based on the information obtained by recognizing at least one or more of the information and the operation input information, the intention of the user is detected, and the operation target corresponding to the operation target is detected based on the detection result. In a multimodal input / output device interface device for returning a required presentation to a user in order to notify that the selection instruction has been recognized, a multimodal interface is provided independently for each operation target. Interface device for input / output devices. 5. An interface device for a multi-modal input / output device according to claim 3, wherein said integrating means converts a result of intention detection in the specific operation target into a position, shape, color, etc. with another operation target group. An interface device for a multi-modal input / output device, which is determined based on a linguistic relationship. 6. The interface device for a multi-modal input / output device according to claim 3, wherein the integrating means determines the specific operation target based on its own intention detection result and the intention detection result of another operation target group. An interface device for a multi-modal input / output device characterized by changing a positional, shape, color, and linguistic relationship with a group of operation targets. 7. The multimodal input / output device interface device according to claim 1, wherein the integration means returns the intention information to the user at a past time or to the user. An interface device for a multi-modal input / output device, wherein the interface device is obtained from the presentation result. 8. The multimodal input / output device interface device according to claim 1, wherein learning of causal relationship information is started or ended based on intention information obtained from said integrating means. Output device interface device. 9. The multimodal input / output device according to claim 1, wherein the output for returning a required presentation to the user by the operation target is performed at the time of the learning. Output device interface device. 10. The interface device for a multi-modal input / output device according to claim 1, wherein said integration means includes at least one of said various input information as information for confirming or canceling an intention detection result. An interface device for a multi-modal input / output device, which is used. 11. Various operation objects having predetermined processing contents are prepared in advance, and a user's selection instruction for these operation objects is provided by user's line-of-sight input information, voice input information, operation input information, image input information. Based on the information obtained by recognizing at least one or more of the information and the operation input information, the intention of the user is detected, and the operation target corresponding to the operation target is detected based on the detection result. An interface method of a multi-modal input / output device for returning a required presentation to a user to notify that the selection instruction has been recognized, wherein the intention is detected using causal relationship information between input information. An interface method for a multimodal input / output device, wherein the causal relationship information is statistically learned through actual input / output with a user. 12. The multi-modal input / output device interface method according to claim 7, wherein the learning is performed by statistically analyzing causal relationship information between input information or between input information and input time through actual input / output with a user. A method for interfacing a multimodal input / output device, the method comprising: