JP2010141843A - Conference system, method of supporting communication conference, conference terminal unit, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conference system, a communication conference support method, a conference terminal device, and a program used therefor that enable a smooth communication conference. A conference terminal device that connects a plurality of remote locations and performs a communication conference. A motion detection unit that detects the vertical movement of the head of the user's own user who operates the device, and the concentration of the user's head in the conference based on the vertical movement of the user's head detected by the motion detection unit Concentration determination means for determining the degree of transmission, transmission means for transmitting the determination result determined by the concentration degree determination means to the communication partner conference terminal device via the network, and transmitted from the communication partner conference terminal device Receiving means for receiving the determination result; and display control means for performing display control based on the determination result received by the receiving means; It mounted.
[Selection] Figure 2

Description

  The present invention relates to a technical field of a conference system, a communication conference support method, a conference terminal device, and a program for performing a conference between a plurality of sites.

  2. Description of the Related Art Conventionally, a conference system connects a plurality of remotely arranged conference terminal devices (hereinafter referred to as terminal devices) via a network, and performs a communication conference such as a video conference or an audio conference between these terminal devices. Known to do.

  In this conference system, when a video conference is performed, data such as video and audio is transmitted and received between the terminal devices. Then, the user of each terminal device conducts a conference while watching and listening to the video and audio of the user of the communication partner terminal device.

  For example, when a video conference is performed between three or more terminal devices, the video of the users of all the communication partner terminal devices is displayed on the display unit of each terminal device. Thereby, the user of each terminal device can perform a video conference while watching the images of the users of all the communication partner terminal devices.

  However, since the video of the user of the communication partner terminal device is displayed on one display unit, when the number of communication partner terminal devices increases, each communication partner terminal image displayed on the display unit of each terminal device increases. The video of the user of the terminal device becomes small. Therefore, it may be difficult for the user of each terminal device to grasp the state of the user of the communication partner terminal device on the display screen.

  On the other hand, in this conference system, when a voice conference is performed, voice data is transmitted and received between the terminal devices, and a user of each terminal device performs a conference with a user of a communication partner terminal device through a voice call. Also at this time, since the user of each terminal device can only listen to the voice of the user of the communication partner terminal device, it may be difficult to grasp the state of the user of the communication partner terminal device.

Therefore, in order to easily grasp the state of the user of the communication partner terminal device, in the technique described in Patent Document 1, the user of the own device (hereinafter referred to as “own user”) photographed in each terminal device. The user's reaction is analyzed by performing image processing on the video of), and the analysis result can be notified to the terminal device of the communication partner.
JP 2001-67482 A

  However, in the technique disclosed in Patent Document 1, since the user's reaction is analyzed by analyzing the captured video, it may be difficult to accurately grasp the user's reaction. For example, the user may move to a position outside the shooting range, or the shooting may be from a direction in which the user's head can be shot but the user's head movement cannot be detected. The reaction of the user of the communication partner terminal device cannot be grasped. If the speaker does not understand the other party's reaction, he / she becomes worried whether the other party is intensively listening to the conversation, whether the content being spoken is transmitted to the other party, or reacting to the content being spoken. It was also possible to interrupt the smooth meeting itself by interrupting and checking the other party's situation. In order to support a smooth communication conference, and the speaker can feel comfortable without having anxiety, it is indispensable that the speaker can accurately grasp the reaction of the other party.

  Accordingly, an object of the present invention is to provide a conference system, a communication conference support method, a conference terminal device, and a program that enable a smooth communication conference.

  In order to solve the above-described problem, the invention according to claim 1 is directed to a conference terminal device in a conference system in which at least one of video and audio data is transmitted and received between conference terminal devices via a network. An image capturing unit that captures an image of the user participating in the network, video data transmitting means for transmitting video data output from the image capturing unit to a conference terminal device of a communication partner via the network, and transmission from the conference terminal device Video data receiving means for receiving the received video data, a display unit for displaying an image according to the video data received by the video data receiving means, and a head of the user of the user apparatus A motion information acquisition unit that acquires head motion information detected by the motion detection unit, and a user's head acquired by the motion information acquisition unit Concentration level determination means for determining the concentration level of the user in the meeting based on the operation information of the user, and the concentration level of the user in the meeting based on the determination result made by the concentration level determination means Transmission means for transmitting information to the conference terminal device of the communication partner, and reception of receiving information indicating the degree of concentration in the conference of other users who are users of the conference terminal device of the communication partner from the conference terminal device of the communication partner And display control means for performing display in accordance with information indicating the degree of concentration of other users in the meeting received by the receiving means.

  Further, the invention according to claim 2 is directed to the conference terminal device in the conference system in which at least one of video and audio data is mutually transmitted and received between the conference terminal devices via the network. Based on the movement information acquisition means for acquiring the movement information of the head detected by the movement detection unit mounted on the user's head, and the movement information of the head of the own user acquired by the movement information acquisition means Concentration level determination means for determining the degree of concentration of the user in the meeting, and information indicating the concentration level of the user in the meeting based on the result of the determination made by the concentration level determination means Transmission means for transmitting to the terminal device, and reception for receiving information indicating the degree of concentration in the conference of another user who is a user of the conference terminal device of the communication partner from the conference terminal device of the communication partner And the step, is characterized in that and a display control means for displaying in accordance with the information indicating the degree of concentration to the conference other users received by the receiving means.

  According to a third aspect of the present invention, in the conference terminal device according to the first aspect, the display control means is an image corresponding to the video data transmitted from the conference terminal device of another user whose concentration degree is not less than a predetermined level. In this case, an image indicating that another user is in a concentrated state is displayed on the display unit.

  According to a fourth aspect of the present invention, in the conference terminal device according to the first or third aspect of the present invention, the display control unit is configured to respond to the video data transmitted from the conference terminal device of another user with a predetermined degree of concentration or more. The displayed image is displayed on the display unit in a size larger than the image corresponding to the video data transmitted from the conference terminal device of another communication partner.

  The invention according to claim 5 is the conference terminal device according to any one of claims 1 to 4, further comprising speech determination means for determining whether or not a speech is made by the own user, wherein the display The control means performs display according to the information indicating the degree of concentration of the other user received by the reception means when the speech determination means determines that the speech is being performed by the own user. When it is determined by the determining means that the user does not speak, the display according to the information indicating the degree of concentration of other users received by the receiving means is not performed.

  The invention according to claim 6 is the conference terminal device according to any one of claims 1 to 5, further comprising a pulse measuring unit that measures the pulse of the own user, wherein the concentration determination unit is configured to perform the operation In addition to the operation information of the head of the user's own detected by the detection unit, the concentration of the user's concentration in the meeting is determined based on the pulse of the user's own measured by the pulse measurement unit. To do.

  The invention according to claim 7 is the conference terminal device according to any one of claims 1 to 5, wherein the concentration determination means is a head based on the motion information acquired by the motion information acquisition means. The vertical deflection width of the head is calculated, the number of vertical movements of the head in a predetermined period is detected, and the concentration level of the user is determined based on the vertical deflection width and the number of vertical movements of the head. It is characterized by this.

  According to an eighth aspect of the present invention, in the conference terminal device according to the sixth aspect of the invention, the concentration degree determining means calculates a vertical deflection width of the head based on the motion information acquired by the motion information acquiring means. Calculating, detecting the number of vertical movements of the head in a predetermined period, calculating the difference between the pulse measured by the pulse measuring unit and the reference pulse, the deflection width in the vertical direction of the head, The degree of concentration of the user is determined based on the number of vertical movements and the difference between the pulse and the reference pulse.

  The invention according to claim 9 is the conference terminal device according to any one of claims 1 to 8, wherein information indicating the degree of concentration of the own user is transmitted to the conference terminal device of the communication partner. A concentration notification presence / absence input unit that accepts an input from the user, and the transmission means receives an input to the communication partner conference terminal device when the input is received by the concentration notification presence / absence input unit, When information indicating the concentration level of the own user is transmitted to the conference terminal device of the communication partner while an input indicating that the information is not transmitted to the conference terminal device of the communication partner is received by the concentration notification notification input unit The information indicating the degree of concentration of the own user is not transmitted to the conference terminal device of the communication partner.

  The invention according to claim 10 is the conference terminal device according to any one of claims 1 to 9, wherein when the concentration level of the own user determined by the concentration level determination means is low, It is characterized by comprising a stimulus control means for performing a control to give a stimulus to the user by sound.

  The invention according to claim 11 further includes a stimulus control setting unit that receives, from the user, a setting as to whether or not to give a stimulus to the user in a state of low concentration in the conference terminal device according to claim 10, When the stimulation control unit is set to give a stimulus to the user by the stimulation control setting unit, the concentration control unit determines that the concentration level of the user is low by the concentration level determination unit. While the control for giving a stimulus to the user by sound is performed, and the setting to not give the user a stimulus by the stimulus control setting unit is made, the concentration degree of the own user is low by the concentration degree determination means If it is determined that the state is in a state, control for giving a stimulus to the user by light or sound is not performed, and the transmission unit is set to not give a stimulus to the user by the stimulus control setting unit. When the information indicating the degree of concentration of the own user is not transmitted to the conference terminal device of the communication partner while the stimulus control setting unit is set to stimulate the own user, Information indicating the degree of concentration is transmitted to the conference terminal device of the communication partner.

  The invention according to claim 12 is a program for causing a computer to function as each means in the conference terminal device according to any one of claims 1 to 11.

  The invention according to claim 13 is a communication conference support method for supporting a communication conference in which at least one of video and audio data is mutually transmitted and received between conference terminal devices via a network. Based on the step of acquiring the motion information of the head detected by the motion detection unit mounted on the head of the user who is the user of the device, and the acquired motion information of the head of the user Determining the concentration level of the user in the conference, transmitting information indicating the concentration level of the user to the conference based on the determination result to the conference terminal device of the communication partner, Receiving from the conference terminal device information indicating the degree of concentration of the other user, who is the user of the conference terminal device of the communication partner, in the conference, and the received information indicating the degree of concentration of the other user in the conference; Performing a display Flip was, and has a.

  The invention according to claim 14 is the conference system in which at least one of video and audio data is mutually transmitted and received between the conference terminal devices via the network, wherein each of the conference terminal devices is the own device. Movement information acquisition means for acquiring movement information of the head detected by the movement detection unit mounted on the user's own user's head, and movement of the user's head acquired by the movement information acquisition means Concentration determination means for determining the degree of concentration of the user in the meeting based on the information, and information indicating the degree of concentration of the user in the meeting based on the determination result made by the concentration degree determination means Transmitting means for transmitting to the conference terminal device of the communication partner, and receiving information indicating the degree of concentration in the conference of other users who are users of the conference terminal device of the communication partner from the conference terminal device of the communication partner Receiving means that is characterized in that and a display control means for displaying in accordance with the information indicating the degree of concentration to the conference other users received by the receiving means.

  According to a fifteenth aspect of the present invention, there is provided a server device that is connected to a plurality of conference terminal devices via a network and manages a conference between the conference terminal devices, and each user's head of the plurality of conference terminal devices. Based on the motion information of the head of the user acquired by the motion information acquisition means, the motion information acquisition means for acquiring the motion information of the head detected by the motion detection unit attached to the user A degree of concentration determination means for determining the concentration degree to the device, and a transmission means for transmitting information indicating the degree of concentration based on the result of the determination made by the concentration degree determination means to the plurality of conference terminal devices. It is characterized by this.

  According to the present invention, the motion detection unit detects the motion of the head of the user, and the concentration level determination unit determines the concentration level of the user in the conference based on the motion information detected by the motion detection unit. The transmitting unit transmits the determination result (information indicating the concentration level) by the concentration level determining unit to the communication terminal conference terminal device. On the other hand, the receiving means receives the information indicating the degree of concentration of the other user who is the user of the conference terminal apparatus transmitted by the conference terminal apparatus of the communication partner, and displays the information indicating the degree of concentration of the other user received by the display control means. Display control based on this.

  Thereby, since the own user can grasp the degree of concentration of the other user who is the communication partner and can change the method of proceeding the conference according to the degree of concentration of the other user, a smooth communication conference is possible.

  In particular, since the motion detector is mounted on the user's head, the movement of the user's head can be directly detected. Thereby, the motion of the user's head can be detected with high accuracy.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a first embodiment of the invention will be described with reference to the drawings.

[First Embodiment]
[Outline of Conference System 1]
First, the outline of the conference system according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of a conference system according to the first embodiment of the present invention.

  As shown in FIG. 1, the conference system 1 according to the first embodiment performs a communication conference such as a video conference between remotely arranged conference terminal devices, and a plurality of conference terminal devices 100 a to 100 d. It is configured by being connected to a network. In the present embodiment, the conference system 1 is configured by the four conference terminal devices 100a to 100d, but is not limited to this. In the following, when any conference terminal device is shown among the conference terminal devices 100a to 100d, it is referred to as “conference terminal device 100”.

  In the conference system 1 configured as described above, a video conference communication conference is performed by exchanging video and audio between the conference terminal devices 100a to 100d. In the conference system 1, a voice conference communication conference can be performed by exchanging audio without exchanging video between the conference terminal devices 100 a to 100 d.

[Outline of conference terminal device]
Next, the outline | summary of the conference terminal device 100 which is the principal part of the conference system 1 which concerns on 1st Embodiment is demonstrated. FIG. 2 is a diagram illustrating a schematic configuration of the conference terminal device 100 according to the first embodiment.

  As shown in FIG. 2, the conference terminal apparatus 100 includes a motion detection unit, a pulse measurement unit, a motion information acquisition unit, a pulse information acquisition unit, a concentration degree determination unit, a transmission unit, a reception unit, a display control unit, a display unit, and an imaging unit. Department.

  The motion detection unit is mounted on the head of a user of the user's own device (hereinafter referred to as “own user”) and detects the motion of the user's head. As the motion detection unit, for example, an acceleration sensor, a speed sensor, or the like can be used. In addition, for example, a headset-type head wearing tool with a built-in motion detection unit can be used for mounting the motion detection unit.

  The motion information acquisition means acquires motion information related to the motion of the head of the own user detected by the motion detection unit. For example, if the motion detector is an acceleration sensor, acceleration information is acquired as motion information, and if the motion detector is a speed sensor, speed information is acquired as motion information. The connection between the motion detection unit and the motion information acquisition unit can be made with a communication cable or the like, but may be a wireless connection.

  The pulse measuring unit is attached to the wrist of the own user and measures the pulse of the own user. The pulse measuring unit includes a pulse meter and a wearing tool for wearing this pulse meter on the wrist of the user.

  The pulse information acquisition means acquires pulse information related to the pulse measured by the pulse measurement unit. For example, the pulse rate per minute is sequentially acquired from the pulse measurement unit as the pulse information. Note that the connection between the pulse measurement unit and the pulse information acquisition unit can be made by a communication cable or the like, but may be a wireless connection.

  The concentration level determination unit determines the concentration level of the user in the conference based on the motion information acquired by the motion information acquisition unit and the pulse information acquired by the pulse information acquisition unit. For example, the concentration degree determination means calculates the vertical swing width of the head based on the motion information acquired by the motion information acquisition means, detects the number of vertical movements of the head during a predetermined period, The difference between the pulse measured by the measurement unit and the reference pulse is calculated, and the degree of concentration of the user is determined based on the information.

  For example, the concentration determination means determines that the concentration is high when the user is in a concentration state, determines that the concentration is low when the user is in a sleep state, and has a medium concentration otherwise. It is determined that

  The transmission means transmits information indicating the degree of concentration of the user to the conference based on the result of the determination made by the concentration level determination means (hereinafter referred to as “concentration information”) to the conference terminal device 100 of the communication partner. To send. The transmission unit also functions as a video data transmission unit that transmits the video data output from the imaging unit to the conference terminal device 100 of the communication partner via the network.

  The receiving unit receives the concentration degree information of the user (hereinafter referred to as “other user”) of the conference terminal device 100 of the communication partner transmitted by the conference terminal device of the communication partner. The receiving unit also functions as a video data receiving unit that receives video data transmitted from the conference terminal device 100 of the communication partner.

  The display control unit controls the display unit to display according to the concentration information of other users received by the receiving unit. For example, the display control means displays an image indicating that the other user is in a concentrated state on the display unit in association with an image corresponding to the video data transmitted from the conference terminal device 100 of the other user whose degree of concentration is a predetermined level or more. Let In addition, the display control unit displays an image corresponding to the video data transmitted from the conference terminal apparatus 100 of another user whose degree of concentration is a predetermined level or more according to the video data transmitted from the conference terminal apparatus 100 of the other communication partner. It is displayed on the display unit with a size larger than the image.

  According to the conference terminal device 100 of the conference system 1 described above, the concentration level determination unit determines the concentration level of the own user based on the operation information of the own user detected by the operation detection unit, and the transmission unit determines the concentration level of the own user. While the information is transmitted to the other user, the receiving means receives the concentration information of the other user, and the display control means controls the display based on the concentration information of the other user received. The degree of concentration of other users in the meeting can be grasped. In addition, since the motion detector is mounted on the head of the user, the motion of the user's head can be detected with high accuracy, and the concentration level can be determined with high reliability.

  In this way, since the own user can grasp the degree of concentration of the other user in the conference, for example, when the own user is a speaker, if the other user has a high degree of concentration, the own user grasps that and feels Can speak well. In addition, when it is determined that the concentration of the user of the conference terminal device 100 of a certain communication partner is high and the concentration of the user of the conference terminal device 100 of another communication partner is determined to be low, Talking to other users with a high degree of concentration and having a conversation, it is possible to attract the interest of other users with a low degree of concentration to the conference, and as a result, the degree of concentration of users with a low degree of concentration as a result. The user can hold a meeting in a good mood.

[Concrete configuration of conference terminal device]
Next, the conference terminal apparatus 100 according to the first embodiment will be described in detail with a specific example. FIG. 3 is a block diagram showing a specific configuration of the conference terminal apparatus according to the first embodiment. The conference terminal device 100 is equivalent to the conference terminal device 100 arranged at each site in FIG. 1, and is appropriately distinguished as conference terminal devices 100a to 100d. In this case, the conference terminal device 100a is referred to as the conference terminal device 100a. The conference terminal device on the speaker side and the conference terminal devices 100b to 100d are set as conference terminal devices on the viewer side.

  As shown in FIG. 3, the conference terminal device 100 includes a device main body 110, a headset 120, a pulsometer 130, an imaging unit 140, and a display unit 150. Note that one or a plurality of headsets 120 and a pulse meter 130 are connected to the apparatus main body 110 according to the number of own users.

[Configuration of Device Main Body 110]
The apparatus main body 110 is a computer device to which a general-purpose computer can be applied and includes a CPU 111, a storage unit 112, a communication IF 113, an operation unit 114, and an image data input unit (not shown). These various components are connected by a system bus 115. The CPU 111 constitutes a control unit 116 together with the storage unit 112.

  The storage unit 112 stores setting data such as a conference control program, user data such as a user name, a user's reference pulse, and various flags related to the conference. Further, the storage unit 112 further stores first and second determination tables (see FIGS. 14 and 15) and information related to the movement of the user's head during the conference (such as a swing width and the number of vertical movements described later). A storage area (not shown) is provided.

  The CPU 111 performs basic control as a computer of the conference terminal device 100 according to the conference control program stored in the storage unit 112. As a result, the control unit 116 includes display control means for controlling the display unit 150, stimulus control means for performing stimulus control on the own user, concentration level determining means for determining the concentration level of the own user, and remarks by the own user. It functions as an utterance determination unit that determines whether or not it is performed, an operation information acquisition unit that acquires operation information from the accelerometer 123, and a pulse information acquisition unit that acquires pulse information from the pulse meter 130.

  The conference control program may be downloaded to the storage unit 112 from, for example, a server connected to the Internet, or after being recorded on a recording medium such as a CD-ROM, via the drive of the recording medium. Then, it may be read into the storage unit 112.

  The communication IF 113 is a NIC (Network Interface Card), for example, and is connected to the network via a router. This communication IF 113 provides the communication terminal conference device 100 with communication data related to the own user, such as video data obtained by imaging the own user, audio data obtained by collecting the own user's voice, and concentration information of the own user. Send. In addition, the communication IF 113 is communication data related to other users transmitted by the conference terminal device 100 of the communication partner, video data obtained by imaging other users, audio data obtained by collecting other users' voices, concentration information of other users, and the like. Receive. The communication IF 113 functions as a transmission unit and a reception unit together with the control unit 116.

  The operation unit 114 includes a concentration degree notification presence / absence input unit 117, a stimulus control setting unit 118, a mouse and a keyboard (not shown). The concentration level notification presence / absence input unit 117 is an input button that receives, from the own user, an input as to whether or not to transmit the concentration level information of the own user to the conference terminal device 100 of the communication partner. The stimulus control setting unit 118 is a setting button for accepting a setting as to whether or not stimulus control such as awakening control or exit control is performed on the sleeping user.

[Configuration of Headset 120 and Pulse Meter 130]
As shown in FIG. 4, the headset 120 is worn on the head of the user and makes a voice call with another user who is a communication partner. It has a function to detect the operation. The headset 120 includes a microphone 121, a speaker 122, and an accelerometer 123 that detects acceleration applied to the headset 120.

  As shown in FIG. 5, the accelerometer 123 detects the acceleration of the vertical movement of the user's head when the user's head moves in the vertical direction. In this embodiment, the accelerometer 123 is provided integrally with the headset 120. However, the present invention is not limited to this. For example, the accelerometer 123 may be provided so as to be fixed to the user's chin or the like.

  The pulsometer 130 measures the pulse of the own user. The pulsometer 130 is fixed to a wristband or the like, and the user can wear the pulsometer 130 around the wrist by winding the wristband around the wrist.

[Configuration of Imaging Unit 140 and Display Unit 150]
The imaging unit 140 is, for example, a CCD camera that performs imaging and outputs video data. The image capturing unit 140 captures an image of the own user participating in the conference, and can transmit the video data of the own user obtained as a result to the conference terminal device 100 of the communication partner via the communication IF 113 or the like. Yes.

  The display unit 150 is a monitor such as an LCD (liquid crystal display). As shown in FIG. 6, the display unit 150 includes presentation material or the like (see FIG. 6A) used in the conference input from the image data input unit or video data transmitted by the conference terminal device 100 as the communication partner. An image of another user based on (see FIG. 6B) is displayed. Depending on the setting, it is possible to fix and display the presentation material or the like or the video of another user, or to sequentially switch and display these screens depending on the speech status of the user.

  A display screen for displaying the video of another user includes a main display area and a sub display area. The main display area is a main display area on the display screen of the display unit 150, and displays an image of another user as shown in FIG. 6B. The sub display area is a sub display area on the display screen of the display unit 150, and as shown in FIG. 7, a terminal device display that displays which of the conference terminal devices 100 participating in the conference is displayed. And a concentration level display region for performing display according to the concentration level information of other users. The concentration display area corresponding to the own device may display according to the concentration information of the own user, but here, whether or not the own user is a speaker is displayed. . Note that the sub-display area may not be displayed by the user's setting operation.

  Display according to the concentration level information of other users in the concentration level display area is performed by displaying an icon indicating a predetermined state as shown in FIG. In the present embodiment, it is displayed that the user of the conference terminal device 100b is in a normal state, the user of the conference terminal device 100c is in a concentrated state, and the user of the conference terminal device 100d is in a sleep state. It is displayed that the conference terminal device 100a is a speaker. Further, when there are a plurality of users in each conference terminal apparatus 100, the number of icons corresponding to the number of users can be displayed. In addition, the display of the state display portion is not limited to the display by the icon shown in FIG. 7A, and may be, for example, the display by characters shown in FIG. 7B or the display by a color representing a predetermined state. When the user state is represented by color, for example, the fact that the user is concentrated is displayed in red, and the user is in sleep state is displayed in blue.

  The above is the description of the configuration of the conference system 1 and the conference terminal device 100 according to the first embodiment.

[Control processing]
Control processing performed by the CPU 111 of the conference terminal apparatus 100 in the conference system according to the first embodiment by reading various programs stored in the storage unit 112 will be described with reference to FIGS.

[Main processing]
In the conference terminal device 100, the main process shown in FIG. 8 is executed from when the power is turned on to when it is turned off.

  In this main process, first, the CPU 111 executes an input process for registering user data, setting a conference, and the like (step S1). Details of this input process will be described later with reference to FIG.

  Next, the CPU 111 controls the accelerometer 123 and the pulse meter 130 to detect the acceleration of the head movement of the user's head as the operation information of the user, and measures the pulse of the user (step S2). In this embodiment, attention is paid to the vertical movement of the user's head, and the detection of acceleration is also based on the vertical movement of the user's head. You may make it detect the acceleration with respect to the motion of the left-right direction of a part. The pulse is measured by measuring an average value per unit time, and the unit time is, for example, 1 minute.

  Next, the CPU 111 acquires the operation information of the user (acceleration when the user's head moves in the vertical direction) and the user's pulse from the accelerometer 123 and the pulse meter 130 (step S3). Then, the CPU 111 calculates the vertical swing width of the head of the user's head, the number of vertical movements of the head, and the like based on the acquired motion information, and stores these as information on the motion of the user's head. 112. In this embodiment, the acceleration of the vertical movement of the head is detected. However, the present invention is not limited to this. For example, the speed at which the head moves is detected using a speedometer, and the detected speed is detected. The deflection width and the number of times may be calculated based on the above.

  Next, the CPU 111 executes concentration degree determination processing for determining the degree of concentration of the user in the meeting based on the operation information acquired from the accelerometer 123 and the pulse information acquired from the pulse meter 130 (step S4). Details of the concentration determination process will be described later with reference to FIG.

  Next, in a predetermined case, the CPU 111 controls the communication IF 113 to transmit the concentration level information of the own user to the conference terminal device 100 of the communication partner, thereby notifying the concentration level of the own user. A notification process is executed (step S5). Details of the concentration degree notification process will be described later with reference to FIG.

  Next, based on the result of the concentration degree determination process, the CPU 111 executes a stimulus control process for performing awakening control that prompts the user to wake up or a leaving control that causes the user to leave the meeting (step S6). Details of the stimulus control process will be described later with reference to FIG.

  Next, the CPU 111 determines whether or not the notification of the concentration level of the other user has been received by determining whether or not the concentration level information of the other user transmitted by the conference terminal device 100 of the communication partner has been received. (Step S7). If the CPU 111 determines that the notification of the concentration level of the other user has been received (step S7: Yes), the process proceeds to step S8 and determines that the notification of the concentration level of the other user has not been received (step S7). : No), the process is returned to step S2.

  In step S8, the CPU 111 controls the display unit 150 to execute a concentration level information reception process for displaying according to the concentration level of other users, and returns the process to step S2. The details of the concentration information receiving process will be described later with reference to FIG.

  The above is the description of the main process performed by the CPU 111. Next, the “input process” in step S2 of the main process will be described in detail with reference to FIG.

[Input processing]
When the input process in step S1 of the main process is started, first, the CPU 111 controls the display unit 150 to display a screen for explaining a conference setting procedure (step S9). That is, the display unit 150 displays a message requesting the user to wear the headset 120 or the pulsometer 130 or a message requesting input of a user name. Is displayed.

  When the pulsometer 130 is worn, a pulse that serves as a reference (hereinafter referred to as “reference pulse”) when the pulsometer 130 determines the degree of concentration of the user is calculated. That is, an average value of a pulse in a predetermined period is calculated, and a range of values obtained by adding and subtracting a predetermined value to the calculated value is a reference pulse. For example, when the value obtained by calculating the 5-minute average of the pulse is 65 beats / minute and the predetermined value is 5 beats / minute, the reference pulse is in a range between 60 beats / minute and 70 beats / minute. .

  Next, when receiving an input of a user name (step S10), the CPU 111 refers to the user data stored in the storage unit 112 and inquires about the input user name (step S11).

  Next, the CPU 111 determines whether or not the user data corresponding to the user name input in step S10 has been registered based on the result of the user name inquiry performed in step S11 (step S12). If the CPU 111 determines that the user data has been registered (step S12: Yes), the process proceeds to step S14. If the CPU 111 determines that the user data has not been registered (step S12: No), the process proceeds to step S13. Migrate processing.

  In step S13, the CPU 111 registers new user data. That is, the CPU 111 stores the user name, the reference pulse measured in step S9, and the like as new user data in the storage unit 112. When this process ends, the process proceeds to step S14.

  In step S14, the CPU 111 sets the registered user data as user data used in the conference. At this time, user data corresponding to the user name input in step S10 is set from the registered user data as user data used in the conference. Even if it is determined in step S12 that the user data has already been registered, new user data is registered by a predetermined operation, and the new user data is set as user data used in the conference. You can also.

  Next, the CPU 111 displays on the display unit 150 a screen for prompting an operation to the concentration notification presence / absence input unit 117 for selecting whether or not to notify other users as communication partners of the concentration level of the user. To operate the concentration degree notification presence / absence input unit 117. The CPU 111 determines whether or not to notify the concentration level of the own user based on the operation of the concentration level notification presence / absence input unit 117 by the own user (step S15).

  Then, if the operation on the concentration level notification presence / absence input unit 117 by the user is an operation to notify the concentration level of the own user (step S15: Yes), the CPU 111 shifts the process to step S16, and the own user If the operation of the concentration level notification presence / absence input unit 117 is an operation not to notify the concentration level of the own user (step S15: No), the process proceeds to step S17.

  In step S <b> 16, the CPU 111 executes a concentration level notification setting process for performing settings for notifying other users of the concentration level of the user. That is, the CPU 111 changes the setting value of the concentration degree notification flag stored in the storage unit 112 from “0” to “1”. When the concentration level notification setting process is not executed, the setting value of the concentration level notification flag stored in the storage unit 112 remains “0”. When this process ends, the process proceeds to step S17.

  In step S <b> 17, the CPU 111 displays a screen that prompts the user to operate the stimulus control setting unit 118 that selects whether or not to perform wake-up control for urging the user when the user is in a sleep state during the meeting. 150, and the user is allowed to operate the stimulus control setting unit 118. Based on the operation of the stimulus control setting unit 118 by the own user, the CPU 111 determines whether or not to perform the awakening control of the own user. Then, if the operation on the stimulus control setting unit 118 by the own user is an operation for performing the awakening control of the own user (step S17: Yes), the CPU 111 shifts the process to step S18.

  In step S <b> 18, the CPU 111 executes a wake-up control setting process for setting a control for prompting the user to wake up. That is, the CPU 111 changes the set value of the awakening control flag stored in the storage unit 112 from “0” to “1”. When the awakening control setting process is not executed, the set value of the awakening control flag stored in the storage unit 112 remains “0”. When this process ends, the CPU 111 shifts the process to step S19.

  On the other hand, if the operation of the stimulus control setting unit 118 by the own user is an operation not to perform the awakening control of the own user (step S17: No), the CPU 111 shifts the process to step S19.

  The CPU 111 displays on the display unit 150 a screen that prompts the operation of the operation unit 114 to select whether or not to perform exit control for leaving the user from the conference when the user is in a sleep state during the conference. To operate the operation unit 114. CPU111 determines whether own user's leaving control is performed based on operation of the operation part 114 by the own user (step S19).

  Then, if the operation of the stimulus control setting unit 118 by the own user is an operation to perform the exit control of the own user (step S19: Yes), the CPU 111 shifts the process to step S20. On the other hand, when the operation of the stimulus control setting unit 118 by the own user is an operation to perform the exit control of the own user (step S19: No), the CPU 111 shifts the process to step S2 of the main process.

  In step S <b> 20, the CPU 111 executes a leaving control setting process for setting a control for leaving the own user who is in a sleeping state from the conference. That is, the CPU 111 changes the setting value of the leaving control flag stored in the storage unit 112 from “0” to “1”. When the exit control setting process is not executed, the set value of the exit control flag stored in the storage unit 112 remains “0”. When this process ends, the CPU 111 shifts the process to step S2 of the main process.

[Concentration determination processing]
Next, the “concentration determination process” in step S4 of the main process will be described in detail with reference to FIG.

  When the concentration determination process in step S1 of the main process is started, first, the CPU 111 moves the head of the user's head down and shakes his / her head up and down (hereinafter referred to as “smile”). It is determined whether or not (step S21). Whether it is “slight” is determined based on the movement of the head of the user, the pulse of the user, and the first determination table (see FIG. 14) stored in the storage unit 112. That is, the swing width W of the head movement of the user's head, the number n of head movements, the period T when the head moves up and down, and the pulse P of the user's user are described in the first determination table. Judgment is made based on whether the requirement is satisfied. In the determination of “slightly”, “slightly” and “slightly 2” are distinguished according to the “slightly” pattern. However, in the following explanation, these are collectively referred to as “slightly”. To do.

Specifically, the swing width W of the vertical movement of the user's head is greater than 5 cm but smaller than 12 cm, and the period T when the head moves up and down is 10 seconds or more and 30 seconds or less. When the number of movements n is 5 or more and the measured pulse P is 80% or less of the reference pulse P ₀ , it is determined that the movement of the user's head is “just 1”. In addition, when the head movement width W is 12 cm or more and the measured pulse P is 80% or less of the reference pulse P ₀ , the movement of the user's head is “slightly 2”. Determined. In addition, when there are a plurality of own users, depending on the setting, it may be determined for a specific user among the own users, or may be determined for all the own users. The specific user is, for example, a user wearing a specific headset 120.

  If the CPU 111 determines that the head movement of the user is “slightly” (step S21: Yes), the CPU 111 proceeds to step S22 and determines that the head movement of the user is not “slightly”. Then (step S21: No), the process proceeds to step S23.

It should be noted that the above-described determination of whether or not it is “slight” can be variously changed according to the desired determination accuracy. For example, the determination can be made based on the motion of the head of the user, the pulse of the user, and the acceleration of the vertical movement of the head. That is, referring to the first table, a determination is made based on the movement of the head of the user and the pulse of the user, and the acceleration of the vertical movement of the head is referred to by referring to the second determination table (see FIG. 15). Based on the determination. Specifically, when the deflection width W is 12 cm or more, the measured pulse P is 80% or less of the reference pulse P ₀ , and the acceleration α is 10 cm / s ² or more downward, the movement of the user's head Is determined to be “collapse 2”. According to this, it is possible to increase the determination requirements for determining “slightly” and to increase the determination accuracy.

  In addition, the determination based on the pulse can be omitted in the determination based on the motion of the head of the user and the pulse of the user. Specifically, referring to the first table, a determination is made based on the deflection width W of the vertical movement of the head, the number n of vertical movements of the head, and the period T when the head moves up and down. Is larger than 5 cm but smaller than 12 cm, the period T is 10 seconds or more and 30 seconds or less, and the number of times n is 5 times or more, it is determined as “slightly 1”, and the deflection width W is 12 cm or more It is determined that it is “slightly 2”. In this case, the determination accuracy may be lowered, but the determination can be simplified.

Further, from the movement of the user's head, the pulse of the user, and the acceleration of the vertical movement of the head, the fluctuation width W of the vertical movement of the head, the acceleration α of the vertical movement of the head, and the pulse P of the user It is also possible to make a determination based on this. Specifically, referring to the first table, a determination is made based on the up and down movement width W of the head and the pulse P of the user, and the acceleration of the vertical movement of the head is referred to the second determination table. When the determination is made based on α, the deflection width W is 12 cm or more, the measured pulse P is 80% or less of the reference pulse P ₀ , and the acceleration α is 10 cm / s ² or more in the downward direction, the user's head It is determined that the operation of is “just 2”.

  In step S22, CPU111 performs the sleep state setting process which sets that the own user is a sleep state. That is, the CPU 111 changes the setting value of the low concentration flag stored in the storage unit 112 from “0” to “1”. When the sleep state setting process is not executed, the setting value of the low concentration level flag stored in the storage unit 112 remains “0”. In addition, when there are a plurality of users whose concentration is determined, the sleep state setting process is executed based on the determination result of a specific user among the users whose concentration is determined, or the concentration The sleep state setting process may be executed based on the determination results of all the determined users. For example, in the case where the sleep state setting process is executed based on the determination results of all the users determined to have the degree of concentration, the determination result of each user is determined for each user whose head movement is determined to be “sloppy”. Based on this, the sleep state setting process is executed.

  In step S <b> 23, the CPU 111 determines whether or not the movement of the user's head is an action of displaying an intention such as consent or consent to the other party by shaking the head up and down (hereinafter referred to as “nodding”). Determine. Whether it is “nodding” is determined based on the movement of the head of the user, the pulse of the user, and the first determination table stored in the storage unit 112. That is, the swing width W of the head movement of the user's head, the number n of head movements, the period T when the head moves up and down, and the pulse P of the user's user are described in the first determination table. Judgment is made based on whether the requirement is satisfied.

Specifically, the swing width W of the head movement of the user's head is 2 cm or more and 5 cm or less, the period T when the head moves up and down is 1 second or less, and the number n of head movements is 1 It is determined that the movement of the user's head is “nodding” when the pulse P is ˜3 times and the measured pulse P is 1.2 times or more of the reference pulse P ₀ . As in the case of “sloppy” determination, when there are a plurality of own users, depending on the setting of the own user, determination may be made for a specific user within the own user, or determination may be made for all own users. Also good. The specific user is, for example, a user wearing a specific headset 120.

  If the CPU 111 determines that the head movement of the user is “nodding” (step S23: Yes), the process proceeds to step S24, and the CPU 111 determines that the head movement of the user is not “nodding”. Then (step S23: No), the process proceeds to step S5 of the main process.

It should be noted that the determination of whether or not the above-mentioned “nodding” can be variously changed according to desired determination accuracy. For example, the determination can be made based on the motion of the head of the user, the pulse of the user, and the acceleration of the vertical movement of the head. That is, the determination based on the head movement of the user and the pulse of the user is made with reference to the first table, and the determination based on the acceleration of the vertical movement of the head is made with reference to the second determination table. Specifically, when the deflection width W is 2 cm or more and 5 cm or less, the measured pulse P is 1.2 times or more of the reference pulse P ₀ , and the acceleration α is 8 cm / s ² or less in the vertical direction, It is determined that the movement of the head is “nodding”. According to this, it is possible to increase the determination requirement for determining that it is “nodding”, and to increase the determination accuracy.

  In addition, the determination based on the pulse can be omitted in the determination based on the motion of the head of the user and the pulse of the user. Specifically, referring to the first table, a determination is made based on the deflection width W of the vertical movement of the head, the number n of vertical movements of the head, and the period T when the head moves up and down. Is 2 cm or more and 5 cm or less, the period T is 1 second or less, and the number n is 1 to 3 times, it is determined as “nodding”. In this case, the determination accuracy may be lowered, but the determination can be simplified.

Further, from the movement of the user's head, the pulse of the user, and the acceleration of the vertical movement of the head, the fluctuation width W of the vertical movement of the head, the acceleration α of the vertical movement of the head, and the pulse P of the user It is also possible to make a determination based on this. Specifically, referring to the first table, a determination is made based on the up and down movement width W of the head and the pulse P of the user, and the acceleration of the vertical movement of the head is referred to the second determination table. When the determination is made based on α, the deflection width W is 2 cm or more and 5 cm or less, the measured pulse P is 1.2 times or more of the reference pulse P ₀ , and the acceleration α is 8 cm / s ² or less in the vertical direction. It is determined that the movement of the user's head is “nodding”.

  In step S24, the CPU 111 executes a concentration state setting process for setting the concentration state of the own user. That is, the CPU 111 changes the setting value of the high concentration flag stored in the storage unit 112 from “0” to “1”. At this time, based on the information related to the movement of the head of the user's own user stored in the storage unit 112, the average number of vertical movements of the head from when the last “nodding” is detected to before a predetermined period is calculated, Information on the calculated average number of vertical movements of the head is stored in the storage unit 112. The period before the predetermined period is, for example, 30 seconds. When the concentration state setting process is not performed, the setting value of the high concentration degree flag stored in the storage unit 112 remains “0”. When there are a plurality of users for which the degree of concentration is determined, concentration state setting processing is executed based on the determination result of a specific user among the users for which the degree of concentration is determined or the degree of concentration is determined according to the setting of the own user. Alternatively, the concentration state setting process may be executed based on the determination results of all the users. For example, when the concentration state setting process is executed based on the determination results of all the users whose concentration levels are determined, the determination result of the user is determined for each determined user whose head movement is “nodding”. Based on this, the concentrated state setting process is executed. When this process ends, the process proceeds to step S5 of the main process.

  Next, the “concentration notification process” in step S5 of the main process will be described in detail with reference to FIG.

[Concentration notification processing]
When the concentration level notification process in step S5 of the main process is started, the CPU 111 first determines whether or not the concentration level notification is set (step S25). That is, the CPU 111 refers to the concentration level notification flag stored in the storage unit 112, determines that the concentration level notification flag is set when the setting value of the concentration level notification flag is “1”, and sets the concentration level notification flag setting value. Is determined not to perform the concentration degree notification. When the CPU 111 determines that the concentration level notification is set (step S25: Yes), the process proceeds to step S26. When the CPU 111 determines that the concentration level notification is not set (step S25: No), the main process is performed. The process proceeds to step S6. If there are a plurality of own users, it is determined whether or not the concentration degree notification is set for all of the own users.

  In step S26, the CPU 111 determines whether the sleep state is set. That is, the CPU 111 refers to the low concentration flag stored in the storage unit 112, determines that the sleep state is set when the setting value of the low concentration flag is “1”, and sets the low concentration flag. When the value is “0”, it is determined that the sleep state is not set. If the CPU 111 determines that the sleep state is set (step S26: Yes), the process proceeds to step S27. If the CPU 111 determines that the sleep state is not set (step S26: No), the process proceeds to step S28. To do. In addition, when there are a plurality of own users, it is determined whether or not the sleep state is set for each of all the own users.

  In step S27, the CPU 111 performs a sleep state notification process for notifying other users that the user is in a sleep state. Specifically, the CPU 111 controls the communication IF 113 and transmits concentration degree information indicating that the user is in a sleep state to the conference terminal device 100 of the communication partner. That is, the degree-of-concentration information with the low-concentration flag “1” is transmitted to the conference terminal device 100 of the communication partner. At this time, the setting value of the low concentration flag stored in the storage unit 112 is changed from “1” to “0”. When this process ends, the process proceeds to step S6 of the main process. In addition, when there are a plurality of own users, the sleep state notification process is performed for each of the own users set to the sleep state.

  In step S <b> 28, the CPU 111 determines whether or not the concentrated state is set. That is, the CPU 111 refers to the high concentration flag stored in the storage unit 112, determines that the user is in a concentrated state when the setting value of the high concentration flag is “1”, and sets the high concentration flag. When the value is “0”, it is determined that the user is not concentrated. If the CPU 111 determines that the own user is in a concentrated state (step S28: Yes), the process proceeds to step S29, and if the CPU 111 determines that the own user is not in a concentrated state (step S28: No), the main processing step. The process proceeds to S6. In addition, when there are a plurality of own users, it is determined whether or not the concentrated state is set for all of the own users.

  In step S29, the CPU 111 notifies other users that the user is in a concentrated state. Specifically, the CPU 111 controls the communication IF 113, and with respect to the conference terminal device 100 of the communication partner, concentration level information indicating that the own user is in a concentrated state, and information on the average number of head vertical movements Send. That is, the degree-of-concentration information with the high-concentration flag “1” is transmitted to the conference terminal device 100 of the communication partner. At this time, the setting value of the high concentration flag stored in the storage unit 112 is changed from “1” to “0”, and the information on the average number of vertical movements of the head stored in the storage unit 112 is deleted. . When this process ends, the process proceeds to step S6 of the main process. In addition, when there are a plurality of own users, the concentrated state notification process is performed for each of the own users set to the concentrated state.

  Next, the “stimulation control process” in step S6 of the main process will be described in detail with reference to FIG.

[Stimulation control processing]
When the stimulus control process in step S6 of the main process is started, first, the CPU 111 determines whether or not a sleep state is set (step S30). That is, the CPU 111 refers to the low concentration flag stored in the storage unit 112 as in step S26 described above, and the sleep state is set when the setting value of the low concentration flag is “1”. When the setting value of the low concentration flag is “0”, it is determined that the sleep state is not set. If the CPU 111 determines that the sleep state is set (step S30: Yes), the process proceeds to step S31. If the CPU 111 determines that the sleep state is not set (step S30: No), the main process step is performed. The process proceeds to S7. When there are a plurality of own users, the low concentration level flag is referred to for each of the own users, and the number of users whose low concentration level setting value is set to “1” is predetermined (for example, 60%) or more. When it is, it is determined that the sleep state is set.

  In step S31, the CPU 111 determines whether or not the awakening control is set. Whether or not the awakening control is set is determined by referring to the awakening control flag stored in the storage unit 112. When the set value of the awakening control flag is “1”, it is determined that the awakening control is set. When the set value of the awakening control flag is “0”, it is determined that the awakening control is not set. If the CPU 111 determines that the awakening control is set (step S31: Yes), the process proceeds to step S32. If the CPU 111 determines that the awakening control is not set (step S31: No), the processing proceeds to step S33. Transition.

  In step S <b> 32, the CPU 111 executes a wake-up control process for performing a wake-up control that prompts the user to wake up. That is, the CPU 111 controls the display unit 150 to control the speaker 122 of the headset 120 while blinking the display screen or displaying a screen (for example, blinking display) prompting awakening as shown in FIG. The speaker 122 increases the output of the speaker 122 or outputs a message prompting awakening, and the process proceeds to step S7 of the main process. Thereby, since the self-user who is a sleep state can be awakened and a self-user can be returned to a meeting, a smooth communication conference can be performed.

  In step S33, the CPU 111 determines whether exit control is set. Whether or not the exit control is set is determined by referring to the exit control flag stored in the storage unit 112. When the set value of the exit control flag is “1”, it is determined that the exit control is set. When the set value of the exit control flag is “0”, it is determined that the exit control is not set. If the CPU 111 determines that the exit control is set (step S33: Yes), the process proceeds to step S34. If the CPU 111 determines that the exit control is not set (step S33: No), the main process step S6 is performed. The process is transferred to.

  In step S <b> 34, the CPU 111 executes an exit control process for leaving the own user from the conference. That is, the CPU 111 controls the display unit 150 to hide the image and the like displayed on the display unit 150 and controls the speaker 122 of the headset 120 to reduce the output of the speaker. The process proceeds to S6.

  Next, the “concentration information reception process” in step S8 of the main process will be described in detail with reference to FIG. Note that the display mode of the display unit according to the present embodiment is different from the general mode in which the presentation material or other user's video is always displayed, and the presentation material and the other user's video according to the amount of speech of the own user. This will be described as a presentation mode for sequentially switching and displaying.

[Processing when receiving concentration information]
When the concentration information reception process in step S8 of the main process is started, first, the CPU 111 determines whether or not a speech is made by the own user (step S35). Whether or not the user speaks is determined based on whether or not the microphone 121 of the headset 120 has collected the voice of the user for a predetermined period or longer. For example, when the microphone 121 of the headset 120 collects the voice of the own user for a period of 10 seconds or longer, it is determined that the user speaks. If the CPU 111 determines that the user has made a statement (step S35: Yes), the CPU 111 proceeds to step S36 and determines that the user has not made a statement (step S35: No). Then, the process proceeds to step S37.

  In step S <b> 36, the CPU 111 controls the display unit 150 to display an image based on the video of another user who is a communication partner and the concentration information of the other user. That is, the CPU 111 divides the main display area of the display unit 150 according to the number of communication terminal conference terminal devices 100 and forms a number of divided display areas according to the number of communication terminal conference terminal devices 100. The other user's video is displayed in this divided display area. In addition, the CPU 111 displays an image based on the concentration information of other users in the sub display area of the display unit 150.

  For example, as shown in FIG. 1, when a communication conference is performed by four conference terminal devices 100a to 100d, as shown in FIG. 6B, the CPU 111 of the conference terminal device 100a displays the display unit 150 of its own device. And the main display area is divided into four vertical and horizontal columns to form four divided display areas. Then, the CPU 111 displays the video of the user of the conference terminal device 100b in the lower right split display region, displays the video of the user of the conference terminal device 100c in the upper left split display region, and displays the conference terminal in the lower left split display region. The image of the user of the device 100d is displayed.

  Further, the CPU 111 displays an icon indicating the degree of concentration of other users in the sub display area. At this time, on the display screen of the display unit, if the received concentration level information is set to the concentration state, an icon shown as “c” in FIG. 7A is displayed, and the received concentration level information is displayed in the sleep state. If it is set, the icon shown in “d” of FIG. 7B is displayed, and when the concentration information is not received, the icon shown in “b” of FIG. 7A is displayed. In the case of a communication terminal conference terminal device in which a plurality of users exist, since the concentration level information is notified for each of all users, when a predetermined number or more of concentration level information for which the concentration state is set are received, The icon shown in “c” is displayed. When a predetermined number or more of sleep state settings in which the concentration state is set are received, the icon shown in “d” is displayed, and the icon shown in “b” is displayed otherwise. . Note that the sub display area may not be displayed by the user's setting operation. Further, in the case of a communication terminal conference terminal device having a plurality of users, an icon corresponding to the concentration degree information can be displayed for each of all the users. By doing in this way, the concentration degree of all the users of the communication terminal conference terminal device can be grasped individually and specifically.

  Thus, when the user is speaking, the video of the other user who is the communication partner is displayed on the display unit 150 of the device, so that the user can make an announcement while watching the state of the other user. You can speak and you can have a smooth communication meeting. When this process ends, the process moves to a step S38.

  In step S37, the CPU 111 controls the display unit 150 to display presentation materials and the like used in the conference as shown in FIG. 6, and the process proceeds to step S2 of the main process.

  In step S38, the CPU 111 determines whether or not to perform related display. Whether or not to perform the related display is determined by referring to the related display flag stored in the storage unit 112 for determining whether or not to perform the related display, and the setting value of the related display flag is “1”. Is determined to be related display, and when the setting value of the related display flag is “0”, it is determined not to be related display. Note that the process of storing the related display flag in the storage unit 112 may be performed in the concentration setting process in step S16 described above. When the CPU 111 determines that the related display is performed (step S38: Yes), the process proceeds to step S39. When the CPU 111 determines that the related display is not performed (step S38: No), the process proceeds to step S40.

  In step S39, the CPU 111 executes related display processing. In other words, the video based on the video data of the other user and the image indicating the concentrated state are displayed in association with each other. Specifically, the information on the average number of vertical movements of the head of the communication partner user transmitted by the conference terminal device 100 of the communication partner is referred to, and this user is displayed on the video of the user with the highest average number of times. An image indicating that it is in a concentrated state is displayed in association with it. For example, as shown in FIG. 17, the user of the conference terminal device 100c is in a concentrated state, and the average value of the number of times the head moves up and down in a predetermined period (the conference terminal device 100 of the communication partner having a plurality of users). In this case, when the average value of each user is added and divided by the number of users), the conference terminal device 100c has an area (upper left in the figure) in which the video of the user of the conference terminal device 100c is displayed. An icon indicating that the user is in a concentrated state is displayed. Thereby, the own user can see the concentration information of the other user displayed on the video of the other user. Further, the period of related display can be set to an arbitrary period by setting. This is to make it easier for the own user to grasp the status of other users.

  Further, in the case of the conference terminal device 100 of a communication partner having a plurality of users, the related display is performed based on the average value of the number of vertical movements of the head of a specific user among the users of the conference terminal device 100 by setting. Or related display based on the average value of the number of times the heads of all the users of the conference terminal device 100 move up and down.

  Next, in step S <b> 40, the CPU 111 determines whether to enlarge and display the user's video indicating the concentration state. Whether or not to perform the enlarged display is determined by referring to the enlarged display flag stored in the storage unit 112 for determining whether or not to perform the enlarged display, and the setting value of the enlarged display flag is “1”. Is determined to be enlarged display, and it is determined not to be enlarged when the set value of the enlarged display flag is “0”. Note that the process of storing the enlarged display flag in the storage unit 112 may be performed in the concentration level setting process in step S16 described above. If the CPU 111 determines that the enlarged display is performed (step S40: Yes), the process proceeds to step S41. If the CPU 111 determines that the enlarged display is not performed (step S40: No), the process proceeds to step S2 of the main process.

  In step S41, the CPU 111 executes an enlarged display process. That is, an image of a user having a particularly high degree of concentration in the concentrated state is displayed in an enlarged manner. Specifically, the average value of the vertical movement of the head of the communication partner user transmitted by the communication partner conference terminal device 100 (in the case of the communication partner conference terminal device 100 in which a plurality of users exist, the average of each user) The user image with the largest average value is displayed in a larger size than the image of the user of the other communication partner. For example, as shown in FIG. 18, when the user of the conference terminal apparatus 100c is in a concentrated state, the area for displaying the video of the user of the conference terminal apparatus 100c is enlarged, and the size of the video of the user is enlarged. Is displayed. Thus, the user can easily understand the state of the highly concentrated user by displaying the user of the conference terminal device 100c so as to be easily understood. Similarly to the related display, the period for the enlarged display can be set to an arbitrary period by setting. This is to make it easier for the own user to grasp the status of other users.

  In addition, when there are a plurality of users of the conference terminal device 100 as communication partners, an enlarged display is made based on the average value of the number of vertical movements of the head of a specific user among the users of the conference terminal device 100 by setting. Or may be enlarged and displayed based on the average value of the number of vertical movements of the heads of all users of the conference terminal device 100.

  When the video of the user of the conference terminal device 100c is enlarged and displayed, the videos of the users of the conference terminal devices 100c and 100d of the communication partners other than the user of the conference terminal device 100c are reduced, and the images of the users of all the communication partners are always reduced. You may make it display.

  The configuration and processing of the conference terminal device according to the first embodiment of the present invention has been described above.

  As described above, the conference terminal device 100 according to the present embodiment includes the accelerometer 123, the control unit 116 that functions as a concentration degree determination unit, and the communication IF 113. The movement of the user's head in the vertical direction (acceleration information) is detected, the control unit 116, which is a concentration determination means, determines the concentration level of the user based on the acceleration information, and the determination result (concentration information) is obtained. It is possible to transmit to other users who are communication partners by the communication IF 113.

  In addition, since the conference terminal apparatus 100 according to the present embodiment includes the control unit 116 that functions as a display control unit, the communication IF 113 receives the concentration information transmitted from another user who is a communication partner, and performs display control. The control unit 116 which is a means can perform display control according to the concentration level information based on the concentration level information received by the communication IF 113.

  Thereby, since the own user who is speaking can make a presentation according to the degree of concentration of other users who are communication partners, a smooth communication conference can be performed.

  In particular, since the accelerometer 123 is mounted on the head of the user, the vertical movement of the user's head can be detected directly and mechanically, thereby increasing the detection accuracy. be able to. That is, when the user's head is imaged by the imaging unit 140 or the like, and the image analysis is performed on the imaging result, the user's head is out of the imaging range or the user's head can be imaged. The detection of the movement of the head of the user is deteriorated due to the imaging from the direction in which the movement of the user's head cannot be detected. However, since the conference terminal apparatus 100 according to the present embodiment directly and mechanically detects the vertical movement of the head of the user's head, the detection accuracy can be increased.

  In addition, by performing predetermined settings in advance in the awakening control setting process (step S17) of the input process, when the CPU 111 determines that the user is in a sleep state, the speaker of the display unit 150 and the headset 120 122 is controlled to urge the user to wake up. Thereby, it is prevented that the user's consciousness of participating in the meeting is reduced.

  The present invention has been described above by taking the first embodiment as an example. However, the present invention is not limited to this embodiment, and various modifications can be made.

  Moreover, you may provide the control button which controls so that a concentration degree determination process and a concentration degree notification process may not be temporarily performed during a meeting in a meeting terminal device. By controlling with this control button, even if the user of each conference terminal device is “nodding” because it is different from the content of the conference, incorrect concentration information is not notified.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to the drawings.
[Outline of Conference System 2]
First, the outline of the conference system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 19 is a diagram showing a schematic configuration of a conference system according to the second embodiment of the present invention.

  As shown in FIG. 19, the conference system 2 according to the second embodiment performs a communication conference such as a video conference between remotely arranged conference terminal devices, and a plurality of conference terminal devices 200 a to 200 d. It is configured by being connected to the server apparatus 300 via a network. In the present embodiment, the conference system 2 includes the four conference terminal devices 200a to 200d, but is not limited to this. In the following, when any conference terminal device is shown among the conference terminal devices 200a to 200d, it is referred to as “conference terminal device 200”.

  In the conference system 2 configured as described above, a video conference communication conference is performed by exchanging video and audio between the conference terminal devices 200 a to 200 d via the server device 300. In the conference system 2, a voice conference communication conference can be performed by exchanging audio without exchanging video between the conference terminal devices 200a to 200d.

  Next, the outline | summary of the conference terminal device 100 and the server apparatus 300 which are the principal parts of the conference system 2 which concerns on 2nd Embodiment is demonstrated. First, an overview of the conference terminal device 200 will be described with reference to FIG. FIG. 20 is a diagram illustrating a schematic configuration of the conference terminal apparatus 200 according to the second embodiment.

[Outline of conference terminal device]
As shown in FIG. 20, the conference terminal device 200 includes an operation detection unit, a pulse measurement unit, a transmission unit, a reception unit, a display unit, and an imaging unit. Note that the operation detection unit, the pulse measurement unit, the display unit, and the imaging unit are the same as those in the first embodiment, and a description thereof will be omitted.

  The transmission unit transmits the motion information detected by the motion detection unit and the pulse of the own user measured by the pulse measurement unit to the server device 300. The transmission unit also functions as a video data transmission unit that transmits the video data output from the imaging unit to the server apparatus 300 via the network.

  The receiving unit receives the concentration information of each user transmitted by the server device 300. The receiving unit also functions as a composite video data receiving unit that receives the composite video data transmitted from the server device 300.

  Next, an overview of the server apparatus 300 will be described with reference to FIG. FIG. 21 is a diagram illustrating a schematic configuration of a server apparatus 300 according to the second embodiment.

[Outline of server device]
As shown in FIG. 21, the server device 300 includes operation information acquisition means, pulse information acquisition means, concentration degree determination means, video composition means, transmission means, and reception means. Note that the operation information acquisition unit, the pulse information acquisition unit, and the concentration level determination unit are the same as those in the first embodiment, and a description thereof will be omitted.

  The video synthesizing unit synthesizes the video data output by the imaging unit of each conference terminal device 200 and synthesizes image data corresponding to the degree of concentration information of each user with the synthesized video. Generate composite video data. As a result, a synthesized video obtained by synthesizing the video of the communication partner user is displayed in the main display area of the display unit 250 of the conference terminal device 200, and an image (icon) corresponding to the concentration degree information is displayed in the sub-display area. The In addition, the video composition unit can synthesize an image indicating that the user is in a concentrated state in association with an image corresponding to the video data transmitted from the conference terminal device 200 of the user having a degree of concentration of a predetermined level or more. . Further, the video synthesizing unit displays an image corresponding to the video data transmitted from the conference terminal device 200 of the user whose concentration degree is not less than a predetermined level, and the video transmitted from each conference terminal device 200 other than the conference terminal device 200 of the user. It is also possible to display on the display unit with a size larger than the image corresponding to the data.

  The receiving unit receives operation information and pulse information of each user of each conference terminal device (hereinafter referred to as “each user”) transmitted by each conference terminal device 200. The receiving unit also functions as a video data receiving unit that receives video data transmitted from each conference terminal device 200.

  The transmission unit transmits the concentration level information of each user to each conference terminal device based on the result of the determination made by the concentration level determination unit. The transmission unit also functions as a composite video data transmission unit that transmits the composite video data generated by the composite video generation unit to each conference terminal device 200 via the network.

  According to the conference terminal device 200 and the server device 300 of the conference system 2 described above, the concentration determination means of the server device 300 is based on the operation information of each user detected by the operation detection unit of each conference terminal device 200. The degree of concentration is determined, and the transmission unit of the server device 300 transmits the concentration level information of each user to each user, while the reception unit of each conference terminal device 200 receives the concentration level information of each user, Since the display control means of the terminal device 200 controls the display based on the concentration level information of each user received, each user can grasp the user's concentration level of the conference terminal device of the communication partner. it can.

In particular, since the motion detector is mounted on each user's head, the motion of each user's head can be detected with high accuracy, and the concentration level can be determined with high reliability.
[Concrete configuration of conference terminal device]
Next, the conference terminal device 200 according to the second embodiment will be described in detail with a specific example. FIG. 22 is a block diagram illustrating a specific configuration of the conference terminal apparatus according to the second embodiment. The conference terminal device 200 is equivalent to the conference terminal device 200 disposed at each site in FIG. 19, and is appropriately distinguished as conference terminal devices 200a to 200d. In this case, the conference terminal device 200a is referred to as the conference terminal device 200a. Assume that the conference terminal device on the speaker side and the conference terminal devices 200b to 200d are conference terminal devices on the viewer side.

  As shown in FIG. 22, the conference terminal device 200 includes a device main body 210, a headset 220, a pulse meter 230, an imaging unit 240, and a display unit 250. The difference from the conference terminal device 100 of the first embodiment is the type of program stored in the storage unit 212. The headset 220, the pulse meter 230, the imaging unit 240, and the display unit 250 are the same as the headset 120, the pulse meter 130, the imaging unit 140, and the display unit 150 according to the first embodiment. Omitted.

[Configuration of Device Main Body 210]
The apparatus main body 210 can be a general-purpose computer, and is a computer device that includes a CPU 211, a storage unit 212, a communication IF 213, an operation unit 214, and an image data input unit (not shown). These various components are connected by a system bus 215. The operation unit 214 is the same as the operation unit 114 according to the first embodiment, and a description thereof will be omitted.

  The storage unit 212 stores a terminal device control program. In addition, the storage unit 212 is provided with a storage area (not shown) for storing information related to the operation of the user's head during the meeting. The CPU 211 performs basic control as a computer of the conference terminal device 100 according to the terminal device control program stored in the storage unit 212. Thereby, the control unit 216 functions as a stimulus control unit and a speech determination unit.

  The terminal device control program may be downloaded to the storage unit 212 from, for example, a server connected to the Internet, or after being recorded on a recording medium such as a CD-ROM, the drive of the recording medium is stored. Via the storage unit 212.

  The communication IF 213 is the same as the communication IF 113 described above. This communication IF 213 provides the server apparatus 300 with communication data related to the own user such as video data obtained by imaging the own user, voice data obtained by collecting the own user's voice, operation information of the own user, and the pulse of the own user. Send. In addition, the communication IF 213 includes, as communication data related to each user transmitted by the server device 300, synthesized video data obtained by synthesizing the imaging of the user serving as the communication partner, voice data obtained by collecting the voice of the user serving as the communication partner, and each user. Receives concentration level information. The communication IF 113 functions as a transmission unit and a reception unit together with the control unit 116.

  The above is the description of the specific configuration of the conference terminal device 200 according to the second embodiment. Subsequently, a specific configuration of the server apparatus 300 according to the second embodiment will be described.

[Specific Configuration of Server Device 300]
Next, the server apparatus 300 according to the second embodiment will be described in detail with a specific example. FIG. 23 is a block diagram illustrating a specific configuration of the server apparatus according to the second embodiment.

  As shown in FIG. 23, the server apparatus 300 is a computer apparatus to which a general-purpose computer can be applied, and includes a CPU 311, a storage unit 312, a communication IF 313, an operation unit 314, and an image data input unit (not shown). These various components are connected by a system bus 315. The operation unit 314 is the same as the operation unit 114 described above, and thus the description thereof is omitted.

  The storage unit 312 stores setting data such as a server device control program, user data as a user name, a user's reference pulse, and various flags relating to a meeting, and further includes determinations for storing first and second determination tables. A table storage area (not shown) is provided.

  The CPU 311 performs basic control of the server device 300 as a computer according to the server device program stored in the storage unit 312. As a result, the control unit 316 includes a concentration level determination unit that determines the concentration level of each user, a motion information acquisition unit that acquires motion information from the accelerometer 223 of the conference terminal device 200, and a pulse from the pulse meter 230 of the conference terminal device 200. It functions as a pulse information acquisition unit that acquires information and a video synthesis unit that synthesizes video data of each user and generates a composite video.

  The server device program may be downloaded to the storage unit 112 from, for example, a server connected to the Internet, or after being recorded on a recording medium such as a CD-ROM, via the drive of the recording medium. Then, it may be read into the storage unit 112.

  The communication IF 313 is the same as the communication IF 113 described above. This communication IF 313 includes, as communication data related to each user transmitted by each conference terminal device 200, video data obtained by imaging each user, audio data obtained by collecting each user's voice, operation information and pulse information of each user, and the like. Receive. In addition, the communication IF 313 transmits, to each conference terminal device 200, as communication data related to each user, synthesized video data obtained by synthesizing video images of each user, audio data obtained by collecting each user's voice, and concentration of each user. Send degree information etc. The communication IF 313 functions as a transmission unit and a reception unit together with the control unit 316.

  The above is the description of the configuration of the conference terminal device 200 and the server device 300 according to the second embodiment of the present invention.

[Control processing]
Next, control processing of the CPU 211 of the conference terminal apparatus 200 and the CPU 311 of the server apparatus 300 according to the second embodiment of the present invention will be described with reference to FIGS.

[Main process of conference terminal apparatus 200]
First, main processing of the CPU 211 of the conference terminal apparatus 200 will be described. In the conference terminal device 200, the main process shown in FIG. 24 is executed from when the power is turned on to when it is turned off.

  First, the CPU 211 of the conference terminal apparatus 200 (hereinafter referred to as “CPU 211”) makes a connection request to the server apparatus 300 (step S42). That is, the CPU 211 controls the communication IF 213 and transmits a request signal for requesting connection between the conference terminal device 200 and the server device 300 to the server device 300.

  In step S43, the CPU 211 stands by while determining whether or not a response signal from the server device 300 has been received. When the CPU 211 determines that a response signal has been received (step S43: Yes), the process proceeds to step S44.

  Next, the CPU 211 performs connection processing between the conference terminal device 200 and the server device 300 (step S44), and then executes input processing such as user data registration and conference setting (step S45). Since this input process is the same as the description of FIG. 9 in the first embodiment, the description is omitted.

  Next, the CPU 211 performs input information transmission processing (step S46). That is, the CPU 211 controls the communication IF 213 and transmits the user data set in the input process of step S45 and the setting data related to the conference as input information to the server apparatus 300.

  Next, the CPU 211 stands by while determining whether or not a notification signal for notifying the completion of the setting related to the conference is received from the server device 300 (step S47), and determines that the setting completion notification is received (step S47: Yes), it is determined that the conference can be started, and the process proceeds to step S48.

  Next, the CPU 211 executes processing during the meeting (step S48). That is, the CPU 211 controls the imaging unit 240 to photograph the user, controls the microphone 221 of the headset 220 to collect the user's voice, and controls the accelerometer 123 of the headset 120 to control the user. The acceleration of the vertical movement of the head (motion information) is detected, and the pulse meter 130 is further controlled to measure the pulse of the user. At this time, based on the detected motion information, the CPU 111 calculates the vertical width of the head of the user's head, the number of times of the head's vertical movement, and the like, and stores these as information regarding the motion of the user's head. Store in the unit 212.

  Next, the CPU 211 controls the communication IF 213, and the communication IF 313 of the server apparatus 300 is recorded with respect to the video data of the user himself, the sound data of the sound of the user of the user, and information about the operation of the user's head. And the information of the pulse which the own user measured is transmitted (step S49).

  Next, the CPU 211 waits while determining whether or not the composite video data obtained by synthesizing each user's concentration information and each user's video data transmitted by the communication IF 313 of the server device 300 has been received (step S50). ). When the CPU 211 determines that the composite video data, the concentration information of each user, and the like have been received (step S50: Yes), the CPU 211 executes a stimulus control process (step S51).

  Next, the CPU 211 wakes up the user to wake up or exits the conference from the conference based on the concentration information of the user in the concentration information of each user received in step S52. A stimulus control process is performed (step S51). This stimulus control process is the same as the description of FIG. 12 in the first embodiment, and a description thereof will be omitted.

  Next, the CPU 211 determines whether or not to end the conference (step S52). When the CPU 211 determines to end the conference (step S52: Yes), the process proceeds to step S53 while determining to end the conference (step S52). Step S52: No), the process is returned to step S47.

  In step S53, the CPU 211 controls the communication IF 213 and requests the server apparatus 300 to end the conference and disconnect the communication between the conference terminal apparatus 200 and the server apparatus 300 (hereinafter, “conference end”). A notification signal "is transmitted, a conference end notification is sent, and the communication between the conference terminal device 200 and the server device 300 is disconnected (step S54).

[Main processing of server device 300]
Next, main processing of the CPU 311 of the server apparatus 300 will be described. In the server apparatus 300, the main process shown in FIG. 25 is executed from when the power is turned on to when it is turned off.

  First, the CPU 311 of the server device 300 (hereinafter referred to as “CPU 311”) waits while determining whether or not there is a connection request from the conference terminal device 200 (step S55), and the connection request from the conference terminal device 200. If it is determined that there is (step S55: Yes), the process proceeds to step S56.

  Next, the CPU 311 controls the communication IF 313 to transmit a response signal to the request signal from the conference terminal device 200 to the conference terminal device 200 (step S56), and then the server device 300 and the conference terminal device 200. Connection is made (step S57).

  Next, the CPU 311 stands by while determining whether or not the setting data from the conference terminal device 200 has been received (step S58). If it is determined that the setting data has been received (step S58: Yes), the process proceeds to step S59. .

  Next, the CPU 311 sets the setting data transmitted by the conference terminal apparatus 200 as data used for the conference and stores it in the storage unit 312 (step S59), and the process proceeds to step S60.

  In step S60, the CPU 311 controls the communication IF 313 to transmit a notification signal for notifying the conference terminal apparatus 200 of the completion of the setting related to the conference, and the process proceeds to step S61.

  Next, the CPU 311 waits while determining whether or not the video data, audio data, operation information, and pulse information from the conference terminal device 200 have been received (step S61), and determines that it has been received (step S61: Yes), the process proceeds to step S62.

  In step S62, the CPU 311 executes a concentration degree determination process for determining the degree of concentration of each user in the meeting based on the operation information and pulse information of each user who is the user of each meeting terminal device 200 received in step S61. Then, the process proceeds to step S62. Note that this concentration level determination processing is the same as the description of FIG. 10 in the first embodiment, and a description thereof will be omitted.

  Next, the CPU 311 generates a composite video for each conference terminal device to be transmitted by synthesizing the video of the user of each conference terminal device transmitted by each conference terminal device 200, and further, the concentration degree information of each user Is synthesized so as to be added to the synthesized video (step S63).

  For example, as shown in FIG. 20, four conference terminal devices 200a to 200d are connected to the server device 300 to perform a communication conference, and the server device 300 transmits composite video data to the conference terminal device 200a therein. In this case, as shown in FIG. 6B, the CPU 311 divides the main display area into four, and forms four divided display areas arranged in two columns. Then, the CPU 311 displays the video of the user of the conference terminal apparatus 200b in the lower right split display area, displays the video of the user of the conference terminal apparatus 200c in the upper left split display area, and displays the video of the user of the conference terminal apparatus 200d. Is displayed in the lower left split display area, the video of the users of the conference terminal devices 200b to 200d is synthesized. Further, as shown in FIG. 7A, the CPU 311 synthesizes an image so that an icon indicating the concentration level of each user is added to the sub display area for display.

  In addition, the CPU 311 associates an image based on the degree-of-concentration information with an image based on the video data, synthesizes the video, or enlarges and displays the video of the user indicating that the user is in a concentrated state by the operation of each user Or synthesize. For example, as shown in FIG. 17, an icon indicating the concentration level of the concentrated user is displayed in association with the user's video, or the size of the concentrated user's video is displayed as shown in FIG. It is displayed larger than the size of.

  Next, the CPU 311 executes a composite video transmission process (step S64). That is, the CPU 311 transmits the composite video data generated in step S63 and the concentration level information of each user to each conference terminal device 200. For example, as shown in FIG. 20, when a communication conference is performed by four conference terminal devices 200 and the server device 300, the CPU 311 synthesizes the video of the users of the conference terminal devices 200b to 200d to the conference terminal device 200a. The synthesized video data and the user concentration information of the conference terminal devices 200a to 200d are transmitted.

  Next, the CPU 311 waits while determining whether or not the conference end notification signal has been received from the conference terminal device 200 (step S65), and determines that the conference end notification signal has been received (step S65: Yes). Communication between server device 300 and conference terminal device 200 is disconnected (step S66), and the conference is terminated.

  The configurations and processes of the conference terminal device and the server device according to the second embodiment of the present invention have been described above. As described above, in the conference terminal device 200 and the server device 300 according to the present embodiment, the headset 220 of each conference terminal device 200 is provided with the accelerometer 223 and the pulse meter 230, so that each conference terminal device has the conference terminal. Operation information and pulse information of the user of the device are acquired.

  In addition, since the conference terminal device 200 includes the communication IF 213 and the server device 300 includes the communication IF 313, operation information, pulse information, and concentration level information are transmitted and received between the conference terminal device 200 and the server device 300. Is done.

  Furthermore, since the server apparatus 300 is provided with the concentration degree control means (CPU 311), the degree of concentration of each user who is a user of each conference terminal apparatus is determined.

  In addition, since the conference terminal device 200 is provided with the display control means (CPU 211), display control based on the determined degree-of-concentration information is performed.

  Thus, according to the conference system 2 of the present embodiment, the acceleration of the vertical movement of the user's head of each conference terminal device detected by the accelerometer 123 of the headset 120 of each conference terminal device 200 and the pulse meter 130 are Based on the measured pulse of the user of each conference terminal device, the concentration degree determining means (CPU 311) of the server device 300 determines the concentration level of the user of each conference terminal device, and the display unit 250 of each conference terminal device 200 is concentrated. Since the display control is performed according to the degree of concentration of each user who is a user of each device determined by the degree determination means, each user can set the degree of concentration of other users who are users of other devices to the conference. In the case of making a presentation within a conference, it is possible to proceed with the presentation according to the degree of concentration of other users, and a smooth communication conference can be performed.

  In particular, since the accelerometer 223 is mounted on the head of the user, the vertical movement of the user's head can be detected directly and mechanically, so that the detection accuracy is increased. be able to.

  The present invention has been described with reference to the first and second embodiments. However, the present invention is not limited to this embodiment, and various modifications can be made.

  For example, each unit such as the concentration degree determination unit and the display control unit does not need to be arranged in the server device or one conference terminal device, and may be arranged in any one of the entire system.

  Further, in the case of a conference terminal device having a plurality of users, the concentration level information of each user is notified to the conference terminal device of the communication partner, but the number of users whose concentration state is set is a predetermined number or more. The concentration state notification process may be performed at the time, and the sleep state notification process may be performed when the number of users whose sleep state is set is equal to or greater than a predetermined number.

It is a figure which shows schematic structure of the conference system which concerns on the 1st Embodiment of this invention. It is a figure which shows schematic structure of the conference terminal device which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the specific structure of the conference terminal device which concerns on the 1st Embodiment of this invention. It is a figure which shows the schematic structural example of the headset which concerns on the 1st Embodiment of this invention. It is a figure which shows the operation | movement of the up-down direction of the own user's head. It is a figure which shows the main display area of the display part which concerns on the 1st Embodiment of this invention. It is a figure which shows the sub area | region of the display part which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the main process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the input process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the concentration degree determination process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the concentration degree notification process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the stimulus control process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a flowchart which shows an example of the information reception process in the conference terminal device which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the 1st determination table which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the 2nd determination table which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the display screen of the display part which carried out the awakening control. It is a figure which shows an example of the display screen which linked and displayed concentration degree information on the image | video of each user. It is a figure which shows an example of the display screen which expanded and displayed the image | video of the user with high concentration degree. It is a figure which shows schematic structure of the conference system which concerns on the 2nd Embodiment of this invention. It is a figure which shows schematic structure of the conference terminal device which concerns on the 2nd Embodiment of this invention. It is a figure which shows schematic structure of the server apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the specific structure of the conference terminal device which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the specific structure of the server apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows an example of the main process in the conference terminal device which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows an example of the main process in the server apparatus which concerns on the 2nd Embodiment of this invention.

Explanation of symbols

1, 2 Conference system 100, 200 Conference terminal device 110, 210 Device main body 111, 211, 311 CPU
112, 212, 312 Storage unit 113, 213, 313 Communication IF
114, 214 Operation unit 115, 215, 315 System bus 116, 216, 316 Control unit 117, 217 Concentration notification presence / absence input unit 118, 218 Stimulation control setting unit 120, 220 Head set 121, 221 Microphone 122, 222 Speaker 123, 223 Accelerometer 130, 230 Pulse meter 140, 240 Imaging unit 150, 250 Display unit 300 Server device

Claims (15)

In the conference terminal device in the conference system for transmitting and receiving at least one of video and audio data between the conference terminal devices via the network,
An imaging unit that images the user participating in the conference;
Video data transmission means for transmitting video data output from the imaging unit to a conference terminal device of a communication partner via the network;
Video data receiving means for receiving video data transmitted from the conference terminal device;
The display unit displays an image according to the video data received by the video data receiving unit, and the head motion information detected by the motion detection unit mounted on the head of the user of the user device. Operation information acquisition means to acquire;
Based on the motion information of the head of the own user acquired by the motion information acquisition means, concentration level determination means for determining the level of concentration of the user in the meeting;
Transmitting means for transmitting information indicating the degree of concentration of the user to the conference based on the result of the determination made by the concentration level determining means to the conference terminal device of the communication partner;
Receiving means for receiving information indicating a degree of concentration in a conference of another user who is a user of the conference terminal device of the communication partner from the conference terminal device of the communication partner;
And a display control means for performing display in accordance with information indicating the degree of concentration of the other user in the meeting received by the receiving means. In the conference terminal device in the conference system for transmitting and receiving at least one of video and audio data between the conference terminal devices via the network,
Motion information acquisition means for acquiring motion information of the head detected by the motion detection unit mounted on the head of the user's own user,
Based on the motion information of the head of the own user acquired by the motion information acquisition means, concentration level determination means for determining the level of concentration of the user in the meeting;
Transmitting means for transmitting information indicating the degree of concentration of the user to the conference based on the result of the determination made by the concentration level determining means to the conference terminal device of the communication partner;
Receiving means for receiving information indicating a degree of concentration in a conference of another user who is a user of the conference terminal device of the communication partner from the conference terminal device of the communication partner;
And a display control means for performing display in accordance with information indicating the degree of concentration of the other user in the meeting received by the receiving means.   The display control means displays an image indicating that the other user is in a concentrated state on the display unit in association with the image corresponding to the video data transmitted from the conference terminal device of the other user having a concentration level equal to or higher than a predetermined level. The conference terminal device according to claim 1, wherein:   The display control means displays an image corresponding to the video data transmitted from the conference terminal device of another user whose degree of concentration is a predetermined level or more, and an image corresponding to the video data transmitted from the conference terminal device of another communication partner. The conference terminal device according to claim 1, wherein the conference terminal device displays a larger size on the display unit. Provided with a speech determination means for determining whether or not a speech by the user is being performed,
The display control means performs display according to the information indicating the degree of concentration of other users received by the reception means when the speech determination means determines that a speech is being made by the own user, The display according to the information indicating the degree of concentration of other users received by the receiving unit is not performed when the speech determining unit determines that the user does not speak. 5. The conference terminal device according to any one of items 1 to 4. It has a pulse measurement unit that measures the pulse of the user,
The concentration degree determining means is based on the own user's pulse measured by the pulse measuring unit in addition to the operation information of the own user's head detected by the operation detecting unit, and is concentrated on the own user's meeting. The conference terminal apparatus according to claim 1, wherein the determination of the degree is performed.   The concentration degree determination means calculates a vertical swing width of the head based on the motion information acquired by the motion information acquisition means, detects the number of vertical movements of the head in a predetermined period, and The conference terminal device according to claim 1, wherein the degree of concentration of the own user is determined based on a swing width and the number of vertical movements of the head.   The concentration degree determination means calculates a vertical swing width of the head based on the motion information acquired by the motion information acquisition means, detects the number of vertical movements of the head in a predetermined period, and the pulse measurement unit The difference between the pulse measured by the reference pulse and the reference pulse is calculated, and the concentration of the user is determined based on the vertical deflection of the head, the number of vertical movements of the head, and the difference between the pulse and the reference pulse. The conference terminal apparatus according to claim 6, wherein the degree is determined. A concentration level notification presence / absence input unit that receives input from the own user whether to transmit information indicating the concentration level of the own user to the conference terminal device of the communication partner;
The transmission means receives information indicating the degree of concentration of the own user to the conference terminal device of the communication partner when the input to transmit to the conference terminal device of the communication partner is received by the concentration notification presence / absence input unit. When the input indicating that transmission is not performed to the communication partner conference terminal device is received by the concentration notification presence / absence input unit, information indicating the concentration level of the own user is transmitted to the conference terminal device of the communication partner. The conference terminal device according to claim 1, wherein the conference terminal device is not transmitted to the conference terminal device.   10. The apparatus according to claim 1, further comprising a stimulus control unit that performs control to give a stimulus to the user by light or sound when the concentration level of the user determined by the concentration determination unit is low. The conference terminal device according to any one of the preceding claims. A stimulus control setting unit for accepting a setting as to whether or not to give a stimulus to the own user in a state of low concentration from the own user;
When the stimulation control unit is set to give a stimulus to the user by the stimulation control setting unit, the concentration control unit determines that the concentration level of the user is low by the concentration level determination unit. While the control for giving a stimulus to the user by sound is performed, and the setting to not give the user a stimulus by the stimulus control setting unit is made, the concentration degree of the own user is low by the concentration degree determination means If it is determined that it is in a state, it does not perform control for stimulating the user with light or sound,
While the transmission means is set to not give a stimulus to the user by the stimulus control setting unit, the transmission means does not transmit information indicating the concentration level of the user to the conference terminal device of the communication partner. The information indicating the concentration degree of the user is transmitted to the conference terminal device of the communication partner when the stimulus control setting unit is set to give a stimulus to the user. 10. The conference terminal device according to 10.   The program for functioning a computer as each means in the conference terminal device of any one of Claim 1 to 11. A communication conference support method for supporting a communication conference in which at least one of video and audio data is mutually transmitted and received between conference terminal devices via a network,
Obtaining head motion information detected by a motion detector mounted on the head of the user, the user of the device;
Determining the concentration of the user in the meeting based on the acquired motion information of the head of the user,
Transmitting information indicating the degree of concentration of the user to the conference based on the result of the determination to the conference terminal device of the communication partner;
Receiving information indicating the degree of concentration in the conference of another user who is a user of the conference terminal device of the communication partner from the conference terminal device of the communication partner;
And displaying according to the received information indicating the degree of concentration of the other user in the conference. In a conference system that transmits and receives at least one of video and audio data between conference terminal devices via a network,
Each of the conference terminal devices
Motion information acquisition means for acquiring motion information of the head detected by the motion detection unit mounted on the head of the user's own user,
Based on the motion information of the head of the own user acquired by the motion information acquisition means, concentration level determination means for determining the level of concentration of the user in the meeting;
Transmitting means for transmitting information indicating the degree of concentration of the user to the conference based on the result of the determination made by the concentration level determining means to the conference terminal device of the communication partner;
Receiving means for receiving information indicating a degree of concentration in a conference of another user who is a user of the conference terminal device of the communication partner from the conference terminal device of the communication partner;
And a display control means for performing display in accordance with information indicating the degree of concentration of other users in the meeting received by the receiving means. In a server device connected to a plurality of conference terminal devices via a network and managing a conference between the conference terminal devices,
Motion information acquisition means for acquiring head motion information detected by the motion detector mounted on the head of each user of the plurality of conference terminal devices;
Based on the motion information of the user's head acquired by the motion information acquisition means, concentration level determination means for determining the level of concentration of the user in the meeting;
A server device comprising: a transmission unit configured to transmit information indicating a concentration level based on a result of the determination performed by the concentration level determination unit to the plurality of conference terminal devices.

Images