JP7596105B2 - Viewing state estimation device, robot system, viewing state estimation method, and viewing state estimation program - Google Patents

Description

The present invention relates to an apparatus, method, and program for estimating the state of a television viewer.

2. Description of the Related Art Conventionally, for a robot that watches a television program or other video together with a viewer, technology has been researched for controlling the movement of the robot in accordance with the viewing state of the viewer.
Techniques for estimating the viewing state include, for example, a method of installing a camera to observe in a room where a television is being watched in order to detect the direction in which the viewer is facing, or a method of having the viewer wear a glasses-type gaze direction acquisition device to acquire gaze direction data of the viewer.

Furthermore, as a control according to the estimated viewing state, for example, Patent Document 1 proposes an apparatus that detects the user's line of sight, determines the display position for projecting an image from an image projection device, and performs geometric correction on the displayed image to display it, thereby displaying an image that is easy for the user to see.
Moreover, Patent Document 2 proposes a device that detects a viewing state from an image including a viewer viewing content, and associates the state with the elapsed time from the start of the content being viewed.

JP 2017-55178 A Patent No. 6614547

Among conventional methods for estimating the viewing state, a method for estimating the gaze direction from the viewer's video in a specific location, such as a laboratory with cameras installed on the ceiling, wall, or above the television, is only carried out experimentally for a certain period of time, and therefore it is difficult to estimate the viewer's gaze direction in an everyday viewing environment.
In addition, the method of having the viewer wear a glasses-type gaze direction acquisition device on their head is different from everyday viewing conditions, and wearing the device creates a sense of discomfort and puts a strain on the viewer. Therefore, this method is also difficult to use in everyday viewing environments.

In the method of Patent Document 1, imaging devices are installed at the four corners of the ceiling of a room, and the user's gaze direction is estimated based on the captured images acquired from the imaging devices. In this case, it is necessary to install imaging devices in the room, but it is difficult to install them in a home. In addition, multiple imaging devices are required to capture an image of the entire room.

In the method of Patent Document 2, the viewing state of a viewer is determined based on vital information of the viewer extracted from a camera image that includes the viewer, but the camera needs to be installed above the display, etc., which differs from everyday viewing conditions. In addition, this method cannot obtain information such as when a viewer is facing another person and not the display while talking, and it also reflects vital information when the viewer is not actually watching, making it impossible to determine the appropriate viewing state.

The present invention aims to provide a viewing state estimation device, a viewing state estimation method, and a viewing state estimation program that can estimate the viewing state of a viewer using a robot without using additional devices such as cameras.

The viewing state estimation device according to the present invention includes a panoramic image unit that acquires an omnidirectional panoramic image synthesized from images captured around the robot, a distance panoramic image unit that generates a distance panoramic image in which distance data corresponding to each pixel of the panoramic image is used as a pixel value, a television detection unit that detects the position of the television from the panoramic image, a viewer detection unit that detects the position of the viewer's face from the panoramic image, a distance acquisition unit that acquires the distances at the television position and the face position from the distance panoramic image, a viewing direction detection unit that calculates the angle between the television and the viewer as seen by the robot based on the size of the panoramic image, the television position, and the face position to identify the positional relationship between the robot, the viewer, and the television, and acquires a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the image of the face position, and a viewing state determination unit that detects an object included in the viewing direction image and determines the state of the viewer based on the type of the object.

The viewing direction detection unit may approximate the distance from the viewer to the viewing direction position by the sum of the distance from the robot to the viewer and the distance from the robot to the viewing direction position.

The viewing state determination unit may calculate the viewing state based on statistical information about the viewer's state within a certain period of time.

The viewing state determination unit may calculate a viewing degree indicating the proportion of the television being watched as the viewing state.

The viewing state determination unit may determine a plurality of viewing states, including a state of watching the television and a state of watching someone else, as the viewing state.

The robot system according to the present invention includes the viewing state estimation device, and an operation control unit that controls the operation of the robot based on the result of comparing the viewing degree output from the viewing state estimation device with a predetermined threshold value.

The operation control unit may gradually change the control of the robot based on multiple thresholds.

The robot system according to the present invention includes the viewing state estimation device and an operation control unit that controls the operation of the robot in accordance with the classification of the plurality of states output from the viewing state estimation device.

The viewing state estimation method according to the present invention includes a panoramic image generation step of acquiring an omnidirectional panoramic image synthesized from images of the robot's surroundings, a distance panoramic image generation step of generating a distance panoramic image in which distance data corresponding to each pixel of the panoramic image is used as a pixel value, a television detection step of detecting a television position from the panoramic image, a viewer detection step of detecting a face position of the viewer from the panoramic image, a distance acquisition step of acquiring the distances at the television position and the face position from the distance panoramic image, a viewing direction detection step of calculating the angle between the television and the viewer as seen by the robot based on the size of the panoramic image, the television position, and the face position, thereby identifying the positional relationship between the robot, the viewer, and the television, and acquiring a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the image of the face position, and a viewing state determination step of detecting an object included in the viewing direction image and determining the state of the viewer based on the type of the object.

The viewing state estimation program according to the present invention is for causing a computer to function as the viewing state estimation device.

According to the present invention, the robot can estimate the viewer's viewing state without using additional cameras or other devices.

FIG. 1 is a diagram illustrating a usage scene of a robot incorporating a viewing state estimating device according to an embodiment. 1 is a block diagram showing a functional configuration of a viewing state estimation device according to an embodiment. 10A and 10B are diagrams illustrating example synthesis position data according to an embodiment. FIG. 4 is a diagram illustrating a functional configuration of a distance panorama image unit in the embodiment. 6A to 6C are diagrams illustrating an example of the operation of a distance image synthesis unit in the embodiment. 11 is a diagram illustrating an example of the operation of an overlapping section calculation unit in the embodiment. FIG. FIG. 4 is a diagram illustrating an example of a distance image database in the embodiment. FIG. 4 is a diagram illustrating a distance panorama image database in the embodiment. FIG. 4 is a diagram illustrating a functional configuration of a viewer detection unit in the embodiment. FIG. 2 is a diagram illustrating a functional configuration of a viewing direction detection unit in the embodiment. FIG. 2 is a diagram showing the positional relationship between a robot, a television, and a viewer in an embodiment. 10A and 10B are diagrams for explaining a method for calculating an angle between a television and a viewer in an embodiment. 4A and 4B are diagrams illustrating a face direction angle according to an embodiment. FIG. 2 is a diagram showing the positional relationship between a robot, a viewer, and a viewing direction position in the embodiment. 10A and 10B are diagrams illustrating a method for acquiring a viewing direction image in an embodiment. FIG. 2 is a diagram illustrating a functional configuration of a viewing state determination unit in the embodiment. FIG. 11 is a diagram showing an example of calculation of a viewer rating in the embodiment.

An example of an embodiment of the present invention will now be described.
FIG. 1 is a diagram illustrating a usage scene of a robot 1 incorporating a viewing state estimating device 10 according to the present embodiment.

The robot 1 is placed, for example, on a table near a viewer watching television. In addition to the viewing state estimation device 10, the robot 1 is equipped with an imaging unit 20 and a distance detection unit 30, and further equipped with an operation control unit 40 that performs operations such as speaking in accordance with the viewing state estimated by the viewing state estimation device 10.

The viewing state estimation device 10 obtains image data of the robot's surroundings from the imaging unit 20 and distance data from the distance detection unit 30, and calculates the distance from the robot 1 to the television, the distance from the robot 1 to the viewer, and the viewer's viewing direction using a method described below, and further obtains an image of the viewer's viewing direction to estimate the viewing state.

The imaging unit 20 is a camera mounted on the robot 1 for acquiring images, and may be configured to rotate horizontally by a motor and capture images of the surroundings of the robot 1 .
The imaging unit 20 is not limited to a method of acquiring images while rotating, and may acquire images by mounting a camera array consisting of multiple cameras on the robot 1, for example.

The distance detection unit 30 acquires distance data based on the infrared light emitted from the irradiation unit and the light reflected from the object and reaching the light receiving unit. The infrared irradiation method is, for example, a pattern method or a TOF (Time Of Flight) method. When an image is acquired by the imaging unit 20, the distance detection unit 30 acquires distance data in a direction corresponding to each pixel of the image in time synchronization.

Here, the viewing state indicates a category of viewing direction, such as watching television, watching someone else, or watching something else.
The operation control unit 40 changes the operation, such as speech, of the robot 1 according to the viewing state estimated by the viewing state estimation device 10 .

FIG. 2 is a block diagram showing a functional configuration of the viewing state estimating device 10 according to the present embodiment.
The viewing state estimation device 10 is an information processing device (computer) equipped with a control unit, a memory unit, an input/output interface, etc., and operates as the following functional units by the control unit executing software (viewing state estimation program) stored in the memory unit.
The viewing state estimation device 10 includes a panoramic image section 11 , a distance panoramic image section 12 , a viewer detection section 13 , a television detection section 14 , a distance acquisition section 15 , a viewing direction detection section 16 , and a viewing state determination section 17 .

The panoramic image unit 11 superimposes multiple images of the surroundings of the robot 1 acquired by the imaging unit 20 to generate a horizontal omnidirectional panoramic image as seen from the robot 1. Note that the open source OpenCV Stitcher class can be used as software for generating the panoramic image, but the generation method is not limited to this.

The panoramic image unit 11 also outputs synthesis position data Dc when a plurality of images are synthesized.
FIG. 3 is a diagram illustrating the composite position data Dc in this embodiment.
The synthesis position data Dc is composed of the numbers of the multiple images that are stitched together when generating a panoramic image, and the coordinates of the synthesis position.
For example, the coordinates (150, 10) of the first image are pasted onto the coordinates (0, 0) of the second image, and the coordinates (0, 0) of the first image are pasted onto the coordinates (x _n , _yn ) of the nth image.
In this example, the numbers of the distance images acquired synchronously are associated with each other.

The distance panoramic image unit 12 uses the distance data acquired by the distance detection unit 30 to generate a distance panoramic image in which the distance data is used as pixel values, similar to the panoramic image generated by the panoramic image unit 11.

FIG. 4 is a diagram showing the functional configuration of the distance panoramic image unit 12 in this embodiment.
The distance panorama image section 12 includes a distance image synthesis section 121 and an overlapping section calculation section 122, and receives the synthesis position data Dc and outputs distance panorama image data Dp.

The distance image synthesis unit 121 uses the synthesis position data Dc input from the panoramic image unit 11 to synthesize multiple distance images obtained from the distance detection unit 30 at the same position as the multiple images synthesized by the panoramic image unit 11.
Here, distance data corresponding to each pixel of the captured image is stored in distance image database 12 A. Note that distance image database 12 A may be stored in a storage unit of viewing state estimation device 10, or may be provided in a common storage unit accessible to each unit of robot 1.

The overlapping section calculation unit 122 averages a plurality of overlapping distance data in a section (pixel) where the combined distance images overlap, and determines the distance data for each pixel.
The method of determining the distance data is not limited to this, and data from any of the distance images may be determined as a representative.

FIG. 5 is a diagram illustrating the operation of the distance image synthesis unit 121 in this embodiment.
For example, if images 1 and 2 are acquired in sequence by the imaging unit 20 and image 2 is synthesized at synthesis position P _c1 (150,0) in image 1, the distance image synthesis unit 121 also synthesizes distance image 1 and distance image 2, which were acquired simultaneously with images 1 and 2, and maps the distance data acquired from the distance image database 12A to each pixel.

FIG. 6 is a diagram illustrating an example of the operation of the overlapping section calculation unit 122 in this embodiment.
5, suppose that the distance data at coordinate _P1 (150,159) in distance image 1 is 1.50 and the distance data at coordinate _P2 (0,159) in distance image 2 is 1.60. In this case, the overlapping section calculation unit 122 determines the average value (1.50+1.60)/2=1.55 as the distance data at position (150,159) in the distance panorama image.
Alternatively, the overlapping section calculation section 122 may use the distance data 1.60 of P ₂ (0,159) in distance image 2 as the distance data for the distance panoramic image.

Similarly, the overlapping section calculation unit 122 determines distance data corresponding to each pixel for the entire overlapping section.
The determined distance data is stored in the distance panorama image database 12B in the storage unit.

FIG. 7 is a diagram illustrating an example of distance image database 12A in this embodiment.
In distance image database 12A, for each distance image number _{n_d} , a position in the distance image (coordinates _{x_d} , _{y_d} ) is associated with distance data _{l_d} , and distance data for each pixel in each distance image is stored.

FIG. 8 is a diagram illustrating an example of the distance panoramic image database 12B in this embodiment.
The distance panorama image database 12B stores distance panorama image data that is composed of an image number _np , a position (coordinates _xp , _yp ) of a panorama image, and distance data _lp .

The viewer detection unit 13 detects the face position of the viewer from the panoramic image generated by the panoramic image unit 11 and obtains the face position on the panoramic image.

FIG. 9 is a diagram showing the functional configuration of the viewer detection unit 13 in this embodiment.
The viewer detection unit 13 includes a person detection unit 131 and a face detection unit 132 .

The human detection unit 131 detects humans from the panoramic image acquired from the panoramic image unit 11. As a means for detecting humans, for example, open source software such as OpenCV (Haarcascade detector full body model) or Faster-RCNN can be used, but the detection method is not limited to these.
If the human detection unit 131 detects a human, it outputs a viewer flag f _h =1, and if it does not detect a human, it outputs a viewer flag f _h =0.

When the person detection unit 131 detects a person, i.e., a viewer ( _fh = 1), the face detection unit 132 performs face detection from the panoramic image generated by the panoramic image unit 11 and acquires the face position on the panoramic image. On the other hand, when no viewer is detected ( _fh = 0), the face detection unit 132 does not perform face detection and proceeds to the process of acquiring a new image from the imaging unit 20 to increase speed.
As a means for face detection, for example, open source software such as OpenCV (Haar-cascade detector) or OpenFace can be used, but the detection method is not limited to these.

Here, the face position is determined by detecting the face portion in a rectangular frame, and the face detection unit 132 calculates a center point _Pf ( _xf , _yf ) from the start point _Psf ( _xsf , _ysf ) and end point _Pef ( _xef , _yef ) of the rectangular frame. The center point _Pf of the face is used to obtain the distance of the face position.
The face detection unit 132 outputs a face flag f _f =1 if a face is detected, and outputs a face flag f _f =0 if a face is not detected.

The television detection unit 14 detects a television from the panoramic image generated by the panoramic image unit 11, and obtains the position of the television on the panoramic image.
As a means for detecting a television, for example, open source software such as Faster-RCNN can be used, but the detection method is not limited to this.

Here, the television position is detected by detecting the television portion as a rectangular frame, and the television detection unit 14 calculates a center point _Ptv ( _xtv , _ytv ) from the start point _Pstv ( _xstv , ystv) and end point _Petv ( _xetv , _yetv ) of the _rectangular frame. The television center point _Ptv is used to obtain the distance to the television position.
The television detection unit 14 outputs a television flag f _tv =1 if a television is detected, and outputs a television flag f _tv =0 if a television is not detected.

The distance acquisition unit 15 acquires distance data for each position of the detected viewer and television from the distance panorama image.
Specifically, the distance acquisition unit 15 acquires the face center point _Pf from the face detection unit 132 and the television center point _Ptv from the television detection unit 14, and acquires the distance _df from the robot 1 to the viewer's face and the distance _dtv from the robot 1 to the television from the distance data stored in the distance panorama image database 12B.
For example, when the face center point _Pf is (1000, 200), distance data ld at point (1000, 200) in the distance panoramic image is obtained, and _df = _ld . When the television center point _Ptv is (3000, 150), distance data _ld at point (3000, 150) in the panoramic distance image is obtained, and _dtv = _ld .

The viewing direction detection unit 16 acquires the distance from the robot 1 to the viewer's face, the distance from the robot 1 to the television, and the angle between the viewer and the television as seen by the robot 1, thereby determining the relative positions of the robot 1, the viewer, and the television, and acquires the viewing direction and viewed image from the angle of the viewer's face.

FIG. 10 is a diagram showing the functional configuration of the viewing direction detection unit 16 in this embodiment.
The viewing direction detection unit 16 includes a television-viewer angle calculation unit 161, a television-viewer distance calculation unit 162, a robot-television angle calculation unit 163, a face direction angle acquisition unit 164, a viewing direction angle calculation unit 165, and a viewing direction image acquisition unit 166.

FIG. 11 is a diagram showing the positional relationship between the robot 1, the television, and the viewer in this embodiment.
In the triangle consisting of robot 1 (point A), viewer (point B) and TV (point C), BC=r, AB=r ₁ , and AC=r ₂ are fixed.
In addition, the angle between the viewer and the television as seen by the robot 1 (∠BAC) = _θr , the angle between the robot and the television as seen by the viewer, ∠ABC = _θh , and the angle between the robot 1 and the viewer as seen by the television, ∠ACB = θtv _, are fixed.

In this positional relationship, when the viewer faces in a direction shifted by a facial direction angle θ _h ' from the robot 1, it is assumed that the viewer is looking at the viewing direction position (point D). Here, it is assumed that AD = AC = _r2 . Also, BD = r'.
At this time, the angle between the viewer and the viewing direction position as seen by the robot 1 is ∠BAD=θ _r '.

The television-viewer angle calculation unit 161 calculates the angle _θr between the television and the viewer as seen by the robot 1 .
FIG. 12 is a diagram for explaining a method for calculating the angle _θr between the television and the viewer in this embodiment.

First, the television-viewer angle calculation unit 161 calculates the number of pixels d1 between the television and the viewer in the panoramic image from the face center point _Pf ( _xf , _yf ) and the television center point _Ptv ( _xtv , _{ytv )} as follows:
d ₁ = |x _tv −x _f |
Next, the television-viewer angle calculation unit 161 calculates the number of pixels d ₂ between the television and the viewer in the panoramic image from the size of the panoramic image (X _p , Y _p ) as follows:
d ₂ = |X _p −d ₁ |

Then, the television-viewer angle calculation unit 161 compares the number of pixels _d1 and _d2 between the television and the viewer,
d=min(d ₁ , d ₂ )
Let us assume that.

The television-viewer angle calculation unit 161 converts the size _Xp of the x-axis in the panoramic image into an angle of 360 degrees around the robot 1, and calculates the angle _θr between the television and the viewer as seen by the robot 1 from the number of pixels d between the television and the viewer as follows:
_θr = d × angle a
Angle a=360/X _p

For example, in a panoramic image, when the x-axis size _Xp = 4320, the x-axis value of the television position _xtv = 3541, and the x-axis value of the face position _xf = 713, the angle _θr between the television and the viewer as seen by the robot 1 is calculated as follows:
d ₁ =3541-713=2828
_d2 =4320-2828=1492
d = _d2 = 1492
θ _r =1492×360/4320≒124 degrees

The television-viewer distance calculation unit 162 calculates the distance _{r between the television and the viewer using the angle θ r} between the television and the viewer as seen by the robot 1, the distance r ₁ from the _robot 1 to the viewer, and _the distance r ₂ from the robot 1 to the television, as follows:
r=√(r ₁ ² + r ₂ ² -2r ₁ r ₂ cosθ _r )

The robot-television angle calculation unit 163 calculates the angle _θh between the robot 1 and the television using the cosine theorem as follows:
r ₂ ² = r ² + r ₁ ² -2rr ₁ cosθ _h
θ _h =cos ⁻¹ [(r ² +r ₁ ² −r ₂ ² )/(2rr ₁ )]

The face direction angle acquisition unit 164 estimates the face direction angle θ _h ′ of the viewer as seen by the robot 1 based on the face image detected by the face detection unit 132 , and acquires it together with the time T.

FIG. 13 is a diagram illustrating the face direction angle θ _h ′ in this embodiment.
The face direction angle θ _h ′ is based on the state (A) in which the viewer's face is facing directly at the robot 1, and is the angle from the reference direction (y-axis) to the direction ( _yh- axis) in which the viewer's face is facing, as shown in (B).
In order to estimate the face direction angle θ _h ′, for example, open source software such as OpenFace can be used, but the estimation method is not limited to this.

The viewing direction angle calculation unit 165 calculates the viewing direction angle θ _r ′ between the viewer as seen by the robot 1 and the direction in which the viewer is looking (viewing direction position) using the cosine theorem as follows:
r ₂ ² = r' ² + r ₁ ² -2r'r ₁ cosθ _h '
r' ² = r ₁ ² + r ₂ ² -2r ₁ r ₂ cosθ _r '
θ _r ′=cos ⁻¹ [(r ₁ /r ₂ )−(r′/r ₂ )cos θ _h ′]
Here, the distance r' from the viewer to the viewing direction position can be calculated, for example, by approximation as follows.

FIG. 14 is a diagram showing the positional relationship between the robot 1, the viewer, and the viewing direction position in this embodiment.
Here, the robot 1 (point A) is placed, for example, on a table between the viewer (point B) and the television, and when the viewing direction position is point D, the distance between point A and line segment BD is sufficiently close.
In this case, let E be the foot of the perpendicular line drawn from point A to line segment BD, and let BE = r ₁ ', DE = r ₂ '.
r'=|r ₁ ² - r ₂ ² |/|r ₁ '-r ₂ '|
In contrast,
r ₁ '≒ r ₁ , r ₂ '≒ r ₂
can be approximated as follows:
r′=r ₁ +r ₂
It is calculated as follows.

The viewing direction image acquisition unit 166 calculates the viewing direction position P _v (x v ) using the viewing direction angle θ _r ', the television-to-viewer angle θ _r , and the television center point P _tv (x _tv , _{y tv} ) in the panoramic image as follows, and acquires the viewing direction image.
x _v = x _tv - (number of pixels corresponding to θ _r '-θ _r )

At this time, the viewing direction image acquisition unit 166 converts the horizontal angle of view _θc of the imaging unit 20 into the number of angle of view pixels _dc , for example, and acquires an image in a range such as ( _xv- ( _dc /2), _xv +( _dc /2)) or ( _xv- ( _dc /2) _{, xv} +( _dc /2)-1 ₎ centered on the viewing direction position Pv(xv).
d _c = (X _p /360) x θ _c
Alternatively, if the size (number of pixels) of the image captured by the imaging section 20 is known, the viewing direction image acquisition section 166 may use this number of pixels as _dc .

FIG. 15 is a diagram illustrating a method for acquiring a viewing direction image in this embodiment.
For example, if the camera's horizontal angle of view θ _c = 40, the x-axis size of the panoramic image X _p = 4320, the x-coordinate of the television center point x _tv = 3800, and θ _r ' - _{θ r} = 0, then d _c = (4320/360) × 40 = 480, x _v = 3800 - 0 = 3800, and the viewing direction image acquisition unit 166 acquires images in the range of (3800 - 240, 3800 + 240) = (3560, 4040).
Also, for example, when _θr ' - _θr = 35, _dc = (4320/360) x 40 = 480, _xv = 3800 - ((4320/360) x 35) = 3380, so the viewing direction image acquisition unit 166 acquires images in the range of (3380 - 240, 3380 + 240) = (3140, 3620).

Furthermore, the viewing direction image acquisition section 166 may use the viewer's face center point _Pf ( _xf , _yf ) instead of the television center point _Ptv to calculate the viewing direction position _Pv ( _xv ) as follows.
x _v = x _f - (number of pixels corresponding to θ _r ')

The viewing state determination unit 17 obtains viewing direction images for a certain period of time and performs statistical processing to determine the viewing state, such as whether or not the viewer is watching television.
In this embodiment, the viewing level is defined as the viewing state.
The viewing intensity is an index of the degree to which a user watches television. A larger value indicates that the user is watching more television, and conversely, a smaller value indicates that the user is not watching television.

FIG. 16 is a diagram showing the functional configuration of the viewing state determining unit 17 in this embodiment.
The viewing state determination unit 17 includes a viewing direction object detection unit 171 and a viewing degree calculation unit 172, and outputs a viewing degree _Iw when a viewing direction image is input.

The viewing direction object detection unit 171 detects objects from the input viewing direction image and extracts keywords. As a means of object detection, for example, open source software such as Faster-RCNN can be used, but the detection method is not limited to this.

The viewership calculation unit 172 uses the keywords extracted by the viewing direction object detection unit 171 to calculate a viewership I _w (T) as an index indicating the degree to which the viewer is actually watching television.

FIG. 17 is a diagram showing an example of calculation of the viewer intensity I _w (T) in this embodiment.
At time T, when the viewing direction object detection unit 171 extracts a keyword indicating a type of video viewing device such as "television", "television", or "monitor", the viewing degree calculation unit 172 determines the viewing state as "television". Furthermore, when a person watching television together is detected, the viewing degree calculation unit 172 determines the viewing state as "others". The viewing state in other cases is defined as "other".

In this embodiment, the ratio of the "television" viewing state to the number of detections during a certain time _Tf is defined as the viewing level _Iw . In the example of Fig. 17, the viewing level _Iw = 0.6, which shows that the subject is paying attention to the television while shifting his/her gaze to other people or other objects.

In this case, the operation control unit 40 may set the threshold value for the robot to talk to the viewer to 0.5, for example, and perform control so that the robot 1 talks to the viewer since the viewing level _Iw is higher than a predetermined level.
In addition, when the calculated viewing level _Iw is, for example, 0.8, and the viewing state ratio of "television" is particularly high, it is highly likely that the viewer is concentrating on watching television. Therefore, the operation control unit 40 may control the robot 1 to refrain from talking to the viewer.

Conversely, when _Iw is 0.3, that is, when the ratio of the television viewing state is low, such as when the time spent watching the television direction is about 3 minutes per 10 minutes, the operation control unit 40 may control the robot 1 to talk to the viewer or to perform actions using gestures in order to arouse the viewer's interest in watching television.
For example, when the viewer is not looking in the direction of the television, the robot 1 looks around the viewer and the surroundings and says things like, "I'd really like to go to this place," or "Shall we change the channel?" to encourage the viewer to pay attention to the television.
In this way, the threshold value may be set in stages, and the operation control unit 40 may control the robot 1 appropriately for each viewer, for example, by only speaking when _Iw is high, and by performing gestures along with speaking when _Iw is low.

In addition, when multiple types of classifications indicating the viewing direction, such as "television," "others," and "other," are acquired as the viewing state, the operation control unit 40 may change the operation of the robot 1 according to this classification. For example, the frequency of each utterance type, such as disclosure, question, confirmation, information, and response, regarding the content of a television program may be adjusted as follows:

When the viewing direction is often toward the television, it is assumed that the viewer is watching the television, so the operation control unit 40 reduces the frequency of questions requesting answers or confirmations that would prevent the viewer from stopping.
When the viewing direction is often toward "other people," it is considered that the viewer is in a state of frequent communication with others, so the operation control unit 40 reduces the frequency of speech for all speech types so as not to interfere with communication with others.
When the viewing direction is often "other," it is considered that the viewer is not paying attention to the television and is not communicating with others, so the operation control unit 40 increases the frequency of disclosures or questions to stimulate interest in television viewing.

According to this embodiment, the viewing state estimation device 10 detects the viewer's face and the television from a panoramic image in all directions around the robot 1, and calculates the angle between the television and the viewer as seen by the robot 1 from the distance between them on the image and the size of the panoramic image. Furthermore, the viewing state estimation device 10 specifies the positional relationship between the robot 1, the viewer, and the television by measuring the distance from the robot 1 to the detected television and viewer. Then, the viewing state estimation device 10 detects an object included in the viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the viewer's face direction angle obtained from the face position image, and judges the viewer's state based on the type of this object.

Therefore, the viewing state estimation device 10 can estimate the viewing state of a viewer by detecting objects in the viewing direction image based on a panoramic image using a robot 1 installed on a table or the like, without installing an imaging device such as a camera on the ceiling or the like in the home, and without having the viewer wear a gaze direction acquisition device.
As a result, the robot 1 can perform actions according to the viewing state, such as being considerate not to talk to the viewer when the viewer is watching television, and making speech and gestures to encourage the viewer to participate in watching television when the viewer is not watching television.

The viewing state estimation device 10 can easily identify the viewing direction position and estimate the viewing state by approximating the distance from the viewer to the viewing direction position by the sum of the distance from the robot 1 to the viewer and the distance from the robot 1 to the viewing direction position.

The viewing state estimation device 10 calculates the viewing state based on statistical information about the viewer's state over a certain period of time, thereby reducing calculation errors in the viewing direction position and improving the reliability of the determined viewing state.

The viewing state estimation device 10 calculates the viewing degree indicating the proportion of television viewing as the viewing state, so that the operation control unit 40 can grasp the degree to which the viewer is actually watching television and appropriately adjust the operation of the robot 1 to the viewer's state according to this degree.
At this time, the operation control unit 40 can easily control the operation of the robot 1 based on the result of comparing the calculated viewing rate with a predetermined threshold value.
Furthermore, the motion control unit 40 can appropriately set the motion variations of the robot 1 according to the state of the viewer by gradually changing the control based on a plurality of threshold values.

The viewing state estimation device 10 determines multiple viewing states, including a state of watching television and a state of watching others, so the operation control unit 40 can grasp the type of object the viewer is looking at and appropriately match the operation of the robot 1 to the viewer's state depending on this type.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

In the above-described embodiment, the viewing state estimation device 10 estimated the type of object the viewer is looking at by object detection in the viewing direction image, but whether or not the viewer is watching television may be determined based on the viewer's face direction angle.
Specifically, for example, the face direction angle θ _h ′ is
θ _h −α＜θ _h '＜θ _h +α
It may be determined that the viewer is facing the television when the above condition is satisfied. Here, α is an adjustment angle (e.g., half the viewing angle of the television) that is set based on the size of the television and the distance from the viewer.

In addition, in the above embodiment, the viewing direction angle θ _r ′ is calculated based on the face direction angle θ _h ′, but the calculation method is not limited to this.
For example, if θ _h ' cannot be obtained properly,
θ _r '=cos ^-1 [(r ₁ ² + r ₂ ² - r' ² )/(2r'r ₁ )]
As shown above, θ _r ' may be calculated without using θ _h '.

In the above embodiment, the distance from the robot 1 to the viewing direction position is equal to the distance _r2 from the robot 1 to the television, but the assumed condition is not limited to this.
For example, depending on the positive/negative or value range of θ _h ′, r ₁ , r ₂ , or another value may be used for the relevant distance, and the value may be set appropriately according to the situation.

In this embodiment, the configuration and operation of the viewing state estimation device 10 have been mainly described, but the present invention is not limited to this, and may be configured as a method or program for estimating the viewing state, including each component.

Furthermore, the functions of the viewing state estimation device 10 may be realized by recording a program for realizing the functions of the viewing state estimation device 10 on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.

The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.

Furthermore, "computer-readable recording medium" may include something that dynamically holds a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or something that holds a program for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such a case. Furthermore, the above program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

LIST OF SYMBOLS 1 Robot 10 Viewing state estimation device 11 Panoramic image section 12 Distance panoramic image section 12A Distance image database 12B Distance panoramic image database 13 Viewer detection section 14 Television detection section 15 Distance acquisition section 16 Viewing direction detection section 17 Viewing state determination section 20 Imaging section 30 Distance detection section 40 Operation control section 121 Distance image synthesis section 122 Overlapped section calculation section 131 Person detection section 132 Face detection section 161 Television-viewer angle calculation section 162 Television-viewer distance calculation section 163 Robot-television angle calculation section 164 Face direction angle acquisition section 165 Viewing direction angle calculation section 166 Viewing direction image acquisition section 171 Viewing direction object detection section 172 Viewing degree calculation section

Claims (10)

a panoramic image unit for acquiring an omnidirectional panoramic image synthesized from images captured around the robot;
a distance panorama image unit for generating a distance panorama image in which distance data corresponding to each pixel of the panorama image is used as a pixel value;
a television detection unit that detects a television position from the panoramic image;
a viewer detection unit that detects a face position of a viewer from the panoramic image;
a distance acquisition unit that acquires distances at the television position and the face position from the distance panoramic image;
a viewing direction detection unit that specifies a positional relationship between the robot, the viewer, and the television by calculating an angle between the television and the viewer as seen by the robot based on a size of the panoramic image, the television position, and the face position, and obtains a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on a face direction angle of the viewer obtained from the image of the face position;
a viewing state determination unit that detects an object included in the viewing direction image and determines a state of the viewer based on a type of the object. The viewing state estimation device according to claim 1, wherein the viewing direction detection unit approximates the distance from the viewer to the viewing direction position by the sum of the distance from the robot to the viewer and the distance from the robot to the viewing direction position. The viewing state estimation device according to claim 1 or claim 2, wherein the viewing state determination unit calculates the viewing state based on statistical information on the state of the viewer within a certain period of time. The viewing state estimation device according to claim 3, wherein the viewing state determination unit calculates a viewing degree indicating the proportion of the television being watched as the viewing state. The viewing state estimation device according to claim 3 , wherein the viewing state determination unit determines, as the viewing state, whether the viewing state is one of a plurality of states including a state in which the viewing state is watching the television and a state in which the viewing state is watching another person. The viewing state estimating device according to claim 4 ;
a motion control unit that controls the motion of the robot based on a result of comparing the viewing degree output from the viewing state estimation device with a predetermined threshold. The robot system according to claim 6 , wherein the operation control unit provides a plurality of the predetermined thresholds, and changes the control of the robot in a stepwise manner depending on a result of comparing the viewing rate with the plurality of thresholds. The viewing state estimating device according to claim 5 ;
and an operation control unit that controls an operation of the robot in accordance with the classification of the plurality of states output from the viewing state estimation device. a panoramic image generating step of acquiring an omnidirectional panoramic image synthesized from images captured around the robot;
a distance panoramic image generating step of generating a distance panoramic image in which distance data corresponding to each pixel of the panoramic image is used as a pixel value;
a television detection step of detecting a television position from the panoramic image;
a viewer detection step of detecting a face position of a viewer from the panoramic image;
a distance acquisition step of acquiring distances at the television position and the face position from the distance panoramic image;
a viewing direction detection step of calculating an angle between the television and the viewer as seen by the robot based on the size of the panoramic image, the television position, and the face position, thereby specifying a positional relationship between the robot, the viewer, and the television, and acquiring a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the image of the face position;
a viewing state determination step of detecting an object included in the viewing direction image and determining a state of the viewer based on a type of the object, the viewing state determination method being executed by a computer. A viewing state estimation program for causing a computer to function as the viewing state estimation device according to any one of claims 1 to 5.

Images