JP7584576B2 - Live communication system using characters - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) Announced on September 18, 2018 via Apple Inc.'s TestFlight (https://developer.apple.com/jp/testflight/). (2) Announced on October 4, 2018 at https://iriam.com/ and https://twitter.com/iriam_official. (3) Announced on October 4, 2018 at https://itunes.apple. com/jp/app/iriam-%E3%82%A4%E3%83%AA%E3%82%A2%E3%83%A0/id1358057296? mt=8. (4) Announced on October 4, 2018 at https://play.google.com/store/apps/details? id=jp.co.duo7.vr.iriam. (5) Announced at Niconico Chokaigi 2019 on April 27, 2019 and April 28, 2019. (6) The results will be announced on Niconico Live Broadcast (https://live.nicovideo.jp/gate/lv319512297 and https://live.nicovideo.jp/gate/lv319512342) on April 27, 2019 and April 28, 2019.

The present invention relates to visual communication technology via a communication network.

With the spread of smartphones, communication via SNS (Social Networking Services) has become common. SNS allows you to easily share a variety of content, including not only text but also photos and videos. SNS is a tool that gives you a strong sense of the "bonds" (connections) between people.

On the other hand, in the field of computer games, there are many games in which the user plays while having a conversation with a character (computer). Users often form strong feelings for characters that do not exist in reality.

JP 2012-231281 A Patent Application No. 2019-002271

Characters are virtual entities created by game designers and do not have the same sense of reality as humans. However, characters have a unique charm that stems from their sense of unreality. The inventors thought that by incorporating both the unique charm of characters and the "realistic connection" that is unique to humans, they might be able to propose a new method of communication.

Specifically, we consider a conversation between a streamer (human) and a viewer (human). The streamer acts as a spokesperson for the character, just like a voice actor in a television anime, and converses with the viewer through the character. If we consider this type of communication, we believe that if the viewer can have a relationship with the character that goes beyond words, this will further strengthen the viewer's attachment to the character.

The present invention was completed based on the above recognition, and its main purpose is to provide technology for deepening non-verbal communication between characters and viewers in communication via characters.

A live communication system according to an embodiment of the present invention includes a distribution device operated by a distributor and a viewing device operated by a viewer.
The distribution device includes an imaging unit that images the broadcaster, a motion information generation unit that generates motion information indicating characteristics of the broadcaster's motion from the captured image of the broadcaster, a first hand detection unit that detects first hand coordinates corresponding to the position of the broadcaster's hand, a first message acquisition unit that accepts a first message from the broadcaster, and a transmission unit that transmits the motion information, the first hand coordinates, and the first message.
The viewing device includes a receiving unit that receives movement information, a first hand coordinate, and a first message, a second hand detection unit that detects a second hand coordinate corresponding to the position of the viewer's hand, a viewing character display unit that displays a moving image of a character on the screen of the viewing device and moves the character according to the movement information, a first message output unit that outputs a first message in accordance with the character's movement, and a hand display unit that displays a first hand, which is an image of the distributor's hand, and a second hand, which is an image of the viewer's hand, on the screen of the viewing device based on the first hand coordinate and the second hand coordinate.
The hand display unit displays a handshake between the first hand and the second hand when the distance between the first hand coordinates and the second hand coordinates is within a first threshold value.

The present invention makes it easier to realize rich communication through characters.

FIG. 2 is a hardware configuration diagram of a live communication system. FIG. 1 is a schematic diagram for explaining a conversation service provided by a live communication system. FIG. 2 is an external view of a conversation booth. FIG. 2 is a functional block diagram of a distribution device and a server. FIG. 2 is a functional block diagram of the viewing device. 11 is a screen diagram of a viewer conversation screen displayed on the viewing device. FIG. FIG. 13 is a screen diagram of a distributor conversation screen displayed on the distribution device. FIG. 11 is a sequence diagram of a process for displaying a character on both the distribution device and the viewing device. 13 is a diagram showing the screen of the viewing device before the viewer shakes hands with the character. FIG. 13 is a diagram showing a screen of a viewing device when a viewer shakes hands with a character. FIG. 1 is a schematic diagram for explaining a handshake movement. 13 is a diagram showing the screen of the viewing device at the start of a conversation with a character. 13 is a diagram showing the screen of the viewing device when a viewer touches a character. FIG. 11 is a sequence diagram showing a process when a good is presented to a character. 13 is a diagram showing a screen of the viewing device when a character is provided with goods. FIG. FIG. 1 is an external view of a digital signage device. FIG. 2 is a functional block diagram of the digital signage device. FIG. 13 is a diagram showing the screen of the viewing device when playing a hand game with a character.

FIG. 1 is a diagram showing the hardware configuration of a live communication system 100. As shown in FIG.
In the live communication system 100, a plurality of distribution devices 104a, 104b, ..., 104m (hereinafter, collectively referred to as "distribution devices 104" when collectively referred to or when not particularly distinguished) and a plurality of viewing devices 108a, 108b, ..., 108n (hereinafter, collectively referred to as "viewing devices 108" when collectively referred to or when not particularly distinguished) are connected to a server 102 via the Internet 106. A smartphone is assumed as the distribution device 104, but it may also be a general-purpose computer such as a laptop PC. A large touch pad is assumed as the viewing device 108, but it may also be another information device such as a smartphone.

A user who uses the distribution device 104 is called a "distributor." A user who uses the viewing device 108 is called a "viewer." The distributor is the provider of the service, and the viewer is the recipient of the service.

A digital signage device 300 is connected to the server 102. The digital signage device 300 broadcasts live or records and plays back conversations between the broadcaster and the viewer. Details will be described later in relation to Figure 16 onwards. The digital signage device 300 is not a required component of the live communication system 100.

FIG. 2 is a schematic diagram for explaining a conversation service provided by the live communication system 100. As shown in FIG.
The live communication system 100 disclosed in Patent Document 2 by the present applicant has been described on the premise that a large number of viewing devices 108 are simultaneously connected to one distribution device 104, and one-to-many communication is performed. The live communication system 100 in this embodiment will be described on the premise that a distributor 110 and a viewer 112 converse with each other through one-to-one communication.

A character is assigned to the distributor 110 in advance. A distribution character 114a that is assigned to the distributor 110 is displayed on the distribution device 104. A viewing character 114b is also displayed on the viewing device 108. The distribution character 114a and the viewing character 114b are exactly the same character. Hereinafter, when there is no particular distinction between the distribution character 114a and the viewing character 114b, they will simply be referred to as "character 114".

When the distributor 110 speaks in front of the distribution device 104, the audio data is transmitted to the viewing device 108. The distribution device 104 detects the movement of the distributor's 110 mouth, and also moves the mouth of the distributed character 114a. In conjunction with this, the viewing device 108 also moves the mouth of the viewed character 114b. To the viewer 112, it appears as if the viewed character 114b is speaking of its own volition, rather than the distributor. In a sense, the distributor 110 acts as a "voice actor" and speaks for the character 114. Hereinafter, the voice message input by the distributor 110 is referred to as the "first message".

When the viewer 112 speaks in front of the viewing device 108, the audio data is transmitted to the distribution device 104 along with an image (moving image) of the viewer 112 himself/herself. Hereinafter, the audio message input by the viewer 112 is referred to as a "second message." In Patent Document 2, it was explained that the viewer 112 inputs the second message by text, but in this embodiment, the distributor 110 and the viewer 112 converse with each other by voice. The distributor 110 can see the appearance of the viewer 112 (described later in relation to FIG. 7), but the viewer 112 only sees the viewing character 114b played by the distributor 110.

The personality of the character 114 is shaped by the basic settings of the character 114 (such as the character's 114 appearance and the variety of facial expressions) and the voice quality and conversational ability of the distributor 110.

The viewer 112 usually enjoys one-to-many conversations with the characters 114 on a smartphone or the like, and develops feelings for the characters 114 (see Patent Document 2). In this embodiment, a "handshake event (an opportunity to meet one-to-one)" with the characters 114 that the viewer 112 has developed feelings for is assumed. The viewer 112 visits a venue (hereinafter referred to as the "event venue") set up by the operator of the live communication system 100. At the event venue, the viewer 112 can enjoy one-to-one voice conversations with the characters 114. However, the "time limit" for the conversation is set to one minute. In the conversation, the viewer can enjoy not only the conversation but also a handshake, using the method described below.

FIG. 3 is an external view of conversation booth 302.
At the event venue, a conversation booth 302 is prepared for each character 114. The viewers 112 line up at the conversation booth 302 corresponding to their favorite character 114. The viewers 112 wear nameplates. When it is their turn, the viewers 112 enter the conversation booth 302. The conversation booth 302 is a semi-closed space separated by a partition, and is configured so that the viewers 112 can easily concentrate on the conversation with the character 114. A viewing device 108 is installed in front of the conversation booth 302, and a viewing character 114b is displayed on the viewing device 108. In the conversation booth 302, the viewers 112 can monopolize the conversation time with the idol-like character 114 who usually has a conversation with a large group of other viewers 112.

A camera 304 for capturing an image of the viewer 112 is attached to the top of the viewing device 108. The captured image of the viewer 112 is transferred to the distribution device 104. A present box 306 is installed at the bottom of the viewing device 108. The viewer 112 can insert presents such as hats or bouquets of flowers (hereinafter referred to as "goods") into an insertion slot 308 of the present box 306. A tag is attached to the goods (physical object), and the ID of the goods (hereinafter referred to as "goods ID") is detected by a detection device 310 inside the insertion slot 308. The insertion of goods will be described later with reference to Figures 14 and 15.

FIG. 4 is a functional block diagram of the distribution device 104 and the server 102.
Each component of the distribution device 104 and the server 102 is realized by hardware including computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer programs may be composed of device drivers, operating systems, various application programs located at higher layers than those, and libraries that provide common functions to these programs. Each block described below shows a functional block rather than a hardware configuration.
The same applies to the audiovisual device 108 and the digital signage device 300.

(Distribution device 104)
The distribution device 104 includes a user interface processing unit 116 , an imaging unit 118 , a data processing unit 120 , a communication unit 122 , and a data storage unit 124 .
The communication unit 122 is responsible for communication processing. The imaging unit 118 images the distributor 110. The user interface processing unit 116 accepts operations from the distributor 110 and is responsible for processing related to the user interface, such as image display and audio output. The data storage unit 124 stores various data including image data of the character 114. The data processing unit 120 executes various processes based on data acquired by the communication unit 122, the user interface processing unit 116, and the imaging unit 118 and data stored in the data storage unit 124. The data processing unit 120 also functions as an interface for the user interface processing unit 116, the imaging unit 118, the communication unit 122, and the data storage unit 124.

The communication unit 122 includes a transmitting unit 148 and a receiving unit 150.

The user interface processing unit 116 includes an input unit 126 that accepts input from the distributor 110 and an output unit 128 that outputs various information to the distributor 110 .
The input unit 126 includes a motion detection unit 130 and a first message acquisition unit 132. The motion detection unit 130 detects facial expressions and body movements of the distributor 110 via the imaging unit 118. The first message acquisition unit 132 acquires a first message (voice message) from the speech of the distributor 110.
The action detection unit 130 includes a first hand detection unit 134. The first hand detection unit 134 detects the movement of the distributor 110's hand.

The output section 128 includes a delivery character display section 138 , a second message output section 140 , a hand display section 142 , a decoration section 312 , an object display section 314 and a live image output section 332 .
The delivery character display unit 138 displays the delivery character 114a. The second message output unit 140 outputs a second message (audio message) by the viewer 112. The hand display unit 142 displays an image (described later) imitating the hands of the character 114 (the distributor 110) and the viewer 112 on the screen. The decoration unit 312 displays various decoration images as visual effects on a given occasion, such as when the character 114 and the viewer 112 shake hands. The handshake and decoration images will be described later in relation to Figures 9, 10, 11, 13, etc. The object display unit 314 displays an image (hereinafter referred to as a "goods image") corresponding to the goods presented by the viewer 112 on the screen. The display of the goods image (virtual object) on the screen will be described later in relation to Figure 15. The live image output unit 332 displays a moving image of the captured image of the viewer 112 (described later in relation to Figure 7).

The data processing unit 120 includes a motion information generating unit 144 .
The motion information generating unit 144 generates "motion information" from the detection result of the motion detecting unit 130. The motion information is numerical data indicating the motion state of the broadcaster 110, such as a feature vector indicating the facial expression of the broadcaster 110 (e.g., a numerical value indicating the angle of the eyebrows, the opening degree of the eyes and mouth, and a numerical value indicating the relative positional relationship (front, back, left, right) between the broadcaster 110 and the broadcasting device 104).

(Server 102)
The server 102 includes a communication unit 152 , a live control unit 154 , and a data storage unit 156 .
The communication unit 152 is responsible for communication processing. The data storage unit 156 stores various data. The live control unit 154 mediates and manages communication between the distribution device 104 and the viewing device 108 based on the data acquired by the communication unit 152 and the data stored in the data storage unit 156. The live control unit 154 also functions as an interface between the communication unit 152 and the data storage unit 156.
The data storage section 156 includes a character storage section 158 that stores data on a plurality of characters 114. The data storage section 156 also registers a goods ID and a goods image in association with each other.

FIG. 5 is a functional block diagram of the viewing device 108.
The viewing device 108 includes a user interface processing unit 162 , a data processing unit 166 , a communication unit 164 , an imaging unit 316 and a data storage unit 168 .
The communication unit 164 is responsible for communication processing. The user interface processing unit 162 accepts operations from the viewer 112 and is responsible for processing related to the user interface, such as image display and audio output. The imaging unit 316 captures images of the viewer 112 via the camera 304. The data storage unit 168 stores various data including image data of the character 114. Both the distribution device 104 and the viewing device 108 store data of the character 114. The data processing unit 166 executes various processes based on data acquired by the communication unit 164 and the user interface processing unit 162 and data stored in the data storage unit 168. The data processing unit 166 also functions as an interface for the user interface processing unit 162, the communication unit 164, the imaging unit 316, and the data storage unit 168.

The communication unit 122 includes a transmitting unit 186 and a receiving unit 190.

The user interface processing unit 162 includes an input unit 170 that receives input from the viewer 112 and an output unit 172 that outputs various information to the viewer 112 .
The input unit 170 includes a second message acquisition unit 174, a second hand detection unit 318, an object insertion unit 320, and an object identification unit 322. The second message acquisition unit 174 acquires a second message (voice message) from the speech of the viewer 112. The second hand detection unit 318 detects the hand movement of the viewer 112. The object insertion unit 320 detects the insertion of a good (physical object) into the insertion slot 308. The object identification unit 322 identifies the type of good by reading the good ID of the good inserted into the insertion slot 308. When the good ID is identified, the transmission unit 186 of the communication unit 164 transmits the good ID to the server 102.

The output section 172 includes a viewing character display section 180 , a first message output section 182 , a hand display section 324 , a decoration section 326 , an object display section 328 and a curtain display section 330 .
The viewing character display unit 180 displays the viewing character 114b on the viewing device 108. The first message output unit 182 outputs the first message by the distributor 110 as audio from the viewing device 108. The hand display unit 324 displays images of the hands of the character 114 (distributor 110) and the viewer 112 on the screen. The decoration unit 326 displays various decoration images. The object display unit 328 displays images of goods corresponding to goods given as presents by the viewer 112 on the screen. The curtain display unit 330 displays images imitating curtains at the start and end of a conversation. Curtains will be described later in relation to FIG. 12.

The data processing unit 166 includes a conversation management unit 290. The conversation management unit 290 manages the conversation time between the viewer 112 and the character 114. The conversation management unit 290 ends the conversation between the viewer 112 and the character 114 when a predetermined time limit, for example, one minute, has elapsed.

FIG. 6 is a diagram showing a viewer conversation screen 196 displayed on the viewing device 108.
The viewing character display unit 180 displays the viewing character 114b selected as the conversation partner as a moving image. The viewer 112 enjoys a conversation with the viewing character 114b on the viewing device 108. The viewer 112 cannot see the distributor 110. The viewer 112 can enjoy the sensation of having a conversation with the viewing character 114b itself, not with the distributor 110. The first message output unit 182 of the viewing device 108 outputs a voice (first message) from the distributor 110 who is in charge of the viewing character 114b. The viewer 112 speaks to the viewing character 114b by voice (second message). The live control unit 154 of the server 102 relays the first message and the second message.

FIG. 7 is a diagram showing the distributor conversation screen 206 displayed on the distribution device 104.
The distributor conversation screen 206 shown in FIG. 7 is displayed on the distribution device 104 of the distributor 110 in charge of the distribution character 114a (corresponding to the viewing character 114b). The distributor conversation screen 206 is divided into a viewer display area 336 and a character display area 334. The distribution character display unit 138 displays the distribution character 114a as a moving image in the character display area 334. The viewing character 114b and the distribution character 114a are the same character and move in the same way. The distributor 110 talks with the viewer 112 while checking the movement of the distribution character 114a, which is his/her alter ego, on the distribution device 104. The distributor 110 inputs a first message by voice to the distribution device 104. The transmission unit 148 transmits the first message to the server 102, and the live control unit 154 of the server 102 transmits the first message to the viewing device 108.

The second message output unit 140 outputs the voice (second message) from the viewer 112. The captured image of the viewer 112 is transmitted from the viewing device 108 to the distribution device 104 via the server 102. The live image output unit 332 displays the captured image (moving image) of the viewer 112 in the viewer display area 336. The distributor 110 can converse with the viewer 112 while observing the viewer's 112 appearance (attitude and facial expression).

The camera 208 (imaging unit 118) of the distribution device 104 captures an image of the distributor 110. The motion detection unit 130 detects the facial expression and motion of the distributor 110 from the captured image, and the motion information generation unit 144 generates motion information. The distribution character display unit 138 determines the motion of the distribution character 114a based on the motion information. The viewing character 114b is also displayed as a video on the viewing device 108 based on the same motion information.

As described above, the viewer 112 converses with the viewable character 114b by voice. The distributor 110 converses with the viewer 112 as the viewable character 114b. The viewer 112 cannot see the distributor 110, but the distributor 110 converses while seeing both the viewer 112 and the viewable character 114a.

FIG. 8 is a sequence diagram of a process for displaying the character 114 on both the distribution device 104 and the viewing device 108.
The motion detection unit 130 of the broadcasting device 104 detects the motion of the broadcaster 110 (S10). The motion information generation unit 144 generates motion information (S12). The motion information is numerical data indicating the facial expression of the broadcaster 110, the angle of the body, the distance of the broadcaster 110 from the imaging unit 118, and the like (S12).

The transmission unit 148 of the distribution device 104 transmits the action information to the viewing device 108 via the server 102 (S14). The distribution character display unit 138 generates a moving image of the distribution character 114a based on the action information (S16) and causes the distribution device 104 to display the moving image of the distribution character 114a (S18).

Meanwhile, the viewing character display unit 180 of the viewing device 108 also generates a moving image of the viewing character 114b based on the received action information (S20), and causes the viewing device 108 to display the viewing character 114b as a moving image (S22).

The distributed character 114a and the viewed character 114b are the same character, and their movements are determined based on the same motion information, so that the distributed character 114a and the viewed character 114b have the same appearance and perform the same movements on the distribution device 104 and the viewing device 108, respectively.

The distribution device 104 does not transmit the moving image (character image) of the character 114 itself to the viewing device 108, but only the movement information that defines the movement of the character 114. Because the amount of data transmitted is smaller than when transmitting the moving image itself, it is easier to quickly link the movements of the distributed character 114a and the viewed character 114b. In other words, because the difference in time between when the movement information is reflected as the movement of the distributed character 114a and when the same movement information is reflected as the movement of the viewed character 114b is small, the viewer 112 can enjoy a pleasant conversation with the character 114 (distributor 110).

The first message (audio message) and the action information may be transmitted from the distribution device 104 to the viewing device 108 in an integrated manner. In this case, the action of the viewing character 114b and the audio message are less likely to be out of sync. In addition, transmitting action information for determining the moving image, rather than the moving image itself, is also effective in reducing the communication load on the Internet 106.

When the broadcaster 110 leans his/her body left or right, the motion detection unit 130 detects the inclination of the body. The motion information generation unit 144 includes the inclination angle and the inclination direction in the motion information. The viewed character display unit 180 leans the viewed character 114b according to the motion information. The broadcast character display unit 138 also leans the broadcast character 114a according to the same motion information. In this way, when the broadcaster 110 leans his/her body left or right, the characters 114 on both the broadcasting side and the viewing side lean their bodies to match the broadcaster 110. When the broadcaster 110 leans his/her body forward or backward, the characters 114 similarly lean their bodies forward or backward. The motion detection unit 130 can recognize the forward or backward lean of the broadcaster 110 from the size of the broadcaster 110 reflected in the captured image.

It is generally said that a person's psychology is reflected in the way their body tilts. For example, it is said that when a person is interested in a conversation, they tend to lean forward (as if they are about to listen), and when they are upset, their body sways from side to side. By reflecting the body movements of the distributor 110 in the character 114, the viewer 112 can sense the psychology of the character 114 (distributor 110).

When the broadcaster 110 changes his/her facial expression, the motion detection unit 130 detects the facial expression of the broadcaster 110, specifically, the size of the eyes, the shape of the mouth, the angle of the eyebrows, the depth and direction of wrinkles between the eyebrows, etc. The motion information generation unit 144 generates a feature vector that defines the facial expression based on these detection values. The feature vector becomes part of the motion information. When the broadcaster 110 closes one eye (winks), the viewing character display unit 180 makes the viewing character 114b wink. The same is true for the broadcast character 114a.

In addition, by numerically expressing various facial expressions, such as sticking out the tongue or turning up the corners of the mouth, using feature vectors (motion information), the facial expression of the distributor 110 can also be reflected in the character 114. When the distributor 110 smiles, the character 114 also smiles. When the distributor 110 purses his lips, the character 114 also purses his lips. By linking the facial expressions of the distributor 110 and the character 114, the viewer 112 can feel the presence of the character 114 even closer.

FIG. 9 is a diagram showing the screen of the viewing device 108 before the viewer 112 shakes hands with the character 114.
When the viewer 112 touches the screen of the viewing device 108 (hereinafter simply referred to as the "viewing screen"), the second hand detection unit 318 of the viewing device 108 detects the coordinates of the touch point (hereinafter simply referred to as the "second hand coordinates 346"). When the distributor 110 touches the screen of the distribution device 104 (hereinafter simply referred to as the "distribution screen"), the first hand detection unit 134 of the distribution device 104 detects the coordinates of the touch point (hereinafter simply referred to as the "first hand coordinates 340"). The viewing device 108 transmits the second hand coordinates 346 to the distribution device 104, and the distribution device 104 transmits the first hand coordinates 340 to the viewing device 108. It is assumed that the same coordinate system is set for the viewing screen and the distribution screen. For example, when the first hand coordinates 340 on the distribution screen are (x1, y1), the first hand coordinates 340 on the viewing screen are also (x1, y1). The same applies to the second hand coordinates 346.

The hand display unit 324 of the viewing device 108 displays a first hand 342 at a first hand coordinate 340. The first hand 342 is an image that imitates the hand of the viewed character 114b (the distributor 110). In Fig. 9, the first hand 342 is displayed separately from the image of the viewed character 114b, but the hand of the viewed character 114b itself may be moved to the position of the first hand coordinate 340. The hand display unit 324 also displays a second hand 344 at a second hand coordinate 346, which is the point touched by the viewer 112. The second hand 344 is an image that imitates the hand of the viewer 112.
The hand display unit 142 of the broadcasting device 104 also displays a first hand 342 and a second hand 344 on the broadcasting screen.

When both the distributor 110 and the viewer 112 are touching the distribution screen and the viewing screen, respectively, they are in a "handshake possible state." On the other hand, when either one of them removes their hand from the screen, they enter a "handshake not possible state." When in a handshake possible state, the hand display unit 142 periodically calculates the distance between the first hand coordinate 340 and the second hand coordinate 346 (hereinafter referred to as "hand distance d"). The same is true for the hand display unit 142 of the distribution device 104.

FIG. 10 is a diagram showing the screen of the viewing device 108 when the viewer 112 shakes hands with the character 114.
When the handshake is possible and the hand distance d is within the first threshold, the hand display unit 324 displays a handshake image 338 of the second hand 344 and the first hand 342 shaking hands. That is, the distributor 110 and the viewer 112 bring their fingers (touch points) close to each other while still in a touching state, thereby establishing a "handshake." When the handshake is established, the decoration unit 326 displays a star-shaped decoration image 360 around the first hand 342 and the second hand 344. The decoration image 360 creates the effect of the handshake being achieved.

In this way, by bringing the distributor 110 and the viewer 112 closer to each other in their touch points, a handshake that transcends the "dimensions" of the viewed character 114b (two-dimensional) and the viewer 112 (three-dimensional) becomes possible. By giving the viewer 112 the sensation of not only talking but also holding the hand of the character 114, or of the character 114 holding his/her hand, the encounter between the viewer 112 and the character 114 can be made even more exciting.
A handshake image 338 is also displayed on the broadcasting device 104 by a similar processing method.

FIG. 11 is a schematic diagram for explaining the handshake movement.
The first hand coordinate 340 at time t1 is expressed as "first hand coordinate 340 (t1)". At time t1, a handshake between the first hand coordinate 340 (t1) and the second hand coordinate 346 (t1) is established. Also, the hand distance d (t1) between the first hand coordinate 340 (t1) and the second hand coordinate 346 (t1) is assumed to be within a second threshold. Even after the handshake is established, the handshake continues as long as the hand distance d is within the second threshold.

The hand display unit 142 displays the handshake image 338 (first hand 342 and second hand 344) at the handshake point 348 (t1), which is the midpoint between the first hand coordinate 340 (t1) and the second hand coordinate 346 (t1). If the first hand coordinate 340 and the second hand coordinate 346 are close enough, a handshake between the first hand 342 and the second hand 344 is established even if the second hand coordinate 346 and the first hand coordinate 340 do not completely match.

Next, the first hand coordinate 340 and the second hand coordinate 346 are each displaced at time t2. The hand distance d (t2) at time t2 is also within the second threshold. The handshake is maintained, and the handshake point 348 is displaced. The handshake image 338 moves from the handshake point 348 (t1) to the handshake point 348 (t2), so the "movement" of the handshake is expressed. When the handshake image 338 moves a predetermined distance or more, or when the handshake image 338 moves at a predetermined speed or more, the decoration unit 326 adds the decoration image 360. According to this control method, the character 114 (distributor 110) and the viewer 112 can continue the handshake by moving the first hand 342 and the second hand 344 together in unison, in other words, by moving their hands in a timely manner so that the hand distance d between the first hand coordinate 340 and the second hand coordinate 346 does not become larger than the second threshold. Additionally, the more the handshake continues, the more decorative images 360 will appear, making the handshake scene more exciting.

The first and second thresholds may be the same, or, as described above, the second threshold may be set to be greater than the first threshold. In this case, it is possible to realize a feeling of operation in which, although a handshake is not possible unless the first hand coordinate 340 and the second hand coordinate 346 are brought close enough, once a handshake is made, it is difficult to come undone. With such a setting, it becomes easier for the viewer 112 to continue shaking hands with the character 114.

In addition, since the same display is also made on the distribution device 104, the distributor 110 also shares the feeling of shaking hands with the viewer 112. In particular, since the distributor 110 can visually check the appearance and expression of the viewer 112 in the viewer display area 336, the distributor 110 can sense the fun and joy that the viewer 112 feels when shaking hands with the character 114.

FIG. 12 is a diagram showing the screen of the viewing device 108 at the start of a conversation with the character 114.
When the viewer 112 enters the conversation booth 302, the curtain display unit 330 of the viewing device 108 displays a curtain 350 on the viewing screen. The viewing character 114b is initially hidden by the curtain 350. When the viewer calls out to the character 114, when a human sensor detects entry into the conversation booth 302, or when an opening command is input from the distributor 110, the curtain display unit 330 displays a moving image of the curtain 350 opening. The character 114 is displayed behind the curtain 350. The curtain 350 opens to reveal the desired character 114, creating the illusion of the viewer 112 meeting the character 114.

The curtain display unit 330 executes the performance of opening the curtain 350 over a predetermined opening time. In this embodiment, the opening time is three seconds. When the curtain 350 opens or closes, the output unit 172 of the viewing device 108 may play predetermined opening music.

After the opening, the conversation management unit 290 measures the conversation time. When the time limit (e.g., one minute) has elapsed, the conversation management unit 290 instructs the curtain display unit 330 to close the curtain 350. The curtain display unit 330 gradually blocks the character 114 by displaying a moving image of the curtain 350 closing. The curtain display unit 330 executes the performance of closing the curtain 350 over a predetermined closing time. In this embodiment, the closing time is set to 10 seconds.

Making the closing time longer than the opening time can make the viewer 112 feel "reluctant to part ways." By slowly closing the curtain and gradually hiding the character 114, a time of reluctance to say goodbye to the character 114 can be created, and the viewer 112 can be made to feel a desire to see the character 114 again. When the curtain 350 closes completely, the character 114 is hidden and the conversation time ends. When the conversation time ends, the broadcaster 110 leaves the viewing device 108, and the next broadcaster 110 enters the conversation booth 302.
It should be noted that distribution device 104 may also be provided with a curtain display unit, and curtain 350 may be displayed on the distribution screen using a similar control method.

FIG. 13 is a diagram showing the screen of the viewing device 108 when the viewer 112 touches the character 114.
The viewer 112 can touch the viewed character 114b, not limited to shaking hands. The viewer 112 can "pat" the viewed character 114b with the second hand 344 (the hand of the viewer 112) by touching the head of the viewed character 114b and rubbing the head. More specifically, the second hand detection unit 318 detects the second hand coordinates 346 at the time of touching and determines whether the detection point of the second hand coordinates 346 is in the head area of the viewed character 114b. When the head area is touched and the second hand coordinates 346 makes a reciprocating motion, the second hand detection unit 318 determines that a "pat" operation is being performed.

When the viewed character 114b is stroked, the decoration unit 326 displays a heart-shaped decoration image 360 near the head. The more the viewed character 114b is stroked, the more decoration images 360 the decoration unit 326 displays. The viewer 112 can express affection for the character 114 by stroking the viewed character 114b with the second hand 344. When the viewer 112 strokes the head of the viewed character 114b, the viewed character display unit 180 changes the movement and facial expression of the viewed character 114b. For example, the reaction of the character 114 to the stroking operation is expressed by movements and facial expressions such as closing the eyes, smiling, tickling, and shrugging the shoulders.

In addition, the viewer 112 may touch and swipe the cheek of the viewed character 114b to perform an action of pulling the cheek of the viewed character 114b. In addition, an area that should not be touched on the viewed character 114b (hereinafter referred to as an "inappropriate area"), such as the eyes or the chest, may be set. When the viewer 112 touches the inappropriate area with the second hand 344, the conversation management unit 290 may shorten the time limit of the conversation, or may terminate the conversation and close it. The conversation management unit 290 may shorten the conversation time every time a touch on an inappropriate area is detected. The decoration unit 326 may display a decoration image 360 indicating a warning when a touch on an inappropriate area is detected. According to such a control method, the viewer 112 will no longer touch the viewed character 114b indiscriminately, and the viewer 112 can be made to respect the personality of the character 114. Viewer 112's efforts to interact with character 114 in a way that does not offend him contribute to making character 114 seem more real.

As mentioned above, when the conversation time ends, the curtain 350 slowly closes. The viewer 112 can express their sadness by touching the character 114 disappearing behind the curtain 350 through the gap in the curtain 350.

FIG. 14 is a sequence diagram showing the process when a gift is given to the character 114. In FIG.
The viewer 112 can also give gifts of goods to his/her favorite character 114. At the event venue, various goods (physical objects) such as bouquets of flowers, hats, ribbons, sweets, etc. are sold. These goods are physical entities, and have tags attached thereto in which goods IDs are recorded. These tags may be RFID (Radio Frequency Identifier) tags or stickers in which QR codes (registered trademarks) are recorded.

The viewer 112 presents the purchased goods to the character 114 by dropping the goods into the present box 306. The object drop unit 320 first detects the passage of the goods using the detection device 310 (S30). The object identification unit 322 optically reads the goods ID (S32). The transmission unit 186 of the viewing device 108 transmits the goods ID to the server 102 (S34). The communication unit 152 of the server 102 transmits data of the goods image (virtual object) to the distribution device 104 and the viewing device 108 (S36, S37).

The object display unit 314 of the distribution device 104 displays the goods image on the distribution screen (S38). Similarly, the object display unit 328 of the viewing device 108 displays the goods image on the viewing screen (S40). The live control unit 154 of the server 102 registers the gifted goods as a purchase history by associating the goods ID, the date and time of provision, the user ID of the viewer 112 who provided the goods, and the character ID of the broadcaster 110 (character 114) who provided the goods (S42). Through the above process, the goods are gifted from the viewer 112 to the character 114.

FIG. 15 is a diagram showing the screen of the viewing device 108 when a good is provided to the character 114.
It is assumed that the viewer 112 purchases a hat (goods as a physical entity) and presents it to the character 114. At this time, the object display unit 314 displays a hat image 212 (goods image as a virtual entity). In FIG. 15, the viewed character 114b is wearing the presented hat image 212. If the character 114 wears the hat that the viewer 112 presented, the viewer 112 can feel a sense of satisfaction as if his sincerity was conveyed to the character 114. Since the three-dimensional hat is converted into the two-dimensional hat image 212, the viewer 112 can feel a greater "connection beyond dimensions" with the character 114. In addition, the viewer 112 can enjoy the change in the fashion of his favorite character 114 by presenting the goods. The viewer 112 may present multiple types of goods to the character 114 at the same time.

A portion of the purchase price of the goods is allocated to the broadcaster 110 who plays the character 114. In addition, the goods actually purchased may be given to the broadcaster 110. By purchasing goods, the viewer 112 can express their affection for and support for the character 114. By using the merchandise income as an incentive, the broadcaster 110 can be encouraged to "make an effort to be loved," such as by improving their conversational skills.

FIG. 16 is an external view of the digital signage device 300. As shown in FIG.
The digital signage device 300 is a large-scale advertisement display device targeted at an unspecified number of passersby. The digital signage device 300 in this embodiment will be described as a free-standing structure that can be set up in a station, a square, or the like.

As described in relation to FIG. 1, the digital signage device 300 is connected to the server 102. The digital signage device 300 broadcasts live the conversation between the distributor 110 (character 114) and the viewer 112. The digital signage device 300 displays the character 114 in the same display format as the viewing device 108. A conversation display area 362 is provided at the bottom of the screen, and both the first message (voice of the distributor 110) and the second message (voice of the viewer 112) are displayed in text. By visually checking the digital signage device 300, passersby can enjoy the conversation between the character 114 (distributor 110) and the viewer 112. Publishing the conversation on the digital signage device 300 is also an opportunity for the distributor 110 to appeal to many people about the appeal of the character 114. However, when publishing the conversation between the viewer 112 and the character 114 on the digital signage device 300, it is a prerequisite to obtain the consent of the viewer 112.

FIG. 17 is a functional block diagram of the digital signage device 300. As shown in FIG.
The digital signage device 300 includes a communication unit 352 , a display unit 354 , and a data processing unit 356 .
The communication unit 352 is responsible for communication processing. The display unit 354 displays images. The data processing unit 356 executes various processes based on data acquired by the communication unit 352. The data processing unit 356 also functions as an interface between the communication unit 352 and the display unit 354.

The communication unit 352 includes a transmitting unit 358 and a receiving unit 368.

The data processing unit 356 includes a character conversion unit 366. The live control unit 154 of the server 102 also transmits the first message and the second message to the digital signage device 300. The character conversion unit 366 converts the first message and the second message from voice to a character string (text).

The display unit 354 includes a character display unit 370 and a message display unit 364. The character display unit 370 receives operation information from the server 102 and displays the character 114. The message display unit 364 displays various converted messages in the conversation display area 362.

<Summary>
The live communication system 100 has been described above based on the embodiment.
According to this embodiment, in the conversation booth 302, the viewer 112 can not only enjoy a one-on-one conversation with the character 114, but can also perform operations that mimic a "handshake," further increasing the intimacy between the viewer 112 and the character 114 (broadcaster 110). A handshake is established when the broadcaster 110 and the viewer 112 touch the screen at the same time and bring their fingers close to each other. In order to establish a handshake, an implicit communication between the viewer 112 and the character 114 (broadcaster 110) is necessary. When a handshake is established, the broadcaster 110 can have a unique sensation, as if they are in sync with the character 114 and have communicated with each other.

In addition, when a handshake is established, the character 114 (distributor 110) and viewer 112 can shake their hands by moving their fingers so that they do not move too far apart. The more the handshake is shaken, the more decorative images 360 are displayed. As the viewer 112 and character 114 (distributor 110) continue to shake hands in sync, the viewer 112 can feel an even greater sense of closeness to the character 114.

In this embodiment, the curtain 350 is used to express the beginning and end of the conversation. The curtain 350 opens to reveal the character 114, effectively heightening the viewer's 112 anticipation of the rare opportunity to have a one-on-one conversation with the character 114. In addition, the curtain 350 slowly closes when the conversation ends, which creates a sense of reluctance to part with the character 114. The viewer 112 is likely to feel the desire to say something to the character 114, who is hidden by the curtain 350. This reluctance to part can also arouse in the viewer 112 a desire to meet the character 114 again.

By touching the character 114 himself, the viewer 112 can enjoy physical contact other than a handshake. On the other hand, if the viewer 112 touches an inappropriate area of the character 114, a penalty such as a shortened conversation time is imposed. The distributor 110 may express his "rejection" by refusing to shake hands with the viewer 112 who has behaved in an inappropriate manner. This control method encourages the viewer 112 to interact with the character 114 in a way that respects the personality of the other person.

By broadcasting live or recording and displaying the conversation between the character 114 and the viewer 112 on the digital signage device 300, the appeal of the live communication system 100 can be widely conveyed to users who are not interested in live communication conversations with the character 114 or who have not yet heard of live communication conversations. In addition, the digital signage device 300 is also an opportunity for the distributor 110 to widely promote the appearance and conversational ability of the character 114, so it is considered that the digital signage device 300 is also useful in increasing pride in the job of the distributor 110.

The viewer 112 can purchase goods and give them to the character 114 as a gift. A goods image (two-dimensional object) corresponding to the goods (three-dimensional object) is displayed on the viewing device 108. This provides the viewer 112 with an opportunity to show affection for the character 114, who is a resident of the virtual world, by giving the character 114 a real-world good.

The present invention is not limited to the above-described embodiments and modifications, and can be embodied by modifying the components without departing from the spirit of the invention. Various inventions may be formed by appropriately combining multiple components disclosed in the above-described embodiments and modifications. In addition, some components may be deleted from all the components shown in the above-described embodiments and modifications.

[Modification]
<Hand games>
FIG. 18 is a diagram showing the screen of the audiovisual device 108 when playing a hand game with the character 114.
In the present embodiment, the distributor 110 and the viewer 112 have been described as displaying the first hand 342 and the second hand 344 by touching the distribution screen and the viewing screen, respectively. As a modified example, the positions of the hands of the distributor 110 and the viewer 112 may be detected by image recognition, rather than touching the distribution screen, etc. The distributor 110 and the viewer 112 may wear white gloves, for example. In this case, the first hand detection unit 134 and the second hand detection unit 318 can more reliably recognize the positions of the hands of the distributor 110 and the viewer 112 from the captured image.

The first hand detection unit 134 detects the hand of the distributor 110 through image recognition, and identifies the first hand coordinate 340 based on the detected position of the hand. The technology for converting the position of an object shown in a captured image into coordinates on the screen by projecting it onto the screen is possible through the application of known technology. Similarly, the second hand detection unit 318 detects the position of the hand of the viewer 112 through image recognition, and identifies the second hand coordinate 346.

The first hand detection unit 134 performs image recognition of not only the position of the distributor's 110 hand but also the shape of the hand (condition of the fingers). The second hand detection unit 318 also performs image recognition of the shape of the viewer's 112 hand. The hand display unit 142 etc. changes the shapes of the first hand 342 and the second hand 344 based on the hand shapes of the distributor 110 and the viewer 112, respectively. With this control method, for example, the viewer 112 can enjoy playing hand games such as "rock-paper-scissors" with the character 114 (distributor 110).

The face direction of the viewer 112 can be seen by the broadcaster 110. The face direction of the broadcaster 110 is reflected as the movement of the character 114. If the face direction as well as the hand shapes can be mutually recognized, it is also possible to play games using the body such as "rock, paper, scissors." The winner of the hand game may be determined independently by the broadcaster 110 and the viewer 112, or a "win/loss determination unit" may be provided in the server 102. For example, in the case of rock-paper-scissors, the win/loss determination unit recognizes "rock," "scissors," or "paper" from the hand shapes of the broadcaster 110 and the viewer 112, respectively, and determines whether the character 114 (broadcaster 110) or the viewer 112 has won.

In addition, the viewer 112 may be given merits or demerits depending on whether he wins or loses. For example, if he wins the hand game, the conversation time may be extended, the character 114 (distributor 110) may say lines requested by the viewer 112, or the character 114 may be given a novelty item such as a key chain.

<Photo taken with character 114>
The distributor 110 may take a commemorative photo with the character 114. The imaging unit 316 of the viewing device 108 captures an image of the viewer 112. The data processing unit 166 of the viewing device 108 may include an "image generation unit." The image generation unit generates a commemorative photo of the two of them by combining the captured image of the viewer 112 with the character 114. The transmission unit 186 of the viewing device 108 may transmit the commemorative photo to a mobile terminal of the viewer 112.

In addition to commemorative photographs, short movies may be created by shooting moving images. The viewer 112 may request the character 114 to pose for the commemorative photograph, or may request lines that they would like to hear. A short movie showing the viewer 112 and the character 114 may be presented to the viewer 112 as a memento of their visit, combining the moving image of the character 114 with an image (moving image) captured by the viewer 112.

<How to move your hands>
The viewer 112 may express his/her involvement with the character 114 by the way he/she moves his/her hand when touching the character 114. For example, when the viewer 112 slowly moves his/her hand from side to side while touching the viewing screen, the second hand 344 may express the viewer 112 gently stroking the character 114. Specifically, the second hand detection unit 318 may detect the movement of the viewer 112's hand, and may determine that the hand movement speed is within a predetermined range as "stroking". In addition, when the viewer 112 moves his/her hand quickly, the second hand 344 may express the viewer 112 hitting the character 114. In this case, the character 114 may be penalized as an inappropriate behavior. The decoration unit 326 may display various decoration images 360 on the screen depending on not only the touch location but also the touch method on the character 114.

＜Other＞
The live control unit 154 may register the character 114 who has engaged in inappropriate behavior on a blacklist. The live control unit 154 may register the viewer 112 who came to meet the character 114, the goods provided by the viewer 112, and whether or not the viewer 112 engaged in inappropriate behavior. When starting a conversation, the live control unit 154 transmits and presents various attribute information regarding the viewer 112 to the distributor 110. By referring to this attribute information, the distributor 110 can confirm various personal information, such as whether the viewer 112 is visiting for the first time or whether the viewer 112 gave a present last time, and therefore can respond appropriately to a wide variety of viewers 112 (fans). For example, if the viewer 112 gave the character 114 a hat in the last conversation, the viewer 112 may feel relieved by thanking the character 114 for the hat, knowing that the character 114 remembered him.

Similarly, for viewers 112 who have engaged in inappropriate behavior (such as touching inappropriate areas or making unpleasant remarks), the character 114 (distributor 110) can convey his or her displeasure to the viewer 112 by refusing to shake hands or shortening the conversation time. This method fosters good manners among the viewers 112 (fans) toward the character 114 (idol), making it easier for the viewers 112 to respect the character 114 as an individual. This manner is also thought to be effective in further increasing the sense of reality and importance of the character 114.

In this embodiment, the first hand 342 is displayed separately from the image of the character 114. The hand display unit 324 or the like may express a handshake between the viewer 112 and the character 114 by moving the character 114 itself. Also, the viewer 112 may touch the hand of the character 114 to complete the handshake.

The output unit 172 of the viewing device 108 may output a sound effect when a handshake is made, or may provide a tactile sensation to the viewer 112 using a built-in vibrator. The output unit 172 may also operate a lighting device (not shown) installed in the conversation booth 302 to change the amount of light or color of lighting when a handshake is made, to create the effect of a handshake. The output unit 172 may vary the effects and other presentation for each character 114, or for the number or duration of handshakes. The decoration unit 326 may display multiple types of decoration images 360 for each character 114.

The distribution device 104 may be equipped with a "text conversion unit." The text conversion unit of the distribution device 104 may convert the first message (audio) and the second message (audio) into text and display them on the distribution screen. With this control method, the distributor 110 can continue the conversation while checking the content of the conversation with the viewer 112. The live control unit 154 may record the conversation between the character 114 (distributor 110) and the viewer 112 as text information, and provide the distributor 110 with a record of the past conversation when the conversation begins. By providing the record of the past conversation, the distributor 110 can more easily respond to a large number of viewers 112 in a detailed manner.

In this embodiment, a handshake has been described, but it is not limited to a handshake; a clap of hands or a high-five may also be used. The viewer 112 places the palm of his/her hand on the viewing screen. The distributor 110 also places the palm of his/her hand on the distribution screen. When the hand coordinates of the two people approach within a predetermined range, the hand display unit 324 or the like may display an animation such as a clap of hands or a high-five. At this time, the decoration unit 326 may also display a decoration image 360.

<Other application examples>
The live communication system 100 can also be used for customer service in unmanned stores. For example, the viewing device 108 may be installed in a self-checkout register, which is currently becoming popular in convenience stores and the like. A store clerk (distributor 110) in a remote location interacts with a visitor (viewer 112) as a character 114. The visitor can enjoy shopping while interacting with the character 114. The visitor may also receive advice from the character 114 (distributor 110) about shopping for clothes and the like. By applying the live communication system 100, the visitor can feel the warmth of the customer service even in an unmanned store. In addition, since the broadcaster 110 does not need to be present at the store, there is an advantage in that the operational burden on the store side can be reduced.

In addition to stores, conversation services using characters 114 from remote locations such as nursing homes and kindergartens are also possible. By combining the extraordinary nature of the characters 114 with the real presence of the distributor 110, the ease of interacting with the characters 114 can be combined with the personality of the distributor 110. Another advantage is that the distributor 110 can serve multiple viewers 112 at the same time.

Remote lessons using the live communication system 100 are also possible. For example, an English teacher may be the broadcaster 110 and students may be the viewers 112. The broadcaster 110 (English teacher) may provide English lesson services to viewers 112 (students) from a remote location as a character 114.

The live communication system 100 also enables discussions between characters 114. For example, assume that user X plays character 114x, and user Y plays character 114y. Character 114y is displayed on user X's communication terminal, and character 114x is displayed on user Y's communication terminal. With this control method, user X and user Y can enjoy anonymous conversation while wearing the "mask" of their character 114. Other possibilities include remote parties (joint get-togethers). Because users meet through characters 114, there is a possibility that they can have conversations centered on personality rather than appearance.

100 live communication system, 102 server, 104 distribution device, 106 Internet, 108 viewing device, 110 distributor, 112 viewer, 114 character, 114a distribution character, 114b viewing character, 116 user interface processing unit, 118 imaging unit, 120 data processing unit, 122 communication unit, 124 data storage unit, 126 input unit, 128 output unit, 130 motion detection unit, 132 first message acquisition unit, 134 first hand detection unit, 138 distribution character display unit, 140 second message output unit, 142 hand display unit, 144 motion information generation unit, 148 transmission unit, 150 reception unit, 152 communication unit, 154 live control unit, 156 data storage unit, 158 character storage unit, 162 user interface processing unit, 164 communication unit, 166 Data processing unit, 168 Data storage unit, 170 Input unit, 172 Output unit, 174 Second message acquisition unit, 180 Viewing character display unit, 182 First message output unit, 186 Transmission unit, 190 Reception unit, 196 Viewer conversation screen, 206 Distributor conversation screen, 208 Camera, 212 Hat image, 290 Conversation management unit, 300 Digital signage device, 302 Conversation booth, 304 Camera, 306 Present box, 308 Insertion port, 310 Detection device, 312 Decoration unit, 314 Object display unit, 316 Imaging unit, 318 Second hand detection unit, 320 Object insertion unit, 322 Object identification unit, 324 Hand display unit, 326 Decoration unit, 328 Object display unit, 330 Curtain display unit, 332 Live image output unit, 334 Character display area, 336 Viewer display area, 338 handshake image, 340 first hand coordinates, 342 first hand, 344 second hand, 346 second hand coordinates, 348 handshake point, 350 curtain, 352 communication unit, 354 display unit, 356 data processing unit, 358 transmission unit, 360 decorative image, 362 conversation display area, 364 message display unit, 366 character conversion unit, 368 reception unit, 370 character display unit, d hand distance

Claims (13)

A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmission unit that transmits the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
A live communication system characterized in that the specified display unit of the viewing device determines whether the specified positional relationship is satisfied only when input from both the distributor and the viewer is maintained . A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmission unit that transmits the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
A live communication system characterized in that the specified display unit of the viewing device cancels the specified display when, after the specified display has started, the first input coordinates and the second input coordinates no longer satisfy the specified positional relationship. A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmitter for transmitting the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
the first input detection unit of the broadcasting device detects the first input coordinates based on a touch point when a broadcaster touches a screen of the broadcasting device;
A live communication system, characterized in that the second input detection unit of the viewing device detects the second input coordinates based on a touch point when a viewer touches a screen of the viewing device. A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmission unit that transmits the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
A live communication system characterized in that the viewing character display unit of the viewing device changes the reaction of the character depending on both or either of the touched area and touch manner of the character when a viewer touches the character on the screen of the viewing device. A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmission unit that transmits the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
The viewing device is a live communication system further characterized in that it includes a conversation management unit that shortens the conversation time between the broadcaster and the viewer when a specified part of the character is touched during a conversation between the broadcaster and the viewer via the character. A distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device includes:
An imaging unit that images a broadcaster;
a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor;
a transmission unit that transmits the motion information and the first input coordinates,
The viewing device includes:
a receiving unit that receives the motion information and the first input coordinates;
a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input;
a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information;
a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied,
The viewing device includes:
an object insertion unit that detects an insertion of a physical object by a viewer;
an object identification unit for detecting an ID of the thrown physical object;
and an object display unit that displays a virtual object corresponding to the detected ID on a screen of the viewing device. A live communication system as described in any one of claims 1 to 6, characterized in that the specified display unit of the viewing device determines that the specified positional relationship is satisfied when the distance between the first input coordinate and the second input coordinate is within a first threshold value. The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device ;
A live communication program characterized in that the function of performing the specified display determines whether the specified positional relationship is satisfied only when input from both the distributor and the viewer is maintained . The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device;
A live communication program characterized in that the function of performing the specified display cancels the specified display when the first input coordinate and the second input coordinate no longer satisfy the specified positional relationship after the specified display has started. The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device;
the function of detecting the first input coordinates detects the first input coordinates based on a touch point when a broadcaster touches a screen of the broadcasting device;
A live communication program, characterized in that the function of detecting the second input coordinates detects the second input coordinates based on a touch point when a viewer touches the screen of the viewing device. The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device;
A live communication program characterized in that the function of moving the character changes the reaction of the character depending on both or either of the touched part and the touch manner of the character when a viewer touches the character on the screen of the viewing device. The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device;
The second program is a live communication program that further causes the viewing device to exhibit a function of shortening the conversation time between the broadcaster and the viewer when a specified part of the character is touched during a conversation between the broadcaster and the viewer via the character. The function of capturing images of the broadcaster,
A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor;
A function of detecting a first input coordinate corresponding to a position of an input by a distributor;
a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates;
receiving the motion information and the first input coordinate;
detecting a second input coordinate corresponding to a position of the viewer's input;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information;
a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device;
The second program is
A function to detect the insertion of physical objects by viewers;
detecting an ID of the inserted physical object;
and a function of displaying a virtual object corresponding to the detected ID on a screen of the viewing device.

Images