JP2019200535A - Movement information utilization apparatus and movement information utilization method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a movement information utilization device and a movement information utilization method capable of expanding the utilization range of acquired movement information. A mobile information utilization apparatus (100) includes a person movement data generation unit (111) that generates flow line data of a person based on an image acquired through a surveillance camera, and a terminal held by the person and a communication relay device. A terminal movement data generation unit 112 that generates movement line data of a terminal based on a communication signal transmitted and received by the mobile terminal, a movement data collation unit 113 that collates movement line data of a person with movement line data of the terminal, and movement data collation unit 113. A person / terminal associating unit 114 for associating a person with a terminal based on the matching result by [Selection diagram] Figure 2

Description

  The present invention relates to a movement information utilization apparatus and a movement information utilization method.

  In recent years, surveillance cameras can be used in various places where an unspecified number of people and objects (collectively referred to as “moving objects”) move, such as indoors in stores, open spaces in parks, premises such as stations and airports, and streets. An imaging device (camera) is installed. 2. Description of the Related Art Conventionally, a technique for acquiring movement information such as a movement path of one or more subjects (moving bodies) captured in an image based on an image acquired via a camera is known (for example, see Patent Document 1 below). reference).

  In recent years, a communication system using a wireless local area network (LAN) such as Wi-Fi (registered trademark) has been rapidly spread, and wireless LAN access points (for LAN connection) can be used in various places. The parent machine is installed. Conventionally, a portable communication source (referred to as “portable communication source”) such as a mobile phone or a tablet PC (personal computer) carried by a mobile body, and a fixed communication source such as an access point (“fixed communication source”). There is known a technique for acquiring movement information of a portable communication source held in a mobile body based on a communication signal transmitted / received to / from (refer to, for example, Patent Document 2 below).

JP 2018-28784 A JP 2017-181166 A

  The movement information of the subject acquired based on the image is used for grasping the movement path and direction of one or a plurality of subjects, tracking a specific subject, and the like. On the other hand, the movement information of the portable communication source acquired based on the communication signal transmitted / received between the portable communication source and the fixed communication source also grasps the movement route and direction of the single or plural portable communication sources, and the specific carrying Used for tracking communication sources. Thus, the current situation is that both types of movement information are used for the same limited purpose, and there is a problem of expanding the utilization range of the acquired movement information.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a movement information utilization apparatus and a movement information utilization method capable of expanding the utilization range of acquired movement information.

The movement information utilization apparatus according to the present invention obtains movement information of a subject (for example, persons P0 to P5 in the embodiment) captured in the image based on an image acquired via a camera that captures a predetermined space. Movement information acquisition means (for example, the person movement data generation unit 111 in the embodiment), a portable communication source (for example, the terminals T0 to T3 in the embodiment) held in a moving body that moves in the predetermined space, and a predetermined Second movement information acquisition means for acquiring movement information of the portable communication source based on a communication signal transmitted / received to / from a fixed communication source (for example, the communication repeaters AP1 to AP3 in the embodiment) installed at a position (For example, the terminal movement data generation unit 112 in the embodiment), the movement information of the subject acquired by the first movement information acquisition unit, and the second movement information acquisition. Based on the collation result by the movement information collating means (for example, the movement data collating unit 113 in the embodiment) for collating the movement information of the portable communication source acquired by the means, the subject and the carrying information A subject / communication source associating means (for example, a person / terminal associating unit 114 in the embodiment) for associating with a communication source.

  In the movement information utilization apparatus according to the present invention, the predetermined portable communication source is held based on a communication signal transmitted from a predetermined portable communication source associated with the predetermined subject by the subject / communication source association unit. Individual information acquisition means (for example, the data acquisition unit 120 in the embodiment) for acquiring individual information related to a predetermined moving body, and subject / individual information that associates the individual information acquired by the individual information acquisition means with the predetermined subject It is preferable to include an association unit (for example, the person / individual information association unit 115 in the embodiment).

  Also, individual information output means (for example, implementation) for outputting the individual information associated with the predetermined object by the subject / individual information association means to a predetermined external device (for example, the sales management apparatus 300 in the embodiment). Preferably, the data output unit 140) in the form is provided.

  The individual information acquisition means is preferably configured to acquire the individual information from a predetermined external storage device (for example, the information storage server 200 in the embodiment).

  The movement information utilization method according to the present invention includes a first movement information acquisition step for acquiring movement information of a subject in the image based on an image acquired through a camera that captures the image in the predetermined space; Second movement information for acquiring movement information of the portable communication source based on a communication signal transmitted / received between the portable communication source held by the moving body that moves the vehicle and the fixed communication source installed at a predetermined position A movement information collation step of collating the movement information of the subject acquired in the acquisition step, the first movement information acquisition step and the movement information of the portable communication source acquired in the second movement information acquisition step. And a subject / communication source association step for associating the subject with the portable communication source based on the collation result in the movement information collation step. And butterflies.

  In the movement information utilization device and the movement information utilization method according to the present invention, the object movement information acquired based on the image and the portable information acquired based on the communication signal transmitted and received between the portable communication source and the fixed communication source. The movement information of the communication source is collated, and the subject is associated with the portable communication source based on the collation result. Therefore, for example, a predetermined subject associated with a predetermined portable communication source can be specified (identified) on the image as corresponding to a mobile object that holds the predetermined portable communication source. As described above, according to the present invention, it is possible to widen the utilization range of the acquired movement information of the subject and the movement information of the portable communication source.

  In the present invention, based on a communication signal transmitted from a predetermined portable communication source associated with a predetermined subject, individual information relating to a predetermined mobile object holding the predetermined portable communication source is acquired, and the acquired individual information Is configured to be associated with a predetermined subject, the individual information associated with the predetermined subject can be effectively used as the individual information of the predetermined subject. For example, it is possible to specify a predetermined subject recognized on the image from the associated individual information and use it for marketing, crime prevention measures, or the like.

  In the present invention, by configuring the individual information associated with the predetermined subject to be output to a predetermined external device, the individual information associated with the predetermined subject is determined as the individual information of the predetermined subject. It is possible to quickly use the external device.

In the present invention, by configuring the individual information to be acquired from a predetermined external storage device, a large amount of individual information is stored in advance in the external storage device, and desired individual information can be quickly acquired therefrom. It becomes possible.

It is the schematic which shows the whole structure of the communication system containing the mobile information utilization apparatus which concerns on one Embodiment of this invention. It is a functional block diagram of the said movement information utilization apparatus. It is a flowchart which shows the flow of the process by the said movement information utilization apparatus. It is a figure explaining the outline | summary of the process which matches the person and terminal by the said mobile information utilization apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 schematically shows the overall configuration of a communication system 10 including a mobile information utilization apparatus 100 according to an embodiment of the present invention. First, referring to FIG. 1, the overall configuration of the communication system 10 is outlined. explain.

  A communication system 10 shown in FIG. 1 is mainly composed of a mobile information utilization apparatus 100 and an information storage server 200. The mobile information utilization apparatus 100 and the information storage server 200 are connected via a wide area information communication network NT such as the Internet, and data can be transmitted and received between them. A monitoring camera SC, communication relay devices AP1 to AP3, an image display device 180, and a sales management device 300 are connected to the movement information utilization device 100 via a wired or wireless communication network.

  The surveillance camera SC captures the inside of a predetermined space (in this embodiment, “store” as an example), and is configured to sequentially transmit captured image data to the mobile information utilization apparatus 100. As the monitoring camera SC, for example, a stereo camera can be used. Although one monitoring camera SC is illustrated in FIG. 1, a plurality of monitoring cameras SC may be arranged so that image data from each monitoring camera SC is transmitted to the movement information utilization apparatus 100.

The communication relay devices AP1 to AP3 function as wireless LAN access points such as Wi-Fi and Bluetooth (registered trademark), for example, and are fixed communication sources installed in or around the store. The communication repeaters AP1 to AP3 are configured to transmit and receive predetermined communication signals to and from terminals (only one terminal T0 is shown in FIG. 1) existing in and near the store. As communication signals transmitted / received between the communication repeaters AP1 to AP3 and the terminal T0, for example, beacons transmitted from the communication repeaters AP1 to AP3 to the surroundings, and from the terminal T0 to the communication repeaters AP1 to AP3, for example. In response to a probe request transmitted from the terminal T0 to a beacon from the communication relay devices AP1 to AP3, the communication relay devices AP1 to AP1
Although authentication (Authentication) etc. which are transmitted to AP3 are mentioned, it is not limited to this. Further, the communication repeaters AP1 to AP3 are configured to sequentially transmit information (for example, radio wave intensity and arrival time) of communication signals received from the terminal T0 to the mobile information utilization apparatus 100.

  The terminal T0 is a portable communication source that moves in the store together with a moving object that moves in the store or a person P0 as a subject that appears in the surveillance camera SC. The terminal T0 is, for example, a mobile phone, a tablet-type or notebook-type PC, and can be connected to the wide area information communication network NT via the communication relay devices AP1 to AP3 and the mobile information utilization device 100. And it is comprised so that various information can be transmitted / received between the various communication apparatuses (illustration omitted) connected to the wide area information communication network NT.

The image display device 180 includes an image display unit configured by a liquid crystal display or the like, and is configured to display information such as image data and analysis data output from the movement information utilization device 100 on the image display unit. The image display device 180 is installed in an arbitrary place inside or outside the store.

  The sales management device 300 is, for example, a register for performing point-of-sale information management (POS), and is installed in a store. The sales management apparatus 300 is connected to the sales management server 400 via the wide area information communication network NT, and transmits sales management information and information output from the mobile information utilization apparatus 100 to the sales management server 400. It is configured as follows.

  The information storage server 200 includes individual information relating to a large number of persons (for example, personal information such as name, gender, date of birth, age, address, etc., and authentication information such as an ID used in communication via the wide area information communication network NT). Information such as criminal history and product purchase history). For example, the individual information is provided from a public organization or acquired based on data communicated via the wide area information communication network NT. Further, the information storage server 200 stores authentication information such as an ID in association with the terminal with which the authentication information is communicated. Specifically, for example, terminal-specific information such as a MAC address included in a communication signal transmitted from a terminal and authentication information communicated from the terminal are stored in association with each other. Therefore, if terminal specific information is known, it is possible to search for authentication information associated with the terminal specific information, and further search and output other individual information of a person who uses the authentication information from the authentication information. It has become. The method (mechanism) for searching for individual information shown here is an example, and the present invention is not limited to this.

  Next, the configuration and function of the movement information utilization apparatus 100 will be described with additional reference to FIG. The mobile information utilization apparatus 100 includes various components such as arithmetic processing elements such as CPU and GPU, storage elements such as semiconductor memory and various IC elements, and software such as control programs stored in the storage elements. Composed. Each functional block shown in FIG. 2 schematically shows each function realized by each component.

  As illustrated in FIG. 2, the movement information utilization apparatus 100 includes a data analysis unit 110, a data acquisition unit 120, a data storage unit 130, and a data output unit 140. The data analysis unit 110 further includes person movement data. A generation unit 111, a terminal movement data generation unit 112, a movement data collation unit 113, a person / terminal association unit 114, a person / individual information association unit 115, and a target person feature extraction unit 116 are configured.

  The data acquisition unit 120 receives various data from the outside, for example, image data captured by the monitoring camera SC, information on communication signals received from the terminal T0 by the communication repeaters AP1 to AP3 (here, information on the radio field intensity of the communication signals). And this is referred to as “terminal signal strength information”), terminal-specific information (information such as a MAC address) included in the communication signal, and individual information output from the information storage server 200.

  The data storage unit 130 has a function of storing data acquired by the data acquisition unit 120 and data generated by the data analysis unit 110.

  The data output unit 140 has a function of outputting the data stored in the data storage unit 130 or the data generated by the data analysis unit 110 to the image display device 180 or the sales management device 300.

  The data analysis unit 110 has a function of performing various analyzes based on the data acquired by the data acquisition unit 120 and the data stored in the data storage unit 130 and generating new data based on the analysis result. Further, the data analysis unit 110 has a function as artificial intelligence (AI) capable of executing machine learning such as deep learning to improve analysis accuracy.

  The person movement data generation unit 111 is based on the image data acquired by the data acquisition unit 120 from the surveillance camera SC, and includes movement information (here, a person's walking movement) included in the image data (in the image). It has a function of generating line information, which is referred to as “person flow line data”. Note that the movement information of the person is not limited to information on the walking flow line of the person.

  Based on the strength information of the terminal signal acquired by the data acquisition unit 120 from the communication relays AP1 to AP3, the terminal movement data generation unit 112 transmits the movement information of the terminals that communicated with the communication relays AP1 to AP3 (here, the movement of the terminal It has a function of generating flow line information, which is referred to as “terminal flow line data”. Note that the movement information of the terminal is not limited to information on the movement flow line of the terminal.

  The movement data matching unit 113 has a function of matching the person flow line data generated by the person movement data generation unit 111 with the terminal flow line data generated by the terminal movement data generation unit 112.

  The person / terminal associating unit 114 has a function of associating a person with a terminal based on the collation result by the movement data collating unit 113.

  The person / individual information association unit 115 has a function of associating individual information acquired by the data acquisition unit 120 from the information storage server 200 with a person.

  Based on the image data acquired by the data acquisition unit 120 from the monitoring camera SC, the target person feature extraction unit 116 uses the image recognition technology to convert the appearance characteristics (features such as appearance and face) of the person included in the image data. It has the function to extract and convert it into data.

  Next, the contents of processing (also referred to as “movement information utilization processing”) executed for utilization of movement information in the movement information utilization apparatus 100 will be described with additional reference to FIGS. 3 and 4. The content of this movement information utilization process corresponds to one embodiment of the movement information utilization method according to the present invention.

  In the movement information utilization process, as shown in FIG. 3, image data is acquired from the surveillance camera SC (step S1), and person flow line data is generated based on the acquired image data (step S2; first movement information). Equivalent to acquisition step). The person flow line data is generated, for example, by extracting a person shown in the image based on the image data, tracking the extracted person, and acquiring the tracking result as a walking flow line (such a method). Not limited). In the upper left part of FIG. 4, as an example of the person flow line data, an image of each walking flow line within a predetermined time of five persons P1 to P5 shown in the image is schematically shown. .

  Next, the terminal signal strength information is acquired (step S3), and terminal flow line data is generated based on the acquired terminal signal strength information (step S4; corresponding to the second movement information acquisition step). For example, the terminal flow line data is obtained by specifying the existence position of each terminal for every predetermined time, and acquiring the transition locus of the existence position as the movement flow line of each terminal (limited to such a method). Not.) The location (positioning) of each terminal is determined based on the principle of three-point positioning using, for example, the strength information of the terminal signal. Specifically, the distance between the terminal and each of the communication repeaters AP1 to AP3 is estimated based on the radio field intensity from one terminal received by each of the communication repeaters AP1 to AP3, and the presence of the terminal is determined from the estimation result. The position (coordinates) is specified (not limited to such a method). It is assumed that the installation positions (coordinates) of the communication repeaters AP1 to AP3 are known. In the upper right part of FIG. 4, as an example of the terminal flow line data, a schematic image of the movement lines of each of the three terminals T1 to T3 communicating with the communication relay devices AP1 to AP3 within a predetermined time is imaged. Is shown.

  Next, person flow line data and terminal flow line data are collated (step S5; equivalent to a movement information collation step). This collation, for example, quantifies the walking flow line of each person P1 to P5 based on the person flow line data, and quantifies the movement flow line of each terminal T1 to T3 based on the terminal flow line data. This is performed by obtaining the degree of correlation (not limited to such a method).

  Next, based on the collation result, the person is associated with the terminal to generate person / terminal association data (step S6; corresponding to the subject / communication source association step). The person / terminal association data associates, for example, those having a high degree of correlation between the walking flow lines of the persons P1 to P5 and the movement flow lines of the terminals T1 to T3. If there is a walking flow line and a movement flow line, they are generated by associating the person of the walking flow line with the terminal of the movement flow line (not limited to such a method). In the lower part of FIG. 4, as an example of the person / terminal association data, the person P2 and the terminal T1, the person P4 and the terminal T2, and the person P5 and the terminal T3, which are associated with each other, are imaged. This is shown schematically. Hereinafter, a person associated with a terminal is referred to as a “terminal corresponding person”.

  Next, the individual information of the terminal corresponding person is acquired (step S7). In the case of this example, the individual information of the person P2 associated with the terminal T1, the individual information of the person P4 associated with the terminal T2, and the individual information of the person P5 associated with the terminal T3 are acquired. Individual information of the terminal-corresponding person is acquired from the information storage server 200. Specifically, for example, in the case of the individual information of the person P2 associated with the terminal T1, first, in the mobile information utilization device 100, the terminal included in the communication signal transmitted from the terminal T1 to the communication repeaters AP1 to AP3. Get unique information. Next, the terminal specific information is transmitted to the information storage server 200 to search whether authentication information and individual information associated with the transmitted terminal specific information are stored. The information storage server 200 executes a search, and when the corresponding individual information is stored, transmits the individual information to the mobile information utilization apparatus 100. The movement information utilization apparatus 100 acquires the transmitted individual information as individual information of the person P2 associated with the terminal T1.

  Next, the terminal-corresponding person is associated with the individual information to generate terminal-corresponding person / individual information associating data (step S8). The terminal-corresponding person / individual information association data is stored, for example, by associating a terminal, a person (for example, appearance feature data extracted from image data regarding the person), and individual information with each other. It can be data.

  Next, the generated data is output (step S9). For example, person / terminal association data or terminal correspondence person / individual information association data is output to the image display device 180, or terminal correspondence person / individual information association data is output to the sales management device 300. To do. Further, the individual information of the terminal-compatible person acquired from the information storage server 200 may be output as it is to the sales management apparatus 300 or the like.

  The individual information (including those included in the terminal correspondence person / individual information association data) of the terminal compatible person output to the sales management apparatus 300 is transmitted from the sales management apparatus 300 to the sales management server 400, Based on this, it is possible to analyze what kind of person has purchased what kind of product. Therefore, the individual information of the terminal-compatible person can be effectively used for marketing or the like. Conventionally, there is an idea of using an image recognition technique to estimate a person's gender, age, clothing preference, etc., and using it for marketing or the like. However, information that can be estimated by image recognition is limited, and the estimated content is not always correct. On the other hand, since the individual information of the terminal corresponding person acquired by the mobile information utilization apparatus 100 is highly reliable, the utility value for marketing or the like is very high.

In addition, the person / terminal association data and the terminal correspondence person / individual information association data output to the image display device 180 are displayed on an image display unit (for example, on an image taken by the monitoring camera SC), so that It is possible to use it to grasp on the image what kind of person is visiting the store. It can also be used to prevent crimes such as shoplifting, and to identify and track the person who executed the crime. For example, when it can be confirmed on the image that a certain person has shoplifted, if there is terminal-corresponding person / individual information association data related to that person, the person is identified from that (individual information included in the data) It becomes possible.

  It is also possible to provide services suitable for a person who has acquired individual information based on terminal-compatible person / individual information association data at the time of visiting the store or at the time of payment. For example, information on purchase history at other stores, etc. is analyzed from the individual information of the target person, and based on the analysis results, products that the person is likely to like are selected and introduced, information on discounts on the products, etc. Or the like, and thereby, the satisfaction of the store visitor can be improved.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, It can change suitably. For example, in the above-described embodiment, the place where the surveillance camera SC and the communication relay devices AP1 to AP3 are installed is a store, but any place other than the store, for example, a plaza such as a park, a station, an airport, etc. These may be installed on the premises, streets, and the like. Further, the number of cameras and fixed communication sources (communication repeaters) can be arbitrarily changed according to the size of the range to be monitored.

  In the above-described embodiment, the individual information of the moving object (person) is stored in the external storage device (information storage server 200). However, the mobile information utilization device may store the individual information. Further, the camera, the fixed communication source, and the mobile information utilization apparatus may be connected via a wide area information communication network. Further, the function of the movement information utilization apparatus may be distributed to a plurality of functions such as transferring the function of the data analysis unit 110 to the information storage server 200.

  In the above-described embodiment, the fixed communication source is a communication repeater that functions as a wireless LAN access point. However, the fixed communication source may be a wireless base station in a telephone line network. In addition, the portable communication source is not limited to a terminal such as a mobile phone, but may be any shape or type as long as it is configured to acquire individual information of a mobile body that holds the communication signal transmitted from the mobile communication source. Any size is acceptable. For example, a communication device formed in a badge shape or a wristwatch shape and worn by a person, or a very small communication device embedded in the body can be used as a portable communication source. Further, the moving body (subject) is not limited to a person, and may be an animal or an object other than a person.

10 Communication System 100 Mobile Information Utilization Device 200 Information Storage Server (External Storage Device)
300 Sales management device (external device)
AP1 to AP3 Communication repeater (fixed communication source)
P0-P5 Person (moving body / subject)
T0 to T3 terminal (portable communication source)
SC Surveillance Camera NT Wide Area Information Communication Network

Claims (5)

First movement information acquisition means for acquiring movement information of a subject shown in the image based on an image acquired via a camera that captures a predetermined space;
Based on a communication signal transmitted / received between a portable communication source held by a mobile body moving in the predetermined space and a fixed communication source installed at a predetermined position, movement information of the portable communication source is acquired. Two movement information acquisition means;
Movement information collating means for collating movement information of the subject acquired by the first movement information acquiring means and movement information of the portable communication source acquired by the second movement information acquiring means;
A movement information utilization apparatus comprising: a subject / communication source association unit that associates the subject with the portable communication source based on a collation result by the movement information collation unit. Based on a communication signal transmitted from a predetermined portable communication source associated with a predetermined subject by the subject / communication source association unit, individual information relating to a predetermined mobile object holding the predetermined portable communication source is acquired. Individual information acquisition means;
The movement information utilization apparatus according to claim 1, further comprising: a subject / individual information association unit that associates the individual information acquired by the individual information acquisition unit with the predetermined subject.   3. The movement information utilization apparatus according to claim 2, further comprising individual information output means for outputting the individual information associated with the predetermined subject by the subject / individual information associating means to a predetermined external device. .   The movement information utilization apparatus according to claim 2 or 3, wherein the individual information acquisition unit is configured to acquire the individual information from a predetermined external storage device. A first movement information acquisition step for acquiring movement information of a subject shown in the image based on an image acquired via a camera that captures a predetermined space;
Based on a communication signal transmitted / received between a portable communication source held by a mobile body moving in the predetermined space and a fixed communication source installed at a predetermined position, movement information of the portable communication source is acquired. Two movement information acquisition steps;
A movement information collation step for collating the movement information of the subject acquired in the first movement information acquisition step with the movement information of the portable communication source acquired in the second movement information acquisition step;
A movement information utilization method comprising: a subject / communication source association step for associating the subject with the portable communication source based on a collation result in the movement information collation step.

Images