TW201911119A - Can be used in systems without cashiers and their components - Google Patents

Abstract

Provide systems and technologies to track the drop and removal of inventory items in the real space area by the subject. A plurality of cameras with overlapping viewing fields generate separate sequences of images of corresponding viewing fields in the real space. In one embodiment, the system includes a first image processor including a subject image recognition engine that receives a corresponding sequence of images from the plurality of cameras. The first image processors process the images to identify the subjects represented in the images in the corresponding sequences of the images. The system includes a second image processor, including a background image recognition engine, and receives a corresponding sequence of images from the plurality of cameras. The second image processor masks the identified subjects to generate a masked image. Continuing here, the second image processors process the occluded images to identify and classify the background changes represented in the images in the corresponding sequences of the images.

Description

System and components that can be used for cashierless checkout

The present invention relates to a system and its components that can be used for cashierless checkout.

Difficulties in image processing occur when images from most cameras deployed in a large space are used to identify and track the subject's actions.

Tracking the actions of subjects in areas of real space, such as people in a shopping store, poses many technical challenges. For example, considering this image processing system deployed in a shopping store, most consumers move among walkways and open spaces between shelves in the shopping store. Consumers remove items from the shelves and place those items in their respective shopping carts or baskets. Consumers can also put items on the shelf if they don't want the item.

When the consumer is performing these actions, different parts of the consumer and different parts of the shelf or other display structures holding the store's inventory will be blocked from images from different cameras, because other consumers, shelves, and product displays And so on. At the same time, there may be many consumers in the store at any given moment, making it difficult to identify and track individuals and their actions at any time.

It is desirable to provide a system that can more efficiently and automatically identify and track the dropping and removing of subjects in a large space; and perform other procedures that support the complex interaction of the subject with its environment, including, for example, cashierless checkout Features.

A system and method for operating the system are provided to track changes in subjects (such as people) in areas through real space, and other complex interactions with subjects in its environment, using image processing. This function of tracking changes by image processing causes complex problems in computer engineering, regarding the type of image data to be processed, what processing of image data should be performed, and how to determine actions from image data with high reliability. The system described in this article can only use images from its camera located above the real space, so that it does not need to refurbish store shelves and floor spaces with sensors, etc., for deployment in a given setting.

Provided is a system and method for tracking down and removal of an inventory item by a subject in an area including a real space of an inventory display structure, which includes using a plurality of cameras disposed on the inventory display structures to generate the The respective sequences of the images of the inventory display structures are in the corresponding viewing domain in the real space, and the viewing domain of each camera overlaps the viewing domain of at least one other camera among the plurality of cameras. Using these sequences of images, a system and method is described as: detecting the drop and removal of inventory items by semantically identifying significant changes in those sequences of images of inventory items on the inventory display structure; and Associate these significant changes in semantics with the subjects represented in the sequences of the image.

Provided is a system and method for tracking down and removal of inventory items of a subject in an area of real space, including using a plurality of cameras disposed on the inventory display structures to generate the inventory display structures The respective sequences of the images are in the corresponding viewing domain in the real space, and the viewing domain of each camera overlaps the viewing domain of at least one other camera among the plurality of cameras. Using these sequences of images, a system and method is described that detects the drop and removal of inventory items by processing foreground data in the sequences of images to identify the subject's posture and inventory items related to the postures.

At the same time, a system and method are described that combine foreground processing and background processing in the same sequence of images. In this combined manner, the provided system and method include using sequences of images to detect inventory by processing foreground data in the sequences of images to identify the subject's posture and inventory items related to the postures. Drop and take of items; and use these sequences of images to detect significant changes in those sequences of images that semantically identify them in relation to inventory items on the inventory display structure, and to detect drop and take of inventory items, and Associate these significant changes in semantics with the subjects represented in the sequences of the image.

In the embodiment described herein, the system uses a plurality of cameras to generate individual sequences of images of corresponding viewing fields in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The system includes a first image processor including a subject image recognition engine that receives a corresponding sequence of images from the plurality of cameras. The first image processors process the images to identify the subjects represented in the images in the corresponding sequences of the images. The system further includes a second image processor, including a background image recognition engine, which receives a corresponding sequence of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images, and process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

In one embodiment, the background image recognition engines include a convolutional neural network. The system includes logic to associate an identified background change with an identified subject.

In one embodiment, the second image processors include a background image storage for storing background images of corresponding sequences of the images. The second image processors further include masking logic to process the images in the sequences of the images instead of the background image data representing the front stage image data of the identified subjects. The background image data is collected from the background images of the corresponding sequences of the images to provide the occluded images.

In one embodiment, the occlusion logic combines a plurality of groups of N occluded images in the sequences of images to generate a sequence of factorized images for each camera. The second image processors process the sequence of factorized images to identify and classify background changes.

In one embodiment, the second image processors include logic to generate a changed data structure for the corresponding sequences of the image. The changed data structure includes the coordinates in the occluded image of the identified background change, the identifier of the inventory item subject of the identified background change, and the category of the identified background change. The second image processors further include coordination logic to process the changed data structure from a plurality of cameras with overlapping viewing fields to find the identified background changes in real space.

In an embodiment, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item is added or removed relative to the background image.

In another embodiment, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item is added or removed relative to the background image. The system further includes logic to associate a background change with the identified subject. Finally, the system includes logic to perform the detection of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure.

In another embodiment, the system includes logic to associate a background change with the identified subject. The system further includes logic to perform detection of the inventory items taken by the identified entities and detection of the inventory items dropped by the identified entities on the inventory display structure.

The system may include a third image processor as described herein, including a front-end image recognition engine that receives a corresponding sequence of images from the plurality of cameras. The third image processors process the images to identify and classify the preceding stage changes represented in the images in the corresponding sequences of the images.

A system and method for operating a system are provided for tracking multiple joint subjects (such as people) in real space. The system uses a plurality of cameras to generate individual sequences of images of corresponding viewing fields in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The sequence processes the images in the sequences to generate an array of joint data structures corresponding to each image. The array of the joint data structure corresponding to a specific image is used to classify the components of a specific image by the joint type, the time of the specific image, and the coordinates of the components in the specific image. The system then transforms the coordinates of the elements in the array of joint data structures corresponding to the images in different sequences into candidate joints with coordinates in real space. Finally, the system identifies clusters of candidate joints, where the clusters include respective sets of candidate joints with coordinates in real space, becoming multi-joint subjects in real space.

In one embodiment, the image recognition engines include a convolutional neural network. The processing of the image by the image recognition engine includes a confidence array of components that generate the image. The confidence array of a particular component of the image includes confidence values for a plurality of joint types of the particular component. The confidence arrays are used to select the joint type of the joint data structure of the specific component according to the confidence array.

In one embodiment of a system for tracking multi-joint subjects, identifying a set of candidate joints includes applying a heuristic function to identify the set of candidate joints as multi-joint subjects based on a physical relationship between the joints of the subject in real space . The process includes storing the sets of joints that it identified as a multi-joint subject. Identifying the set of candidate joints includes determining whether a candidate joint identified in an image acquired at a particular time matches a member of one of those sets of candidate joints that it identified as a multi-joint subject in a previous image.

In an embodiment, the sequences of the images are synchronized such that the images in each of the sequences of the images captured by the plurality of cameras are representative of the time scale of the subject's movement through the space. Real space at a single point in time.

The coordinate system in the real space of a member of the set of candidate joints identified as the multi-joint subject identifies the position in the region of the multi-joint subject. In some embodiments, the processing includes simultaneous tracking of the positions of the plurality of multi-joint subjects in the area of real space. In some embodiments, the process includes determining when one of the plurality of multi-articulated subjects leaves the region of real space. In some embodiments, the process includes determining a direction in which the multi-joint subject faces at a given point in time. In the embodiment described herein, the system uses a plurality of cameras to generate individual sequences of images of corresponding viewing fields in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The system processes the images in the sequences of the images it receives from the plurality of cameras to identify the subjects represented in the images and generates categories of the identified subjects. Finally, the system processes the categories of the identified subjects of the collection of images in the sequences of the images to detect the removal of the inventory item by the identified subject and the placement of the inventory item on the shelf by the identified subject.

In one embodiment, the category identifies whether the identified entity holds inventory items. This category also identifies whether the hand of the identified subject is close to the shelf or whether the hand of the identified subject is close to the identified subject. The category of whether the hand is close to the identified subject may include whether the hand of the identified subject is close to a basket related to the identified subject, and whether it is close to the body of the identified subject.

With the described technique, an image representing the subject's hand in the viewing domain can be processed to generate categories of the subject's hand in a plurality of images in a time series. The categories of hands from the sequence of images can be processed using a convolutional neural network (in some embodiments) to identify actions by the subject. These actions may be the dropping and removing of inventory items (such as those described in the embodiment described herein), or other types of actions that can be decrypted by processing the image of the hand.

With the described technique, the images are processed to identify subjects in the viewing domain and locate the joints of the subjects. The positioning of the joints of the subjects can be processed as described herein to identify bounding boxes in the corresponding images that include the hands of the subjects. The data in the bounding boxes can be the processed category of the subject's hand in the corresponding image. The category of hand from an identified subject, which is generated from a sequence of images in this manner, can be processed to identify actions by that subject.

In a system including a plurality of image recognition engines (such as both a front desk and a background image recognition engine), the system can perform: detection of the inventory items taken by the identified subjects and drop of inventory items by the identified subjects in inventory The first set of inspections on the display structure, and the second set of inspections on the inventory display structure by which the identified entities take inventory items away and the identified entities drop the inventory items. The selection logic used to process the first and second sets of tests can be used to generate a log data structure. The log data structure includes a list of inventory items for identified entities.

In the embodiment described herein, the sequences of images from the cameras in the plurality of cameras are synchronized. The same camera and the same sequence of images are used by both the front stage and the background image processor in a preferred embodiment. As a result, the redundant detection of put-down and take-out of inventory items is performed using the same input data to allow high confidence (and high accuracy) in the obtained data.

In one of the techniques described herein, the system includes logic that detects the placement and removal of inventory items by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. This can be done using the front-end image recognition engine in conjunction with the subject image recognition engine described in the text.

In another technique described herein, the system includes logic to detect drop and take-off of inventory items by: semantically identifying inventory items on inventory display structures (such as shelves) over time Significant changes and associated semantically significant changes with the subjects represented in the sequences of the image. This can be accomplished using a background image recognition engine in conjunction with a subject image recognition engine as described in the text.

In applying the system described in the article, both pose analysis and semantic difference analysis can be combined and performed on the same sequence of synchronized images from the camera's array.

Methods and computer program products that can be executed by a computer system are also described herein.

Other forms and advantages of the present invention can be found in the drawings, detailed description, and review of the scope of patent application (which is continued below).

The following description is presented to enable any person skilled in the art to implement and use the invention and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily understood by those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Therefore, the present invention is not intended to be limited to the illustrated embodiments but is to be accorded the widest scope consistent with the principles and features disclosed herein. System Overview

A system and various embodiments of the main technology are described with reference to FIGS. 1-28A / 28B. The system and program are described with reference to FIG. 1 and are schematic diagrams of the architecture of the system according to the embodiment. Because Figure 1 is an architectural diagram, some details have been omitted to improve the clarity of the description.

The discussion of Figure 1 is as follows. First, the components of the system will be described, followed by their interconnection. Next, the use of components in the system is described in more detail.

FIG. 1 provides a block diagram diagram of a system 100. The system 100 includes a camera 114, network nodes controlling the image recognition engines 112a, 112b, and 112n, a tracking engine 110, a calibrator 120, a main body database 140, and a network node (or node) deployed on the network. Training database 150, heuristic database 160 (for joint heuristics, for putting down and removing heuristics, and other heuristics for coordinating and combining the output of most image recognition engines as described below), Calibration database 170, and communication network or network 181. The network node can control only one image recognition engine, or several image recognition engines, as described in the text. The system can also include an inventory database and other supporting data.

As used herein, a network node is an addressable hardware device or a virtual device attached to a network and capable of transmitting, receiving, or passing information to or from other network nodes through a communication channel. Examples of electronic devices that can be deployed as hardware network nodes include computers, workstations, laptops, handheld computers, and all types of smartphones. Network nodes can be implemented in a cloud-based server system. More than one virtual device configured as a network node can be implemented using a single physical device.

For the sake of simplicity, only three network nodes controlling the image recognition engine are displayed in the system 100. However, any number of network nodes controlling the image recognition engine can be connected to the tracking engine 110 through the network 181. At the same time, the image recognition engine, tracking engine, and other processing engines described herein may be executed using more than one network node in a decentralized architecture.

The interconnection of the elements of the system 100 will now be described. The network 181 couples the network nodes 101a, 101b, and 101c, which are, respectively, the master image recognition engines 112a, 112b, and 112n, the network node 102, the adjuster 120, and the main data of the master tracking engine 110. The database 140, the training database 150, the joint heuristics database 160, and the adjustment database 170. The camera 114 is connected to the tracking engine 110 through network nodes controlling the image recognition engines 112a, 112b, and 112n. In one embodiment, the camera 114 is installed in a shopping store (such as a supermarket) so that its collection of cameras 114 (two or more) with overlapping viewing fields is placed above each aisle to capture the reality in the store Imagery of space. In FIG. 1, two cameras are disposed above the aisle 116a, two cameras are disposed above the aisle 116b, and three cameras are disposed above the aisle 116n. The camera 114 is installed above a walkway with overlapping viewing fields. In this embodiment, the camera is configured with the goal that a consumer moving in the aisle of a shopping store appears in the viewing domain of two or more cameras at any point in time.

The cameras 114 may be synchronized with each other in time such that their images are captured simultaneously (or close in time) and at the same image capture rate. The cameras 114 can transmit the respective continuous streams of the images to the network nodes of the master image recognition engines 112a-112n at a predetermined rate. The images captured by all the cameras that simultaneously (or closely in time) cover the area of real space are synchronized, because the synchronized images can be identified in the processing engine, such as representing a fixed position in real space Different perspectives of the subject. For example, in one embodiment, the camera transmits image frames to the network nodes of the respective master image recognition engines 112a-112n at a rate of 30 frames per second (fps). Each frame has a time stamp, camera identification (abbreviated as "camera_id"), and frame identification (abbreviated as "frame_id"), along with image data.

The cameras installed above the aisle are connected to respective image recognition engines. For example, in FIG. 1, two cameras installed above the aisle 116a are connected to the network node 101a of the master image recognition engine 112a. Similarly, the two cameras installed above the aisle 116b are connected to the network node 101b of the master image recognition engine 112b. Each of the image recognition engines 112a-112n controlled by the network nodes or nodes 101a-101n separately processes the image frames received from one of the cameras in the illustrated example.

In one embodiment, each of the image recognition engines 112a, 112b, and 112n is implemented as a deep learning algorithm, such as a convolutional neural network (abbreviated as CNN). In this embodiment, the CNN is trained using the training database 150. In the embodiment described herein, the image recognition of subjects in real space is based on identifying and clustering identifiable joints in the images, where the group of joints can be assigned to a separate subject. For this joint-based analysis, the training database 150 has a large collection of images for each different type of joint of the subject. In an exemplary embodiment of a shopping store, the subject is a consumer moving in a hallway between shelves. In an exemplary embodiment, during training of a CNN, the system 100 is referred to as a "training system". After using the training database 150 to train the CNN, the CNN is switched to a generation mode to process images of consumers in a shopping store in real time. In an exemplary embodiment, during generation, the system 100 is referred to as a runtime system (also known as an inference system). The CNN in each image recognition engine generates an array of joint data structures of the images in their respective streams of the images. In one embodiment as described herein, the array of joint data structures is generated for each processed image, so that each of its image recognition engines 112a-112n generates an output stream of the array of joint data structures. These arrays of joint data structures from cameras with overlapping viewing fields are further processed to form groups of joints, and these groups of joints are identified as subjects.

Camera 114 is tuned before switching CNN to production mode. The calibrator 120 calibrates the cameras and stores the calibration data in the calibration database 170.

The tracking engine 110 (hosted on the network node 102) receives a continuous stream of arrays of joint data structures from the subjects of the image recognition engines 112a-112n. The tracking engine 110 processes an array of joint data structures and transforms the coordinates of elements in the array of joint data structures corresponding to the images in different sequences into candidate joints with coordinates in real space. For each set of synchronized images, the combination of candidate joints identified throughout the real space can be considered (for analogy) as a galaxy as a candidate joint. For each subsequent point in time, the movement of the candidate joint is recorded so that its galaxy changes over time. The output of the tracking engine 110 is stored in the subject database 140.

The tracking engine 110 uses logic to identify groups or sets of candidate joints with coordinates in the real space as subjects in the real space. For the purpose of analogy, each set of candidate points is like a cluster of candidate joints at each point in time. The cluster of candidate joints can move over time.

The logic to identify a set of candidate joints includes a heuristic function based on the physical relationship between subjects in real space. These heuristic functions are used to identify sets of candidate joints as subjects. Heuristic functions are stored in a heuristic database 160. The output of the tracking engine 110 is stored in the subject database 140. Therefore, the sets of candidate joints include individual candidate joints that have a relationship with other individual candidate joints based on heuristic parameters, and candidates that have been identified (or can be identified) as a given set of individual subjects. A subset of joints.

The actual communication path through the network 181 may be a point-to-point through a public and / or private network. Communication can occur over a variety of networks 182, such as private networks, VPNs, MPLS circuits, or the Internet; and appropriate application programming interfaces (APIs) and data exchange formats such as Representational State Transfer (REST) ), JavaScript ^TM Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java ^TM Message Service (JMS), and / or Java platform module system. All these communications can be encrypted. Communication is usually through a network such as LAN (Local Area Network), WAN (Wide Area Network), Telephone Network (Public Switched Telephone Network PSTN)), Session Initiation Protocol (SIP), Wireless Network, Point-to-Point Network, Star network, token ring network, hub network, Internet, including mobile Internet via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. In addition, multiple authorization and authentication technologies such as username / password, open authorization (OAuth), Kerberos, SecureID, digital credentials, and more can be used to ensure communication.

The techniques disclosed in this article can be implemented in the context of any computer-implemented system, including database systems, multi-tenant environments, or connected database implementations, such as Oracle ™ -compatible connected implementations, IBM DB2 Enterprise Server ™ Compatible Connected Implementation, MySQL ™ or PostgreSQL ™ Compatible Connected Implementation or Microsoft SQL Server ™ Compatible Connected Implementation; or NoSQL ™ Non-Related Database Implementation, such as Vampire ™ Compatible Unconnected implementation, Apache Cassandra ™ compatible unconnected implementation, BigTable ™ compatible unconnected implementation, or HBase ™ or DynamoDB ™ compatible unconnected implementation. In addition, the disclosed techniques can be implemented using: different programming models, such as MapReduce ™, large block synchronous programming, MPI primitives, etc .; or different scalable batch and stream management systems, such as Apache Storm ™ , Apache Spark ™, Apache Kafka ™, Apache Flink ™, Truviso ™, Amazon Elasticsearch Service ™, Amazon Web Services ™ (AWS), IBM Info-Sphere ™, Borealis ™, and Yahoo! S4 ™. Camera configuration

The camera 114 is configured to track a multi-joint cell (or subject) in a three-dimensional (abbreviated as 3D) real space. In an exemplary embodiment of a shopping store, the real space may include an area of a shopping store in which sales items are stacked in a shelf. Points in real space can be represented by the (x, y, z) coordinate system. Points in the area of the real space in which the system is deployed are covered by the viewing field of the two or more cameras 114.

In a shopping store, shelves and other inventory display structures can be configured in a variety of ways, such as along a wall of a shopping store, or in a row forming a walkway, or a combination of both configurations. Figure 2 shows the configuration of the shelf (which forms the aisle 116a), as seen from one end of the aisle 116a. Two cameras (camera A 206 and camera B 208) are placed above the aisle 116a at a predetermined distance from the roof 230 and the floor 220 of the shopping store above the inventory display structure, such as a shelf. The camera 114 includes a camera disposed above and having a viewing area covering various parts of the inventory display structure and floor area in a real space. The coordinates in the real space of the members of the set of candidate joints identified as the subject identify positions in the subject's floor area. In an exemplary embodiment of a shopping store, the real space may include all floors 220 in the shopping store for inventory to be accessed there. The camera 114 is placed and oriented so that the area of its floor 220 and shelf can be seen by at least two cameras. The camera 114 also covers at least a portion of the shelves 202 and 204 and the floor space in front of the shelves 202 and 204. The camera angle is selected to have both a steep perspective (straight down) and an angled perspective (which provides a more complete body image of the consumer). In an exemplary embodiment, the camera 114 is configured to be eight (8) feet high or higher throughout the shopping store. Figure 13 presents a diagram of this embodiment.

In FIG. 2, the cameras 206 and 208 have overlapping viewing domains, which cover the space between the shelf A 202 and the shelf B 204, and the viewing domains 216 and 218 overlap each other. The position in real space is represented as the (x, y, z) point in the real space coordinate system. "X" and "y" represent positions on a two-dimensional (2D) plane (which may be the floor 220 of a shopping store). The value "z" is the height of a point above a 2D plane on the floor 220 in a configuration.

FIG. 3 illustrates the aisle 116a viewed from the top of FIG. 2, which further shows an example configuration of the positions of the cameras 206 and 208 above the aisle 116a. The cameras 206 and 208 are placed closer to the opposite end of the aisle 116a. Camera A 206 is placed a predetermined distance from shelf A 202, and camera B 208 is placed a predetermined distance from shelf B 204. In another embodiment, more than two cameras are placed above the aisle, and the cameras are placed at equal distances from each other. In this embodiment, two cameras are placed near opposite ends and a third camera is placed in the middle of the aisle. It should be understood that several different camera configurations are possible. Camera calibration

The camera adjuster 120 performs two types of adjustments: internal and external. During the internal adjustment, the internal parameters of the camera 114 are adjusted. Examples of internal camera parameters include focus length, principal points, skew, fisheye coefficient, and so on. Various techniques for internal camera calibration can be used. One such technology was proposed by Zhang in "A flexible new technique for camera calibration", which was published in IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22, No. 11, November 2000.

In external calibration, external camera parameters are adjusted to generate mapping parameters for transforming 2D image data into 3D coordinates in real space. In one embodiment, a subject, such as a person, is introduced into real space. The subject moves through the real space on a path through the viewing field of each camera 114. At any given point in the real space, the master system appears in the viewing domain of at least two cameras that form a 3D scene. However, the two cameras have different views of the same 3D scene in their respective two-dimensional (2D) image planes. Features in a 3D scene, such as the subject's left wrist, are viewed by two cameras at different positions in their respective 2D image planes.

Point correspondences are established between each pair of cameras and the overlapping viewing domain of a given scene. Because each camera has a different perspective on the same 3D scene, the point corresponds to a two-pixel position (a position from the positions of the cameras with overlapping viewing fields), which represents the projection of the same point in the 3D scene. Many points are identified for each 3D scene using the results of the image recognition engines 112a-112n for external calibration purposes. The image recognition engine recognizes the position of the joint as the (x, y) coordinates of pixels in the 2D image plane of each camera 114, such as the number of columns and rows. In one embodiment, the joint is one of 19 different types of joints of the subject. As the subject moves through the viewing fields of the different cameras, the tracking engine 110 receives the (x, y) coordinates of each of the 19 different types of joints of the subject it uses to adjust the camera 114 from each image.

For example, consider the image from camera A and the image from camera B, both of which were acquired at the same time point and have overlapping viewing areas. There are pixels in the image from camera A, which correspond to the pixels in the synchronized image from camera B. Consider that it has a certain point of an object or surface in the viewpoint of camera A and camera B and that point is captured in the pixels of the two image frames. In external camera calibration, most such points are identified and referred to as corresponding points. Because there is a subject in the viewing area of camera A and camera B during calibration, key joints of this subject are identified, for example, the center of the left wrist. If these key joints are visible in the image frames from both camera A and camera B, it is assumed that these represent the corresponding points. This process is repeated over many image frames to create a large set of corresponding points for all camera pairs with overlapping viewing fields. In one embodiment, the images are streamed out of all cameras at a rate of 30 FPS (frames per second) or more and a resolution of 720 pixels in full RGB (red, green, and blue) colors. These images have the form of a one-dimensional array (also known as a flat array).

The large number of images collected for the subject above can be used to determine corresponding points between cameras with overlapping viewing fields. Consider two cameras A and B with overlapping viewing fields. The plane passing through the centers of the cameras A and B and the joint positions (also referred to as feature points) in the 3D scene is referred to as an "epipolar plane". The intersection of this core surface with the 2D image planes of cameras A and B defines the "epipolar line". Given these corresponding points, a transformation is determined, which can accurately map the corresponding points from camera A to the epipolar lines in the viewing domain of camera B whose corresponding points are ensured to cross the image frame of camera B. This transformation is generated using the image frames collected above for the subject. This transformation is known in the art to be non-linear. The general form is further known as the need to compensate for the radial deformation of the lens of each camera, as well as the non-linear coordinate transformation of movement to and from the projected space. In the external camera calibration, the approximation of the ideal nonlinear transformation is determined by solving the nonlinear optimization problem. This non-linear optimization function is used by the tracking engine 110 to identify the same joints in the output (an array of joint data structures) of different image recognition engines 112a-112n, and processes the images of the camera 114 with overlapping viewing fields. The results of the internal and external camera calibration are stored in the calibration database 170.

Various techniques can be used to determine the relative position of points in the image of the camera 114 in real space. For example, Longuet-Higgins published "Computer Algorithms for Reconstructing Scenes from Two Projections" in Nature, Volume 293, September 10, 1981. This paper proposes to calculate the three-dimensional structure of the scene from the perspective projection of the relevant pair, when the spatial relationship between the two projections is unknown. Longuet-Higgins' paper proposes a technique to determine the position of each camera in real space relative to other cameras. In addition, their technology allows triangulation of subjects in real space, using images from cameras 114 with overlapping viewing fields to identify the value of the z coordinate (height from the floor). Any point in the real space (for example, the end of a shelf in a corner of the real space) is designated as the (0, 0, 0) point on the (x, y, z) coordinate system of the real space.

In one embodiment of the technology, externally adjusted parameters are stored in two data structures. The first data structure stores essential parameters. The essential parameter is the projection transformation from 3D coordinates to 2D image coordinates. The first data structure contains the essential parameters of each camera, as shown below. The data values are all floating point numbers. This data structure stores a 3x3 essential matrix, expressed as "K" and deformation coefficient. The deformation coefficient includes six radial deformation coefficients and two tangential deformation coefficients. Radial deformation occurs when the light ray is bent closer to the edge of the lens (compared to bending at its optical center). Tangential deformation occurs when the lens and the image plane are not parallel. The following data structure only shows the value of the first camera. Similar data is stored for all cameras 114.

The second data structure stores each pair of cameras: 3x3 basic matrix (F), 3x3 basic matrix (E), 3x4 projection matrix (P), 3x3 rotation matrix (R), and 3x1 transformation vector (t). This information is used to convert points in the reference frame of one camera to the reference frame of another camera. For each pair of cameras, eight homography coefficients are also stored to map the plane of the floor 220 from one camera to another. The base matrix is a relationship between two images of the same scene, which restricts where the projection of points from that scene can occur in the two images. The basic matrix is also the relationship between two images of the same scene, with the camera being adjusted. The projection matrix provides a vector space projection from the 3D real space to the subject. The rotation matrix is used to perform rotations in Euclidean space. The transformation vector "t" represents a geometric transformation, which moves each point of a figure or a space in the predetermined direction by the same distance. The homography_floor_factor is used to combine images of features of the subject on the floor 220 as viewed by cameras with overlapping viewing fields. The second data structure is shown below. Similar information is stored for all pairs of cameras. As previously indicated, x represents a floating point number of values. Network configuration

FIG. 4 presents a network architecture 400 for controlling an image recognition engine. The system includes a plurality of network nodes 101a-101n in the illustrated embodiment. In this embodiment, the network node is also referred to as a processing platform. Processing platforms 101a-101n and cameras 412, 414, 416, ... 418 are connected to a network 481.

FIG. 4 shows a plurality of cameras 412, 414, 416, ... 418 connected to the network. A large number of cameras can be deployed in specific systems. In one embodiment, cameras 412 to 418 are connected to network 481 using Ethernet-based connectors 422, 424, 426, and 428, respectively. In this embodiment, the Ethernet-based connector has a data transfer speed of one billion bits per second, which is also called a billion-bit Ethernet. It should be understood that in other embodiments, the camera 114 is connected to the network, using other types of network connections that may have faster or slower data transfer rates than gigabit Ethernet. Meanwhile, in an alternative embodiment, a group of cameras may be directly connected to the processing platforms, and the processing platforms may be coupled to a network.

The storage subsystem 430 is a storage basic programming and data structure, which provides functions of some embodiments of the present invention. For example, each module implementing the function of the multiple image recognition engine may be stored in the storage subsystem 430. The storage subsystem 430 is an example of a computer-readable memory, which includes a non-transitory data storage medium having computer instructions stored in the memory, which can be executed by a computer to perform the data processing and image processing described in the text All or any combination, including logic to: identify changes in real space, track subjects, and drop and remove inventory items in areas of real space through procedures as described in the text. In other examples, computer instructions may be stored in other types of memory, including portable memory, which includes non-transitory data storage media or media that can be read by a computer.

These software modules are typically executed by the processor subsystem 450. The host memory subsystem 432 generally includes several memories, including a main random access memory (RAM) 434 (for storing instructions and data during program execution) and a read-only memory (ROM) 436 (for storing fixed instructions) ). In one embodiment, the RAM 434 is used as a buffer to store the video stream from the camera 114 which is connected to the platform 101a.

The file storage subsystem 440 provides persistent storage for program and data files. In an exemplary embodiment, the storage subsystem 440 includes four 120 gigabyte (GB) solid state drives (SSDs) in a RAID 0 (redundant array of independent hard drives) configuration (identified by the number 442). . In the exemplary embodiment (where CNN is used to identify the joints of the subject), RAID 0 442 is used to store training data. During training, training data other than RAM 434 was read from RAID 0 442. Similarly, when an image is recorded for training purposes, its data not in RAM 434 is stored in RAID 0 442. In the exemplary embodiment, a hard disk drive (HDD) 446 is stored in 10 megabytes. It has a slower access speed than RAID 0 442 storage. A solid state drive (SSD) 444 contains an operating system and related files for the image recognition engine 112a.

In an example configuration, three cameras 412, 414, and 416 are connected to the processing platform 101a. Each camera has a dedicated graphics processing unit GPU 1 462, GPU 2 464, and GPU 3 466 for processing images transmitted by the camera. It should be understood that less than or more than three cameras may be connected to each processing platform. Therefore, fewer or more GPUs are configured in the network node, so that each camera has a dedicated GPU to process the image frame received from the camera. The processor subsystem 450, the storage subsystem 430, and the GPUs 462, 464, and 466 all use a bus subsystem 454 to communicate.

Several peripheral devices, such as a network interface subsystem, a user interface output device, and a user interface input device, are also connected to the streamline subsystem 454, which forms part of the processing platform 101a. These subsystems and devices are intentionally not shown in FIG. 4 to improve the clarity of the description. Although the busbar subsystem 454 is shown schematically as a single busbar, alternative embodiments of the busbar subsystem may use a majority of the busbars.

In an embodiment, the camera 412 may be implemented using Chameleon3 1.3 MP Color USB3 Vision (Sony ICX445), which has a resolution of 1288x964, a frame rate of 30 FPS, and a resolution of 1.3 megapixels per image (MegaPixels). 300-∞ zoom lens with working distance (mm), viewing field with 98.2 域-23.8˚ 1/3 "sensor. Convolutional neural network

The image recognition engine in the processing platform receives a continuous stream of images at a predetermined rate. In one embodiment, the image recognition engines include a convolutional neural network (abbreviated as CNN).

FIG. 5 illustrates the processing of the image frame by the CNN represented by the number 500. The input image 510 is a matrix composed of image pixels arranged in columns and rows. In one embodiment, the input image 510 has a width of 1280 pixels, a height of 720 pixels, and three channels of red, blue, and green (also referred to as RGB). The channels are imagined as three 1280x720 two-dimensional images stacked on top of each other. Therefore, the input image has a dimension of 1280x720x3 as shown in FIG. 5.

The 2x2 filter 520 is convolved with the input image 510. In this embodiment, no padding is applied when the filter is convolved with the input. Following this, a non-linear function is applied to the convolved image. In this embodiment, the corrected linear unit (ReLU) activation is used. Other examples of non-linear functions include sigmoid, hyperbolic tangent (tanh), and changes in ReLU, such as leaking ReLU. The search is performed to find the hyperparameter values. The hyper-parameters are C ₁ , C ₂ , ....., C _N , where C _N represents the number of channels of the convolution layer "N". Typical values of N and C are shown in FIG. 5. There are twenty-five (25) layers in CNN, as represented by N equal to 25. The value of C is the number of channels in each of the convolutional layers of layers 1 to 25. In other embodiments, additional features are added to the CNN 500, such as residual connections, squeeze excitation modules, and multiple resolutions.

In a typical CNN used for image classification, the size (width and height dimensions) of an image is reduced as the image is processed through a convolutional layer. It is helpful for feature recognition because the goal is to predict the category of the input image. However, in the illustrated embodiment, the size (i.e., image width and height dimensions) of the input image is not reduced because the goal is not only to identify joints (also called features) in the image frame, but to It also identifies its location in the image, so it can be mapped to coordinates in real space. Therefore, as shown in FIG. 5, as the process proceeds through the CNN's convolutional layer, the width and height dimensions of the image remain unchanged, in this example.

In one embodiment, the CNN 500 identifies 19 possible joints of the subjects on each element of the image. These possible joints can be grouped into two categories: foot joints and non-foot joints. A joint type of type 19 is for all non-joint features of the subject (ie, elements that are not classified as joint images). Foot joints: Ankle joint (left and right) Non-foot joints: neck nose eyes (left and right) ear (left and right) shoulder (left and right) elbow (left and right) wrist (left and right) hip (Left and right) knee (left and right) not a joint

As can be seen, for the purposes of this specification, a "joint" is a traceable feature of a subject in real space. The joint may correspond to a physiological joint on the subjects, or other features (such as eyes, or nose).

The first set of analyses for a stream of input images is to identify traceable features of a subject in real space. In one embodiment, this is called "joint analysis". In this embodiment, the CNN used for joint analysis is referred to as a "joint CNN". In one embodiment, joint analysis is performed thirty times per second on a thirty frame per second received from the corresponding camera. The analysis is synchronized in time (ie, 1/30 ^{th of a} second), and images from all cameras 114 are analyzed in the corresponding joint CNN to identify the joints of all subjects in real space. The results of this analysis of images from multiple cameras from multiple moments are stored as "snapshots".

The snapshot may be in the form of a dictionary containing an array of joint data structures from the images of all cameras 114 at a time, which represents a cluster of candidate joints in the area of real space covered by the system. In one embodiment, the snapshot is stored in the subject database 140.

In this example CNN, the softmax function is applied to each element of the image in the final layer of the convolutional layer 530. The softmax function transforms an arbitrary real-valued K-dimensional vector into a real-valued K-dimensional vector in the range [0, 1] (which is increased up to 1). In one embodiment, the image element is a single pixel. The softmax function converts an arbitrary real-valued 19-dimensional array (also known as a 19-dimensional vector) of each pixel into a real-valued 19-dimensional confidence array in the range [0, 1] (which is added up to 1). The 19 dimensions of the pixels in the image frame correspond to the 19 channels in the final layer of the CNN, which further corresponds to the 19 types of joints of the subjects.

A large number of picture elements can be classified as one of each of the 19 types of joints in an image, based on the number of subjects in the viewing domain of the camera from which the image originated.

The image recognition engines 112a-112n process the image to generate a confidence array of components for the image. The confidence array of a particular component of the image includes confidence values for a plurality of joint types of the particular component. Each of the image recognition engines 112a-112n (respectively) generates an output matrix 540 of the confidence array for each image. Finally, each image recognition engine generates an array of joint data structures of each output matrix 540 corresponding to the confidence array of each image. The array of the joint data structure corresponding to a specific image is used to classify the components of a specific image by the joint type, the time of the specific image, and the coordinates of the components in the specific image. The joint type of the joint data structure of a specific component in each image is selected based on the value of the confidence array.

The joints of the subjects can be regarded as being distributed in the output matrix 540 and become a heat map. Thermal maps can be parsed to display image elements with the highest value (peak) for each joint type. Ideally, for a given picture element with a high value for a particular joint type, the surrounding picture elements outside a certain distance from the given picture element will have a lower value for the joint type, so that it has the value of the joint type The position of a particular joint can be identified in the image space coordinates. Accordingly, the confidence array of the image element will have the highest confidence value for the joint and lower confidence values for the remaining 18 types of joints.

In one embodiment, the batches of images from each camera 114 are processed by respective image recognition engines. For example, six consecutive time-stamped images are sequentially processed in a batch to take advantage of cache coherence. The parameters for one layer of the CNN 500 are loaded into memory and applied to the batch of six image frames. The parameters for the next layer are then loaded into memory and applied to the batch of six images. This is repeated for all convolutional layers 530 in CNN 500. Cache coherence reduces processing time and improves the performance of the image recognition engine.

In one such embodiment, for three-dimensional (3D) convolutions, the performance of the CNN 500 is further improved by sharing information across image frames in the batch. This is to assist in more accurate joint identification and reduce false positives. For example, a feature in an image frame (where the pixel value across most image frames in a given batch does not change) is likely to be a static object (such as a shelf). A change in the value of the same pixel across an image frame in a given batch indicates that this pixel is likely to be a joint. As a result, CNN 500 can focus more on processing the pixel to correctly identify its joints identified by the pixel. Joint data structure

The output of the CNN 500 is a matrix of confidence arrays for each image of each camera. The matrix of the confidence array is transformed into an array of joint data structures. The joint data structure shown in FIG. 6 is used to store information of each joint. The joint data structure 600 identifies the x and y positions of the components in a specific image in the 2D image space of the camera (the image is received from the camera). The joint number system identifies the types of joints it has identified. For example, in one embodiment, the values range from 1 to 19. A value of 1 indicates that the joint is a left ankle, a value of 2 indicates that the joint is a right ankle, and so on. The type of joint is selected using a confidence array for that element in the output matrix 540. For example, in one embodiment, if the value corresponding to the left ankle joint is the highest in the confidence array for the image element, the value of the joint number is "1".

The confidence number indicates the degree of confidence in the CNN 500 when predicting the joint. If the value of the confidence number is high, it means that CNN is confident in its prediction. The integer Id is assigned to the joint data structure to uniquely identify it. Following the above mapping, the output matrix 540 of the confidence array of each image is converted into an array of joint data structures of each image.

The image recognition engines 112a-112n receive a sequence of images from the camera 114 and process the images to generate a corresponding array of joint data structures as described above. An array of joint data structures for a particular image is used to classify the components of a particular image by the type of joint, the time of the particular image, and the coordinates of the elements in the particular image. In an embodiment, the image recognition engines 112a-112n are convolutional neural network CNN 500, the joint type is one of the 19 types of joints of the subjects, and the specific image time is determined by the source camera 114 for the specific image The time stamp of the generated image, and the coordinates (x, y) identify the position of the component on the 2D image plane.

In one embodiment, joint analysis includes performing a combination of k nearest neighbors, Gaussian mixtures, various image morphological transformations, and joint CNNs on each input image. The result contains an array of joint data structures that can be stored in the form of bit masks in the ring buffer, which maps the number of images to the bit masks at each time. Tracking engine

The tracking engine 110 is configured to receive an array of joint data structures generated by the image recognition engines 112a-112n, corresponding to images in a sequence of images from cameras with overlapping viewing fields. The array of the joint data structure of each image is transmitted from the image recognition engines 112a-112n to the tracking engine 110 via the network 181 as shown in FIG. The tracking engine 110 transforms the coordinates of elements in an array of joint data structures corresponding to the images in different sequences into candidate joints with coordinates in real space. The tracking engine 110 includes logic to identify a set of candidate joints (cluster of joints) with coordinates in the real space as a subject in the real space. In one embodiment, the tracking engine 110 accumulates an array of joint data structures of image recognition engines from all cameras at a given time, and stores this information as a dictionary in the subject database 140 for identifying a cluster of candidate joints . The dictionary can be configured in the form of key-value pairs, where the key is the camera id and the value is an array of joint data structures from the camera. In this embodiment, the dictionary is used for heuristic-based analysis to determine candidate joints and the assignment of joints to the subject. In this embodiment, the high-level input, processing, and output of the tracking engine 110 are illustrated in Table 1. Table 1: Inputs, processing, and outputs from the tracking engine 110 in an example embodiment. Cluster joints as candidate joints

The tracking engine 110 receives an array of joint data structures along two dimensions: time and space. Along the time dimension, the tracking engine sequentially receives a time stamped array of joint data structures processed by the image recognition engines 112a-112n of each camera. The joint data structure includes most instances of the same joints of the same subject during a period of time, in images from cameras with overlapping viewing fields. The (x, y) coordinates of the components in a particular image will usually be different in the ordered time stamp array of the joint data structure due to the movement of the subject to which the particular joint belongs. For example, twenty picture elements classified as a left wrist joint can appear in many sequential and time-stamped images from a particular camera, each left wrist joint having a real space that can change or remain constant with the image Its location. As a result, the twenty left wrist joint data structures 600 in the sequential time-stamped array of many joint data structures may represent the same twenty joints in real space during a certain period.

Because most cameras with overlapping viewing fields cover locations in real space, the same joint can appear in more than one image of camera 114 at any given moment. The cameras 114 are synchronized in time, so the tracking engine 110 receives joint data structures from specific joints of most cameras with overlapping viewing fields at any given moment. This is a spatial dimension (the second of two dimensions: time and space), and the tracking engine 110 receives data in an array of joint data structures along the spatial dimension.

The tracking engine 110 uses the initial set of heuristics stored in the heuristic database 160 to identify candidate joint data structures from an array of joint data structures. The goal is to minimize overall metrics over a period of time. The overall metric calculator 702 calculates an overall metric. The overall metric is the sum of the many values described below. Intuitively, the value of the overall metric is minimal when the joints in the array of joint data structures received by the tracking engine 110 along the temporal and spatial dimensions are correctly assigned to individual subjects. For example, consider an embodiment of a shopping store with consumers moving in the aisle. If Consumer A's left wrist is incorrectly assigned to Consumer B, the value of the overall metric will increase. Therefore, minimizing the overall measurement of each joint of each consumer is an optimization problem. One option to solve this problem is to try all possible connections of the joint. However, this can become difficult to handle as the number of consumers increases.

The second way to solve this problem is to use heuristics to reduce the possible combinations of members of a member of the set of candidate joints that it identified as a single subject. For example, the left wrist joint must not belong to that subject in space away from other joints of a subject due to the known physiological characteristics of the relative positions of the joints. Similarly, a left wrist joint with a small change in position with the image is unlikely to belong to a subject with the same node from the same position in the image that is far away in time, because the subject is not expected to be able to move at extremely high speed mobile. These initial heuristics are used to establish temporal and spatial boundaries for clusters of candidate joints that can be classified as specific subjects. The joints in the joint data structure within a specific temporal and spatial boundary are considered "candidate joints" for the collection of candidate joints assigned to subjects as they appear in real space. These candidate joints include joints identified in an array of joint data structures from most images from the same camera, which span a period of time (time dimension) and span different cameras (spatial dimensions) with overlapping viewing domains. Foot joint

Joints can be partitioned for the purpose of a procedure to group joints into clusters (become foot and non-foot joints as shown in the list of joints above). The left and right ankle joint types in the current example are considered foot joints for the purpose of this procedure. The tracking engine 110 may begin using the foot joints to identify a set of candidate joints for a particular subject. In the embodiment of the shopping store, the consumer's feet are on the floor 220 as shown in FIG. 2. The distance from the camera 114 to the floor 220 is known. Therefore, the tracking engine 110 can assume a known depth (distance along the z-axis) when combining the joint data structure of its foot joint from an array of data joint data structures corresponding to the images of cameras with overlapping viewing fields. . The depth of the value of the foot joint is zero, that is, (x, y, 0) in the (x, y, z) coordinate system of the real space. Using this information, the image tracking engine 110 applies homography mapping to combine the joint data structures of the foot joints from cameras with overlapping viewing fields to identify candidate foot joints. Using this mapping, the positions of the joints in (x, y) coordinates in image space are transformed to the positions in (x, y, z) coordinates in real space, resulting in candidate foot joints. This procedure is performed separately to identify candidate left and right foot joints using individual joint data structures.

Following this, the tracking engine 110 may combine the candidate left foot joint and the candidate right foot joint (assigned to a set of candidate joints) to generate a subject. Other joints from the candidate joint galaxy can be linked to the subject to build a cluster of some or all joint types of the resulting subject.

If there is only one left candidate foot joint and one right candidate foot joint, it means that there is only one subject in the specific space at the specific time. The tracking engine 110 generates a new subject with left and right candidate foot joints belonging to its joint set. The subject is stored in the subject database 140. If there are a majority of candidate left and right foot joints, the overall metric calculator 702 attempts to combine each candidate left foot joint to each candidate right foot joint to generate a subject such that the value of its overall metric is minimized. Non-foot joints

To identify candidate non-foot joints from an array of joint data structures within specific temporal and spatial boundaries, the tracking engine 110 uses a non-linear transformation (also from any given camera A to its neighboring camera B (with overlapping viewing domains) (also (Called the base matrix). The non-linear transformation is calculated using a single multi-joint body and stored in the calibration database 170 as described above. For example, for two cameras A and B with overlapping viewing fields, candidate non-foot joints are identified as follows. Non-foot joints in the array corresponding to the joint data structure of the components in the image frame from camera A are mapped to the epipolar lines in the synchronized image frame from camera B. The joints identified by the joint data structure in the array of joint data structures of the specific image of camera A (also known as features in the machine vision literature) will appear on the corresponding epipolar lines if they appear in the image of camera B . For example, if the joint in the joint data structure from camera A is the left wrist joint, the left wrist joint on the epipolar line in the image of camera B represents the same left wrist joint from the viewpoint of camera B. These two points in the images of cameras A and B are projections of the same points in a 3D scene in real space and are called "conjugate pairs".

Machine vision technologies (such as those published in the paper by Longuet-Higgins, entitled "Computer Algorithms for Reconstructing Scenes from Two Projections", were applied in Nature, Volume 293, September 10, 1981) to The conjugate pairs of corresponding points determine the height of the joint from the floor 220 in the real space. The application of the above method requires a predetermined mapping between cameras with overlapping viewing fields. This data is stored in the calibration database 170 and becomes a non-linear function determined during the calibration of the camera 114 described above.

The tracking engine 110 receives an array of joint data structures corresponding to images in a sequence of images from cameras with overlapping viewing fields, and transforms the coordinates of elements in the array of joint data structures corresponding to images in different sequences into Candidate non-foot joints for coordinates in real space. The identified candidate non-foot joints are clustered into a collection of subjects with coordinates in real space using the overall metric calculator 702. The overall metric calculator 702 calculates an overall metric value and attempts to minimize the value by examining different combinations of non-foot joints. In one embodiment, the overall metric is the sum of heuristics organized in four categories. The logic used to identify the set of candidate joints includes a heuristic function based on the physical relationship between the joints of the subject in the real space to identify the set of candidate joints as the subject. An example of the physical relationship between joints is considered in the heuristics described below. Heuristics of the first kind

The first type of heuristics includes measurements to determine the similarity between two proposed subject-joint positions in the same camera perspective at the same or different times. In one embodiment, these measures are floating point values, where higher values indicate that the two lists of joints are likely to belong to the same subject. Considering an exemplary embodiment of a shopping store, these measures determine the distance between the same joints of a consumer in a camera from one image to the next image along the time dimension. Given camera A in the viewing domain of camera 412, the first set of measurements determines the distance between each of the joints of person A from an image from camera 412 to an image below camera 412. These measures are applied to the joint data structure 600 in an array of joint data structures for each image from the camera 114.

In one embodiment, two example metrics in the first type of heuristic are listed below: 1. Between the left ankle joint of two subjects on the floor and the right ankle joint of two subjects on the floor The reciprocal of the Euclidean 2D coordinate distances between them (using the x, y coordinate values for a particular image from a particular camera) is added together. 2. The sum of the Euclidean 2D coordinate distances between each pair of non-foot joints of the subject in the image frame. Heuristics of the second kind

The second type of heuristics includes measurements to determine the similarity between the two proposed subject-joint positions from the viewing field of most cameras at the same time. In one embodiment, these measures are floating point values, where higher values indicate that the two lists of joints are likely to belong to the same subject. Considering the exemplary embodiment of the shopping store, the second set of measurements determines the distance between the same joints of the consumer in the image frames from two or more cameras (with overlapping viewing fields) at the same time.

In one embodiment, two example metrics of the second type of heuristic are listed below: 1. Between the left ankle joint of two subjects on the floor and the right ankle joint of two subjects on the floor The reciprocal of the Euclidean 2D coordinate distances between them (using the x, y coordinate values for a particular image from a particular camera) is added together. The ankle joint position of the first subject is projected to the camera, where the second subject is made visible by homography mapping. 2. The sum of all pairs of joints of the inverse of Euclidean 2D coordinates between a line and a point, where the line is from the first camera with the first subject in its viewing field to the second camera with the second subject in it The epipolar line of the joint of the image of the second camera in the viewing domain, and the point is the joint of the second subject in the image of the second camera. The third kind of heuristics

A third type of heuristics includes measurements to determine the similarity between all joints between the proposed subject-joint position in the same camera perspective at the same time. Considering an exemplary embodiment of a shopping store, this type of measurement determines the distance between the joints of the consumer in a frame from a camera. The fourth kind of heuristics

The fourth type of heuristics includes measurements to determine the degree of disparity between the proposed subject-joint positions. In one embodiment, these metrics are floating point values. Higher values indicate that the two lists of joints are more likely not to be the same subject. In an embodiment, two exemplary measures in this category include: 1. The distance between the neck joints of two proposed subjects. 2. The sum of the distance between two pairs of joints between two subjects.

In one embodiment, the thresholds that can be determined empirically are applied to the metrics listed above: 1. Thresholds are used to determine when a metric value is small enough to consider whether a joint belongs to A known subject. 2. Threshold value, which is used to determine when there are too many potential candidate subjects. One joint may belong to a score with too good measure similarity. 3. Threshold value, used to determine when the set of joints has a sufficiently high degree of similarity to be considered as a new subject, which has not previously appeared in real space. 4. Threshold value to determine when the subject is no longer in real space. 5. Threshold value, used to determine when the tracking engine 110 has made an error and has confused the two subjects.

The tracking engine 110 includes logic to store a collection of joints that it identifies as a subject. The logic used to identify a set of candidate joints includes logic to determine whether a candidate joint identified in an image acquired at a particular time matches a member of one of the sets of candidate joints that it identified as a subject in a previous image . In one embodiment, the tracking engine 110 compares the current joint position of the subject with the previously recorded joint position of the same subject at regular intervals. This comparison allows the tracking engine 110 to update the joint position of the subject in the real space. In addition, using this approach, the tracking engine 110 identifies false positives (ie, misidentified subjects) and removes subjects that no longer appear in the real space.

Consider the example of the shopping store embodiment, in which the tracking engine 110 generates the consumer (subject) at an earlier time, however, after a certain time, the tracking engine 110 does not have the current joint position of the specific consumer. It indicates that the consumer was incorrectly generated. The tracking engine 110 deletes incorrectly generated subjects from the subject database 140. In one embodiment, the tracking engine 110 also uses the above procedure to remove positively identified subjects from real space. Considering the example of a shopping store, when a consumer leaves the shopping store, the tracking engine 110 deletes the corresponding consumer record from the main body database 140. In one such embodiment, the tracking engine 110 updates the consumer's record in the main body database 140 to indicate that "the consumer has left the store."

In one embodiment, the tracking engine 110 attempts to identify the subject by applying both foot and non-foot heuristics simultaneously. This leads to "islands" of the joints of these subjects. As the tracking engine 110 processes further arrays of joint data structures along time and space dimensions, the size of the island increases. Finally, the island of the joint is merged into the other islands of the joint to form the subject, which is then stored in the subject database 140. In one embodiment, the tracking engine 110 maintains a record of unassigned joints for a predetermined period of time. During this time, the tracking engine attempts to assign unassigned joints to existing subjects or generate new multi-joint cells from these unassigned joints. The tracking engine 110 discards the unassigned joints after a predetermined period of time. It should be understood that in other embodiments, different heuristics than those listed above are used to identify and track subjects.

In one embodiment, the user interface output device of the node 102 connected to the master tracking engine 110 displays the position of each subject in the real space. In one such embodiment, the display of the output device is renewed at regular intervals to the new positions of the subjects. Main data structure

The joints of the main body are connected to each other using the above-mentioned measures. In doing so, the tracking engine 110 generates new subjects and updates the positions of existing subjects by updating their respective joint positions. FIG. 8 shows a main data structure 800 for storing a main body. The data structure 800 stores data related to the subject as a key-value dictionary. The key is a box_number and the value is another key-value dictionary, where the key is the camera_id and the value is a list of 18 joints (of the subject), with its position in real space. The subject data is stored in the subject database 140. Each new subject is also assigned a unique identifier that is used to access the subject's data in the subject database 140.

In one embodiment, the system recognizes the joints of the subject and generates the skeleton of the subject. The bone is projected into the real space to indicate the position and orientation of the subject in the real space. This is also called "posture estimation" in the field of machine vision. In one embodiment, the system displays the orientation and position of the subject in the real space on a graphical user interface (GUI). In one embodiment, the image analysis is anonymous, that is, the unique identifier assigned to the subject generated by the joint analysis does not identify the personal identity details (such as name, email address) of any particular subject in real space , Address, credit card number, bank account number, driver's license number, etc.). Program flow

Several flowcharts illustrating logic are described in the text. Logic may be implemented as: using a processor configured as described above, which is programmed using a computer program stored in a memory accessible and executable by those processors; and in its configuration, By using dedicated logic hardware (including field programmable integrated circuit), and by combining dedicated logic hardware and computer programs. With all flowcharts in the text, it should be understood that many steps can be combined, performed in parallel, or performed in different sequences without affecting the functions achieved. In some cases, as the reader will understand: the reconfiguration of steps will achieve the same result, only when some other changes are performed simultaneously. In other cases, as the reader will understand: the reconfiguration of steps will achieve the same result, only when certain conditions are met. Furthermore, it should be understood that the flowchart in the text only shows the steps which have an understanding of the embodiment, and it should be understood that various other steps to perform other functions can be performed before, after and between those shown.

FIG. 9 is a flowchart illustrating the procedure steps for tracking a subject. The process starts at step 902. The camera 114 with a viewing field having a region in the real space is adjusted at step 904. The video program is executed by the image recognition engines 112a-112n in step 906. In one embodiment, a video program is executed on each camera to process batches of image frames received from the respective cameras. The output of all video programs from the respective image recognition engines 112a-112n is provided as input to a scene program performed by the tracking engine 110 in step 908. The scene program recognizes the new subject and updates the joint position of the existing subject. In step 910, it is checked whether there are more image frames to be processed. If there are more image frames, the process continues at step 906, otherwise the process ends at step 914.

A more detailed procedure of the procedure step 904 "Tuning the camera in real space" is presented in the flowchart of FIG. 10. The calibration procedure starts at step 1002 by identifying the (0, 0, 0) point of the (x, y, z) coordinates of the real space. At step 1004, the first camera having the position (0, 0, 0) in its viewing domain is adjusted. More details on camera calibration were previously filed in this application. At step 1006, the next camera having an overlapping viewing field with the first camera is adjusted. In step 1008, it is checked whether there are more cameras to be adjusted. The process is repeated at step 1006 until all cameras 114 are calibrated.

In next program step 1010, the subject is introduced into real space to identify conjugate pairs of corresponding points between cameras with overlapping viewing fields. Some details of this procedure are described above. The process is repeated at step 1012 for each pair of overlapping cameras. If there are no more cameras, the process ends (step 1014).

The flowchart in Figure 11 shows more detailed steps of step 906 of the "video program". At step 1102, k consecutive timestamped images per camera are selected as a batch for further processing. In an embodiment, the value of k = 6 is calculated according to the available memory of the video program in the network nodes 101a-101n, and these network nodes 101a-101n are the main control image recognition engines 112a-112n. . In the next step 1104, the size of the image is set to an appropriate size. In one embodiment, the image has a width of 1280 pixels, a height of 702 pixels, and three channels of RGB (representing red, green, and blue). At step 1106, a complex trained convolutional neural network (CNN) processes the images and generates an array of joint data structures for each image. The output of the CNN is an array of joint data structures per image (step 1108). This output is passed to the scene program at step 1110.

FIG. 12A is a flowchart showing the first part of the more detailed steps of the “scene program” step 908 in FIG. 9. The scene program combines the output from most video programs in step 1202. In step 1204, the joint data structure is checked to identify a foot joint or a non-foot joint. If the joint data structure belongs to a foot joint, the homography mapping is applied to combine the joint data structure corresponding to the images from the cameras with overlapping viewing fields, at step 1206. This program identifies candidate foot joints (left and right foot joints). At step 1208, a heuristic is applied to the candidate foot joints identified in step 1206 to identify the set of candidate foot joints as the subject. In step 1210, it is checked whether the set of candidate foot joints belongs to an existing subject. If not, a new subject is generated in step 1212. Otherwise, the existing subject is updated at step 1214.

Flowchart 12B is the second part showing more detailed steps of "Scene Program" step 908. At step 1240, the non-foot joint data structure is combined from a majority array of joint data structures corresponding to images in a sequence of images from cameras with overlapping viewing fields. This is accomplished by mapping corresponding points from the first camera to the second image from the second camera with overlapping viewing domains. Some details of this procedure are described above. At step 1242, a heuristic is applied to candidate non-foot joints. In step 1246, it is determined whether the candidate non-foot joint belongs to the existing subject. If so, the existing subject is updated at step 1248. Otherwise, the candidate non-foot joint is processed again at step 1250 (after a predetermined time) to match it with the existing subject. In step 1252, it is checked whether the non-foot joint belongs to an existing subject. If so, the subject is updated at step 1256. Otherwise, the joint is discarded at step 1254.

In an exemplary embodiment, the procedures for identifying new subjects, tracking subjects, and removing subjects (which have left real space or were incorrectly generated) are implemented as performed by a runtime system (also known as an inference system) Part of the "monolithic cohesion algorithm". Monoliths are clusters of joints referred to above as the subject. The monomer cohesion algorithm recognizes the monomers in the real space and updates the positions of the joints in the real space to track the movement of the monomers.

FIG. 14 presents an illustration of a video program 1411 and a scene program 1415. In the illustrated embodiment, four video programs are shown, each processing images from one or more cameras 114. The video program processes the images described above and identifies the joints in each frame. In one embodiment, each video program recognizes the 2D coordinates, the number of confidences, the number of joints, and the unique ID for each frame and each joint. The output 1452 of all video programs is provided as input 1453 to the scene program 1415. In one embodiment, the scene program generates a joint key-value dictionary for each moment, where the key is a camera identifier and the value is an array of joints. The joints are reprojected into the perspective of a camera with overlapping viewing fields. The re-projected joints are stored as a key-value dictionary and can be used to generate a foreground subject mask for each image in each camera, as discussed below. The key in this dictionary is the combination of joint id and camera id. The values in this dictionary are the 2D coordinates of the joints that are reprojected into the viewpoint of the target camera.

The scene program 1415 produces an output 1457 which contains a list of all subjects in real space at a certain moment. The list includes a key-value dictionary for each subject. The key is the unique identifier of the subject and the value is another key-value dictionary, the key is the number of boxes, and the value is the camera-subject joint key-value dictionary. The camera-subject joint key-value dictionary is a per-subject dictionary, where the key is a camera identifier and the value is a list of joints. Image analysis to identify and track inventory items for each subject

The system and various embodiments for tracking down and taking off of the inventory item of the subject in the area of the real space are described with reference to FIGS. 15A to 25. The system and program are described with reference to FIG. 15A, which is a schematic diagram of the architecture of the system according to an embodiment. Since FIG. 15A is an architectural diagram, some details are omitted to improve the clarity of the description. Architecture of multiple CNN pipelines

FIG. 15A is a high-level architecture of a pipeline of a convolutional neural network (also known as a multi-CNN pipeline), which processes the image frames received from the camera 114 to generate a shopping cart data structure of each subject in the real space. The system described herein includes a per-camera image recognition engine as described above to identify and track multi-joint subjects. Alternative image recognition engines can be used, including examples where only one "joint" is recognized and tracked in each volume, or other features covering space and time or other types of image data are used to identify and track the processed Subject in real space.

The multiple CNN pipeline runs on each camera in parallel, moving images from the respective cameras to the image recognition engines 112a-112n, and passing through each camera's circular buffer 1502. In one embodiment, the system is composed of three subsystems: a first image processor subsystem 2602, a second image processor subsystem 2604, and a third image processor subsystem 2606. In one embodiment, the first image processor subsystem 2602 includes an image recognition engine 112a-112n, which is implemented as a convolutional neural network (CNN) and is referred to as a joint CNN 112a-112n. As described in relation to FIG. 1, the cameras 114 may be synchronized with each other in time such that their images are captured simultaneously (or temporally close) and at the same image capture rate. The images captured by all the cameras that simultaneously (or close in time) cover the area of real space are synchronized, because the synchronized images can be identified in the processing engine, such as representing a fixed in real space Different perspectives of the subject at a moment in position.

In one embodiment, the camera 114 is installed in a shopping store (such as a supermarket) so that its collection of cameras (two or more) with overlapping viewing areas is placed above each aisle to capture the real space in the store Image. There are N cameras in real space, however, for simplicity, only one camera is shown as the camera ( i ) in FIG. 17A, where the value of i ranges from 1 to N. Each camera generates a sequence of images corresponding to the real space of its respective viewing field.

In one embodiment, the image frames corresponding to the sequence of images from the cameras are transmitted to the respective image recognition engines 112a-112n at a rate of 30 frames per second (fps). Each image frame has a time stamp, camera identification (abbreviated as "camera_id"), and frame identification (abbreviated as "frame_id"), along with image data. The image frames are stored in a circular buffer 1502 (also referred to as a ring buffer) per camera 114. The circular buffer 1502 stores a collection of consecutive time stamped image frames from the respective cameras 114.

The joint CNN processes the sequence of image frames for each camera and identifies 18 different types of joints for each subject appearing in its respective viewing domain. The outputs of the joints CNN 112a-112n corresponding to cameras with overlapping viewing fields are combined to map the positions of the joints from the 2D image coordinates of each camera to the 3D coordinates of real space. The joint data structure 800 (where j is equal to 1 to x) of each subject (j) identifies the position of the joint of the subject (j) in the real space. Details of the master data structure 800 are presented in FIG. 8. In an exemplary embodiment, the joint data structure 800 is a second-order key-value dictionary of the joints of each subject. The first key is the frame_number and the value is the second key-value dictionary, the key is the camera_id and the value is a list of joints assigned to the subject.

A data set containing a subject identified by the joint data structure 800 and a corresponding image frame of a sequence of image frames from each camera is provided as a delimited frame generator 1504 input to a third image processor subsystem 2606. The third image processor subsystem further includes a foreground image recognition engine. In one embodiment, the front-end image recognition engine semantically recognizes important objects (i.e., shoppers, hands, and inventory items) in the front-end, because it is related to, for example, images from various cameras over time. Dropping and removing of passing inventory items. In the example implementation shown in FIG. 15A, the foreground image recognition engine is implemented as WhatCNN 1506 and WhenCNN 1508. The bounding box generator 1504 implements logic to process the data set to specify the bounding box of the image of the identified subject's hand in the images in its sequence of images. The bounding box generator 1504 identifies the position of the hand joint in each source image frame of each camera, using the position of the hand joint in the multi-joint data structure 800 corresponding to each source image frame. In an embodiment, where the coordinates of the joints in the main data structure indicate the positions of the joints in the 3D real space coordinates, the bounding box generator maps the positions of the joints from the 3D real space coordinates to the images of the respective source images 2D coordinates in the box.

The bounding box generator 1504 generates a bounding box for a hand joint in an image frame in a circular buffer of each camera 114. In one embodiment, the bounding box is a 128 pixel (width) x 128 pixel (height) portion of the image box, and the hand joint is located at the center of the bounding box. In other embodiments, the size of the bounding box is 64 pixels x 64 pixels or 32 pixels x 32 pixels. For m subjects from the image frame of the camera, there can be a maximum of 2m hand joints, so there are 2m bounding frames. However, actually, less than 2m hands are visible in the image frame, because it is blocked by other subjects or other objects. In an exemplary embodiment, the subject's hand position is inferred from the positions of the elbow and wrist joints. For example, the position of the subject ’s right hand is extrapolated, which uses the position of the right elbow (recognized as p1) and the right wrist (recognized as p2) as extrapolation_amount * (p2-p1) + p2, where the extrapolation_amount is 0.4. In another embodiment, the joint CNNs 112a-112n are trained using left and right hand images. Therefore, in this embodiment, the joints CNN 112a-112n directly identify the position of the hand joint in the image frame of each camera. The hand position of each image frame is used by the bounding frame generator 1504 to generate a bounding frame for each identified hand joint.

WhatCNN 1506 is a convolutional neural network that is trained to process indicated bounding boxes in an image to generate categories of hands of identified subjects. WhatCNN 1506 was trained to process image frames from a camera. In the exemplary embodiment of the shopping store, for each hand joint in each image frame, WhatCNN 1506 identifies whether the hand joint is empty. WhatCNN 1506 also identifies the number of SKUs (inventory holding units) of the inventory items in the hand joints, the confidence value indicating that the items in the hand joints are non-SKU items (that is, they are not part of the shopping store inventory), and the hand in the image frame Background of the joint position.

The outputs of the WhatCNN model 1506 of all cameras 114 are processed by a single WhenCNN model 1508 for a predetermined time window. In the example of a shopping store, WhenCNN 1508 performs a time series analysis on the two hands of the subject to identify whether the subject takes the store inventory item from the shelf or puts the store inventory item on the shelf. A shopping cart data structure 1510 (also referred to as a log data structure including a list of inventory items) is generated for each subject to keep a record of store inventory items in a shopping cart (or basket) associated with that subject.

The second image processor subsystem 2604 receives the same data set as a given input to the third image processor. The same data set includes the subject identified by the joint data structure 800 and a sequence of image frames from each camera. The corresponding image frame. Subsystem 2604 includes a front-end image recognition engine that semantically identifies important differences in the front-end (i.e., inventory display structures such as shelves) because it is related to, for example, the passage of time in images from various cameras over time Dropping and taking away of inventory items. The selection logic component (not shown in FIG. 15A) uses the confidence score to select the output from either the second image processor or the third image processor to generate a shopping cart data structure 1510.

FIG. 15B shows the coordination logic module 1522, which combines the results of most WhatCNN models and will provide input to a single WhenCNN model. As mentioned above, two or more cameras with overlapping viewing fields capture images of subjects in real space. A joint of a single subject may appear in the image frame of most cameras in the respective image channel 1520. The separated WhatCNN model identifies the SKU of the inventory item in the subject's hand (represented by the hand joint). The coordination logic module 1522 combines the output of the WhatCNN model into a single merged input of the WhenCNN model. WhenCNN model 1508 operates on the merged input to generate a shopping cart for the subject.

Detailed implementations of the system including the multiple CNN pipeline of FIG. 15A are presented in FIGS. 16, 17, and 18. In the example of a shopping store, the system tracks the drop and removal of inventory items by the subject in areas of real space. The area of the real space is a shopping store, which has inventory items placed on shelves organized in the aisle as shown in FIGS. 2 and 3. It should be understood that shelves containing inventory items can be organized in a number of different configurations. For example, the shelves may be configured in a straight line with its back side against the wall of a shopping store and its front side facing an open area in a real space. The plurality of cameras 114 having overlapping viewing fields in real space generates a sequence of images of their corresponding viewing fields. The viewing field of one camera overlaps the viewing field of at least one other camera as shown in FIGS. 2 and 3. Joint CNN-Identification and Update of Subjects

FIG. 16 is a flowchart of processing steps performed by the joints CNN 112a-112n to identify a subject in a real space. In the example of a shopping store, the subject is a consumer in a store moving in a hallway between a shelf and other open spaces. The program starts at step 1602. Note: As mentioned above, the camera is calibrated before the sequence of images from the camera is processed to identify the subject. The details of camera calibration are presented above. The camera 114 with an overlapping viewing field captures an image of the real space in which the subject appears (step 1604). In one embodiment, the camera is configured to generate a synchronized sequence of images. The sequence of images from each camera is stored in a separate circular buffer 1502 for each camera. A circular buffer (also known as a ring buffer) is a sequence of images in a sliding window that stores time. In one embodiment, the circular buffer stores 110 image frames from corresponding cameras. In another embodiment, each circular buffer 1502 stores an image frame for a time period of 3.5 seconds. It should be understood that, in other embodiments, the number of image frames (or time periods) may be larger or smaller than the exemplary values listed above.

The joints CNN 112a-112n receive a sequence of image frames from the corresponding camera 114 (step 1606). Each joint CNN processes batches of images from the corresponding cameras through most convolutional network layers to identify the joints of the subjects from the image frames of the corresponding cameras. The architecture and processing of the image with an example convolutional neural network is presented in FIG. 5. Because the camera 114 has overlapping viewing fields, the subject's joint system is identified by more than one joint CNN. The two-dimensional (2D) coordinates of the joint data structure 600 generated by the joint CNN are mapped to the three-dimensional (3D) coordinates of the real space to identify the joint position in the real space. The details of this mapping are presented in the discussion of FIG. 7, where the tracking engine 110 transforms the coordinates of elements in an array of joint data structures corresponding to the images in different image sequences into candidate joints with coordinates in real space.

The joints of the subject are organized into two classes (foot joints and non-foot joints) to group these joints into clusters, as discussed above. The left and right ankle joint types in the current example are considered foot joints for the purpose of this procedure. At step 1608, a heuristic is applied to assign candidate left foot joints and candidate right foot joints to the set of candidate joints to generate a subject. Continuing here, in step 1610, it is determined whether the newly identified subject already exists in the real space. If not, the new subject is generated in step 1614, otherwise, the existing subject is updated in step 1612.

Other joints from the candidate joint galaxy can be linked to the subject to build a cluster of some or all joint types of the resulting subject. At step 1616, heuristics are applied to non-foot joints to assign those to the identified subjects. The overall metric calculator 702 calculates an overall metric value and attempts to minimize the value by examining different combinations of non-foot joints. In one embodiment, the overall metric is the sum of heuristics organized in four categories, as described above.

The logic used to identify the set of candidate joints includes a heuristic function based on the physical relationship between the joints of the subject in the real space to identify the set of candidate joints as the subject. At step 1618, the existing master system is updated using the corresponding non-foot joints. If there are more images for processing (step 1620), steps 1606 to 1618 are repeated, otherwise the process ends at step 1622. The first data set was generated at the end of the above procedure. The first data set identifies the subject and the location of the identified subject in real space. In one embodiment, the first data set is related to FIG. 15A and is presented above as a joint data structure 800 for each subject. WhatCNN-Classification of Hand Joints

FIG. 17 is a flowchart illustrating processing steps for identifying an inventory item in the hands of an identified subject in a real space. In the example of a shopping store, the subject is a consumer in a shopping store. When consumers move in aisles and open spaces, they pick up inventory items stacked on shelves and put those items in their shopping carts or baskets. The image recognition engine identifies a subject in a collection of images in a sequence of images it receives from a plurality of cameras. The system includes logic to process a collection of images in those sequences that include images of the identified subject to detect removal of inventory items by the identified subject and drop of inventory items on the shelf by the identified subject.

In one embodiment, the logic for processing a collection of images includes (for identified subjects) logic for processing the images to generate categories of images of the identified subjects. The categories include whether the identified entity holds inventory items. The categories include a first proximity category, which indicates the position of the identified subject's hand relative to the shelf. The categories include a second proximity category, which indicates the position of the hand of the identified subject relative to the body of the identified subject. The categories further include a third proximity category, which indicates the position of the identified subject's hand relative to a basket associated with the identified subject. Finally, these categories include identifiers of possible inventory items.

In another embodiment, the logic used to process the collection of images includes (for identified subjects) logic to identify the delimitation of the data of the hand in the image in the collection of images representing the identified subjects frame. The data in the bounding box is processed to generate categories of data in the bounding box for the identified subjects. In this embodiment, the category identifies whether the identified entity holds inventory items. The categories include a first proximity category, which indicates the position of the identified subject's hand relative to the shelf. The categories include a second proximity category, which indicates the position of the hand of the identified subject relative to the body of the identified subject. The categories include a third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject. Finally, these categories include identifiers of possible inventory items.

The procedure starts at step 1702. At step 1704, the position of the subject's hand (represented by the hand joint) in the image frame is identified. The delimited frame generator 1504 recognizes the hand position of the subject from each frame of each camera, using the joint positions identified in the first data set generated by the joints CNN 112a-112n as described in FIG. Continuing here, in step 1706, the bounding box generator 1504 processes the data set to specify the bounding box of the image of the hand of the identified multi-joint subject in the images in its sequence including the images. Details of the bounding box generator are presented in the discussion of FIG. 15A above.

The second image recognition engine receives a sequence of images from the plurality of cameras and processes a designated bounding box in the images to generate a category of the hand of the identified subject (step 1708). In one embodiment, each of the image recognition engines used to classify the subjects based on the image of the hand includes a trained convolutional neural network, which is called WhatCNN 1506. WhatCNN is configured in a multiple CNN pipeline, as described above in relation to FIG. 15A. In one embodiment, the input to WhatCNNj is a multi-dimensional array BxWxHxC (also known as a BxWxHxC tensor). "B" is the batch size, which indicates the number of image frames in the batch of images processed by WhatCNN. "W" and "H" indicate the width and height of the bounding box in the pixel, and "C" is the number of channels. In one embodiment, there are 30 images in a batch (B = 30), so the size of the bounding box is 32 pixels (width) x 32 pixels (height). There can be six channels, each of which represents red, green, blue, foreground mask, forearm mask and upper arm mask. The foreground mask, forearm mask, and upper arm mask are additional and selective input data sources for WhatCNN in this example. It is the information that CNN can include in processing to classify RGB image data. The foreground mask may be generated using, for example, a mixture of Gaussian algorithms. The forearm mask can be a line between the wrist and elbow, which provides the background generated by using the information in the joint data structure. Similarly, the upper arm mask may be a line between the elbow and the shoulder, which is generated using information in the joint data structure. Different values of the B, W, H, and C parameters can be used in other embodiments. For example, in another embodiment, the size of the bounding box is larger, for example, 64 pixels (width) x 64 pixels (height) or 128 pixels (width) x 128 pixels (height).

Each WhatCNN 1506 is processing batches of images to generate categories for the hands of the identified subjects. The categories include whether the identified entity holds inventory items. The categories include one or more categories, which indicate that the positions of the hands relative to the shelf and relative to the subject cannot be detected for being dropped and removed. In this example, the first proximity category indicates the position of the identified subject's hand relative to the shelf. The categories include a second proximity category in this example, which indicates the position of the hand of the identified subject relative to the body of the identified subject, where the subject may hold inventory items during shopping. The categories further include a third proximity category in this example, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, where "basket" is used by the subject to Bags, baskets, carts, or other items held in stock during shopping. Finally, these categories include identifiers of possible inventory items. The last layer of WhatCNN 1506 generates logits, which are the predicted raw values. Logit is represented as a floating point value and further processed (as described below) for generating classification results. In one embodiment, the output of the WhatCNN model includes a multi-dimensional array BxL (also known as a BxL tensor). "B" is the batch size, and "L = N + 5" is the number of logit outputs per image frame. "N" is the number of SKUs, which represents "N" unique inventory items for sale in the shopping store.

The output "L" for each image frame is the original activation from WhatCNN 1506. Logit "L" is processed at step 1710 to identify inventory items and background. The first “N” logit represents the confidence that the entity is holding one of the “N” inventory items. Logite "L" includes an additional five (5) Logites, which are explained below. First Logit represents the confidence that the image of its item in the hands of the subject is not one of the store's SKU items (also known as non-SKU items). The second logit indicates whether the entity has confidence in the project. A large positive value indicates that the WhatCNN model has a high level of confidence that the entity is holding the item. A large negative value indicates that the model is confident that the entity does not hold any items. A near-zero value for the second logit indicates that the WhatCNN model does not have the confidence to predict whether the subject holds the item.

The next three Logitets represent the first, second, and third proximity categories, including: the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, and the second proximity category, which Indicates the position of the hand of the identified subject relative to the body of the identified subject, and a third proximity category, which indicates the position of the hand of the identified subject relative to the basket associated with the identified subject. Therefore, these three Loggetts represent a background with the location of a logit's hand, each indicating that the background of their hand is close to a shelf, close to a basket (or shopping cart), or close to the body's confidence. In one embodiment, WhatCNN is trained using a training data set containing hand images in three backgrounds: close to a shelf, close to a basket (or shopping cart), and close to the subject's body. In another embodiment, the "closeness" parameter is used by the system to classify the background of the hand. In this embodiment, the system determines the distance between the identified subject's hand and the shelf, basket (or shopping cart), and the subject's body to classify the background.

The output of WhatCNN is "L" logit, which includes: N SKU logit, 1 non-SKU logit, 1 holding logit, and 3 background logit, as described above. SKU logit (first N logit) and non-SKU logit (first logit following those N logit) are processed by the softmax function. As described above with reference to FIG. 5, the softmax function transforms an arbitrary real-valued K-dimensional vector into a real-valued K-dimensional vector in the range [0, 1] (which is increased up to 1). The softmax function calculates the probability distribution of items covering N + 1 items. The output value is between 0 and 1, and the sum of all probabilities is equal to one. The softmax function (for multi-class classification) returns the probability of each class. The category with the highest probability is the predicted category (also known as the target category).

Holding Logit is handled by S-shaped functions. The sigmoid function has real values as inputs and produces output values in the range of 0 to 1. The output of the sigmoid function identifies whether the hand is empty or holding the item. The three background logit are processed by the softmax function to identify the background of the hand joint position. In step 1712, it is checked whether there are more images to be processed. If so, steps 1704-1710 are repeated, otherwise the procedure ends at step 1714. WhenCNN to identify dropped and removed items-time series analysis

In one embodiment, the system implements logic to perform time series analysis of categories that cover subjects, to detect the removal and drop of the identified subjects based on the pre-stage image processing of the subjects. Time series analysis identifies the poses of the subjects and the inventory items associated with the poses represented in the sequence of the images.

The output of WhatCNN 1506 in the multiple CNN pipeline is provided as input to WhenCNN 1508, which processes these inputs to detect removal and drop by the identified subjects. Finally, the system includes logic (which is in response to the detected take and drop) to generate a log data structure that includes a list of inventory items for each identified entity. In the example of a shopping store, the log data structure is also referred to as the shopping cart data structure 1510 per subject.

Figure 18 presents a program that implements logic to generate a shopping cart data structure for each subject. The program starts at step 1802. The input of WhenCNN 1508 is prepared in step 1804. The inputs to the WhenCNN are a multidimensional array BxCxTxCams, where B is the batch size, the number of C video tracks, T is the number of frames considered for a time window, and Cams is the number of cameras 114. In one embodiment, the batch size "B" is 64 and the value of "T" is 110 image frames or the number of image frames in 3.5 seconds.

For each image frame and each subject identified by each camera, a list of 10 logits per hand joint (20 logits for both hands) is generated. The holding and background logit is part of the "L" logit produced by WhatCNN 1506, as described above.

The above data structure is generated for each hand in the image frame and also includes the other hand about the same subject. For example, if the data is for the left hand joint of the subject, the corresponding value for the right hand is included as "other" logit. The fifth logit (item number 3 in the above list called log_sku) is the logarithm of the SKU logit in the above "L" logit. The sixth logit is the log of the other hand SKU logit. The "roll" function generates the same information before and after the current box. For example, the seventh logit (called roll (log_sku, -30)) is the logarithm of SKU logit, which is 30 frames earlier than the current box. The eighth log is the logarithm of the SKU log, which is 30 frames later than the current frame. The ninth and tenth information in the list are similar information of the other hand, 30 frames earlier and 30 frames later than the current frame. A similar data structure for the other hand was also generated, resulting in a total of 20 logits per subject per camera per frame. Therefore, the number of channels in the input for WhenCNN is 20 (that is, C = 20 in the multi-dimensional array BxCxTxCams).

For all the image frames (for example, B = 64) in the batch of image frames from each camera, a similar data structure (identified in the image frame) of 20 hands per subject is generated. A time window (T = 3.5 seconds or 110 image frames) is used to search forward and backward for image frames in a sequence of image frames for the subject's hand joints. At step 1806, the 20 handed Loggies per subject per frame are merged from the multi-CNN pipeline. In one embodiment, the batch of image frames (64) can be imagined as the smaller window of the image frame, which is placed in the middle of the larger window of the image frame 110, with additional image frames for the two sides Search forward and backward. The input BxCxTxCams for WhenCNN 1508 is composed of 20 logits of both hands of the subjects identified in the batch "B" of the image frames from all cameras 114 (called "Cams"). The combined inputs are provided to a single trained convolutional neural network (referred to as WhenCNN model 1508).

The output of the WhenCNN model includes 3 logit, which represents confidence in the three possible actions of the identified subject: removing the inventory item from the shelf, putting the inventory item back on the shelf, and no action. The three output logits are processed by the softmax function to predict the actions performed. The three categories of Rogette are generated at regular intervals for each subject, and the results are stored for each person (with simultaneous stamping). In one embodiment, three Loggies are generated per twenty frames per subject. In this embodiment, a window of 110 image frames is formed around the current image frame at intervals of every twenty image frames per subject.

A time series analysis of these three Loggies per subject during a period of time is performed (step 1808) to identify the gesture corresponding to the real event and the time it occurred. Non-Maximum Suppression (NMS) algorithms are used for this purpose. When an event is detected multiple times (from the same camera and from most cameras) by WhenCNN 1508 (ie, by dropping and removing an item of a subject), the NMS removes the redundant event for a subject. NMS is a re-scoring technique that includes two main tasks: a "matching loss" that penalizes redundant detection and a "joint processing" of neighbors to know if there is a better detection nearby.

The real events taken and dropped for each subject are further processed by: Calculating the average of the SKU Logitech's 30 image frames before the image frame with the real event. Finally, the maximum argument (abbreviated as arg max or argmax) is used to determine the maximum. The inventory item classified by the argmax value is used to identify the inventory item from the shelf that is dropped or removed. The inventory items are added to the logarithm of the SKU (also known as a shopping cart or basket) for each entity, at step 1810. Procedure steps 1804 to 1810 are repeated, if there is more category information (checked in step 1812). During a period of time, this process results in an update of the shopping cart or basket for each subject. The process ends at step 1814. WhatCNN with scenes and video programs

FIG. 19 presents an embodiment of the system in which data from the scene program 1415 and the video program 1411 are provided as input to a WhatCNN model 1506 to generate a hand image category. Note: The output of each video program is provided to a separate WhatCNN model. The output from the scene program 1415 is a joint dictionary. In this dictionary, the key is a unique joint identifier and the value is the unique subject identifier associated with the joint. If no subject is associated with a joint, it is not included in the dictionary. Each video program 1411 receives a joint dictionary from the scene program and stores it in a ring buffer, which maps the number of frames to the returned dictionary. Using the returned key-value dictionary, the video program selects at each moment its subset of images close to the hand associated with the identified subject. These parts of the image frame around the hand joints can be referred to as zone proposals.

In the example of a shopping store, the zone proposal is an image frame from the position of the hand of one or more cameras with the subject in its respective viewing area. Zone proposals are generated by each camera in the system. It includes empty hands as well as hands that carry items in shopping store inventory and items that are not part of shopping store inventory. The video program selects the part of the image frame containing the hand joints at each moment. A similar fragment of the foreground mask is generated. The above (the image part of the hand joint and the foreground mask) are sequentially connected to the joint dictionary (indicating the subject to which each hand joint belongs) to generate a multi-dimensional array. This output from the video program is provided as input to the WhatCNN model.

The classification results of the WhatCNN model are stored in the region proposal data structure (generated by the video program). All zones for a moment are then provided as input to the scene program. The scenario program stores the result in a key-value dictionary, where the key is the subject identifier and the value is a key-value dictionary, where the key is the camera identifier and the value is the logit of the zone. This aggregate data structure is then stored in a ring buffer, which maps the number of frames to the aggregate structure at each time. WhenCNN with scenes and video programs

FIG. 20 presents an embodiment of the system in which WhenCNN 1508 receives output from a scene program, following the hand image classification performed by the WhatCNN model of each video program, as explained in FIG. 19. A zone proposal data structure for a period of time (eg, for one second) is provided as input to the scene program. In one embodiment, the camera captures images at a rate of 30 frames per second, and the input includes 30 time periods and corresponding zone proposals. The scenario program subtracts 30 zone proposals (per lot) from a single integer (which represents the inventory item SKU). The output of the scenario program is a key-value dictionary, where the key is the subject identifier and the value is a SKU integer.

WhenCNN model 1508 performs time series analysis to determine the evolution of this dictionary over time. This results in the identification of items taken from the shelves and items placed on the shelves in the shopping store. The output of the WhenCNN model is a key-value dictionary, where the key is the subject identifier and the value is the logit generated by WhenCNN. In one embodiment, a set of heuristics 2002 is used to determine the shopping cart data structure 1510 of each subject. Heuristics are applied to the output of WhenCNN, the joint positions of the subject as indicated by its respective joint data structure, and the plan view. Planograms are pre-calculated maps of items on the shelf. Heuristics 2002 determined (for each take or drop) whether the inventory item was placed on or removed from the shelf, whether the inventory item was placed in a shopping cart (or basket) ) Remove, or whether the inventory item is close to the body of the identified subject. What-CNN model example architecture

Figure 21 presents an example architecture of a WhatCNN model 1506. In this example architecture, there are a total of 26 convolutional layers. The dimensions of different layers with respect to the respective width (pixel), height (pixel), and number of channels are also proposed. The first convolution layer 2113 receives the input 2111 and has a width of 64 pixels, a height of 64 pixels, and has 64 channels (written as 64x64x64). The details for the input of WhatCNN are presented above. The direction of the arrow indicates the flow of data from one layer to the next. The second convolution layer 2115 has a dimension of 32x32x64. Continuing from the second layer, there are eight convolutional layers (shown in a square box 2117), each with a dimension of 32x32x64. Only two layers 2119 and 2121 are shown in the square box 2117 for the purpose of clarification. This series is followed by another eight convolution layers 2123 with a dimension of 16x16x128. Two such convolutional layers 2125 and 2127 are shown in FIG. 21. Finally, the last eight convolutional layers 2129 have dimensions of 8x8x256 each. Two convolution layers 2131 and 2133 are shown in a square box 2129 to facilitate clarification.

There is a fully connected layer 2135 with 256 inputs from the last convolutional layer 2133, which produces N + 5 outputs. As mentioned above, "N" is the number of SKUs, which represents "N" unique inventory items for sale in a shopping store. The five additional logits include: the first logit, which indicates that the item in the image is a non-SKU item, and the second logit, which indicates whether the subject has confidence in the item. The next three Rockets represent the first, second, and third proximity categories, as described above. WhatCNN's final output is shown at 2137. The example architecture uses batch normalization (BN). The distribution of the layers in a convolutional neural network (CNN) changes during training and it changes with each layer. This reduces the convergence speed of the optimization algorithm. Batch normalization (Ioffe and Szegedy 2015) is a technique to overcome this problem. ReLU (corrected linear unit) enables non-linearities that are used for each layer, except for the final output where softmax is used.

Figures 22, 23, and 24 are graphical visualizations of different parts of an implementation of WhatCNN 1506. The graphics are adapted from the graphical visualization of the WhatCNN model produced by TensorBoard ™. TensorBoard ™ is a suite of visualization tools for viewing and understanding deep learning models such as convolutional neural networks.

Figure 22 shows a high-level architecture of a convolutional neural network model that detects one-handed ("one-handed" model 2210). WhatCNN model 1506 contains two of these convolutional neural networks to detect left and right hands separately. In the embodiment shown, the architecture includes four blocks, called block 0 2216, block 1 2218, block 2 2220, and block 3 2222. A block is a higher-level abstraction and contains most nodes that it represents a convolutional layer. The blocks are arranged in a sequence from lower to higher so that their output from one block is input to subsequent blocks. The architecture also includes a collection layer 2214 and a convolution layer 2212. Between these blocks, different non-linearities can be used. In the illustrated embodiment, ReLU nonlinearity is used as described above.

In the illustrated embodiment, the input to the one-handed model 2210 is a BxWxHxC tensor, which is defined as described above in WhatCNN 1506. "B" is the batch size, "W" and "H" indicate the width and height of the input image, and "C" is the number of channels. The output of the one-handed model 2210 is combined with the second one-handed model and passed to the fully connected network.

During training, the output of the one-handed model 2210 is compared with the ground truth. The prediction error calculated between the output and the ground truth is used to update the weighting of the convolutional layer. In the illustrated embodiment, Stochastic Gradient Descent (SGD) is used to train WhatCNN 1506.

FIG. 23 presents further details of block 0 2216 of the one-handed convolutional neural network model of FIG. 22. It contains four convolutional layers, labeled conv0, conv1 2318, conv2 2320, and conv3 2322 in the square box 2310. Further details of the convolution layer conv0 are proposed in the square box 2310. This input is processed by the convolutional layer 2312. The output of the convolutional layer is processed by the batch normalization layer 2314. ReLU nonlinearity 2316 is applied to the output of the batch normalization layer 2314. The output of the convolution layer conv0 is passed to the next layer conv1 2318. The output of the final convolution layer conv3 is processed through the addition of 2324. This operation sums the output from layer conv3 2322 to its unmodified input that has undergone a jump connection 2326. It has been published in a paper by He et al. Entitled "Identification Mapping in Deep Residual Networks" (published at https://arxiv.org/pdf/1603.05027.pdf, July 25, 2016). Backward signals can be propagated directly from one block to any other block. This signal propagates unchanged through the convolutional neural network. This technique improves the training and testing performance of deep convolutional neural networks.

As described in Figure 21, the output of the convolutional layer of WhatCNN is processed by a fully connected layer. The outputs of the two one-handed models 2210 are combined and passed as input to the fully connected layer. FIG. 24 is an example implementation of a fully connected layer (FC) 2410. The input to the FC layer is processed by a reforming operator 2412. The reshaping operator changes the shape of the tensor before passing it to the next layer 2420. Reshaping includes flattening the output from the convolutional layer, that is, reshaping the output from the multi-dimensional matrix to a one-dimensional matrix or vector. The output of the reforming operator 2412 is passed to a matrix multiplication operator, which is labeled as MatMul 2422. The output from MatMul 2422 is passed to the matrix positive addition operator, which is labeled xw_plus_b 2424. For each input "x", the operator 2424 multiplies the input by the matrix "w" and the vector "b" to produce the output. "W" is a trainable parameter associated with the input "x", and "b" is another trainable parameter called offset or intercept. The output 2426 from the fully connected layer 2410 is a BxL tensor, as explained above in the description of WhatCNN 1506. "B" is the batch size, and "L = N + 5" is the number of logit outputs per image frame. "N" is the number of SKUs, which represents "N" unique inventory items for sale in the shopping store. WhatCNN model training

Training data sets of hands holding different inventory items in different backgrounds and images of empty hands in different backgrounds are generated. To achieve this, human actors hold various unique SKU inventory items in many different ways at different locations in the test environment. The scope of his background includes: close to the actor's body, close to the shelves of the store, and close to the actor's shopping cart or basket. The actor also performed the above action empty-handed. This procedure is done for both left and right hands. Most actors perform these actions simultaneously in the same test environment to simulate their natural blocking in a real shopping store.

The camera 114 takes an image of an actor performing the above-mentioned actions. In one embodiment, twenty cameras are used in this program. The joint CNNs 112a-112n and the tracking engine 110 process the images to identify the joints. The bounding box generator 1504 generates a bounding box similar to a hand region of production or inference. Instead of classifying these hand regions via WhatCNN 1506, the images are saved to a storage disk. Saved images are viewed and marked. The image is assigned three tags: the inventory item SKU, the background, and whether the hand holds something. This procedure is performed for a large number of images (up to millions of images).

The image files are organized according to the data collection scene. The naming convention for image files is to identify the content and background of the images. FIG. 25 shows an image file name in an exemplary embodiment. The first part of the filename (referred to by the number 2502) identifies the scene of the data collection and also includes the time stamp of the image. The second part of the file name, 2504, identifies the source camera. In the example shown in Figure 25, the image is captured by "Camera 4". The third part of the file name 2506 identifies the number of frames from the source camera. In the example shown, the filename indicates that it is the 94,600 image frame from camera 4. The fourth part of the file name 2508 is to identify the range of the x and y coordinate areas in the source image frame (this hand area image is obtained from the source image frame). In the example shown, the region is defined between the x-coordinate values from pixels 117 to 370 and the y-coordinate values from pixels 370 to 498. The fifth part of the file name 2510 identifies the personal id of the actor in the scene. In the example shown, the individual in the scene has the id "3". Finally, the sixth part 2512 of the file name identifies the SKU number of the inventory item (item = 68), which is identified in the image.

In the training mode of WhatCNN 1506, forward pass and backward pass are performed as opposed to generation mode, of which only forward pass is performed. During training, WhatCNN generates classes of the identified subjects' hands in forward pass. WhatCNN's output is compared to ground truth. In backward propagation, the gradient of one or more cost functions is calculated. The gradient is then propagated to the convolutional neural network (CNN) and the fully connected (FC) neural network so that its prediction error is reduced, causing the output to be closer to the ground truth. In one embodiment, Stochastic Gradient Descent (SGD) is used to train WhatCNN 1506.

In one embodiment, 64 images are randomly selected from the training data and amplified. The purpose of image augmentation is to diversify the training data, which leads to better performance of the model. Image augmentation includes random flip, random rotation, random hue shift, random Gaussian noise, random contrast change, and random trimming. The amount of amplification is a hyperparameter and is tuned by a hyperparameter search. The amplified images were classified by WhatCNN 1506 during training. This classification is compared to ground truth, and the coefficients or weights of WhatCNN 1506 are updated by calculating the gradient loss function and multiplying the gradient by the learning rate. The above procedure is repeated multiple times (for example, about 1000 times) to form a period. Fulfilled between 50 and 200 periods. During each period, the learning rate is slightly reduced, following the cosine annealing schedule. WhenCNN model training

WhenCNN 1508 is trained similar to WhatCNN 1506 described above, using backward propagation to reduce prediction errors. Actors perform a variety of actions in a training environment. In the exemplary embodiment, the training is performed in a shopping store with inventory items stacked on its shelves. Examples of actions performed by actors include: removing inventory items from shelves, putting inventory items back on shelves, putting inventory items in shopping carts (or baskets), retrieving inventory items from shopping carts, left and right hands Switch items between, and put inventory items into the actor's hideout. Hidden place refers to the position on the body of an actor who claims to have inventory items beside his left and right hands. Some examples of hidden places include: an inventory item that is squeezed between the forearm and upper arm, between the forearm and chest, and between the neck and shoulders.

The camera 114 is a video recording all the actions described above during the training. The videos are viewed and all video frames are marked to indicate time stamps and actions performed. These tags are called action tags for individual video frames. These image frames are processed through the multiple CNN pipelines to WhatCNN 1506 as described above for generation or inference. The output of WhatCNN (along with the associated action labels) is then used to train WhenCNN 1508 to use these action labels as ground truth. Stochastic gradient descent (SGD) with cosine annealing schedule is used for training as described above for training of WhatCNN 1506.

In addition to image augmentation (for WhatCNN training), time augmentation is also applied to image frames during the training of WhenCNN. Some examples include mirroring, adding Gaussian noise, swapping left and right-handed logits, shortening time, shortening time series by discarding image frames, extending time series by copying frames, and discarding time series. Data points are used to simulate defects in the base model, which generates inputs for WhenCNN. Mirroring includes reversing the time series and individual labels, for example, when the action is reversed, the action is changed to the take action. Use background image processing to predict inventory events

A system and various embodiments for tracking changes by a subject in an area of real space are described with reference to FIGS. 26 to 28B. system structure

FIG. 26 presents a high-level overview of a system according to an embodiment. Because FIG. 26 is an architectural diagram, some details are omitted to improve the clarity of the description.

The system proposed in FIG. 26 receives image frames from a plurality of cameras 114. As described above, in one embodiment, the cameras 114 may be synchronized with each other in time, so that their images are captured simultaneously (or close in time) and at the same image capture rate. The images captured by all the cameras that simultaneously (or close in time) cover the area of real space are synchronized, because the synchronized images can be identified in the processing engine, such as representing a fixed in real space Different perspectives of the subject at a moment in position.

In one embodiment, the camera 114 is installed in a shopping store (such as a supermarket) so that its collection of cameras (two or more) with overlapping viewing areas is placed above each aisle to capture the real space in the store Image. There are "n" cameras in real space. Each camera produces a sequence of images corresponding to the real space of its respective viewing field.

The subject recognition subsystem 2602 (also referred to as the first image processor) processes image frames received from the camera 114 to identify and track subjects in real space. The first image processor includes a subject image recognition engine. The subject image recognition engine receives a corresponding sequence of images from a plurality of cameras, and processes the images to identify a subject represented by the images in the corresponding sequence of images. In one embodiment, the system includes a per-camera image recognition engine as described above for identifying and tracking multi-joint subjects. Alternative image recognition engines can be used, including examples where only one "joint" is recognized and tracked in each volume, or other features covering space and time or other types of image data are used to identify and track the processed Subject in real space.

The "semantic difference" subsystem 2604 (also known as the second image processor) includes a background image recognition engine that receives corresponding sequences of images from the plurality of cameras and semantically identifies the background (ie, inventory displays such as shelves Structure), as it relates to, for example, the dropping and removal of inventory items over time in images from various cameras. The second image processor receives the output of the subject recognition subsystem 2602 and the image frame from the camera 114 as inputs. The second image processor masks the identified subjects in the foreground to generate a masked image. The occluded images are generated by replacing the bounding box corresponding to the foreground subject with background image data. Continuing here, the background image recognition engines process the occluded images to identify and classify the background changes represented in the images in the corresponding sequences of the images. In one embodiment, the background image recognition engines include a convolutional neural network.

Finally, the second image processor processes the first set of detected background changes for detection by the identified subject to remove the inventory item and detection by the identified subject to drop the inventory item on the inventory display structure. The first set of inspections is also referred to as background inspection of the drop and take away of inventory items. In the example of a shopping store, the first detection is to identify inventory items that have been removed from or placed on a shelf by a consumer or store employee. The semantic difference subsystem includes logic to associate an identified background change with an identified subject.

The area proposal subsystem 2606 (also known as the third image processor) includes a front-end image recognition engine that receives corresponding sequences of images from the plurality of cameras 114 and semantically identifies the front-end (i.e., shoppers, And inventory items) because they are related to, for example, dropping and removing inventory items over time in images from various cameras. Subsystem 2606 also receives output from the subject identification subsystem 2602. The third image processors process a sequence of images from the camera 114 to identify and classify anterior stage changes represented in the images in the corresponding sequences of the images. The third image processor is the second set of processing the identified foreground changes to detect the inventory items taken by the identified subject and the detection of the inventory items dropped by the identified subject on the inventory display structure. The second set of inspections is also referred to as front-end inspection of drop and take-off of inventory items. In the example of a shopping store, the second set of detections identifies the removal of inventory items by consumers and store employees and placing the inventory items on the inventory display structure.

The system described in FIG. 26 includes a selection logic component 2608 for processing the first and second sets of inspections to generate a log data structure including a list of inventory items for identified entities. For removal or drop in real space, the selection logic 2608 selects the output from either the semantic difference subsystem 2604 or the zone proposal subsystem 2606. In one embodiment, the selection logic 2608 uses the confidence score generated by the semantic difference subsystem for the first set of detections and the confidence score generated by the district proposal subsystem for the second set of detections. The output of the subsystem with a higher confidence score for a particular test is selected and used to generate a log data structure 1510 (also known as a shopping cart data structure) that includes a list of inventory items associated with the identified front-end subjects. Subsystem components

Figure 27 presents a subsystem component that implements the system to track changes to subjects in an area through real space. The system includes a plurality of cameras 114 that generate respective sequences of images of corresponding viewing fields in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera among the plurality of cameras as described above. In an embodiment, a sequence of image frames corresponding to the images generated by the plurality of cameras 114 is stored in a circular buffer 1502 (also referred to as a ring buffer) of each camera 114. Each image frame has a time stamp, camera identification (abbreviated as "camera_id"), and frame identification (abbreviated as "frame_id"), along with image data. The circular buffer 1502 stores a collection of consecutive time stamped image frames from the respective cameras 114. In one embodiment, the camera 114 is configured to generate a synchronized sequence of images.

The same camera and the same sequence of images are used by both the front stage and the background image processor in a preferred embodiment. As a result, the redundant detection of put-down and take-out of inventory items is performed using the same input data to allow high confidence (and high accuracy) in the obtained data.

The subject recognition subsystem 2602 (also referred to as the first image processor) includes a subject image recognition engine that receives a corresponding sequence of images from the plurality of cameras 114. The subject image recognition engine processes the images to identify subjects represented in the images in the corresponding sequences of the images. In one embodiment, the subject image recognition engine is implemented as a Convolutional Neural Network (CNN), which is referred to as a joint CNN 112a-112n. The outputs of the joints CNN 112a-112n corresponding to cameras with overlapping viewing fields are combined to map the positions of the joints from the 2D image coordinates of each camera to the 3D coordinates of real space. The joint data structure 800 (where j is equal to 1 to x) of each subject (j) identifies the positions of the joints of the subject (j) in the real space and in the 2D space of each image. Some details of the master data structure 800 are presented in FIG. 8.

The background image storage 2704 in the semantic difference subsystem 2604 stores occluded images (also referred to as background images, where the foreground subject has been removed by occlusion) for a corresponding sequence of images from the camera 114. The background image storage 2704 is also called a background buffer. In one embodiment, the size of the masked image is the same as the size of the image frame in the circular buffer 1502. In one embodiment, the occluded image is stored in a background image storage 2704, which is an image frame in a sequence corresponding to the image frame of each camera.

The semantic difference subsystem 2604 (or the second image processor) includes a mask generator 2724 that generates a mask of the foreground subject represented by the images in the corresponding sequence of images from the camera. In one embodiment, a mask generator processes a sequence of images from each camera. In the example of a shopping store, the front-end subject is a consumer or store employee in front of a background shelf containing items for sale.

In one embodiment, the joint data structure 800 and the image frame from the circular buffer 1502 are provided as inputs to the mask generator 2724. The joint data structure identifies the position of the front stage main body in each image frame. The mask generator 2724 generates a bounding frame for each foreground subject identified in the image frame. In this embodiment, the mask generator 2724 uses the x and y coordinates of the joint positions in the 2D image frame to determine the four boundaries of the bounding frame. The minimum value of x (from all x values for a subject's joints) is the left vertical boundary that defines the bounding box of the subject. The minimum value of y (from all y values for joints of a subject) defines the bottom vertical boundary of the bounding box. Similarly, the maximum values of the x and y coordinates identify the right vertical and top horizontal boundaries of the bounding box. In the second embodiment, the mask generator 2724 uses a convolutional neural network-based person detection and localization algorithm to generate the bounding box of the foreground subject. In this embodiment, the mask generator 2724 does not use the joint data structure 800 to generate the bounding box of the foreground subject.

The semantic difference subsystem 2604 (or the second image processor) includes masking logic to process the images in the sequences of the image and replace the background image data representing the background images from the corresponding sequences of the image with the background image data The subject's front stage image data has been identified to provide a masked image, which results in a new background image for processing. When the circular buffer receives the image frame from the camera 114, the mask logic processes the images in those sequences of the image to replace the foreground image data defined by the image mask with the background image data. The background image data is taken from the background images of the corresponding sequences of the image to generate the corresponding masked images.

Consider the example of a shopping store. Initially, at time t = 0, when there are no consumers in the store, the background image in the background image storage 2704 is the same as its corresponding image frame in the sequences of each camera image. Now consider time t = 1, the consumer moves in front of the shelf to purchase items in the shelf. The mask generator 2724 generates the bounding box of the consumer and transmits it to the mask logic component 2702. The mask logic component 2702 replaces the pixels in the bounding frame with the corresponding pixels in the image frame at t = 1 by corresponding pixels in the background image frame at t = 0. This results in a corresponding line in the circular buffer 1502 in the image frame t = 1 t = 1 of the image has been masked. The occluded image does not include pixels for the foreground subject (or consumer), and is now replaced by pixels from the background image frame at t = 0. The masked image at t = 1 is stored in the background image storage 2704 and acts as the background image of the image frames below t = 2 in those sequences of images from the corresponding camera.

In one embodiment, the mask logic component 2702 combines (such as by averaging or summing by pixels) a plurality of sets of N masked images in the sequences of images to generate a sequence of factorized images for each camera . In this embodiment, the second image processors process the factorized image sequence to identify and classify background changes. The factorized image can be generated, for example, by taking the average of the pixels in the N masked images in the sequence of masked images per camera. In one embodiment, the value of N is equal to the frame rate of the camera 114. For example, if the frame rate is 30 FPS (frame per second), the value of N is 30. In this embodiment, the masked images for a time period of one second are combined to generate a factorized image. Obtaining average pixel values is to minimize pixel fluctuations due to sensor noise and luminosity changes in areas of real space.

The second image processors process the sequence of factorized images to identify and classify background changes. The factorized images in the sequences of factorized images are compared with the previous factorized images of the same camera by the bit mask calculator 2710. The pair of factorized images 2706 is provided as input to a bitmask calculator 2710 to generate a bitmask, which identifies changes in the corresponding pixels of the two factorized images. The bit mask has a pixel position of 1, where the difference between the RGB (red, green, and blue channel) values of the corresponding pixels (current and previous factorized images) is greater than the "difference threshold." The difference threshold value is adjustable. In one embodiment, the difference threshold is set at 0.1.

The bit mask and the pair of factorized images (current and previous) from the sequence of factorized images from each camera are provided as inputs to a background image recognition engine. In one embodiment, the background image recognition engine includes a convolutional neural network and is referred to as a change CNN 2714a-2714n. A single change CNN is a sequence of factorized images per camera. In another embodiment, the masked images from the corresponding sequence of images are not combined. The bit mask is calculated from the pairs that have masked the image. In this embodiment, the pairs of masked images and the bit mask are then provided as inputs to the changed CNN.

In this example, the input for changing the CNN model is composed of seven (7) channels, including three image channels (red, green, and blue) per factorized image and one channel for the bit mask. The modified CNN includes a majority of convolutional layers and one or more fully connected (FC) layers. In one embodiment, the modified CNN includes the same number of convolutions and FC layers as the joint CNNs 112a-112n shown in FIG. 5.

The background image recognition engine (change CNN 2714a-2714n) recognizes and classifies changes in the factorized images and generates a changed data structure for the corresponding sequences of the images. The changed data structure includes the coordinates in the occluded image of the identified background change, the identifier of the inventory item subject of the identified background change, and the category of the identified background change. The categories of the identified background changes in the changed data structure classify whether the identified inventory item has been added or removed relative to the background image.

Because the item can be removed or placed on the shelf simultaneously by one or more subjects, the changing CNN results in the number "B" which is predicted by its bounding box overlapping each output position. The bounding box prediction corresponds to the change in the factorized image. Consider that the shopping store has a unique "C" inventory item, each identified by a unique SKU. The change CNN predicts the SKU of the subject of the changed inventory item. Finally, the change CNN identifies a change (or type of inventory event) for each location (pixel) in the output, which indicates whether the identified item was removed from the shelf or dropped on the shelf. The above three pairs of outputs from changing the CNN are described by the formula "5 * B + C + 1". Each bounding box "B" prediction system contains five (5) numbers, so "B" is multiplied by 5. These five numbers are the "x" and "y" coordinates representing the center of the bounding box, the width and height of the bounding box. The fifth number represents the confidence score of the changed CNN model for the prediction of the bounding box. "B" is a hyperparameter that can be adjusted to improve the performance of changing the CNN model. In one embodiment, the value of "B" is equal to four. The consideration comes from changing the width and height (in pixels) of the output of the CNN, which are respectively represented by W and H. The output of the changed CNN is then expressed as "W * H * (5 * B + C + 1)". The bounding box output model is based on the object detection system proposed by Redmon and Farhadi in their paper "YOLO9000: Better, Faster, Stronger" (published on December 25, 2016). The paper is available at https://arxiv.org/pdf/1612.08242.pdf.

The output of the CNN 2714a-2714n corresponding to changes in the sequence of images from cameras with overlapping viewing fields is combined by the coordination logic component 2718. The coordination logic component processes the change data structure from multiple sets of cameras with overlapping viewing fields to find the identified background changes in real space. The coordination logic component 2718 selects a bounding box, which represents an inventory item with the same SKU and the same inventory event type (taken or dropped), from most cameras with overlapping viewing fields. The selected bounding box is then triangulated in the 3D real space (using the triangulation technique described above) to identify the location of the inventory item in the 3D real space. The position of the shelf in the real space is compared with the triangulated position of the inventory item in the 3D real space. False positive predictions are discarded. For example, if the triangulated position of the bounding box does not map to the position of a shelf in real space, the output is discarded. Its triangulated position mapped to the bounding box in the 3D real space of the shelf is considered as a true prediction of the inventory event.

In an embodiment, the categories of the identified background changes in the changed data structure generated by the second image processor classify whether the identified inventory item has been added or removed relative to the background image . In another embodiment, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image, and the system includes The context changes the logic associated with the identified subject. The system performs the detection of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure.

The log generator 2720 implements logic to associate a change identified by a true prediction of the change with an identified subject near the location of the change. In an embodiment using a joint recognition engine to identify the subject, the log generator 2720 uses the joint data structure 800 to determine the position of the hand's hand joint in the 3D real space. A subject is identified whose hand joints fall within a threshold distance from the changed position at the moment of change. The log generator associates the change with the identified subject.

In one embodiment, as described above, the N occluded images are combined to generate a factorized image, which is provided as input to the changed CNN. Consideration: N is equal to the frame rate (frames per second) of the camera 114. Therefore, in this embodiment, the position of the subject's hand during a second time period is compared to the changed position to associate the changes with the identified subject. If more than one subject's hand joint position falls within a threshold distance from the changed position, the association of the change with the subject is postponed to the output of the foreground image processing subsystem 2606.

The foreground image processing (area proposal) subsystem 2606 (also known as the third image processor) includes a foreground image recognition engine that receives the sequences of images from the plurality of cameras. The third image processors include logic to identify and classify the previous stage changes represented in the images in the corresponding sequences of images. The area proposal subsystem 2606 generates a second set of detections by which the identified entities take inventory items away and detections by which the identified entities drop inventory items on the inventory display structure. As shown in FIG. 27, the subsystem 2606 includes a bounding box generator 1504, WhatCNN 1506, and WhenCNN 1508. The joint data structure 800 and the image frames from each camera of the circular buffer 1502 are provided as inputs to the bounding frame generator 1504. Details of the bounding box generator 1504, WhatCNN 1506, and WhenCNN 1508 were proposed earlier.

The system described in FIG. 27 includes selection logic to process the first and second sets of inspections to produce a log data structure that includes a list of inventory items for identified entities. The first set of detections of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure are generated by the log generator 2720. The first set detected is determined using the output of the second image processor and the joint data structure 800 (as described above). The second set of detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the inventory display structure are determined using the output of the third image processor. For each real inventory event (remove or drop), the selection logic controller 2608 selects one from either the second image processor (semantic difference subsystem 2604) or the third image processor (region proposal subsystem 2606). Output. In one embodiment, the selection logic selects an output from an image processor with a higher confidence score for the inventory event. Program flow of semantic differences in background images

Figures 28A and 28B present detailed steps performed by the semantic difference subsystem 2604 to track changes to subjects in an area through real space. In the example of a shopping store, the subjects are consumers and store employees in the store moving in the aisles between the shelves and other open spaces. The program starts at step 2802. As described above, the camera 114 is calibrated before the sequence of images from the camera is processed to identify the subject. The details of camera calibration are presented above. The camera 114 with overlapping viewing fields captures images of real space in which the subject appears. In one embodiment, the camera is configured to generate a synchronized sequence of images at a rate of N frames per second. The sequence of images from each camera is stored in a separate circular buffer 1502 for each camera, at step 2804. A circular buffer (also known as a ring buffer) is a sequence of images in a sliding window that stores time. The background image storage 2704 is initialized with the initial image frame in the sequence of the image frame of each camera without the foreground subject (step 2806).

When the subject moves in front of the shelf, the bounding frame of each subject is generated using its corresponding joint data structure 800 (as described above) (step 2808). At step 2810, the occluded image is generated by replacing pixels in the bounding box of each image frame with pixels from the same location from the background image from the background image store 2704. The occluded image corresponding to each of the images in the sequences of each camera image is stored in the background image storage 2704. The i-th occluded image is used as a background image to replace pixels in subsequent (i + 1) image frames in the sequence of the image frame of each camera.

At step 2812, the N masked images are combined to generate a factorized image. At step 2814, the difference thermal map is generated by comparing the pixel values of the multiple logarithmic image. In one embodiment, the difference between the pixels at the position (x, y) in the 2D space of the two factorized images ( fi 1 and fi 2) is calculated in Equation 1 as shown below:

Differences between pixels at the same x and y positions in 2D space are determined using the respective intensity values of the red, green, and blue (RGB) channels (as shown in the equation). The above equation provides a numerical value (also known as Euclidean modulus) of the difference between corresponding pixels in two factorized images.

The differential thermal map may contain noise due to changes in sensor noise and luminosity in areas of real space. In FIG. 28B, in step 2816, a bit mask is generated to the difference heat map. Meaningful changes are identified by a cluster of 1 (one) in the bit mask. These clusters correspond to changes that identify inventory items that have been removed from or placed on a shelf. However, noise in the difference heat map can introduce random ones into the bit mask. In addition, most changes (taken from the shelf or most items placed on the shelf) can introduce an overlapping cluster of 1. In the next step (2818) of the program flow, the image shape operation is applied to the bit mask. Image morphology operations remove noise (unwanted ones) and also try to separate overlapping clusters of ones. This results in a cleaner bit mask that contains a cluster corresponding to one of the semantically meaningful changes.

Two inputs are provided to the morphological operation. The first input is the bit mask and the second input is called a structural element or kernel. The two basic form operations are "erosion" and "swell". The kernel consists of 1s arranged in a rectangular matrix of various sizes. Kernels of different shapes (eg, circular, elliptical, or cross-shaped) are generated by adding zeros at specific positions in the matrix. Differently shaped kernels are used for image morphology operations to achieve the desired results when cleaning bit masks. During the erosion operation, the kernel slides (or moves) over the bit mask. Pixels (either 1 or 0) in the bit mask are considered as 1 if all pixels below the kernel are 1. Otherwise, it is eroded (changed to 0). Erosion can be used to remove isolated ones in the bit mask. However, erosion also reduces these clusters of 1 by erosion edges.

The expansion operation is the opposite of erosion. In this operation, when the kernel slides over the bit mask, the values of all pixels in the bit mask area overlapped by the kernel are changed to 1, if at least one pixel below the kernel If the value is 1. Dilation is applied to the bit mask after erosion to increase the size cluster by one. Because noise is removed during erosion, dilation does not introduce random noise to the bit mask. A combination of erosion and expansion operations is applied to obtain a cleaner bit mask. For example, a subsequent line of computer code applies a 3x3 filter of 1 to the bit mask to perform an "open" operation, which applies an erosion operation followed by an expansion operation to remove noise and restore the bit mask. The size of the cluster of 1 is as described above. The above computer code uses the OpenCV (Open Source Computer Vision) library for programming functions of real-time computer vision applications. The library is available at

The "close" operation applies an expansion operation followed by an erosion operation. It can be used to close small holes inside the 1 cluster. The following code uses a 30x30 cross-shaped filter to apply a close operation to that bit mask.

This bit mask and two factorized images (before and after) are provided as input to a convolutional neural network per camera (referred to as the modified CNN as above). Change the output of CNN to change the data structure. At step 2822, the output from the altered CNN with overlapping viewing fields is combined using the triangulation technique described earlier. The changed position in the 3D real space matches the position of the shelf. If the location of the inventory event is mapped to a location on the shelf, the change is considered a real event (step 2824). Otherwise, the change is false positive and discarded. The real event is related to the foreground subject. At step 2826, the foreground subject is identified. In one embodiment, the joint data structure 800 is used to determine the position of the hand joint within a threshold distance of the change. If the foreground subject is identified at step 2828, the change is associated with the identified subject, at step 2830. If no foreground subject is identified in step 2828, for example, the hand joint positions of most subjects within a range of the threshold of the change. Redundancy detection of the change by the zone proposal subsystem is then selected, at step 2832. The process ends at step 2834. Training changes CNN

Seven channel input training data sets are generated to train the changing CNN. Act as one or more consumers to fulfill the take and drop action by pretending to shop in a shopping store. The main system moves in the aisle, takes inventory items from the shelves and puts the items back on the shelves. Images of actors performing take-off and drop-down operations are collected in a circular buffer 1502. These images are processed to produce a factorized image, as described above. The multi-pair factorized image 2706 and the corresponding bit mask output by the bit mask calculator 2710 are manually inspected to visually identify changes between the two factorized images. For factorized images with changes, bounding boxes are manually drawn around the change. This is the smallest bounding box containing a cluster of 1s corresponding to the change in the bit mask. The SKU number of the changed inventory item is identified and included in the label for the image (along with the bounding box). The type of event identifying the removal or drop of inventory items is also included in the label of the bounding box. The label of each bounding box therefore identifies (its position on the factorized image) the item's SKU and the event type. A factorized image may have more than one bounding box. The above procedure is repeated for every change in all the collected factorized images of the training data set. A pair of factorized images (together with the bit mask) form seven channel inputs for the changed CNN.

During the training of the altered CNN, forward pass and backward pass are performed. In the forward pass, the change CNN is to identify and classify the background changes, which are represented in the factorized images in the corresponding sequences of the images in the training data set. The change CNN is the first set of processing the identified background change to perform the detection of the inventory item taken by the identified subject and the detection of the inventory item dropped by the identified subject on the inventory display structure. During backward propagation, the output of the altered CNN is compared to the ground truth (as indicated in the label of the training data set). The gradient of one or more cost functions is calculated. The gradient is then propagated to the convolutional neural network (CNN) and the fully connected (FC) neural network so that its prediction error is reduced, causing the output to be closer to the ground truth. In one embodiment, a softmax function and a cross-entropy loss function are used to train a CNN that changes the class prediction portion of the output. The category prediction portion of the output includes the SKU identifier for the inventory item and the event type (ie, removed or dropped).

The second loss function is used to train the predicted CNN against the bounded box. This loss function calculates the intersection over union (IOU) between the prediction frame and the ground truth frame. The area of the intersection of the bounding boxes predicted by the changing CNN with the true bounding box label is divided by the area of the associated set of the same bounding box. If the overlap between the prediction frame and the ground truth frame is large, the value of IOU is high. If more than one predicted bounding box overlaps the ground truth bounding box, one with the highest IOU value is selected to calculate the loss function. The details of the loss function were proposed by Redmon et al. In their paper "You Only Look Once: Unified, Real-Time Object Detection" and published on May 9, 2016. The paper is available at https://arxiv.org/pdf/1506.02640.pdf. Specific implementation

In various embodiments, the system (as described above) for tracking down and taking off the inventory items of the subject in the area of real space also includes one or more of the following features. District proposal

The area is proposed as a frame image from the hand positions of all the different cameras covering the person. Zone proposals are generated by each camera in the system. It includes empty hands as well as hands carrying store items. 1.1 WhatCNN model

District proposals can be used as input for image classification using deep learning algorithms. This classification engine is called the "WhatCNN" model. It is a classification model in hand. It is the classification of things. Even if part of the object is blocked by the hand, the image classification in the hand is still operable. Smaller items can be blocked up to 90% by hand. The area of image analysis by the WhatCNN model is intentionally kept small (in some embodiments) because it is computationally expensive. Each camera can have a dedicated GPU. This is performed for each frame with each hand image from each camera. In addition to the above-mentioned image analysis by the WhatCNN model, confidence weighting is also assigned to the image (one camera, one point in time). The classification algorithm outputs a logit that includes a complete list of inventory keeping units (SKUs) to generate a list of product and service identifiers for the store for n items and an extra for empty hands (n + 1).

The scene program now returns its results to each video program by transmitting a key-value dictionary to each video. Here, the key is a unique joint ID and the value is a unique personal ID associated with the joint. If no one is associated with the joint, it is not included in the dictionary.

Each video program receives a key-value dictionary from the scene program and stores it into a ring buffer, which maps the number of frames to the returned dictionary.

Using the returned key-value dictionary, the video selects a subset of its images near the hands associated with known people at each moment. These areas are numpy fragments. We also obtained similar clips around the foreground mask and the original feature array of the joint CNN. These combined regions are connected together in order to form a single multi-dimensional numpy array and stored in a data structure, the data structure is kept: the numpy array and the personal ID associated with the region, and the floristic from the individual Which hand.

All proposed areas are then fed into the FIFO queue. This queue accepts regions and pushes its numpy array into memory on the GPU.

When the array reaches the GPU, it is fed into a CNN dedicated to classification, called WhatCNN. The output of this CNN is a flat array of floating points of size N + 1, where N is the number of unique SKUs in the store, and the last category represents no category (or empty hand). These floating points in this array are called logit.

The results of WhatCNN are stored back into the zone data structure.

All zones for a moment are then passed back from each video program to the scene program.

The scene program receives all regions from all videos at a time and stores the result in a key-value dictionary, where the key is a personal ID and the value is a key-value dictionary, where the key is a camera ID The value is the Logit of the district.

This aggregate data structure is then stored in a ring buffer, which maps the number of frames to the aggregate structure at each time. 1.2 WhenCNN model

Images from different cameras processed by the WhatCNN model are combined during a period of time (most cameras during a period of time). The additional input to this model is the hand position in 3D space, triangulated from most cameras. Another input for this algorithm is the distance between the hand and the store's planogram. In some embodiments, a planogram can be used to identify whether the hand is approaching a shelf containing a specific item (eg, a cheerios box). Another input to this algorithm is the foot position on the store.

In addition to using SKUs to classify objects, the second classification model uses time series analysis to determine whether the object was picked up from the shelf or placed on the shelf. The images are analyzed during a period of time to determine whether the item that was in the hand in the previous image frame has been put back into the shelf or picked up from the shelf.

For a second time (30 frames per second) period and three cameras, the system will have 90 category outputs, and increase confidence for the same hand. This combined with image analysis significantly increases the chances of correctly identifying an object in the hand. Coverage analysis improves the quality of the output, albeit at the output of some very low confidence levels in the respective boxes. This step may have, for example, output confidence from 80% accuracy to 95% accuracy.

This model also includes the output from the shelf model as its input to identify what items the person has picked up.

The scenario program waits for 30 or more aggregate structures to accumulate (which represents at least one second of real time), and then performs further analysis to reduce the aggregate structure down to a single integer for each individual ID-hand pair, where the integer represents the Unique ID of the SKU in the store. For a moment, this information is stored in a key-value dictionary, where the key is a personal ID-hand pair and the value is a SKU integer. This dictionary is stored in a ring buffer as time passes, which maps the number of boxes to the dictionaries for that moment.

Additional analysis can then be performed to observe how this dictionary changes over time to identify when an individual took something and what it took away. This model (WhenCNN) is issued SKU logit and logit for each Brin problem: something is taken away? Something is placed.

The output of WhenCNN is stored in a ring buffer, which maps the number of boxes to a key-value dictionary, where the key is a personal ID and the value is an extended logit issued by WhenCNN.

Another set of heuristics is then run on the stored results of both WhenCNN and the person's stored joint positions, and on pre-calculated maps of items on the store shelves. This collection of heuristics determines where the removal and dropping of the heuristics caused the item to be added to or removed from. For each take / drop, the heuristics determine that the take or drop is from the shelf to the shelf, from the basket, or from the individual. The output is inventory for each person, which is stored as an array, where the array value on the index of the SKU is the number of those SKUs owned by the individual.

When a shopper approaches the store's exit, the system can transfer the inventory list to the shopper's mobile phone. The phone then displays the user's inventory and asks to confirm the charge from the credit card information stored in it. If the user accepts, their credit card will be charged. If it does not have a credit card known in the system, it will be required to provide credit card information.

Alternatively, shoppers can also approach the kiosk in the store. The system identifies when the shopper approaches the service desk and sends a message to the service desk to display the shopper's inventory. The help desk requires the shopper to accept a charge for the inventory. If a shopper accepts, they can then swipe their credit card or insert cash to pay. Figure 16 presents a diagram of the WhenCNN model for district proposals. Misplaced items

This feature identifies misplaced items when those items are returned to a random shelf by an individual. This causes the problem of object recognition, because the foot and hand positions relative to the plan view will be incorrect. As a result, the system builds modified planograms over time. Based on previous time series analysis, the system was able to determine whether an individual had placed items back on the shelf. Next time, when an item is picked up from the shelf position, the system learns that at least one misplaced item is in the hand position. Accordingly, the algorithm will have some confidence that the individual may pick up misplaced items from the shelf. If the misplaced item is picked up from the shelf, the system subtracts the item from the location, and the shelf no longer has the item. The system can also use an app to inform the store clerk about the misplaced item so that the store clerk can move the item to its correct shelf. 3. Semantic differences (shelf model)

Alternative techniques for background image processing include background subtraction algorithms to identify changes to items on the shelves (items are removed or placed). This is based on changes in pixel level. If someone is in front of the shelf, the algorithm stops so that it does not take into account pixel changes due to the presence of people. Background subtraction is a noise process. Therefore, cross-camera analysis is performed. If enough cameras agree that there is a "semanticly meaningful" change on that shelf, the system records that it has changed in that part of the shelf.

The next step is to identify whether the change is a "drop" or "take away" change. In this regard, a time series analysis of the second classification model is used. A zone proposal for this particular part of the shelf is generated and passed through a deep learning algorithm. This is easier than hand image analysis, because the object is not blocked inside the hand. A fourth input is provided to the algorithm, in addition to three typical RGB inputs. The fourth channel is background information. The output of this shelf or semantic difference is re-input to the second classification model (time series analysis model).

The semantic differences in this method include the following steps: 1. The images from the camera are compared with earlier images from the same camera. 2. The corresponding pixels between the two images are compared via the Euclidean distance in the RGB space. 3. The distance above a certain threshold is marked, which results in a new image of the pixel just marked. 4. The collection of image shape filters is used to remove noise from the marked image. 5. We then search for a large collection of labeled pixels and form a bounding box around them. 6. For each bounding box, we then observe the original pixels in the two images to obtain two image snapshots. 7. These two image snapshots are then pushed into CNN, which is trained to classify whether the image area represents the item being taken away or the item being placed and what the item is. 3. Store Audit

The inventory of each shelf is maintained by the system. Items are updated when they are picked up by consumers. At any point in time, the system can produce an audit of store inventory. 4. Most items in hand

Different images are used for most projects. The two projects in hand were treated differently than one. Some algorithms can only predict one item instead of several items. As a result, CNNs are trained so that their number of items for "two" can be different from a single item in their hands. 5. Data collection system

Pre-defined shopping scripts are used to collect good quality data from the images. These images are used for algorithm training. 5.1 Shopping script

Data collection includes the following steps: 1. Scripts are automatically generated to inform human actors what actions to take. 2. These actions are randomly sampled from a set of actions including: removing item X, dropping item X, and holding item X for Y seconds. 3. When performing these actions, the actor moves and orients itself in as many ways as possible, while still succeeding in the intended action. 4. During the sequence of actions, the collection of cameras records the actors from many perspectives. 5. After the actors have completed the script, the camera videos are bundled together and stored with the original script. 6. This script is used as input tags for machine learning models (such as CNN) trained on actors' videos. 6. Product Line

The system and its parts can be used for cashierless checkout, which is supported by the following applications. 6.1 Store app

The store application has several main possibilities: providing data analysis visualization, supporting loss prevention, and providing a platform to assist consumers by showing where the retailer is in the store and what items they have collected. The level of permission for employees and access to applications can be determined by the retailer. 6.1.1 Standard analysis

Information is collected by the platform and can be used in a variety of ways. 1. Derived data is used to perform a variety of analyses of: stores, the shopping experience they provide, and consumer interactions with products, the environment, and others. A. This information is stored and used in the background to perform analysis of store-consumer interactions. The store app will display some visualizations of this information to the retailer. Other data are stored and asked (when the data point is desired). 2. Heat map: The platform visualizes the following: the retailer's floor plan, shelf layout, and other store environments, with overlays showing the level of multiple activities. 1. Example: 1. Map for a place where people walked but did not touch any product. 2. Map for the place where people stand when interacting with the product. 3. Misplaced items: The platform is the owner of the SKU that tracks the store. When a project is placed in an incorrect location, the platform will know where the project is and establish a log. At certain thresholds, or immediately, store employees can be alerted about misplaced items. Alternatively, employees can access misplaced item maps in the store application. When convenient, employees can then quickly find and correct misplaced items. 6.1.2 Standard Assistance • The store app will show the floor plan of the store. • It will display a graphic to represent everyone in the store. • When the graphic is selected (via contact, pressing, or other means), relevant information for store employees will be displayed. For example: a shopping cart item (the item it has collected) will appear in the list. • If the platform has a level of confidence below a predetermined threshold for a specific project and for a period of time (about it being owned by someone (shopping cart)), its graphic (currently a dot) will indicate The difference. The app uses color changes. Green indicates high confidence and yellow / orange indicates lower confidence. • Store employees with a store app can be told about this lower confidence. They can confirm that the consumer's shopping cart is correct. • Through the store app, retailer employees will be able to adjust (add or delete) shopping cart items for consumers. 6.1.3 Standard LP • If a shopper is using a store app, they only need to leave the store and be charged. However, if it weren't, they would have to use the guest app to pay for items in their shopping cart. • If shoppers bypass the visitor app on their way out of the store, their graphics indicate that they must be approached before leaving. The app uses a color change to red. Personnel also receive notification of potential losses. • Through the store app, retailer employees will be able to adjust (add or delete) shopping cart items for consumers. 6.2 Non-Store Apps

The following analysis features represent the additional capabilities of the platform. 6.2.1 Standard analysis 1. Product interaction: 粒度 Granular decomposition of product interaction, such as: a. Interaction time and conversion ratio for each product. B. A / B comparison (color, style, etc.). Some of the smaller products on the display stand have many options, such as color, taste, and so on. • Is rose gold manipulated more than silver? • Does the blue jar attract more interaction than the red jar? 2. Directional impressions: Learn where the location-based impression is and where the shoppers ’attention is difference. If he is watching his product 15 feet away (20 seconds), the impression should not consider where he is located, but where he is watching. 3. Consumer Identification: Remember repeat shoppers and their associated email addresses (collected by retailers in multiple ways) and shopping profiles. 4. Group dynamics: Decide when shoppers are watching other people interact with the product. • Do you interact with the product after answering the individual? • Do those people enter the store together, or may they be strangers? • Individuals or crowds spend more time in the store? 5. Consumers come back: Provide consumer target information Announce store experience. This feature may have slightly different implementations from retailer to retailer, depending on specific habits and strategies. It may require integration and / or development from retailers to take advantage of this feature.购物 • Shoppers will be asked if they want to receive notifications about products they may be interested in. This step can be integrated with the e-mail store method. • After leaving the store, consumers can receive an email with a product that they spend time in the store. Interaction thresholds for duration, contact, and witnessing (direction impressions) will be determined. When the threshold is met, the products will enter her list and be delivered to her shortly after she leaves the store.

In addition, or alternatively, the shopper may be sent an email after a period of time, which provides discounted products or other special information. These products will be items that they have expressed interest in (but have not purchased). 6.3 Guest application

The shopper app automatically helps people check out when they leave the store. However, the platform does not require shoppers to have or use a shopper app to use the store.

When shoppers / people do not own or use the shopper app, they go to the help desk (iPad / tablet or other screen) or they go to a pre-installed self-checkout machine. The display (integrated with the platform) will automatically show the consumer's shopping cart.

Shoppers will have the opportunity to see what they show. If they agree with the information on the display, they can either put cash into the machine (if the capability is built into the hardware (eg, a self-checkout machine)) or they swipe a credit or debit card. They can then leave the store.

If they disagree with the display, store personnel are told to use their choice to raise a challenge with a touch screen, button, or other means. (See Store Assistance under the Store App) 6.4 Shopper App

Through the use of the application (shopper application), consumers can leave the store with the product and be automatically charged and provide a digital receipt. Shoppers must open their app at any time while in the shopping area of a store. The platform will identify unique images that are displayed on the shopper's device. The platform will bind them to their account (Consumer Association) and whether or not they leave the app open, they will be able to remember who they are all the time they are in the shopping area of the store.

When a shopper collects items, the shopper application will display those items in the shopper's shopping cart. If shoppers want, they can view product information about the items they picked up (ie, added to their shopping cart). Product information is stored in the store's system or added to the platform. The ability to update this information, such as offering product discounts or displaying prices, is an option that retailers can request / purchase or develop.

When a shopper drops an item, it is removed from its shopping cart on the backend and on the shopper application.

If the shopper application is opened and then closed after the consumer association is completed, the platform will maintain the shopper's shopping cart and charge them correctly (once they leave the store).

Shopper apps also have mapping information about their development guidelines. It can tell the consumer where to find items in the store, if the consumer requests the information by typing in a search item. At a later date, we will get the shopper's shopping list (by typing the app manually or through other smart systems) and show the fastest route through the store to collect all the desired items. Other filters (such as "bagging preferences") can be added. The bagging preference filter allows shoppers not to follow the fastest route, but to collect the strongest items first, and then the more fragile items later. 7. Types of consumers

Member Consumers-The first type of consumer uses an application to log into the system. The consumer is prompted with a picture and when she / he presses, the system will link it to the consumer's internal id. If the consumer has an account, the account is automatically charged (when the consumer walks out of the store). This is a member-based store.

Visitor consumers-Not every store will have a membership system, or consumers may not have a smartphone or credit card. This type of consumer will ask the help desk. The help desk will display the items the consumer has and will ask the consumer to deposit money. The help desk will already know about all the items that the consumer has purchased. For this type of consumer, the system is able to identify whether the consumer has not paid for an item in the shopping cart and prompt the cash register on the door (before the consumer arrives there) to let the cash register know about the outstanding payment s project. The system can also prompt for an item that has not yet been paid, or the system has low confidence in a project. This is called predictive path finding.

The system assigns a color code (green or yellow) to a consumer walking in the store based on the confidence level. Green coded consumers are either logged into the system or the system has high confidence about them. Yellow-coded consumers have one or more items that have not been predicted with high confidence. The clerk can watch the yellow dots and press them to identify the problem item, approach the consumer, and resolve the problem. 8. Analysis

A large group of analytical information was collected about the consumer, such as how much time the consumer spent in front of a particular shelf. In addition, the system tracks where the consumer is looking (about the impression of the system), as well as items that the consumer picks up and puts back on the shelf. These analyses are currently available for e-commerce but not yet available for retail stores. 9. Functional modules

The following is a list of functional modules: 1. Use a system capture array that synchronizes the images in the camera store. 2. A system for identifying the joints in an image and the joints of other people. 3. A system for generating new people using joint sets. 4. A system for removing ghosts using joint collections. 5. A system for tracking each other over time by tracking joint collections. 6. A system for generating zone suggestions for each person present in the store, which indicates the SKU number (WhatCNN) of the item in hand. 7. A system to perform acquisition / drop analysis for zone proposals, which indicates whether the item in hand is picked up or placed on a shelf (WhenCNN). 8. A system to generate a per-person inventory array using zone proposal and get / drop analysis (output of WhenCNN combined with heuristics, pre-calculated maps of person's stored joint positions, and items on store shelves) . 9. A system to identify, track, and update the location of misplaced items on the shelf. 10. A system for tracking (change / acquire) items on the shelf using pixel-based analysis. 11. A system for performing inventory inspections in stores. 12. A system to identify most items in hand. 13. A system for collecting project image data from stores using shopping scripts. 14. A system for performing checkouts and receiving payments from member consumers. 15. A system for fulfilling checkouts and receiving payments from guest consumers. 16. A system to perform loss prevention by identifying outstanding items in the shopping cart. 17. A system for tracking consumers using color codes to assist shop assistants in identifying incorrectly identified items in a consumer's shopping cart. 18. A system for generating consumer shopping analysis, including location-based impressions, directional impressions, A / B analysis, consumer identification, group dynamics, and more. 19. A system for using shopping analytics to generate targeted consumer returns. 20. A system for generating thermal map overlays of a store to visualize different activities.

The techniques described in this article can support billless cashiers. go to the shop. Take things away. go away.

Cashierless checkout is a pure machine vision and deep learning based system. Shoppers skip the line and get what they want faster and easier. No RFID tags. No changes to the store's back-end system. Can ^{(3 rd party Point of Sale and} Inventory Management systems) integration with third-party point of sale and inventory management systems. Real-time 30 FPS analysis for each video feed. Pre-built, cutting-edge GPU clusters. Identify shoppers and the items they interact with. In the exemplary embodiment, there is no internet dependency. Most state-of-the-art deep learning models (including proprietary custom algorithms) are used for the first time to resolve gaps in machine vision technology.

Techniques & Capabilities include the following: 1. Standard cognitive machine learning pipeline solves: a) People detection. B) Individual tracking. C) Most cameras agree personally. D) Hand detection. E) Project classification. F) Project ownership resolution.

Combining these technologies, we can: 1. Track everyone through their instant shopping experience. 2. Learn what shoppers have in their hands, where they stand, and what they put back. 3. Learn what direction and how long shoppers face. 4. Identify misplaced items and perform 24/7 visual sales audit.

Detects what really is in the hands of shoppers and in their baskets. Learn from your store:

Custom neural networks trained for specific stores and projects. Training materials are reusable across all store locations. Standard deployment:

The ceiling camera must be installed with double coverage of all areas of the store. For general walkways, cameras between 2 and 6 are required.

Pre-built GPU clusters fit into one or two server racks in the back office.

Example systems can be integrated with or include points for sales and inventory management systems.

First system, method and computer program product using an array of synchronized cameras to capture images in a store.

A second system, method, and computer program product for identifying a joint in an image and a joint of each other.

A third system, method, and computer program product using joint sets to generate newcomers.

A fourth system, method, and computer program product for removing ghost people using joint sets.

A fifth system, method, and computer program product that tracks each other over time by tracking joint collections.

A sixth system, method, and computer program product for generating a zone proposal for each person present in the store, which indicates the SKU number (WhatCNN) of the item in hand.

The seventh system, method, and computer program product to perform acquisition / drop analysis for district proposals, which indicates whether the item in hand is picked up or placed on a shelf (WhenCNN).

Eighth system, method, and computer program product (e.g., heuristics, person's stored joint positions, and items on store shelves with pre-calculated maps) of the eighth system, method, and computer program product for use area proposal and acquisition / drop analysis Combined with the output of WhenCNN).

Ninth system, method, and computer program product for identifying, tracking, and updating locations of misplaced items on a shelf.

A tenth system, method, and computer program product that uses pixel-based analysis to track changes (acquisition / drop) to items on the shelf.

An eleventh system, method, and computer program product for performing inventory inspection of a store.

Twelfth system, method and computer program product for identifying most items in hand.

Thirteenth system, method and computer program product using shopping script to collect image data of items from a store.

A fourteenth system, method, and computer program product for performing checkout and receiving payments from member consumers.

A fifteenth system, method, and computer program product for performing checkout and receiving payments from guest consumers.

A sixteenth system, method, and computer program product that performs loss prevention by identifying outstanding items in a shopping cart.

A seventeenth system, method, and computer program product that uses, for example, a color code to track a consumer to assist a store clerk to identify incorrectly identified items in the consumer's shopping cart.

An eighteenth system, method, and computer program product for generating consumer shopping analysis, the analysis includes one or more position-based impressions, directional impressions, A / B analysis, consumer identification, group dynamics, and many more.

The nineteenth system, method, and computer program product that uses shopping analysis to generate targeted consumer returns.

Twentieth system, method and computer program product for generating thermal map overlays of a store to visualize different activities.

Twenty-first system, method and computer program for hand detection.

Twenty-two systems, methods and computer programs for item classification.

Twenty-third system, method and computer program for project ownership resolution.

Twenty-fourth system, method and computer program for project person detection.

Twenty-fifth system, method and computer program for tracking individual items.

Twenty-sixth method and computer program for personal consent of most cameras.

The twenty-seventh system, method, and computer program product for essentially a cashier-free checkout as described herein.

Any one of systems 1-26 in combination with any other system or systems in systems 1-26 listed above.

The article described is a method to track down and take off the inventory item of the subject in the area of real space, including:

Using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera overlapping the viewing fields of at least one other camera of the plurality of cameras;

Receiving the sequences of images from the plurality of cameras, and using a first image recognition engine to process the images to generate a first data set that identifies the subjects and locations of the identified subjects in the real space;

Processing the first data set to specify a bounding box of an image of the identified subject's hand in the images of the sequences in which it includes the image;

Receiving the sequences of images from the plurality of cameras, and processing the designated bounding boxes in the images to use a second image recognition engine to generate a classification of the hands of the identified subjects, the classification including the Whether the identified subject is holding inventory items, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, and the second proximity category, which indicates the hand of the identified subject relative to the identified subject The position of the body, the third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and an identifier of a possible inventory item; and

The types of hands that process the set of images in the sequences of the identified subject are detected to detect the removal of the inventory item by the identified subject and the drop of the inventory item by the identified subject on the inventory display structure.

In the method described herein, the first data sets may include a set of candidate joints with coordinates in real space for each identified subject.

The method described herein may include processing the first data sets to specify bounding boxes including specifying the bounding boxes based on the positions of the joints in the sets of candidate joints for each subject.

In the method described herein, one or both of the first and second image recognition engines may include a convolutional neural network.

The method described may include using a convolutional neural network to process these classes of bounding boxes.

Describe a computer program product (and product) that includes computer-readable memory that contains non-transitory data storage media, and computer instructions stored in the memory can be executed by a computer to track areas of real space by The entity's inventory items are dropped and taken away by any of the procedures described in the text.

Describe a system including a plurality of cameras that generate a sequence of images including a subject's hand; and a processing system that is coupled to the plurality of cameras, the processing system including a hand image recognition engine that receives the sequences of images The category of the hand is generated in time series, and logic is used to process the categories of the hand from the sequences of the image to identify the action by the subject, where the action is the dropping and removing of the inventory item one.

The system may include logic to identify the positions of the joints of the subject in the images in the sequences of the images, and corresponding images to identify the hands that include the subject based on the identified joints. Bounding box in.

The appendix of the list of computer programs is a part of an example of a computer program used to implement some parts of the system provided in this application. The appendix includes examples of heuristics to identify the subject's joints and inventory items. The appendix proposes computer code to update the main body's shopping cart data structure. The appendix also includes computer routines to calculate the learning rate during the training of a convolutional neural network. The appendix includes computer program routines to store the classification results of the subject's hands from the convolutional neural network in the data structure of each subject and each hand of each image frame from each camera.

100‧‧‧ system

101a, 101b, 101c‧‧‧ network nodes

102‧‧‧Network Node

110‧‧‧Tracking Engine

112a-112n‧‧‧Image recognition engine

114‧‧‧ Camera

116a‧‧‧Aisle

116b‧‧‧Aisle

116n‧‧‧Aisle

120‧‧‧ Tuner

140‧‧‧Main database

150‧‧‧ training database

160‧‧‧ Heuristics Database

170‧‧‧ Calibration database

181‧‧‧Internet

202‧‧‧Shelf A

204‧‧‧Shelf B

206‧‧‧ Camera A

208‧‧‧Camera B

216‧‧‧viewing domain

218‧‧‧viewing domain

220‧‧‧ Floor

230‧‧‧ roof

412-418‧‧‧ Camera

422-428‧‧‧ Ethernet-based connector

430‧‧‧Storage Subsystem

432‧‧‧Host Memory Subsystem

434‧‧‧ Random Access Memory (RAM)

436‧‧‧Read Only Memory (ROM)

440‧‧‧File Storage Subsystem

442‧‧‧RAID 0

444‧‧‧Solid State Drive (SSD)

446‧‧‧HDD

450‧‧‧Processor Subsystem

454‧‧‧Bus Subsystem

462‧‧‧GPU 1

464‧‧‧GPU 2

466‧‧‧GPU 3

481‧‧‧Internet

510‧‧‧ input image

520‧‧‧Filter

530‧‧‧ Convolutional layer

540‧‧‧ output matrix

702‧‧‧Overall Measurement Calculator

800‧‧‧ main data structure

1411‧‧‧Video Program

1415‧‧‧Scene Program

1452‧‧‧ output

1453‧‧‧Enter

1457‧‧‧output

1502‧‧‧Circular buffer

1504‧‧‧Delimited Box Generator

1506‧‧‧WhatCNN

1508‧‧‧WhenCNN

1510‧‧‧Shopping cart data structure

1520‧‧‧Video Channel

1522‧‧‧ Coordination logic module

2002‧‧‧ Heuristics

2111‧‧‧Input

2113‧‧‧First Convolutional Layer

2115‧‧‧Second Convolution Layer

2117‧‧‧square box

2119, 2121‧‧‧th floor

2123‧‧‧eight convolutional layers

2125, 2127‧‧‧ Convolutional layer

2129‧‧‧eight convolutional layers

2135‧‧‧Fully connected layer

2210‧‧‧ One-Handed Model

2212‧‧‧ Convolutional Layer

2214‧‧‧Common use layer

2216‧‧‧block 0

2218‧‧‧Block 1

2220‧‧‧Block 2

2222‧‧‧Block 3

2310‧‧‧square box

2312‧‧‧ Convolutional Layer

2314‧‧‧ Batch normalization layer

2316‧‧‧ReLU Nonlinear

2318‧‧‧conv1

2320‧‧‧conv2

2322‧‧‧conv3

2324‧‧‧ Addition

2326‧‧‧jump connection

2410‧‧‧Fully Connected Layer (FC)

2412‧‧‧ Reshaping Operator

2420‧‧‧Next floor

2422‧‧‧MatMul

2424‧‧‧ Operator

2426‧‧‧ Output

2502‧‧‧ Part I

2504‧‧‧ Part Two

2506‧‧‧Part III

2508‧‧‧Part 4

2510‧‧‧Part 5

2512‧‧‧Part VI

2602‧‧‧First Image Processor Subsystem

2604‧‧‧Second Image Processor Subsystem

2606‧‧‧Third Image Processor Subsystem

2608‧‧‧Select logic component

2702‧‧‧Mask logic components

2704‧‧‧Background image storage

2706‧‧‧factorized image

2710‧‧‧Bit Mask Calculator

2714a-2714n‧‧‧ Change CNN

2718‧‧‧ Coordination logic component

2720‧‧‧Log Generator

2724‧‧‧Mask generator

FIG. 1 illustrates an architectural level diagram of a system in which a tracking engine uses joint data generated by an image recognition engine to track a subject.

FIG. 2 is a side view of the aisle in a shopping store illustrating the camera configuration.

FIG. 3 is a top view of the walkway of FIG. 2 in a shopping store illustrating a camera configuration.

FIG. 4 is a camera and computer hardware configuration configured to control the image recognition engine of FIG. 1.

FIG. 5 illustrates a convolutional neural network, which illustrates the identification of joints in the image recognition engine of FIG. 1.

FIG. 6 shows an example data structure for storing joint information.

FIG. 7 illustrates the tracking engine of FIG. 1 with an overall metric calculator.

FIG. 8 shows an example data structure of a subject for storing information including related joints.

FIG. 9 is a flowchart illustrating the steps of a procedure for tracking a subject by the system of FIG. 1. FIG.

FIG. 10 is a flowchart showing a more detailed procedure step of the camera calibration step of FIG. 9.

FIG. 11 is a flowchart showing more detailed program steps of the video program steps of FIG. 9.

FIG. 12A is a flowchart showing the first part of a more detailed program step of the scenario program of FIG. 9.

FIG. 12B is a flowchart showing the second part of the more detailed program steps of the scenario program of FIG. 9.

FIG. 13 is a diagram of an environment in which the embodiment of the system of FIG. 1 is used.

FIG. 14 is a diagram of a video and scene program in the embodiment of the system of FIG. 1. FIG.

FIG. 15A is a schematic diagram showing a structure of a shopping cart data having a majority convolutional neural network (CNN) including joints CNN, WhatCNN, and WhenCNN to generate each subject in real space.

FIG. 15B shows most image channels from most cameras and coordination logic for those subjects and their respective shopping cart data structures.

FIG. 16 is a flowchart illustrating the procedure steps for identifying and updating a subject in a real space.

FIG. 17 is a flowchart showing the steps of a procedure for processing a hand joint of a subject to identify an inventory item.

FIG. 18 is a flowchart showing program steps for time series analysis of inventory items of each hand joint to generate a shopping cart data structure for each subject.

FIG. 19 is a diagram of a WhatCNN model in the embodiment of the system of FIG. 15A.

FIG. 20 is a diagram of a WhenCNN model in the embodiment of the system of FIG. 15A.

Figure 21 presents an example architecture of a WhatCNN model that identifies the dimensions of the convolutional layer.

Figure 22 presents a high-level block diagram of an embodiment of WhatCNN for categories of hand images.

FIG. 23 presents details of the first block of the high-order block diagram of the WhatCNN model proposed in FIG. 22.

FIG. 24 presents the operators in the fully connected layer in the example WhatCNN model proposed in FIG. 22.

FIG. 25 is an example name of an image file stored as part of a training data set for a WhatCNN model.

FIG. 26 is a high-level architecture of a system for tracking changes by subjects in a region of real space, where the selection logic is based on the first detection using background semantic diffing and the redundant detection using the foreground area proposal Choose between.

FIG. 27 presents the components of a subsystem that implements the system of FIG. 26.

FIG. 28A is a flowchart showing the first part of the detailed procedure steps for determining the generation of the inventory event and the shopping cart data structure.

FIG. 28B is a flowchart showing the second part of the detailed procedure steps for determining the occurrence of the inventory event and the shopping cart data structure.

Claims (177)

A system for tracking multi-joint subjects in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generate respective sequences of images of corresponding viewing fields in the real space, and each of the cameras should view The domain is overlapped with the viewing domain of at least one other camera of the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system includes: an image recognition engine receiving the images from the plurality of cameras; These sequences process images to generate corresponding arrays of joint data structures, and the arrays of joint data structures corresponding to specific images are obtained by the type of joint, the time of the specific image, and the coordinates of the components in the specific image Classify the elements of the specific images; a tracking engine configured to receive the arrays of joint data structures corresponding to the images in a sequence of images from cameras with overlapping viewing domains, and will correspond to the The blocks of the joint data structure of the image the blocks of the elements in the arrays Transforming the coordinates into candidate joints with coordinates in the real space; and unitary logic to identify a set of candidate joints with coordinates in the real space as multi-joint subjects in the real space. For example, the system of claim 1 in the patent application scope, wherein the image recognition engines include a convolutional neural network. For example, the system of claim 1 in which the image recognition engine processes the image to generate a confidence array for the components of the image, wherein the confidence array of the specific component of the image includes the confidence value of the multiple joint types of the specific component, and according to The confidence array selects a joint type of the joint data structure for the specific component. For example, the system of claim 1 in which the logic used to identify the set of candidate joints includes a heuristic function based on the physical relationship between the joints of the subject in the real space to identify the set of candidate joints as multiple joints main body. For example, the system of claim 4 includes logic to store the sets of joints identified as multi-joint subjects, and the logic to identify sets of candidate joints includes logic to determine Whether the identified candidate joints in the time-acquired image meet members of one of those sets of candidate joints that have been identified as multiple joint subjects in the previous image. For example, the system of claim 1, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the system of claim 1 in the patent scope, wherein the plurality of cameras include a camera disposed above and having a viewing field covering various parts of the area in real space, and which are identified as candidate joints of a multi-joint subject The coordinates in the real space of the members of the set identify positions in the region of the multi-joint subject. For example, the system of claim 1 includes logic to track the position of a plurality of multi-joint subjects in the area of real space. For example, the system of claim 8 includes logic to determine when the multi-joint subject among the plurality of multi-joint subjects leaves the area of real space. For example, the system of claim 1 includes logic to track the positions of the majority of candidate joints in the region of real space where they are members of a set of candidate joints identified as a specific multi-joint subject. A method for tracking multi-joint subjects in a region of real space, comprising: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing field of each camera and the plurality of cameras The viewing fields of at least one of the other cameras overlap; processing the images in the sequences of the image to generate corresponding arrays of joint data structures, the arrays corresponding to the joint data structure of a particular image are determined by the joint type, the particular The time of the image and the coordinates of the components in the specific image to classify the components of the specific image; transform the coordinates of the components in the arrays of the joint data structures corresponding to the images in different sequences Are candidate joints having coordinates in the real space; and a set of candidate joints having coordinates in the real space are identified as multi-joint subjects in the real space. For example, the method of claim 11 in which the processing image includes using a convolutional neural network. For example, the method of claim 11 in which the processing image includes generating a confidence array for the components of the image, wherein the confidence array of a specific component of the image includes a confidence value of a plurality of joint types of the specific component, and according to the confidence array To select the joint type of the joint data structure for the specific component. For example, the method of claim 11, wherein identifying the set of candidate joints includes applying a heuristic function based on the physical relationship between the joints of the subjects in the real space to identify the set of candidate joints as multi-joint subjects. For example, the method of claim 14 includes storing the sets of joints that are identified as multi-joint subjects, and identifying the set of candidate joints includes determining whether the candidate joints identified in the images obtained at a specific time meet It is identified as a member of one of these sets of candidate joints of the multi-joint subject in the previous image. For example, the method of claim 11 in which the sequences of the image are synchronized. For example, the method of claim 11 of the patent scope, wherein the plurality of cameras include a camera disposed above and having a viewing field covering various parts of the area in real space, and which are identified as candidate joints of a multi-joint subject The coordinates in the real space of the members of the set identify positions in the region of the multi-joint subject. For example, the method of claim 11 includes tracking the position of a plurality of multi-joint subjects in the area of real space. For example, the method of claim 18 of the scope of patent application includes determining when the articulated subject among the plurality of articulated subjects leaves the area of real space. The method as claimed in item 11 of the patent application scope includes tracking the position of a majority of candidate joints that are members of a set of candidate joints identified as a particular multi-joint subject in that region of real space. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program The multi-articulated subject of the program includes: using a sequence of images from a plurality of cameras having corresponding viewing fields in real space, the viewing field of each camera overlapping the viewing field of at least one other camera of the plurality of cameras ; Processing the images in the sequences of the image to generate corresponding arrays of joint data structures, the arrays corresponding to the joint data structure of a particular image are by joint type, time of the particular image, and components in the particular image The coordinates of the specific images to classify the components of the particular image; transform the coordinates of the components in the arrays corresponding to the joint data structure of the images in different sequences into candidate joints with the coordinates in the real space ; And will have Identifying a set of candidate subject in the real space of the joint as much as for the joint body. For example, the product in the scope of patent application No. 21, wherein processing the image includes using a convolutional neural network. For example, the product of the scope of application for patent No. 21, wherein the processed image includes generating a confidence array for the component of the image, wherein the confidence array of the specific component of the image includes the confidence value of the multiple joint types of the specific component, and according to the confidence array To select the joint type of the joint data structure for the specific component. For example, the product in the scope of application for patent No. 21, wherein identifying the set of candidate joints includes applying a heuristic function based on the physical relationship between the joints of the subjects in the real space to identify the set of candidate joints as multi-joint subjects. For example, the product in the 24th scope of the patent application includes storing these sets of joints identified as multi-joint subjects, and the set of identifying candidate joints includes determining whether the candidate joints identified in the images obtained at a specific time meet It is identified as a member of one of these sets of candidate joints of the multi-joint subject in the previous image. For example, the product in the scope of patent application 21, wherein the sequences of the image are synchronized. For example, the product under the scope of patent application No. 21, wherein the plurality of cameras include a camera disposed above and having a viewing field covering various parts of the area in real space, and which are identified as candidate joints of a multi-joint subject The coordinates in the real space of the members of the set identify positions in the region of the multi-joint subject. For example, the product in the scope of patent application No. 21 includes tracking the position of a plurality of multi-joint subjects in the area of real space. For example, the product under the scope of patent application No. 28 includes determining when the articulated subject among the plurality of articulated subjects leaves the area of real space. For example, the product under the scope of patent application No. 21 includes tracking the positions of the majority of candidate joints that are members of a set of candidate joints identified as a specific multi-joint subject in the region of real space. A system for tracking the dropping and taking out of inventory items of a subject in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, each camera The viewing field overlaps with the viewing field of at least one other camera among the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system including a plurality of image recognition engines receiving the plurality of cameras from the plurality of cameras; A corresponding sequence of images, the image recognition engine in the plurality of image recognition engines processes the images in the corresponding sequences to identify a subject displayed in the images; and logic for processing which includes the identified A collection of images in the sequences of the subject's image to detect that the inventory item is taken away by the identified subject and the inventory item is dropped by the identified subject on the shelf. For example, the system for applying item 31 of the patent scope, wherein the logic for processing the collection of images includes: For the identified subjects, the logic for processing the images to generate the categories of the images of the identified subjects, the categories Including: whether the identified subject is holding inventory items, a first proximity category, which indicates the position of the identified subject's hand relative to the shelf, and a second proximity category, which indicates that the identified subject's hand is relative to the The location of the identified subject's body, the third proximity category, are identifiers indicating the position of the identified subject's hand relative to the basket associated with the identified subject, and the possible inventory item. For example, the system of claim 32 includes logic to perform time series analysis of the categories covering the images to detect the removal and the drop by the identified subjects. For example, the system of claim 31, wherein the logic for processing the collection of images includes: For identified subjects, logic to identify the hand in the images in the collections representing the images of the identified subjects The bounding boxes of the data, and the types of data in the bounding boxes that are used to process the data in the bounding boxes to generate the identified subjects. For example, if the system of claim 34 is applied, the categories include: whether the identified subject is holding inventory items, and the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity Degree category, which indicates the position of the identified subject's hand relative to the body of the identified subject, a third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and possibly Identifier of the inventory item. For example, the system of claim 34 includes logic to perform time series analysis of the categories covering the data in the bounding boxes in the collections of the images to detect the Take them away and let them go. The system of claim 31 includes a circular buffer that is coupled to the cameras of the plurality of cameras to store a collection of images in the sequences of images from the plurality of cameras. For example, the system of claim 31, wherein the logic for processing a collection of images includes a convolutional neural network. For example, the system of claim 31, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the system of claim 31, wherein the plurality of cameras include cameras arranged above and having viewing fields covering respective parts of the area in real space. The system of claim 31, including logic, includes logic that generates a log data structure in response to the detected take and drop, and includes a list of inventory items for each identified entity. A method for tracking a subject dropping and removing inventory items in an area of real space, the method comprising: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera being Overlaps with the viewing field of at least one other camera of the plurality of cameras; receives a corresponding sequence of images from the plurality of cameras, and uses an image recognition engine in the plurality of image recognition engines to process the images in the corresponding sequences And identifying the subjects represented in the images, wherein the plurality of image recognition engines are part of a processing system coupled to the plurality of cameras; and processing the images in the sequences that include the images of the identified subjects Collect to detect that the inventory item is taken by the identified subject and that the inventory item is dropped by the identified subject on the shelf. For example, the method of applying for the scope of patent scope No. 42, wherein the set of processed images includes: For the identified subjects, the categories of the images of the identified subjects are generated, and the categories include: whether the identified subjects are holding inventory Item, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity category, which indicates the position of the identified subject's hand relative to the body of the identified subject, the third proximity The degree category indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and an identifier of a possible inventory item. For example, the method of claim 43 of the scope of patent application includes performing a time series analysis of the categories covering the images to detect the removal and the drop by the identified subjects. For example, the method of applying for the scope of patent scope No. 42, wherein the set of processed images includes: for the identified subjects, a delimited frame identifying the data of the hand in the images in the sets representing the images of the identified subjects, and The data in the bounding boxes are processed to generate categories of data in the bounding boxes for the identified subjects. For example, the method of applying for item 45 of the patent scope, wherein the categories include: whether the identified subject is holding inventory items, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity Degree category, which indicates the position of the identified subject's hand relative to the body of the identified subject, a third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and possibly Identifier of the inventory item. For example, the method of applying scope 45 of the patent includes performing time series analysis of the categories of data in the bounding boxes in the collections of the images to detect the removal by the identified subjects. And those to put down. The method of claim 42 includes a circular buffer that is coupled to the cameras in the plurality of cameras to store a collection of images in the sequences of images from the plurality of cameras. The method of claim 42 includes applying a convolutional neural network to process a collection of images. For example, the method of claim 42 in which the cameras in the plurality of cameras are configured to generate a synchronized sequence of images. For example, the method of claim 42 in the patent application range, wherein the plurality of cameras include cameras disposed above and having viewing fields covering respective parts of the area in real space. The method of applying for a patent scope item 42 includes generating a log data structure in response to a detected take and drop, which includes a list of inventory items for each identified entity. A system for tracking the dropping and taking out of inventory items of a subject in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, each camera The viewing field overlaps with the viewing field of at least one other camera among the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system including: a first image recognition engine receiving the plurality of complex numbers; The sequences of camera images are processed to generate a first data set that identifies the subjects and locations of the identified subjects in the real space; logic to process the first data set to indicate that it includes images The bounding box of the image of the hand of the identified subject in the images in the sequences; The second image recognition engine receives the sequences of the images from the plurality of cameras, which processes the ones in the images The bounding box has been specified to generate a classification of the hands of the identified subjects, the classification includes: Whether the identified subject is holding inventory items, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, and the second proximity category, which indicates the hand of the identified subject relative to the identified subject The position of the body, the third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and an identifier of a possible inventory item; and logic to process the image of the identified subject The types of hands in the collection of images in these sequences detect the removal of the inventory item by the identified subject and the placement of the inventory item by the identified subject on the shelf. For example, the system of claim 53 includes a circular buffer that is coupled to the cameras in the plurality of cameras to store a collection of images in the sequences of images from the plurality of cameras. For example, the system of claim 53 in the patent application scope, wherein the first data sets include a set of candidate joints for each identified subject having coordinates in real space. If the system of claim 53 is applied, the logic for processing the first data sets to specify the bounding boxes is to specify the bounding boxes based on the positions of the joints in the sets of candidate joints for each subject. . For example, the system of claim 53 in which the second image recognition engine includes a convolutional neural network. For example, the system of claim 53, wherein the logic for processing the classes of the bounding box includes a convolutional neural network. For example, the system of claim 53, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the system of claim 53, wherein the plurality of cameras include cameras disposed above and having viewing fields covering respective parts of the area in real space. For example, the system for applying for the scope of patent No. 53 includes logic to generate a log data structure including a list of inventory items for each identified entity. A method for tracking subjects dropping and taking inventory items in an area of real space, including: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera and the The viewing fields of at least one other camera of the plurality of cameras overlap; receive the sequences of images from the plurality of cameras, and use a first image recognition engine to process the images to generate the recognition of the already in the real space A first data set identifying the subject's subject and location; processing the first data set to specify a bounding box of the image of the identified subject's hand in the images in the sequences that include the image; receiving data from the plurality of cameras The sequences of the images, and processing the indicated bounding boxes in the images to use a second image recognition engine to generate a classification of the hands of the identified subjects, the classification including: whether the identified subjects are holding Inventory item, first proximity category, which indicates the position of the identified subject's hand relative to the shelf, second Proximity category, which indicates the position of the identified subject's hand relative to the body of the identified subject, and third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and An identifier of a possible inventory item; and the type of hand that processes the collection of images in those sequences of the image of the identified subject to detect the removal of the inventory item by the identified subject and the placement of the inventory item by the identified subject on the shelf. The method of claim 62 includes a circular buffer that is coupled to the cameras in the plurality of cameras to store a collection of images in the sequences of images from the plurality of cameras. For example, the method according to item 62 of the patent application, wherein the first data sets include a set of candidate joints for each identified subject having coordinates in real space. For example, the method of claim 62, wherein processing the first data sets to specify bounding boxes includes specifying bounding boxes based on the positions of the joints in the sets of candidate joints for each subject. For example, the method according to item 62 of the patent application, wherein the second image recognition engines include a convolutional neural network. The method of claim 62 includes applying a convolutional neural network to deal with these classes of bounding boxes. For example, the method of claim 62, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the method according to item 62 of the patent application, wherein the plurality of cameras include cameras arranged above and having a viewing field covering various parts of the area in the real space. If the method of applying for the scope of patent No. 62 includes generating a log data structure including a list of inventory items for each identified entity. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program The subject lowers and removes the inventory item, the procedure includes: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing field of each camera and at least one other camera of the plurality of cameras The viewing domains overlap; (i) receiving the corresponding sequences of images from the plurality of cameras, using the image recognition engine in the plurality of image recognition engines to process the images in the corresponding sequences and identifying the subjects represented in the images, Wherein the plurality of image recognition engines are part of a processing system coupled to the plurality of cameras; and processing a collection of images in the sequences including images of the identified subjects to detect removal of inventory items by the identified subjects And the inventory item is dropped by the identified entity On. For example, if the product of the scope of patent application No. 71, the set of processed images includes: For the identified subjects, the types of the images of the identified subjects are generated, and the categories include: whether the identified subjects are holding inventory Item, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity category, which indicates the position of the identified subject's hand relative to the body of the identified subject, the third proximity The degree category indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and an identifier of a possible inventory item. For example, the product of the scope of patent application No. 72 includes performing a time series analysis of the categories covering the images to detect the removal and the drop of the identified subjects. For example, the product in the scope of patent application 71, wherein the set of processed images includes: for the identified subject, a delimited frame identifying the data of the hand in the images in the sets representing the images of the identified subjects, and The data in the bounding boxes are processed to generate categories of data in the bounding boxes for the identified subjects. For example, if the product under the scope of patent application No. 74, the categories include: whether the identified subject is holding inventory items, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity Degree category, which indicates the position of the identified subject's hand relative to the body of the identified subject, a third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and possibly Identifier of the inventory item. If the product under the scope of patent application No. 74 includes performing the time series analysis of the categories of the data in the bounding boxes in the collections covering the images to detect the removal by the identified subjects And those to put down. The product of claim 71 includes a circular buffer that is coupled to the cameras in the plurality of cameras to store a collection of images in the sequences of images from the plurality of cameras. For example, the product under the scope of patent application 71 includes the use of a convolutional neural network to process a collection of images. For example, the product in the scope of patent application 71, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the product of the scope of patent application No. 71, wherein the plurality of cameras include cameras disposed above and having a viewing field covering each part of the area in the real space. For example, the product under the scope of patent application 71 includes a log data structure generated in response to the detected take-off and drop, which includes a list of inventory items for each identified entity. A computer program product comprising: computer-readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track down and remove subjects in areas of real space Inventory items include: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera overlapping the viewing fields of at least one other camera among the plurality of cameras; receiving The sequences of images from the plurality of cameras, and using a first image recognition engine to process the images to generate a first data set that identifies the subjects and locations of the identified subjects in the real space; process the first data Set the bounding box that indicates the image of the hand of the identified subject in the images that include the sequences of the images; receive the sequences of the images from the plurality of cameras and process the existing ones of the images Specify the bounding box to use the second image recognition engine to generate the The classification of the identified subject's hand, the classification includes: whether the identified subject is holding inventory items, the first proximity category, which indicates the position of the identified subject's hand relative to the shelf, the second proximity category, which Indicates the position of the identified subject's hand relative to the body of the identified subject, the third proximity category, which indicates the position of the identified subject's hand relative to the basket associated with the identified subject, and the identification of possible inventory items And the types of hands that process the collection of images in those sequences of the image of the identified subject to detect the removal of the inventory item by the identified subject and the placement of the inventory item by the identified subject on the shelf. For example, the product under the scope of patent application No. 82 includes a circular buffer, which is a collection of images in the sequences coupled to the cameras in the plurality of cameras to store images from the plurality of cameras. For example, the product in the scope of patent application No. 82, wherein the first data sets include a set of candidate joints for each identified subject having coordinates in real space. For example, applying for a product in the 82nd patent scope, wherein processing the first data sets to specify the bounding boxes includes specifying the bounding boxes based on the positions of the joints in the sets of candidate joints for each subject. For example, the product under the scope of patent application No. 82, wherein the second image recognition engines include a convolutional neural network. For example, the product under the scope of patent application No. 82 includes the use of convolutional neural networks to deal with these categories of bounding boxes. For example, the product under the scope of patent application No. 82, wherein the cameras in the plurality of cameras are configured to generate a synchronization sequence of images. For example, the product under the scope of patent application No. 82, wherein the plurality of cameras include cameras disposed above and having a viewing field covering various parts of the area in real space. If the product of the scope of patent application is No. 82, it includes a log data structure that generates a list including inventory items for each identified entity. A system comprising: a camera that generates a sequence including an image of a hand; a processing system that is coupled to the camera, the processing system including a hand image recognition engine that receives the sequences of the image to generate a category of the hand In time series, and logic, the categories of the hand used to process the sequences from the image to identify actions by the subject. For example, the system for applying for the scope of the patent No. 91, wherein the actions are to put down and remove the inventory items. For example, the system of claim 91 includes logic to identify the positions of the joints of the subjects in the sequences in the sequences of the images, and to identify that they include the subjects based on the identified joints. The bounding box in the corresponding image of the hands. A system for tracking changes in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, and the viewing fields of each camera Overlaps with the viewing field of at least one other camera of the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system includes: a first image processor including a subject image recognition engine that receives a plurality of images from a plurality of cameras; The corresponding sequence of camera images is to process the images to identify the subjects represented in those images in the corresponding sequence of images; a second image processor that includes a background image recognition engine that receives the corresponding images from multiple cameras A sequence that masks the identified subjects to produce a masked image, and processes the masked images to identify and classify the background changes represented in the images in the corresponding sequences of images. For example, the system of claim 94, wherein the background image recognition engines include a convolutional neural network. For example, the system for applying scope of patent No. 94 includes logic for associating an identified background change with an identified subject. For example, the system of claim 94, wherein the second image processors include background image storage for storing background images of the corresponding sequences of the image; mask logic for processing the sequences of the image The background image data of the corresponding sequences of the background images from the images replaces the front stage image data representing the identified subjects to provide the occluded images. For example, the system of claim 97, wherein the masking logic is a combination of N masked images in the sequences of images to generate a sequence of factorized images for each camera, and the second image processors This sequence of factorized images is processed to identify and classify background changes. For example, the system of claim 94, wherein the second image processors include logic for generating changed data structures for the corresponding sequences of the images, and the changed data structures include the identified background changes. The coordinates in the occluded image, the identifiers of the inventory item subjects of the identified background changes, and the categories of the identified background changes; and coordination logic to process the changed data structure from the collection of cameras with overlapping viewing fields to Find out the identified background changes in real space. For example, if the system of claim 94 is applied, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image. If the system of claim 99 is applied, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image; and includes logic For associating the background change with the identified subject, and for performing the detection of the inventory items taken by the identified subjects and the detection of the inventory items being dropped by the identified subjects on the shelf. For example, the system for applying for item 94 of the patent scope includes logic for associating background changes with identified entities, and for performing inspections of items removed by the identified entities and dropping inventory items by the identified entities. Inspection on shelves. For example, the system of claim 94, wherein the first image processors identify the positions of the hands of the identified subjects; and include logic for comparing the changed positions with the hands of the identified subjects These positions are used to associate the background change with the identified subjects, and to perform the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped on the shelves by the identified subjects. For example, the system for applying for the item No. 94 includes a third image processor, which includes a front-end image recognition engine that receives corresponding sequences of images from the plurality of cameras, and processes the images to identify and classify the corresponding sequences of images. The foreground changes shown in these images. For example, the system for applying for the scope of patent No. 104 includes logic for associating a background change with an identified subject, and for performing an inspection of the inventory items taken by the identified subjects and dropping the inventory items by the identified subjects. The first set of inspections on the shelves; logic for associating front desk changes with identified entities, and for performing inspections of inventory items taken by the identified entities and dropping inventory items by the identified entities at A second set of inspections on the shelf; and selection logic for processing the first and second sets of inspections to generate a log data structure that includes a list of inventory items for identified entities. For example, the system of claim 94, wherein the sequences of the images from the cameras in the plurality of cameras are synchronized. A system for tracking the dropping and taking out of inventory items of a subject in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, each camera The viewing field overlaps with the viewing field of at least one other camera of the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system including: a first image processor including a subject image recognition The engine receives a corresponding sequence of images from a plurality of cameras, which processes the images to identify the subjects represented in the images in the corresponding sequence of the images; a second image processor, which includes a background image recognition engine, receives the A corresponding sequence of camera images that masks the identified subjects to produce a masked image, processes the masked images to identify and classify the background changes represented in the images in the corresponding sequences of images, and Handle identified background changes to perform removal from identified subjects The first set of detection of goods items and detection of items being put on the shelf by the identified subject; a third image processor, which includes a front-end image recognition engine that receives the corresponding sequences of images from the plurality of cameras, which are processed Images to identify and classify the foreground changes represented in those images in those corresponding sequences of images, and to process the identified foreground changes to perform the detection of inventory items taken by the identified subject and drop the inventory items by the identified subject in A second set of inspections on the shelf; and selection logic for processing the first and second sets of inspections to generate a log data structure that includes a list of inventory items for identified entities. A system for tracking the dropping and taking out of inventory items of a subject in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, each camera The viewing field overlaps with the viewing field of at least one other camera of the plurality of cameras; and a processing system coupled to the plurality of cameras, the processing system includes logic for detecting inventory items by: Drop and take away: semantically identify significant changes in inventory items on the inventory display structure and associate those significant changes with the subjects represented in the sequences of the images. For example, a system with a scope of 108 patent applications, wherein the processing system includes logic to detect the dropping and fetching of inventory items by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images go. A system for tracking the dropping and taking out of inventory items of a subject in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, each camera The viewing domain overlaps the viewing domain of at least one other camera among the plurality of cameras; and a processing system coupled to the plurality of cameras, the processing system includes logic for identifying a subject ’s posture and The inventory items related to the gestures represented in the sequences of the images are used to detect the drop and removal of the inventory items. A method for tracking changes in areas of real space, including: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera and at least one of the plurality of cameras The viewing domains of other cameras overlap; using a first image processor including a subject image recognition engine to process the images to identify subjects represented in the images in the corresponding sequences of the images; using a second image processing Device, which includes a background image recognition engine to mask the identified subjects in the images in the sequences of the image to generate masked images, and to process the masked images to identify and classify the corresponding sequences in the image The background shown in those images changes. For example, the method of claim 111, wherein the background image recognition engines include a convolutional neural network. For example, the method of applying for item 111 of the patent scope includes associating the identified background change with the identified subject. For example, the method of applying for item No. 111, wherein using the second image processors includes storing background images corresponding to the corresponding sequences of the images; processing the images in the sequences of the images and using the corresponding sequences from the images. The background image data of the background images replaces the front stage image data representing the identified subjects to provide the occluded images. For example, the method of claim 114, wherein processing the images in the sequences of images includes combining a set of N occluded images in the sequences of images to generate a sequence of factorized images for each camera, and the These second image processors process the sequence of factorized images to identify and classify background changes. For example, the method of claim 111, wherein using the second image processors includes generating changed data structures for the corresponding sequences of the images, and the changed data structures include the masked images whose identified background changes. , The identifiers of the inventory item subjects of the identified background changes, and the categories of the identified background changes; and processing the changed data structure from the set of cameras with overlapping viewing domains to find those in real space A background change has been identified. For example, the method of applying for item 116 of the patent scope, wherein the categories of the identified background change in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image. If the method of applying for the scope of patent No. 116, wherein the categories of the identified background change in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image; and includes using The background change is associated with the identified entities, and the detection of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the shelves are performed. For example, the method of applying for item No. 111 of the patent scope includes: associating a background change with the identified subject, and performing a test of the inventory items taken by the identified subjects and a test of the inventory items being dropped by the identified subjects on the shelf. For example, the method of applying for item No. 111, wherein using the first image processors includes identifying the positions of the hands of the identified subjects; and including comparing the positions of the changes with the positions of the hands of the identified subjects Position to associate the background change with the identified subject, and to perform the detection of the inventory items taken away by the identified subjects and the detection of the inventory items dropped on the shelf by the identified subjects. For example, the method of applying for item No. 111 includes using a third image processor, which includes a front-end image recognition engine that receives corresponding sequences of images from the plurality of cameras, and processes the images to identify and classify the corresponding images. The foreground changes represented in the images in the sequence. For example, the method of applying for the scope of patent No. 121 includes associating a background change with an identified subject, and performing the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the shelf. The first set; the second set of causing the front desk to change the association with the identified subjects, and performing the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the shelf; and the processing inspection These first and second sets are used to generate a log data structure including a list of inventory items of identified entities. The method of claim 111 includes synchronizing the sequences of images from the cameras in the plurality of cameras. A method for tracking subjects dropping and taking inventory items in an area of real space, including: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera and the The viewing fields of at least one of the plurality of cameras overlap; and detecting drop and removal of inventory items by semantically identifying significant changes in inventory items on the inventory display structure and making those semantically significant Changes are associated with the subjects represented in those sequences of the image. For example, the method of applying for the scope of 124 items of patent includes detecting the dropping and removing of the inventory item by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. A method for tracking subjects dropping and taking inventory items in an area of real space, including: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera and the The viewing field of at least one other of the plurality of cameras overlaps; and detecting the dropping and removing of the inventory item by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program The multi-joint subject of the program includes: using a plurality of cameras to generate respective sequences of images of a corresponding viewing field in the real space, the viewing field of each camera and the viewing of at least one other camera of the plurality of cameras Domain overlap; use a first image processor, which includes a subject image recognition engine, to process the image to identify the subject represented in the images in the corresponding sequences of the image; use a second image processor, which includes a background An image recognition engine for masking the identified subjects in the images in the sequences of the images to generate masked images, for processing the masked images to identify and classify the images in the corresponding sequences of the images The indicated background changes. For example, a computer program product with a scope of application of item 127, wherein the background image recognition engines include a convolutional neural network. For example, a computer program product with the scope of patent application No. 127 includes associating the identified background change with the identified subject. For example, a computer program product under the scope of application patent No. 127, wherein using the second image processors includes storing background images corresponding to the corresponding sequences of the images; processing the images in the sequences of the images and using the corresponding sequences from the images The background image data of the background images replaces the front stage image data representing the identified subjects to provide the occluded images. For example, a computer program product with the scope of patent application No. 127, wherein processing the images in the sequences of images includes combining a set of N occluded images in the sequences of images to generate a sequence of factorized images for each camera, And the second image processors recognize and classify background changes by processing the sequence of factorized images. For example, a computer program product with the scope of patent application No. 127, wherein using the second image processors includes generating changed data structures for the corresponding sequences of the images, the changed data structures including the masked background changes that have been identified The coordinates in the image, the identifiers of the inventory item subjects with the identified background changes, and the categories of the identified background changes; and processing the changed data structure from a set of cameras with overlapping viewing domains to find out the real space These identified background changes. For example, the computer program product with the scope of patent application No. 132, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image. In the case of a computer program product with a scope of application of item 133, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image; and Including associating the background change with the identified subjects, and performing the detection of the inventory items taken by the identified subjects and the detection of the inventory items being dropped by the identified subjects on the shelf. For example, the computer program product under the scope of application for patent No. 127 includes associating the background change with the identified subjects, and performing the detection of the inventory items taken by the identified subjects and the inventory items dropped by the identified subjects on the shelves. Detection. For example, a computer program product under the scope of patent application No. 127, wherein using the first image processors includes identifying the positions of the hands of the identified subjects; and including comparing the changed positions with the hands of the identified subjects These positions are used to associate the background change with the identified subjects, and to perform the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped on the shelves by the identified subjects. For example, a computer program product with a scope of application of No. 127 includes the use of a third image processor, which includes a front-end image recognition engine that receives corresponding sequences of images from the plurality of cameras, and processes the images to identify and classify the images. Foreground changes represented in the images in the corresponding sequences. For example, a computer program product under the scope of patent application No. 137 includes associating a background change with an identified subject, and performing the detection of the inventory items taken by the identified subjects, and the identified items being put on the shelf by the identified subjects. The first set of inspections; making the front desk change to associate with the identified entities, and performing the second set of inspections of the inventory items taken away by the identified entities and the detection of the inventory items dropped on the shelves by the identified entities; and The detected first and second sets are processed to generate a log data structure including a list of inventory items of identified entities. For example, a computer program product with a scope of application of item 127 includes synchronizing the sequences of images from the cameras in the plurality of cameras. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program By dropping and taking away the inventory items of the subject, the program includes: using a plurality of cameras to generate separate sequences of images of corresponding viewing domains in the real space, the viewing domains of each camera and those of the plurality of cameras The viewing fields of at least one other camera overlap; and detecting the dropping and removing of the inventory items by: semantically identifying significant changes in the inventory items on the inventory display structure and making the semantically significant changes to the images. The subjects represented in these sequences are associated. For example, a computer program product with a scope of 140 patent applications includes detecting the placement and removal of inventory items by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program By dropping and taking away the inventory items of the subject, the program includes: using a plurality of cameras to generate separate sequences of images of corresponding viewing domains in the real space, the viewing domains of each camera and those of the plurality of cameras The viewing fields of at least one other camera overlap; and detecting the dropping and removing of the inventory item by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. A system for tracking a subject dropping and taking inventory items in an area including a real space of an inventory display structure, including: a plurality of cameras, which are arranged on the inventory display structures, and the cameras in the plurality of cameras generate A respective sequence of images of the inventory display structure in a corresponding viewing domain in the real space, the viewing domain of each camera overlapping the viewing domain of at least one other camera among the plurality of cameras; and a processing system, which is Coupled to the plurality of cameras, the processing system includes logic to detect the drop and removal of the inventory item by identifying the posture of the subject and the inventory items related to the postures represented in the sequences of the images. For example, a system with a scope of 143 patent applications, wherein the logic used to detect the drop and removal of inventory items by identifying the posture of the subject and the inventory items related to the postures includes a front-end image recognition engine, which is processed by The foreground data in the sequences of the images are used to identify the posture, and further include logic to detect the drop and removal of the inventory items by: semantically identifying the inventory on the inventory display structure that includes the background image recognition engine A significant change in the item, the background image recognition engine recognizes the change by processing background data in the sequences of the image. A system for tracking changes in an area of real space, including: a plurality of cameras, the cameras in the plurality of cameras generating respective sequences of images of corresponding viewing fields in the real space, and the viewing fields of each camera Overlaps with the viewing field of at least one other camera of the plurality of cameras; a processing system coupled to the plurality of cameras, the processing system includes: a first image processor including a subject image recognition engine that receives a plurality of images from a plurality of cameras; The corresponding sequence of camera images is to process the images to identify the subjects represented in those images in the corresponding sequence of images; a second image processor that includes a background image recognition engine that receives the corresponding images from multiple cameras A sequence that masks the identified subjects to generate a masked image, processes the masked images to identify and classify the background changes represented in the images in the corresponding sequences of images; and a third image processor , Which includes a front-end image recognition engine that receives The plurality of complex images corresponding sequence of the camera, an image processing system to change its reception at the plurality of corresponding sequence identification and classification of images in the plurality of images indicated. For example, the system for applying item No. 145, wherein the front-end image recognition engines and the background image recognition engines include a convolutional neural network. For example, the system for applying for patent scope No. 145 includes logic to associate the identified background change and the identified foreground change with the identified subject. For example, the system for applying item No. 145, wherein the second image processors include: background image storage for storing background images of the corresponding sequences of the image; mask logic for processing the sequences of the image And replace the background image data representing the identified subjects with the background image data of the background images from the corresponding sequences of the images to provide the occluded images. For example, the system of claim 148, wherein the masking logic is a combination of N masked images in the sequences of images to generate a sequence of factorized images for each camera, and the second image processor This sequence of factorized images is processed to identify and classify background changes. For example, the system of claim 145, wherein the second image processors include logic for generating changed data structures for the corresponding sequences of the images, and the changed data structures include the identified background changes. The coordinates in the occluded image, the identifiers of the inventory item subjects of the identified background changes, and the categories of the identified background changes; and coordination logic to process the changed data structure from the collection of cameras with overlapping viewing fields to Find out the identified background changes in real space. For example, in the system of applying for the scope of patent No. 150, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image. If the system of applying for patent scope item 150, wherein the categories of the identified background change in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image; and includes logic For associating the background change with the identified entities, and for performing the detection of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure. For example, the system for applying for item No. 145 includes: logic for associating background changes and identified front desk changes with identified subjects, and for performing inspections by the identified subjects to remove inventory items, and The inspection of the identified entity dropping the inventory item on the inventory display structure. For example, the system of claim 145, wherein the first image processors identify the positions of the hands of the identified subjects; and include: logic for comparing the positions of the changes with the positions of the identified subjects The positions of the hands are used to associate the background change with the identified subjects, and to perform the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the inventory display structure. For example, the system for applying for item No. 145 includes logic for associating a background change with an identified subject, and for performing inspections of items removed by the identified subjects and dropping inventory items by the identified subjects. The first set of inspections on the inventory display structure; logic for associating front desk changes with identified entities, and for performing inspections of inventory items taken by the identified entities and dropping inventory by the identified entities The second set of inspections of items on the inventory display structure; Selection logic for processing the first and second sets of inspections to generate a log data structure that includes a list of inventory items for identified entities. For example, the system of claim No. 145, wherein the sequences of the images from the cameras in the plurality of cameras are synchronized. A method for tracking down and removing inventory items from a subject in an area of real space, including: using a plurality of cameras, which are arranged on the inventory display structures to generate corresponding viewing areas in the real space The respective sequences of the images of the inventory display structure, the viewing fields of each camera and the viewing fields of at least one other camera of the plurality of cameras overlap; identify the subject by processing foreground data in the sequences of the images Postures and inventory items related to those postures are used to detect the drop and removal of inventory items. For example, the method of applying for item 157 of the patent scope includes: (i) semantically identifying significant changes in the inventory items on the inventory display structure by processing background information in the sequences of the images to detect the drop and removal of the inventory items. For example, the method of claiming scope of patent application 158 includes using a first image processor including a subject image recognition engine to process the image to identify the subject represented in the images in the corresponding sequences of the image; ； wherein the Detecting the drop and removal of inventory items by semantically identifying significant changes in the inventory items includes the use of a second image processor, which includes a background image recognition engine to mask the identified subjects in the images in those sequences of images To generate a masked image, to process the masked image to identify and classify the background changes represented in the images in the corresponding sequences of the images; and to identify the subject ’s pose and the poses Relevant inventory items to detect inventory items to detect the dropping and removal of inventory items include the use of a third image processor, which includes a front-end image recognition engine that receives corresponding sequences of images from the plurality of cameras for processing the images to identify And classify the foreground changes represented in the images in the corresponding sequences of the images . For example, the method in the 159th patent application range, wherein the background image recognition engines and the foreground image recognition engines include a convolutional neural network. For example, the method of applying for item 159 of the patent scope includes associating the identified background change and foreground change with the identified subject. For example, the method of claiming the scope of patent application No. 159, wherein the second image processors include storing background images for corresponding sequences of the images; processing the images in the sequences of the images and using the corresponding sequences from the images The background image data of the background image replaces the front stage image data representing the identified subjects to provide the occluded images. For example, the method of claim 162, wherein processing the images in the sequences of images includes combining a set of N occluded images in the sequences of images to generate a sequence of factorized images for each camera, and the These second image processors process the sequence of factorized images to identify and classify background changes. For example, the method of claiming range 162 of the patent application, wherein using the second image processors includes generating changed data structures for the corresponding sequences of the images, and the changed data structures include the masked images whose identified background changes. , The identifiers of the inventory item subjects of the identified background changes, and the categories of the identified background changes; and processing the changed data structure from the set of cameras with overlapping viewing domains to find those in real space A background change has been identified. For example, in the method of applying for the scope of patent No. 164, the categories of the identified background changes in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image. If the method of applying for the scope of patents No. 164, wherein the categories of the identified background change in the changed data structure indicate whether the identified inventory item has been added or removed relative to the background image; and includes The background change is associated with the identified entities, and the detection of the inventory items taken away by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure are performed. For example, the method of claiming scope 159, wherein using the first image processors includes identifying positions of the identified subject's hand; and including comparing the changed positions with the identified subject's hand. Position to associate the background change with the identified subject, and perform the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the inventory display structure. For example, the method of applying for item 159 of the patent scope includes associating a background change with an identified entity, and performing the detection of the inventory items taken by the identified entities, and the identification of the inventory items on the inventory display structure by the identified entities. The first set of inspections; Associate the front desk to change with the identified entities, and perform the second set of inspections of the inventory items taken by the identified entities and the detection of the inventory items dropped by the identified entities on the inventory display structure And processing the first and second sets of inspections to produce a log data structure including a list of inventory items for identified entities. The method of claim 157 includes synchronizing the sequences of images from the cameras in the plurality of cameras. A computer program product comprising: computer readable memory, which contains non-transitory data storage media; 之 computer instructions stored in the memory, which can be executed by a computer to track a region of real space by a program The multi-joint subject of the program includes: using a sequence of images from a corresponding viewing field in the real space of a plurality of cameras, the viewing field of each camera overlapping the viewing field of at least one other camera of the plurality of cameras ; Use a first image processor, which includes a subject image recognition engine, which processes the image to identify the subjects represented in the images in the corresponding sequences of the images; use a second image processor, which includes background image recognition An engine for masking the identified subjects in the images in the sequences of the images to generate masked images, for processing the masked images to identify and classify the representations in the images in the corresponding sequences of the images Background changes; and using a third image processor that includes a foreground image Recognition engine, receives the corresponding sequence from a plurality of images of the plurality of cameras, for processing an image corresponding to the plurality of SEQ ID foreground and classification of the plurality of video images represented in the image changed. For example, a computer program product with a scope of application of item 170, wherein the use of the first image processors includes identifying the positions of the hands of the identified subjects; and including comparing the positions of the changes with the hands of the identified subjects These positions are used to associate the background change with the identified subject, and to perform the detection of the inventory items taken by the identified subjects and the detection of the inventory items dropped by the identified subjects on the inventory display structure. For example, a computer program product under the scope of application for patent No. 170 includes associating a background change with an identified subject, and performing inspections of the inventory items taken by the identified subjects and dropping the inventory items by the identified subjects in the inventory display structure. The first set of inspections on the market; Associate the front desk to change with the identified entities, and perform the inspection of the inventory items taken by the identified entities and the inspection of the inventory items on the inventory display structure by the identified entities. Two sets; and processing the first and second sets of inspections to produce a log data structure that includes a list of inventory items for identified entities. A method for training a neural network to detect the drop and removal of inventory items by a subject in an area of real space, including: using a plurality of cameras to generate individual sequences of images of script actors, the script actors performing Poses with inventory items in corresponding viewing domains in the real space; and use the sequences of images of script actors to train neural networks to identify the pose of the subject and the representations in the sequences with the images These posture-related inventory items are used to detect the drop and removal of inventory items. A method for tracking down and removing inventory items of a subject in an area of real space, including: using an image recognition engine to identify the subject and the inventory items in the area of the real space represented in the sequences by the subject Drop and remove; and display a graphic that shows a map of subjects in that area of real space, including colored images that represent individual subjects, and includes assigning colors based on the status of those individual subjects Give those colored images. A method for tracking down and removing inventory items of a subject in an area of real space, including: using an image recognition engine to identify the subject and the inventory items in the area of the real space represented in the sequences by the subject Drop and remove; and assign those inventory items to individual subjects; display a graphic that is a map of the subject in the area showing real space, including its colored image representing the individual subject, and includes The level of confidence that their inventory items assigned to the respective subjects are correctly identified to assign colors to the colored images. A method for a subject to drop and take inventory items in an area that tracks real space in a store with store inventory, including: using an image recognition engine to identify the real space represented in the sequences of the subject and the image of the area The inventory items of the entity are dropped and taken away; and the inventory items are assigned to individual entities; and the measured drop and taken are used to generate an inventory of the store. A method for detecting a directional impression of a subject in a region of real space, comprising: using an image processor to process a sequence of images to identify the subject in the region, and to determine the direction of gaze of the identified subject.