CN110969112B - Pedestrian identity alignment method under camera-crossing scene - Google Patents

Abstract

The invention proposes a pedestrian identity alignment method in a cross-camera scene, which solves the pedestrian association problem under the cross-camera, and is based on the continuous and correct tracking of the target pedestrian under a single camera. After the target leaves the camera field of view, Entering the blind spot, the tracking is intermittent, but when it reappears under the camera, it can re-identify the pedestrian and keep its identity unchanged and continue tracking. When the detector detects a new pedestrian, it is added to the candidate pool to be associated, and then a pair of pedestrians are selected from the candidate pool to complete the image preprocessing using the F‑CCT model, and the processed image will be used as the SAM‑Dets model. Enter the data and find the appearance fit of the two pedestrians. After the pedestrians in the candidate pool are paired to calculate the appearance fit, the minimum cost flow graph model is established according to the fit of each pair of pedestrians and combined with the space-time relationship to solve the optimal pedestrian correlation solution; finally, the pedestrians are kept original according to the correlation results. Identify or assign a new identity.

Description

Alignment of pedestrian identities across camera scenes

technical field

The invention belongs to the fields of machine vision and intelligent security, and in particular relates to a pedestrian identity alignment method in a cross-camera scene.

Background technique

With the continuous development of economy and society, people's demand for safety is increasing. Therefore, the field of intelligent security is constantly being discussed and developed, and the scope of application is getting larger and larger, and technologies such as pedestrian detection and tracking have become a hot issue in current research. For the pedestrian tracking problem, it is naturally the research under the cross-camera that has practical significance, and requires the processing of multiple pedestrians. Although the pedestrian tracking technology in single-camera scenarios is relatively mature, the presence of blind spots in multi-cameras, especially in non-overlapping fields of view, makes the spatiotemporal information of the target unreliable. The identification, tracking and retrieval of the same target has caused great trouble. Therefore, research on pedestrian tracking technology in cross-camera scenarios is emerging. Among them, the most important part is how to match pedestrian identities under different cameras.

Cross-camera pedestrian identity alignment mainly focuses on pedestrians and focuses on multi-camera and multi-target tracking problems in non-overlapping visual fields. The current common solution mechanism for this problem is divided into two steps: First, use the detection and tracking algorithm to obtain the running trajectory of the target under a single camera. Secondly, use the association algorithm to associate and integrate the independent pedestrian running trajectories between cameras, so as to obtain the complete motion trajectory of each target. The limitation of the above mechanism is that it can only process offline data, which is essentially suitable for retrieval scenarios and cannot support online tracking. The reason is that after the target pedestrian leaves the field of view of the current camera, due to the existence of the blind spot, when the target enters the field of view of the next camera, the spatiotemporal information is missing, which increases the difficulty of correctly handing over the target pedestrian from the previous camera to the next camera. . This mechanism also has a side effect: the results of cross-camera pedestrian tracking rely heavily on the pedestrian tracking effect under a single camera.

The key to achieving cross-camera identity alignment of pedestrians is to correctly associate the same target pedestrians in different fields of view. For most of the current cross-camera pedestrian tracking algorithms, the learning ability of pedestrian features is limited, and more robust pedestrian features cannot be learned. Therefore, the accuracy of the following pedestrian similarity measurement is finally affected, resulting in an unsatisfactory data association result. Therefore, it is difficult to adapt to the complex environment of cross-camera pedestrian tracking.

Although the existing research related to cross-camera identity alignment can effectively solve some pedestrian tracking for offline data, it cannot meet the requirements for real-time online tracking, and it cannot effectively track unknown pedestrians entering and leaving the area. .

SUMMARY OF THE INVENTION

In order to overcome the gaps and deficiencies in the prior art, the solution of the present invention is to solve the pedestrian association problem under cross-cameras. The target is given a new identity. It is based on the continuous and correct tracking of the target pedestrian under a single camera. After the target leaves the camera's field of view, it enters the blind spot and the tracking is intermittent, but when it reappears under the camera, the pedestrian can be re-identified and maintained. Its identity does not change and continues tracking. When the detector detects a new pedestrian, it is added to the candidate pool to be associated, and then a pair of pedestrians are selected from the candidate pool to complete the image preprocessing using the F-CCT model, and the processed image will be used as the SAM-Dets model. Enter the data and find the appearance fit of the two pedestrians. After the pedestrians in the candidate pool are paired to calculate the appearance fit, the minimum cost flow graph model is established according to the fit of each pair of pedestrians and combined with the space-time relationship to solve the optimal pedestrian correlation solution; finally, the pedestrians are kept original according to the correlation results. Identify or assign a new identity to the tracker to continue tracking.

The present invention specifically adopts the following technical solutions:

A method for aligning pedestrian identities across camera scenes, comprising the following steps:

Step S1: each of the plurality of cameras adds the pedestrian images detected by the detector to the candidate pool to be associated; Step S2: calculates the appearance fit of two pedestrian images belonging to different cameras in the candidate pool to be associated;

Step S3: After the pedestrian images in the candidate pool to be associated are paired to calculate the appearance fit degree, according to the appearance fit degree of each pair of pedestrians, combined with the spatiotemporal relationship, a minimum cost flow graph model is established to solve the optimal pedestrian association untie;

Step S4: According to the association result of step S3, the pedestrian is maintained with the original identification or given a new identification.

Preferably, in step S1, the detector is Faster R-CNN.

Preferably, in step S2, calculating the degree of appearance fit specifically includes the following steps:

Step A21: Use the fuzzy C-means clustering F-CCT model to complete image preprocessing, and set the overall feature of pedestrian image A to be X={x ₁ , x ₂ ,...,x _N }, and the overall feature of pedestrian image B The characteristic is Y={y ₁ , y ₂ ,...,y _N };

Step A22: The image processed in step S21 is used as the input data of the pedestrian association model SAM-Dets fused with fine-grained representation: X is the input vector, Y is the weight vector, and the pedestrian association model SAM-Dets fused with fine-grained representation is used as the input vector. -Dets encode the local fine-grained feature f ₁ of pedestrian A; take Y as the input vector and X as the weight vector, encode the local fine-grained feature f of pedestrian B by the pedestrian association model SAM-Dets fused with fine-grained representation ₂ ;

Step A23: Take f _s =(f ₁ -f ₂ ) ² as the input value of two convolutional layers C with kernel size of 1×1×4096, take softmax as the output function, and output a two-dimensional vector (q ₁ , q ₂ ), representing the probability value that the input two objects belong to the same person in the real world, as the appearance fit degree.

Preferably, in step A22, the structure of the pedestrian association model SAM-Dets fused with fine-grained representation includes: K attention branches and splicing layers; each of the attention branches includes the following six layers, wherein:

The first layer is the convolutional layer A, which is used to extract the high-level features of the overall pedestrian features of the input;

The second layer is the activation layer, and the activation function is softmax;

The third layer is the dimension expansion layer;

The fourth layer is the summation layer, which adds the overall pedestrian characteristics to the results obtained in the third layer;

The fifth layer is the global average pooling layer, which is used to reduce the feature dimension;

The sixth layer is the fully connected layer, which is used to complete the inner product calculation of the input vector and the weight vector in the weight matrix;

The splicing layer splices the results obtained by the K attention branches by channel, and outputs the local fine-grained features of pedestrians.

Preferably, the size of the convolution kernel of the convolutional layer A is 1×1, and the stride is 1; the dimension expansion layer expands the channel dimension to 512 dimensions; the size of the global average pooling layer is 1×1 , with a step size of 1.

Preferably, in step S2, calculating the degree of appearance fit specifically includes the following steps:

Step B21: Use the fuzzy C-means clustering F-CCT model to complete image preprocessing, set the overall feature of pedestrian image A as X={x ₁ ,x ₂ ,...,x _N }, the overall feature of pedestrian image B The characteristic is Y={y ₁ , y ₂ ,...,y _N };

Step B22: Extract the pedestrian abstract features from the pedestrian image A and the pedestrian image B through the DR-ResNet basic network;

Step B23: Use the convolutional layer B to further extract the high-level features of the target pedestrian as the input data of the classification model and the pedestrian association model SAM-Dets fused with fine-grained representation; the classification model outputs the respective identities of the pedestrian image A and the pedestrian image B respectively Identifying the representation number, the pedestrian association model SAM-Dets fused with fine-grained representation outputs the local fine-grained feature f ₁ possessed by pedestrian A and the local fine-grained feature f ₂ possessed by pedestrian B.

Step B24: take f _s =(f ₁ -f ₂ ) ² as the input value of the two convolutional layers C with a kernel size of 1×1×4096, take softmax as the output function, and output a two-dimensional vector (q ₁ , q ₂ ), representing the probability value that the input two objects belong to the same person in the real world, as the appearance fit degree.

Preferably, the DR-ResNet basic network includes two identical deep convolutional Siamese neural basic network modules R-ResNet with shared weights; the structure of the deep convolutional Siamese neural basic network module R-ResNet includes forty-nine layer convolutional layers, three parallel convolutional layers, and end convolutional layers:

Among them, the convolution kernel size of the first convolution layer is (7, 7, 64), the max-pooling is (3, 3), and the sliding step size is 2;

The convolution kernel sizes from the second convolutional layer to the fourth convolutional layer are (1,1,64), (3,3,64), (1,1,256) respectively, and the activation functions all use the ReLu function; the three layers The convolution layer and the activation function form a convolution block, and the input value of the convolution block is used both as the input value of the second convolution layer and the input value of the activation function of the third layer of the convolution block; the fifth convolution Layers to the seventh convolutional layer, and the eighth convolutional layer to the tenth convolutional layer all use the same structure as the second to the fourth convolutional layer;

The convolution kernel sizes of the eleventh convolutional layer to the thirteenth convolutional layer are (1, 1, 128), (3, 3, 128), (1, 1, 512), and the activation functions all use the ReLu function ; The three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as the input value of the eleventh convolution layer and the input of the activation function of the third layer of the convolution block. value; the fourteenth convolutional layer to the sixteenth convolutional layer, the seventeenth convolutional layer to the nineteenth convolutional layer, and the twentieth convolutional layer to the twenty-second convolutional layer all adopt the same value as the tenth convolutional layer. The same structure from a convolutional layer to the thirteenth convolutional layer;

The convolution kernel sizes of the twenty-third convolutional layer to the twenty-fifth convolutional layer are (1, 1, 256), (3, 3, 256), (1, 1, 1024), respectively, and the activation functions are all used ReLu function; the three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as the input value of the twenty-third convolution layer and the activation function of the third layer of the convolution block. The input value of ; the twenty-sixth convolutional layer to the twenty-eighth convolutional layer, the twenty-ninth convolutional layer to the thirty-first convolutional layer, the thirty-second convolutional layer to the thirty-fourth convolutional layer , the thirty-fifth convolutional layer to the thirty-seventh convolutional layer, and the thirty-eighth convolutional layer to the fortieth convolutional layer all adopt the same structure as the twenty-third convolutional layer to the twenty-fifth convolutional layer ;

The convolution kernel sizes of the forty-first convolutional layer to the forty-third convolutional layer are (1, 1, 512), (3, 3, 512), (1, 1, 2048), respectively, and the activation functions are all used ReLu function; the three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as both the input value of the 41st convolution layer and the activation function of the third layer of the convolution block. The input value of ; the forty-fourth convolutional layer to the forty-sixth convolutional layer, the forty-seventh convolutional layer to the forty-ninth convolutional layer are convolutional with the forty-first convolutional layer to the forty-second convolutional layer layers of the same structure;

After the forty-ninth convolutional layer, there are three parallel convolutional layers. Each convolutional layer uses 2048 convolution kernels. The sizes of the first parallel convolutional layer to the third parallel convolutional layer are ( 3, 3, 1024), (5, 5, 1024) and (7, 7, 1024), the channels of the three parallel convolutional layers are merged through a connection layer, and the subsequent max-pooling is (4, 4);

The last layer is an end convolutional layer with 1024 convolution kernels and a size of (2, 2, 2048);

The convolutional layer B uses 2 convolution kernels with a size of (1, 1, 4096);

The structure of the pedestrian association model SAM-Dets fused with fine-grained representation includes: K attention branches and splicing layers; each of the attention branches includes the following six layers, wherein:

The first layer is the convolution layer A, the size of the convolution kernel is 1×1, and the stride is 1; it is used to extract the high-level features of the input pedestrian overall features;

The second layer is the activation layer, and the activation function is softmax;

The third layer is the dimension expansion layer, which expands the channel dimension to 512 dimensions;

The fourth layer is the summation layer, which adds the overall pedestrian characteristics to the results obtained in the third layer;

The fifth layer is a global average pooling layer with a size of 1×1 and a step size of 1, which is used to reduce the feature dimension;

The sixth layer is the fully connected layer, which is used to complete the inner product calculation of the input vector and the weight vector in the weight matrix;

The splicing layer splices the results obtained by the K attention branches by channel, and outputs the local fine-grained features of pedestrians.

Preferably, in step S3, according to the appearance fit of each pair of pedestrians, combined with the space-time relationship, a minimum cost flow graph model is established, and the specific process of solving the optimal pedestrian association solution includes the following steps:

则当t_p时刻，为视野内行人集

和出视野行人集

中每一个目标新增进出两个节点，更新新增节点与源点、汇点间有向边连接；Step S31: Set the cost flow graph after the real-time alignment is completed at a given time t _p -1 as

Then at time t _p , it is the set of pedestrians in the field of view

and out of sight pedestrian set

In each target, two new nodes are added, and the new node is updated with the directed edge connection between the source node and the sink node;

和出视野行人集

中两两目标间的行人外观适配度，更新相应节点间有向边，得到t_p时刻新的费用流图

Step S32: According to the set of pedestrians in the field of view

and out of sight pedestrian set

The pedestrian appearance adaptation degree between the two targets in the middle, update the directed edges between the corresponding nodes, and obtain a new cost flow graph at time t _p

剩余未对齐的目标节点，并将视野内行人集

剩余未对齐的目标节点为新进入的目标行人，得到费用流图

等待下一时刻更新对齐。Step S33: Delete all alignment target nodes and pedestrian sets in the field of view

Remaining unaligned target nodes and set pedestrians within the field of view

The remaining unaligned target nodes are the newly entered target pedestrians, and the cost flow graph is obtained

Wait for the next moment to update the alignment.

Preferably, in step S4, after the operation of maintaining the original identification or giving a new identification identification to the pedestrian, the tracker is handed over to the tracker to continue tracking; the tracker adopts the KCF algorithm to track; the KCF algorithm is for each Pedestrians are assigned a tracker.

Preferably, the tracking visual field area of the camera is divided into a core area and a critical area, and in step S1, only pedestrians in the critical area are detected by the detector.

Compared with the prior art, the present invention and its preferred solution realize online tracking of pedestrians across cameras, with accurate identification and high efficiency, and are not affected by intermittent tracking after the target leaves the field of view of the camera and enters the blind spot.

Among them, the realization of the tracking function is realized by the existing mature FasterR-CNN to realize the pedestrian detection, and the KCF algorithm is used to realize the online pedestrian tracking; and the core of the realization of the present invention and the preferred solution is to establish the minimum cost flow graph model according to the space-time information and the pedestrian similarity value. The task of pedestrian identity alignment is instantly completed, and the F-CCT and SAM-Dets models or the fusion of the SAM-Dets model and the DR-ResNet network model are integrated to solve the problem of pedestrian appearance fitness measurement.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments:

1 is a schematic diagram of the overall flow of Embodiment 1 of the present invention;

2 is a schematic structural diagram of a SAM-Dets model according to an embodiment of the present invention;

3 is a schematic diagram of a network structure of a SAM-Dets model according to an embodiment of the present invention;

4 is a schematic diagram of a fine-grained association model for pedestrians in Embodiment 2 of the present invention;

5 is a schematic diagram 1 of the effect of a pedestrian identity alignment model in a cross-camera scene according to Embodiment 2 of the present invention;

FIG. 6 is a schematic diagram 2 of the effect of a pedestrian identity alignment model in a cross-camera scene according to Embodiment 2 of the present invention.

Detailed ways

In order to make the features and advantages of this patent more obvious and easy to understand, two embodiments are given below, which are described in detail as follows:

As shown in FIG. 1, in the first embodiment of the present invention, the overall solution for realizing pedestrian identity alignment across camera scenes includes the following steps:

Step S1: When the pedestrian enters the field of view of the camera, each of the plurality of cameras adds the pedestrian image detected by the detector to the candidate pool to be associated.

In this embodiment, the detector selects Faster R-CNN, a representative method of target detection based on deep learning, which uses RPN when selecting candidate frames to detect the convolutional features of the entire image shared by the network, so that the classification and regression tasks have the same Convolutional features.

Step S2: Calculate the appearance fit of two pedestrian images belonging to different cameras in the candidate pool to be associated.

In an embodiment, calculating the appearance fit specifically includes the following steps:

Step A21: Use the fuzzy C-means clustering F-CCT model to complete image preprocessing, and set the overall feature of pedestrian image A to be X={x ₁ , x ₂ ,...,x _N }, and the overall feature of pedestrian image B The feature is Y={y ₁ , y ₂ ,...,y _N }; use fuzzy clustering algorithm to divide the clustering domain of the image, and achieve localization between the clustering domains by matching the clustering domains of the source and target images. Color brightness migration, and the introduction of membership factor to improve the effect of color brightness migration.

Step A22: The image processed in step S21 is used as the input data of the pedestrian association model SAM-Dets fused with fine-grained representation: X is the input vector, Y is the weight vector, and the pedestrian association model SAM-Dets is fused with fine-grained representation. Encode the local fine-grained feature f ₁ of pedestrian A; take Y as the input vector and X as the weight vector, encode the local fine-grained feature f ₂ of pedestrian B through the pedestrian association model SAM-Dets fused with fine-grained representation.

As shown in Figure 2, in this embodiment, the pedestrian association model SAM-Dets fused with fine-grained representation is composed of multiple attentions. Each branch in the model has the same functional module, and the input data is extracted by the basic network. Pedestrian global features. Each attention branch goes through the three modules of local detector, global pooling and linear embedding at a time, and finally the results of the K branches are spliced by channel to obtain the complete output of the attention model.

In the local detector module, first obtain the high-level features of the input information, and use the softmax function to normalize the high-level features to obtain the attention weights that fit the value range of the probability distribution. The dimensions are enlarged, and finally, the weighted sum function is used to obtain the probability distribution of attention distribution of the corresponding features. In the global pooling module and the linear embedding module, the attention distribution probability distribution filters and retains the corresponding pedestrian features, and finally outputs the high-level pedestrian features with the attention distribution.

As shown in Figure 3, specifically, the network structure of the pedestrian association model SAM-Dets fused with fine-grained representation includes: K attention branches and splicing layers; each attention branch includes the following six layers, where:

The first layer is the convolutional layer A, which is used to extract the high-level features of the overall pedestrian features of the input;

The second layer is the activation layer, and the activation function is softmax;

The third layer is the dimension expansion layer;

The fourth layer is the summation layer, which adds the overall pedestrian characteristics to the results obtained in the third layer;

The fifth layer is the global average pooling layer, which is used to reduce the feature dimension;

The sixth layer is the fully connected layer, which is used to complete the inner product calculation of the input vector and the weight vector in the weight matrix;

The splicing layer concatenates the results obtained by the K attention branches by channel, and outputs the local fine-grained features of pedestrians.

The size of the convolution kernel of convolutional layer A is 1×1 and the stride is 1; the dimension expansion layer expands the channel dimension to 512 dimensions; the size of the global average pooling layer is 1×1 and the stride is 1.

Step A23: Convert the similarity value of a pair of pedestrians input to be calculated into a similarity comparison of f ₁ and f ₂ features. A non-parameter layer Square layer is introduced to solve the square difference of f ₁ and f ₂ features, as the similarity comparison layer of f ₁ and f ₂ , and the Square layer is recorded as: f _s =(f ₁ -f ₂ ) ² ; f _s =(f ₁ -f ₂ ) ² is used as the input value of two convolutional layers C with kernel size of 1×1×4096, and softmax is used as the output function to output a two-dimensional vector (q ₁ ,q ₂ ), Represents the probability value that the input two objects belong to the same person in the real world, as the appearance fit.

Further, according to the obtained similarity probability value between a pair of pedestrians as the weight of the graph, the newly entered pedestrian and the target pedestrian to be associated are regarded as two different sets of vertices, respectively, to establish a matching graph with weights; by solving The solution of the maximum weight matching graph problem is obtained, and the solution of the data association between the newly entered pedestrian and the target pedestrian waiting to be associated is obtained.

Step S3: After the pedestrian images in the candidate pool to be associated are paired to calculate the appearance fit degree, according to the appearance fit degree of each pair of pedestrians, combined with the spatiotemporal relationship, a minimum cost flow graph model is established to solve the optimal pedestrian association untie.

It specifically includes the following steps:

则当t_p时刻，为视野内行人集

和出视野行人集

中每一个目标新增进出两个节点，更新新增节点与源点、汇点间有向边连接；Step S31: Set the cost flow graph after the real-time alignment is completed at a given time t _p -1 as

Then at time t _p , it is the set of pedestrians in the field of view

and out of sight pedestrian set

In each target, two new nodes are added, and the new node is updated with the directed edge connection between the source node and the sink node;

和出视野行人集

中两两目标间的行人外观适配度，更新相应节点间有向边，得到t_p时刻新的费用流图

Step S32: According to the set of pedestrians in the field of view

and out of sight pedestrian set

The pedestrian appearance adaptation degree between the two targets in the middle, update the directed edges between the corresponding nodes, and obtain a new cost flow graph at time t _p

剩余未对齐的目标节点，并将视野内行人集

剩余未对齐的目标节点为新进入的目标行人，得到费用流图

等待下一时刻更新对齐。Step S33: Delete all alignment target nodes and pedestrian sets in the field of view

Remaining unaligned target nodes and set pedestrians within the field of view

The remaining unaligned target nodes are the newly entered target pedestrians, and the cost flow graph is obtained

Wait for the next moment to update the alignment.

Step S4: According to the association result of Step S3, keep the original identification or assign a new identification to the pedestrian, and hand it over to the tracker to continue tracking; the tracker adopts the KCF algorithm to track; the KCF algorithm assigns each pedestrian a tracker. The algorithm forms a circulant matrix in the target area, and then uses some properties such as the circulant matrix can be diagonalized in Fourier space, and obtains a general prediction formula through regression ridge regression.

Meanwhile, in this embodiment, the tracking field of view area of the camera is divided into a core area and a critical area, and in step S1, only pedestrians in the critical area are detected by the detector. This is to consider that the entry and exit of any target must first pass through the critical area. Therefore, only the pedestrians in the critical area have a reasonable spatial transfer relationship to ensure the universality of the subsequent pedestrian alignment solution to the greatest extent.

In the second embodiment of the present invention, as shown in FIG. 4 , it provides step S2: another preferred implementation of calculating the appearance adaptation degree of two pedestrian images belonging to different cameras in the candidate pool to be associated The scheme specifically includes the following steps:

Step B21: Use the fuzzy C-means clustering F-CCT model to complete image preprocessing, set the overall feature of pedestrian image A as X={x ₁ ,x ₂ ,...,x _N }, the overall feature of pedestrian image B The characteristic is Y={y ₁ , y ₂ ,...,y _N };

Step B22: Extract the pedestrian abstract features from the pedestrian image A and the pedestrian image B through the DR-ResNet basic network;

Step B23: Use the convolutional layer B to further extract the high-level features of the target pedestrian as the input data of the classification model and the pedestrian association model SAM-Dets fused with fine-grained representation; the classification model outputs the respective identification representations of the pedestrian image A and the pedestrian image B respectively , the pedestrian association model SAM-Dets fused with fine-grained representation outputs the local fine-grained feature f ₁ of pedestrian A and the local fine-grained feature f ₂ of pedestrian B.

Step B24: take f _s =(f ₁ -f ₂ ) ² as the input value of the two convolutional layers C with a kernel size of 1×1×4096, take softmax as the output function, and output a two-dimensional vector (q ₁ , q ₂ ), representing the probability value that the input two objects belong to the same person in the real world, as the appearance fit degree.

Among them, the DR-ResNet basic network includes two identical deep convolutional Siamese neural basic network modules R-ResNet with shared weights; the structure of the deep convolutional Siamese neural basic network module R-ResNet includes forty-nine convolutional layers, Three parallel convolutional layers, and end convolutional layers:

Among them, the convolution kernel size of the first convolution layer is (7, 7, 64), the max-pooling is (3, 3), and the sliding step size is 2;

The convolution kernel sizes from the second convolutional layer to the fourth convolutional layer are (1,1,64), (3,3,64), (1,1,256) respectively, and the activation functions all use the ReLu function; the three layers The convolution layer and the activation function form a convolution block, and the input value of the convolution block is used both as the input value of the second convolution layer and the input value of the activation function of the third layer of the convolution block; the fifth convolution Layers to the seventh convolutional layer, and the eighth convolutional layer to the tenth convolutional layer all use the same structure as the second to the fourth convolutional layer;

The convolution kernel sizes of the eleventh convolutional layer to the thirteenth convolutional layer are (1, 1, 128), (3, 3, 128), (1, 1, 512), and the activation functions all use the ReLu function ; The three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as the input value of the eleventh convolution layer and the input of the activation function of the third layer of the convolution block. value; the fourteenth convolutional layer to the sixteenth convolutional layer, the seventeenth convolutional layer to the nineteenth convolutional layer, and the twentieth convolutional layer to the twenty-second convolutional layer all adopt the same value as the tenth convolutional layer. The same structure from a convolutional layer to the thirteenth convolutional layer;

The convolution kernel sizes of the twenty-third convolutional layer to the twenty-fifth convolutional layer are (1, 1, 256), (3, 3, 256), (1, 1, 1024), respectively, and the activation functions are all used ReLu function; the three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as the input value of the twenty-third convolution layer and the activation function of the third layer of the convolution block. The input value of ; the twenty-sixth convolutional layer to the twenty-eighth convolutional layer, the twenty-ninth convolutional layer to the thirty-first convolutional layer, the thirty-second convolutional layer to the thirty-fourth convolutional layer , the thirty-fifth convolutional layer to the thirty-seventh convolutional layer, and the thirty-eighth convolutional layer to the fortieth convolutional layer all adopt the same structure as the twenty-third convolutional layer to the twenty-fifth convolutional layer ;

The convolution kernel sizes of the forty-first convolutional layer to the forty-third convolutional layer are (1, 1, 512), (3, 3, 512), (1, 1, 2048), respectively, and the activation functions are all used ReLu function; the three-layer convolution layer and the activation function form a convolution block, and the input value of the convolution block is used as both the input value of the 41st convolution layer and the activation function of the third layer of the convolution block. The input value of ; the forty-fourth convolutional layer to the forty-sixth convolutional layer, the forty-seventh convolutional layer to the forty-ninth convolutional layer are convolutional with the forty-first convolutional layer to the forty-second convolutional layer layers of the same structure;

After the forty-ninth convolutional layer, there are three parallel convolutional layers. Each convolutional layer uses 2048 convolution kernels. The sizes of the first parallel convolutional layer to the third parallel convolutional layer are ( 3, 3, 1024), (5, 5, 1024) and (7, 7, 1024), the channels of the three parallel convolutional layers are merged through a connection layer, and the subsequent max-pooling is (4, 4);

The last layer is an end convolutional layer with 1024 convolution kernels and a size of (2, 2, 2048);

Convolutional layer B uses 2 convolution kernels of size (1, 1, 4096).

As shown in FIG. 5 and FIG. 6 , the solution of this embodiment and the existing offline cross-camera tracking solution both achieve the performance of cross-camera tracking. The difference is that the solution of this embodiment can directly realize online tracking. .

This patent is not limited to the above-mentioned best embodiment, anyone can come up with other various forms of pedestrian identity alignment methods in cross-camera scenes under the inspiration of this patent. Modifications should all fall within the scope of this patent.

Claims (1)

1. A pedestrian identity alignment method under a camera-crossing scene is characterized by comprising the following steps:

step S1: the cameras respectively add the pedestrian images detected by the cameras through the detector into a candidate pool to be associated;

step S2: calculating the appearance adaptation degree of two pedestrian images belonging to different cameras in a candidate pool to be associated;

step S3: after the pedestrian images in the candidate pool to be associated are pairwise paired and the appearance adaptation degrees are calculated, according to the appearance adaptation degrees of each pair of pedestrians, a minimum cost flow diagram model is established by combining the space-time relationship, and the optimal pedestrian association solution is solved;

step S4: according to the correlation result of the step S3, the operation of keeping the original identification or giving a new identification is carried out on the pedestrian;

in step S1, the detector is Faster R-CNN;

in step S2, the specific steps of calculating the degree of appearance suitability are step a 21-step a23, or step B21-step B24:

wherein the step A21-the step A23 specifically comprises the following steps:

step A21: image preprocessing is finished by using a fuzzy C mean value clustering F-CCT model, and the integral characteristic of the pedestrian image A is set as X ═ X ₁ ,x ₂ ,...,x _N The overall characteristic of the pedestrian image B is Y ═ Y ₁ ,y ₂ ,...,y _N }；

Step A22: will be provided withThe image processed in the step S21 is used as input data of a pedestrian correlation model SAM-Dets with fine-grained representation fused: using X as an input vector and Y as a weight vector, and encoding the local fine-grained characteristic f of the pedestrian A through the pedestrian correlation model SAM-Dets fused with the fine-grained characterization ₁ (ii) a And with Y as an input vector and X as a weight vector, coding local fine-grained features f of the pedestrian B through the pedestrian correlation model SAM-Dets fused with the fine-grained characterization ₂ ；

Step A23: will f is _s ＝(f ₁ -f ₂ ) ² As input values of two convolution layers C with a kernel size of 1 × 1 × 4096, softmax is used as an output function to output a two-dimensional vector (q) ₁ ,q ₂ ) Representing the probability value of inputting two objects belonging to the same person in the real world as the appearance suitability;

in step a22, the structure of the fused fine-grained representation pedestrian association model SAM-Dets includes: k attention branches and splice layers; each of the attention branches includes the following six layers, wherein:

the first layer is a convolution layer A and is used for extracting high-level features of the input pedestrian overall features;

the second layer is an activation layer, and the activation function is softmax;

the third layer is a dimension expansion layer;

the fourth layer is a summation layer, and the overall pedestrian characteristics are added with the result obtained by the third layer;

the fifth layer is a global average pooling layer for reducing feature dimension;

the sixth layer is a full connection layer and is used for finishing inner product calculation of the input vector and the weight vector in the weight matrix;

the splicing layer splices results obtained by the K attention branches according to channels and outputs the local fine-grained characteristic of the pedestrian;

the convolution kernel size of the convolution layer A is 1 multiplied by 1, and the step length is 1; the dimension expanding layer expands the channel dimension into 512 dimensions; the size of the global average pooling layer is 1 multiplied by 1, and the step length is 1;

the step B21-the step B24 are specifically:

step B21: image preprocessing is finished by using a fuzzy C mean value clustering F-CCT model, and the integral characteristic of the pedestrian image A is set as X ═ X ₁ ,x ₂ ,...,x _N The overall characteristic of the pedestrian image B is Y ═ Y ₁ ,y ₂ ,...,y _N }；

Step B22: extracting pedestrian abstract characteristics from the pedestrian image A and the pedestrian image B through a DR-ResNet basic network;

step B23: further extracting high-level features of the target pedestrian by using the convolutional layer B, and using the high-level features as input data of a classification model and a pedestrian correlation model SAM-Dets fusing fine-grained representation; the classification model respectively outputs the identity identification representation numbers of the pedestrian image A and the pedestrian image B, and the pedestrian correlation model SAM-Dets fused with the fine-grained representation outputs the local fine-grained feature f of the pedestrian A ₁ Local fine-grained feature f of pedestrian B ₂ ；

Step B24: will f is _s ＝(f ₁ -f ₂ ) ² As input values of two convolution layers C with a kernel size of 1 × 1 × 4096, softmax is used as an output function to output a two-dimensional vector (q) ₁ ,q ₂ ) Representing the probability value of inputting two objects belonging to the same person in the real world as the appearance suitability;

in step B22, the DR-ResNet basis network comprises two weight-shared identical deep convolution twin neural basis network modules R-ResNet; the structure of the deep convolution twin neural basic network module R-ResNet comprises forty-nine convolution layers, three parallel convolution layers and a tail end convolution layer:

wherein the convolution kernel size of the first convolution layer is (7,7, 64), the max-firing is (3,3), and the sliding step length is 2;

the sizes of convolution kernels of the second convolution layer to the fourth convolution layer are (1,1,64), (3,3,64) and (1,1,256), and the ReLu function is adopted as the activation function; forming a convolution block by the three layers of convolution layers and the activation function, and taking an input value of the convolution block as an input value of a second convolution layer and an input value of a third layer of activation function of the convolution block; the fifth to seventh convolutional layers, and the eighth to tenth convolutional layers all adopt the same structures as the second to fourth convolutional layers;

the sizes of convolution kernels of the eleventh convolution layer to the thirteenth convolution layer are (1,1, 128), (3,3, 128) and (1,1, 512), and the ReLu function is adopted as the activation function; the three convolutional layers and the activation function form a convolutional block, and the input value of the convolutional block is used as the input value of the first convolutional layer and also used as the input value of the third layer of the activation function of the convolutional block; the fourteenth to sixteenth convolutional layers, the seventeenth to nineteenth convolutional layers, and the twentieth to twenty second convolutional layers all adopt the same structures as the eleventh to thirteenth convolutional layers;

the sizes of convolution kernels from the twenty-third convolution layer to the twenty-fifth convolution layer are (1,1,256), (3,3, 256) and (1,1, 1024), and the ReLu function is adopted as the activation function; the three convolutional layers and the activation function form a convolutional block, and the input value of the convolutional block is used as the input value of a twenty-third convolutional layer and also used as the input value of a third layer of the activation function of the convolutional block; the twenty-sixth to twenty-eighth, twenty-ninth to thirty-first, thirty-second to thirty-fourth, thirty-fifth to thirty-seventh, and thirty-eighth to forty-fourth convolutional layers all adopt the same structure as the twenty-third to twenty-fifth convolutional layers;

the sizes of convolution kernels from the forty-th convolution layer to the forty-third convolution layer are (1,1, 512), (3,3, 512) and (1,1, 2048), and the activation functions all adopt ReLu functions; the three convolutional layers and the activation function form a convolutional block, and the input value of the convolutional block is used as the input value of a forty-th convolutional layer and also used as the input value of a third layer of the activation function of the convolutional block; the forty-fourth to forty-sixth and forty-seventh to forty-ninth buildup layers have the same structure as the forty-fourth to forty-second buildup layers;

after the forty-ninth convolutional layer, three parallel convolutional layers are formed, each convolutional layer uses 2048 convolutional cores, the sizes of the first parallel convolutional layer to the third parallel convolutional layer are respectively (3, 1024), (5, 1024) and (7, 1024), the channels of the three parallel convolutional layers are combined through a connecting layer, and then the max-pooling is (4, 4);

the last layer is the end convolution layer with size (2, 2048) using 1024 convolution kernels;

the convolutional layer B uses 2 convolutional kernels and has the size of (1,1, 4096);

the structure of the pedestrian correlation model SAM-Dets fused with the fine-grained representation comprises the following steps: k attention branches and splice layers; each of the attention branches includes the following six layers, wherein:

the first layer is convolution layer A, the convolution kernel size is 1 multiplied by 1, and the step length is 1; the high-level features are used for extracting the input overall features of the pedestrians;

the second layer is an active layer, and the active function is softmax;

the third layer is a dimension expanding layer, and the dimension of the channel is expanded into 512 dimensions;

the fourth layer is a summation layer, and the overall pedestrian characteristics are added with the result obtained by the third layer;

the fifth layer is a global average pooling layer with the size of 1 multiplied by 1 and the step length of 1, and is used for reducing the characteristic dimension;

the sixth layer is a full connection layer and is used for finishing inner product calculation of the input vector and the weight vector in the weight matrix;

the splicing layer splices results obtained by the K attention branches according to channels and outputs the local fine-grained characteristic of the pedestrian;

in step S3, according to the degree of adaptation of each pair of pedestrians, a minimum cost flow graph model is established in combination with the spatio-temporal relationship, and a specific process of solving an optimal pedestrian association solution includes the following steps:

step S31: let given t _p The cost flow graph at time-1 after the instant alignment is completed is

When t is _p Time of day, is the collection of pedestrians in the field of view

And go out of field pedestrian collection

Each target newly adds in and out two nodes, and the newly added nodes are updated to be connected with the directed edges between the source points and the sinks;

step S32: according to the pedestrian set in the field of vision

And go out of field pedestrian collection

The pedestrian appearance adaptation degree between every two middle targets is updated, the directed edges between the corresponding nodes are updated, and t is obtained _p New cost flow graph of time of day

Step S33: deleting all aligned target nodes and set of pedestrians in view

The target nodes which are not aligned remain, and the pedestrians in the visual field are collected

Obtaining a cost flow graph by taking the residual unaligned target nodes as newly entered target pedestrians

Waiting for updating alignment at the next moment;

in step S4, the pedestrian is given the operation of keeping the original identification or giving a new identification, and then tracked by the tracker; the tracker adopts a KCF algorithm for tracking; the KCF algorithm allocates a tracker for each pedestrian;

the tracking visual field area of the camera is divided into a core area and a critical area, and in step S1, only pedestrians in the critical area are detected by the detector.

Images