CN114677754B - Behavior recognition method, device, electronic device and computer-readable storage medium - Google Patents

Behavior recognition method, device, electronic device and computer-readable storage medium Download PDF

Info

Publication number
CN114677754B
CN114677754B CN202210239612.0A CN202210239612A CN114677754B CN 114677754 B CN114677754 B CN 114677754B CN 202210239612 A CN202210239612 A CN 202210239612A CN 114677754 B CN114677754 B CN 114677754B
Authority
CN
China
Prior art keywords
image frame
activation
area
center
frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210239612.0A
Other languages
Chinese (zh)
Other versions
CN114677754A (en
Inventor
苏海昇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Intelligent Technology Co Ltd
Priority to CN202210239612.0A priority Critical patent/CN114677754B/en
Publication of CN114677754A publication Critical patent/CN114677754A/en
Application granted granted Critical
Publication of CN114677754B publication Critical patent/CN114677754B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例公开了一种行为识别方法、装置、设备及计算机可读存储介质。该方法包括：对获得的图像帧序列的至少一个图像帧中，各个图像帧的重要区域分类识别，得到与至少一个图像帧对应的识别结果；重要区域根据至少一个图像帧中目标对象所在的检测框区域确定；在识别结果表征存在预设异常事件的情况下，确定每个图像帧的与识别结果对应的类激活图的激活中心，得到至少一个图像帧的至少一个激活中心；一个激活中心表征对应的一个图像帧的类激活图中的异常位置；基于至少一个激活中心，识别出各个图像帧的异常目标对象。通过本公开，能够提高识别准确度。

The disclosed embodiment discloses a behavior recognition method, device, equipment and computer-readable storage medium. The method includes: classifying and identifying important areas of each image frame in at least one image frame of the obtained image frame sequence, and obtaining a recognition result corresponding to at least one image frame; the important area is determined according to the detection frame area where the target object in at least one image frame is located; when the recognition result indicates the existence of a preset abnormal event, the activation center of the class activation map corresponding to the recognition result of each image frame is determined to obtain at least one activation center of at least one image frame; an activation center represents an abnormal position in the class activation map of a corresponding image frame; based on at least one activation center, the abnormal target object of each image frame is identified. Through the present disclosure, the recognition accuracy can be improved.

Description

Behavior recognition method, behavior recognition device, electronic equipment and computer readable storage medium
Technical Field
The present disclosure relates to terminal technologies, and in particular, to a behavior recognition method, apparatus, electronic device, and computer readable storage medium.
Background
Anomaly detection in video is an important problem in the field of computer vision, and has wide application in the field of intelligent recognition, such as detecting illegal behaviors, traffic accidents, and some unusual events. Thousands of cameras are deployed worldwide. However, most cameras merely record dynamics at each moment, and do not have the ability to automatically recognize (often requiring special personnel to be responsible for manual viewing). Due to the vast number of videos, it is obviously impractical to filter the content in the video by human only. We need to automatically detect abnormal events occurring in video using computer vision and deep learning techniques.
Identifying abnormal events in video is extremely difficult. Possible challenges include scarcity of annotation data due to small probability events, large inter/intra class variances, subjectively defined differences in outliers, low resolution of video images, and so forth. As a human, we have performed recognition of an abnormality by common sense, for example, if people gather on a street where traffic is not usually present, an abnormality may occur, for example, a violent event such as a fight. They have no general knowledge about the machine, only visual features. In general, the stronger the visual feature, the better the anomaly detection performance that is desired.
Some recognition methods related to abnormal events exist in the related art, but when the recognition method in the related art is adopted for performing the abnormal recognition, the recognition accuracy is low.
Disclosure of Invention
The embodiment of the disclosure provides a behavior recognition method, a behavior recognition device, electronic equipment and a computer readable storage medium, which can improve recognition accuracy.
The technical scheme of the embodiment of the disclosure is realized as follows:
the embodiment of the disclosure provides a behavior recognition method, which comprises the following steps:
classifying and identifying important areas of each image frame in at least one image frame of the obtained image frame sequence to obtain an identification result corresponding to the at least one image frame, wherein the important areas are determined according to the detection frame area where the target object in the at least one image frame is located;
Determining an activation center of a class activation graph corresponding to the identification result of each image frame under the condition that the identification result represents a preset abnormal event, and obtaining at least one activation center of the at least one image frame;
an abnormal target object for the respective image frames is identified based on the at least one activation center.
The embodiment of the disclosure provides a behavior recognition device, which comprises a recognition unit, a recognition unit and a recognition unit, wherein the recognition unit is used for classifying and recognizing important areas of each image frame in at least one image frame of an obtained image frame sequence to obtain a recognition result corresponding to the at least one image frame;
The device comprises a determining unit, a determining unit and a judging unit, wherein the determining unit is used for determining the activation center of a class activation image corresponding to the identification result of each image frame under the condition that the identification result represents the existence of a preset abnormal event to obtain at least one activation center of at least one image frame;
the identifying unit is further configured to identify an abnormal target object of each image frame based on the at least one activation center.
The embodiment of the disclosure provides electronic equipment, which comprises a memory and a processor, wherein the memory is used for storing an executable computer program, and the processor is used for realizing the behavior recognition method when executing the executable computer program stored in the memory.
The embodiment of the disclosure provides a computer readable storage medium storing a computer program for causing a processor to execute the behavior recognition method.
The behavior recognition method, device, equipment and computer readable storage medium provided by the embodiment of the disclosure are used for classifying and recognizing important areas of each image frame in at least one image frame of an obtained image frame sequence to obtain a recognition result corresponding to the at least one image frame, determining the important areas according to a detection frame area where a target object in the at least one image frame is located, determining an activation center of a class activation image corresponding to the recognition result of each image frame under the condition that the recognition result represents a preset abnormal event to obtain at least one activation center of the at least one image frame, wherein the activation center represents an abnormal position in the class activation image of the corresponding one image frame, and recognizing the abnormal target object of each image frame based on the at least one activation center. By adopting the technical scheme, on one hand, the important areas of the determined image frames are classified and identified, so compared with the identification of the data according to the whole image of each image frame, the search range during identification is effectively reduced, and meanwhile, the interference to the identification is reduced, so that the identification accuracy is improved; on the other hand, the abnormal center (activation center) of each image frame is positioned by determining the abnormal position in the class activation diagram corresponding to the identification result of each image frame, the abnormal target object in the corresponding image frame is identified according to the obtained abnormal center, and the class activation diagram corresponding to the identification result can characterize the target object related to the identification result in the image frame, so that the identified abnormal target object of each image frame is more accurate, and the identification accuracy of the abnormal target object is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the technical aspects of the disclosure.
FIG. 1 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 2 is a schematic illustration of one presentation effect of an exemplary class activation graph and activation center provided by an embodiment of the present disclosure;
FIG. 3 is a schematic illustration of a display effect of an abnormal target object in an exemplary image frame provided by an embodiment of the present disclosure;
FIG. 4 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 5 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 6 is another display effect schematic of an exemplary class activation diagram provided by an embodiment of the present disclosure;
FIG. 7A is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 7B is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 8 is a schematic illustration of a display effect of a class activation diagram and a target activation center corresponding to an exemplary image frame provided by an embodiment of the present disclosure;
FIG. 9 is a schematic flow chart of an alternative behavior recognition method provided by an embodiment of the present disclosure;
FIG. 10 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 11 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure;
FIG. 12 is a schematic illustration of corresponding regions in an image frame at generalized region locations of two different exemplary second regions provided by embodiments of the present disclosure;
fig. 13 is a schematic structural diagram of a behavior recognition device according to an embodiment of the present disclosure;
fig. 14 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present disclosure more apparent, the present disclosure will be further described in detail with reference to the accompanying drawings, and the described embodiments should not be construed as limiting the present disclosure, and all other embodiments obtained by those skilled in the art without making inventive efforts are within the scope of protection of the present disclosure.
The inventor of the present disclosure has found through research that, in a conventional behavior recognition method, data of a full graph of an input video sequence is usually enhanced or other preprocessed and then sent to a behavior recognition model for prediction, however, this method is only applicable to human-centered video behavior recognition, and such data are commonly found in a published video academic dataset. The video shot by the monitoring camera often contains more information, the covered visual field is larger, and meanwhile, the occurrence position and the human body scale of the target event are random, so that the simple input of the full graph serving as a model is obviously unreasonable, and more irrelevant information in the picture can interfere with behavior recognition, thereby influencing the accuracy of behavior recognition and the recognition accuracy of a behavior executor.
In this regard, the inventors of the present disclosure have studied that it is possible to input a partial area instead of a whole image into a behavior recognition model to classify the behaviors, so that the interference of most of irrelevant information in a picture can be reduced, and the accuracy of behavior recognition is improved, but on this basis, it is also necessary to consider the aspect of how to accurately position a specific behavior executor after the search range of the behavior recognition model is narrowed.
For this reason, the embodiments of the present disclosure provide a behavior recognition method capable of improving recognition accuracy of abnormal behavior recognition and an abnormal target object (behavior executor) associated with the abnormal behavior recognition. Before explaining the embodiments of the present disclosure in further detail, terms and terminology involved in the embodiments of the present disclosure are explained, and the terms and terminology involved in the embodiments of the present disclosure are applicable to the following explanation.
1) CNN (Convolutional Neural Networks), a convolutional neural network, is a deep neural network with a convolutional structure, is essentially an input-to-output mapping, can learn a large number of mapping relationships between inputs and outputs, is trained by a known mode, has mapping capability between input and output pairs, and is commonly used in the field of image recognition.
2) CAM (Class Activation Map), referred to as class activation thermodynamic diagrams, or class activation thermodynamic diagrams for short, is a two-dimensional diagram, essentially a two-dimensional feature score network associated with a particular output class, with each location of the grid representing the degree of importance for that class. For a picture that is input to the CNN model and classified as a "dog," the degree of similarity of each location in the picture to the "dog" class may be presented in a thermodynamic diagram. Class activation maps help to know which part of a picture causes the convolutional neural network to make the final decision.
The following describes exemplary applications of the electronic device provided by the embodiments of the present disclosure, where the electronic device provided by the embodiments of the present disclosure may be implemented as various types of user terminals (hereinafter referred to as terminals) such as AR glasses, notebook computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable game devices), and the like, and may also be implemented as servers.
In the following, an exemplary application when the electronic device is implemented as a terminal will be described. Fig. 1 is a schematic flow chart of an alternative behavior recognition method provided in an embodiment of the present disclosure, and will be described with reference to the steps shown in fig. 1.
S101, classifying and identifying important areas of each image frame in at least one image frame of the obtained image frame sequence to obtain an identification result corresponding to the at least one image frame, wherein the important areas are determined according to the detection frame area where the target object in the at least one image frame is located.
The electronic device may input the important areas of each image frame in the at least one image frame of the image frame sequence into the behavior recognition model at the same time to perform classification recognition to obtain a recognition result corresponding to the at least one image frame under the condition that the important areas of each image frame in the at least one image frame are obtained, and the important areas of each image frame are determined according to the detection frame areas where all the target objects in the at least one image frame are located, for example, the electronic device may input the important areas of each image frame in the 8 image frames of each obtained image frame sequence into the behavior recognition model at the same time to perform classification recognition to obtain a recognition result corresponding to the 8 image frames.
Here, the electronic device may crop out important areas of each of the at least one image frame, and classify and identify the cropped important areas.
In the embodiment of the disclosure, the detection frames are used to mark the positions of target objects in the image frame, where each target object corresponds to one detection frame, for example, there are two detection frames corresponding to two target objects one to one in fig. 3 described below.
In some embodiments, the electronic device may obtain the at least one image frame and the important region of each image frame in the image frame sequence directly from the other device, so as to perform classification identification according to the important region of each image frame in the at least one image frame, thereby obtaining an identification result corresponding to the at least one image frame.
In the embodiment of the disclosure, the behavior recognition model is a pre-trained neural network model, such as Two-Stream model, 3D (3-dimensional) ConvNets model, 2D (2-dimensional) ConvNets model, and the like, which is not limited by the embodiment of the disclosure. The electronic device can determine whether abnormal events such as frame play, alarming, traffic accidents and the like exist in the at least one image frame by classifying and identifying important areas of each image frame in the at least one image frame.
In some embodiments, the image frame sequence may be a video stream acquired by an acquisition device such as a camera disposed in a specific scene, where the at least one image frame may be a continuous frame in the video stream, and the specific scene may be a street, a mall, a scenic spot, or the like, which is not limited by the embodiments of the present disclosure.
S102, under the condition that the identification result represents that a preset abnormal event exists, determining an activation center of a class activation diagram corresponding to the identification result of each image frame to obtain at least one activation center of at least one image frame, wherein one activation center represents an abnormal position in the class activation diagram of the corresponding image frame.
When the electronic device recognizes that a preset abnormal event exists in the at least one image frame (namely, a category of the preset abnormal event is recognized), an activation center of a class activation image corresponding to each image frame in the at least one image frame can be determined, so that at least one activation center corresponding to the at least one image frame is obtained, wherein the class activation image is a class activation image related to the category corresponding to the recognition result, and an activation center characterizes an abnormal position in the class activation image of the corresponding one image frame. For example, as shown in fig. 2, in the case where the preset abnormal event of the framing is identified according to 8 consecutive image frames of y1, y2, y3, y4, y5, y6, y7 and y8, the areas with the greatest information provided for the class of the preset abnormal event identified by the model in the respective image frames may be generated, where the areas r1, r2, r3, r4, r5, r6, r7 and r8 are respectively represented by the areas r1, r2, r3, r8 in the class activation diagrams w1, w2, w3, w4, w5, w6, w7 and w8, and the areas r1 in the class activation diagrams w1 represent the areas with the greatest information provided for the class of the preset abnormal event identified by the model in the image frames y 1; and, the activation centers of the 8 class activation maps, such as j1, j2, j3, j4, j5, j6, j7 and j8 in fig. 2, are also determined (since the sizes of the 8 class activation maps are consistent with the sizes of the 8 corresponding image frames, the activation centers of the class activation maps are directly marked in the corresponding image frames in fig. 2, and as shown in fig. 2, the activation centers are marked with pentagonal shapes).
In the embodiment of the present disclosure, the preset abnormal event may be, for example, a frame, a crowd alarm, a traffic accident, etc., which is not limited in the embodiment of the present disclosure.
S103, identifying abnormal target objects of the image frames based on at least one activation center.
In the embodiment of the disclosure, in the case that the activation center of each image frame in the at least one image frame is determined, for each image frame, the electronic device may determine whether an abnormal target object exists in the image frame according to the activation center of the image frame, and identify the existing abnormal target object if the abnormal target object exists.
In some embodiments, since each activation center corresponds to an image frame, for each activation center, the electronic device may determine a location of the activation center in the corresponding one of the image frames, and if the location is located in any one of the detection frame areas in the image frames, determine a target object corresponding to the any one of the detection frame areas as an abnormal target object in the image frame, so that for the at least one activation center, an abnormal target object of each of the at least one image frames corresponding to the at least one activation center may be obtained. For example, as shown in fig. 3, for the image frame a, in the case where the activation center 31 is located at two different human body detection frames 32 and 33, it may be determined that two pedestrians corresponding to the human body detection frames 32 and 33 are abnormal target objects in the image frame a.
Here, for each activation center, in the case where the position of the activation center in the corresponding one of the image frames does not belong to any of the detection frame areas in the image frame, it may be determined that there is no abnormal target object in the image frame.
In the embodiment of the disclosure, the important areas of the determined image frames are classified and identified, so that compared with the identification of the data according to the whole image of each image frame, the search range of each image frame during identification is effectively reduced, and meanwhile, the identification accuracy is improved, on the other hand, the abnormal center (activation center) of each image frame is positioned by determining the abnormal position of the class activation image corresponding to the identification result of each image frame, the abnormal target object in the corresponding image frame is identified according to the obtained abnormal center, and the class activation image corresponding to the identification result can characterize the target object related to the identification result in the image frame, so that the abnormal target object of each image frame identified later can be more accurate, the identification accuracy of the abnormal target object is improved, and meanwhile, the search range of the model is reduced during identification without influencing the identification of the subsequent abnormal target object.
Fig. 4 is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure, as shown in fig. 4, S102 may be implemented through S1021-S1023, which will be described with reference to fig. 4.
S1021, under the condition that the identification result represents that a preset abnormal event exists, determining a group of activation values corresponding to the identification result of each image frame, wherein the group of activation values are used for generating a class activation diagram of the image frame, and each pixel position in the class activation diagram corresponds to one activation value in the group of activation values one by one.
When the electronic device recognizes that a preset abnormal event exists in at least one image frame through the behavior recognition model, for each image frame, the electronic device can determine a group of activation values corresponding to the preset abnormal event of the image frame through a class activation mapping method, wherein the group of activation values are used for drawing a class activation image corresponding to the preset abnormal event of the image frame, each pixel position in the class activation image corresponds to one pixel position in the group of activation values one by one, and in the process of drawing the class activation image of the image frame according to the group of activation values, for example, when one activation value represents that the color of one pixel point corresponding to the activation value is red, red is generated at the pixel position corresponding to the activation value, and the class activation image corresponding to the group of activation values is drawn through the method, namely, the contribution of different pixel points to the recognition result is represented by different dark colors in the class activation image.
In the embodiment of the disclosure, the size of the class activation image of the image frame drawn according to the set of activation values is consistent with the size of the image frame, so that the abnormal target object in the original image frame can be accurately positioned according to the abnormal position in the class activation image of the image frame.
In some embodiments, fig. 5 is a schematic flow chart of an alternative behavior recognition method provided in the embodiments of the present disclosure, as shown in fig. 5, S1021 may be implemented through S201-S203, which will be described in connection with fig. 5.
S201, generating a group of initial class activation mapping values corresponding to the identification result for each image frame.
For each image frame, the electronic device may generate a set of initial class activation map values for the image frame corresponding to the recognition result if the recognition result for the image frame is obtained.
In some embodiments, the step S201 may be implemented by generating, for each image frame, a set of initial class activation mapping values corresponding to a recognition result based on weights of classification layers corresponding to the recognition result and feature maps output by convolution layers in a behavior recognition model, where the recognition result is obtained by classifying and recognizing an important area by using the behavior recognition model, and the behavior recognition model includes the classification layers and the convolution layers. In some embodiments, the electronic device may employ a behavior recognition model including a classification layer and a plurality of convolution layers to classify and identify a significant region of each of the at least one image frame to identify an event category of a preset anomaly event. For each image frame, the last convolution layer in the behavior recognition model can generate a y Zhang Tezheng diagram of an important area of the image frame, each feature diagram mainly extracts features related to one of y categories, for example, y feature diagrams can be expressed as a 1,A2,…,Ay, y neurons are correspondingly included in the classification layer, one neuron corresponds to one category, each neuron corresponds to y weight values, for example, w 1,w2,…,wy, and in the case that the event category of the preset abnormal event is c categories, a set of initial category activation mapping values corresponding to the image frame can be calculated by the following formula (1):
wherein i represents the ith neuron, A weight value of class c corresponding to the ith neuron is represented, a i represents a feature map corresponding to the ith neuron,An initial set of class activation value map values is characterized.
S202, up-sampling the initial class activation mapping value to obtain a group of intermediate activation values, wherein the group of intermediate activation values are used for generating a class activation diagram corresponding to an important area of the image frame.
S203, fusing the intermediate activation value with the original pixel value of the image frame to obtain a group of activation values.
Here, for an image frame, the class activation map that can be drawn by the obtained set of initial class activation mapping values is consistent with the size of the feature map generated by the last convolution layer, so the electronic device may perform an upsampling operation on the set of initial class activation mapping values to generate a set of intermediate activation values, and the size of the class activation map that can be drawn by the generated set of intermediate activation values is consistent with the size of the important area of the image frame, so as to obtain a class activation map corresponding to the important area of the image frame; on the basis, the electronic equipment can further perform fusion processing on the group of intermediate activation values and a group of original pixel values of the image frame to obtain a group of activation values corresponding to the image frame, and the size of a class activation image which can be drawn by the obtained group of activation values is consistent with the size of an original image of the image frame, so that superposition of the original image of the image frame and the class activation image corresponding to an important area of the image frame is realized, and finally, the class activation image which corresponds to the image frame and is consistent with the size of the image frame is obtained. For example, fig. 6 shows 8 class activation graphs w10, w20, w30, w40, w50, w60, w70 and w80, which are generated in a one-to-one correspondence with 8 consecutive image frames of y10, y20, y30, y40, y50, y60, y70 and y80 and correspond to the racking event, in the case that the racking event is identified according to 8 consecutive image frames of y10, y20, y30, y50, y60, y70 and y80 (not shown in fig. 6), wherein the corresponding areas r10, r20, r30, r40, r50, r60, r70 and r80 are marked, and the area r10 in the class activation graph w10 represents the area of the respective image frames that provides the most information when the model identifies the class of the preset abnormal event, for example, the area r10 in the class activation graph w10 represents the most information provided for the identified model of the racking event in the image frame y 10.
Explanation of S1022 to S1023 in FIG. 4 is continued as follows:
s1022, determining the maximum activation value in a group of activation values of each image frame, and determining the pixel position corresponding to the maximum activation value as the abnormal position.
For each image frame, the electronic device may determine a maximum activation value of a set of activation values corresponding to the image frame and determine a pixel location corresponding to the maximum activation value as an outlier location for the image frame. Since the class activation map is a two-dimensional image, the anomaly location (pixel location) may be represented using two-dimensional coordinates, e.g., X (i, j), to characterize the location of the anomaly location in the class activation map or in the image frame.
S1023, obtaining an activation center of the image frame based on the abnormal position of the image frame, and correspondingly obtaining at least one activation center for at least one image frame.
For each image frame, the electronic device may determine an abnormal position in the class activation map of the image frame as an activation center of the class activation map of the image frame, so that for the at least one image frame, at least one activation center may be correspondingly obtained.
Fig. 7A is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure, as shown in fig. 7A, S103 may be implemented through S1031-S1032, and will be described with reference to fig. 7A by taking fig. 1 as an example.
S1031, determining a target activation center according to at least one activation center.
Because the positions of the activation centers in different types of activation graphs may be offset, the electronic device can determine a target activation center according to the obtained activation centers under the condition that all the activation centers corresponding to at least one image frame are determined, so that the drift phenomenon of the activation centers can be relieved, the abnormal target object of each image frame in the at least one image frame determined by the target activation center is more accurate, and the recognition accuracy of the abnormal target object is improved.
In some embodiments, the electronic device may calculate an average value of all activation centers corresponding to the at least one image frame, and take the average value as the target activation center. Here, since each activation center is actually a two-dimensional coordinate of one pixel point, the electronic device may obtain the target activation center by calculating a coordinate average value of these two-dimensional coordinates, for example, an average value of x coordinates and an average value of y coordinates may be calculated, respectively.
S1032, identifying abnormal target objects in each image frame according to the target activation center.
The electronic device may determine, for each image frame, whether an abnormal target object exists in the image frame according to the target activation center when determining that one target activation center corresponding to the at least one image frame, and identify, when determining that the abnormal target object exists, the existing abnormal target object.
Here, for each image frame, the electronic device may determine a position of the target activation center in the image frame, and if the position is located in any detection frame region in the image frame, determine a target object corresponding to the any detection frame region as an abnormal target object in the image frame, so that for the at least one image frame, an abnormal target object of each image frame in the at least one image frame may be obtained. For example, 32 and 33 in fig. 3 described above are abnormal target objects in the image frame a.
Fig. 7B is a schematic flow chart of an alternative behavior recognition method according to an embodiment of the present disclosure, as shown in fig. 7B, S103 may also be implemented through S1033-S1034, and fig. 1 will be taken as an example, and description will be made with reference to fig. 7B.
S1033, clustering at least one activation center in an unsupervised clustering mode to obtain a target activation center.
S1034, identifying abnormal target objects in each image frame according to the target activation center.
Under the condition that at least one activation center corresponding to at least one image frame one by one is determined, the electronic equipment can determine a target activation center according to the at least one activation center in an unsupervised clustering mode, and can relieve the drift phenomenon of the activation center, so that the abnormal target object of each image frame in the at least one image frame determined by the target activation center in the follow-up process is more accurate, and the recognition accuracy of the abnormal target object is improved.
In some embodiments, the electronic device may use a K-Means clustering method to cluster all activation centers corresponding to the at least one image frame, so that drift points can be removed by Means of unsupervised clustering, so that an abnormal target object of each image frame in the at least one image frame determined by using the target activation center later is more accurate, and the recognition accuracy of the abnormal target object is improved.
Here, the unsupervised clustering method may be other clustering methods, which is not specifically limited in the embodiment of the present disclosure.
In some embodiments, S1031 described above may also be implemented by S1033 described above.
In some embodiments, S1033 described above may be implemented by S301-S302:
s301, clustering at least one activation center in an unsupervised clustering mode to obtain a center cluster with the largest number of the activation centers.
The electronic device may cluster all the active centers corresponding to the at least one image frame in an unsupervised clustering manner, so as to obtain a plurality of clusters (clusters), where each cluster includes at least one active center, and then, a cluster with the largest number of active centers included in the plurality of clusters may be selected from the plurality of clusters, so as to obtain a center cluster.
S302, determining an average value of the activation centers contained in the center cluster, and determining the average value as a target activation center.
Under the condition that the central cluster is determined, the electronic equipment can calculate the average value of all the activation centers contained in the central cluster, and takes the average value as a target activation center. Similarly, since each activation center is actually a two-dimensional coordinate of one pixel, the electronic device may obtain the target activation center by calculating a coordinate average value of these two-dimensional coordinates, for example, obtain the target activation center X (i, j). FIG. 8 shows that in the case where the presence of the preset abnormal event of the framing is identified according to 8 consecutive image frames of y11, y22, y33, y44, y55, y66, y77 and y88, the areas corresponding to the 8 consecutive image frames of y11, y22, y33, y44, y55, y66, y77 and y88 are generated, one by one, and the 8 class activation images w11, w22, w33, w44, w55, w66, w77 and w88 corresponding to the framing event are generated, wherein the class activation images w11, w22, w33, w44, w55, w66, w77 and w88 are marked in correspondence with the areas r11, r22, r33, r44, r55, r66, r77 and r88, and represent the areas of the respective image frames that are most informative when the model identifies the class of the preset abnormal event, for example, the areas r11 in the class activation image w11 represent the areas of the image frames that the j type of the frame is most informative when the model is identified, and the areas of j in the class activation images w11, and the areas of the classes of j11, and the areas of the center 8, such as the areas of the center 8, w11, w33, w55, w66, w77 and w88 are marked in correspondence with the areas of the class activation images w11, w33, w55, w66, w77 and w 88.
In some embodiments, for at least one image frame in S101 described above, and the important areas of each image frame in the at least one image, the electronic device may also obtain itself by some method, such as the method shown in fig. 9. Fig. 9 is a schematic flowchart of an alternative behavior recognition method according to an embodiment of the present disclosure, as shown in fig. 9, S001-S003 may be further performed before S101, and will be described with reference to fig. 9 by taking fig. 1 as an example.
S001, carrying out target detection on the current image frame of the image frame sequence, and determining at least one detection frame area corresponding to at least one existing target object.
The electronic device may obtain a video stream of a specific scene acquired in real time from an acquisition device such as a camera provided in the specific scene, and perform real-time target detection on the video stream acquired in real time by using an image frame as a unit, in the target detection process, the electronic device may use a pre-trained target detection network for the current image frame to detect whether a target object exists in the current image frame, and in the case that the target object exists, mark all the target objects in the current image frame with a detection frame, so as to obtain detection frame areas corresponding to all the target objects in the current image frame, for example, two detection frame areas 32 and 33 corresponding to two target objects one by one exist in the image frame a in fig. 3.
In the embodiment of the present disclosure, the target detection network may be a CNN network, or may be another network, or the like, which is not limited in the embodiment of the present disclosure.
For an image frame, there may or may not be a target object, and the present disclosure is described with respect to the case where there is a target object in the image frame.
S002, determining a center detection frame area in at least one detection frame area.
S003, when at least one image frame is detected, determining important areas of each image frame in the at least one image frame based on at least one central detection frame area corresponding to the at least one image frame.
For each image frame, the electronic device may determine one detection frame area as a central detection frame area from among all detection frame areas in the image frame, so that when the electronic device obtains at least one image frame, at least one central detection frame area may be correspondingly obtained, and the important area of each image frame in the at least one image frame may be determined according to the at least one central detection frame area.
In the embodiment of the disclosure, the electronic device may determine that at least one image frame has been detected if it is determined that the preset condition is satisfied. The electronic device may determine that the preset condition is met when the detected duration reaches the preset duration, and obtain detected image frames when the detected duration reaches the preset duration, where the preset duration may be set according to actual needs, for example, the preset duration may be 6 seconds, then the electronic device may use the detected image frames in every 6 seconds as detected image frames, that is, the electronic device may determine the important areas of each image frame in the at least one image frame every 6 seconds, and the electronic device may further determine that the preset condition is met when the number of detected image frames reaches the preset number, and obtain detected image frames, where the preset number may be set according to actual needs, for example, when the preset number is 8, the electronic device detects 8 image frames each time, that is, determines the 8 image frames as detected image frames, that is, each time the electronic device detects 8 image frames, and determines the important areas of each image frame in the 8 image frames.
In some embodiments, fig. 10 is a schematic flow chart of an alternative behavior recognition method provided in the embodiments of the present disclosure, as shown in fig. 10, S002 may be implemented through S401-S404, which will be described with reference to fig. 10.
S401, expanding at least one detection frame area by a first preset proportion to obtain at least one first area corresponding to the at least one detection frame area.
For all detection frame regions in each image frame, the electronic device may expand the areas of all detection frame regions by a first preset ratio to obtain first regions corresponding to all detection frame regions, e.g., in case that N detection frame regions exist in each image frame, the electronic device may expand the areas of the N detection frame regions by a first preset ratio to obtain N first regions, where N is an integer greater than 0. The enlarging the area of one detection frame region by the first preset ratio means enlarging the range included in the detection frame region by the first preset ratio, thereby obtaining a first region having a larger included range.
In the embodiment of the present disclosure, the first preset ratio may be set according to actual needs, for example, the first preset ratio may be 1.5, which is not limited in the embodiment of the present disclosure.
S402, determining a matching result between any two first areas based on the area intersection ratio between any two first areas in at least one first area, wherein the matching result represents whether any two first areas are matched.
For the first areas corresponding to all the detection frame areas in each image frame, the electronic device can calculate the area intersection ratio between any two first areas, and determine whether the any two first areas are matched according to the size relation between the obtained area intersection ratio and a preset threshold value. For example, in the case where one image frame corresponds to 3 first areas (first area 1, first area 2, and first area 3), the electronic device may calculate an area intersection ratio between first area 1 and first area 2, an area intersection ratio between first area 1 and first area 3, and an area intersection ratio between first area 2 and first area 3, and determine that there is no match between first area 1 and first area 2 in the case where the area intersection ratio between first area 1 and first area 2 is smaller than a preset threshold, and determine that first area 1 is matched with first area 3 in the case where the area intersection ratio between first area 1 and first area 3 is greater than or equal to a preset threshold, and determine that first area 2 is matched with first area 3 in the case where the area intersection ratio between first area 2 and first area 3 is greater than or equal to a preset threshold.
In the embodiment of the present disclosure, the preset threshold may be set according to actual needs, for example, may be 0, which is not limited to the embodiment of the present disclosure.
S403, based on the matching result, determining the matching times of each first area, and determining the first area with the largest matching times.
For each image frame, under the condition that the matching results among all the first areas of the image frame are determined, the electronic device can determine the number of other first areas matched with each first area according to the matching results, take the number as the matching times, and perform descending order or ascending order arrangement on the matching times of all the first areas, and determine one or more first areas with the largest matching times according to the ordering results. Continuing with the above example, since it is determined that the first region 1 and the first region 2 are not matched by the area overlap ratio, the first region 1 is matched with the first region 3, and the first region 2 is matched with the first region 3, it can be determined that the other first regions to which the first region 1 is matched are the first region 3, the other first regions to which the first region 2 is matched are the first region 3, and the other first regions to which the first region 3 is matched are the first region 1 and the first region 2, and thus it is possible to obtain that the number of times of matching of the first region 1 is 1, the number of times of matching of the first region 2 is 1, the number of times of matching of the first region 3 is 2, and the first region 3 is the first region with the largest number of times of matching.
S404, determining a dense center according to the first area with the largest matching times, and determining a detection frame area corresponding to the dense center as a center detection frame area.
For each image frame, when the electronic device obtains the first area with the largest matching frequency of the image frame, the dense center of the image frame can be determined according to the first area with the largest matching frequency, and the detection frame area corresponding to the dense center is determined as the center detection frame area of the image frame. For example, in the case where the detection frame region corresponding to the dense center is 33 as an example in fig. 3, the detection frame region may be 33 as the center detection frame region of the image frame a.
In some embodiments, the determining the dense center according to the first region with the largest matching number in S404 may be implemented by determining the first region with the largest matching number as the dense center when there is one first region with the largest matching number, and determining the first region with the largest area among the two or more first regions with the largest matching number as the dense center when there are two or more first regions with the largest matching number, so that the accuracy of the obtained dense center may be improved.
In some embodiments, fig. 11 is a schematic flow chart of an alternative behavior recognition method provided in the embodiments of the present disclosure, as shown in fig. 11, S003 may be implemented through S501-S503, which will be described in connection with fig. 11.
S501, expanding at least one central detection frame area corresponding to at least one image frame by a second preset proportion to obtain at least one second area corresponding to at least one detection frame area when at least one image frame is detected.
For all detection frame regions of each of the at least one image frame, the electronic device may expand the areas of all detection frame regions by a second set proportion to obtain second regions corresponding to all detection frame regions, e.g., in case that N detection frame regions are present in each image frame, the electronic device may expand the areas of the N detection frame regions by a second preset proportion to obtain N second regions, where N is an integer greater than 0. The enlarging the area of one detection frame region by the second preset ratio means enlarging the range included in the detection frame region by the second preset ratio, thereby obtaining a second region having a larger included range.
In the embodiment of the present disclosure, the second preset ratio may be set according to actual needs, for example, the second preset ratio may be 2, which is not limited in the embodiment of the present disclosure.
S502, determining the position of the summarized region according to the position of each second region in at least one second region in the affiliated image frame.
In some embodiments, for all the obtained second areas corresponding to the at least one image frame, the electronic device may determine, according to the position information of each second area in the corresponding image frame, one summarized area position corresponding to all the second areas corresponding to the at least one image frame, and make the area corresponding to the summarized area position include any one of the second areas corresponding to the at least one image frame.
Here, the summarized region position may be a region coordinate, and for all the second regions corresponding to the at least one image frame, the summarized region position may be a union of the region coordinates of all the second regions corresponding to the at least one image frame, for example, may be a minimum union. For example, when 2 second regions corresponding to 2 image frames are obtained, and the region coordinates of the first second region m11 in the 1 st image frame are (x 11, y 11), (x 12, y 12), the region coordinates of the second region m12 in the 2 nd image frame are (x 21, y 21), (x 22, y 22), and the minimum coordinate union of the region coordinates of x11< x12< x21< x22, y21< y22< y11< y12 is (x 11, y 21), (x 22, y 12), wherein the minimum coordinate union is the region corresponding to the 1 st image frame and includes the first second region m11, and the minimum coordinate union is the region corresponding to the 2 nd image frame and includes the second region m12. For example, fig. 12 shows an area 111 corresponding to the minimum coordinate union between the second area m11 (x 11, y 11) (x 12, y 12) in the 1 st image frame and the second area m12 in the 2 nd image frame, where the area 111 includes an area 112 corresponding to the area coordinate (x 11, y 11) (x 12, y 12), and an area 113 corresponding to the area coordinate (x 21, y 21), (x 22, y 22).
S503, for each image frame, determining the corresponding region of the summarized region position in the image frame as the important region of the image frame, wherein the important region of each image frame comprises the second region corresponding to the image frame.
In the case that one summarized region position corresponding to all the regions of interest of the at least one image frame is obtained, for each image frame in the at least one image frame, the electronic device may determine a region of the summarized region position in the image frame, and take the determined region as an important region of the image frame, where the important region of the image frame includes a second region of the image frame.
The disclosure further provides a behavior recognition device, and fig. 13 is a schematic structural diagram of the behavior recognition device provided by the embodiment of the disclosure, as shown in fig. 13, the behavior recognition device 1 comprises a recognition unit 11, a recognition unit 11 and a determination unit 12, wherein the recognition unit is used for classifying and recognizing important areas of each image frame in at least one image frame of an obtained image frame sequence to obtain a recognition result corresponding to the at least one image frame, the important areas are determined according to a detection frame area where a target object in the at least one image frame is located, the determination unit 12 is used for determining an activation center of a class activation diagram corresponding to the recognition result of each image frame to obtain at least one activation center of the at least one image frame when the recognition result represents a preset abnormal event, and the recognition unit 11 is also used for recognizing the abnormal target object of the each image frame based on the at least one activation center.
In some embodiments of the present disclosure, the determining unit 12 is further configured to determine a set of activation values of each image frame corresponding to the identification result, where the set of activation values is used to generate a class activation map of each image frame, and each pixel position in the class activation map corresponds to one activation value in the set of activation values one by one, determine a largest activation value in the set of activation values of each image frame, and determine a pixel position corresponding to the largest activation value as the abnormal position, and obtain an activation center of each image frame based on the abnormal position of each image frame, and for the at least one image frame, the at least one activation center is obtained correspondingly.
In some embodiments of the present disclosure, the identifying unit 11 is further configured to determine a position of each activation center in a corresponding one of the image frames, and determine, when the position is located in any one of the detection frame areas in the image frames, a target object corresponding to the any one of the detection frame areas as an abnormal target object in the image frames, and for the at least one activation center, obtain the abnormal target object of each of the image frames correspondingly.
In some embodiments of the present disclosure, the identifying unit 11 is further configured to determine a target activation center according to the at least one activation center, and identify an abnormal target object in each image frame according to the target activation center.
In some embodiments of the present disclosure, the identifying unit 11 is further configured to cluster the at least one activation center by using an unsupervised clustering manner to obtain a target activation center, and identify an abnormal target object in each image frame according to the target activation center.
In some embodiments of the present disclosure, the identifying unit 11 is further configured to cluster the at least one activation center by using an unsupervised clustering manner, to obtain a center cluster with the largest number of activation centers, determine an average value of activation centers included in the center cluster, and determine the average value as the target activation center.
In some embodiments of the present disclosure, the determining unit 12 is further configured to generate a set of initial class activation mapping values of each image frame corresponding to the identification result, perform upsampling processing on the initial class activation mapping values to obtain a set of intermediate activation values, where the set of intermediate activation values is used to generate a class activation map corresponding to an important area of each image frame, and perform fusion processing on the set of intermediate activation values and original pixel values of each image frame to obtain the set of activation values.
In some embodiments of the present disclosure, the behavior recognition apparatus 1 further includes a detection unit 13, where the detection unit 13 is configured to perform, before the recognition result corresponding to at least one image frame is obtained by classifying and recognizing an important area of each image frame in the at least one image frame of the obtained image frame sequence, object detection on a current image frame in the image frame sequence, and determine at least one detection frame area corresponding to at least one object that exists, the determination unit 12 is further configured to determine a center detection frame area in the at least one detection frame area, and determine, in a case where the at least one image frame has been detected, the important area of each image frame in the at least one image frame based on the at least one center detection frame area corresponding to the at least one image frame.
In some embodiments of the present disclosure, the determining unit 12 is further configured to enlarge the at least one detection frame area by a first preset ratio to obtain at least one first area corresponding to the at least one detection frame area, determine a matching result between any two first areas based on an area intersection ratio between the any two first areas in the at least one first area, where the matching result indicates whether the any two first areas are matched, determine a matching number of each first area based on the matching result, and determine a first area with a largest matching number, determine a dense center according to the first area with the largest matching number, and determine a detection frame area corresponding to the dense center as the center detection frame area.
In some embodiments of the present disclosure, the determining unit 12 is further configured to enlarge at least one central detection frame area corresponding to the at least one image frame by a second preset proportion, obtain at least one second area corresponding to the at least one detection frame area, determine a summarized area position according to a position of each second area in the at least one second area in the image frame to which the at least one second area belongs, determine, for each image frame, an area corresponding to the summarized area position in the each image frame as the important area of the each image frame, where the important area of the each image frame includes the second area corresponding to the each image frame.
In some embodiments of the present disclosure, the determining unit 12 is further configured to determine, when there is one first area with the largest matching number, the first area with the largest matching number as the dense center, and determine, when there are two or more first areas with the largest matching number, the first area with the largest area among the two or more first areas with the largest matching number as the dense center.
The embodiment of the present disclosure further provides an electronic device, and fig. 14 is a schematic structural diagram of the electronic device provided by the embodiment of the present disclosure, as shown in fig. 14, where the electronic device 2 includes a memory 21 and a processor 22, and the memory 21 and the processor 22 are connected through a bus 23, the memory 21 is configured to store an executable computer program, and the processor 22 is configured to implement a method provided by the embodiment of the present disclosure when executing the executable computer program stored in the memory 21, for example, a behavior recognition method provided by the embodiment of the present disclosure.
The disclosed embodiments provide a computer readable storage medium storing a computer program for, when executed by a processor 22, implementing a method provided by the disclosed embodiments, for example, a behavior recognition method provided by the disclosed embodiments.
In some embodiments of the present disclosure, the storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, or various devices including one or any combination of the above.
In some embodiments of the present disclosure, a computer program may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment, in the form of software, software modules, scripts, or code.
As an example, a computer program may, but need not, correspond to a file in a file system, may be stored as part of a file that holds other programs or data, e.g., in one or more scripts in a hypertext markup language (HTML, hyper Text Markup Language) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
As an example, a computer program may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.
In summary, by adopting the technical implementation scheme, on one hand, because the important areas of the determined image frames are classified and identified, compared with the identification according to the data enhancement of the whole image of each image frame, the search range during identification is effectively reduced, and meanwhile, the interference during identification is reduced, so that the identification accuracy is improved; on the other hand, the abnormal center (activation center) of each image frame is positioned by determining the abnormal position in the class activation diagram corresponding to the identification result of each image frame, the abnormal target object in the corresponding image frame is identified according to the obtained abnormal center, and the class activation diagram corresponding to the identification result can characterize the target object related to the identification result in the image frame, so that the identified abnormal target object of each image frame is more accurate, and the identification accuracy of the abnormal target object is improved.
The foregoing is merely exemplary embodiments of the present disclosure and is not intended to limit the scope of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and scope of the present disclosure are intended to be included within the scope of the present disclosure.

Claims (12)

1.一种行为识别方法，其特征在于，包括：1. A behavior recognition method, characterized by comprising: 对获得的图像帧序列的至少一个图像帧中，各个图像帧的重要区域分类识别，得到与所述至少一个图像帧对应的识别结果；所述重要区域根据所述至少一个图像帧中目标对象所在的检测框区域确定；In at least one image frame of the acquired image frame sequence, important regions of each image frame are classified and identified to obtain an identification result corresponding to the at least one image frame; the important region is determined according to a detection frame region where a target object is located in the at least one image frame; 在所述识别结果表征存在预设异常事件的情况下，确定每个图像帧的与所述识别结果对应的一组激活值；所述一组激活值用于生成所述每个图像帧的类激活图，且所述类激活图中的每个像素位置与所述一组激活值中的一个激活值一一对应；确定所述每个图像帧的一组激活值中最大的激活值，并将所述最大的激活值所对应的像素位置确定为异常位置；基于所述每个图像帧的所述异常位置，得到所述每个图像帧的激活中心，对于所述至少一个图像帧，对应得到至少一个激活中心；一个激活中心表征对应的一个图像帧的所述类激活图中的异常位置；In the case where the recognition result represents the presence of a preset abnormal event, a group of activation values corresponding to the recognition result of each image frame is determined; the group of activation values is used to generate a class activation map of each image frame, and each pixel position in the class activation map corresponds one-to-one to an activation value in the group of activation values; the maximum activation value in the group of activation values of each image frame is determined, and the pixel position corresponding to the maximum activation value is determined as the abnormal position; based on the abnormal position of each image frame, the activation center of each image frame is obtained, and for the at least one image frame, at least one activation center is obtained; one activation center represents the abnormal position in the class activation map of the corresponding image frame; 基于所述至少一个激活中心，识别出所述各个图像帧的异常目标对象；Based on the at least one activation center, identifying abnormal target objects in each of the image frames; 所述基于所述至少一个激活中心，识别出所述各个图像帧中的异常目标对象，包括：The step of identifying abnormal target objects in each image frame based on the at least one activation center includes: 确定每个激活中心在对应的一个图像帧中的位置；在所述位置位于所述图像帧中的任一检测框区域的情况下，将所述任一检测框区域对应的目标对象，确定为所述图像帧中的异常目标对象，对于所述至少一个激活中心，对应得到所述各个图像帧的异常目标对象。Determine the position of each activation center in a corresponding image frame; when the position is located in any detection frame area in the image frame, determine the target object corresponding to any detection frame area as an abnormal target object in the image frame, and for the at least one activation center, obtain the abnormal target object of each image frame. 2.根据权利要求1所述的方法，其特征在于，所述基于所述至少一个激活中心，识别出所述各个图像帧中的异常目标对象，还包括：2. The method according to claim 1, characterized in that the step of identifying abnormal target objects in each image frame based on the at least one activation center further comprises: 根据所述至少一个激活中心，确定出目标激活中心；Determining a target activation center according to the at least one activation center; 根据所述目标激活中心，识别出所述各个图像帧中的异常目标对象。According to the target activation center, abnormal target objects in each image frame are identified. 3.根据权利要求1所述的方法，其特征在于，所述基于所述至少一个激活中心，识别出所述各个图像帧中的异常目标对象，还包括：3. The method according to claim 1, characterized in that the step of identifying abnormal target objects in each image frame based on the at least one activation center further comprises: 通过无监督聚类方式对所述至少一个激活中心进行聚类，得到目标激活中心；Clustering the at least one activation center by an unsupervised clustering method to obtain a target activation center; 根据所述目标激活中心，识别出所述各个图像帧中的异常目标对象。According to the target activation center, abnormal target objects in each image frame are identified. 4.根据权利要求3所述的方法，其特征在于，所述通过无监督聚类方式对所述至少一个激活中心进行聚类，得到目标激活中心，包括：4. The method according to claim 3, characterized in that clustering the at least one activation center by an unsupervised clustering method to obtain a target activation center comprises: 通过无监督聚类方式对所述至少一个激活中心进行聚类，得到包含的激活中心数量最多的中心簇；Clustering the at least one activation center by an unsupervised clustering method to obtain a center cluster containing the largest number of activation centers; 确定所述中心簇中包含的激活中心的平均值，并将所述平均值确定为所述目标激活中心。An average value of the activation centers included in the center cluster is determined, and the average value is determined as the target activation center. 5.根据权利要求1所述的方法，其特征在于，所述确定每个图像帧的与所述识别结果对应的一组激活值，包括：5. The method according to claim 1, wherein determining a set of activation values corresponding to the recognition result for each image frame comprises: 生成所述每个图像帧的与所述识别结果对应的一组初始类激活映射值；Generating a set of initial class activation map values corresponding to the recognition result for each image frame; 对所述初始类激活映射值进行上采样处理，得到一组中间激活值；所述一组中间激活值用于生成与所述每个图像帧的重要区域对应的类激活图；Upsampling the initial class activation map values to obtain a set of intermediate activation values; the set of intermediate activation values is used to generate a class activation map corresponding to the important area of each image frame; 将所述一组中间激活值与所述每个图像帧的原始像素值进行融合处理，得到所述一组激活值。The set of intermediate activation values is fused with the original pixel value of each image frame to obtain the set of activation values. 6.根据权利要求1-5任一项所述的方法，其特征在于，在所述对获得的图像帧序列的至少一个图像帧中，各个图像帧的重要区域分类识别，得到与所述至少一个图像帧对应的识别结果之前，所述方法还包括：6. The method according to any one of claims 1 to 5, characterized in that before classifying and identifying the important areas of at least one image frame in the obtained image frame sequence and obtaining the identification result corresponding to the at least one image frame, the method further comprises: 对所述图像帧序列中的当前图像帧进行目标检测，确定存在的至少一个目标对象对应的至少一个检测框区域；Performing target detection on a current image frame in the image frame sequence to determine at least one detection frame area corresponding to at least one existing target object; 确定所述至少一个检测框区域中的中心检测框区域；Determine a central detection frame area among the at least one detection frame area; 在已检测所述至少一个图像帧的情况下，基于所述至少一个图像帧对应的至少一个中心检测框区域，确定所述至少一个图像帧中各个图像帧的所述重要区域。In a case where the at least one image frame has been detected, the important area of each image frame in the at least one image frame is determined based on at least one central detection frame area corresponding to the at least one image frame. 7.根据权利要求6所述的方法，其特征在于，所述确定所述至少一个检测框区域中的中心检测框区域，包括：7. The method according to claim 6, wherein determining a central detection frame area in the at least one detection frame area comprises: 将所述至少一个检测框区域扩大第一预设比例，得到与所述至少一个检测框区域对应的至少一个第一区域；Enlarging the at least one detection frame area by a first preset ratio to obtain at least one first area corresponding to the at least one detection frame area; 基于所述至少一个第一区域中任意两个第一区域之间的面积交并比，确定所述任意两个第一区域之间的匹配结果；所述匹配结果表征所述任意两个第一区域是否匹配；Determine a matching result between any two first regions based on an area intersection-and-union ratio between any two first regions in the at least one first region; the matching result indicates whether the any two first regions match; 基于所述匹配结果，确定每个第一区域的匹配次数，并确定匹配次数最大的第一区域；Based on the matching results, determining the number of matches for each first region, and determining the first region with the largest number of matches; 根据所述匹配次数最大的第一区域，确定密集中心，并将所述密集中心对应的检测框区域，确定为所述中心检测框区域。A dense center is determined according to the first area with the largest number of matches, and a detection frame area corresponding to the dense center is determined as the central detection frame area. 8.根据权利要求6所述的方法，其特征在于，所述基于所述至少一个图像帧对应的至少一个中心检测框区域，确定所述至少一个图像帧中各个图像帧的所述重要区域，包括：8. The method according to claim 6, wherein determining the important area of each image frame in the at least one image frame based on at least one central detection frame area corresponding to the at least one image frame comprises: 将所述至少一个图像帧对应的至少一个中心检测框区域扩大第二预设比例，得到与所述至少一个检测框区域对应的至少一个第二区域；Enlarging at least one central detection frame area corresponding to the at least one image frame by a second preset ratio to obtain at least one second area corresponding to the at least one detection frame area; 根据所述至少一个第二区域中各个第二区域在所属的图像帧中的位置，确定出概括区域位置；Determine a generalized region position according to a position of each second region in the image frame to which it belongs; 对于每个图像帧，将所述概括区域位置在所述每个图像帧中对应的区域，确定为所述每个图像帧的所述重要区域；所述每个图像帧的重要区域包含所述每个图像帧对应的第二区域。For each image frame, the area corresponding to the position of the generalized area in each image frame is determined as the important area of each image frame; the important area of each image frame includes the second area corresponding to each image frame. 9.根据权利要求7所述的方法，其特征在于，所述根据所述匹配次数最大的第一区域，确定密集中心，包括：9. The method according to claim 7, characterized in that determining the dense center according to the first area with the largest number of matches comprises: 在存在一个所述匹配次数最大的第一区域的情况下，将所述匹配次数最大的第一区域确定为所述密集中心；In the case where there is a first region with the largest number of matches, determining the first region with the largest number of matches as the dense center; 在存在两个或两个以上的所述匹配次数最大的第一区域的情况下，将所述两个或两个以上的所述匹配次数最大的第一区域中面积最大的第一区域，确定为所述密集中心。In the case that there are two or more first regions with the largest number of matches, the first region with the largest area among the two or more first regions with the largest number of matches is determined as the dense center. 10.一种行为识别装置，其特征在于，包括：10. A behavior recognition device, comprising: 识别单元，用于对获得的图像帧序列的至少一个图像帧中，各个图像帧的重要区域分类识别，得到与所述至少一个图像帧对应的识别结果；所述重要区域根据所述至少一个图像帧中目标对象所在的检测框区域确定；an identification unit, configured to classify and identify important areas of each image frame in at least one image frame of the acquired image frame sequence, and obtain an identification result corresponding to the at least one image frame; the important area is determined according to a detection frame area where a target object is located in the at least one image frame; 确定单元，用于在所述识别结果表征存在预设异常事件的情况下，确定每个图像帧的与所述识别结果对应的一组激活值；所述一组激活值用于生成所述每个图像帧的类激活图，且所述类激活图中的每个像素位置与所述一组激活值中的一个激活值一一对应；确定所述每个图像帧的一组激活值中最大的激活值，并将所述最大的激活值所对应的像素位置确定为异常位置；基于所述每个图像帧的所述异常位置，得到所述每个图像帧的激活中心，对于所述至少一个图像帧，对应得到至少一个激活中心；一个激活中心表征对应的一个图像帧的所述类激活图中的异常位置；A determination unit, configured to determine a set of activation values corresponding to the recognition result of each image frame when the recognition result indicates the presence of a preset abnormal event; the set of activation values is used to generate a class activation map of each image frame, and each pixel position in the class activation map corresponds one-to-one to an activation value in the set of activation values; determine the maximum activation value in the set of activation values of each image frame, and determine the pixel position corresponding to the maximum activation value as the abnormal position; based on the abnormal position of each image frame, obtain the activation center of each image frame, and for at least one image frame, obtain at least one activation center; one activation center represents the abnormal position in the class activation map of the corresponding image frame; 所述识别单元，还用于基于所述至少一个激活中心，识别出所述各个图像帧的异常目标对象；The identification unit is further configured to identify abnormal target objects in each image frame based on the at least one activation center; 所述识别单元，还用于确定每个激活中心在对应的一个图像帧中的位置；在所述位置位于所述图像帧中的任一检测框区域的情况下，将所述任一检测框区域对应的目标对象，确定为所述图像帧中的异常目标对象，对于所述至少一个激活中心，对应得到所述各个图像帧的异常目标对象。The recognition unit is also used to determine the position of each activation center in a corresponding image frame; when the position is located in any detection frame area in the image frame, the target object corresponding to any detection frame area is determined as an abnormal target object in the image frame, and for the at least one activation center, the abnormal target object of each image frame is obtained. 11.一种电子设备，其特征在于，包括：11. An electronic device, comprising: 存储器，用于存储可执行计算机程序；a memory for storing executable computer programs; 处理器，用于执行所述存储器中存储的可执行计算机程序时，实现权利要求1至9任一项所述的方法。A processor, configured to implement the method according to any one of claims 1 to 9 when executing the executable computer program stored in the memory. 12.一种计算机可读存储介质，其特征在于，存储有计算机程序，用于引起处理器执行时，实现权利要求1至9任一项所述的方法。12. A computer-readable storage medium, characterized in that it stores a computer program for causing a processor to execute the method according to any one of claims 1 to 9.
CN202210239612.0A 2022-03-11 2022-03-11 Behavior recognition method, device, electronic device and computer-readable storage medium Active CN114677754B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210239612.0A CN114677754B (en) 2022-03-11 2022-03-11 Behavior recognition method, device, electronic device and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210239612.0A CN114677754B (en) 2022-03-11 2022-03-11 Behavior recognition method, device, electronic device and computer-readable storage medium

Publications (2)

Publication Number Publication Date
CN114677754A CN114677754A (en) 2022-06-28
CN114677754B true CN114677754B (en) 2025-04-15

Family

ID=82071674

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210239612.0A Active CN114677754B (en) 2022-03-11 2022-03-11 Behavior recognition method, device, electronic device and computer-readable storage medium

Country Status (1)

Country Link
CN (1) CN114677754B (en)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114896307B (en) * 2022-06-30 2022-09-27 北京航空航天大学杭州创新研究院 Time series data enhancement method, device and electronic device
US12082969B1 (en) 2023-02-22 2024-09-10 BrightHeart SAS Systems and methods for improving detection of fetal congenital heart defects
US12343197B2 (en) 2023-02-22 2025-07-01 BrightHeart SAS Systems and methods for improving detection of fetal congenital heart defects
US12148162B2 (en) 2023-02-22 2024-11-19 BrightHeart SAS Systems and methods for detecting cardiovascular anomalies using spatiotemporal neural networks
US12493962B2 (en) 2023-02-22 2025-12-09 BrightHeart SAS Systems and methods for detecting cardiovascular anomalies using spatiotemporal neural networks
CN116824626A (en) * 2023-05-31 2023-09-29 广东省科学院动物研究所 A method for artificial intelligence identification of animal abnormality
US11861838B1 (en) * 2023-06-07 2024-01-02 BrightHeart SAS Systems and methods for system agnostic automated detection of cardiovascular anomalies and/or other features
CN116524414B (en) * 2023-06-26 2023-10-17 广州英码信息科技有限公司 Method, system and computer readable storage medium for identifying racking behavior
CN119131368B (en) * 2024-11-08 2025-04-18 北京天融信网络安全技术有限公司 Image key feature area detection method, electronic device and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095257A (en) * 2021-04-20 2021-07-09 上海商汤智能科技有限公司 Abnormal behavior detection method, device, equipment and storage medium
CN113205512A (en) * 2021-05-26 2021-08-03 北京市商汤科技开发有限公司 Image anomaly detection method, device, equipment and computer readable storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2674311C (en) * 2007-02-08 2015-12-29 Behavioral Recognition Systems, Inc. Behavioral recognition system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095257A (en) * 2021-04-20 2021-07-09 上海商汤智能科技有限公司 Abnormal behavior detection method, device, equipment and storage medium
CN113205512A (en) * 2021-05-26 2021-08-03 北京市商汤科技开发有限公司 Image anomaly detection method, device, equipment and computer readable storage medium

Also Published As

Publication number Publication date
CN114677754A (en) 2022-06-28

Similar Documents

Publication Publication Date Title
CN114677754A (en) Behavior recognition method, device, electronic device, and computer-readable storage medium
CN111738231B (en) Target object detection method and device, computer equipment and storage medium
CN107862270B (en) Face classifier training method, face detection method and device and electronic equipment
CN114169425B (en) Method and device for training target tracking model and target tracking
CN113936256A (en) Image target detection method, device, equipment and storage medium
CN108256404B (en) Pedestrian detection method and device
CN108875708A (en) Video-based behavior analysis method, device, equipment, system and storage medium
CN112562159B (en) Access control method and device, computer equipment and storage medium
CN116129522A (en) Behavior recognition method and device, electronic equipment and computer storage medium
CN110390226B (en) Crowd event identification method and device, electronic equipment and system
CN108345892A (en) A kind of detection method, device, equipment and the storage medium of stereo-picture conspicuousness
CN111652181B (en) Target tracking method, device and electronic equipment
CN113065379B (en) Image detection method and device integrating image quality and electronic equipment
WO2024060558A1 (en) Feasible region prediction method and apparatus, and system and storage medium
CN114140841A (en) Point cloud data processing method, neural network training method and related equipment
CN114743264B (en) Shooting behavior detection method, device, equipment and storage medium
WO2022217551A1 (en) Target detection method and apparatus
CN114972182B (en) Object detection method and device
CN107563299B (en) A pedestrian detection method using ReCNN to fuse contextual information
CN120182921B (en) A security monitoring method, system, device and medium based on edge computing
CN116311535B (en) Risky behavior analysis method and system based on human interaction detection
CN117994851A (en) A method, device and equipment for detecting falls of the elderly based on multi-task learning
CN118397538A (en) Intelligent blind guiding method, device and system
Xiao et al. Vrstnn: Visual-relational spatio-temporal neural network for early hazardous event detection in automated driving systems
CN111062311B (en) Pedestrian gesture recognition and interaction method based on depth-level separable convolution network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant